Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changes to support SpanInstabase in Labor #69

Merged
merged 2 commits into from
Feb 6, 2021
Merged

Conversation

jrising
Copy link
Member

@jrising jrising commented Feb 4, 2021

This PR has three related changes:

  • Adds a buffer for asynchronous results to PassCallApplication. This is needed to allow SpanInstabase to be embedded within the FractionSum calculation.
  • Creating a RecursiveCalculation ABC, to better ensure passing of messages to contained subcalculations.
  • Creating a Reword calculation, used to label the operations that need to be kept for labor.

These are in response to https://gitlab.com/ClimateImpactLab/Impacts/impact-calculations/-/issues/141.
Strictly speaking, only the first item is necessary. However, only that change results in three different outputs named rebased, rebased2, and rebased22. We can clean up the calculation structure used in labor and keep only the results needed by allowing calculation results to be renamed in situ. That requires the Reword calculation, and it is at this point where I noticed that a lot of similar recursive calculations don't properly pass messages along, and so I made Reword and several other calculations derive from the new RecursiveCalculation for ease.

@jrising jrising requested a review from brews February 4, 2021 16:16
@jrising
Copy link
Member Author

jrising commented Feb 4, 2021

My test for labor works, but...

@brews I've lost the ability to run pytest in my setup (as of a couple weeks ago). I get the following error:

tests/test_system/conftest.py:7: in <module>
    import adaptation.csvvfile
E   ModuleNotFoundError: No module named 'adaptation'

I've tried to run pytest in different ways, but even if I get that line running, it messes up how other imports are interpreted. Any thoughts?

@brews
Copy link
Member

brews commented Feb 4, 2021

@jrising This looks like impact-calculations might be cloned but not installed in the python environment. You can do pip install -e . from the impact-calculation root directory. If you're working in conda or other virtual environment, you need to have that activated first so it installs in the correct environment.

Installing impact-calculations as a pip package is helpful because then people can import and run it from python or bash without worrying about paths (e.g. they can run it from a directory within their repository of sector-specific configs). ... You can also have people run conda list or pip list and the package version shows exactly what commit they're working on.

@jrising
Copy link
Member Author

jrising commented Feb 4, 2021

@brews Okay, that got me working again. Thanks. But my test failed with a segfault on the ag system test...

collected 209 items                                                                                                                                  

tests/test_cli.py ....EEE                                                                                                                      [  3%]
tests/test_covariates.py ..                                                                                                                    [  4%]
tests/test_csvvfile.py .                                                                                                                       [  4%]
tests/test_forecastreader.py .                                                                                                                 [  5%]
tests/test_linextrap.py .                                                                                                                      [  5%]
tests/test_pvalses.py .                                                                                                                        [  6%]
tests/test_season_conditionals.py ...                                                                                                          [  7%]
tests/test_seasonal_climategen.py ..xxx.                                                                                                       [ 10%]
tests/test_system_legacy.py ...ssss                                                                                                            [ 13%]
tests/test_weather.py .......................                                                                                                  [ 24%]
tests/test_weatherreader.py .                                                                                                                  [ 25%]
tests/test_datastore/test_irregions.py ........                                                                                                [ 29%]
tests/test_interpret/test_configs.py ....                                                                                                      [ 31%]
tests/test_interpret/test_variables.py .                                                                                                       [ 31%]
tests/test_system/test_montecarlo_labor.py .............................                                                                       [ 45%]
tests/test_system/test_montepoint_agcorn.py Fatal Python error: Segmentation fault

Can you run this on the cloud?

@brews
Copy link
Member

brews commented Feb 4, 2021

@jrising Yeah, I can run the tests. I'll post back when they're done.

Bummer that the CLI test also fail. From your virtual environment can you run imperics --help like just from a bash session? (This might be an issue for another time)

Copy link
Member

@brews brews left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool!

I have some suggestions but I don't see anything breaking!

openest/generate/calculation.py Outdated Show resolved Hide resolved
openest/generate/calculation.py Outdated Show resolved Hide resolved
openest/generate/calculation.py Outdated Show resolved Hide resolved
@jrising jrising merged commit 751efee into master Feb 6, 2021
@jrising jrising deleted the passcallbuffer branch February 6, 2021 04:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants