Error while extracting energy impact with full uncertainty

[ruixue-li]

Hi @jrising and @brews ,

I'm trying to extract some energy impact with full uncertainty with this config, and this command:

python -u quantiles.py /home/liruixue/repos/energy-code-release-2020/projection_inputs/configs/GMFD/TINV_clim/break2_Exclude/semi-parametric/Extraction_Configs/sacagawea/impactpc//full/aggregated/median/energy-extract-impactpc-aggregated-median_OTHERIND_electricity.yml --only-iam=low --suffix=_test_impactpc_median_fulluncertainty_low_fulladapt-aggregated FD_FGLS_inter_OTHERIND_electricity_TINV_clim-aggregated -FD_FGLS_inter_OTHERIND_electricity_TINV_clim-histclim-aggregated

But I'm getting the following error at the end:

...
/shares/gcp/outputs/energy_pixel_interaction/impacts-blueghost/median_OTHERIND_electricity_TINV_clim_GMFD_dm/median/rcp85/MIROC5/high/SSP2 not low
/shares/gcp/outputs/energy_pixel_interaction/impacts-blueghost/median_OTHERIND_electricity_TINV_clim_GMFD_dm/median/rcp85/CanESM2/low/SSP3
/shares/gcp/outputs/energy_pixel_interaction/impacts-blueghost/median_OTHERIND_electricity_TINV_clim_GMFD_dm/median/rcp85/CanESM2/high/SSP4 not low
/shares/gcp/outputs/energy_pixel_interaction/impacts-blueghost/median_OTHERIND_electricity_TINV_clim_GMFD_dm/median/rcp85/CanESM2/high/SSP5 not low
/shares/gcp/outputs/energy_pixel_interaction/impacts-blueghost/median_OTHERIND_electricity_TINV_clim_GMFD_dm/median/rcp85/CanESM2/high/SSP3 not low
/shares/gcp/outputs/energy_pixel_interaction/impacts-blueghost/median_OTHERIND_electricity_TINV_clim_GMFD_dm/median/rcp85/CanESM2/high/SSP2 not low
Observations: 15546
Creating file: ('SSP4', 'rcp85')
('SSP4', 'rcp85') is not in delta method output. Skipping model specification...
Creating file: ('SSP4', 'rcp45')
('SSP4', 'rcp45') is not in delta method output. Skipping model specification...
Creating file: ('SSP5', 'rcp85')
('SSP5', 'rcp85') is not in delta method output. Skipping model specification...
Creating file: ('SSP1', 'rcp45')
('SSP1', 'rcp45') is not in delta method output. Skipping model specification...
Creating file: ('SSP2', 'rcp85')
('SSP2', 'rcp85') is not in delta method output. Skipping model specification...
Creating file: ('SSP2', 'rcp45')
('SSP2', 'rcp45') is not in delta method output. Skipping model specification...
Creating file: ('SSP3', 'rcp85')
Outputing row: ('all', 1981)
/home/liruixue/miniconda3/envs/risingverse-py27/lib/python2.7/site-packages/scipy/stats/_distn_infrastructure.py:1913: RuntimeWarning: invalid value encountered in greater
  cond0 = self._argcheck(*args) & (scale > 0) & (loc == loc)
/home/liruixue/miniconda3/envs/risingverse-py27/lib/python2.7/site-packages/scipy/stats/_distn_infrastructure.py:1746: RuntimeWarning: invalid value encountered in greater
  cond0 = self._argcheck(*args) & (scale > 0)
/home/liruixue/miniconda3/envs/risingverse-py27/lib/python2.7/site-packages/scipy/stats/_distn_infrastructure.py:877: RuntimeWarning: invalid value encountered in greater
  return (self.a < x) & (x < self.b)
/home/liruixue/miniconda3/envs/risingverse-py27/lib/python2.7/site-packages/scipy/stats/_distn_infrastructure.py:877: RuntimeWarning: invalid value encountered in less
  return (self.a < x) & (x < self.b)
/home/liruixue/miniconda3/envs/risingverse-py27/lib/python2.7/site-packages/scipy/stats/_distn_infrastructure.py:1747: RuntimeWarning: invalid value encountered in greater
  cond1 = self._open_support_mask(x) & (scale > 0)
/home/liruixue/miniconda3/envs/risingverse-py27/lib/python2.7/site-packages/scipy/stats/_distn_infrastructure.py:1748: RuntimeWarning: invalid value encountered in greater_equal
  cond2 = (x >= self.b) & cond0
Traceback (most recent call last):
  File "quantiles.py", line 114, in <module>
    writer.writerow(myrowstuff + list(distribution.inverse(encoded_evalqvals)))
  File "/home/liruixue/repos/prospectus-tools/gcp/extract/lib/weights_vcv.py", line 54, in inverse
    roots.append(brentq(func, left, right))
  File "/home/liruixue/miniconda3/envs/risingverse-py27/lib/python2.7/site-packages/scipy/optimize/zeros.py", line 754, in brentq
    r = _zeros._brentq(f, a, b, xtol, rtol, maxiter, args, full_output, disp)
RuntimeError: Failed to converge after 100 iterations.

This error does not occur if I limit to one or a set of regions, or if I extract the non-aggregated files.

I'm wondering if any of you have any idea what's going on? I could work around this and get what I need by specifying regions, but I'd just like to make sure that I'm not doing anything that I shouldn't be doing, or if there's an obvious and easy fix for it.

Thank you very much!

[jrising]

It appears that you're getting a bunch of NAs for some of the regions. Checking into it.

[jrising]

It looks like there are missing values in the following regions:
TWN.4
TWN.3
TWN.1
TWN.2
NZL.10.42
HMD
CHN.21.226
CHN.21
ATF
ZAF.9
SGS
CL-
BRA.8.837
BRA.8.836
AUS.1
AUS.3
ESP.6.27.191
ESP.6.27.192
SP-
IOT
BVT
ATA

These are generally very small regions. I have a fix where the code just warns you that there are missing values and so it has to skip the region, but it might be worthwhile to find out why there are these values.

[ruixue-li]

Hi @jrising ,

I see, thanks! If it's not too much trouble, do you mind putting in that fix first, so that it doesn't hold up other stuff while I look into why those regions are missing?

[jrising]

Sure. I've just committed it to master. Here are the changes:
a22539b