w2w not successful in incorporating LCZ into geo files #132

GLlorin · 2024-09-15T08:48:30Z

Describe your issue

Hello!

I am trying to use the global LCZ map produced by Demuzere et al. (2022), cut to the Philippines (~5-25N, 115-130E), in simulations over Metro Manila to test the performance of different urban canopy models. However, I am running into trouble at the first step of using the w2w python utility to incorporate the TIFF file into the prepared geo_em.d0X.nc files.

Following its recommended use, I run this line in the directory where I have my LCZ file (PAR.tif) and my two domain geo files:

> w2w ./ PAR.tif geo_em.d02.nc

It runs well until the check and cleaning starts, giving the following messages and ending in an error:

--> Start sanity check and clean-up ...
Check 1: Urban class removed from geo_em.d02_NoUrban.nc? OK
Check 2: LCZ Urban extent present in geo_em.d02_LCZ_extent.nc?
WARNING: LCZ-based urban extent missing
Check 3: Urban LCZ classes exists in geo_em.d02_LCZ_params.nc? OK: LCZ Classes ([]) present
Check 4: FRC_URB2D present in geo_em.d02_LCZ_params.nc? OK: FRC_URB2D values range between 0.00 and 0.00
Check 5: URB_PARAMS matrix present in file geo_em.d02_LCZ_params.nc? OK
Check 6: Do URB_PARAM variable values follow expected range in geo_em.d02_LCZ_params.nc?

OK for LP_URB2D

OK for MH_URB2D

OK for STDH_URB2D

OK for HGT_URB2D

OK for LB_URB2D

OK for LF_URB2D
Traceback (most recent call last):
File "/home/gagena/.conda/envs/w2w/bin/w2w", line 8, in
sys.exit(main())
^^^^^^
File "/home/gagena/.conda/envs/w2w/lib/python3.11/site-packages/w2w/w2w.py", line 207, in main
checks_and_cleaning(
File "/home/gagena/.conda/envs/w2w/lib/python3.11/site-packages/w2w/w2w.py", line 1653, in checks_and_cleaning
if np.nanmax(np.abs((100 - hi_sum).values)) > 0.1:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/gagena/.conda/envs/w2w/lib/python3.11/site-packages/numpy/lib/nanfunctions.py", line 476, in nanmax
res = np.fmax.reduce(a, axis=axis, out=out, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: zero-size array to reduction operation fmax which has no identity

I know this error was already brought up in Issue #38, but when I tried to follow the recommended course of action from the thread by executing:

pip install --upgrade --force-reinstall git+https://github.com/matthiasdemuzere/w2w.git@add_tests

I run into this error:

Collecting git+https://github.com/matthiasdemuzere/w2w.git@add_tests
Cloning https://github.com/matthiasdemuzere/w2w.git (to revision add_tests) to /tmp/pip-req-build-tv1ghnl6
Running command git clone --filter=blob:none --quiet https://github.com/matthiasdemuzere/w2w.git /tmp/pip-req-build-tv1ghnl6
WARNING: Did not find branch or tag 'add_tests', assuming revision or ref.
Running command git checkout -q add_tests
error: pathspec 'add_tests' did not match any file(s) known to git
error: subprocess-exited-with-error

× git checkout -q add_tests did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× git checkout -q add_tests did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

I have uploaded my geo_em.d0X.nc files and the TIFF file I have cropped from the global map by Demuzere et al. (2022) in this GDrive. I've also included the produced geo_em.d02_LCZ_params.nc file which confirms that the LU_INDEX was not updated. Please advise me on my next steps.

Thank you very much!

w2w --version

w2w 0.5.0

nc-config --all

This netCDF 4.8.1 has been built with the following features:

  --cc            -> /home/spack/opt/spack/linux-rocky9-broadwell/gcc-11.4.1/intel-oneapi-compilers-2023.2.1-mzcqsqcfemyks4cypbbrruzwz5kd5vy4/compiler/2023.2.1/linux/bin/intel64/icc
  --cflags        -> -I/home/spack/opt/spack/linux-rocky9-broadwell/intel-2021.10.0/netcdf-c-4.8.1-amx4rlwhjr54v2uhfxk3i7xvvy4prqkq/include
  --libs          -> -L/home/spack/opt/spack/linux-rocky9-broadwell/intel-2021.10.0/netcdf-c-4.8.1-amx4rlwhjr54v2uhfxk3i7xvvy4prqkq/lib -lnetcdf
  --static        -> -lhdf5_hl -lhdf5 -lm

  --has-c++       -> no
  --cxx           ->

  --has-c++4      -> no
  --cxx4          ->

  --has-fortran   -> yes
  --fc            -> /home/spack/opt/spack/linux-rocky9-broadwell/gcc-11.4.1/intel-oneapi-compilers-2023.2.1-mzcqsqcfemyks4cypbbrruzwz5kd5vy4/compiler/2023.2.1/linux/bin/intel64/ifort
  --fflags        -> -I/home/spack/opt/spack/linux-rocky9-broadwell/intel-2021.10.0/netcdf-fortran-4.5.4-akdrb6wpoo4lgieevwzr7dgvp7j5gbs2/include -I/home/spack/opt/spack/linux-rocky9-broadwell/intel-2021.10.0/netcdf-fortran-4.5.4-akdrb6wpoo4lgieevwzr7dgvp7j5gbs2/include
  --flibs         -> -L/home/spack/opt/spack/linux-rocky9-broadwell/intel-2021.10.0/netcdf-fortran-4.5.4-akdrb6wpoo4lgieevwzr7dgvp7j5gbs2/lib -lnetcdff -lnetcdf -lnetcdf -lm
  --has-f90       ->
  --has-f03       -> yes

  --has-dap       -> no
  --has-dap2      -> no
  --has-dap4      -> no
  --has-nc2       -> yes
  --has-nc4       -> yes
  --has-hdf5      -> yes
  --has-hdf4      -> no
  --has-logging   -> no
  --has-pnetcdf   -> no
  --has-szlib     -> no
  --has-cdf5      -> yes
  --has-parallel4 -> no
  --has-parallel  -> no
  --has-nczarr    -> yes

  --prefix        -> /home/spack/opt/spack/linux-rocky9-broadwell/intel-2021.10.0/netcdf-c-4.8.1-amx4rlwhjr54v2uhfxk3i7xvvy4prqkq
  --includedir    -> /home/spack/opt/spack/linux-rocky9-broadwell/intel-2021.10.0/netcdf-c-4.8.1-amx4rlwhjr54v2uhfxk3i7xvvy4prqkq/include
  --libdir        -> /home/spack/opt/spack/linux-rocky9-broadwell/intel-2021.10.0/netcdf-c-4.8.1-amx4rlwhjr54v2uhfxk3i7xvvy4prqkq/lib
  --version       -> netCDF 4.8.1

Installed Packages

affine==2.4.0
attrs==24.2.0
certifi==2024.8.30
cftime==1.6.4
click==8.1.7
click-plugins==1.1.1
cligj==0.7.2
ecmwflibs==0.6.3
findlibs==0.0.5
h5netcdf==1.3.0
h5py==3.11.0
netCDF4==1.7.1.post2
numpy @ file:///home/conda/feedstock_root/build_artifacts/numpy_1707225376651/work/dist/numpy-1.26.4-cp311-cp311-linux_x86_64.whl#sha256=d08e1c9e5833ae7780563812aa73e2497db1ee3bd5510d3becb8aa18aa2d0c7c
packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1718189413536/work
pandas==1.5.3
pyparsing==3.1.4
pyproj==3.6.1
python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1709299778482/work
pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1706886791323/work
rasterio==1.3.11
rioxarray==0.17.0
scipy==1.14.1
six @ file:///home/conda/feedstock_root/build_artifacts/six_1620240208055/work
snuggs==1.4.7
tqdm==4.66.5
w2w==0.5.0
xarray @ file:///home/conda/feedstock_root/build_artifacts/xarray_1674166302925/work

Traceback

No response

The text was updated successfully, but these errors were encountered:

jkittner · 2024-09-15T13:02:01Z

I am not sure why you're encountering the error, but your re-installation command does not work because the branch does not exist anymore.

Try this instead for the latest version on main

pip install  --upgrade --force-reinstall git+https://github.com/matthiasdemuzere/w2w.git

or a version where we've made a few changes (which is not merged)

pip install  --upgrade --force-reinstall git+https://github.com/matthiasdemuzere/w2w.git@add_wrf_version

If you're error still persists, you will probably have to provide your geo_em files, for someone to have a look.

GLlorin · 2024-09-16T05:19:48Z

Thank you so much for replying so quickly. It took me some time to apply both suggestions but essentially, while both suggested re-installation commands worked, I still encounter the same error message I posted about when the checks and cleaning are being made. For some reason, the LCZ data is not being incorporated into the new geo_em.d0X.nc files.

It may not have been seen but I do provide all files I have in this drive as the TIFF file was quite large: https://drive.google.com/drive/u/1/folders/1J6sm3QQZaSg0JaYaPG4mezDLTx_k7eA5

Please advise me on further courses of action I can take, thank you!

matthiasdemuzere · 2024-09-18T10:36:30Z

Hi @GLlorin,

Thanks for reporting this.

The first thing I noticed is the large size of your PAR.tif file, that seems to cover an area that is many times larger than your target WRF d02 domain. When using that file, my local machine actually did not have anough memory :)

So, I just cropped a smaller area out of the global LCZ map, using the code below (you can get v3 of global LCZ map here):

import rioxarray as rxr
import xarray as xr

fn_lcz = "/mnt/SSD2TB/data/ZENODO/LCZ_GLOBAL/v3/lcz_filter_v3.tif"
fn_geo = "/mnt/SSD2TB/data/w2w-debug/132_Manilla/geo_em.d02.nc"

lcz = rxr.open_rasterio(fn_lcz)
ds = xr.open_dataset(fn_geo)

dxy = 0.5
xmin= float(ds['XLONG_M'].min()) - dxy
xmax = float(ds['XLONG_M'].max()) + dxy
ymin = float(ds['XLAT_M'].min()) - dxy
ymax = float(ds['XLAT_M'].max()) + dxy
print(xmin, xmax, ymin, ymax)

# Crop LCZ map
lcz_sel = lcz.sel(x=slice(xmin, xmax), y=slice(ymax, ymin))
print(lcz_sel)

# Store to drive
fn_lcz_sel = "/mnt/SSD2TB/data/w2w-debug/132_Manilla/lcz.tif"
lcz_sel.rio.to_raster(fn_lcz_sel)

When I execute w2w with this smaller file, I can actually reproduce your error:

w2w . lcz.tif geo_em.d02.nc 'v4.3'

--> Set data, arguments and files 
> Using layer 0 of the LCZ GeoTIFF. Can be changed with -l (--lcz-band)
--> Check LCZ integrity, in terms of class labels, projection and extent
> LCZ labels as expected (1 to 17)
> LCZ provided as WGS84 (EPSG:4326)
> LCZ domain is covering WRF domain
--> Replace WRF urban LC with surrounding natural LC
Looping through urban grid pixels: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 182/182 [00:00<00:00, 194.71it/s]
--> Create LCZ-based geo_em file
> Processing LP_URB2D ...
> Processing MH_URB2D ...
> Processing STDH_URB2D ...
> Processing HGT_URB2D ...
> Processing LB_URB2D ...
> Processing LF_URB2D ...
> Processing HI_URB2D ...
--> Create LCZ-based urban extent geo_em file (excluding other LCZ-based info)
--> Expanding land categories of parent domain(s) to 41

--> Start sanity check and clean-up ...
> Check 1: Urban class removed from geo_em.d02_NoUrban.nc? OK 
> Check 2: LCZ Urban extent present in geo_em.d02_LCZ_extent.nc?
 WARNING: LCZ-based urban extent missing 
> Check 3: Urban LCZ classes exists in geo_em.d02_LCZ_params.nc? OK: LCZ Classes ([]) present 
> Check 4: FRC_URB2D present in geo_em.d02_LCZ_params.nc? OK: FRC_URB2D values range between 0.00 and 0.00 
> Check 5: URB_PARAMS matrix present in file geo_em.d02_LCZ_params.nc? OK 
> Check 6: Do URB_PARAM variable values follow expected range in geo_em.d02_LCZ_params.nc?
   + OK for LP_URB2D 
   + OK for MH_URB2D 
   + OK for STDH_URB2D 
   + OK for HGT_URB2D 
   + OK for LB_URB2D 
   + OK for LF_URB2D 
Traceback (most recent call last):
  File "/home/matthias/pyvens/w2w/bin/w2w", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/matthias/pyvens/w2w/lib/python3.11/site-packages/w2w/w2w.py", line 234, in main
    checks_and_cleaning(
  File "/home/matthias/pyvens/w2w/lib/python3.11/site-packages/w2w/w2w.py", line 1728, in checks_and_cleaning
    if np.nanmax(np.abs((100 - hi_sum).values)) > 0.1:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/matthias/pyvens/w2w/lib/python3.11/site-packages/numpy/lib/_nanfunctions_impl.py", line 495, in nanmax
    res = np.fmax.reduce(a, axis=axis, out=out, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: zero-size array to reduction operation fmax which has no identity

I can also confirm to get this error in both these versions:

pip install  --upgrade --force-reinstall git+https://github.com/matthiasdemuzere/w2w.git

pip install  --upgrade --force-reinstall git+https://github.com/matthiasdemuzere/w2w.git@add_wrf_version

Based on the output, it seems to go wrong when introducing the LCZ labels into the LU_INDEX, because of:

WARNING: LCZ-based urban extent missing

At the same time, Check 3 seems also misleading: > Check 3: Urban LCZ classes exists in geo_em.d02_LCZ_params.nc? OK: LCZ Classes ([]) present. The LCZ class array is empty, so this check should not be ok.

I'll try to look into this further ...

Ropshin · 2025-01-08T08:27:53Z

Dear @matthiasdemuzere ,

I am using the global LCZ map produced by Demuzere et al. (2022) and encountered the same problem as Mr.GLlorin when research Hong Kong. Additionally, I followed the instructions in #38 and reinstalled the matthiasdemuzere version. However, the same problem persists.

I was wondering if you have found a solution to this issue? After inspection, I can confirm that the LCZ import failed, as the resulting file still contains 21 classes instead of 41 classes. I hope this information might be helpful for your research.

Thank you so much for your contributions to this project. Wish you have a great day! My error message is as follows:

--> Set data, arguments and files

Using layer 0 of the LCZ GeoTIFF. Can be changed with -l (--lcz-band)
--> Check LCZ integrity, in terms of class labels, projection and extent
LCZ labels as expected (1 to 17)
LCZ provided as WGS84 (EPSG:4326)
LCZ domain is covering WRF domain
--> Replace WRF urban LC with surrounding natural LC
Looping through urban grid pixels: 100%|███| 2809/2809 [00:07<00:00, 399.22it/s]
--> Create LCZ-based geo_em file
Processing LP_URB2D ...
Processing MH_URB2D ...
Processing STDH_URB2D ...
Processing HGT_URB2D ...
Processing LB_URB2D ...
Processing LF_URB2D ...
Processing HI_URB2D ...
--> Create LCZ-based urban extent geo_em file (excluding other LCZ-based info)
--> Expanding land categories of parent domain(s) to 41

--> Start sanity check and clean-up ...

Check 1: Urban class removed from geo_em.d03_NoUrban.nc? OK
Check 2: LCZ Urban extent present in geo_em.d03_LCZ_extent.nc?
WARNING: LCZ-based urban extent missing
Check 3: Urban LCZ classes exists in geo_em.d03_LCZ_params.nc? OK: LCZ Classes ([]) present
Check 4: FRC_URB2D present in geo_em.d03_LCZ_params.nc? OK: FRC_URB2D values range between 0.00 and 0.00
Check 5: URB_PARAMS matrix present in file geo_em.d03_LCZ_params.nc? OK
Check 6: Do URB_PARAM variable values follow expected range in geo_em.d03_LCZ_params.nc?

OK for LP_URB2D
OK for MH_URB2D
OK for STDH_URB2D
OK for HGT_URB2D
OK for LB_URB2D
OK for LF_URB2D
Traceback (most recent call last):
File "/home/ropshin/miniconda3/envs/w2w/bin/w2w", line 8, in
sys.exit(main())
^^^^^^
File "/home/ropshin/miniconda3/envs/w2w/lib/python3.11/site-packages/w2w/w2w.py", line 206, in main
checks_and_cleaning(
File "/home/ropshin/miniconda3/envs/w2w/lib/python3.11/site-packages/w2w/w2w.py", line 1621, in checks_and_cleaning
if np.nanmax(np.abs((100 - hi_sum).values)) > 0.1:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ropshin/miniconda3/envs/w2w/lib/python3.11/site-packages/numpy/lib/nanfunctions.py", line 476, in nanmax
res = np.fmax.reduce(a, axis=axis, out=out, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: zero-size array to reduction operation fmax which has no identity

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

w2w not successful in incorporating LCZ into geo files #132

w2w not successful in incorporating LCZ into geo files #132

GLlorin commented Sep 15, 2024 •

edited

Loading

jkittner commented Sep 15, 2024

GLlorin commented Sep 16, 2024

matthiasdemuzere commented Sep 18, 2024

Ropshin commented Jan 8, 2025

w2w not successful in incorporating LCZ into geo files #132

w2w not successful in incorporating LCZ into geo files #132

Comments

GLlorin commented Sep 15, 2024 • edited Loading

Describe your issue

w2w --version

nc-config --all

Installed Packages

Traceback

jkittner commented Sep 15, 2024

GLlorin commented Sep 16, 2024

matthiasdemuzere commented Sep 18, 2024

Ropshin commented Jan 8, 2025

GLlorin commented Sep 15, 2024 •

edited

Loading