Cannot generate composites on bigger extents using MTG FCI Level-1c files

[AlejandroGorgues]

Describe the bug
I am using the fci_l1c_nc reader to generate the composites.
When the extent used is big or the full extent of the scene is used, two things could happen:

• A Segmentation fault (core dumped) message
• A NetCDF error:

NetCDF: HDF error
Traceback (most recent call last):
  File "/usr/src/app/test/MTGSatpy.py", line 169, in <module>
    satpy.run_script()
  File "/usr/src/app/test/MTGSatpy.py", line 160, in run_script
    self.get_composites()
  File "/usr/src/app/test/MTGSatpy.py", line 115, in get_composites
    local_scene.save_datasets(writer='geotiff', filename=str(file_no_projected), fill_value=0)
  File "/usr/local/lib/python3.12/dist-packages/satpy/scene.py", line 1296, in save_datasets
    return writer.save_datasets(dataarrays, compute=compute, **save_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/satpy/writers/__init__.py", line 756, in save_datasets
    return compute_writer_results([results])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/satpy/writers/__init__.py", line 594, in compute_writer_results
    da.compute(delayeds)
  File "/usr/local/lib/python3.12/dist-packages/dask/base.py", line 660, in compute
    results = schedule(dsk, keys, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/netCDF4/_netCDF4.pyx", line 5079, in netCDF4._netCDF4.Variable.__getitem__
  File "src/netCDF4/_netCDF4.pyx", line 6051, in netCDF4._netCDF4.Variable._get
  File "src/netCDF4/_netCDF4.pyx", line 2164, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: HDF error

To Reproduce

import hdf5plugin
from satpy.scene import Scene
from satpy import find_files_and_readers
# data_input corresponds to the path where the .nc resides
files = find_files_and_readers(base_dir=data_input, reader='fci_l1c_nc')
# output file path
file_no_projected = Path(data_output, f'NO_PROJECTED_{composite["COMPOSITE_NAME"]}.tif')
scn = Scene(filenames=files, reader='fci_l1c_nc')
# composite is a dictionary that contains the key SATPY_NAME associated to the name in the satpy library (ir_105, vis_06 or colorized_ir_clouds)
scn.load([composite["SATPY_NAME"]], upper_right_corner="NE")
""" 
model contains general data that includes the extend that surrounds the sea:
"EXTENT_EUROPE":{
        "XMAX": 20.0,
        "XMIN": -45.0,
        "YMAX": 55.0,
        "YMIN": 20.0
    },
"""
local_scene = scn.crop(ll_bbox=(model["EXTENT_EUROPE"]["XMIN"], model["EXTENT_EUROPE"]["YMIN"], \
                    model["EXTENT_EUROPE"]["XMAX"], model["EXTENT_EUROPE"]["YMAX"]))
local_scene.save_datasets(writer='geotiff', filename=str(file_no_projected))

Expected behavior
The geotiff file associated to the product requested

Actual results
Included above.
Also on both results mentioned previously, I executed the code with debug_on and debug_off, showing the same messages:

[DEBUG: 2025-01-07 15:18:19 : satpy.writers] Reading ['/usr/local/lib/python3.12/dist-packages/satpy/etc/writers/geotiff.yaml']
[DEBUG: 2025-01-07 15:18:19 : rasterio.env] GDAL data found in package: path='/usr/local/lib/python3.12/dist-packages/rasterio/gdal_data'.
[DEBUG: 2025-01-07 15:18:19 : rasterio.env] PROJ data found in package: path='/usr/local/lib/python3.12/dist-packages/rasterio/proj_data'.
[DEBUG: 2025-01-07 15:18:19 : satpy.writers] Adding enhancement configuration from file: /usr/local/lib/python3.12/dist-packages/satpy/etc/enhancements/generic.yaml
[DEBUG: 2025-01-07 15:18:19 : satpy.writers] Adding enhancement configuration from file: /usr/local/lib/python3.12/dist-packages/satpy/etc/enhancements/fci.yaml
[DEBUG: 2025-01-07 15:18:19 : satpy.writers] Data for DataID(name='vis_06', wavelength=WavelengthRange(min=0.59, central=0.64, max=0.69, unit='µm'), resolution=500, calibration=<1>, modifiers=()) will be enhanced with options:
        [{'name': 'linear_stretch', 'method': <function stretch at 0x7f010b1f72e0>, 'kwargs': {'stretch': 'crude', 'min_stretch': 0.0, 'max_stretch': 100.0}}, {'name': 'gamma', 'method': <function gamma at 0x7f010b954040>, 'kwargs': {'gamma': 1.5}}]
[DEBUG: 2025-01-07 15:18:19 : trollimage.xrimage] Applying stretch crude with parameters {'min_stretch': 0.0, 'max_stretch': 100.0}
[DEBUG: 2025-01-07 15:18:19 : trollimage.xrimage] Applying gamma 1.5
[DEBUG: 2025-01-07 15:18:19 : rasterio.env] Entering env context: <rasterio.env.Env object at 0x7f010b147c20>
[DEBUG: 2025-01-07 15:18:19 : rasterio.env] Starting outermost env
[DEBUG: 2025-01-07 15:18:19 : rasterio.env] No GDAL environment exists
[DEBUG: 2025-01-07 15:18:19 : rasterio.env] New GDAL environment <rasterio._env.GDALEnv object at 0x7f0107ab9330> created
[DEBUG: 2025-01-07 15:18:19 : rasterio._filepath] Installing FilePath filesystem handler plugin...
[DEBUG: 2025-01-07 15:18:19 : rasterio._env] GDAL data found in package: path='/usr/local/lib/python3.12/dist-packages/rasterio/gdal_data'.
[DEBUG: 2025-01-07 15:18:19 : rasterio._env] PROJ data found in package: path='/usr/local/lib/python3.12/dist-packages/rasterio/proj_data'.
[DEBUG: 2025-01-07 15:18:19 : rasterio._env] Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x7f0107ab9330>.
[DEBUG: 2025-01-07 15:18:19 : rasterio.env] Entered env context: <rasterio.env.Env object at 0x7f010b147c20>
[DEBUG: 2025-01-07 15:18:19 : rasterio._io] Skipped delete for overwrite, dataset does not exist: '/usr/src/app/test/satpy_tif_mtg/NO_PROJECTED_vis_06.tif'
[DEBUG: 2025-01-07 15:18:19 : rasterio._base] Nodata success: 0, Nodata value: 0.000000
[DEBUG: 2025-01-07 15:18:19 : rasterio._base] Nodata success: 0, Nodata value: 0.000000
[DEBUG: 2025-01-07 15:18:19 : rasterio.env] Exiting env context: <rasterio.env.Env object at 0x7f010b147c20>
[DEBUG: 2025-01-07 15:18:19 : rasterio.env] Cleared existing <rasterio._env.GDALEnv object at 0x7f0107ab9330> options
[DEBUG: 2025-01-07 15:18:19 : rasterio._env] Stopped GDALEnv <rasterio._env.GDALEnv object at 0x7f0107ab9330>.
[DEBUG: 2025-01-07 15:18:19 : rasterio.env] Exiting outermost env
[DEBUG: 2025-01-07 15:18:19 : rasterio.env] Exited env context: <rasterio.env.Env object at 0x7f010b147c20>
[INFO: 2025-01-07 15:18:19 : satpy.writers] Computing and writing results...

Screenshots
N/A

Environment Info:

• OS: GNU/Linux
• Satpy Version: 0.53.0
• PyResample Version: 1.31.0
• Readers and writers dependencies (when relevant):

Readers
=======
abi_l1b:  ok
abi_l1b_scmi:  ok
abi_l2_nc:  ok
acspo:  ok
agri_fy4a_l1:  ok
agri_fy4b_l1:  ok
ahi_hrit:  ok
ahi_hsd:  ok
ahi_l1b_gridded_bin:  ok
ahi_l2_nc:  ok
ami_l1b:  ok
amsr2_l1b:  ok
amsr2_l2:  ok
amsr2_l2_gaasp:  ok
amsub_l1c_aapp:  ok
ascat_l2_soilmoisture_bufr:  cannot find module 'satpy.readers.ascat_l2_soilmoisture_bufr' (('Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes.\n           Error: ', ModuleNotFoundError("No module named 'eccodes'")))
atms_l1b_nc:  ok
atms_sdr_hdf5:  ok
avhrr_l1b_aapp:  ok
avhrr_l1b_eps:  ok
avhrr_l1b_gaclac:  cannot find module 'satpy.readers.avhrr_l1b_gaclac' (No module named 'pygac')
avhrr_l1b_hrpt:  ok
avhrr_l1c_eum_gac_fdr_nc:  ok
caliop_l2_cloud:  cannot find module 'satpy.readers.caliop_l2_cloud' (cannot import name 'Dataset' from 'satpy.dataset' (/usr/local/lib/python3.12/dist-packages/satpy/dataset/__init__.py))
clavrx:  ok
cmsaf-claas2_l2_nc:  ok
electrol_hrit:  ok
epic_l1b_h5:  ok
fci_l1c_nc:  ok
fci_l2_bufr:  cannot find module 'satpy.readers.eum_l2_bufr' (Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes)
fci_l2_grib:  cannot find module 'satpy.readers.eum_l2_grib' (Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes)
fci_l2_nc:  ok
fy3a_mersi1_l1b:  ok
fy3b_mersi1_l1b:  ok
fy3c_mersi1_l1b:  ok
generic_image:  cannot find module 'satpy.readers.generic_image' (No module named 'rioxarray')
geocat:  ok
gerb_l2_hr_h5:  ok
ghi_l1:  ok
ghrsst_l2:  ok
gld360_ualf2:  ok
glm_l2:  ok
gms5-vissr_l1b:  cannot find module 'satpy.readers.gms.gms5_vissr_l1b' (No module named 'numba')
goci2_l2_nc:  ok
goes-imager_hrit:  ok
goes-imager_nc:  ok
gpm_imerg:  ok
grib:  cannot find module 'satpy.readers.grib' (No module named 'pygrib')
hsaf_grib:  cannot find module 'satpy.readers.hsaf_grib' (No module named 'pygrib')
hsaf_h5:  ok
hy2_scat_l2b_h5:  ok
iasi_l2:  ok
iasi_l2_cdr_nc:  ok
iasi_l2_so2_bufr:  cannot find module 'satpy.readers.iasi_l2_so2_bufr' (('Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes.\n           Error: ', ModuleNotFoundError("No module named 'eccodes'")))
ici_l1b_nc:  ok
insat3d_img_l1b_h5:  ok
jami_hrit:  ok
li_l2_nc:  ok
maia:  ok
mcd12q1:  ok
meris_nc_sen3:  ok
mersi2_l1b:  ok
mersi3_l1b:  ok
mersi_ll_l1b:  ok
mersi_rm_l1b:  ok
mhs_l1c_aapp:  ok
mimicTPW2_comp:  ok
mirs:  ok
modis_l1b:  ok
modis_l2:  ok
modis_l3:  ok
msi_safe:  ok
msi_safe_l2a:  ok
msu_gsa_l1b:  ok
mtsat2-imager_hrit:  ok
mviri_l1b_fiduceo_nc:  ok
mwi_l1b_nc:  ok
mws_l1b_nc:  ok
nucaps:  ok
nwcsaf-geo:  ok
nwcsaf-msg2013-hdf5:  ok
nwcsaf-pps_nc:  ok
oceancolorcci_l3_nc:  ok
oci_l2_bgc:  ok
olci_l1b:  ok
olci_l2:  ok
oli_tirs_l1_tif:  ok
omps_edr:  ok
osisaf_nc:  ok
safe_sar_l2_ocn:  ok
sar-c_safe:  cannot find module 'satpy.readers.sar_c_safe' (No module named 'rioxarray')
satpy_cf_nc:  ok
scatsat1_l2b:  cannot find module 'satpy.readers.scatsat1_l2b' (cannot import name 'Dataset' from 'satpy.dataset' (/usr/local/lib/python3.12/dist-packages/satpy/dataset/__init__.py))
seadas_l2:  ok
seviri_l1b_hrit:  ok
seviri_l1b_icare:  ok
seviri_l1b_native:  ok
seviri_l1b_nc:  ok
seviri_l2_bufr:  cannot find module 'satpy.readers.eum_l2_bufr' (Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes)
seviri_l2_grib:  cannot find module 'satpy.readers.eum_l2_grib' (Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes)
sgli_l1b:  ok
slstr_l1b:  ok
smos_l2_wind:  ok
tropomi_l2:  ok
vii_l1b_nc:  ok
vii_l2_nc:  ok
viirs_compact:  ok
viirs_edr:  ok
viirs_edr_active_fires:  ok
viirs_edr_flood:  ok
viirs_l1b:  ok
viirs_l2:  ok
viirs_sdr:  ok
viirs_vgac_l1c_nc:  ok
virr_l1b:  ok

Writers
=======
awips_tiled:  ok
cf:  ok
geotiff:  ok
mitiff:  ok
ninjogeotiff:  ok
ninjotiff:  cannot find module 'satpy.writers.ninjotiff' (No module named 'pyninjotiff')
simple_image:  ok

Versions
======
platform: Linux-6.1.61-85.141.amzn2023.x86_64-x86_64-with-glibc2.39
python: 3.12.3

cartopy: not installed
dask: 2024.12.1
fsspec: 2024.12.0
gdal: 3.9.2
geoviews: not installed
h5netcdf: 1.4.1
h5py: 3.12.1
netcdf4: 1.7.2
numpy: 1.26.4
pyhdf: 0.11.6
pyproj: 3.7.0
rasterio: 1.4.2
xarray: 2025.1.0

Additional context

1. I tested this process in the past (around september) with older versions such as 0.51.0 and 0.52.0 and I made it work, however now even with older versions, I cannot generate the products.
2. I tested this process with smaller extents than the mentioned in the code and it worked.
3. I think that due to the volatility of the MTG base files, some configuration could be changed, but not updated on the satpy library

[gerritholl]

Not enough RAM? Try adapting some of the parameters suggested at https://satpy.readthedocs.io/en/stable/faq.html

[AlejandroGorgues]

Hello!
I have adjusted the parameters as suggested in the documentation and tried all the combinations between the different suggestions in the list below.

• Change GDAL_CACHEMAX to '15%' or '30%'
• Change num_workers for dask operations to 4 (this is the max num of cores that I have)
• Change OMP_NUM_THREADS to 2 (half the num_workers)
• Added num_threads=2 while saving the product as geotiff
• Added generate=False while loading the composite

Despite these changes, the two error messages mentioned earlier appears.

Also, if I resample the scene using a gradient_search or a nearest for example, to get the ir_105, the GeoTIFF is sometimes generated successfully. However, at other times, the same errors appears

I also have around 13Gb available memory (16Gb total)