Interpolation of RF predictions with cosZD, for homogeneous performance

[moralejo]

This should solve the issue of RF "performance jumps" at high zenith angles, when the pointing goes through the middle points between training nodes. See #1317

NOTE: this is a different implementation of the interpolation approach proposed by @gabemery, see branch https://github.com/cta-observatory/cta-lstchain/tree/dl2_RF_interpolate)

If the interpolation option is activated we call the RF predictors twice, for each of the two closest MC training nodes, then interpolate (or extrapolate) the values to the actual telescope pointing for each event.

Currently the training sample pointings are obtained from the path to the training sample(provided via config file). A better solution would be to add to the .sav files an array with the zenith and azimuth values of the MC training nodes.

[codecov]

Codecov Report

Attention: Patch coverage is 43.85965% with 64 lines in your changes missing coverage. Please review.

Project coverage is 73.27%. Comparing base (9aaa78b) to head (95632d0).
Report is 9 commits behind head on main.

Files with missing lines	Patch %	Lines
lstchain/reco/dl1_to_dl2.py	20.33%	47 Missing ⚠️
lstchain/scripts/lstchain_dl1_to_dl2.py	69.09%	17 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1320      +/-   ##
==========================================
- Coverage   73.52%   73.27%   -0.25%     
==========================================
  Files         134      134              
  Lines       14215    14321     +106     
==========================================
+ Hits        10451    10494      +43     
- Misses       3764     3827      +63     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[vuillaut]

Hi @moralejo
Thank you for making this implementation.
Early feedback:
would it be possible to add the training and testing nodes of the current production as an ascii file in the lstchain data files to allow for testing without necessarily being on the cluster?
(astropy tables can be dumped to ascii easily)
I completely agree these pointing nodes should be saved with the models during training and then loaded from there in the future.

[moralejo]

Hi @moralejo Thank you for making this implementation. Early feedback: would it be possible to add the training and testing nodes of the current production as an ascii file in the lstchain data files to allow for testing without necessarily being on the cluster? (astropy tables can be dumped to ascii easily)

Will do. What is better, just the zenith and azimuth values, or a list of the directory names?

[vuillaut]

Hi @moralejo Thank you for making this implementation. Early feedback: would it be possible to add the training and testing nodes of the current production as an ascii file in the lstchain data files to allow for testing without necessarily being on the cluster? (astropy tables can be dumped to ascii easily)

Will do. What is better, just the zenith and azimuth values, or a list of the directory names?

I think the table of values that you can use directly at inference would be great (with appropriate metadata such as the date of export).

[moralejo]

Will do. What is better, just the zenith and azimuth values, or a list of the directory names?

I think the table of values that you can use directly at inference would be great (with appropriate metadata such as the date of export).

Can you clarify exactly what metadata (and how? Just lines starting with # or whatever?)

[moralejo]

What if we just write in the config the two arrays of zeniths and azimuths?
(considering it is just for local testing, not needed for normal execution)

I did not implement it yet, it would be like this:

"random_forest_zd_interpolation": {

"interpolate_energy": true,
"interpolate_gammaness": true,
"interpolate_direction": true,
"DL1_training_dir": null,
"DL1_training_pointings": [
  [ 69.19,   27.055],
  [290.81,   27.055],
  [ 63.197,  63.108],
  [ 69.537,  34.367],
  [ 66.706,  19.807],
  [293.294,  19.807],
  [ 67.397,  48.911],
  [296.803,  63.108],
  [292.603,  48.911],
  [ 65.502,  56.069],
  [ 58.737,  12.829],
  [290.463,  34.367],
  [ 60.505,  69.992],
  [328.91,    7.093],
  [ 68.802,  41.666],
  [291.198,  41.666],
  [294.498,  56.069],
  [301.263,  12.829],
  [ 31.09,    7.093],
  [299.495,  69.992]
]

}

Copilot reviewed 2 out of 3 changed files in this pull request and generated no suggestions.

Files not reviewed (1)

• lstchain/data/lstchain_standard_config.json: Language not supported

Comments skipped due to low confidence (1)

lstchain/reco/dl1_to_dl2.py:215

• The word 'dictionnary' is misspelled. It should be 'dictionary'.

config: dictionnary containing configuration

Copilot reviewed 2 out of 3 changed files in this pull request and generated no comments.

Files not reviewed (1)

• lstchain/data/lstchain_standard_config.json: Language not supported

Comments suppressed due to low confidence (1)

lstchain/scripts/lstchain_dl1_to_dl2.py:164

• Ensure that the training_pointings array is correctly populated and used. If the directory does not exist, the interpolation should be switched off.

if dl1_training_dir.is_dir():