esmvaltool run --help does not show all possible command line options

[schlunma]

Currently, esmvaltool run --help returns

> esmvaltool run --help
...

FLAGS
    --config_file=CONFIG_FILE
        Type: Optional[]
        Default: None
        Configuration file to use. If not provided the file ${HOME}/.esmvaltool/config-user.yml will be used.
    --resume_from=RESUME_FROM
        Type: Optional[]
        Default: None
        Resume one or more previous runs by using preprocessor output files from these output directories.
    --max_datasets=MAX_DATASETS
        Type: Optional[]
        Default: None
        Maximum number of datasets to use.
    --max_years=MAX_YEARS
        Type: Optional[]
        Default: None
        Maximum number of years to use.
    --skip_nonexistent=SKIP_NONEXISTENT
        Default: False
        If True, the run will not fail if some datasets are not available.
    --offline=OFFLINE
        Type: Optional[]
        Default: None
        If True, the tool will not download missing data from ESGF.
    --diagnostics=DIAGNOSTICS
        Type: Optional[]
        Default: None
        Only run the selected diagnostics from the recipe. To provide more than one diagnostic to filter use the syntax 'diag1 diag2/script1' or '("diag1", "diag2/script1")' and pay attention to the quotes.
    --check_level=CHECK_LEVEL
        Default: 'default'
        Configure the sensitivity of the CMOR check. Possible values are: `ignore` (all errors will be reported as warnings), `relaxed` (only fail if there are critical errors), default (fail if there are any errors), strict (fail if there are any warnings).
    Additional flags are accepted.

These are clearly not all possible arguments, see for example here:

ESMValCore/esmvalcore/experimental/config/_config_validators.py

Lines 254 to 289 in 4b8bf66

	_validators = {
	# From user config
	'log_level': validate_string,
	'exit_on_warning': validate_bool,
	'output_dir': validate_path,
	'download_dir': validate_path,
	'auxiliary_data_dir': validate_path,
	'extra_facets_dir': validate_pathtuple,
	'compress_netcdf': validate_bool,
	'save_intermediary_cubes': validate_bool,
	'remove_preproc_dir': validate_bool,
	'max_parallel_tasks': validate_int_or_none,
	'config_developer_file': validate_config_developer,
	'profile_diagnostic': validate_bool,
	'run_diagnostic': validate_bool,
	'output_file_type': validate_string,
	# From CLI
	"resume_from": validate_pathlist,
	"skip-nonexistent": validate_bool,
	"diagnostics": validate_diagnostics,
	"check_level": validate_check_level,
	"offline": validate_bool,
	'max_years': validate_int_positive_or_none,
	'max_datasets': validate_int_positive_or_none,
	# From recipe
	'write_ncl_interface': validate_bool,
	# oldstyle
	'rootpath': validate_oldstyle_rootpath,
	'drs': validate_oldstyle_drs,
	# config location
	'config_file': validate_path,
	}

@ESMValGroup/esmvaltool-coreteam Is there a certain strategy which options are shows in the help and which not? I think it would definitely make sense to show additional options (not all of them!), like run_diagnostic which even appears in the documentation.

[valeriupredoi]

related to this #936 - we really need to fix the command helper and run

[zklaus]

Can we single-source this? Eg embed the output of esmvaltool --help in the documentation or perhaps a version of esmvalcore._main.ESMValTool.run.__doc__ or similar?

[schlunma]

Yes, I agree with that!

As far as I can tell Fire uses the docstrings of the corresponding classes and functions (e.g., this one for the run subcommand) to create the --help output, so using esmvalcore._main.ESMValTool.run.__doc__ as single source would definitely make sense!

Adding additional optional arguments for the --help command can probably be implemented by explicitly adding optional arguments to run (right now, many of them are passed in the kwargs).

[bouweandela]

Is there a certain strategy which options are shows in the help and which not?

Historically some options are command-line options and others are configuration file options. At some point, it became possible to specify most (all?) options either on the command line or in the configuration file and the distinction got a bit lost. The best solution seems to be have a look at all available options and think about which options make sense as a command line option and document those with the --help flag.

[valeriupredoi]

fire has a PR improving this google/python-fire#364 - seems like their dev process is somewhat slower though wrt PR acceptance. I am also wondering if we should ask them about the issue with command line args integrity ie --config_user vs --config-user

[schlunma]

I think the problem here is not fire, but rather that we do not explicitly list all possible options in the main function but use **kwargs instead:

ESMValCore/esmvalcore/_main.py

Lines 325 to 335 in 54385ac

	def run(self,
	recipe,
	config_file=None,
	resume_from=None,
	max_datasets=None,
	max_years=None,
	skip_nonexistent=None,
	offline=None,
	diagnostics=None,
	check_level=None,
	**kwargs):