From 3b005bc33593e424708deea30f78345c34229d69 Mon Sep 17 00:00:00 2001 From: Ivan Ruiz Manuel <72193617+irm-codebase@users.noreply.github.com> Date: Mon, 2 Dec 2024 15:34:54 +0100 Subject: [PATCH] Update documentation: templates now in YAML section --- docs/creating/data_tables.md | 3 +- docs/creating/index.md | 3 +- docs/creating/techs.md | 2 +- docs/creating/templates.md | 144 ---------- docs/creating/yaml.md | 311 ++++++++++++++++----- docs/examples/national_scale/index.md | 2 +- docs/examples/urban_scale/index.md | 2 +- docs/migrating.md | 5 +- docs/user_defined_math/helper_functions.md | 2 +- mkdocs.yml | 1 - 10 files changed, 244 insertions(+), 231 deletions(-) delete mode 100644 docs/creating/templates.md diff --git a/docs/creating/data_tables.md b/docs/creating/data_tables.md index 2319ed04..d0d194c6 100644 --- a/docs/creating/data_tables.md +++ b/docs/creating/data_tables.md @@ -18,7 +18,6 @@ In brief it is: * [**drop**](#selecting-dimension-values-and-dropping-dimensions): dimensions to drop from your rows/columns, e.g., a "comment" row. * [**add_dims**](#adding-dimensions): dimensions to add to the table after loading it in, with the corresponding value(s) to assign to the dimension index. * [**rename_dims**](#renaming-dimensions-on-load): dimension names to map from those defined in the data table (e.g `time`) to those used in the Calliope model (e.g. `timesteps`). -* [**template**](#using-a-template): Reference to a [template](templates.md) from which to inherit common configuration options. When we refer to "dimensions", we mean the sets over which data is indexed in the model: `nodes`, `techs`, `timesteps`, `carriers`, `costs`. In addition, when loading from file, there is the _required_ dimension `parameters`. @@ -271,7 +270,7 @@ In this section we will show some examples of loading data and provide the equiv cost_storage_cap.data: 150 ``` - 1. To limit repetition, we have defined [templates](templates.md) for our costs. + 1. To limit repetition, we have defined [templates](yaml.md#reusing-definitions-through-templates) for our costs. !!! info "See also" Our [data table loading tutorial][loading-tabular-data] has more examples of loading tabular data into your model. diff --git a/docs/creating/index.md b/docs/creating/index.md index 35ddefb2..b242aef8 100644 --- a/docs/creating/index.md +++ b/docs/creating/index.md @@ -35,7 +35,7 @@ We distinguish between: - the model **definition** (your representation of a physical system in YAML). Model configuration is everything under the top-level YAML key [`config`](config.md). -Model definition is everything else, under the top-level YAML keys [`parameters`](parameters.md), [`techs`](techs.md), [`nodes`](nodes.md), [`templates`](templates.md), and [`data_tables`](data_tables.md). +Model definition is everything else, under the top-level YAML keys [`parameters`](parameters.md), [`techs`](techs.md), [`nodes`](nodes.md), and [`data_tables`](data_tables.md). It is possible to define alternatives to the model configuration/definition that you can refer to when you initialise your model. These are defined under the top-level YAML keys [`scenarios` and `overrides`](scenarios.md). @@ -84,5 +84,4 @@ The rest of this section discusses everything you need to know to set up a model - An overview of [YAML as it is used in Calliope](yaml.md) - though this comes first here, you can also safely skip it and refer back to it as a reference as questions arise when you go through the model configuration and definition examples. - More details on the [model configuration](config.md). - The key parts of the model definition, first, the [technologies](techs.md), then, the [nodes](nodes.md), the locations in space where technologies can be placed. -- How to use [technology and node templates](templates.md) to reduce repetition in the model definition. - Other important features to be aware of when defining your model: defining [indexed parameters](parameters.md), i.e. parameter which are not indexed over technologies and nodes, [loading tabular data](data_tables.md), and defining [scenarios and overrides](scenarios.md). diff --git a/docs/creating/techs.md b/docs/creating/techs.md index 60d0479d..f2a2a73a 100644 --- a/docs/creating/techs.md +++ b/docs/creating/techs.md @@ -15,7 +15,7 @@ This establishes the basic characteristics in the optimisation model (decision v ??? info "Sharing configuration with templates" To share definitions between technologies and/or nodes, you can use configuration templates (the `template` key). - This allows a technology/node to inherit definitions from [`template` definitions](templates.md). + This allows a technology/node to inherit definitions from [`template` definitions](yaml.md#reusing-definitions-through-templates). Note that `template` is different to setting a `base_tech`. Setting a base_tech does not entail any configuration options being inherited; `base_tech` is only used when building the optimisation problem (i.e., in the `math`). diff --git a/docs/creating/templates.md b/docs/creating/templates.md deleted file mode 100644 index c723c8c8..00000000 --- a/docs/creating/templates.md +++ /dev/null @@ -1,144 +0,0 @@ - -# Inheriting from templates: `templates` - -For larger models, duplicate entries can start to crop up and become cumbersome. -To streamline data entry, technologies, nodes, and data tables can inherit common data from a `template`. - -## Templates in technologies - -If we want to set interest rate to `0.1` across all our technologies, we could define: - -```yaml -templates: - interest_rate_setter: - cost_interest_rate: - data: 0.1 - index: monetary - dims: costs -techs: - ccgt: - template: interest_rate_setter - ... - ac_transmission: - template: interest_rate_setter - ... -``` - -## Templates in nodes - -Similarly, if we want to allow the same technologies at all our nodes: - -```yaml -templates: - standard_tech_list: - techs: {ccgt, battery, demand_power} # (1)! -nodes: - region1: - template: standard_tech_list - ... - region2: - template: standard_tech_list - ... - ... - region100: - template: standard_tech_list -``` - -1. this YAML syntax is shortform for: - ```yaml - techs: - ccgt: - battery: - demand_power: - ``` - -## Templates in data tables - -Data tables can also store common options under the `templates` key, for example: - -```yaml -templates: - common_data_options: - rows: timesteps - columns: nodes - add_dims: - parameters: source_use_max -data_tables: - pv_data: - data: /path/to/pv_timeseries.csv - template: common_data_options - add_dims: - techs: pv - wind_data: - data: /path/to/wind_timeseries.csv - template: common_data_options - add_dims: - techs: wind - hydro_data: - data: /path/to/hydro_timeseries.csv - template: common_data_options - add_dims: - techs: hydro -``` - -## Inheritance chains - -Inheritance chains can also be created. -That is, templates can inherit from other templates. -E.g.: - -```yaml -templates: - interest_rate_setter: - cost_interest_rate: - data: 0.1 - index: monetary - dims: costs - investment_cost_setter: - template: interest_rate_setter - cost_flow_cap: - data: 100 - index: monetary - dims: costs - cost_area_use: - data: 1 - index: monetary - dims: costs -techs: - ccgt: - template: investment_cost_setter - ... - ac_transmission: - template: interest_rate_setter - ... -``` - -## Overriding template values - -Template properties can always be overridden by the inheriting component. -This can be useful to streamline setting costs, e.g.: - -```yaml -templates: - interest_rate_setter: - cost_interest_rate: - data: 0.1 - index: monetary - dims: costs - investment_cost_setter: - template: interest_rate_setter - cost_interest_rate.data: 0.2 # this will replace `0.1` in the `interest_rate_setter`. - cost_flow_cap: - data: null - index: monetary - dims: costs - cost_area_use: - data: null - index: monetary - dims: costs -techs: - ccgt: - template: investment_cost_setter - cost_flow_cap.data: 100 # this will replace `null` in the `investment_cost_setter`. - ... -``` diff --git a/docs/creating/yaml.md b/docs/creating/yaml.md index 6264afa2..4ede042f 100644 --- a/docs/creating/yaml.md +++ b/docs/creating/yaml.md @@ -2,9 +2,90 @@ All model configuration/definition files (with the exception of tabular data files) are in the YAML format, "a human friendly data serialisation standard for all programming languages". +## A quick introduction to YAML + Configuration for Calliope is usually specified as `option: value` entries, where `value` might be a number, a text string, or a list (e.g. a list of further settings). -## Abbreviated nesting +!!! info "See also" + See the [YAML website](https://yaml.org/) for more general information about YAML. + +### Data types + +Using quotation marks (`'` or `"`) to enclose strings is optional, but can help with readability. +The three ways of setting `option` to `text` below are equivalent: + +```yaml +option1: "text" +option2: 'text' +option3: text +``` + +Without quotations, the following values in YAML will be converted to different Python types: + +- Any unquoted number will be interpreted as numeric (e.g., `1`, `1e6` `1e-10`). +- `true` or `false` values will be interpreted as boolean. +- `.inf` and `.nan` values will be interpreted as the float values `np.inf` (infinite) and `np.nan` (not a number), respectively. +- `null` values will interpreted as `None`. + +### Comments + +Comments can be inserted anywhere in YAML files with the `#` symbol. +The remainder of a line after `#` is interpreted as a comment. +Therefore, if you have a string with a `#` in it, make sure to use explicit quotation marks. + +```yaml +# This is a comment +option1: "text with ##hashtags## needs quotation marks" +``` + +### Lists and dictionaries + +Lists in YAML can be of the form `[...]` or a series of lines starting with `-`. +These two lists are equivalent: + +```yaml +key: [option1, option2] +``` + +```yaml +key: + - option1 + - option2 +``` + +Dictionaries can be of the form `{...}` or a series of lines _without_ a starting `-`. +These two dictionaries are equivalent: + +```yaml +key: {option1: value1, option2: value2} +``` + +```yaml +key: + option1: value1 + option2: value2 +``` + +To continue dictionary nesting, you can add more `{}` parentheses or you can indent your lines further. +We prefer to use 2 spaces for indenting as this makes the nested data structures more readable than the often-used 4 spaces. + +We sometimes also use lists of dictionaries in Calliope, e.g.: + +```yaml +key: + - option1: value1 + option2: value2 + - option3: value3 + option4: value4 +``` + +Which is equivalent in Python to `#!python {"key": [{"option1": value1, "option2": value2}, {"option3": value3, "option4": value4}]}`. + +## Calliope's additional YAML features + +To make model definition easier, we add some extra features that go beyond regular YAML formatting. + +### Abbreviated nesting Calliope allows an abbreviated form for long, nested settings: @@ -20,7 +101,7 @@ can be written as: one.two.three: x ``` -## Relative file imports +### Relative file imports Calliope also allows a special `import:` directive in any YAML file. This can specify one or several YAML files to import, e.g.: @@ -125,9 +206,160 @@ scenarios: * The imported files may include further files, so arbitrary degrees of nested configurations are possible. * The `import` statement can either give an absolute path or a path relative to the importing file. -## Overriding one file with another +### Reusing definitions through templates + +For larger models, duplicate entries can start to crop up and become cumbersome. +To streamline data entry, any section can inherit common data from a `template` which is defined in the top-level `templates` section. + +???+ example "Example 1: templates in technologies" + + If we want to set interest rate to `0.1` across all our technologies, we could define: + + ```yaml + templates: + interest_rate_setter: + cost_interest_rate: + data: 0.1 + index: monetary + dims: costs + techs: + ccgt: + template: interest_rate_setter + ... + ac_transmission: + template: interest_rate_setter + ... + ``` + +??? example "Example 2: templates in nodes" + + Similarly, if we want to allow the same technologies at all our nodes: + + ```yaml + templates: + standard_tech_list: + techs: {ccgt, battery, demand_power} # (1)! + nodes: + region1: + template: standard_tech_list + ... + region2: + template: standard_tech_list + ... + ... + region100: + template: standard_tech_list + ``` + + This YAML syntax is shortform for: + + ```yaml + nodes: + region1: + techs: + ccgt: + battery: + demand_power: + ... + ... + ``` + +??? example "Example 3: templates in data tables" + + Storing common options under the `templates` key is also useful for data tables, for example: -While generally, as stated above, if an the imported file and the current file define the same option, Calliope will raise an exception. + ```yaml + templates: + common_data_options: + rows: timesteps + columns: nodes + add_dims: + parameters: source_use_max + data_tables: + pv_data: + data: /path/to/pv_timeseries.csv + template: common_data_options + add_dims: + techs: pv + wind_data: + data: /path/to/wind_timeseries.csv + template: common_data_options + add_dims: + techs: wind + hydro_data: + data: /path/to/hydro_timeseries.csv + template: common_data_options + add_dims: + techs: hydro + ``` + +Inheritance chains can also be created. +That is, templates can inherit from other templates. + +??? example "Example 4: template inheritance chain" + + ```yaml + templates: + interest_rate_setter: + cost_interest_rate: + data: 0.1 + index: monetary + dims: costs + investment_cost_setter: + template: interest_rate_setter + cost_flow_cap: + data: 100 + index: monetary + dims: costs + cost_area_use: + data: 1 + index: monetary + dims: costs + techs: + ccgt: + template: investment_cost_setter + ... + ac_transmission: + template: interest_rate_setter + ... + ``` + +Template properties can always be overridden by the inheriting component. +That is, the 'local' value has priority over the inherited template value. +This can be useful to streamline setting costs for different technologies. + +??? example "Example 5: overriding template values" + + In this example, a technology overrides a single templated cost. + + ```yaml + templates: + interest_rate_setter: + cost_interest_rate: + data: 0.1 + index: monetary + dims: costs + investment_cost_setter: + template: interest_rate_setter + cost_interest_rate.data: 0.2 # this will replace `0.1` in the `interest_rate_setter`. + cost_flow_cap: + data: null + index: monetary + dims: costs + cost_area_use: + data: null + index: monetary + dims: costs + techs: + ccgt: + template: investment_cost_setter + cost_flow_cap.data: 100 # this will replace `null` in the `investment_cost_setter`. + ... + ``` + +### Overriding one file with another + +Generally, if an the imported file and the current file define the same option, Calliope will raise an exception. However, you can define `overrides` which you can then reference when loading your Calliope model (see [Scenarios and overrides](scenarios.md)). These `override` settings will override any data that match the same name and will add new data if it wasn't already there. @@ -167,74 +399,3 @@ Will lead to: one.four: y four.five.six: y ``` - -## Data types - -Using quotation marks (`'` or `"`) to enclose strings is optional, but can help with readability. -The three ways of setting `option` to `text` below are equivalent: - -```yaml -option: "text" -option: 'text' -option: text -``` - -Without quotations, the following values in YAML will be converted to different Python types: - -- Any unquoted number will be interpreted as numeric. -- `true` or `false` values will be interpreted as boolean. -- `.inf` and `.nan` values will be interpreted as the float values `np.inf` (infinite) and `np.nan` (not a number), respectively. -- `null` values will interpreted as `None`. - -## Comments - -Comments can be inserted anywhere in YAML files with the `#` symbol. -The remainder of a line after `#` is interpreted as a comment. -Therefore, if you have a string with a `#` in it, make sure to use explicit quotation marks. - - -## Lists and dictionaries - -Lists in YAML can be of the form `[...]` or a series of lines starting with `-`. -These two lists are equivalent: - -```yaml -key: [option1, option2] -``` - -```yaml -key: - - option1 - - option2 -``` - -Dictionaries can be of the form `{...}` or a series of lines _without_ a starting `-`. -These two dictionaries are equivalent: - -```yaml -key: {option1: value1, option2: value2} -``` - -```yaml -key: - option1: value1 - option2: value2 -``` - -To continue dictionary nesting, you can add more `{}` parentheses or you can indent your lines further. -We prefer to use 2 spaces for indenting as this makes the nested data structures more readable than the often-used 4 spaces. - -We sometimes also use lists of dictionaries in Calliope, e.g.: - -```yaml -key: - - option1: value1 - option2: value2 - - option3: value3 - option4: value4 -``` - -Which is equivalent in Python to `#!python {"key": [{"option1": value1, "option2": value2}, {"option3": value3, "option4": value4}]}`. - -!!! info "See also" - See the [YAML website](https://yaml.org/) for more general information about YAML. diff --git a/docs/examples/national_scale/index.md b/docs/examples/national_scale/index.md index dc4bfe28..0b820b7f 100644 --- a/docs/examples/national_scale/index.md +++ b/docs/examples/national_scale/index.md @@ -201,7 +201,7 @@ As the name suggests, it applies no cost or efficiency losses to this transmissi We can see that those technologies which rely on `free_transmission` inherit a lot of this information from elsewhere in the model definition. `free_transmission` is defined in `templates`, which makes it inheritable. -[Templates](../../creating/templates.md) allow us to avoid excessive repetition in our model definition. +[Templates](../../creating/yaml.md#reusing-definitions-through-templates) allow us to avoid excessive repetition in our model definition. Technologies and nodes can inherit from anything defined in `templates`. items in `templates` can also inherit from each other, so you can create inheritance chains. diff --git a/docs/examples/urban_scale/index.md b/docs/examples/urban_scale/index.md index 241385ac..f0c54898 100644 --- a/docs/examples/urban_scale/index.md +++ b/docs/examples/urban_scale/index.md @@ -226,7 +226,7 @@ Gas is made available in each node without consideration of transmission. --8<-- "src/calliope/example_models/urban_scale/model_config/techs.yaml:transmission" ``` -To avoid excessive duplication in model definition, our transmission technologies inherit most of the their parameters from [templates](../../creating/templates.md): +To avoid excessive duplication in model definition, our transmission technologies inherit most of the their parameters from [templates](../../creating/yaml.md#reusing-definitions-through-templates): ```yaml --8<-- "src/calliope/example_models/urban_scale/model_config/techs.yaml:transmission-templates" diff --git a/docs/migrating.md b/docs/migrating.md index f361358d..930bb7af 100644 --- a/docs/migrating.md +++ b/docs/migrating.md @@ -309,7 +309,7 @@ Instead, links are defined as separate transmission technologies in `techs`, inc ``` !!! note - You can use [`templates`](creating/templates.md) to minimise duplications in the new transmission technology definition. + You can use [`templates`](creating/yaml.md#reusing-definitions-through-templates) to minimise duplications in the new transmission technology definition. ### Renaming parameters/decision variables without core changes in function @@ -746,8 +746,7 @@ This means you could define different output carriers for a `supply` technology, ### `templates` for nodes -The new [`templates` key](creating/templates.md) can be applied to `nodes` as well as `techs`. -This makes up for the [removal of grouping node names in keys by comma separation](#comma-separated-node-definitions). +The new [`templates` key](creating/yaml.md#reusing-definitions-through-templates) makes up for the [removal of grouping node names in keys by comma separation](#comma-separated-node-definitions). So, to achieve this result: diff --git a/docs/user_defined_math/helper_functions.md b/docs/user_defined_math/helper_functions.md index 8b08dad8..9a4c7b59 100644 --- a/docs/user_defined_math/helper_functions.md +++ b/docs/user_defined_math/helper_functions.md @@ -8,7 +8,7 @@ Helper functions generally require a good understanding of their functionality, ## inheritance -Using `inheritance(...)` in a `where` string allows you to grab a subset of technologies / nodes that all share the same [`template`](../creating/templates.md) in the technology's / node's `template` key. +Using `inheritance(...)` in a `where` string allows you to grab a subset of technologies / nodes that all share the same [`template`](../creating/yaml.md#reusing-definitions-through-templates) in the technology's / node's `template` key. If a `template` also inherits from another `template` (chained inheritance), you will get all `techs`/`nodes` that are children along that inheritance chain. So, for the definition: diff --git a/mkdocs.yml b/mkdocs.yml index 6b41e180..2db9eee6 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -104,7 +104,6 @@ nav: - Model configuration: creating/config.md - Technologies: creating/techs.md - Nodes: creating/nodes.md - - Inheriting from templates: creating/templates.md - Indexed parameters: creating/parameters.md - Loading data tables: creating/data_tables.md - Scenarios and overrides: creating/scenarios.md