Merge branch 'main' into update-calliope-terminology-overview

.github/workflows/release.yml

@@ -13,7 +13,7 @@ jobs:
     steps:
 
     - name: Download built package from another workflow
-      uses: dawidd6/action-download-artifact@v3
+      uses: dawidd6/action-download-artifact@v6
       with:
         name: ${{ env.PACKAGENAME }}
         workflow: pr-ci.yml

.pre-commit-config.yaml

@@ -3,7 +3,7 @@ default_language_version:
 
 repos:
   - repo: https://github.com/astral-sh/ruff-pre-commit # https://beta.ruff.rs/docs/usage/#github-action
-    rev: v0.6.3
+    rev: v0.7.2
     hooks:
       - id: ruff
         args: [--fix, --exit-non-zero-on-fix]

CHANGELOG.md

@@ -3,7 +3,18 @@
 ### User-facing changes
 
 |changed| "An overview of the Calliope terminology" information admonition to remove self-references and improve understandability.
-Now also includes a visual depiction of how the different defined components connect together.
+Now also includes a visual depiction of how the different defined components connect together (#699).
+
+|fixed| Area-based parameters have appropriate documented units of `area` rather than `area^2` (#701).
+
+|fixed| Technology capacity lower bound constraints so that `[cap-type]_min` (e.g., `flow_cap_min`) is not always enforced if the `purchased_units` variable is active (#643).
+
+|changed| Single data entries defined in YAML indexed parameters will not be automatically broadcast along indexed dimensions.
+To achieve the same functionality as in `<v0.7.dev4`, the user must set the new `init` configuration option `broadcast_param_data` to True (#615).
+
+|changed| Helper functions are now documented on their own page within the "Defining your own math" section of the documentation (#698).
+
+|new| `where(array, condition)` math helper function to apply a where array _inside_ an expression, to enable extending component dimensions on-the-fly, and applying filtering to different components within the expression (#604, #679).
 
 |new| Data tables can inherit options from `templates`, like `techs` and `nodes` (#676).
 
@@ -18,6 +29,10 @@ Now also includes a visual depiction of how the different defined components con
 |changed| `data_sources` -> `data_tables` and `data_sources.source` -> `data_tables.data`.
 This change has occurred to avoid confusion between data "sources" and model energy "sources" (#673).
 
+### Internal changes
+
+|fixed| Avoided gurobi 12.0 incompatibility with pyomo by setting the lower bound to v6.8.2.
+
 ## 0.7.0.dev4 (2024-09-10)
 
 ### User-facing changes

docs/creating/config.md

@@ -41,6 +41,10 @@ To test your model pipeline, `config.init.time_subset` is a good way to limit yo
     Various capabilities are available to adjust the temporal resolution of a model on-the-fly, both by resampling or using externally-provided clustering.
     See our [time adjustment page](../advanced/time.md) for more details.
 
+!!! info "See also"
+    The full set of available configuration options is documented in the [configuration schema][model-configuration-schema].
+    This provides you with a description of each configuration option and the default which will be used if you do not provide a value.
+
 ## Deep-dive into some key configuration options
 
 ### `config.build.backend`
@@ -84,7 +88,7 @@ In fact, you can use a set of results from using `plan` model to initialise both
 ### `config.solve.solver`
 
 Possible options for solver include `glpk`, `gurobi`, `cplex`, and `cbc`.
-The interface to these solvers is done through the Pyomo library. Any [solver compatible with Pyomo](https://pyomo.readthedocs.io/en/6.5.0/solving_pyomo_models.html#supported-solvers) should work with Calliope.
+The interface to these solvers is done through the Pyomo library. Any [solver compatible with Pyomo](https://pyomo.readthedocs.io/en/latest/reference/topical/appsi/appsi.solvers.html) should work with Calliope.
 
 For solvers with which Pyomo provides more than one way to interface, the additional `solver_io` option can be used.
 In the case of Gurobi, for example, it is usually fastest to use the direct Python interface:

docs/creating/parameters.md

@@ -53,3 +53,29 @@ Which will add the new dimension `my_new_dim` to your model: `model.inputs.my_ne
 
 !!! warning
     The `parameter` section should not be used for large datasets (e.g., indexing over the time dimension) as it will have a high memory overhead on loading the data.
+
+## Broadcasting data along indexed dimensions
+
+If you want to set the same data for all index items, you can set the `init` [configuration option](config.md) `broadcast_param_data` to True and then use a single value in `data`:
+
+=== "Without broadcasting"
+
+    ```yaml
+    my_indexed_param:
+      data: [1, 1, 1, 1]
+      index: [my_index_val1, my_index_val2, my_index_val3, my_index_val4]
+      dims: my_new_dim
+    ```
+
+=== "With broadcasting"
+
+    ```yaml
+    my_indexed_param:
+      data: 1  # All index items will take on this value
+      index: [my_index_val1, my_index_val2, my_index_val3, my_index_val4]
+      dims: my_new_dim
+    ```
+
+!!! warning
+    The danger of broadcasting is that you maybe update `index` as a scenario override without realising that the data will be broadcast over this new index.
+    E.g., if you start with `!#yaml {data: 1, index: monetary, dims: costs}` and update it with `!#yaml {index: [monetary, emissions]}` then the `data` value of `1` will be set for both `monetary` and `emissions` index values.

docs/examples/national_scale/index.md

@@ -1,3 +1,10 @@
+---
+costs:
+    file: "src/calliope/example_models/national_scale/data_tables/costs.csv"
+    header: [0, 1]
+    index_col: 0
+---
+
 # National Scale Example Model
 
 This example consists of two possible power supply technologies,
@@ -25,6 +32,29 @@ It does not contain much data, but the scaffolding with which to construct and r
 
 ## Model definition
 
+### Referencing tabular data
+
+As of Calliope v0.7.0 it is possible to load tabular data completely separately from the YAML model definition.
+To do this we reference data tables under the `data_tables` key:
+
+```yaml
+--8<-- "src/calliope/example_models/national_scale/model.yaml:data-tables"
+```
+
+In the Calliope national scale example model, we load both timeseries and cost data from file.
+As an example, the data in the cost CSV file looks like this:
+
+{{ read_csv(page.meta.costs.file, header=page.meta.costs.header, index_col=page.meta.costs.index_col) }}
+
+You'll notice that in each row there is reference to a technology, and in each column to a cost parameter and a comment on the units being used.
+Therefore, we reference `techs` in our data table _rows_, and `parameters` and `comment` in our data table _columns_.
+The `comment` information is metadata that we don't need in our Calliope model object, so we _drop_ it on loading the table.
+Since all the data refers to the one cost class `monetary`, we don't add that information in the CSV file, but instead add it on as a dimension when loading the file.
+Where there is no data for that combination of technology and cost parameter, the value is Not-a-Number (NaN) and this combination will be ignored on loading the table.
+
+!!! info
+    You can read more about loading data from file in [our dedicated tutorial][loading-tabular-data].
+
 ### Indexed parameters
 
 Before we dive into the technologies and nodes in the model, we have defined some parameters that are independent of both of these:
@@ -110,23 +140,6 @@ The costs are more numerous as well, and include monetary costs for all relevant
 * carrier conversion capacity
 * variable operational and maintenance costs
 
-### Interlude: inheriting from templates
-
-You will notice that the above technologies _inherit_ `cost_dim_setter`.
-Templates allow us to avoid excessive repetition in our model definition.
-In this case, `cost_dim_setter` defines the dimension and index of costs, allowing us to keep our definition of technology costs to only defining `data`.
-By defining `data`, the technologies override the `null` setting applied by `cost_dim_setter`.
-We also use it to set the `interest_rate` for all technologies, which will be used to annualise any investment costs each technology defines.
-
-Technologies and nodes can inherit from anything defined in `templates`.
-items in `templates` can also inherit from each other, so you can create inheritance chains.
-
-`cost_dim_setter` looks like this:
-
-```yaml
---8<-- "src/calliope/example_models/national_scale/model_config/techs.yaml:cost-dim-setter"
-```
-
 ### Storage technologies
 
 The second location allows a limited amount of battery storage to be deployed to better balance the system.
@@ -184,8 +197,16 @@ Transmission technologies look different to other technologies, as they link the
 `free_transmission` allows local power transmission from any of the csp facilities to the nearest location.
 As the name suggests, it applies no cost or efficiency losses to this transmission.
 
+### Interlude: inheriting from templates
+
 We can see that those technologies which rely on `free_transmission` inherit a lot of this information from elsewhere in the model definition.
 `free_transmission` is defined in `templates`, which makes it inheritable.
+[Templates](../../creating/templates.md) allow us to avoid excessive repetition in our model definition.
+
+Technologies and nodes can inherit from anything defined in `templates`.
+items in `templates` can also inherit from each other, so you can create inheritance chains.
+
+The `free_transmission` template looks like this:
 
 ```yaml
 --8<-- "src/calliope/example_models/national_scale/model_config/techs.yaml:free-transmission"

docs/examples/urban_scale/index.md

@@ -47,6 +47,8 @@ The import section in our file looks like this:
 --8<-- "src/calliope/example_models/urban_scale/model.yaml:import"
 ```
 
+## Model definition
+
 ### Referencing tabular data
 
 As of Calliope v0.7.0 it is possible to load tabular data completely separately from the YAML model definition.
@@ -56,7 +58,7 @@ To do this we reference data tables under the `data_tables` key:
 --8<-- "src/calliope/example_models/urban_scale/model.yaml:data-tables"
 ```
 
-In the Calliope example models, we only load timeseries data from file, including for [energy demand](#demand-technologies), [electricity export price](#revenue-by-export) and [solar PV resource availability](#supply-technologies).
+In the Calliope urban scale example model, we only load timeseries data from file, including for [energy demand](#demand-technologies), [electricity export price](#revenue-by-export) and [solar PV resource availability](#supply-technologies).
 These are large tables of data that do not work well in YAML files!
 As an example, the data in the energy demand CSV file looks like this:
 
@@ -69,8 +71,6 @@ Since all the data refers to the one parameter `sink_use_equals`, we don't add t
 !!! info
     You can read more about loading data from file in [our dedicated tutorial][loading-tabular-data].
 
-## Model definition
-
 ### Indexed parameters
 
 Before we dive into the technologies and nodes in the model, we have defined some parameters that are independent of both of these:
@@ -127,21 +127,6 @@ The definition of this technology in the example model's configuration looks as
 --8<-- "src/calliope/example_models/urban_scale/model_config/techs.yaml:pv"
 ```
 
-### Interlude: inheriting from templates
-
-You will notice that the above technologies _inherit_ `interest_rate_setter`.
-Templates allow us to avoid excessive repetition in our model definition.
-In this case, `interest_rate_setter` defines an interest rate that will be used to annualise any investment costs the technology defines.
-
-Technologies / nodes can inherit from anything defined in `templates`.
-items in `templates` can also inherit from each other, so you can create inheritance chains.
-
-`interest_rate_setter` looks like this:
-
-```yaml
---8<-- "src/calliope/example_models/urban_scale/model_config/techs.yaml:interest-rate-setter"
-```
-
 ### Conversion technologies
 
 The example model defines two conversion technologies.
@@ -241,7 +226,7 @@ Gas is made available in each node without consideration of transmission.
 --8<-- "src/calliope/example_models/urban_scale/model_config/techs.yaml:transmission"
 ```
 
-To avoid excessive duplication in model definition, our transmission technologies inherit most of the their parameters from _templates_:
+To avoid excessive duplication in model definition, our transmission technologies inherit most of the their parameters from [templates](../../creating/templates.md):
 
 ```yaml
 --8<-- "src/calliope/example_models/urban_scale/model_config/techs.yaml:transmission-templates"

docs/installation.md

@@ -62,7 +62,7 @@ However, we recommend to not use this solver where possible, since it performs r
 Indeed, our example models use the free and open source [CBC](#cbc) solver instead, but installing it on Windows requires an extra step.
 [CBC](#cbc) (open-source) or [Gurobi](#gurobi) (commercial) are recommended for large problems, and have been confirmed to work with Calliope.
 The following subsections provide additional detail on how to install a solver.
-This list is not exhaustive; any solvers [supported by Pyomo](https://pyomo.readthedocs.io/en/stable/solving_pyomo_models.html#supported-solvers) can be used.
+This list is not exhaustive; any solvers [supported by Pyomo](https://pyomo.readthedocs.io/en/latest/reference/topical/appsi/appsi.solvers.html) can be used.
 
 ### CBC
 

docs/reference/api/helper_functions.md

@@ -4,3 +4,6 @@ search:
 ---
 
 ::: calliope.backend.helper_functions
+    options:
+      docstring_options:
+        ignore_init_summary: true