Apps macro wps (#261)

source/Reviewers/howtocommit.rst

@@ -166,6 +166,18 @@ then you will need to upgrade the test-suite.
             where `vnX.Y_tZZZZ` is the `AFTER_TAG` of the latest upgrade macro.
             The upgrade is expected to fail for the `fab_jules`, `metadata_checker` and `umdp3_checker` apps.
 
+        .. tab-item:: LFRic Apps + Core
+
+            .. code-block:: RST
+
+                apply_macros.py vnX.Y_tZZZZ [--apps=/path/to/apps] [--core=/path/to/core] [--jules=/path/to/jules]
+
+            where `vnX.Y_tZZZZ` is the `AFTER_TAG` of the latest upgrade macro and the others are paths to the relevant sources. Apps defaults to the current location. Core and Jules default to reading the `dependencies.sh` file in the Apps source. A copy of `apply_macros.py` is available at `$UMDIR/SimSys_Scripts/lfric_macros`.
+
+            .. note::
+
+                All LFRic Core tickets with macros are expected to be linked with LFRic Apps, though they may not have required an LFRic Apps development branch (although an Apps ticket should be provided). This is fine - if there is no LFRic Apps branch just checkout the LFRic Apps trunk. Then run the apply_macros script as described above and this will share the upgrade macro across both LFRic Apps and LFRic Core as needed.
+
 .. dropdown:: New rose-stem app?
 
     If the ticket introduces a new rose-stem app, but doesn't otherwise have a macro
@@ -191,11 +203,12 @@ then you will need to upgrade the test-suite.
             rose macro --validate -M path/to/working_copy/rose-meta
 
     .. note::
-       For UM tickets, if there are linked `jules-shared
-       <https://code.metoffice.gov.uk/trac/jules/browser/main/trunk/rose-meta/jules-shared>`_
-       metadata changes these will need to be added to the metadata
-       path. Please see the :ref:`rose config-edit
-       example<metadata_changes>`.
+
+        LFRic Apps tickets will require an LFRic Core source to use. You can do this by checking out an appropriate working copy, and exporting the environment variable `ROSE_META_PATH=/path/to/core`.
+
+        For UM tickets, if there are linked `jules-shared
+        <https://code.metoffice.gov.uk/trac/jules/browser/main/trunk/rose-meta/jules-shared>`_
+        metadata changes then a suitable Jules source will need to be included in the `ROSE_META_PATH` as described above.
 
 
 .. dropdown:: Temporary Logical?
@@ -492,7 +505,6 @@ Supporting data is stored in the filesystems of our machines and changes to use
         .. note::
               The numbered run directory must be included in the suite name, eg. `name-of-suite/run1`.
 
-
         5. Verify the checksums updated properly by retriggering the failed checksums. First retrigger
         ``export-source``, and then when complete ``export-source_xc40`` if new checksums are present
         there (there is no need to retigger spice). You may need to change the maximum window extent

source/WorkingPractices/TestSuites/ukca.rst

@@ -5,5 +5,5 @@ Changes in UKCA that touch `src/science` or `src/control/core` must be tested
 with both the UM and LFRic by following the :ref:`linked tickets guidance <multirepo>`.
 
 For further guidance on testing and working with UKCA, including standard
-suites and box models see the 
-`UKCA trac wiki<https://code.metoffice.gov.uk/trac/ukca/wiki/WorkingPractices>`_.
+suites and box models see the
+`UKCA trac wiki <https://code.metoffice.gov.uk/trac/ukca/wiki/WorkingPractices>`_.

source/WorkingPractices/documentation.rst

@@ -15,7 +15,7 @@ Most notably:
 
 LFRic Apps and Core also use doxygen to document the code and all changes should
 include appropriate doxygen changes to go with them. Doxygen guidelines are
-available on the `LFRic Technical pages<https://code.metoffice.gov.uk/trac/lfric/wiki/LFRicTechnical/DoxygenUsage>`_.
+available on the `LFRic Technical pages <https://code.metoffice.gov.uk/trac/lfric/wiki/LFRicTechnical/DoxygenUsage>`_.
 
 .. toctree::
     :hidden:

source/WorkingPractices/inputs.rst

@@ -53,7 +53,7 @@ metadata :ref:`is available <metadata_guidance>`.
   work with the model.
 
 Changes to the metadata which don't involve namelist changes may or may not
-require an upgrade macro. If you are unsure whether a UM change needs an
+require an upgrade macro. If you are unsure whether a change needs an
 upgrade macro, then run the following command on your branch:
 
 .. code-block::
@@ -63,10 +63,25 @@ upgrade macro, then run the following command on your branch:
 If all of the tests pass then there is no requirement to add an upgrade macro.
 If any of the metadata tests fail, then the developer should add a blank upgrade
 macro which contains no upgrade commands but simply points the rose stem suite
-to the new metadata.
+to the new metadata. The SSD team are also available to advise on whether an upgrade macro is necessary.
 
-..
-  The above should probably be extended to LFRic eventually.
+.. tip::
+
+  When editing metadata you should always check that the new metadata appears as expected in the gui, including testing that invalid settings raise appropriate warnings. The command to open the gui is in general:
+
+  .. code-block::
+
+    rose edit -C rose-stem/app/APP-NAME
+
+  For LFRic Apps a few extra changes are required:
+
+  .. code-block::
+
+    export ROSE_PYTHONPATH=$PYTHONPATH
+    export ROSE_META_PATH=/path/to/valid/core
+    rose edit -C rose-stem/app/APP-NAME
+
+  This requires an LFRic Core working copy at an appropriate revision to be available. It is also necessary to run from the top level of the Apps working copy to ensure rose metadata paths are valid.
 
 How to add an upgrade macro to your branch
 ------------------------------------------

source/WorkingPractices/macros.rst

@@ -3,10 +3,11 @@
 Upgrade Macros
 ==============
 
-To create an upgrade macro, the developer must edit a ``versions.py`` file which is used
-to update the various apps in the rose stem suite to accept the namelist changes. The upgrade
-macros also form the basis of the ``rose app-upgrade`` script applied by a user wishing to
-upgrade from one version of a model to the next.
+.. important::
+
+    When developing Upgrade Macros, they must be tested using a test branch (see :ref:`testing`).
+
+To create an upgrade macro, the developer must edit a ``versions.py`` file which is used to update the various apps in the rose stem suite to accept the namelist changes. The upgrade macros also form the basis of the ``rose app-upgrade`` script applied by a user wishing to upgrade from one version of a model to the next.
 
 The  ``versions.py`` file containing upgrade macros can be found in the following locations:
 
@@ -20,6 +21,11 @@ The  ``versions.py`` file containing upgrade macros can be found in the followin
 
         ``rose-meta/jules-standalone/versions.py``
 
+    .. tab-item:: LFRic Core + Apps
+
+        | ``applications/<APPLICATION>/rose-meta/lfric-<APPLICATION>/versions.py``
+        | Variations on this theme occur, eg. LFRic Apps science sections or Components in LFRic Core
+
 
 Within the file a blank upgrade macro will typically look like this:
 
@@ -73,3 +79,51 @@ This command can then be run on a **test** branch (see :ref:`testing`).
   namelists and changing the value that a particular variable takes.
   A `tutorial <http://metomi.github.io/rose/doc/html/tutorial/rose/furthertopics/upgrading.html>`_
   is also available.
+
+
+Upgrade Macros in LFRic
+-----------------------
+
+.. warning::
+
+    Namelist files in application example directories are not currently updated by the Apply Macros script. This feature is intended to be introduced, but for now, developers still need to manually update those files.
+
+The organisation of LFRic metadata is different from other repositories (UM + Jules) as the metadata is stored with the Science or Application section it is associated with and is then imported by other apps that require it. This helps modularise the LFRic code but complicates macro chains.
+
+To solve this, macros in LFRic Apps are applied using a wrapper script which will read the added macros and combine them into the versions.py files for the apps where that metadata is imported. Therefore when adding macros, the macro should be added in the versions.py file in the same metadata directory as the metadata change being made. It will then be shared as appropriate by the ``apply_macros.py`` script.
+
+For example, if a change to metadata is made in ``science/gungho/rose-meta/lfric-gungho``, the macro should be added to the ``versions.py`` file in that directory. This will then be copied to other ``versions.py`` files that import gungho metadata, eg. lfric_atm, transport etc.
+
+It is expected that all metadata changes in LFRic Core will require change to the rose-apps in LFRic Apps, but changes to Apps must not affect Core. Therefore, the apply_macros script requires a working copy of LFRic Apps to work, but will source it's own copy of Core if required. If your only changes are to LFRic Core metadata, then you will require a linked LFRic Apps ticket and test branch, but potentially not a development branch.
+
+.. important::
+
+    Some complex macro commands may be dependent on the order in which they are applied. As macros are copied by the wrapper script, the order they are applied will always be determined by the reverse metadata import order. For example, lfric_atm imports gungho metadata, which itself imports components/driver. If all 3 sections have an associated macro, then the macro commands would be applied in the order: components/driver, gungho, lfric_atm.
+
+.. tip::
+
+    The wrapper script will read the ``dependencies.sh`` file in your LFRic Apps working copy and will checkout a temporary copy of the LFRic Core source if required. Some Core metadata changes will also modify the Core rose apps. In this case make sure to also commit these changes back to the core branch.
+
+To add upgrade macros to LFRic the following steps can be followed:
+
+
+
+1. Checkout an LFRic Apps working copy and update the core source in ``dependencies.sh`` if you have LFRic Core changes.
+2. Add your upgrade macros. These **must** be added to the versions.py file in the same directory as the metadata being changed.
+3. Run the Upgrade Macro script in a test branch (see :ref:`testing`). This is located in the `SimSys_Scripts github repo <https://github.com/MetOffice/SimSys_Scripts>`_ (at meto an up to date clone is available in $UMDIR/SimSys_Scripts). The syntax for running is:
+
+.. code-block::
+
+    SimSys_Scripts/lfric_macros/apply_macros.py vnXX.Y_tTTTT -a Apps -c Core -j Jules
+
+.. important::
+
+    **Test branches must be used for running the Apply Macros script. Do not commit the changes made by apply_macros.py to a Dev Branch**
+
+The Apps, Core and Jules options are paths to sources for each of these. Apps will default to the present location (so it is recommended to launch from an Apps working copy). Core and Jules will default to reading the ``dependencies.sh`` file in the Apps source if not provided.
+
+The ``vnXX.Y_tTTTT`` option must match the After Tag of your upgrade macro. When setting this, the version is the last released version of LFRic Apps. If it's a linked Apps-Core ticket, then set the ticket number as the one where the most metadata changes are being made.
+
+.. tip::
+
+    The apply_macros script requires python >= 3.9. At the Met Office this can be achieved by ``module load scitools``.

source/WorkingPractices/testing.rst

@@ -79,12 +79,12 @@ commands, noting that ``--jules-path`` is only required if you have
 +-------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
 | JULES | ``./bin/upgrade_jules_test_apps vnX.X_tXXXX``                                                                                                                                |
 +-------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| LFRic | ``apply_macros.py vnX.Y_tZZZZ [--apps=/path/to/apps] [--core=/path/to/core] [--jules=/path/to/jules]``                                                                       |
++-------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
 
-..
-    | LFRic | *Macro capabilities are currently in development*                                                                                              |
-    +-------+------------------------------------------------------------------------------------------------------------------------------------------------+
+.. tip::
 
-.. todo: add an LFRic upgrade macro command... and make these more consistent?
+    The `apply_macros.py` script is located in the `SimSys_Scripts github repo <https://github.com/MetOffice/SimSys_Scripts>`_ (at meto an up to date clone is available in $UMDIR/SimSys_Scripts).
 
 .. warning::
    Please ensure that Cylc7 is used with `update_all.py` @vn13.5. This is fixed at HoT and either Cylc7 or Cylc8 can be used.