cronjobs: expand script to run stand-alone (#1215)

With this PR instructions to set up the source code repo have been turned into executable script chunks. Hence, if the scripts are executed on a new server, the GitHub repo cloning is done as well as needed directories are created. Furthermore, some minor script tuning: * comment improvements * define USER env by id command (Docker compliant) * replace `~` with env var * do not clone repo within scripts * simplify instructions * compile_addons_git.sh: run 2 make jobs in parallel * accommodate for script path defined in grass-addons-index.sh * README.md: add intro to docker section; make it stand-alone Co-authored-by: Vaclav Petras <[email protected]>
OSGeo · Nov 12, 2024 · fe58963 · fe58963
1 parent 2070d84
commit fe58963
Show file tree

Hide file tree

Showing 10 changed files with 266 additions and 185 deletions.
diff --git a/utils/cronjobs_osgeo_lxd/README.md b/utils/cronjobs_osgeo_lxd/README.md
@@ -102,6 +102,69 @@ It is controlled via `cron_job_list_grass` on the server in:
 grasslxd:/home/neteler/cronjobs/
 ```
 
-## History
+## Running the scripts in a Docker container
 
-The cronjobs here have been initially written in 2002 and subsequently been updated.
+Using a standard Docker image, we will clone the required repositories
+inside the Docker container, set it up and run the cronjob bash scripts.
+This setup facilitates automated builds, snapshots and handling of multiple
+GRASS GIS versions in a Dockerised environment for continuous integration
+or release management.
+
+```bash
+# using ubuntu:22.04 as ubuntu-latest is still missing PDAL
+docker run -it ubuntu:22.04 bash
+```
+
+Run the following within docker:
+
+```bash
+export DEBIAN_FRONTEND=noninteractive
+export TZ=Etc/UTC
+apt update -y && apt install git gettext python3-sphinx doxygen graphviz zip -y
+
+USER=`id -u -n`
+MAINDIR=/home/$USER
+SOURCE=$MAINDIR/src/
+mkdir -p $SOURCE $MAINDIR/cronjobs/
+
+# GRASS GIS addons: get a shallow clone into docker container
+cd $SOURCE
+git clone --depth=1 --branch grass8 https://github.com/OSGeo/grass-addons.git grass8-addons
+# add links to cronjob scripts
+(cd $MAINDIR/cronjobs/ \
+    && ln -s $SOURCE/grass8-addons/utils/cronjobs_osgeo_lxd/*.sh .)
+(cd $MAINDIR/cronjobs/ \
+    && ln -s $SOURCE/grass8-addons/utils/cronjobs_osgeo_lxd/*.py .)
+
+# needed for script runs in docker, to avoid major path complexity in scripts
+ln -s /home/root/src /root/src
+ln -s $MAINDIR/cronjobs /root/cronjobs
+
+# repo release branches: get a shallow clone into docker container
+for REPO in releasebranch_7_8 releasebranch_8_3 releasebranch_8_4 ; do
+ git clone -b $REPO --single-branch --depth=1 \
+     https://github.com/OSGeo/grass.git $REPO
+done
+
+# repo main branch: get a shallow clone into docker container
+BRANCH=main
+git clone --single-branch --depth=1 https://github.com/OSGeo/grass.git $BRANCH
+
+# install dependencies
+apt install $(cat $SOURCE/$BRANCH/.github/workflows/apt.txt) -y
+
+# define python3 = python
+update-alternatives --install /usr/bin/python python /usr/bin/python3 1
+
+cd $SOURCE/grass8-addons/utils/cronjobs_osgeo_lxd/
+
+# run all cronjob scripts
+bash cron_grass_current_stable_build_binaries.sh \
+  && bash cron_grass_current_stable_src_snapshot.sh \
+  && bash cron_grass_legacy_build_binaries.sh \
+  && bash cron_grass_legacy_src_snapshot.sh \
+  && bash cron_grass_old_build_binaries.sh \
+  && bash cron_grass_old_src_snapshot.sh \
+  && bash cron_grass_preview_build_binaries.sh \
+  && bash cron_grass_preview_src_snapshot.sh
+```
diff --git a/utils/cronjobs_osgeo_lxd/compile_addons_git.sh b/utils/cronjobs_osgeo_lxd/compile_addons_git.sh
@@ -151,8 +151,8 @@ for c in "db" "display" "general" "gui/wxpython" "imagery" "misc" "raster" "rast
         fi
     fi
     echo "<tr><td><tt>$c/$m</tt></td>" >> "$ADDON_PATH/logs/${INDEX_FILE}.html"
-    make MODULE_TOPDIR="$TOPDIR" clean > /dev/null 2>&1
-    make MODULE_TOPDIR="$TOPDIR" \
+    make -j2 MODULE_TOPDIR="$TOPDIR" clean > /dev/null 2>&1
+    make -j2 MODULE_TOPDIR="$TOPDIR" \
         BIN="$path/bin" \
         HTMLDIR="$path/docs/html" \
         MANBASEDIR="$path/docs/man" \

diff --git a/utils/cronjobs_osgeo_lxd/cron_grass_current_stable_build_binaries.sh b/utils/cronjobs_osgeo_lxd/cron_grass_current_stable_build_binaries.sh
@@ -8,36 +8,34 @@
 ###################################################################
 # how it works:
 # - it updates locally the GRASS source code from github server
-# - configures, compiles
+# - configures source code and then compiles it
 # - packages the binaries
 # - generated the install scripts
 # - generates the pyGRASS 8 HTML manual
 # - generates the user 8 HTML manuals
 # - injects DuckDuckGo search field
 
 # Preparations, on server (neteler@grasslxd:$):
-#  - Install PROJ incl Datum shift grids
-#  - Install GDAL
-#  - Install apt-get install texlive-latex-extra python3-sphinxcontrib.apidoc
-#  - Clone source from github:
-#    mkdir -p ~/src ; cd ~/src
-#    git clone https://github.com/OSGeo/grass.git releasebranch_8_4
-#    cd releasebranch_8_4
-#    git checkout releasebranch_8_4
-#  - Prepare target directories:
-#    cd /var/www/code_and_data/
-#    mkdir grass84
-#    cd /var/www/html/
-#    ln -s /var/www/code_and_data/grass84 .
+# - install dependencies:
+#     cd $HOME/src/releasebranch_8_4/ && git pull && sudo apt install $(cat .github/workflows/apt.txt)
+# - install further dependencies:
+#     apt-get install texlive-latex-extra python3-sphinxcontrib.apidoc
+# - run this script
+# - one time only: cross-link code into web space on grasslxd server:
+#     cd /var/www/html/
+#     ln -s /var/www/code_and_data/grass84 .
 #
 #################################
 # variables for build environment (grass.osgeo.org specific)
-MAINDIR=/home/neteler
+USER=`id -u -n`
+MAINDIR=/home/$USER
 PATH=$MAINDIR/bin:/bin:/usr/bin:/usr/local/bin
 
+# https://github.com/OSGeo/grass/tags
 GMAJOR=8
 GMINOR=4
 GPATCH="0dev"  # required by grass-addons-index.sh
+BRANCH=releasebranch_${GMAJOR}_${GMINOR}
 DOTVERSION=$GMAJOR.$GMINOR
 VERSION=$GMAJOR$GMINOR
 GVERSION=$GMAJOR
@@ -53,13 +51,12 @@ LDFLAGSSTRING='-s'
 # define GRASS GIS build related paths:
 # where to find the GRASS sources (git clone):
 SOURCE=$MAINDIR/src/
-BRANCH=releasebranch_${GMAJOR}_$GMINOR
 GRASSBUILDDIR=$SOURCE/$BRANCH
 TARGETMAIN=/var/www/code_and_data
 TARGETDIR=$TARGETMAIN/grass${VERSION}/binary/linux/snapshot
 TARGETHTMLDIR=$TARGETMAIN/grass${VERSION}/manuals/
 
-# progman not built for older dev versions or old stable, only for preview
+# progman not built for older dev versions or old stable, only for preview version
 #TARGETPROGMAN=$TARGETMAIN/programming${GVERSION}
 
 MYBIN=$MAINDIR/binaries
@@ -80,6 +77,12 @@ halt_on_error()
 # function to configure for compilation
 configure_grass()
 {
+# be sure the targetdir exists
+mkdir -p $TARGETDIR
+
+# be sure to be on the right branch
+cd $SOURCE/$BRANCH/
+git checkout $BRANCH
 
 # cleanup from previous run
 rm -f config_$GMAJOR.$GMINOR.git_log.txt
@@ -135,6 +138,7 @@ git fetch --all --prune && git checkout $BRANCH && git pull --rebase || halt_on_
 git status
 
 # for the "contributors list" in old CMSMS (still needed for hugo?)
+mkdir -p $TARGETMAIN/uploads/grass/
 cp -f *.csv $TARGETMAIN/uploads/grass/
 
 # configure for compilation
@@ -164,7 +168,7 @@ $MYMAKE sphinxdoclib
 echo "Copy over the manual + pygrass HTML pages:"
 mkdir -p $TARGETHTMLDIR
 mkdir -p $TARGETHTMLDIR/addons # indeed only relevant the very first compile time
-# don't destroy the addons
+# don't destroy the addons during update
 \mv $TARGETHTMLDIR/addons /tmp
 rm -f $TARGETHTMLDIR/*.*
 (cd $TARGETHTMLDIR ; rm -rf barscales colortables icons northarrows)
@@ -212,9 +216,8 @@ cd $GRASSBUILDDIR/
 #(cd $TARGETPROGMAN/ ; ln -s index.html main.html)
 #### end unused
 
-# note: from G82+ onwards the gettext POT files are managed in git and OSGeo Weblate
-
-##### generate i18N stats for HTML page path (WebSVN):
+##### generate i18N stats for HTML page path:
+# note: the gettext POT files are managed in git and OSGeo Weblate
 ## Structure:  grasslibs_ar.po 144 translated messages 326 fuzzy translations 463 untranslated messages.
 cd $GRASSBUILDDIR
 (cd locale/ ;
@@ -274,21 +277,21 @@ cd $GRASSBUILDDIR
 
 # update addon repo (addon repo has been cloned twice on the server to
 # have separate grass7 and grass8 addon compilation)
-(cd ~/src/grass$GMAJOR-addons/; git checkout grass$GMAJOR; git pull origin grass$GMAJOR)
+(cd $SOURCE/grass$GMAJOR-addons/; git checkout grass$GMAJOR; git pull origin grass$GMAJOR)
 # compile addons
 cd $GRASSBUILDDIR
-sh ~/cronjobs/compile_addons_git.sh $GMAJOR \
+sh $MAINDIR/cronjobs/compile_addons_git.sh $GMAJOR \
    $GMINOR \
-   ~/src/grass$GMAJOR-addons/src/ \
-   ~/src/$BRANCH/dist.$ARCH/ \
-   ~/.grass$GMAJOR/addons \
-   ~/src/$BRANCH/bin.$ARCH/grass \
+   $SOURCE/grass$GMAJOR-addons/src/ \
+   $SOURCE/$BRANCH/dist.$ARCH/ \
+   $MAINDIR/.grass$GMAJOR/addons \
+   $SOURCE/$BRANCH/bin.$ARCH/grass \
    1
 mkdir -p $TARGETHTMLDIR/addons/
 # copy individual addon html files into one target dir if compiled addon
-# has own dir e.g. ~/.grass8/addons/db.join/ with bin/ docs/ etc/ scripts/
+# has own dir e.g. $MAINDIR/.grass8/addons/db.join/ with bin/ docs/ etc/ scripts/
 # subdir
-for dir in `find ~/.grass$GMAJOR/addons -maxdepth 1 -type d`; do
+for dir in `find $MAINDIR/.grass$GMAJOR/addons -maxdepth 1 -type d`; do
     if [ -d $dir/docs/html ] ; then
         if [ "$(ls -A $dir/docs/html/)" ]; then
             for f in $dir/docs/html/*; do
@@ -297,7 +300,7 @@ for dir in `find ~/.grass$GMAJOR/addons -maxdepth 1 -type d`; do
         fi
     fi
 done
-sh ~/cronjobs/grass-addons-index.sh $GMAJOR $GMINOR $GPATCH $TARGETHTMLDIR/addons/
+sh $MAINDIR/cronjobs/grass-addons-index.sh $GMAJOR $GMINOR $GPATCH $TARGETHTMLDIR/addons/
 # copy over hamburger menu assets
 cp $TARGETHTMLDIR/grass_logo.png \
    $TARGETHTMLDIR/hamburger_menu.svg \
@@ -306,13 +309,13 @@ cp $TARGETHTMLDIR/grass_logo.png \
    $TARGETHTMLDIR/addons/
 chmod -R a+r,g+w $TARGETHTMLDIR 2> /dev/null
 
-# copy over logs from ~/.grass$GMAJOR/addons/logs/
+# copy over logs from $MAINDIR/.grass$GMAJOR/addons/logs/
 mkdir -p $TARGETMAIN/addons/grass$GMAJOR/logs/
-cp -p ~/.grass$GMAJOR/addons/logs/* $TARGETMAIN/addons/grass$GMAJOR/logs/
+cp -p $MAINDIR/.grass$GMAJOR/addons/logs/* $TARGETMAIN/addons/grass$GMAJOR/logs/
 
 # generate addons modules.xml file (required for g.extension module)
-~/src/$BRANCH/bin.$ARCH/grass --tmp-project EPSG:4326 --exec ~/cronjobs/build-xml.py --build ~/.grass$GMAJOR/addons
-cp ~/.grass$GMAJOR/addons/modules.xml $TARGETMAIN/addons/grass$GMAJOR/modules.xml
+$SOURCE/$BRANCH/bin.$ARCH/grass --tmp-project EPSG:4326 --exec $MAINDIR/cronjobs/build-xml.py --build $MAINDIR/.grass$GMAJOR/addons
+cp $MAINDIR/.grass$GMAJOR/addons/modules.xml $TARGETMAIN/addons/grass$GMAJOR/modules.xml
 
 # regenerate keywords.html file with addons modules keywords
 export ARCH
@@ -332,13 +335,13 @@ python3 $HOME/src/grass$GMAJOR-addons/utils/create_manuals_sitemap.py --dir=/var
 # cleanup
 cd $GRASSBUILDDIR
 $MYMAKE distclean  > /dev/null || (echo "$0: an error occurred" ; exit 1)
-rm -rf lib/html/ lib/latex/
+rm -rf lib/html/ lib/latex/ /tmp/addons
 
 echo "Finished GRASS $VERSION $ARCH compilation."
 echo "Written to: $TARGETDIR"
 echo "Copied HTML ${GVERSION} manual to https://grass.osgeo.org/grass${VERSION}/manuals/"
 echo "Copied pygrass progman ${GVERSION} to https://grass.osgeo.org/grass${VERSION}/manuals/libpython/"
-## echo "Copied HTML ${GVERSION} progman to https://grass.osgeo.org/programming${GVERSION}"
 echo "Copied Addons ${GVERSION} to https://grass.osgeo.org/grass${VERSION}/manuals/addons/"
+## echo "Copied HTML ${GVERSION} progman to https://grass.osgeo.org/programming${GVERSION}"
 
 exit 0
diff --git a/utils/cronjobs_osgeo_lxd/cron_grass_current_stable_src_snapshot.sh b/utils/cronjobs_osgeo_lxd/cron_grass_current_stable_src_snapshot.sh
@@ -1,6 +1,6 @@
 #!/bin/sh
 
-# script to build GRASS GIS new current sources package from the main branch
+# script to build GRASS GIS new current sources package from the `release_branch_8_4` branch
 # (c) 2002-2024, GPL 2+ Markus Neteler <[email protected]>
 #
 # GRASS GIS github, https://github.com/OSGeo/grass
@@ -10,19 +10,21 @@
 # - it updates locally the GRASS source code from github server
 # - packages the source code tarball
 #
-# Preparations, on server (neteler@grasslxd:$):
-#   mkdir -p ~/src
-#   cd ~/src
-#   git clone https://github.com/OSGeo/grass.git release_branch_8_4
+# To be executed on server (neteler@grasslxd:$)
+# - install dependencies:
+#   cd $HOME/src/release_branch_8_4/ && git pull && sudo apt install $(cat .github/workflows/apt.txt)
+# - run this script
 #
 ###################################################################
-# variables for packaging environment (grass.osgeo.org specific)
-MAINDIR=/home/neteler
+# variables for src packaging environment (grass.osgeo.org specific)
+USER=`id -u -n`
+MAINDIR=/home/$USER
 PATH=$MAINDIR/bin:/bin:/usr/bin:/usr/local/bin
 
 # https://github.com/OSGeo/grass/tags
 GMAJOR=8
 GMINOR=4
+BRANCH=releasebranch_${GMAJOR}_${GMINOR}
 GVERSION=$GMAJOR.$GMINOR.git
 DOTVERSION=$GMAJOR.$GMINOR
 GSHORTGVERSION=$GMAJOR$GMINOR
@@ -33,7 +35,6 @@ set -e
 ###################
 # where to find the GRASS sources (git clone):
 SOURCE=$MAINDIR/src/
-BRANCH=main
 # where to put the resulting .tar.gz file:
 TARGETMAIN=/var/www/code_and_data/
 TARGETDIR=$TARGETMAIN/grass${GSHORTGVERSION}/source/snapshot
@@ -56,10 +57,12 @@ halt_on_error()
 # create a source code snapshot:
 CWD=`pwd`
 
+# setup source code repo
 mkdir -p $TARGETDIR
 cd $SOURCE/$BRANCH/
-date
 
+# be sure to be on the right branch
+git checkout $BRANCH
 # clean up from previous run
 touch include/Make/Platform.make
 $MYMAKE distclean > /dev/null 2>&1
@@ -73,9 +76,6 @@ rm -f config_*.git_log.txt ChangeLog
 # reset i18N POT files to git, just to be sure
 git checkout locale/templates/*.pot
 
-## hard reset local git repo (just in case)
-#git checkout main && git reset --hard HEAD~1 && git reset --hard origin
-
 echo "git update..."
 git fetch --all --prune       || halt_on_error "git fetch error!"
 # we dont have upstream in this cronjob repo