From baf7302a322b0a600bb19ced82ba33e8379cb89b Mon Sep 17 00:00:00 2001 From: Helena Rasche Date: Tue, 17 Oct 2023 12:35:48 +0200 Subject: [PATCH 1/5] add FAQ about upgrading through 22.05 --- topics/admin/faqs/galaxy-update-22.05.md | 95 ++++++++++++++++++++++++ 1 file changed, 95 insertions(+) create mode 100644 topics/admin/faqs/galaxy-update-22.05.md diff --git a/topics/admin/faqs/galaxy-update-22.05.md b/topics/admin/faqs/galaxy-update-22.05.md new file mode 100644 index 00000000000000..4dcffe35b823bf --- /dev/null +++ b/topics/admin/faqs/galaxy-update-22.05.md @@ -0,0 +1,95 @@ +--- +title: Updating from 22.01 to 23.0 with Ansible +area: ansible +box_type: tip +layout: faq +contributors: [hexylena] +--- + +Galaxy introduced a number of changes in 22.05 and 23.0 that are extremely important to be aware of during the upgrade process. Namely a new database migration system, and a new required running environment (gunicorn instead of uwsgi). + +Unfortunately the scripts to migrate to the new database migration system were only present in 22.05, and then were subsequently removed, so it is **mandatory** to upgrade to 22.05 if you want to go further. + +Here is the recommended update procedure with ansible: + +1. Update to 22.01 normally +2. Change the release to 22.05, and run the upgrade + 1. Galaxy will probably not start correctly here, ignore it. + 2. Run the database migration manually + + ``` + GALAXY_CONFIG_FILE=/srv/galaxy/config/galaxy.yml sh manage_db.sh -c /srv/galaxy/config/galaxy.yml upgrade + ``` + +3. Update your system's ansible, you probably need something with a major version greater than 2. +3. Set the release to `23.0` and make other required changes. There are a lot of useful changes, but the easiest procedure is probably something like: + + 1. git clone https://github.com/hexylena/git-gat/ + 2. git checkout step-4 + 3. `vimdiff group_vars/galaxyservers.yml git-gat/group_vars/galaxyservers.yml` (repeat for group_vars/all.yml, group_vars/dbservers.yml, galaxy.yml, requirements.yml, hosts, etc.) + + But the main change is the swap from uwsgi to gravity+gunicorn + + ```diff + - uwsgi: + - socket: 127.0.0.1:8080 + - buffer-size: 16384 + - processes: 1 + - threads: 4 + - offload-threads: 2 + - static-map: + - - /static={{ galaxy_server_dir }}/static + - - /favicon.ico={{ galaxy_server_dir }}/static/favicon.ico + - static-safe: client/galaxy/images + - master: true + - virtualenv: "{{ galaxy_venv_dir }}" + - pythonpath: "{{ galaxy_server_dir }}/lib" + - module: galaxy.webapps.galaxy.buildapp:uwsgi_app() + - thunder-lock: true + - die-on-term: true + - hook-master-start: + - - unix_signal:2 gracefully_kill_them_all + - - unix_signal:15 gracefully_kill_them_all + - py-call-osafterfork: true + - enable-threads: true + - mule: + - - lib/galaxy/main.py + - - lib/galaxy/main.py + - farm: job-handlers:1,2 + + gravity: + + process_manager: systemd + + galaxy_root: "{{ galaxy_root }}/server" + + galaxy_user: "{{ galaxy_user_name }}" + + virtualenv: "{{ galaxy_venv_dir }}" + + gunicorn: + + # listening options + + bind: "unix:{{ galaxy_mutable_config_dir }}/gunicorn.sock" + + # performance options + + workers: 2 + + # Other options that will be passed to gunicorn + + # This permits setting of 'secure' headers like REMOTE_USER (and friends) + + # https://docs.gunicorn.org/en/stable/settings.html#forwarded-allow-ips + + extra_args: '--forwarded-allow-ips="*"' + + # This lets Gunicorn start Galaxy completely before forking which is faster. + + # https://docs.gunicorn.org/en/stable/settings.html#preload-app + + preload: true + + celery: + + concurrency: 2 + + loglevel: DEBUG + + handlers: + + handler: + + processes: 2 + + pools: + + - job-handlers + + - workflow-schedulers + ``` + + Some other important changes include: + - uchida.miniconda is replaced with galaxyproject.conda + - usegalaxy_eu.systemd is no longer needed + - `galaxy_user_name` is defined in all.yml in the latest git-gat + - git-gat also separates out the DB serving into a `dbservers.yml` host group + +4. Backup your `venv`, `mv /srv/galaxy/venv/ /srv/galaxy/venv-old/`, as your NodeJS is probably out of date and Galaxy doesn't handle that gracefully +5. Do any local customs for luck (knocking on wood, etc.) +6. Run the playbook From 60bbf6acb09608a4229b11e753c4d910ef8e4040 Mon Sep 17 00:00:00 2001 From: Helena Date: Tue, 17 Oct 2023 13:04:28 +0200 Subject: [PATCH 2/5] Update topics/admin/faqs/galaxy-update-22.05.md Co-authored-by: Marius van den Beek --- topics/admin/faqs/galaxy-update-22.05.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/topics/admin/faqs/galaxy-update-22.05.md b/topics/admin/faqs/galaxy-update-22.05.md index 4dcffe35b823bf..3f4001c6d30513 100644 --- a/topics/admin/faqs/galaxy-update-22.05.md +++ b/topics/admin/faqs/galaxy-update-22.05.md @@ -8,7 +8,7 @@ contributors: [hexylena] Galaxy introduced a number of changes in 22.05 and 23.0 that are extremely important to be aware of during the upgrade process. Namely a new database migration system, and a new required running environment (gunicorn instead of uwsgi). -Unfortunately the scripts to migrate to the new database migration system were only present in 22.05, and then were subsequently removed, so it is **mandatory** to upgrade to 22.05 if you want to go further. +The scripts to migrate to the new database migration system are only compatible with release 22.05, and then were subsequently removed, so it is **mandatory** to upgrade to 22.05 if you want to go further. Here is the recommended update procedure with ansible: From cb9cf15579e222e25750242591381dace425fec8 Mon Sep 17 00:00:00 2001 From: Helena Rasche Date: Tue, 17 Oct 2023 13:23:13 +0200 Subject: [PATCH 3/5] add missing vars --- topics/admin/faqs/galaxy-update-22.05.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/topics/admin/faqs/galaxy-update-22.05.md b/topics/admin/faqs/galaxy-update-22.05.md index 3f4001c6d30513..3362f8c1538456 100644 --- a/topics/admin/faqs/galaxy-update-22.05.md +++ b/topics/admin/faqs/galaxy-update-22.05.md @@ -90,6 +90,15 @@ Here is the recommended update procedure with ansible: - `galaxy_user_name` is defined in all.yml in the latest git-gat - git-gat also separates out the DB serving into a `dbservers.yml` host group + You may also need to add + + ``` + galaxy_systemd_mode: 'gravity' + galaxy_manage_systemd: yes + ``` + + to your galaxy configuration to have systemd units properly installed. + 4. Backup your `venv`, `mv /srv/galaxy/venv/ /srv/galaxy/venv-old/`, as your NodeJS is probably out of date and Galaxy doesn't handle that gracefully 5. Do any local customs for luck (knocking on wood, etc.) 6. Run the playbook From ca0889b4fb99153a3a9a5cca362d96c731b65cb4 Mon Sep 17 00:00:00 2001 From: Helena Rasche Date: Tue, 17 Oct 2023 13:43:08 +0200 Subject: [PATCH 4/5] finally alive --- topics/admin/faqs/galaxy-update-22.05.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/topics/admin/faqs/galaxy-update-22.05.md b/topics/admin/faqs/galaxy-update-22.05.md index 3362f8c1538456..fbd12fb8aa174b 100644 --- a/topics/admin/faqs/galaxy-update-22.05.md +++ b/topics/admin/faqs/galaxy-update-22.05.md @@ -26,7 +26,14 @@ Here is the recommended update procedure with ansible: 1. git clone https://github.com/hexylena/git-gat/ 2. git checkout step-4 - 3. `vimdiff group_vars/galaxyservers.yml git-gat/group_vars/galaxyservers.yml` (repeat for group_vars/all.yml, group_vars/dbservers.yml, galaxy.yml, requirements.yml, hosts, etc.) + 3. Diff and sync (e.g. `vimdiff group_vars/galaxyservers.yml git-gat/group_vars/galaxyservers.yml`) for the main configuration files: + + - group_vars/all.yml + - group_vars/dbservers.yml + - galaxy.yml + - requirements.yml + - hosts + - templates/nginx/galaxy.j2 But the main change is the swap from uwsgi to gravity+gunicorn @@ -102,3 +109,7 @@ Here is the recommended update procedure with ansible: 4. Backup your `venv`, `mv /srv/galaxy/venv/ /srv/galaxy/venv-old/`, as your NodeJS is probably out of date and Galaxy doesn't handle that gracefully 5. Do any local customs for luck (knocking on wood, etc.) 6. Run the playbook +7. Things might go wrong with systemd units + - try running `galaxyctl -c /srv/galaxy/config/galaxy.yml update` as root + - you may also need to `rm /etc/systemd/system/galaxy.service` which is then no longer needed + - you'll have a `galaxy.target` and you can instead `systemctl daemon-reload` and `systemctl start galaxy.target` From 8759b620d9d3674c3aaecdf0e24513646f3d219e Mon Sep 17 00:00:00 2001 From: Helena Rasche Date: Tue, 17 Oct 2023 13:50:22 +0200 Subject: [PATCH 5/5] ok not needed? --- topics/admin/faqs/galaxy-update-22.05.md | 9 --------- 1 file changed, 9 deletions(-) diff --git a/topics/admin/faqs/galaxy-update-22.05.md b/topics/admin/faqs/galaxy-update-22.05.md index fbd12fb8aa174b..1ae76720aec2dc 100644 --- a/topics/admin/faqs/galaxy-update-22.05.md +++ b/topics/admin/faqs/galaxy-update-22.05.md @@ -97,15 +97,6 @@ Here is the recommended update procedure with ansible: - `galaxy_user_name` is defined in all.yml in the latest git-gat - git-gat also separates out the DB serving into a `dbservers.yml` host group - You may also need to add - - ``` - galaxy_systemd_mode: 'gravity' - galaxy_manage_systemd: yes - ``` - - to your galaxy configuration to have systemd units properly installed. - 4. Backup your `venv`, `mv /srv/galaxy/venv/ /srv/galaxy/venv-old/`, as your NodeJS is probably out of date and Galaxy doesn't handle that gracefully 5. Do any local customs for luck (knocking on wood, etc.) 6. Run the playbook