mobu is versioned with semver. Dependencies are updated to the latest available version during each release. Those changes are not noted here explicitly.
Find changes for the upcoming release in the project's changelog.d.
- All app config, including autostart config (and excluding secrets, which still come from env vars) now comes from a single YAML file, provisioned by a single
ConfigMapin Phalanx.
- Update to newer rubin-nublado-client that reports exceptions better
- cell["id"] is not always set: be more cautious
- Replace internal client with rubin-nublado-client from
nubladorepo.
- Remove
exclude_dirsoption fromNotebookRunneroptions, which means it can no longer be set in the autostart config.exclude_dirsmust be set in an in-repomobu.yamlconfig file.
-
NotebookRunnerbusiness will skip notebooks in environments that do not have the services required for them to run. Required services ban be declared by adding metadata to a notebook. -
Allow specification of log level for individual flocks
- Follow redirections by hand for hub logins as well as lab ones to get XSRF
- Use new safir with upstreamed GitHub model changes
- GitHub CI and refresh app config are now each a separate, all-or-nothing set of config that comes from a mix of a yaml file and env vars. This requires some new and different Helm values in Phalanx (see https://mobu.lsst.io/operations/github_ci_app.html#add-phalanx-configuration)
- The GitHub CI app now takes the scopes it assigns from config values, rather than hardcoding a list of scopes.
- The existing refresh functionality is now a GitHub app integration (from a simple webhook integration). This requires new Phalanx secrets to be sync'd, and a new GitHub app to be added to repos that want the functionality. Special care has been taken to not leave these checks in a forever-in-progress state, even in the case of (graceful) mobu shutdown/restart
- A GitHub app integration to generate GitHub actions checks for commits pushed to notebook repo branches that are part of active PRs. These checks trigger and report on a solitary Mobu run of the changed notebooks in the commit.
- Python 3.12.3 -> 3.12.4
NotebookRunnerflocks can now pick up changes to their notebooks without having to restart the whole mobu process. This refresh can happen via:- GitHub
pushwebhook post to/mobu/github/webhookwith changes to a repo and branch that matches the flock config monkeyflocker refresh <flock>POSTto/mobu/flocks/{flock}/refresh
- GitHub
- NotebookRunner business now runs all notebooks in a repo, at tht root and in all subdirs recursively, by default.
- Add
exclude_dirsoption to NotebookRunner business to list directories in which notebooks will not be run.
- Correctly extract cookies from the middle of the redirect chain caused by initial authentication to a Nublado lab. This fixes failures seen with labs containing jupyterhub 4.1.3.
- Add
GitLFSBusinessfor testing Git LFS by storing and retrieving a Git LFS-managed artifact.
- Properly handle the XSRF tokens for JupyterHub and the Jupyter lab by storing separate tokens for the hub and lab after initial login and sending the appropriate XSRF token in the
X-XSRFTokenheader to the relevant APIs. This fixes a redirect loop at the Jupyter lab when running 4.1.0 or later.
- mobu now uses uv to maintain frozen dependencies and set up a development environment.
- Drop support for cachemachine and Nublado v2. The
cachemachine_image_policyanduse_cachemachineconfiguration options are no longer supported and should be deleted. - Rename the existing
TAPQueryRunnerbusiness toTAPQuerySetRunnerto more accurately capture what it does. Add a newTAPQueryRunnerbusiness that runs queries chosen randomly from a list. Based on work by @stvoutsin. - Rename
JupyterPythonLooptoNubladoPythonLoopto make it explicit that it requires Nublado and will not work with an arbitrary JupyterHub.
- Convert all configuration options that took intervals in seconds to
timedelta. Bare numbers will still be interpreted as a number of seconds, but any format Pydantic recognizes as atimedeltamay now be used.
- All environment variables used to configure mobu now start with
MOBU_, and several have changed their names. The new settings areMOBU_ALERT_HOOK,MOBU_AUTOSTART_PATH,MOBU_ENVIRONMENT_URL,MOBU_GAFAELFAWR_TOKEN,MOBU_NAME,MOBU_PATH_PREFIX,MOBU_LOGGING_PROFILE, andMOBU_LOG_LEVEL. This is handled by the Phalanx application, so no configuration changes should be required.
- Rather than dumping the full monkey data when summarizing flocks, which can cause long enough delays that in-progress calls fail due to the huge amount of timing data, extract only the success and failure count from the running business. This should be considerably faster and avoid timeout problems.
- Improve error reporting by catching exceptions thrown while sending code to the lab WebSocket for execution.
- The timeout when talking to JupyterHub and Jupyter labs can now be configured in the business options (as
jupyter_timeout). The default is now 60s instead of 30s.
- When reporting httpx failures to Slack, put the response body into an attachment instead of a block so that it will be collapsed if long.
- Fix reporting of WebSocket open timeouts to Slack.
- Configuration of whether to use cachemachine and, if so, what image policy to use is now done at the business level instead of globally. This allows the same mobu instance to test both Nublado v2 and Nublado v3.
- The maximum allowable size for a WebSocket message from the Jupyter lab is now configurable per business and defaults to 10MB instead of 4MB.
- Revert change in 5.0.0 to number all cells, and go back to counting only code cells for numbering purposes. This matches the way cell numbers are displayed in the Jupyter lab UI.
- When reporting errors to Slack, mobu 5.0.0 mistakenly started stripping ANSI escape sequences from the code being executed, which should be safe since it comes from local notebooks or configuration, instead of the error output, which is where Jupyter labs like to add formatting. Strip ANSI escape sequences from the error output instead of the code.
- mobu now uses httpx instead of aiohttp for all HTTP requests (including websockets for WebSocket connections and httpx-sse for EventStream connections) and makes use of the Safir framework for parsing and reporting HTTP client exceptions. Alerts for failing web requests will be somewhat different and hopefully clearer.
- mobu now sends keep-alive pings on the WebSocket connection to the lab, hopefully allowing successful execution of cells that take more than five minutes to run.
- Nublado-based businesses can now set
debugto true in the image specification to request that debugging be enabled in the spawned Jupyter lab. - mobu now catches timeouts attempting to open a WebSocket to the lab and reports them to Slack with more details.
- Slack alerts from monkeys now include the flock and monkey name as a field in the alert.
- Unexpected business exceptions now include an "Exception type" heading and use "Failed at" instead of "Date" to match the display of expected exceptions.
- The prefix for mobu routes (
/mobuby default) can now be configured withSAFIR_PATH_PREFIX. - Uncaught exceptions from mobu's route handlers are now also reported to Slack.
- The code to determine the Docker reference and description of the running Nublado image is now more robust against unexpected output.
- Node and cell information in Slack error reports for Nublado errors are now formatted as full blocks rather than fields, since they are often too wide to fit nicely in the limited width of a Slack Block Kit field.
- The default
error_idle_timefor Nublado-based business is back to 60 seconds instead of 10 minutes. The problem the longer timeout was working around should be fixed in the new Nublado lab controller. - Nublado-based notebooks now request the
JUPYTER_IMAGE_SPECenvironment variable instead ofJUPYTER_IMAGEto get the running image for error reporting purposes. This is now the preferred environment variable andJUPYTER_IMAGEis deprecated. - mobu now uses the Ruff linter instead of flake8, isort, and pydocstyle.
- Settings are now handled with Pydantic and undergo much stricter validation. In particular, the Slack web hook URL must now be a valid URL if provided.
- In order to enable stricter and more useful Pydantic validation of flock specifications, the syntax for creating a flock has changed.
businessis now a dictionary, therestartoption has been moved under it, the type of business is specified withtype, and the business configuration options have moved under that key asoptions. Options that are not applicable to a given business type are now rejected. - The
jupyter.url_prefixoption is now justurl_prefix, andjuyter.imageis now justimage. The names of the setting underimagehave changed. - The
TAPQueryRunneroptionstap_syncandtap_query_setare now justsyncandquery_set. lab_settle_timeis no longer supported as a configuration option for the businesses that spawn a Nublado lab. It defaulted to 0 and we never set it.JupyterJitterLoginLoophas been retired. Instead, set thejitteroption onJupyterPythonLoop.JupyterLoginLoophas been merged withJupyterPythonLoop. The only difference in the former is that no lab session was created and no code was run, which seems pointless and not worth the distinction.JupyterPythonLoopruns a simple addition by default, which should be an improvement overJupyterLoginLoopin every likely situation.
- When the production logging profile is used, the messages from monkeys are no longer reported to the main mobu log, only to the individual monkey logs. This should produce considerably less noise in external log aggregators.
- The notebook being run is now included in all Slack error reports, not just for code execution failures.
- The API documentation now shows only the relevant options for the type of business when showing how to create a flock.
- Add support for running a business once and returning its results, via a POST to the new
/runendpoint. - Add support for the new Nublado lab controller (see SQR-066.
- The time a business pauses after a failure before it is restarted is now configurable with the
error_idle_timeoption and defaults to 10 minutes (instead of 1 minute) for Nublado businesses, since this is how long JupyterHub will wait for a lab to spawn before giving up.
- The
dp0.2TAPQueryRunnerquery set is now lighter-weight and will consume less memory and CPU to execute, hopefully reducing timeout errors. - Cell numbering in error reports is now across all cells, not just code cells.
TAPQueryRunnerno longer creates a TAP client in its__init__method, since creating a TAP client makes HTTP requests to the TAP server that can fail and failure would potentially crash mobu. Instead, it creates the TAP client instartupand handles exceptions properly so that they're reported to Slack.- Business failures during
startupare now counted as a failed execution so that a business that fails repeatedly instartupdoesn't report 100% success in the flock summary. - The code run by
JupyterPythonLoopandNotebookRunnerto get the Kubernetes node on which the lab is running now useslsst.rsp.get_nodeinstead of the deprecatedrubin_jupyer_utils.lab.notebook.utils.get_node.
- Slightly improve logging when monkeys are shut down due to errors.
- mobu's internals have been extensively refactored following the design in SQR-072 to hopefully make future maintenance easier.