Log SMC progress via Python logger when progress bar deactivated

[jucor]

Description

Output terminals that do not support cursor movement ANSI control sequences behave very badly with Rich ProgressBars, and thus need progressbar=false. See discussion in #7233 (comment)

However, even in those cases it is still very worthy to know how the SMC tasks are progressing.

Therefore, in this PR, when progress bars are deactivated, any change to the state of the chains is logged to the default logger, using the exact same state dictionary as used for the progressbars. The fact we only log when a change happens means we do not flood the log.

The use of the logging facility also allows the caller to add timestamps in the logging handlers for a sense of elapsing time, or to filter out these progress updates.

Related Issue

• Closes BUG: SMC progress bars do not display correctly when called from RStudio #7409
• Related to Replace fastprogress progress bars with rich #7233

Checklist

• Checked that the pre-commit linting/style checks pass
• Included tests that prove the fix is effective or that the new feature works (no test for logging)
• Added necessary documentation (docstrings and/or example notebooks) (no documentation necessary)
• If you are a pro: each commit corresponds to a relevant logical change

Type of change

• New feature / enhancement
• Bug fix
• Documentation
• Maintenance
• Other (please specify):

---

📚 Documentation preview 📚: https://pymc--7410.org.readthedocs.build/en/7410/

[jucor]

@fonnesbeck That might be of interest to you :)

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.18%. Comparing base (1c532f8) to head (08674b3).
Report is 4 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #7410   +/-   ##
=======================================
  Coverage   92.18%   92.18%           
=======================================
  Files         103      103           
  Lines       17258    17270   +12     
=======================================
+ Hits        15909    15921   +12     
  Misses       1349     1349           

Files	Coverage Δ
pymc/smc/sampling.py	99.28% <100.00%> (+0.04%)	⬆️

... and 1 file with indirect coverage changes

[jucor]

@zaxtax @ricardoV94 Would you please be able to review, given how you were the reviewers for #7233 and thus already know that part of the code? Thank you :)

[ricardoV94]

I'm still not convinced the change to rich progressbar was a net positive :)

[aloctavodia]

LGTM!

[jucor]

@aloctavodia @ricardoV94 @aseyboldt Sorry to nudge, but do you think this could be merged, please?

[zaxtax]

That sounds reasonable!

…

On Sun, 11 Aug 2024, 20:30 Julien Cornebise, ***@***.***> wrote: @aloctavodia <https://github.com/aloctavodia> @ricardoV94 <https://github.com/ricardoV94> @aseyboldt <https://github.com/aseyboldt> Do you think this could be merged, please? — Reply to this email directly, view it on GitHub <#7410 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAACCUNA3CU62J427ET2ZRDZQ6USXAVCNFSM6AAAAABKVNWFZWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBSHA2DQMBWGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jucor]

👋 Any help merging this please? I believe @ricardoV94 requested a review by @aseyboldt on July 12th, but I imagine Adrian is possibly swamped with less trivial PRs :) Any chance to merge this please?

[ricardoV94]

Is PyMC log info showing by default? Asking because usually progressbar is disabled when people don't want any output.

[jucor]

Good point, thanks for raising it. I believe the progressbar argument means just that: "do you want a progressbar?", and does not say anything about any other way to log.
I've checked the codebase, and found several examples confirming this:

• In SMC initialization:

  pymc/pymc/smc/sampling.py

  Line 202 in 2856062

  _log.info("Initializing SMC sampler...")

  outputs to the logger regardless of progressbar.
• In the tuning phase of MCMC:

  pymc/pymc/tuning/starting.py

  Line 160 in 2856062

  pm._log.warning(

  outputs to the logger regardless of the value of progressbar.
• and a few others in this search of the codebase.

Seems logical: besides the pretty display of a progressbar, any other information is just part of the use of the standard logging facility, and can be muted or filtered using the core library's logging module.

So if anything, we could argue that I should maybe modify this PR to log to the logger even if progressbar is True. However I do not think that such a "logging + progressbar" is warranted in this case, because:

• the progressbar provides more information than the logging
• and the logging occurs during the inner loop and thus, if printed, would possibly mess up the rendering of the progressbar (I do not know how gracefully rich handles text that is printed by non-rich calls between two updates to its progressbar)