omero admin mail does not wait

[pwalczysko]

When trying to send an email to some 300 users on the nightshade server, with the option --wait -1 the action times out after 25 secs with the error. The connection is done remotely via OMERO.cli as a client on local laptop.

$ omero admin email --everyone  "OMERO server not accessible" < email-ns.users.txt
Using session for [email protected]:4064. Idle timeout: 10 min. Current group: Swedlow Lab
Traceback (most recent call last):
  File "/Users/pwalczysko/miniconda3/envs/env563/bin/omero", line 8, in <module>
    sys.exit(main())
  File "/Users/pwalczysko/miniconda3/envs/env563/lib/python3.6/site-packages/omero/main.py", line 125, in main
    rv = omero.cli.argv()
  File "/Users/pwalczysko/miniconda3/envs/env563/lib/python3.6/site-packages/omero/cli.py", line 1784, in argv
    cli.invoke(args[1:])
  File "/Users/pwalczysko/miniconda3/envs/env563/lib/python3.6/site-packages/omero/cli.py", line 1222, in invoke
    stop = self.onecmd(line, previous_args)
  File "/Users/pwalczysko/miniconda3/envs/env563/lib/python3.6/site-packages/omero/cli.py", line 1299, in onecmd
    self.execute(line, previous_args)
  File "/Users/pwalczysko/miniconda3/envs/env563/lib/python3.6/site-packages/omero/cli.py", line 1381, in execute
    args.func(args)
  File "/Users/pwalczysko/miniconda3/envs/env563/lib/python3.6/site-packages/omero/plugins/admin.py", line 1601, in email
    failonerror=True, failontimeout=True)
  File "/Users/pwalczysko/miniconda3/envs/env563/lib/python3.6/site-packages/omero/clients.py", line 1001, in submit
    closehandle=True)
  File "/Users/pwalczysko/miniconda3/envs/env563/lib/python3.6/site-packages/omero/clients.py", line 1020, in waitOnCmd
    callback.loop(loops, ms)  # Throw LockTimeout
  File "/Users/pwalczysko/miniconda3/envs/env563/lib/python3.6/site-packages/omero/callbacks.py", line 260, in loop
    5000, int(waited))
omero.LockTimeout: exception ::omero::LockTimeout
{
    serverStackTrace = None
    serverExceptionClass = None
    message = Command unfinished after 25.0 seconds
    backOff = 5000
    seconds = 25
}

This is the same error experienced when no --wait parameter is passed at all.

Happy to investigate further, but need some hints about how to establish a testing system

cc @joshmoore @sbesson

[joshmoore]

merge-ci's OMERO-test-integration job is set up to create annotations rather than actually send emails.

omero config set omero.mail.config true
omero config set omero.mail.fake true
omero config set omero.mail.port 2525

[sbesson]

Looking at implementations, it looks like the key difference is that OMERO.py uses client.submit() with its built-in callback handling and the maximum waiting time capped at 25 independently of the value of --wait:

https://github.com/ome/omero-py/blob/24fcf1de625574dcdf5894f1ff1b35b0797cb198/src/omero/plugins/admin.py#L1586-L1609

On the other hand, OMERO.web uses the underlying session.submit with no callback:

https://github.com/ome/omero-web/blob/34c68446920830b8b68fdf3f62542abfe2b72a68/omeroweb/webadmin/views.py#L1161-L1177

I assume we could use an internal system with mail set-up pub-omero for reproducing the initial scenario. This might require setting up temporary email addresses