Add MongoDB query timeout

[jnm]

We have an issue where zombie Mongo queries are able to run for several hours—long after any client that requested them has perished. Unfortunately, a global query timeout seems impossible, but maybe we could wrap pymongo somehow so that it sends maxTimeMS with all queries: https://stackoverflow.com/a/60542564/2402324

Whatever solution we arrive at we should implement in KoBoCAT as well.

Somewhat related to kobotoolbox/kobocat#696 in that both that and this work together to cause MongoDB slowdowns, which lead to users getting 502s

[jnm]

Elevated the priority because the servers are really struggling under the load.

As I mentioned earlier, I don't see a way to set maxTimeMS on every query used with our pymongo MONGO_DB, but maybe the easiest thing to do is to make a helper function that wraps MONGO_DB.instances.find() and adds the maxTimeMS argument.

The limit should be CELERY_TASK_TIME_LIMIT (converted to milliseconds) + some grace period

kpi/kobo/settings/base.py

Lines 422 to 426 in 7edbc13

	# Default to a 30-minute soft time limit and a 35-minute hard time limit
	CELERY_TASK_TIME_LIMIT = int(os.environ.get('CELERYD_TASK_TIME_LIMIT', 2100))
	CELERY_TASK_SOFT_TIME_LIMIT = int(os.environ.get(
	'CELERYD_TASK_SOFT_TIME_LIMIT', 1800))

[jnm]

It's more of a sysadmin thing, but for reference, here's a quick and really-very-dirty 🙈 method to reduce load from runaway queries:

root@mongo:/# while true; do mongo -u root -p "$MONGO_INITDB_ROOT_PASSWORD" admin --eval 'db.currentOp(true).inprog.forEach(function(op){ if(op.secs_running > 2110) { print(op.opid); db.killOp(op.opid) } });'; sleep 10; done