Check that notifications don't exceed SQS message limit

[whabanks]

Summary | Résumé

Notifications payloads sent to /v2/notifications/{type} are now checked to ensure that their size in bytes does not exceed the SQS message payload size limit.

• Add logging that identifies the largest piece of data when oversized payloads are attempted.
• Add a message informing the API user of the size limit and pointing them to the largest item in their payload as a guide.
• Add a recursive utility method to flatten a dict for easier processing in some situations.

SQS has a payload limit 262,144 bytes, or 256Kib. When Celery attempts to consume inflights that are greater than this size limit it will fail and the message is returned to the inflight inbox. This loop continues until the inflight is expired thus creating a lot of unnecessary Celery errors in our logs.

Related work

• Add error message when notification size > 256KB notification-admin#2025

Test instructions | Instructions pour tester la modification

Test 1

1. Send an email with an attachment via the API using attachment_content.txt for the personalisation.
2. Note the following error is returned:

{
    "status_code": 413,
    "errors": [
        {
            "error": "BadRequestError",
            "message": "Notification size cannot exceed 256Kb. Consider reducing the size of: personalisation.link_to_file.filename"
        }
    ]
}

Test 2

1. Add a variable to the template used in test 1
2. Replace the filename content from the above attachment_content.txt with something reasonable like "filename" and move the previous content into the variable added in step 1
3. Attempt the send
4. Note that the pre-existing personalisation size validation logic still works as expected.

[jzbahrai]

tested here: https://app.zenhub.com/workspaces/notify-planning-614b3ad91bc2030015ed22f5/issues/gh/cds-snc/notification-planning/1411

tbd

[smcmurtry]

Test 1
Running Test 1 on the main branch of notification-api running locally, there no celery errors and everything works as expected, I receive the email with the attachment. So unless I'm missing something, I don't think the SQS message payload limit needs to limit our attachment size to 256Kb.

Test 2
Running Test 2 on the main branch locally, I get the error message:

{'status_code': 400,
 'errors': [{'error': 'ValidationError',
   'message': 'Personalisation variables size of 422422 bytes is greater than allowed limit of 51200 bytes.'}]}

So we have an existing error that handles this case already.

[whabanks]

Closing as investigations determined that we no longer need this additional validation / error messages bubbled up to the front end.