Reduce alarms noise filtering out duplicate key errors

[jimleroyer]

Description

We get too much noise in the #notification-ops channel with unique key violation errors. These are expected as we use standard SQS queues which do not guarantee to send one and only one message. Hence finetuning the alarm to filter out a minimal number of duplicate should be be OK while raising an alarm if too many are reported in a short period of time (i.e. a few minutes).

As a person on support,
I need to have less alarms concerning inconsequential duplicate errors key
so that I can achieve focus on higher priority alarms.

WHY are we building?

Less noise in the notification-ops channel and more focus on important alarms.

WHAT are we building?

Better alarms filter.

VALUE created by our solution

More focus and less distraction.

Acceptance Criteria** (Definition of done)

• Low volume unique key violation errors are not getting reported as an alarm anymore.
• Medium and high volume unique key violation errors are getting reported as an alarm anymore.

QA Steps

• Verify that sparse unique key violation errors are not getting reported anymore.
• If possible, reproduce high volume of key violation errors to trigger the error.

Additional context

Patrick wrote:

Yeah, it’s just a whole wack of the expected unique violation warnings:
(psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "notifications_pkey"
I think it would be a good idea to filter these out into their own alert category so we don’t get all the alarm messages for them:
cds-snc/notification-api#1458 (comment)

[yaelberger-commits]

@jimleroyer Is this still happening, duplicate key errors?

[yaelberger-commits]

Hey team! Please add your planning poker estimate with Zenhub @andrewleith @jimleroyer @jzbahrai @Pensai @sastels @smcmurtry

[jimleroyer]

@yaelberger-commits This topic came back this week. Definitely still valid and stealing support focus.

[yaelberger-commits]

Revisit this in January or February to evaluate if changes to New Relic did the job or if more effort is needed

[yaelberger-commits]

@jimleroyer @jzbahrai Has this issue been resolved or are we still seeing too many duplicate key errors?

[yaelberger-commits]

@jimleroyer @jzbahrai Has this issue been resolved or are we still seeing too many duplicate key errors?