feature/grace seconds

[IhorMarchenkoRakuten]

This is a feature to add support for grace duration in seconds. Once grace is set, alerts will be ignored until the seconds are over and then start working usually with throttling. The default for grace is 0 sec.

[kevincobain2000]

Doesn't work

func main() {
  os.Setenv("APP_NAME", "123")
  os.Setenv("MS_TEAMS_ALERT_ENABLED", "test")
  os.Setenv("MS_TEAMS_WEBHOOK", "true")
  os.Setenv("THROTTLE_DURATION", "1")
  os.Setenv("THROTTLE_GRACE_SECONDS", "20")
  t := n.NewThrottler()
  t.CleanThrottlingCache()

  err := errors.New("ignore 000")
  alert := n.NewAlert(err, nil)
  alert.Notify()

  err = errors.New("ignore 000")
  alert = n.NewAlert(err, nil)
  alert.Notify()
  alert.Notify()

  err = errors.New("ignore 000")
  alert = n.NewAlert(err, nil)
  alert.Notify()
} 

Output

func (a *Alert) Notify() (err error) {
  fmt.Println("should alert:", a.shouldAlert())

╰─$ go run main.go
should alert: true <-- this should have been false, because the grace is 20 seconds
should alert: false

[chetra-tep]

Not in the correct place here. Grace's cache key should be set when Throttle's key is set.

[IhorMarchenkoRakuten]

Thank you, let me try it

[IhorMarchenkoRakuten]

Hi @chetra-tep,
I have optimized the function and tried different approaches for the Cache key. The key is set in the same function just before the Throttle key. So if I put the function into the Throttle function, then it will be more difficult to check the Grace period on the first iteration.

I think there is not much difference if it's in another function since they are still in the same parent function IsThrottledOrGraced, and will be executed sequentially. Is there any scenario when the key shouldn't be handled at that place?

Please let me know what you think or if I misunderstood anything

[kevincobain2000]

I ll test it first.

[kevincobain2000]

Doesn't work

func main() {
	os.Setenv("APP_NAME", "123")
	os.Setenv("MS_TEAMS_ALERT_ENABLED", "test")
	os.Setenv("MS_TEAMS_WEBHOOK", "true")
	os.Setenv("THROTTLE_DURATION", "1")
	os.Setenv("THROTTLE_GRACE_SECONDS", "5")
	t := n.NewThrottler()
	t.CleanThrottlingCache()

	err := errors.New("ignore 000")
	alert := n.NewAlert(err, nil)
	alert.Notify()              // false, caught by grace
	time.Sleep(6 * time.Second) // wait for grace to go over

	alert.Notify() // true, first alert
	alert.Notify() // false, caught by throttle
	time.Sleep(61 * time.Second)
	alert.Notify()              // false, caught by grace
	time.Sleep(6 * time.Second) // wait for grace to go over
	alert.Notify()              // true
	alert.Notify()              // false, caught by throttle
}

should alert: false
should alert: true
should alert: false
should alert: false
should alert: false <-- this should be true
should alert: false

[IhorMarchenkoRakuten]

thank you, I've adjusted the grace reset, so it should work now

[kevincobain2000]

Will test. Thanks.

[kevincobain2000]

Looks good