Event Handler failure after 17.x

[programmerq]

After upgrading to Teleport 17.0.5, the event handler started throwing this error in a loop:

ERRO Unexpected error in watch loop. Reconnecting in 5s... error:[
ERROR REPORT:
Original Error: *interceptors.RemoteError rpc error: code = Unknown desc = json: cannot unmarshal object into Go struct field checkpointKey.iterator of type string
Stack Trace:
    github.com/gravitational/teleport/apiav0.0.0/client/client.go:2526 github.com/gravitational/teleport/api/client.(*Client).SearchUnstructuredEvents
    github.com/gravitational/teleport/integrations/event-handler/legacy_events_watcher.go:269 main.(*LegacyEventsWatcher).getEventsInWindow
    github.com/gravitational/teleport/integrations/event-handler/legacy_events_watcher.go:193 main.(*LegacyEventsWatcher).getEvents
    github.com/gravitational/teleport/integrations/event-handler/legacy_events_watcher.go:139 main.(*LegacyEventsWatcher). fetch
    github.com/gravitational/teleport/integrations/event-handler/legacy_events_watcher.go:320 main.(*LegacyEventsWatcher).ExportEvents.funcl
    runtime/asm_amd64.s:1700 runtime.goexit
User Message: rpc error: code = Unknown desc = json: cannot unmarshal object into Go struct field checkpointKey.iterator of type string] event-handler/events_job.go:97

Both the cluster and the event handler were running 17.0.5.

I attempted to reproduce the issue unsuccessfully. Creating this issue to track this or similar stack traces. Will fill further details if a set of steps to reproduce are found.

[fspmarshall]

From initial investigation, it looks like this error likely originates from here. The iterator field typically contains a map[string]any that has been json-encoded into a string. The resulting string should then be properly quoted automatically when the outer checkpointKey is marshaled.

At a glance, this error seems to be indicating that the object in the iterator field has somehow become "unquoted" s.t. the outer logic now fails to unmarshal it as a string, but from some initial poking around I can't find any possible culprits for that.

If we see more folks hitting this, getting them to dump the contents of the event-handler's cursor file as it's stored on disk might give us more to go on.