RFC/Feature Request: vtorc should make it easy to find audit info related to recoveries

[ejortegau]

Feature Description

Context & prior art

Classical Orchestrator had under /web/audit-recovery/uid/<recoveryid> some details about recoveries and steps taken as part of each one of them. This allowed an operator to easily see whether the recovery had succeeded or not, as well as some information about each step taken:

Furthermore, if one had more than one Orchestrator instance (and set them up with the shared MySQL database - not sure how this worked on the raft setup with independent local databases as I never used it), no matter which Orchestrator instance was checked, the same information would be visible.

In vtorc, however, this information is (at least currently) not so clearly surfaced to the user anywhere. The current vtorc web UI only seems to show a list of recoveries with no details on each one:

Furthermore, since there is no shared state, and no cluster leader, users need to check each individual vtorc instance to see all the recoveries. This is IMHO a considerable feature gap when it comes to observability.

Feature Request

There should be some easy means to centrally view what recoveries took place and how each step of them went.

RFC

There are multiple ways to try to solve the issue above:

• Emit clearly distiniguishable log entries for recoveries and delegate visualization to the user's logging stack (e.g. kibana). No other changes needed/most likely. For example, all log entries of interest related to recoveries could have a prefix like <Problem> Recovery <keyspace>/<shard>: (e.g. DeadPrimary Recovery commerce/80-: ). This is probably the lowest effort one to implement, but assumes the existence of logging processing & visualization infrastructure.
• vtorc implements a new web UI endpoint for the information. This comes with some challenges:
  • For users running multiple vtorc instances:
    • The user needs to visit the new web UI for each instance.
    • Or the vtcorcs need to know about each other and the one serving the web request fetches the information from the rest to present a single view. This woul impliy that, in addition to the UI, vtorc should also expose recovery details view APi endpoint that can be queried by the other vtorcs to consolidate information from all of them.
  • Data persistence: vtorc currently uses an sqlite DB which defaults to in-memory storage. Care should be taken to ensure it's kept across vtorc process restarts if the user wants to not lose the recovery details across restarts. E.g., use an actual sqlite file instead of inmemory, and, if running in k8s, use a persistent volume claim for its path.
• vtadmin adds functionality to query all vtorc instances, aggregates the information about recoveries from all instances and shows it to the user in a new web UI. This requires registering all vtorc instances somewhere so that vrtadmin can find them. I guess that would be the topology? In this scenario, the user should still take care of data persistency for the vtorc DB.

The last two approaches are more feature-rich/complete for the user as there's no need/assumption of existence of logging pipeline, but obviously require more work. Also, perhaps the first approach could be done as a stop gap measure while the second or third one are done.

Thoughts?

Use Case(s)

As a vites operator, I want to be able to easilky check whether vtorc detected and attempted to recover a problem, and the steps taken during that recovery.

[timvaillancourt]

vtorc implements a new web UI endpoint for the information. This comes with some challenges

VTOrc seems to be designed to be very ephemeral. As you pointed out, the backend data is ephemeral, also the vtorc daemon's themselves (ie: their hostname/pod) are ephemeral. I like this choice and hope an individual VTOrc remains unimportant. I think this architecture makes them not ideal for a stable UI, however

I guess what I'm saying here is if an improved UI is made, I feel it should be located somewhere "stable" and more built for the purpose such as VTAdmin

Data persistence: vtorc currently uses an sqlite DB which defaults to in-memory storage. Care should be taken to ensure it's kept across vtorc process restarts if the user wants to not lose the recovery details across restarts. E.g., use an actual sqlite file instead of inmemory, and, if running in k8s, use a persistent volume claim for its path.

The backend of VTOrc is assumed to be ephemeral now and this RFC is requesting some of that data is not ephemeral. I suggest the data that requires persistence is separated vs. breaking those assumptions for all backend data

  --audit-file-location string                                  File location where the audit logs are to be stored
 --audit-purge-duration duration                               Duration for which audit logs are held before being purged. Should be in multiples of days (default 168h0m0s)
 --audit-to-backend                                            Whether to store the audit log in the VTOrc database
 --audit-to-syslog                                             Whether to store the audit log in the syslog

There is an "auditing" system in VTOrc. Currently it lets you output audit data to logs, a file or the ephemeral backend DB. The usefulness of backend and to some degree file are questionable to me in such an ephemeral daemon/pod, but 🤷. The quality/detail of the audit events could be improved in my opinion, but to me this existing "audit" concept feels like the logical place to build upon

Thinking aloud, could support for remote, persistent audit backends and support for reading it in VTAdmin address this? Also, is a web UI required? Could --audit-to-syslog and log pipeline UIs be enough? 🤔

[mattlord]

Today you can use --logtostderr or --alsologtostderr and redirect STDERR wherever you like. Typically the log messages would get sent to something like Loki with annotations added for context. That’s then the aggregated audit log of system actions across all processes that you can then search. If that’s not enough and you want something specific then I think that you should be very clear about what you’d like and why. I’m not suggesting a new http endpoint is a bad idea, just that this is the place to get into the specifics of what and why (including why alternatives don’t meet the needs).

I think that we should start by improving what we have before we consider adding more complexity and tradeoffs that some may object to. To put it another way, we should explore any gaps today for option 1. That may just be improving the usefulness of vtorc log messages, which is always a good thing. We should also improve the output of the existing vtorc http endpoint. These are both useful improvements that nobody can reasonably object to and which would make the product much better IMO w/o any downsides.

[GuptaManan100]

Hey @ejortegau me and Deepthi discussed this, and I think we can for now go ahead with the first proposal of having distinguishable log entries for now.

Currently VTOrcs don't register themselves in the topo-server, which means they aren't discoverable by any other running service that doesn't explicitly have its endpoint, so that makes the other two options a little trickier to implement, so let's start with the first.

[timvaillancourt]

To me we're tackling 2 x topics:

1. How to persist recovery (to my mind "audit"-system) data
2. How to display that data

Focusing on the 1st point, the audit system has a "file", "backend" and "syslog" options. If it doesn't exist already, what if an interface such as type AuditBackend interface existed?

Perhaps the existing "file", "backend" and "syslog" options could be migrated to implement this interface and the user could provide their own implementations like durability policies. This implementation could also be used in VTAdmin to read this data

I personally feel VTOrc should remain ephemeral, so I feel a persistent backend both components can access would be a remote datastore. Could we just persist the recovery data in-Vitess, perhaps in the sidecar database of the shard in a simple schema with JSON fields? Of course, this might not scale at read-time. Or at worst, in a remote store such as a plain MySQL (either as a custom interface or a built-in implementation)?

As an example, in our AWS-based infra, something like an RDS database or S3 would be viable. If it could be stored in the shards, even better