Config & KV thoughts

[auniverseaway]

I continue to grapple with the best way to handle arbitrary data stored alongside content.

As of right now, the only data we need to store is auth information. The other needs are theoretical: ERP-like integrations, workflow related needs, etc. Other data like production domain or when something was last published we can lean on existing AEM features (helix admin api, .helix/config, etc.).

The overarching idea for config has always been:

"I want to store arbitrary data associated with my content, but it should not publicly viewable like metadata currently is."

The closest analogy here is AEM page properties or SharePoint metadata (AKA columns).

From an auth perspective, I think KV matches well as it has performance characteristics that meet our needs. Things get a little dicey when you think about multi-tenancy and rate limits. Today, there is one KV for all auth information for all customers. Do we want to continue this? If we move to the rest API, it would require an additional request to lookup the KV ID... thus removing the performance benefit we enjoy today.

The other solutions are not exactly appetizing: D1, which is a SQL database may have rate limits solved, but will the performance be there? Would we still be "stuck" sharing a single DB for all customers? Do we care? What does this mean for moving to a multi-cloud solution?

Once again, I'm still continuing to punt a lot of these thoughts down the road. We will use a single KV because it's fast, easy, and we're still well below our rate limits. If & when that becomes an issue, we can look at other means to store this data. It's also important to know that even if we do decide to use KV for extra content data, that will not be requested at the volume we do now for auth... where we lookup a KV value for every request. KV for content will be much more on-demand and have less volume.

It's also still on the table that we use S3 for anything below org/repo. This would be slower, but it makes it much easier to version at the same time as a document.

[auniverseaway]

After talking with @chrischrischris, I think I may have another idea...

What if we simply put another table in the page for configs? This gets us some free stuff:

1. Collaboration
2. Diffing / track changes (when we implement)
3. Fewer requests / performance
4. A starter UI (it's just a block in the editor)

Challenges this raises:

1. Concurrent reading / writing - If a service wants to update only the page config while someone else is editing the content, this poses a potential collision.
2. Exposing this data - We would want to strip this block out when being delivered from content.da.live.
3. Do we make an API to request this content separately? This would require more massaging of the html document source.

This seems like the most ideal outcome for page-level configs where performance needs to be good, but not "fires every request" good. We could maybe even have inheritance if we used the metadata pattern with a sheet and URL globs.