ell serialization refactor / distributed 2

[alex-dixon]

Rough overview:

• Production environments may benefit from an architecture where lmp versions and invocation traces are written from multiple machines that do not share the same file system
• Ideally, these setups need not miss out on any of ell’s features. This includes realtime updates to ell studio
• Data stores like postgres do not offer an easy way to subscribe to realtime changes. We could have an api that works off postgres change data capture. I’ve worked with this extensively to implement materialized views from changes to tables using debezium, Kafka/redpanda, and Apache Spark. It is doable, but at the end of it this would at most support realtime for one datastore. Ideally ell and its feature set should be available with other data stores, even nosql ones. Allowing studio to wire itself to an event bus lets us “read our own writes” without getting into the particulars of realtime options per data store. This makes it much easier to integrate data stores because the realtime component is solved for all of them by default.
• The event bus itself is pluggable in the same way as SQLite or postgres is for storage backends.
• We establish patterns for adding new functionality via conditional import of modules and Ell’s —extras. We remove barriers to adoption and promote what is more or less single file contributions to support additional event buses or data stores. See the celery library for an example of this pattern scaling to dozens of different backends: https://github.com/celery/celery/tree/main/celery/backends
• We continue to separate of ell studio from the core and lay groundwork for async support by introducing a serialization interface that is fully async and does not use sqlmodels as input or necessarily depend on sqlalchemy. Of all the changes this is the only one that touches the core and is probably the hardest to get right. The interfaces are written and ready for use but I have held off integrating them until we gain consensus on the code that is written here.
• Even with serialization code added to core, ell users should notice 0 changes when updating the library except perhaps the addition of new configuration options. All changes are internal.