Migrations: Default for auto fill-in

[MentalGear]

Hey team,
Great first impression, especially local-first, relationships defined as queries and access control in schemas!

Looking at migrations

I was wondering if the default of a schema can not be simply used to fill new optional attributes automatically ?
Would make migrations way more streamlined.

[matlin]

Glad you've enjoyed using Triplit so far!

Yeah that's great idea. We're planning to add some basic automatic data migrations that you can optionally apply after getting an error message from schema push. However, we're not planning on making it fully automatic because many data edits (less so this particular case you've mentioned) can be unsafe when considering that they're only being applied to the server and not to any clients that also have data locally. E.g. you could change a optional attribute to required and backfill the data appropriately but it would then break clients that are still using the old definition.

[MentalGear]

Thank you for the detailed info! I agree that some changes, which could be automatically generated, still need manual approval.

However I have some concerns regarding the currently proposed migration pattern:

Only when you're confident that all clients have been updated to handle the new schema and all existing data has been updated to reflect the target schema, should you proceed with a backwards incompatible change.

This seems almost impossible to comply with for a local first distributed system. There might be clients that do not come back online for weeks, which would mean completing migrations would take a very long time in which the dev team still has to consider how on-going migrations might impact app logic. Also what if you want to start another migration while the previous one is still ongoing? This seems to get messy fast.

Of course you could use a "cut off" date after which old clients are simply dismissed, but from the user perspective, being not able to sync data any more (rendering the app basically useless for network usage) only because they haven't opened it for a month, seems -especially for lo-fi - unacceptable.

---

Schema Versioning & Migrations

I'm wondering if instead the traditional up/down deterministic migration logic, as used by migration tools like TypeORM etc, would not be the better pattern - especially for lofi.

Define a migration with 'up' which holds the new schema plus all code that needs to run to make the db compliant with the new schema. (In triplit's case, even just an upwards transition could be needed since clients can update themselves).

These can be collected in a migration file, that the server can check against with the client schema version.

Now, all clients, no matter their offline time, they can still upgrade their database and submit the new update. Also, as devs we certainly feel much more relaxed knowing that we don't have to continuously monitor how many clients have yet migrated to declare a migration as "finished" at an more or less arbitrary cut-off date.

Scenario:

• Outdated Client sends an update to the Server
• Server notices that the schema is outdated, sends back new schema & migration logic
• client runs migration logic and then sends the new update to the server which accepts it

[matlin]

You're exactly right to call out all of these details! It's a challenging problem that doesn't have any "free lunch" like running a single node database.

This seems almost impossible to comply with for a local first distributed system. There might be clients that do not come back online for weeks, which would mean completing migrations would take a very long time in which the dev team still has to consider how on-going migrations might impact app logic.

This is an unfortunate, intractable truth of distributed systems. The best solution is to maintain backwards compatibility at all times. We're working on tooling to make this easier to manage in your code but it's only the way to guarantee that older clients can continue to operate. This is why we warn against backwards incompatible changes and don't make any on your behalf. In development it's fine and we'll continue to streamline that flow but in production you're better of making some additive change to your schema and data instead.

• Outdated Client sends an update to the Server
• Server notices that the schema is outdated, sends back new schema & migration logic
• client runs migration logic and then sends the new update to the server which accepts it

This mostly works and something like this will likely be part of our solution to making your schema code simpler as it evolves but the main issue is that older clients can't simply migrate their schemas to a backwards incompatible version without also updating client code like queries and mutations. So in a sense this pattern would require keeping all client's code perfectly up to date which as you've pointed out is impossible with some clients being offline for long periods of time.

certainly feel much more relaxed knowing that we don't have to continuously monitor how many clients have yet migrated to declare a migration as "finished" at an more or less arbitrary cut-off date.

It's still very early so not ready to speak to our solution quite yet but my hunch is that this can be fully automated.

[MentalGear]

the main issue is that older clients can't simply migrate their schemas to a backwards incompatible version without also updating client code like queries and mutations. So in a sense this pattern would require keeping all client's code perfectly up to date

I assume by client updates, only the sync and schema parts (the sync protocol) are what needs updating, not the entire application. So the actual transfer should be a rather trivial, but I agree re: UX changing queries could be annoying.

While a full up/down migration schema would certainly be the goal, I'm wondering if a first version with an up only approach would be useful. Old clients can still submit their data to the server that upcasts it, but the app may choice, if "outdated" data is currently loaded, to notify that a reload is necessary to get the most recent data (sometimes this manual update might even be preferably to just having new data/properties pop up).

Overall, I'm sure you will find the right solution (as you have with the rest of the API), I'm just advocating for maybe a bit more of an incremental step.

Side-note: One thing I was a bit surprised over was that so much work is already being put into "React Native" support, when it seems like quite some of the core features that a full lo-fi distributed collab database would require are not fully in place yet ( migrations, presence, even collab text ). Everything else seems to make sense in planning, but this seems a bit out of order regarding prioritization.

[matlin]

Basically you can't just update queries or schemas without also updating your application code because your app expects queries to return data of a certain shape and insert data of a certain shape. If just sync a backwards incompatible schema over the air you should expect it to break your clients in some way.

You can apply safe backwards compatible changes to your server (that's support already) but ultimately the traditional, single node database server migrations don't apply here because of the distributed nature at play. This is similar to the problem many protocols face e.g. Protocol Buffers.

On prioritization, one way we prioritize is by making things possible first and then making them easy. It's quite possible to use existing APIs and libraries for presence, collaborative text, file uploads, etc alongside Triplit but it wasn't possible to run it in React Native without our intervention. So we made it work in React Native. In the future, we will make those other APIs more easily accessible via Triplit.

Focus is really important to build high quality products so we intentionally keep a narrow API and just keep working on making it better and more usable before adding on more tangential features.