Plans for Beaker 0.8 and Rotonde

[pfrazee]

Hey all. I wanted to share some things about where we're headed with the next major release of Beaker. We're making some significant updates and we want the Rotonde community to be involved. There are three topics to cover: data models, webdb, and applications.

Data models

The way that Rotonde currently works is by writing a single portal.json file which contains all of the activity of the users. This is a fine solution, but it has some nonobvious downsides:

• Problem 1. Unlike Git, Dat does not currently do any diffing between file updates. That means, each time you change portal.json, your audience has to redownload the file in its entirety. Over time, the portal.json will get pretty big, so this will hurt the speed of updates. (Dat may add diffing to updates someday, but it has costs as well.)
• Problem 2. the portal.json file has to be loaded entirely into memory to be used. This will also become a performance drag as the portal.jsons become larger.

The ideal solution is actually to break the data model up into multiple files, where the posts are put in individual .jsons. This solves both problems because you effectively partition the data.

We realized pretty quickly that breaking up into multiple files made things a little bit harder to develop. To deal with that, we created an indexing crawler called IngestDB.

The idea of Ingest is not only to make working with multiple json files easier, but to greatly simplify application dev overall by treating Dat like a database. So, rather than reading and writing files directly, you can instead interact with an ingest DB, like this:

var postUrl = await db.posts.add(bobsDat, {
  text: 'Hello!',
  createdAt: Date.now()
})
var recentPosts = await db.posts
  .orderBy('createdAt')
  .reverse() // most recent first
  .limit(30)
  .toArray()

From the application's perspective, it feels very much like using a database, but under the hood, Ingest translates records to .json files in the users' dats. So, each record has its own URL, and is stored in its own file. (See "how it works" to learn more about the underlying mechanics.)

Ingest is fairly portable; it works from within a web app in Beaker and from nodejs. We wanted to make sure bots and services could also use it. On top of that, we decided to build an Ingest instance into Beaker core, and we're calling that webdb.

WebDB

WebDB is basically an Ingest instance with a core set of data types programmed into it. We're adding it to simplify the job for developers. Here's roughly what the API will look like (it's still being worked on):

await webdb.profiles.get(bob.url)
await webdb.posts.create(alicesDat, {text: 'Hello, world!'})

Why are we creating WebDB? Two reasons.

One, ease of use. Ingest is not difficult to program, but it's more difficult than selecting from a set of semantic APIs. We want a nice set of APIs for devs of all skill levels to sit down and be productive with. Conceptual accessibility and DX is important to us.

Two, webdb creates a compatibility guarantee. By setting up common data model, we can help devs write apps that work together well. If things work correctly, anybody that writes an app using webdb will be able to read the data from another app that uses it. We'll leave room in the data model for some extensibility as well.

The meta goal is to create an apps ecosystem that's highly customizable, easy to cross-integrate, and easy to build with.

You won't have to use webdb, and it's entirely up to the rotonde community what to do. The idea is to offer 3 levels of abstraction and yall choose what's best for you: webdb at the highest level, your own Ingest for a little more power, or DatArchive for the most control.

Applications

One other thing I want to explain about 0.8, and that's applications.

Our meta goal for applications is for them to be powerful (see this and this), very easy to develop, and very easy to customize.

We've added a formal "application" concept, which is a type of dat which you can install. Installation lets you access privileged APIs. Those include the bookmarks, dat library, history, etc.; they are APIs you might normally expect for a web extension.

Installation also gives you access to the user-defined DNS system, which is called the App Scheme.

To explain -- it's very important to us that users are able to customize their applications. You should be able to to modify any application you use.

One way that apps will create friction against customization is by owning their URL space. The issue is, even if you can fork the code behind dat://someapp.com, links using that domain will still go to the original version. You need to be able to change the app assigned to dat://someapp.com to your fork.

So, we created a new URL scheme, app://, which maps user-chosen names to dat apps. This will get set when users install an Application. The app will suggest a default name to install at, and then users will choose the specific location during the install flow. That solves our problem of customization: you can fork an app, and set your fork to override the app:// URL.

Here is a slightly outdated gif of this in action:

Over time, we also think users will be able to share what they have mapped to their app names, and then p2p system becomes somewhat collaborative about deciding what apps should go at what names.

Rotonde as an app

Rotonde's architecture is close to working with the apps system already.

In the apps architecture, the app code and the user content will be in 2 separate dats, just as recently happened in Rotonde. Users would then choose a Rotonde distro to be installed at app://rotonde.

The rest of the mechanics are up to yall; one option would be, to view other ppl's rotonde feeds, you visit app://rotonde/{user-dat-key}. Then if you visit the user's dat directly (dat://{user-dat-key}) it would load its own version of the rotonde software, as it currently works.

This would make it possible for people to choose their personal app://rotonde, and also create personalized homepages at their dat://.

Wrapping up

That was a lot! I hope everything was clear. Please feel free to @ me with questions and feedback, and let me know if you have any concerns about the plans.

[neauoire]

Okay.
This is everything I ever wanted.

[Yanzoo]

Exciting stuff! I really like the idea of having rotonde working as an app, as well as the solutions for the eventual issues rotonde could've have in case it's structure stays as it is. This opens a load of possibility, really exciting!

[alphaflood]

I was actually planning my own .json flat file db for use with dats the other night. Glad to see IngestDB and WebDB will be on the table as well. Appreciate your efforts, and can't wait for 0.8.

[ghost]

does storing every post in its own json file will make things better long term?

filesystem is a database afterall

[karloscarweber]

I'm very happy to see this, especially access to privileged APIs in installed apps. I have a question though. Will IngestDB or WebDB have a private storage space? Let's say there is something that you don't want to publish with the rest of your site.

[pfrazee]

@karloscarweber Not at first. I'm still figuring out the best way to do private storage with ingest/webdb.