Skip to content

Commit

Permalink
Add serde support for ConvexValue to help with serializing system…
Browse files Browse the repository at this point in the history
… metadata (#22841)

It's very tedious to manually write the conversions between our in-memory structs (like `TableMetadata`) and `ConvexValue`. This change uses Serde to automate some parts of this conversion. Let's use `TableMetadata` as our running example:
```rust
pub struct TableMetadata {
    pub name: TableName,
    pub number: TableNumber,
    pub state: TableState,
}

pub enum TableState {
    Active,
    Hidden,
    Deleting,
}
```

To start, the developer defines two `Serialized*` types in Rust that define the serialization format they'd like for this in-memory structure. Think of these definitions in Rust as similar to a Protobuf definition.

```rust
#[derive(Serialize, Deserialize)]
pub struct SerializedTableMetadata {
    pub name: String,
    pub number: i64,
    pub state: SerializedTableState,
}

#[derive(Serialize, Deserialize)]
#[serde(tag = "type")]
pub enum SerializedTableState {
    Active {},
    Hidden {},
    Deleting {},
}
```

The developer may have the `Serialized*` structs diverge from the in-memory one to, say, implement backwards compatibility. Note that we're using an empty struct variant for the enum to tell serde to serialize the enum as an object with its variant tagged in the `type` field.

Then, implement `TryFrom` conversions both ways. Since this is going between two Rust data types, this should be easier than implementing conversions to and from `ConvexObject`:
```rust
impl TryFrom<TableMetadata> for SerializedTableMetadata {
    type Error = anyhow::Error;
    ...
}

impl TryFrom<SerializedTableMetadata> for TableMetadata {
    type Error = anyhow::Error;
    ...
}

impl TryFrom<TableState> for SerializedTableState {
    type Error = anyhow::Error;
    ...
}

impl TryFrom<SerializedTableState> for TableState {
    type Error = anyhow::Error;
    ...
}
```

Finally, connect the two types together with the `codegen_convex_serialization!` macro. This macro generates conversions to and from `ConvexValue` and `ConvexObject`, and it also defines a roundtrips proptest.
```rust
codegen_convex_serialization!(TableMetadata, SerializedTableMetadata);
```

I've ported over all of `bootstrap_model` to this new approach, and I think it cleans stuff up a lot!

GitOrigin-RevId: ede5e8a87d608bb72a7007948c91b28472ca4aca
  • Loading branch information
sujayakar authored and Convex, Inc. committed Mar 17, 2024
1 parent f5b0b0f commit 23a8d40
Show file tree
Hide file tree
Showing 37 changed files with 4,399 additions and 2,396 deletions.
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

284 changes: 0 additions & 284 deletions crates/common/src/bootstrap_model/index/database_index.rs

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
use std::convert::TryFrom;

use serde::{
Deserialize,
Serialize,
};

/// Represents state of currently backfilling index.
/// We currently do not checkpoint. Will extend the struct when we do.
#[derive(Debug, Clone, PartialEq, Eq)]
#[cfg_attr(any(test, feature = "testing"), derive(proptest_derive::Arbitrary))]
pub struct DatabaseIndexBackfillState;

#[derive(Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct SerializedDatabaseIndexBackfillState {}

impl TryFrom<DatabaseIndexBackfillState> for SerializedDatabaseIndexBackfillState {
type Error = anyhow::Error;

fn try_from(_config: DatabaseIndexBackfillState) -> anyhow::Result<Self> {
Ok(Self {})
}
}

impl TryFrom<SerializedDatabaseIndexBackfillState> for DatabaseIndexBackfillState {
type Error = anyhow::Error;

fn try_from(_config: SerializedDatabaseIndexBackfillState) -> anyhow::Result<Self> {
Ok(Self)
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
use std::convert::TryFrom;

use serde::{
Deserialize,
Serialize,
};

use super::indexed_fields::IndexedFields;
use crate::paths::FieldPath;

#[derive(Debug, Clone, PartialEq, Eq)]
#[cfg_attr(any(test, feature = "testing"), derive(proptest_derive::Arbitrary))]
pub struct DeveloperDatabaseIndexConfig {
/// Ordered field(s) to index. The "unindexed" primary key ordering of
/// documents by [`DocumentId`] is represented by an empty vector.
pub fields: IndexedFields,
}

#[derive(Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct SerializedDeveloperDatabaseIndexConfig {
fields: Vec<String>,
}

impl TryFrom<DeveloperDatabaseIndexConfig> for SerializedDeveloperDatabaseIndexConfig {
type Error = anyhow::Error;

fn try_from(config: DeveloperDatabaseIndexConfig) -> anyhow::Result<Self> {
Ok(Self {
fields: Vec::<FieldPath>::from(config.fields)
.into_iter()
.map(String::from)
.collect(),
})
}
}

impl TryFrom<SerializedDeveloperDatabaseIndexConfig> for DeveloperDatabaseIndexConfig {
type Error = anyhow::Error;

fn try_from(config: SerializedDeveloperDatabaseIndexConfig) -> anyhow::Result<Self> {
Ok(Self {
fields: config
.fields
.into_iter()
.map(|p| p.parse())
.collect::<anyhow::Result<Vec<FieldPath>>>()?
.try_into()?,
})
}
}
Loading

0 comments on commit 23a8d40

Please sign in to comment.