Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs(progress): Add docs for JSON progress format #961

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
+311 −27
Open

docs(progress): Add docs for JSON progress format #961

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9f2af62
docs: Describe `--json-fd`
antheas Dec 11, 2024
06aef6d
Generate JSON schema for progress
cgwalters Jan 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 57 additions & 17 deletions docs/src/bootc-via-api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,27 +4,67 @@ At the current time, bootc is primarily intended to be
driven via a fork/exec model. The core CLI verbs
are stable and will not change.

## Using `bootc edit` and `bootc status --json`
The core API is meant to be used as a "state machine": run a command,
such as `bootc upgrade --check` or `bootc switch`, and then query the
updated state with `bootc status --json`. The status command contains all
relevant state about the current deployment, including cached results
from `bootc upgrade --check`.

## `bootc status` JSON Schema

`bootc status --json --format-version=1` outputs a machine-readable JSON
that contains all relevant information about the current
deployment. This API is considered stable. You can find a
[JSON schema](https://json-schema.org/) describing the
version `org.containers.bootc/v1` here:
[host-v1.schema.json](host-v1.schema.json). This schema was generated
directly from the Rust bootc code.
You can either reference or feed it to a code generator such as
[go-jsonschema](https://github.com/omissis/go-jsonschema) to generate
client bindings.

In order to be forwards compatible with a future introduction of
a v2 or newer format, please include an explicit version in your
status request with `--format-version=1` as referenced above.
(Available since bootc 0.1.15, `--format-version=0` in bootc 0.1.14).

## Interactive progress with `--json-fd`

While the `bootc status` tooling allows a client to discover the state
of the system, during interactive changes such as `bootc upgrade`
or `bootc switch` it is possible to monitor the status of downloads
or other operations at a fine-grained level with `--json-fd`.

The format of data output over `--json-fd` is [JSON Lines](https://jsonlines.org)
which is a series of JSON objects separated by newlines (the intermediate
JSON content is guaranteed not to contain a literal newline).

The current API version is `org.containers.bootc/progress/v1`. You can find
the JSON schema describing this version here:
[progress-v1.schema.json](progress-v1.schema.json).

Deploying a new image with either switch or upgrade consists
of three stages: `pulling`, `importing`, and `staging`. The `pulling` step
downloads the image from the registry, offering per-layer and progress in
each message. The `importing` step imports the image into storage and consists
of a single step. Finally, `staging` runs a variety of staging
tasks. Currently, they are staging the image to disk, pulling bound images,
and removing old images.

Note that new stages or fields may be added at any time.

Importing and staging are affected by disk speed and the total image size. Pulling
is affected by network speed and how many layers invalidate between pulls.
Therefore, a large image with a good caching strategy will have longer
importing and staging times, and a small bespoke container image will have
negligible importing and staging times.

## Using `bootc edit`

While bootc does not depend on Kubernetes, it does currently
also offere a Kubernetes *style* API, especially oriented
also offer a Kubernetes *style* API, especially oriented
towards the [spec and status and other conventions](https://kubernetes.io/docs/reference/using-api/api-concepts/).

In general, most use cases of driving bootc via API are probably
most easily done by forking off `bootc upgrade` when desired,
and viewing `bootc status --json --format-version=1`.

## JSON Schema

The current API `org.containers.bootc/v1` is stable.
In order to support the future introduction of a v2
or newer format, please change your code now to explicitly
request `--format-version=1` as referenced above. (Available
since bootc 0.1.15, `--format-version=0` in bootc 0.1.14).

There is a [JSON schema](https://json-schema.org/) generated from
the Rust source code available here: [host-v1.schema.json](host-v1.schema.json).

A common way to use this is to run a code generator such as
[go-jsonschema](https://github.com/omissis/go-jsonschema) on the
input schema.
230 changes: 230 additions & 0 deletions docs/src/progress-v1.schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,230 @@
{
"$schema": "http://json-schema.org/draft-07/schema#",
"title": "Event",
"description": "An event emitted as JSON.",
"oneOf": [
{
"description": "An incremental update to a container image layer download",
"type": "object",
"required": [
"api_version",
"bytes",
"bytes_cached",
"bytes_total",
"description",
"id",
"steps",
"steps_cached",
"steps_total",
"subtasks",
"task",
"type"
],
"properties": {
"api_version": {
"description": "The version of the progress event format.",
"type": "string"
},
"bytes": {
"description": "The number of bytes already fetched.",
"type": "integer",
"format": "uint64",
"minimum": 0.0
},
"bytes_cached": {
"description": "The number of bytes fetched by a previous run.",
"type": "integer",
"format": "uint64",
"minimum": 0.0
},
"bytes_total": {
"description": "Total number of bytes. If zero, then this should be considered \"unspecified\".",
"type": "integer",
"format": "uint64",
"minimum": 0.0
},
"description": {
"description": "A human readable description of the task if i18n is not available.",
"type": "string"
},
"id": {
"description": "A human and machine readable unique identifier for the task (e.g., the image name). For tasks that only happen once, it can be set to the same value as task.",
"type": "string"
},
"steps": {
"description": "The initial position of progress.",
"type": "integer",
"format": "uint64",
"minimum": 0.0
},
"steps_cached": {
"description": "The number of steps fetched by a previous run.",
"type": "integer",
"format": "uint64",
"minimum": 0.0
},
"steps_total": {
"description": "The total number of steps (e.g. container image layers, RPMs)",
"type": "integer",
"format": "uint64",
"minimum": 0.0
},
"subtasks": {
"description": "The currently running subtasks.",
"type": "array",
"items": {
"$ref": "#/definitions/SubTaskBytes"
}
},
"task": {
"description": "A machine readable type (e.g., pulling) for the task (used for i18n and UI customization).",
"type": "string"
},
"type": {
"type": "string",
"enum": [
"ProgressBytes"
]
}
}
},
{
"description": "An incremental update with discrete steps",
"type": "object",
"required": [
"api_version",
"description",
"id",
"steps",
"steps_cached",
"steps_total",
"subtasks",
"task",
"type"
],
"properties": {
"api_version": {
"description": "The version of the progress event format.",
"type": "string"
},
"description": {
"description": "A human readable description of the task if i18n is not available.",
"type": "string"
},
"id": {
"description": "A human and machine readable unique identifier for the task (e.g., the image name). For tasks that only happen once, it can be set to the same value as task.",
"type": "string"
},
"steps": {
"description": "The initial position of progress.",
"type": "integer",
"format": "uint64",
"minimum": 0.0
},
"steps_cached": {
"description": "The number of steps fetched by a previous run.",
"type": "integer",
"format": "uint64",
"minimum": 0.0
},
"steps_total": {
"description": "The total number of steps (e.g. container image layers, RPMs)",
"type": "integer",
"format": "uint64",
"minimum": 0.0
},
"subtasks": {
"description": "The currently running subtasks.",
"type": "array",
"items": {
"$ref": "#/definitions/SubTaskStep"
}
},
"task": {
"description": "A machine readable type (e.g., pulling) for the task (used for i18n and UI customization).",
"type": "string"
},
"type": {
"type": "string",
"enum": [
"ProgressSteps"
]
}
}
}
],
"definitions": {
"SubTaskBytes": {
"description": "An incremental update to e.g. a container image layer download. The first time a given \"subtask\" name is seen, a new progress bar should be created. If bytes == bytes_total, then the subtask is considered complete.",
"type": "object",
"required": [
"bytes",
"bytesCached",
"bytesTotal",
"description",
"id",
"subtask"
],
"properties": {
"bytes": {
"description": "Updated byte level progress",
"type": "integer",
"format": "uint64",
"minimum": 0.0
},
"bytesCached": {
"description": "The number of bytes fetched by a previous run (e.g., zstd_chunked).",
"type": "integer",
"format": "uint64",
"minimum": 0.0
},
"bytesTotal": {
"description": "Total number of bytes",
"type": "integer",
"format": "uint64",
"minimum": 0.0
},
"description": {
"description": "A human readable description of the task if i18n is not available. (e.g., \"OSTree Chunk:\", \"Derived Layer:\")",
"type": "string"
},
"id": {
"description": "A human and machine readable identifier for the task (e.g., ostree chunk/layer hash).",
"type": "string"
},
"subtask": {
"description": "A machine readable type for the task (used for i18n). (e.g., \"ostree_chunk\", \"ostree_derived\")",
"type": "string"
}
}
},
"SubTaskStep": {
"description": "Marks the beginning and end of a dictrete step",
"type": "object",
"required": [
"completed",
"description",
"id",
"subtask"
],
"properties": {
"completed": {
"description": "Starts as false when beginning to execute and turns true when completed.",
"type": "boolean"
},
"description": {
"description": "A human readable description of the task if i18n is not available. (e.g., \"OSTree Chunk:\", \"Derived Layer:\")",
"type": "string"
},
"id": {
"description": "A human and machine readable identifier for the task (e.g., ostree chunk/layer hash).",
"type": "string"
},
"subtask": {
"description": "A machine readable type for the task (used for i18n). (e.g., \"ostree_chunk\", \"ostree_derived\")",
"type": "string"
}
}
}
}
}
18 changes: 15 additions & 3 deletions lib/src/cli.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -359,6 +359,12 @@ pub(crate) enum ImageOpts {
Cmd(ImageCmdOpts),
}

#[derive(Debug, Clone, clap::ValueEnum, PartialEq, Eq)]
pub(crate) enum SchemaType {
Host,
Progress,
}

/// Hidden, internal only options
#[derive(Debug, clap::Subcommand, PartialEq, Eq)]
pub(crate) enum InternalsOpts {
Expand All @@ -371,7 +377,10 @@ pub(crate) enum InternalsOpts {
},
FixupEtcFstab,
/// Should only be used by `make update-generated`
PrintJsonSchema,
PrintJsonSchema {
#[clap(long)]
of: SchemaType,
},
/// Perform cleanup actions
Cleanup,
/// Proxy frontend for the `ostree-ext` CLI.
Expand Down Expand Up @@ -1090,8 +1099,11 @@ async fn run_from_opt(opt: Opt) -> Result<()> {
.await
}
InternalsOpts::FixupEtcFstab => crate::deploy::fixup_etc_fstab(&root),
InternalsOpts::PrintJsonSchema => {
let schema = schema_for!(crate::spec::Host);
InternalsOpts::PrintJsonSchema { of } => {
let schema = match of {
SchemaType::Host => schema_for!(crate::spec::Host),
SchemaType::Progress => schema_for!(crate::progress_jsonl::Event),
};
let mut stdout = std::io::stdout().lock();
serde_json::to_writer_pretty(&mut stdout, &schema)?;
Ok(())
Expand Down
7 changes: 4 additions & 3 deletions lib/src/progress_jsonl.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
//! see <https://jsonlines.org/>.

use anyhow::Result;
use schemars::JsonSchema;
use serde::Serialize;
use std::borrow::Cow;
use std::os::fd::{FromRawFd, OwnedFd, RawFd};
Expand All @@ -20,7 +21,7 @@ pub const API_VERSION: &str = "org.containers.bootc.progress/v1";
/// An incremental update to e.g. a container image layer download.
/// The first time a given "subtask" name is seen, a new progress bar should be created.
/// If bytes == bytes_total, then the subtask is considered complete.
#[derive(Debug, serde::Serialize, serde::Deserialize, Default, Clone)]
#[derive(Debug, serde::Serialize, serde::Deserialize, Default, Clone, JsonSchema)]
#[serde(rename_all = "camelCase")]
pub struct SubTaskBytes<'t> {
/// A machine readable type for the task (used for i18n).
Expand All @@ -44,7 +45,7 @@ pub struct SubTaskBytes<'t> {
}

/// Marks the beginning and end of a dictrete step
#[derive(Debug, serde::Serialize, serde::Deserialize, Default, Clone)]
#[derive(Debug, serde::Serialize, serde::Deserialize, Default, Clone, JsonSchema)]
#[serde(rename_all = "camelCase")]
pub struct SubTaskStep<'t> {
/// A machine readable type for the task (used for i18n).
Expand All @@ -64,7 +65,7 @@ pub struct SubTaskStep<'t> {
}

/// An event emitted as JSON.
#[derive(Debug, serde::Serialize, serde::Deserialize)]
#[derive(Debug, serde::Serialize, serde::Deserialize, JsonSchema)]
#[serde(
tag = "type",
rename_all = "PascalCase",
Expand Down
Loading
Loading