📝 Update documentation to agree with recent architecture overhaul

Makefile

@@ -10,6 +10,9 @@ builder:
 	docker build --platform linux/arm64 --build-arg TAG=arm64 -f distribution/docker/images/builder/Dockerfile -t webgrid/rust-musl-builder:arm64-root .
 	docker build --platform linux/amd64 --build-arg TAG=amd64 -f distribution/docker/images/builder/Dockerfile -t webgrid/rust-musl-builder:amd64-root .
 
+mkdocs-docker:
+	docker build -f distribution/docker/images/mkdocs/Dockerfile -t mkdocs .
+
 core:
 	cd core && ./build.sh --release
 

README.md

@@ -68,7 +68,6 @@ To run a basic grid in Docker you can use Docker Compose. Below is a bare-bones
 
 ```bash
 # Create prerequisites
-docker volume create webgrid
 docker network create webgrid
 
 # Download compose file
@@ -88,8 +87,11 @@ For deployment to Kubernetes a Helm repository is available. The default values
 # Add the repository
 helm repo add webgrid https://webgrid.dev/
 
+# List all available versions
+helm search repo --versions --devel webgrid/demo
+
 # Install the chart
-helm install example webgrid/webgrid
+helm install example webgrid/demo --version "<pick-a-version-from-the-list>"
 
 # Make it accessible locally for evaluation
 kubectl port-forward service/example-webgrid 8080:80

distribution/docker/images/mkdocs/Dockerfile

@@ -0,0 +1,17 @@
+FROM alpine
+
+RUN apk add --no-cache python3 py3-pip git curl bash
+
+RUN pip3 install --no-cache \
+            'mkdocs-git-revision-date-localized-plugin>=0.4' \
+            'mkdocs-material' \
+            'mkdocs-mermaid2-plugin' \
+            'mkdocs-codeinclude-plugin' \
+            'mkdocs-material-extensions' \
+            'mkdocs-simple-hooks' \
+            'git+http://github.com/TilBlechschmidt/mkdocs-helm'
+
+RUN curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | VERIFY_CHECKSUM=false bash
+
+WORKDIR /docs
+CMD mkdocs serve -a 0.0.0.0:8000

docs/architecture/database.md

@@ -1,5 +1,8 @@
 # Database
 
+!!! warning "Outdated"
+	Due to a recent architectural overhaul, the information on this page is no longer applicable for versions beyond `v0.5.1-beta`.
+
 ## Keys
 
 All metadata is stored in a key-value in-memory database called [Redis](https://redis.io).  Below is a list of all keys that are currently in use, annotated with their type and format (if applicable).

docs/architecture/index.md

@@ -4,6 +4,4 @@ No overview of the architecture exists yet :(
 
 ![No content](https://http.cat/204)
 
-Feel free to explore the following pages on your own!
-
-You can find the [core documentation here](/rust-doc/webgrid/)
+While the following pages contain some information, large portions of it are outdated due to the recent architecture overhaul. It will take me a bit to write about the new architecture. In the meantime, you can take a look at the mostly documented code for the core services [over here](/rust-doc/webgrid/).

docs/architecture/workflows.md

@@ -1,5 +1,8 @@
 # Workflows
 
+!!! warning "Outdated"
+	Due to a recent architectural overhaul, the information on this page is no longer applicable for versions beyond `v0.5.1-beta`.
+
 Below is a list of common process workflows that happen within WebGrid. They may help understanding how the grid operates internally.
 
 ## Scheduling workflow

docs/faq.md

@@ -43,7 +43,7 @@ Let's just say that we haven't reached any performance limits yet in our interna
 
 ### What is this sorcery?! 🧙‍♂️
 
-No magic, just standardization and a efficient architecture 😉
+No magic, just standardization and an efficient architecture 😉
 
 WebGrid relies on the [WebDriver](https://www.w3.org/TR/webdriver1/) specification. When you execute a Selenium Test, you are effectively speaking to an implementation of this protocol. Almost all browser vendors provide such an API for their browser. Both Selenium Grid and WebGrid are just intermediates who delegate your requests to a browser.
 

docs/features/api.md

@@ -11,29 +11,19 @@ http://<your-webgrid-address>/api
 
 ## Session metadata
 
-When creating a session, you can attach additional metadata which you can later use to query for your session. To do so, set one or both of the following keys in your client libraries `DesiredCapabilities` object.
+If you have attached metadata to your session as described [over here](./capabilities.md#attaching-metadata), you can search for your session by using regular expressions. To do so, you can use a query that looks like this:
 
-=== "Java"
-    ```java
-    DesiredCapabilities desiredCapabilities = new DesiredCapabilities();
-    ...
-    final Map<String, String> webgridOptions = new HashMap<>();
-    webgridOptions.put("name", "<your-test-name>");
-    webgridOptions.put("build", "<your-build-id>");
-    desiredCapabilities.setCapability("webgrid:options", webgridOptions);
-    ...
-    ```
-
-To query session objects which match your name, use a query like this:
-
-```javascript
+```graphql
 query {
-  sessions(name: "test-name") {
-    id
-    metadata {
-      name
-      build
+  session {
+    query(fields: [
+      { key: "project", regex: "^tardis$" },
+      { key: "answer", regex: "\\d+" },
+    ]) {
+      id
     }
   }
 }
-```
+```
+
+You can also fetch the latest sessions or retrieve details of a session given its identifier. For more details, consult the self-documenting API at `/api`.

docs/features/capabilities.md

@@ -0,0 +1,129 @@
+# Extensions & Metadata
+
+While WebGrid supports all standard capabilities, there is a number of extensions. These allow you to control features like screen recording and idle timeouts. Additionally, they can be used to attach metadata to a session. Later on, this metadata can be used to search for your session!
+
+To get started, you have to create a map which contains all WebGrid specific settings. This map will later on be stored in the `webgrid:options` key of the capability object. The way to do this differs for each selenium library but below are a few examples to get you started.
+
+=== "Java"
+    ```java
+    DesiredCapabilities desiredCapabilities = new DesiredCapabilities();
+    final Map<String, Object> webgridOptions = new HashMap<>();
+
+    // Put anything in your options map! See below for a list of whats available.
+
+    desiredCapabilities.setCapability("webgrid:options", webgridOptions);
+    ```
+
+=== "Rust"
+    ```rust
+    let mut caps = DesiredCapabilities::firefox();
+
+    // Put anything in your options map! See below for a list of whats available.
+    caps.add_subkey("webgrid:options", /* Option key */, /* Option value */)?;
+
+    WebDriver::new(endpoint, &caps).await?
+    ```
+
+## Disabling screen recording
+
+We have optimized the heck out of screen recordings! They use almost no bandwidth and minimal CPU. For this reason, they are enabled globally if you have configured a storage backend. However, should you for some reason decide that you do not want to record a session, you can set the `disableRecording` flag in the `webgrid:options` capabilities.
+
+=== "Java"
+    ```java
+    webgridOptions.put("disableRecording", true);
+    ```
+
+=== "Rust"
+    ```rust
+    caps.add_subkey("webgrid:options", "disableRecording", true);
+    ```
+
+!!! tip "Globally disable recordings"
+    If you do not want recordings for any sessions, just do not configure a storage backend. Refer the corresponding installation guide on how to not do so!
+
+## Overwriting idle timeout
+
+To conserve resources, each session terminates automatically when it does not receive a command from a client within a certain time period. This is especially useful in scenarios where the client may have crashed. Since the protocol is not connection oriented, there is no other way to detect such a situation. The default timeout is set to about 10 minutes. When setting up the grid, you have to opportunity to set a different global default. However, maybe just some of your clients need to stay idle for a long time while others do not. For such situations, you can overwrite the idle timeout on a per-session basis by setting the `idleTimeout` key in the `webgrid:options` capabilities to any numeric value in seconds.
+
+=== "Java"
+    ```java
+    webgridOptions.put("idleTimeout", 3600);
+    ```
+
+=== "Rust"
+    ```rust
+    caps.add_subkey("webgrid:options", "idleTimeout", 3600);
+    ```
+
+!!! warning "Very long timeouts"
+    Setting a very long timeout may cause issues. As it is virtually impossible for the grid to detect a client that has crashed or otherwise disconnected in a non-clean fashion, such a sessions may become "orphaned" and stick around blocking resources for the timeout you set.
+
+## Attaching metadata
+
+When you run hundreds of sessions on the grid and do not have a way to store session identifiers, it can become hard to identify that one session after the fact — or maybe you want to run statistics on how many sessions each project has created in the last week. To solve this, you can attach arbitrary key-value metadata to each session by passing a map to the `metadata` key in the `webgrid:options` capabilities.
+
+=== "Java"
+    ```java
+    final Map<String, String> metadata = new HashMap<>();
+    metadata.put("project", "tardis");
+    metadata.put("pipeline", "#42");
+
+    webgridOptions.put("metadata", metadata);
+    ```
+
+=== "Rust"
+    ```rust
+    let mut metadata = HashMap::new();
+    metadata.insert("project", "tardis");
+    metadata.insert("pipeline", "#42");
+
+    caps.add_subkey("webgrid:options", "metadata", metadata);
+    ```
+
+### Modifying metadata at runtime
+
+Setting metadata up-front is nice. However, in certain scenarios you might want to attach additional or modify existing metadata while the session is running. For this, an extension command is available at `/<session-id>/webgrid/metadata`. You can either make a POST request to this URL with a JSON object as the request body, or use your libraries support for extension commands if available. In the future, support for legacy libraries which do not support extension commands will be added (using cookies with special names).
+
+=== "Rust"
+    Below is an example implementation of a metadata modification extension command for the [`thirtyfour`](https://github.com/stevepryde/thirtyfour) Rust library.
+    ```rust
+    struct WebgridMetadataCommand {
+        fields: HashMap<String, String>,
+    }
+
+    impl WebgridMetadataCommand {
+        pub fn new() -> Self {
+            Self {
+                fields: HashMap::new(),
+            }
+        }
+
+        pub fn with_field(key: String, value: String) -> Self {
+            let mut instance = Self::new();
+            instance.add(key, value);
+            instance
+        }
+
+        pub fn add(&mut self, key: String, value: String) {
+            self.fields.insert(key, value);
+        }
+    }
+
+    impl ExtensionCommand for WebgridMetadataCommand {
+        fn parameters_json(&self) -> Option<serde_json::Value> {
+            serde_json::to_value(self.fields.clone()).ok()
+        }
+
+        fn method(&self) -> thirtyfour::RequestMethod {
+            thirtyfour::RequestMethod::Post
+        }
+
+        fn endpoint(&self) -> String {
+            "/webgrid/metadata".into()
+        }
+    }
+
+    // Usage
+    let metadata_command = WebgridMetadataCommand::with_field("answer".into(), "42".into());
+    driver.extension_command(metadata_command).await.ok();
+    ```

docs/features/screen-recording.md

@@ -46,14 +46,21 @@ In order to view or embed a video you need to retrieve its unique session identi
 
     ```graphql
     query {
-        sessions {
-            id
-            alive
+      session {
+        latest {
+        id
+        metadata {
+            client {
+            key
+            value
+            }
+        }
         }
+      }
     }
     ```
 
-    It returns you a list of all sessions with their corresponding identifiers and whether or not they are currently alive. To identify your session uniquely using metadata, head over to the [API documentation](./api.md#session-metadata).
+    It returns you a list of all sessions with their corresponding identifiers and user-assigned metadata. To identify your session uniquely using metadata, head over to the [API documentation](./api.md#session-metadata).
 
 ## Embedding
 

docs/getting-started.md

@@ -7,10 +7,9 @@ Below are guides to get you started as quickly as possible on your specific plat
 If you want to run a grid locally for simple testing purposes that require multiple isolated browsers or just want to evaluate this tool this is the right choice.
 Make sure you have [Docker](https://www.docker.com/get-started) installed and configured properly.
 
-Before you can start the grid in Docker you have to create a network and volume for it:
+Before you can start the grid in Docker you have to create a network for it:
 
 ```bash
-docker volume create webgrid
 docker network create webgrid
 ```
 
@@ -35,8 +34,11 @@ You can change the name of the release in the second command or add other option
 # Add the repository
 helm repo add webgrid https://webgrid.dev/
 
+# List all available versions
+helm search repo --versions --devel webgrid/demo
+
 # Install the chart
-helm install example webgrid/webgrid
+helm install example webgrid/demo --version "<pick-a-version-from-the-list>"
 ```
 
 ## Using the grid

docs/guides/k3s.md

@@ -4,16 +4,17 @@ This guide will explain how to get a basic but feature-complete instance of WebG
 
 ## Installing with Helm
 
-Since video recording is disabled by default, we have to set a config value to enable it. For this guide it is not required to modify any other variables as the defaults are suited for K3s. However, this may differ if you deploy to a non-K3s cluster. Consult the [storage documentation](../kubernetes/storage.md) for more details.
-
-To set the value, we will use the `--set` command line option but a values file would work the same way.
+To get started quickly, we will be deploying the `demo` chart. It contains the core grid along with external services that are required to get additional features like video recordings running.
 
 ```bash
 # Add the repository
 helm repo add webgrid https://webgrid.dev/
 
+# List all available versions
+helm search repo --versions --devel webgrid/demo
+
 # Install the chart
-helm install k3s-guide-deployment webgrid/webgrid --set recording.enabled=true
+helm install k3s-guide-deployment webgrid/demo --version "<pick-a-version-from-the-list>"
 ```
 
 ## Gaining access
@@ -53,7 +54,7 @@ The grid is now fully operational and available on port `30007` of your cluster!
 docker run --rm -it \
     -e ENDPOINT=http://<your-grid-endpoint>:30007 \
     -e FORKS=3 \
-    ghcr.io/tilblechschmidt/parallelseleniumtest:sha-14540b77
+    ghcr.io/tilblechschmidt/parallelseleniumtest:sha-fa30ad9
 ```
 
 After experimenting a bit with it, you can visit your grid in a browser — just put the same URL in that you use for your Selenium clients — to view the screen recordings for your sessions!

docs/index.md

@@ -68,7 +68,6 @@ To run a basic grid in Docker you can use Docker Compose. Below is a bare-bones
 
 ```bash
 # Create prerequisites
-docker volume create webgrid
 docker network create webgrid
 
 # Download compose file
@@ -88,8 +87,11 @@ For deployment to Kubernetes a Helm repository is available. The default values
 # Add the repository
 helm repo add webgrid https://webgrid.dev/
 
+# List all available versions
+helm search repo --versions --devel webgrid/demo
+
 # Install the chart
-helm install example webgrid/webgrid
+helm install example webgrid/demo --version "<pick-a-version-from-the-list>"
 
 # Make it accessible locally for evaluation
 kubectl port-forward service/example-webgrid 8080:80

docs/kubernetes/configuration.md

@@ -1,8 +1,8 @@
 # Configuration
 
-To get you started as quickly as possible the helm chart uses a set of default values that work in most clusters. However, these defaults are only an entrypoint — it is very likely that these need to be adapted to your specific needs.
+To get you started as quickly as possible the helm chart uses a set of default values that work in most clusters. However, these defaults are only an entrypoint — it is very likely that these need to be adapted to your specific needs. The Helm repository contains two charts: `webgrid` and `demo`. The latter wraps the former and provides some additional tools like an instance of [Minio](http://min.io) for storing video recordings and a web UI to access the database.
 
-Other documentation topics may ask you to change values in order to enable advanced features like screen recordings. Refer the sections below to learn how to do so.
+Other documentation topics may ask you to change values in order to enable advanced features like screen recordings. Refer the sections below to learn how to do so. If you are using the `demo` Chart and want to change values, put them under the `webgrid:` key so they are forwarded to the `webgrid` Chart!
 
 ## Changing the defaults
 

docs/kubernetes/scaling.md

@@ -13,11 +13,11 @@ To change the number of concurrent sessions allowed (in this case `5`) merge and
 ```yaml
 config:
   orchestrator:
-    maxSessions: 5
+    permits: 5
 ```
 
 !!! note
-    Make sure that your K8s service account has a sufficient quota available to create the required number of pods for the sessions.
+    Make sure that your K8s service account has a sufficient quota available to create the required number of jobs and pods for the sessions.
 
 ## Traffic congestion