Skip to content

Commit

Permalink
Various documentation updates (#7)
Browse files Browse the repository at this point in the history
  • Loading branch information
jonmmease authored Oct 10, 2022
1 parent 041f3c2 commit a053c00
Show file tree
Hide file tree
Showing 6 changed files with 19 additions and 23 deletions.
10 changes: 5 additions & 5 deletions DEVELOP.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
## Release process
Releases of vl-convert are handled using [cargo-workspaces](https://github.com/pksunkara/cargo-workspaces), which can be installed with:
Releases of VlConvert crates are handled using [cargo-workspaces](https://github.com/pksunkara/cargo-workspaces), which can be installed with:

```
$ cargo install cargo-workspaces
Expand All @@ -8,13 +8,13 @@ $ cargo install cargo-workspaces
## Tagging and publish to crates.io
Check out the main branch, then tag and publish a new version of the `vl-convert` and `vl-convert-rs` crates with:

(replacing `0.1.0-rc3` with the desired version)
(replacing `0.1.0` with the desired version)
```
$ cargo ws publish --all --force "vl-convert*" custom 0.1.0-rc3
$ cargo ws publish --all --force "vl-convert*" custom 0.1.0
```

## Publish Python packages to PyPI
The `cargo ws publish ...` command above will push a commit to the `main` branch. This push to `main` will trigger CI, including the "Publish to PyPI" job. This job must be approved manually in the GitHub interface. After it is approved it will run and publish Python packages to PyPI.
The `cargo ws publish ...` command above will push a commit to the `main` branch. This push to `main` will trigger CI, including the "Publish to PyPI" job. This job must be approved manually in the GitHub interface. After it is approved it will run and publish the Python packages to PyPI.

## Create GitHub Release
Create a new GitHub release using the `v0.1.3-rc1` tag.
Create a new GitHub release using the `v0.1.0` tag.
19 changes: 9 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,13 @@
# VlConvert
VlConvert provides a Rust library, CLI utility, and Python library for converting high-level Vega-Lite visualization specifications into low-level Vega visualization specifications, all with no web browser or system dependencies required.
VlConvert provides a Rust library, CLI utility, and Python library for converting high-level [Vega-Lite](https://vega.github.io/vega-lite/) visualization specifications into low-level [Vega](https://vega.github.io/vega/) visualization specifications, all with no web browser or system dependencies required.

# Getting started
## CLI
First, download the `vl-convert` release archive for your operating system from GitHub release page and unzip into the current directory.

Install `vl-convert` using [cargo](https://doc.rust-lang.org/cargo/) with:
```
$ cargo install vl-convert
$ ./vl-convert --help
```
```
vl-convert: A utility for converting Vega-Lite specifications into Vega specification
Usage: vl-convert [OPTIONS] --input-vegalite-file <INPUT_VEGALITE_FILE> --output-vega-file <OUTPUT_VEGA_FILE>
Expand All @@ -30,7 +29,7 @@ Options:

Example: Convert a Vega-Lite specification file named `in.vl.json` into a Vega specification file named `out.vg.json`. Perform the conversion using version 5.5 of the Vega-Lite JavaScript library and pretty-print the resulting JSON.
```
./vl-convert -i ./in.vl.json -o ./out.vg.json --vl-version 5.5 --pretty
$ ./vl-convert -i ./in.vl.json -o ./out.vg.json --vl-version 5.5 --pretty
```

## Python
Expand All @@ -40,7 +39,7 @@ Install the `vl-convert-python` pacakge using pip
$ pip install vl-convert-python
```

Then in Python, import the library, create a `VlConverter object`, and use the converter to convert a Vega-Lite specification string to a Vega specification string.
Then in Python, import the library, create a `VlConverter` object, and use the `vegalite_to_vega` method to convert a Vega-Lite specification string to a Vega specification string.

```python
from vl_convert import VlConverter
Expand Down Expand Up @@ -82,12 +81,12 @@ print(vg_spec)
```

# Motivation
VlConvert was motivated by the needs of VegaFusion, which extracts data transformations from Vega specifications and evaluates them on the server. Using VlConvert, VegaFusion can input Vega-Lite specifications directly. That said, VlConvert is designed to be used by the wider Vega-Lite ecosystem, independent of VegaFusion.
VlConvert was motivated by the needs of [VegaFusion](https://vegafusion.io/), which extracts data transformations from Vega specifications and evaluates them on the server. Using VlConvert, VegaFusion can input Vega-Lite specifications directly. That said, VlConvert is designed to be used by the wider Vega-Lite ecosystem, independent of VegaFusion.

# How it works
VlConvert relies on the standard Vega-Lite JavaScript library to perform the Vega-Lite to Vega conversion. It uses the [Deno](https://deno.land/) project (in particular [`deno_core`](https://github.com/denoland/deno/tree/main/core)) to run Vega-Lite using the v8 JavaScript runtime, embedded into a Rust library called `vl-convert-rs`. This Rust library is then wrapped as a CLI application (`vl-convert`) and a Python library using PyO3 (`vl-convert-python`).

In addition to easy embedding of the v8 JavaScript runtime, another advantage of building on `deno_core` is that it's possible to customize the module loading logic. vl-convert takes advantage of this to inline the minified JavaScript source code for multiple versions of Vega-Lite, and all their dependencies, into the Rust library itself. This way, no internet connection is required to use vl-convert, and the executable and Python library are truly self-contained.
In addition to easy embedding of the v8 JavaScript runtime, another advantage of building on `deno_core` is that it's possible to customize the module loading logic. `vl-convert-rs` takes advantage of this to inline the minified JavaScript source code for multiple versions of Vega-Lite, and all their dependencies, into the Rust library itself. This way, no internet connection is required to use vl-convert, and the executable and Python library are truly self-contained.

# Future work: Image Export
An exciting future possibility is to use the VlConvert infrastructure to enable zero-dependency image export of Vega(-Lite) visualizations. This is already possible outside the web browser context using NodeJs, and Deno is designed to be largely compatible with NodeJs. The main difficulty is that Vega relies on the node [canvas](https://www.npmjs.com/package/canvas) library for computing text metrics and performing png image export. Node canvas is not compatible with Deno, so image export doesn't work out of the box.
An exciting future possibility is to use the VlConvert infrastructure to enable zero-dependency image export of Vega(-Lite) visualizations. This is already possible outside the web browser context using Node.js, and Deno is designed to be largely compatible with Node.js. The main difficulty is that Vega relies on the node [canvas](https://www.npmjs.com/package/canvas) library for computing text metrics and performing png image export. Node canvas is not compatible with Deno, so image export doesn't work out of the box.
3 changes: 1 addition & 2 deletions vl-convert-python/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ $ pip install vl-convert-python
```

# Usage
From Python, import the `vl_convert` package, create a `VlConverter` object, and use the converter to convert a Vega-Lite specification string to a Vega specification string.
From Python, import the library, create a `VlConverter` object, and use the `vegalite_to_vega` method to convert a Vega-Lite specification string to a Vega specification string.

```python
from vl_convert import VlConverter
Expand Down Expand Up @@ -50,7 +50,6 @@ print(vg_spec)
}
```


## Python development setup
Create development conda environment
```
Expand Down
2 changes: 1 addition & 1 deletion vl-convert-rs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,4 @@ This crate is a dependency of the [`vl-convert`](https://crates.io/crates/vl-con
The [`VlConverter`] struct may be used to perform Vega-Lite to Vega specification conversions. See the documentation of [`VlConverter`] for example usage.

## JavaScript Vendoring and Code Generation
Note: The `vl-convert-rs/vendor` directory and `vl-convert-rs/src/module_loader/import_map.rs` file in this crate is generated by the internal [`vl-convert-vendor`](https://github.com/jonmmease/vl-convert/tree/main/vl-convert-vendor) crate.
Note: The `vl-convert-rs/vendor` directory and `vl-convert-rs/src/module_loader/import_map.rs` file in this crate are generated by the internal [`vl-convert-vendor`](https://github.com/jonmmease/vl-convert/tree/main/vl-convert-vendor) crate.
6 changes: 3 additions & 3 deletions vl-convert-vendor/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# vl-convert-vendor
Helper crate that downloads multiple versions of Vega-Lite, and their dependencies, using [Deno vendor](https://deno.land/[email protected]/tools/vendor). It also generates the `vl-convert-rs/src/module_loader/import_map.rs` file which inlines the source code of all the downloaded dependencies using the `include_str!` macro.
# Overview
`vl-convert-vendor` is a helper crate that downloads multiple versions of Vega-Lite, and their dependencies, using [Deno vendor](https://deno.land/[email protected]/tools/vendor). It also generates the `vl-convert-rs/src/module_loader/import_map.rs` file which inlines the source code of all the downloaded dependencies using the `include_str!` macro.

This crate relies on the Deno command-line program being available on the system `PATH`.

Expand All @@ -15,7 +15,7 @@ $ cargo install deno
# Adding a new version of Vega-Lite
vl-convert inlines the source code of supported versions of Vega-Lite so that no internet connection is required at runtime. As a consequence, vl-convert must be updated each time a new version of Vega-Lite is released. Here are the steps to add support for a new version of Vega-Lite (called version X.Y.Z in this example)

1. Identify the skypack CDN URL for the new version by opening https://cdn.skypack.dev/[email protected] in a web browser. Copy the *minified* URL displayed in the header comment of this page. This URL will start with https://cdn.skypack.dev/pin/[email protected].
1. Identify the Skypack CDN URL for the new version by opening https://cdn.skypack.dev/[email protected] in a web browser. Copy the *minified* URL displayed in the header comment of this page. This URL will start with https://cdn.skypack.dev/pin/[email protected].
2. Update the `VL_PATHS` const variable at the top of `vl-convert-vendor/src/main.rs` to include a new tuple of the form `("X.Y", "https://cdn.skypack.dev/pin/[email protected]...")`. Note that only the major and minor version are included in the first element of the tuple.
3. Run the `vl-convert-vendor` binary from the `vl-convert-vendor` directory using `cargo run`. This will download the new version of Vega-Lite, and it's dependencies, using `deno vendor`. It will also generate a new version of `vl-convert-rs/src/module_loader/import_map.rs` that includes the new version.
4. Update the default value of the `vl_version` command line argument in `vl-convert/src/main.rs` to `X.Y`.
Expand Down
2 changes: 0 additions & 2 deletions vl-convert/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,6 @@ Install `vl-convert` using cargo with:
$ cargo install vl-convert
```

Alternatively, download the `vl-convert` release archive for your operating system from the GitHub [release page](https://github.com/jonmmease/vl-convert/releases), unzip it, and add it to your system PATH.

## CLI Usage
Display the documentation for the `vl-convert` command
```plain
Expand Down

0 comments on commit a053c00

Please sign in to comment.