Add SA compression, fix some tests and refactor some code

* bitarray compression

* add coverage for bitarray and small README

* document library

* cargo formatting + better panic message in testing

* add is_empty method for clippy

* cargo formatting

* function to compress data without having to store everything in memory

* compressed integration + some major refactoring

* dynamic resuired bits per value

* update tests and documentation

* update coverage config

* fmt + clippy

* format --all

* improve testing

* formatting

* fix bitarray tests

* some additional smaller tests

* update codecov config

* ignore main files when covering code

* test failing reader and writer in test module

* small cleanup

* small cleanup

* some debug information

* allow preliminary enzyme numbers (x.x.x.nx)

* encode functional annotations on load

* more debug info + some movement of SA code

.github/workflows/coverage.yml

@@ -19,6 +19,33 @@ jobs:
           toolchain: nightly
           override: true
 
+
+
+      - name: Run cargo test (bitarray)
+        uses: actions-rs/cargo@v1
+        with:
+          command: test
+          args: --all-features --no-fail-fast -p bitarray
+        env:
+          CARGO_INCREMENTAL: 0
+          RUSTFLAGS: '-Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code -Coverflow-checks=off -Cpanic=abort -Zpanic_abort_tests'
+          RUSTDOCFLAGS: '-Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code -Coverflow-checks=off -Cpanic=abort -Zpanic_abort_tests'
+
+      - name: Gather coverage information (bitarray)
+        id: coverage-bitarray
+        uses: actions-rs/[email protected]
+
+      - name: Upload coverage reports to Codecov (bitarray)
+        uses: codecov/[email protected]
+        with:
+          token: ${{ secrets.CODECOV_TOKEN }}
+          file: ${{ steps.coverage-bitarray.outputs.report }}
+          flags: bitarray
+          verbose: true
+          fail_ci_if_error: true
+
+
+
       - name: Run cargo test (fa-compression)
         uses: actions-rs/cargo@v1
         with:
@@ -42,6 +69,82 @@ jobs:
           verbose: true
           fail_ci_if_error: true
 
+
+
+      - name: Run cargo test (sa-builder)
+        uses: actions-rs/cargo@v1
+        with:
+          command: test
+          args: --all-features --no-fail-fast -p sa-builder
+        env:
+          CARGO_INCREMENTAL: 0
+          RUSTFLAGS: '-Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code -Coverflow-checks=off -Cpanic=abort -Zpanic_abort_tests'
+          RUSTDOCFLAGS: '-Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code -Coverflow-checks=off -Cpanic=abort -Zpanic_abort_tests'
+
+      - name: Gather coverage information (sa-builder)
+        id: coverage-sa-builder
+        uses: actions-rs/[email protected]
+
+      - name: Upload coverage reports to Codecov (sa-builder)
+        uses: codecov/[email protected]
+        with:
+          token: ${{ secrets.CODECOV_TOKEN }}
+          file: ${{ steps.coverage-sa-builder.outputs.report }}
+          flags: sa-builder
+          verbose: true
+          fail_ci_if_error: true
+
+
+
+      - name: Run cargo test (sa-compression)
+        uses: actions-rs/cargo@v1
+        with:
+          command: test
+          args: --all-features --no-fail-fast -p sa-compression
+        env:
+          CARGO_INCREMENTAL: 0
+          RUSTFLAGS: '-Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code -Coverflow-checks=off -Cpanic=abort -Zpanic_abort_tests'
+          RUSTDOCFLAGS: '-Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code -Coverflow-checks=off -Cpanic=abort -Zpanic_abort_tests'
+
+      - name: Gather coverage information (sa-compression)
+        id: coverage-sa-compression
+        uses: actions-rs/[email protected]
+
+      - name: Upload coverage reports to Codecov (sa-compression)
+        uses: codecov/[email protected]
+        with:
+          token: ${{ secrets.CODECOV_TOKEN }}
+          file: ${{ steps.coverage-sa-compression.outputs.report }}
+          flags: sa-compression
+          verbose: true
+          fail_ci_if_error: true
+
+
+
+      - name: Run cargo test (sa-index)
+        uses: actions-rs/cargo@v1
+        with:
+          command: test
+          args: --all-features --no-fail-fast -p sa-index
+        env:
+          CARGO_INCREMENTAL: 0
+          RUSTFLAGS: '-Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code -Coverflow-checks=off -Cpanic=abort -Zpanic_abort_tests'
+          RUSTDOCFLAGS: '-Zprofile -Ccodegen-units=1 -Cinline-threshold=0 -Clink-dead-code -Coverflow-checks=off -Cpanic=abort -Zpanic_abort_tests'
+
+      - name: Gather coverage information (sa-index)
+        id: coverage-sa-index
+        uses: actions-rs/[email protected]
+
+      - name: Upload coverage reports to Codecov (sa-index)
+        uses: codecov/[email protected]
+        with:
+          token: ${{ secrets.CODECOV_TOKEN }}
+          file: ${{ steps.coverage-sa-index.outputs.report }}
+          flags: sa-index
+          verbose: true
+          fail_ci_if_error: true
+
+
       - name: Run cargo test (sa-mappings)
         uses: actions-rs/cargo@v1
         with:
@@ -63,4 +166,4 @@ jobs:
           file: ${{ steps.coverage-sa-mappings.outputs.report }}
           flags: sa-mappings
           verbose: true
-          fail_ci_if_error: true
+          fail_ci_if_error: true

.gitignore

@@ -1 +1,4 @@
 target/
+data/
+
+.DS_Store

Cargo.toml

@@ -1,10 +1,10 @@
 [workspace]
 resolver = "2"
 
-members = [
+members = [ "bitarray",
     "fa-compression",
     "libsais64-rs",
-    "sa-builder",
+    "sa-builder", "sa-compression",
     "sa-index",
     "sa-mappings",
     "sa-server"

README.md

@@ -2,18 +2,20 @@
 
 ![Codecov](https://img.shields.io/codecov/c/github/unipept/unipept-index?token=IZ75A2FY98&logo=Codecov)
 
-The unipept index written entirely in `Rust`. This repository consists of multiple different Rust projects that depend on 
-each other. More information about each project can be found in their respective `README.md` file.
+The unipept index written entirely in `Rust`. This repository consists of multiple different Rust projects that depend on each other. More information about each project can be found in their respective `README.md` file.
 
 ## Installation
 
-Clone this repository with the following command:
+> [!NOTE]  
+> To build and use the Unipept Index, you need to have Rust installed. If you don't have Rust installed, you can get it from [rust-lang.org](https://www.rust-lang.org/).
+
+Clone this repository by executing the following command:
 
 ```bash
 git clone https://github.com/unipept/unipept-index.git
 ```
 
-And build the projects using:
+Next, build everything by executing the following command in the root of the repository.
 
 ```bash
 cargo build --release

bitarray/Cargo.toml

@@ -0,0 +1,8 @@
+[package]
+name = "bitarray"
+version = "0.1.0"
+edition = "2021"
+
+# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
+
+[dependencies]

bitarray/README.md

@@ -0,0 +1,21 @@
+# Bitarray
+
+![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/unipept/unipept-index/test.yml?logo=github)
+![Codecov](https://img.shields.io/codecov/c/github/unipept/unipept-index?token=IZ75A2FY98&flag=bitarray&logo=codecov)
+![Static Badge](https://img.shields.io/badge/doc-rustdoc-blue)
+
+The `bitarray` offers a special array where each item is represented by a specified amount of bits (smaller than 64). The bitarray uses a pre-alocated vector and allows you to `set` or `get` a value from the array.
+
+## Example
+
+```rust
+use bitarray;
+
+fn main() {
+    let bitarray = BitArray::<40>::with_capacity(4);
+
+    bitarray.set(0, 0b0001110011111010110001000111111100110010);
+
+    assert_eq!(bitarray.get(0), 0b0001110011111010110001000111111100110010);
+}
+```