docs: Added documentation for MySQL.

.github/workflows/docs.yaml

@@ -0,0 +1,36 @@
+on:
+  push:
+    branches:
+      - main
+      - mariadb
+
+jobs:
+  deploy:
+    name: Deploy Documentation
+    environment: 'docs-dev'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          submodules: true
+      - name: Set up AWS credentials
+        uses: aws-actions/configure-aws-credentials@v4
+        with:
+          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+          aws-region: us-east-1
+      - uses: actions/setup-python@v5
+        with:
+          python-version: 3.x
+      - name: Install plantuml
+        run: sudo apt-get install -y plantuml
+      - run: echo "cache_id=$(date --utc '+%V')" >> $GITHUB_ENV
+      - uses: actions/cache@v4
+        with:
+          key: mkdocs-material-${{ env.cache_id }}
+          path: .cache
+          restore-keys: |
+            mkdocs-material-
+      - run: pip install mkdocs-material pymdown-extensions plantuml_markdown
+      - run: mkdocs build
+      - run: aws s3 sync ./site "s3://${{ env.BUCKET_NAME || 'dev.docs.cfa.codes' }}/${{ env.PREFIX || 'cmr-entity-resolution' }}"

README.md

@@ -13,15 +13,16 @@ An entity resolution solution for automated record clearance.
 This solution is provided as a [docker compose][docker-compose] file and can be
 launched locally by running the following command:
 
-*Note: If you would like to launch the webapp to search for entities using a web
-interface, you will need to add `-p webapp` to include the `webapp` profile.*
+> [!NOTE]
+> If you would like to launch the webapp to search for entities using a web
+> interface, you will need to add `-p webapp` to include the `webapp` profile.
 
 ```bash
 docker compose up -d
 ```
 
 The configuration uses a persistent volume for the database and message queue
-containers to ensure data is persisted through updates to the image or
+container to ensure data is persisted through updates to the image or
 configuration.
 
 ### Applying changes
@@ -70,7 +71,25 @@ information on how to get data in and out of Senzing.
 See our [collection of examples][examples] to see a demonstration of the system
 in action.
 
+## Documentation
+
+Necessary documentation to operate, use, maintain, and contribute to this
+solution is included in this repository. The majority of these documents are
+written in Markdown and can be rendered directly in GitHub or you favorite IDE.
+However, the documentation as a whole is meant to be converted to a static site
+using [MkDocs].
+
+In order to view the documentation in its intended form locally, you can use the
+included docker container. Simply run the following:
+
+```bash
+docker compose --profile docs up -d
+```
+
+The documentation should then be available at <http://localhost:8000>.
+
 [docker-compose]: https://docs.docker.com/compose/
 [entity-spec]: https://senzing.zendesk.com/hc/en-us/articles/231925448-Generic-Entity-Specification-Data-Mapping
 [examples]: docs/examples.md
+[mkdocs]: https://www.mkdocs.org/
 [import-export]: docs/importing-exporting.md

docker-compose.yml

@@ -260,6 +260,16 @@ services:
     read_only: true
     restart: always
 
+  docs:
+    profiles:
+      - docs
+    build:
+      dockerfile: docs.dockerfile
+    ports:
+      - "8000:8000"
+    volumes:
+      - .:/docs
+
 networks:
   senzing:
     name: ${SENZING_DOCKER_NETWORK:-senzing-network}

docs.dockerfile

@@ -0,0 +1,6 @@
+FROM squidfunk/mkdocs-material:latest
+
+# Install PlantUML so we can render UML diagrams.
+RUN pip install markdown-callouts plantuml_markdown
+RUN apk add --no-cache plantuml --repository=https://dl-cdn.alpinelinux.org/alpine/edge/community \
+    && rm -rf /var/cache/apk/*

docs/api.md

@@ -105,5 +105,5 @@ Example response:
 
 If the request succeeds, the API will return a 201 status code.
 
-[compose]: ../docker-compose.yml
+[compose]: https://github.com/codeforamerica/cmr-entity-resolution/blob/main/docker-compose.yml
 [config]: importing-exporting.md

docs/examples.md

@@ -10,6 +10,8 @@ functionality of this entity resolution system.
 ## Sources
 
 * [Import from Informix][import-from-informix]
+* [Import from MySQL][import-from-mysql]
 
 [export-to-mongo]: examples/export-to-mongo.md
 [import-from-informix]: examples/import-from-informix.md
+[import-from-mysql]: examples/import-from-mysql.md

docs/examples/assets/config.mariadb.yml → docs/examples/assets/config.mysql.yml

@@ -18,7 +18,6 @@ sources:
   mariadb:
     type: MySQL
     host: maraidb
-#    host: 127.0.0.1
     database: people
     table: people
     username: root

docs/examples/assets/docker-compose.mariadb.yml → docs/examples/assets/docker-compose.mysql.yml

@@ -3,7 +3,7 @@ services:
     image: mariadb
     environment:
       # See https://hub.docker.com/_/mariadb
-      MARIADB_ROOT_PASSWORD: ${INFORMIX_PASSWORD:-password}
+      MARIADB_ROOT_PASSWORD: ${MARIADB_ROOT_PASSWORD:-password}
       MARIADB_DATABASE: people
     networks:
       - senzing

docs/examples/export-to-mongo.md

@@ -6,8 +6,9 @@ exporting the resulting data. If you choose to import your own CSV file, ensure
 that you also provide an appropriate configuration file rather than the one
 specified here.
 
-> _Note: All commands listed in this document are run from the root directory of
-> this [repository][repo]._
+> [!NOTE]
+> All commands listed in this document are run from the root directory of this
+> [repository][repo].
 
 ## Launching
 
@@ -45,7 +46,8 @@ Now that we have the system up and running, we can import our data. We'll do so
 by copying the file into the appropriate directory, setting the configuration
 file path, and running the importer.
 
-See [README.md][readme:import] for more information on importing and exporting. 
+See [Importing & Exporting][import-export] for more information on importing
+and exporting. 
 
 ```bash
 cp docs/examples/assets/import.csv data/import/import.csv
@@ -74,6 +76,6 @@ see your data in MongoDB. The default username and password are `admin` and
 
 [mongo]: https://www.mongodb.com/
 [mongo-express]: https://github.com/mongo-express/mongo-express
-[readme:import]: ../../README.md#importing--exporting
-[readme:launching]: ../../README.md#launching
+[import-export]: ../importing-exporting.md
+[readme:launching]: /#launching
 [repo]: https://github.com/codeforamerica/cmr-entity-resolution

docs/examples/import-from-informix.md

@@ -6,8 +6,9 @@ and load a sample dataset to be imported into Senzing. You can also use your own
 Informix database, but you must also provide an appropriate configuration file
 rather than the one specified here.
 
-> _Note: All commands listed in this document are run from the root directory of
-> this [repository][repo]._
+> [!NOTE]
+> All commands listed in this document are run from the root directory of this
+> [repository][repo].
 
 ## Launching
 
@@ -45,7 +46,8 @@ already been loaded into the Informix database, so we just need to point the
 importer at it. We do so using a configuration file with the source set to the 
 database, and passing that to the importer.
 
-See [README.md][readme:import] for more information on importing and exporting. 
+See [Importing & Exporting][import-export] for more information on importing and
+exporting. 
 
 ```bash
 export IMPORTER_CONFIG_FILE="$(pwd)/docs/examples/assets/config.informix.yml"
@@ -64,7 +66,7 @@ export EXPORTER_CONFIG_FILE="$(pwd)/docs/examples/assets/config.informix.yml"
 docker compose up exporter
 ```
 
+[import-export]: ../importing-exporting.md
 [informix]: https://www.ibm.com/products/informix
-[readme:import]: ../../README.md#importing--exporting
-[readme:launching]: ../../README.md#launching
+[readme:launching]: /#launching
 [repo]: https://github.com/codeforamerica/cmr-entity-resolution

docs/examples/import-from-mysql.md

@@ -0,0 +1,77 @@
+# Example: Import from MySQL
+
+> [!TIP]
+> The MySQL source type can be used for any MySQL compatible database, as shown
+> in this example using MariaDB.
+
+This example demonstrates importing data from a [MySQL][mysql] database.
+Following the steps below will launch a [MariaDB] container locally, and load a
+sample dataset to be imported into Senzing. You can also use your own
+MySQL compatible database, but you must also provide an appropriate
+configuration file rather than the one specified here.
+
+> [!NOTE]
+> All commands listed in this document are run from the root directory of this
+> [repository][repo].
+
+## Launching
+
+If you've launched the entity resolution system by following
+[README.md][readme:launching], then you're already part way there! If you
+haven't, don't worry! It's worth taking a look at that documentation, but you
+don't have to run any of those commands. We'll launch the system along with our
+MariaDB container.
+
+Before launching, the following environment variables can be set to configure
+the container:
+
+| Variable              | Default  | Description                 |
+|-----------------------|----------|-----------------------------|
+| MARIADB_ROOT_PASSWORD | password | Password for the root user. |
+
+Whether you already have the entity resolution system up and you just want to
+add the MariaDB container, or your need to launch the full stack, you can do so
+with the following:
+
+```bash
+docker compose \
+  -f docker-compose.yml \
+  -f docs/examples/assets/docker-compose.mysql.yml \
+  up -d
+```
+
+This will launch and configure all the required containers and load the sample
+dataset into MariaDB.
+
+## Importing
+
+Now that we have the system up and running, we can import our data. The data's
+already been loaded into the database, so we just need to point the importer at
+it. We do so using a configuration file with the source set to the database, and
+passing that to the importer.
+
+See [Importing & Exporting][import-export] for more information on importing and
+exporting. 
+
+```bash
+export IMPORTER_CONFIG_FILE="$(pwd)/docs/examples/assets/config.mysql.yml"
+docker compose up importer
+```
+
+Once the importer container exits, your data is now in Senzing!
+
+## Exporting
+
+To verify that the import succeeded, we can export the results from Senzing
+to a CSV file. Our config file already has this setup.
+
+```bash
+export EXPORTER_CONFIG_FILE="$(pwd)/docs/examples/assets/config.mysql.yml"
+docker compose up exporter
+```
+
+[import-export]: ../importing-exporting.md
+[mariadb]: https://mariadb.org/
+[mysql]: https://www.mysql.com/
+[readme:launching]: /#launching
+[repo]: https://github.com/codeforamerica/cmr-entity-resolution

docs/filters.md

@@ -53,6 +53,6 @@ The following options are available for this filter.
   inverse: false
 ```
 
-[non_human]: ../lib/filter/non_human.yml
+[non_human]: https://github.com/codeforamerica/cmr-entity-resolution/blob/main/lib/filter/non_human.yml
 [source]: sources.md
 [transformations]: transformations.md

docs/importing-exporting.md

@@ -4,8 +4,9 @@ Data can be imported from a number of [sources] and exported to several
 [destinations]. Before you begin, you will need a configuration file with
 settings for your source, destination, [filters], and [transformations].
 
-> _Note: All commands listed in this document are run from the root directory of
-> this project._
+> [!NOTE]
+> All commands listed in this document are run from the root directory of
+> this project.
  
 ## Using the CLI
 
@@ -36,7 +37,7 @@ correct path of your configuration file to `--config`.
 
 ```bash
 ./bin/exporter export --config config/config.yml
-````
+```
 
 ## Using docker
 

docs/index.md

@@ -0,0 +1,4 @@
+--8<-- "README.md"
+
+[examples]: examples.md
+[import-export]: importing-exporting.md

docs/license.md

@@ -0,0 +1 @@
+--8<-- "LICENSE"

docs/sources.md

@@ -62,6 +62,7 @@ The following options are available for this source.
 | database |           | YES      | Database to read from.                                                        |
 | host     |           | YES      | Informix host to connect to.                                                  |
 | password |           | YES      | Password for the database user.                                               |
+| port     | 9089      | NO       | Port to connect to on the database server.                                    |
 | security | nil       | NO       | Set to "SSL" in order to utilize TLS[^1].                                     |
 | schema   | $username | NO       | Schema that the database is attached to. Defaults to the value of `username`. |
 | table    |           | YES      | Table that contains the records to be imported                                |
@@ -91,10 +92,56 @@ sources:
 
 Check out the [Import from Informix][informix-example] to see this in action.
 
+## MySQL
+
+Query an [MySQL] or compatible (such as [MariaDB]) database for records to
+import.
+
+### Configuration
+
+The following options are available for this source.
+
+| Option   | Default   | Required | Description                                                                   |
+|----------|-----------|----------|-------------------------------------------------------------------------------|
+| database |           | YES      | Database to read from.                                                        |
+| host     |           | YES      | Informix host to connect to.                                                  |
+| password |           | YES      | Password for the database user.                                               |
+| port     | 3306      | NO       | Port to connect to on the database server.                                    |
+| security | nil       | NO       | Set to "SSL" in order to utilize TLS[^1].                                     |
+| table    |           | YES      | Table that contains the records to be imported                                |
+| username |           | YES      | User with access to the database                                              |
+
+### Example
+
+```yaml
+sources:
+  informix:
+    type: MySQL
+    host: localhost
+    database: people
+    table: people
+    username: mysql
+    password: password
+    field_map:
+      party_id: RECORD_ID
+      last_name: PRIMARY_NAME_LAST
+      first_name: PRIMARY_NAME_FIRST
+      gender: GENDER
+      birth_date: DATE_OF_BIRTH
+      dr_lic_num: DRIVERS_LICENSE_NUMBER
+      dr_lic_state: DRIVERS_LICENSE_STATE
+      ssn: SSN_NUMBER
+```
+
+Check out the [Import from MySQL][mysql-example] to see this in action.
+
 [entity-spec]: https://senzing.zendesk.com/hc/en-us/articles/231925448-Generic-Entity-Specification-Data-Mapping
 [filters]: filters.md
 [informix]: https://www.ibm.com/products/informix
 [informix-example]: examples/import-from-informix.md
+[mariadb]: https://mariadb.org/
+[mysql]: https://www.mysql.com/
+[mysql-example]: examples/import-from-mysql.md
 [senzing-config]: configuring-senzing.md
 [transformations]: transformations.md
 [^1]: Transport Layer Security