Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CONTRIBUTING.md #80

Merged
merged 1 commit into from
Sep 4, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 84 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# Contributing to dbt_data_privacy

Thank you for your interest in contributing to dbt_data_privacy!

## Before you start

We utilize a unique approach to implement unit testing for dbt packages.
This approach is necessary because Jinja2 does not inherently support unit testing.
The dbt Developer Blog provides an excellent article on implementing unit testing for dbt packages.
This article serves as a valuable resource for understanding the concept of unit testing within the context of dbt packages.

[An introduction to unit testing your dbt Packages \| dbt Developer Blog](https://docs.getdbt.com/blog/unit-testing-dbt-packages)

Unit testing becomes increasingly crucial as a dbt package grows in complexity, ensuring that the package functions as intended.

## How to develop

### Directory structure

We primarily utilize the following directories for developing the dbt package.
The `macros/` directory houses the core logic of dbt_data_privacy in the form of dbt Macros.
The `integration_tests/` directory contains test files for both unit and integration testing.

- `macros/`: Contains the dbt Macros that constitute dbt_data_privacy.
- `integration_tests/`: Houses integration tests for dbt_data_privacy, encompassing both unit and integration tests.

### How to set up the development environment

Begin by cloning the repository to your local machine.
Navigate to the `integration_tests/` directory to execute unit and integration tests.
The [./integration_tests/Makefile](./integration_tests/Makefile) provides a set of helpful commands.
The `make setup` command installs dependencies and configures the development environment.

```shell
cd dbt_data_privacy/integration_tests

# Install dependencies
make setup
```

### How to run unit testing

A dedicated Makefile target is available for running unit tests.
This target allows you to run unit tests with a specified target and a vars file.

```shell
make run-unit-tests
```

Note that unit testing cannot be parallelized due to the sequential nature of test execution.
If an error occurs in a test file, the unit testing process halts and returns an error.
Consequently, it is not possible to capture all failed tests in a single command.

### How to implement unit tests

When adding or modifying dbt Macros in the `macros/` directory, it is essential to implement corresponding unit tests.
Place the test file in the `integration_tests/macros/` directory.

For example, the dbt Macro located at [macros/codegen/generate_privacy_protected_model_sql.sql](macros/codegen/generate_privacy_protected_model_sql.sql) is responsible for generating a secured dbt Model.
The corresponding unit test for this macro can be found at [integration_tests/macros/tests/codegen/test_generate_privacy_protected_model_sql.sql](integration_tests/macros/tests/codegen/test_generate_privacy_protected_model_sql.sql).
This illustrates the convention of placing a dbt Macro and its associated unit test file in a structured manner.

Custom dbt Macros, such as `assert_equals`, are available in the [integration_tests/macros/test_utils/](integration_tests/macros/test_utils/) directory to facilitate effective unit testing of dbt Macros.
Leverage these macros to create more comprehensive and robust unit tests.

### How to run integration testing

In addition to unit testing, we also perform integration testing for dbt_data_privacy.
This involves building dbt models, including secured models generated by the dbt_data_privacy package, on a data warehouse such as BigQuery.

1. Generate secured models using the provided command.
2. Compile the dbt Models to ensure the generated secured models are valid.
3. Execute a `dbt build` command to thoroughly test the generated models.

```shell
# 1. Generate secured models
make generate

# 2. Compile dbt Models if generated secured models are valid
make compile

# 2. Run a `dbt build` command to test generated models
make run-integration-tests
```
Loading