From 3beecabc8701687528f2f9b7f77b49d1a7ab7b51 Mon Sep 17 00:00:00 2001 From: Yu Ishikawa Date: Tue, 3 Sep 2024 09:54:48 +0900 Subject: [PATCH] Add `CONTRIBUTING.md` Signed-off-by: Yu Ishikawa --- CONTRIBUTING.md | 84 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 84 insertions(+) create mode 100644 CONTRIBUTING.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..56156be --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,84 @@ +# Contributing to dbt_data_privacy + +Thank you for your interest in contributing to dbt_data_privacy! + +## Before you start + +We utilize a unique approach to implement unit testing for dbt packages. +This approach is necessary because Jinja2 does not inherently support unit testing. +The dbt Developer Blog provides an excellent article on implementing unit testing for dbt packages. +This article serves as a valuable resource for understanding the concept of unit testing within the context of dbt packages. + +[An introduction to unit testing your dbt Packages \| dbt Developer Blog](https://docs.getdbt.com/blog/unit-testing-dbt-packages) + +Unit testing becomes increasingly crucial as a dbt package grows in complexity, ensuring that the package functions as intended. + +## How to develop + +### Directory structure + +We primarily utilize the following directories for developing the dbt package. +The `macros/` directory houses the core logic of dbt_data_privacy in the form of dbt Macros. +The `integration_tests/` directory contains test files for both unit and integration testing. + +- `macros/`: Contains the dbt Macros that constitute dbt_data_privacy. +- `integration_tests/`: Houses integration tests for dbt_data_privacy, encompassing both unit and integration tests. + +### How to set up the development environment + +Begin by cloning the repository to your local machine. +Navigate to the `integration_tests/` directory to execute unit and integration tests. +The [./integration_tests/Makefile](./integration_tests/Makefile) provides a set of helpful commands. +The `make setup` command installs dependencies and configures the development environment. + +```shell +cd dbt_data_privacy/integration_tests + +# Install dependencies +make setup +``` + +### How to run unit testing + +A dedicated Makefile target is available for running unit tests. +This target allows you to run unit tests with a specified target and a vars file. + +```shell +make run-unit-tests +``` + +Note that unit testing cannot be parallelized due to the sequential nature of test execution. +If an error occurs in a test file, the unit testing process halts and returns an error. +Consequently, it is not possible to capture all failed tests in a single command. + +### How to implement unit tests + +When adding or modifying dbt Macros in the `macros/` directory, it is essential to implement corresponding unit tests. +Place the test file in the `integration_tests/macros/` directory. + +For example, the dbt Macro located at [macros/codegen/generate_privacy_protected_model_sql.sql](macros/codegen/generate_privacy_protected_model_sql.sql) is responsible for generating a secured dbt Model. +The corresponding unit test for this macro can be found at [integration_tests/macros/tests/codegen/test_generate_privacy_protected_model_sql.sql](integration_tests/macros/tests/codegen/test_generate_privacy_protected_model_sql.sql). +This illustrates the convention of placing a dbt Macro and its associated unit test file in a structured manner. + +Custom dbt Macros, such as `assert_equals`, are available in the [integration_tests/macros/test_utils/](integration_tests/macros/test_utils/) directory to facilitate effective unit testing of dbt Macros. +Leverage these macros to create more comprehensive and robust unit tests. + +### How to run integration testing + +In addition to unit testing, we also perform integration testing for dbt_data_privacy. +This involves building dbt models, including secured models generated by the dbt_data_privacy package, on a data warehouse such as BigQuery. + +1. Generate secured models using the provided command. +2. Compile the dbt Models to ensure the generated secured models are valid. +3. Execute a `dbt build` command to thoroughly test the generated models. + +```shell +# 1. Generate secured models +make generate + +# 2. Compile dbt Models if generated secured models are valid +make compile + +# 2. Run a `dbt build` command to test generated models +make run-integration-tests +```