Release v0.1.0b1 (#237)

## Description Release v0.1.0b1 ## Checklist: - [x] Tests added/updated. - [x] Run Demo Job Locally. - [x] Documentation updated. - [x] Changelogs updated in [CHANGELOG.cdf-tk.md](https://github.com/cognitedata/cdf-project-templates/blob/main/CHANGELOG.cdf-tk.md). - [x] Template changelogs updated in [CHANGELOG.templates.md](https://github.com/cognitedata/cdf-project-templates/blob/main/CHANGELOG.templates.md). - [x] Version bumped. [_version.py](https://github.com/cognitedata/cdf-project-templates/blob/main/cognite/cognite_toolkit/_version.py) and [pyproject.toml](https://github.com/cognitedata/cdf-project-templates/blob/main/pyproject.toml) per [semantic versioning](https://semver.org/). ## Description Please describe the change you have made. ## Checklist: - [ ] Tests added/updated. - [ ] Run Demo Job Locally. - [ ] Documentation updated. - [ ] Changelogs updated in [CHANGELOG.cdf-tk.md](https://github.com/cognitedata/cdf-project-templates/blob/main/CHANGELOG.cdf-tk.md). - [ ] Template changelogs updated in [CHANGELOG.templates.md](https://github.com/cognitedata/cdf-project-templates/blob/main/CHANGELOG.templates.md). - [ ] Version bumped. [_version.py](https://github.com/cognitedata/cdf-project-templates/blob/main/cognite/cognite_toolkit/_version.py) and [pyproject.toml](https://github.com/cognitedata/cdf-project-templates/blob/main/pyproject.toml) per [semantic versioning](https://semver.org/).
cognitedata · Dec 15, 2023 · 649b19a · 649b19a
2 parents 3b7e68f + 1d42ba3
commit 649b19a
Show file tree

Hide file tree

Showing 252 changed files with 5,163 additions and 2,506 deletions.
diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
@@ -3,6 +3,7 @@ Please describe the change you have made.
 
 ## Checklist:
 - [ ] Tests added/updated.
+- [ ] Run Demo Job Locally.
 - [ ] Documentation updated.
 - [ ] Changelogs updated in [CHANGELOG.cdf-tk.md](https://github.com/cognitedata/cdf-project-templates/blob/main/CHANGELOG.cdf-tk.md).
 - [ ] Template changelogs updated in [CHANGELOG.templates.md](https://github.com/cognitedata/cdf-project-templates/blob/main/CHANGELOG.templates.md).

diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
@@ -11,7 +11,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4
-      - uses: actions/setup-python@v4
+      - uses: actions/setup-python@v5
         with:
           python-version: ${{ env.PYTHON_VERSION }}
       - uses: snok/install-poetry@v1
@@ -28,7 +28,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4
-      - uses: actions/setup-python@v4
+      - uses: actions/setup-python@v5
         with:
           python-version: ${{ env.PYTHON_VERSION }}
       - uses: snok/install-poetry@v1
@@ -40,3 +40,42 @@ jobs:
           poetry install
       - name: Run pytest
         run: pytest tests
+  dry-run-demo:
+    runs-on: ubuntu-latest
+    environment: dev
+    env:
+        CDF_CLUSTER: ${{ secrets.CDF_CLUSTER }}
+        CDF_PROJECT: ${{ secrets.CDF_PROJECT }}
+        IDP_CLIENT_ID: ${{ secrets.IDP_CLIENT_ID }}
+        IDP_CLIENT_SECRET: ${{ secrets.IDP_CLIENT_SECRET }}
+        IDP_TOKEN_URL: ${{ secrets.IDP_TOKEN_URL }}
+    name: Dry Run Demo
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ env.PYTHON_VERSION }}
+      - uses: snok/install-poetry@v1
+        with:
+          version: 1.6.1
+          virtualenvs-create: false
+      - name: Install dependencies and build the package
+        run: |
+          poetry install
+          poetry build
+      - name: Install cdf-tk
+        run: pip install .
+      - name: Initialize project
+        run: cdf-tk init demo_project
+      - name: "Pre-processing for demo environment"
+        run: python ./demo/preproc.py
+      - name: "Build the templates"
+        run: cdf-tk build --build-dir=./build --env=demo ./demo_project
+      - name: "Verify and create access rights"
+        run: cdf-tk auth verify
+      - name: "Test clean --dry-run"
+        run: |
+          cdf-tk clean --env=demo ./build --dry-run
+      - name: "Deploy the templates --dry-run"
+        run: |
+          cdf-tk deploy --drop --env=demo ./build --dry-run
diff --git a/.github/workflows/demo.yml b/.github/workflows/demo.yml
@@ -18,7 +18,7 @@ jobs:
     name: Loadmaster
     steps:
       - uses: actions/checkout@v4
-      - uses: actions/setup-python@v4
+      - uses: actions/setup-python@v5
         with:
           python-version: ${{ env.PYTHON_VERSION }}
       - uses: snok/install-poetry@v1
@@ -34,21 +34,21 @@ jobs:
       - name: Initialize project
         run: cdf-tk init demo_project
       - name: "Pre-processing for demo environment"
-        run: ./demo/preproc.sh
+        run: python ./demo/preproc.py
       - name: "Build the templates"
         run: cdf-tk build --build-dir=./build --env=demo ./demo_project
       # be careful, this works as promised
-      #- name: "Delete existing resources including data"
-      #  run: |
-      #    cdf-tk clean --env=demo ./build
+#      - name: "Delete existing resources including data"
+#        run: |
+#          cdf-tk clean --env=demo ./build --include data_models
       - name: "Verify and create access rights"
         run: cdf-tk auth verify
       - name: "Allow some time for data modeling to finish syncing of deletions"
         run: |
           sleep 30
       - name: "Deploy the templates"
         run: |
-          cdf-tk deploy --drop --drop-data --env=demo ./build
+          cdf-tk deploy --drop --env=demo ./build
       - name: "Run transformations and other post-processing"
         run: |
-          poetry run python ./demo/postproc.py
+          poetry run python ./demo/postproc.py
diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml
@@ -2,7 +2,7 @@ name: Release pypi package
 on:
   push:
     branches:
-      - release*
+      - release
 env:
   PYTHON_VERSION: '3.11'
 jobs:
@@ -11,7 +11,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4
-      - uses: actions/setup-python@v4
+      - uses: actions/setup-python@v5
         with:
           python-version: ${{ env.PYTHON_VERSION }}
       - uses: snok/install-poetry@v1
@@ -28,7 +28,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4
-      - uses: actions/setup-python@v4
+      - uses: actions/setup-python@v5
         with:
           python-version: ${{ env.PYTHON_VERSION }}
       - uses: snok/install-poetry@v1
@@ -48,7 +48,7 @@ jobs:
     name: Test build templates
     steps:
       - uses: actions/checkout@v4
-      - uses: actions/setup-python@v4
+      - uses: actions/setup-python@v5
         with:
           python-version: 3.11
       - uses: snok/install-poetry@v1
@@ -73,7 +73,7 @@ jobs:
     needs: [lint, test, build]
     steps:
       - uses: actions/checkout@v4
-      - uses: actions/setup-python@v4
+      - uses: actions/setup-python@v5
         with:
           python-version: ${{ env.PYTHON_VERSION }}
       - uses: snok/install-poetry@v1
@@ -88,4 +88,4 @@ jobs:
         env:
           TWINE_USERNAME: __token__
           TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
-        run: twine upload --verbose dist/* || echo 'Version exists'
+        run: twine upload --verbose dist/*
diff --git a/.gitignore b/.gitignore
@@ -278,4 +278,6 @@ new_project/
 # If you need to update the cognite_toolkit template files for local.yaml and config.yaml, comment below
 local.yaml
 config.yaml
-demo_project/
+!demo/config.yaml
+demo_project/
+tests/pytest-project/
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -13,7 +13,7 @@ repos:
           - --fixable=E,W,F,I,T,RUF,TID,UP
           - --target-version=py39
           - --exclude=cognite/client/_proto,cognite/client/_proto_legacy
-    rev: v0.1.5
+    rev: v0.1.7
 
   - repo: https://github.com/psf/black
     rev: 23.11.0

diff --git a/.vscode/launch.json b/.vscode/launch.json
@@ -8,7 +8,7 @@
             "name": "Python: build",
             "type": "python",
             "request": "launch",
-            "program": "./cognite_toolkit/cdf.py",
+            "program": "./cdf-tk-dev.py",
             "args": [
                 "--verbose",
                 "--override-env",
@@ -25,7 +25,7 @@
             "name": "Python: deploy",
             "type": "python",
             "request": "launch",
-            "program": "./cognite_toolkit/cdf.py",
+            "program": "./cdf-tk-dev.py",
             "args": [
                 "deploy",
                 "--dry-run",
@@ -42,7 +42,7 @@
             "name": "Python: clean",
             "type": "python",
             "request": "launch",
-            "program": "./cognite_toolkit/cdf.py",
+            "program": "./cdf-tk-dev.py",
             "args": [
                 "clean",
                 //"--dry-run",
@@ -56,7 +56,7 @@
             "name": "Python: cdf.py",
             "type": "python",
             "request": "launch",
-            "program": "./cognite_toolkit/cdf.py",
+            "program": "./cdf-tk-dev.py",
             "args": [
                 "clean",
                 "-r"

diff --git a/CHANGELOG.cdf-tk.md b/CHANGELOG.cdf-tk.md
@@ -15,14 +15,48 @@ Changes are grouped as follows:
 - `Fixed` for any bug fixes.
 - `Security` in case of vulnerabilities.
 
+## [0.1.0b1] - 2023-12-15
+
+### Added
+
+- Warnings if a configuration file is using `snake_case` when then resource type is expecting `camelCase`.
+- Added support for validation of `space` for data models.
+- Check for whether template variables `<change_me>` are present in the config files.
+- Check for whether data set id is present in the config files.
+- Print table at the end of `cdf-tk deploy` with the resources that were created, deleted, and skipped.
+- Support for Extraction Pipelines and Extraction Pipeline configuration for remotely configured Extractors 
+- Separate loader for Transformation Schedule resources.
+
+### Removed
+
+- In the `deploy` command `drop_data` option has been removed. To drop data, use the `clean` command instead.
+
+### Changed
+
+- Require all spaces to be explicitly defined as separate .space.yaml file.
+- The `data_set_id` for `Transformations` must now be set explicitly in the yaml config file for the `Transformation`
+  under the `data_set_id` key. Note that you also need to explicitly define the `data_set` in its own yaml config file.
+- All config files have been merged to a single config file, `config.yaml`. Upon calling `cdf-tk init` the `config.yaml`
+  is created in the root folder of the project based on the `default.config.yaml` file of each module.
+- DataSetID is no longer set implicitly when running the `cdf-tk deploy` command. Instead, the `data_set_id` must be
+  set explicitly in the yaml config file.
+
+### Fixed
+
+- When running `cdf-tk deploy` with `--dry-run` a `ValueError` was raised if not all datasets were pre-existing.
+  This is now fixed by skipping dataset validation when running with `--dry-run`.
+- When having a `auth` group with mixed capabilities of all scoped and resource scoped, the all scoped capabilities
+  were not removed when running `cdf-tk deploy`. This is now fixed.
+- Loading `Transformation` did not support setting `dataSetExternalId` in the yaml config file. This is now fixed.
+
 ## [0.1.0a3] - 2023-12-01
 
 ### Changed
 
 - Refactored load functionality. Loading raw tables and files now requires a `yaml` file with metadata.
 - Fix container comparison to detect identical containers when loading data models (without --drop flag).
 - Clean up error on resource does not exist when deleting (on `deploy --drop` or using clean command).
-  
+
 ### Added
 
 - Support for loading `data_sets`.
@@ -43,6 +77,8 @@ Changes are grouped as follows:
 - Missing .sql files for transformations will now raise an error in the build step.
 - The build step will now raise a number of warnings for missing externalIds in the yaml files,
   as well as if the naming conventions are not followed.
+- System section in `environments.yaml` to track local state of `cdf-toolkit`.
+- Introduced a `build_environment.yaml` in the `/build` folder to track how the build was run.
 
 ### Fixed
 

diff --git a/CHANGELOG.templates.md b/CHANGELOG.templates.md
@@ -1,4 +1,5 @@
 # Changelog
+
 All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
@@ -13,16 +14,39 @@ Changes are grouped as follows:
 - `Removed` for now removed features.
 - `Fixed` for any bug fixes.
 - `Security` in case of vulnerabilities.
-
-## [0.2.0] - 2023-12-01
-
+
+## [0.1.0b1] - 2023-12-15
+
+### Added
+
+- Explicitly define model `space` in `experimental/cdf_asset_source_model/` and `experimental/example_pump_model/`.
+- The module `my_example_module` has been added to the `custom_modules` folder.
+- Added globally defined schedule variables that can be used across all modules.
+- A complete example of an Asset data pipeline in `examples/cdf_asset_data_pipeline/` shows how to configure an Extractor, monitor the status of the Extraction Pipeline, and load the data into the asset hierarchy using Transformations.
+- DataSet to all example modules: `cdf_apm_simple_data_model`, `cdf_asset_source_model`, `cdf_oid_example_data`, 
+  `example_pump_data_model`, `example_pump_asset_hierarchy`.
+
 ### Changed
 
 - **BREAKING** All externalIds and names have been changed to follow the naming conventions for resources
   in `examples/cdf_oid_example_data`, `examples/cdf_apm_simple_data_model`, `modules/cdf_apm_base`,
   `modules/cdf_infield_common`, and `modules/cdf_infield_location`.
+- **BREAKING** Transformation Schedules broken out into separate files, following naming convention `<transformation_name>.schedule.yaml`. 
+- All cognite templates have been moved into `cognite_templates` folder, while `local_templates` is renamed to `custom_templates`.
+- Move cdf_apm_base into separate folder.
+- The file `local.yaml` has been renamed `environments.yaml` to better reflect its purpose.
+- Removed demo `sourceId` from `cdf_infield_location` module.
+- Changed the isPaused flag to use a module-level variable instead of hardcoded in `cdf_apm_simple_data_model`.
+- Combined the child and parent transformations `sync_assets_from_hierarchy_to_apm` in `cdf_infield_location`. 
+  This has the benefit of not having to wait for the parent transformation to finish before starting the child transformation, 
+  thus no longer a dependency between the two transformations.
+
+### Fixed
+
+- Removed transformation identity provider variables from modules and reused the global cicd_ prefixed ones.
+- Ensure all transformations in `cognite_modules` are prefixed with `tr_` and all spaces are prefixed with `sp_`.
 
-## [0.1.2] - 2023-11-29
+## [0.1.0a3] - 2023-11-29
 
 ### Changed
 
@@ -34,9 +58,9 @@ Changes are grouped as follows:
 - Add space yaml files for existing data models when explicit space definition was introduced.
 - Fix use of integer value in version for data models.
 - Fix wrong reference to `apm_simple` in `examples/cdf_apm_simple_data_model` and `modules/cdf_infield_location`.
-- Examplify use of a single config yaml file for multiple file resources in `examples/cdf_oid_example_data/files/files.yaml`.
+- Exemplify use of a single config yaml file for multiple file resources in `examples/cdf_oid_example_data/files/files.yaml`.
 
-## [0.1.1] - 2023-11-23
+## [0.1.0a2] - 2023-11-23
 
 ### Changed
 
@@ -51,6 +75,6 @@ Changes are grouped as follows:
 - cdf_infield_common module and the auth applications-configuration.yaml did not load group source id
    correctly due to source_id being used instead of sourceId. This is now fixed.
 
-## [0.1.0] - 2023-11-21
+## [0.1.0a1] - 2023-11-21
 
 Initial release