Release 0.1.0b5 (#284)

# Description Please describe the change you have made. ## Checklist - [ ] Tests added/updated. - [ ] Run Demo Job Locally. - [ ] Documentation updated. - [ ] Changelogs updated in [CHANGELOG.cdf-tk.md](https://github.com/cognitedata/cdf-project-templates/blob/main/CHANGELOG.cdf-tk.md). - [ ] Template changelogs updated in [CHANGELOG.templates.md](https://github.com/cognitedata/cdf-project-templates/blob/main/CHANGELOG.templates.md). - [ ] Version bumped. [_version.py](https://github.com/cognitedata/cdf-project-templates/blob/main/cognite/cognite_toolkit/_version.py) and [pyproject.toml](https://github.com/cognitedata/cdf-project-templates/blob/main/pyproject.toml) per [semantic versioning](https://semver.org/).
cognitedata · Jan 11, 2024 · 7b658cd · 7b658cd
2 parents b9bf049 + 264e9da
commit 7b658cd
Show file tree

Hide file tree

Showing 42 changed files with 2,112 additions and 1,873 deletions.
diff --git a/CHANGELOG.cdf-tk.md b/CHANGELOG.cdf-tk.md
@@ -15,6 +15,23 @@ Changes are grouped as follows:
 - `Fixed` for any bug fixes.
 - `Security` in case of vulnerabilities.
 
+## [0.1.0b5] - 2024-01-11
+
+### Added
+
+- Support for custom environment variables injected into build files when calling the command `cdf-tk deploy`.
+- All resources that are unchanged are now skipped when running `cdf-tk deploy`.
+- Support for loading `Group` Capabilities with scope `idScope` of type string. This means you can now set the
+  `idScope` to the external id of a `dataSet` and it will be automatically replaced by the dataset id
+  `cdf-tk deploy`.
+
+### Fixed
+
+- Fixed bug when calling any command loading a `.env` file and the path is not relative to the current working
+  directory. This is now fixed.
+- Calling `cdf-tk init --upgrade` overwrote all variables and comments set in the `config.yaml` file. This is now
+  fixed.
+
 ## [0.1.0b4] - 2024-01-08
 
 ### Added

diff --git a/CHANGELOG.templates.md b/CHANGELOG.templates.md
@@ -1,96 +1,100 @@
-# Changelog
-
-All notable changes to this project will be documented in this file.
-
-The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
-and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
-
-Changes are grouped as follows:
-
-- `Added` for new features.
-- `Changed` for changes in existing functionality.
-- `Deprecated` for soon-to-be removed features.
-- `Improved` for transparent changes, e.g. better performance.
-- `Removed` for now removed features.
-- `Fixed` for any bug fixes.
-- `Security` in case of vulnerabilities.
-
-## [0.1.0b4] - 2024-01-08
-
-No changes to templates.
-
-## [0.1.0b3] - 2024-01-02
-
-No changes to templates.
-
-## [0.1.0b2] - 2023-12-17
-
-### Fixed
-
-- In the package `example_pump` ensure all transformations are prefixed with `tr_`.
-
-## [0.1.0b1] - 2023-12-15
-
-### Added
-
-- Explicitly define model `space` in `experimental/cdf_asset_source_model/` and `experimental/example_pump_model/`.
-- The module `my_example_module` has been added to the `custom_modules` folder.
-- Added globally defined schedule variables that can be used across all modules.
-- A complete example of an Asset data pipeline in `examples/cdf_asset_data_pipeline/` shows how to configure an
-  Extractor, monitor the status of the Extraction Pipeline, and load the data into the asset hierarchy using Transformations.
-- DataSet to all example modules: `cdf_apm_simple_data_model`, `cdf_asset_source_model`, `cdf_oid_example_data`,
-  `example_pump_data_model`, `example_pump_asset_hierarchy`.
-
-### Changed
-
-- **BREAKING** All externalIds and names have been changed to follow the naming conventions for resources
-  in `examples/cdf_oid_example_data`, `examples/cdf_apm_simple_data_model`, `modules/cdf_apm_base`,
-  `modules/cdf_infield_common`, and `modules/cdf_infield_location`.
-- **BREAKING** Transformation Schedules broken out into separate files, following naming convention `<transformation_name>.schedule.yaml`.
-- All cognite templates have been moved into `cognite_templates` folder, while `local_templates` is renamed to `custom_templates`.
-- Move cdf_apm_base into separate folder.
-- The file `local.yaml` has been renamed `environments.yaml` to better reflect its purpose.
-- Removed demo `sourceId` from `cdf_infield_location` module.
-- Changed the isPaused flag to use a module-level variable instead of hardcoded in `cdf_apm_simple_data_model`.
-- Combined the child and parent transformations `sync_assets_from_hierarchy_to_apm` in `cdf_infield_location`.
-  This has the benefit of not having to wait for the parent transformation to finish before starting the child transformation,
-  thus no longer a dependency between the two transformations.
-
-### Fixed
-
-- Removed transformation identity provider variables from modules and reused the global cicd_ prefixed ones.
-- Ensure all transformations in `cognite_modules` are prefixed with `tr_` and all spaces are prefixed with `sp_`.
-
-## [0.1.0a3] - 2023-11-29
-
-### Changed
-
-- Remove unused template_version variable from groups and use of group metadata.
-- Split up cdf_oid_example_data into data sets and RAW databases per source system.
-
-### Fixed
-
-- Add space yaml files for existing data models when explicit space definition was introduced.
-- Fix use of integer value in version for data models.
-- Fix wrong reference to `apm_simple` in `examples/cdf_apm_simple_data_model` and `modules/cdf_infield_location`.
-- Exemplify use of a single config yaml file for multiple file resources in `examples/cdf_oid_example_data/files/files.yaml`.
-
-## [0.1.0a2] - 2023-11-23
-
-### Changed
-
-- Changed format of infield external_ids to be more readable, moving `_dataset` (ds) and `_space` to the beginning of
-  the external_id.
-- `examples/cdf_apm_simple/raw` and `examples/example_dump_asst_hierarchy/raw` now explicitly
-  defines database and table name in `.yaml` files for each table.
-- Added `data_set` to `examples/example_dump_asst_hierarchy/`, which was implicitly defined in
-  before.
-
-### Fixed
-
-- cdf_infield_common module and the auth applications-configuration.yaml did not load group source id
-   correctly due to source_id being used instead of sourceId. This is now fixed.
-
-## [0.1.0a1] - 2023-11-21
-
-Initial release
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+Changes are grouped as follows:
+
+- `Added` for new features.
+- `Changed` for changes in existing functionality.
+- `Deprecated` for soon-to-be removed features.
+- `Improved` for transparent changes, e.g. better performance.
+- `Removed` for now removed features.
+- `Fixed` for any bug fixes.
+- `Security` in case of vulnerabilities.
+
+## [0.1.0b5] - 2024-01-11
+
+No changes to templates.
+
+## [0.1.0b4] - 2024-01-08
+
+No changes to templates.
+
+## [0.1.0b3] - 2024-01-02
+
+No changes to templates.
+
+## [0.1.0b2] - 2023-12-17
+
+### Fixed
+
+- In the package `example_pump` ensure all transformations are prefixed with `tr_`.
+
+## [0.1.0b1] - 2023-12-15
+
+### Added
+
+- Explicitly define model `space` in `experimental/cdf_asset_source_model/` and `experimental/example_pump_model/`.
+- The module `my_example_module` has been added to the `custom_modules` folder.
+- Added globally defined schedule variables that can be used across all modules.
+- A complete example of an Asset data pipeline in `examples/cdf_asset_data_pipeline/` shows how to configure an
+  Extractor, monitor the status of the Extraction Pipeline, and load the data into the asset hierarchy using Transformations.
+- DataSet to all example modules: `cdf_apm_simple_data_model`, `cdf_asset_source_model`, `cdf_oid_example_data`,
+  `example_pump_data_model`, `example_pump_asset_hierarchy`.
+
+### Changed
+
+- **BREAKING** All externalIds and names have been changed to follow the naming conventions for resources
+  in `examples/cdf_oid_example_data`, `examples/cdf_apm_simple_data_model`, `modules/cdf_apm_base`,
+  `modules/cdf_infield_common`, and `modules/cdf_infield_location`.
+- **BREAKING** Transformation Schedules broken out into separate files, following naming convention `<transformation_name>.schedule.yaml`.
+- All cognite templates have been moved into `cognite_templates` folder, while `local_templates` is renamed to `custom_templates`.
+- Move cdf_apm_base into separate folder.
+- The file `local.yaml` has been renamed `environments.yaml` to better reflect its purpose.
+- Removed demo `sourceId` from `cdf_infield_location` module.
+- Changed the isPaused flag to use a module-level variable instead of hardcoded in `cdf_apm_simple_data_model`.
+- Combined the child and parent transformations `sync_assets_from_hierarchy_to_apm` in `cdf_infield_location`.
+  This has the benefit of not having to wait for the parent transformation to finish before starting the child transformation,
+  thus no longer a dependency between the two transformations.
+
+### Fixed
+
+- Removed transformation identity provider variables from modules and reused the global cicd_ prefixed ones.
+- Ensure all transformations in `cognite_modules` are prefixed with `tr_` and all spaces are prefixed with `sp_`.
+
+## [0.1.0a3] - 2023-11-29
+
+### Changed
+
+- Remove unused template_version variable from groups and use of group metadata.
+- Split up cdf_oid_example_data into data sets and RAW databases per source system.
+
+### Fixed
+
+- Add space yaml files for existing data models when explicit space definition was introduced.
+- Fix use of integer value in version for data models.
+- Fix wrong reference to `apm_simple` in `examples/cdf_apm_simple_data_model` and `modules/cdf_infield_location`.
+- Exemplify use of a single config yaml file for multiple file resources in `examples/cdf_oid_example_data/files/files.yaml`.
+
+## [0.1.0a2] - 2023-11-23
+
+### Changed
+
+- Changed format of infield external_ids to be more readable, moving `_dataset` (ds) and `_space` to the beginning of
+  the external_id.
+- `examples/cdf_apm_simple/raw` and `examples/example_dump_asst_hierarchy/raw` now explicitly
+  defines database and table name in `.yaml` files for each table.
+- Added `data_set` to `examples/example_dump_asst_hierarchy/`, which was implicitly defined in
+  before.
+
+### Fixed
+
+- cdf_infield_common module and the auth applications-configuration.yaml did not load group source id
+   correctly due to source_id being used instead of sourceId. This is now fixed.
+
+## [0.1.0a1] - 2023-11-21
+
+Initial release
diff --git a/README.md b/README.md
@@ -20,7 +20,7 @@ More details about the tool can be found at
 [developer.cognite.com](http://developer.cognite.com/sdks/toolkit).
 
 You can find an overview of the modules and packages in the
-[module and package documentation](http://developer.cognite.com/sdks/toolkit/modules).
+[module and package documentation](https://developer.cognite.com/sdks/toolkit/references/module_reference).
 
 See [./CONTRIBUTING.md](./CONTRIBUTING.md) for information about how to contribute to the `cdf-tk` tool or
 templates.
diff --git a/cognite_toolkit/_version.py b/cognite_toolkit/_version.py
@@ -1 +1 @@
-__version__ = "0.1.0b4"
+__version__ = "0.1.0b5"
diff --git a/cognite_toolkit/cdf.py b/cognite_toolkit/cdf.py
@@ -37,8 +37,8 @@
     CUSTOM_MODULES,
     ENVIRONMENTS_FILE,
     BuildEnvironment,
+    ConfigYAML,
     build_config,
-    generate_config,
     read_yaml_file,
 )
 from cognite_toolkit.cdf_tk.utils import CDFToolConfig
@@ -159,7 +159,11 @@ def common(
 
     if dotenv_file.is_file():
         if verbose:
-            print(f"Loading .env file: {dotenv_file.relative_to(Path.cwd())!s}")
+            try:
+                path_str = dotenv_file.relative_to(Path.cwd())
+            except ValueError:
+                path_str = dotenv_file.absolute()
+            print(f"Loading .env file: {path_str!s}")
         load_dotenv(dotenv_file, override=override_env)
 
     ctx.obj = Common(
@@ -672,7 +676,7 @@ def main_init(
     if not dry_run and not upgrade:
         target_dir.mkdir(exist_ok=True)
     if upgrade:
-        print("  Will upgrade modules and files in place, config.yaml files will not be touched.")
+        print("  Will upgrade modules and files in place.")
     print(f"Will copy these files to {target_dir}:")
     print(files_to_copy)
     print(f"Will copy these module directories to {target_dir}:")
@@ -744,21 +748,20 @@ def main_init(
             print(f"You project in {target_dir} was upgraded.")
         else:
             print(f"A new project was created in {target_dir}.")
-        if upgrade:
-            print("  All default variables from the modules have been upgraded.")
-            print("  Please check you config.yaml file for new default variables that may need to be changed.")
-
     config_filepath = target_dir / "config.yaml"
     if not dry_run:
         if clean or not config_filepath.exists():
-            config_str, _ = generate_config(target_dir)
-            config_filepath.write_text(config_str)
+            config_yaml = ConfigYAML.load(target_dir)
+            config_filepath.write_text(config_yaml.dump_yaml_with_comments(indent_size=2))
             print(f"Created your config.yaml file in {target_dir}.")
         else:
             current = config_filepath.read_text()
-            config_str, difference = generate_config(target_dir, existing_config=current)
-            config_filepath.write_text(config_str)
-            print(str(difference))
+            config_yaml = ConfigYAML.load(target_dir, existing_config_yaml=current)
+            config_filepath.write_text(config_yaml.dump_yaml_with_comments(indent_size=2))
+            print(str(config_yaml))
+            if ctx.obj.verbose:
+                for added in config_yaml.added:
+                    print(f"  [bold green]ADDED:[/] {added}")
 
 
 @describe_app.callback(invoke_without_command=True)