Merge pull request #66 from fivetran/T-371104

updates coming out of applying ns2 internally

.buildkite/hooks/pre-command

@@ -21,4 +21,5 @@ export CI_SNOWFLAKE_DBT_USER=$(gcloud secrets versions access latest --secret="C
 export CI_SNOWFLAKE_DBT_WAREHOUSE=$(gcloud secrets versions access latest --secret="CI_SNOWFLAKE_DBT_WAREHOUSE" --project="dbt-package-testing-363917")
 export CI_DATABRICKS_DBT_HOST=$(gcloud secrets versions access latest --secret="CI_DATABRICKS_DBT_HOST" --project="dbt-package-testing-363917")
 export CI_DATABRICKS_DBT_HTTP_PATH=$(gcloud secrets versions access latest --secret="CI_DATABRICKS_DBT_HTTP_PATH" --project="dbt-package-testing-363917")
-export CI_DATABRICKS_DBT_TOKEN=$(gcloud secrets versions access latest --secret="CI_DATABRICKS_DBT_TOKEN" --project="dbt-package-testing-363917")
+export CI_DATABRICKS_DBT_TOKEN=$(gcloud secrets versions access latest --secret="CI_DATABRICKS_DBT_TOKEN" --project="dbt-package-testing-363917")
+export CI_DATABRICKS_DBT_CATALOG=$(gcloud secrets versions access latest --secret="CI_DATABRICKS_DBT_CATALOG" --project="dbt-package-testing-363917")

.buildkite/pipeline.yml

@@ -69,5 +69,6 @@ steps:
             - "CI_DATABRICKS_DBT_HOST"
             - "CI_DATABRICKS_DBT_HTTP_PATH"
             - "CI_DATABRICKS_DBT_TOKEN"
+            - "CI_DATABRICKS_DBT_CATALOG"
     commands: |
       bash .buildkite/scripts/run_models.sh databricks

CHANGELOG.md

@@ -1,3 +1,20 @@
+# dbt_netsuite v0.8.0
+[PR #66](https://github.com/fivetran/dbt_netsuite/pull/66) applies the following changes:
+
+## 🚨 Breaking Changes 🚨 (Netsuite.com Endpoint Only)
+- Adds `transaction_id` and `transaction_line_id` to the Netsuite (1) income statement and balance sheet models. These fields were already present in the Netsuite2 versions of these models. These columns are included in newly added Primary Key tests on the Netsuite (1) income statement and balance sheet models.
+> **Note**: Ensure that neither of these columns are included in your `balance_sheet_transaction_detail_columns` or `income_statement_transaction_detail_columns` variables.
+
+## Under the Hood
+- Aligns join types in Netsuite end models with the joins in Netsuite2 end models.
+- Adds new account type <> account category mappings in the `int_netsuite__transactions_with_converted_amounts`/`int_netsuite2__tran_with_converted_amounts` model. 
+  - `Prepaid Expense` account types are treated as `Deferred Expense` accounts. 
+  - `Non Posting` and `Statistical` account types will be placed in a new `Other` category.
+- Adds an `accepted_values` test on the transaction detail end models that will raise a **warning** if unexpected account types are encountered. 
+- Adds a [DECISIONLOG](https://github.com/fivetran/dbt_netsuite/DECISIONLOG.md).
+- Updates README to include the `netsuite2__using_jobs` variable.
+- Adds uniqueness and not-null tests to the Netsuite (1) income statement and balance sheet models. These tests were already present in the Netsuite2 models.
+
 # dbt_netsuite v0.7.1
 
 ## 🎉 Feature Updates 🎉

DECISIONLOG.md

@@ -0,0 +1,11 @@
+# Decision Log
+
+In creating this package, which is meant for a wide range of use cases, we had to take opinionated stances on a few different questions we came across during development. We've consolidated significant choices we made here, and will continue to update as the package evolves. 
+
+## Testing Expected Account Type Values 
+
+In the `int_netsuite2_tran_with_converted_amounts`/`int_netsuite2_tran_with_converted_amounts` models, account types are bucketed into broader account _categories_. We bucket account types through a `case when` statement that hard-codes the mapping of each account type into its appropriate category. Thus, because this is hard-coded, we have added an `accepted_values` test on the transaction detail end models. It will raise a **warning** if unexpected account types are encountered. 
+
+## Creation of 'Other' Account Category for Non Posting and Statistical Account Types
+
+As mentioned above, in the `int_netsuite2_tran_with_converted_amounts`/`int_netsuite2_tran_with_converted_amounts` models, account types are bucketed into broader account _categories_. There is no standard category for `Non Posting` and `Statistical` account types and transactions from these kinds of accounts are excluded from all financial reports. However, they are used for other workflows in Netsuite, so we have bucketed `Non Posting` and `Statistical` account types into a new `Other` account category.

README.md

@@ -99,7 +99,7 @@ Include the following netsuite package version in your `packages.yml` file:
 ```yaml
 packages:
   - package: fivetran/netsuite
-    version: [">=0.7.0", "<0.8.0"]
+    version: [">=0.8.0", "<0.9.0"]
 ```
 ## Step 3: Define Netsuite.com or Netsuite2 Source
 As of April 2022 Fivetran made available a new Netsuite connector which leverages the Netsuite2 endpoint opposed to the original Netsuite.com endpoint. This package is designed to run for either or, not both. By default the `netsuite_data_model` variable for this package is set to the original `netsuite` value which runs the netsuite.com version of the package. If you would like to run the package on Netsuite2 data, you may adjust the `netsuite_data_model` variable to run the `netsuite2` version of the package.
@@ -123,8 +123,9 @@ It's possible that your Netsuite connector does not sync every table that this p
 vars:
     netsuite2__multibook_accounting_enabled: false # True by default. Disable `accountingbooksubsidiary` and `accountingbook` if you are not using the Multi-Book Accounting feature
     netsuite2__using_vendor_categories: false # True by default. Disable `vendorcategory` if you don't categorize your vendors
+    netsuite2__using_jobs: false # True by default. Disable `job` if you don't use jobs
 ```
-> **Note**: The Netsuite dbt package currently only supports disabling transforms of [Multi-Book Accounting](https://docs.oracle.com/en/cloud/saas/netsuite/ns-online-help/book_3831565332.html) related tables (`accountingbooksubsidiary` and `accountingbook`) and the `vendorcategory` source table. Please create an issue to request additional tables and/or [features](https://docs.oracle.com/en/cloud/saas/netsuite/ns-online-help/bridgehead_N233872.html) to exclude. 
+> **Note**: The Netsuite dbt package currently only supports disabling transforms of [Multi-Book Accounting](https://docs.oracle.com/en/cloud/saas/netsuite/ns-online-help/book_3831565332.html) related tables (`accountingbooksubsidiary` and `accountingbook`) and the `vendorcategory` and `job` source tables. Please create an issue to request additional tables and/or [features](https://docs.oracle.com/en/cloud/saas/netsuite/ns-online-help/bridgehead_N233872.html) to exclude. 
 > 
 > To determine if a table or field is activated by a feature, access the [Records Catalog](https://docs.oracle.com/en/cloud/saas/netsuite/ns-online-help/article_159367781370.html).
 

dbt_project.yml

@@ -1,6 +1,6 @@
 config-version: 2
 name: 'netsuite'
-version: '0.7.1'
+version: '0.8.0'
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 
 models:

integration_tests/ci/sample.profiles.yml

@@ -16,13 +16,13 @@ integration_tests:
       pass: "{{ env_var('CI_REDSHIFT_DBT_PASS') }}"
       dbname: "{{ env_var('CI_REDSHIFT_DBT_DBNAME') }}"
       port: 5439
-      schema: netsuite_integration_tests
+      schema: netsuite_integration_tests_1
       threads: 8
     bigquery:
       type: bigquery
       method: service-account-json
       project: 'dbt-package-testing'
-      schema: netsuite_integration_tests
+      schema: netsuite_integration_tests_1
       threads: 8
       keyfile_json: "{{ env_var('GCLOUD_SERVICE_KEY') | as_native }}"
     snowflake:
@@ -33,7 +33,7 @@ integration_tests:
       role: "{{ env_var('CI_SNOWFLAKE_DBT_ROLE') }}"
       database: "{{ env_var('CI_SNOWFLAKE_DBT_DATABASE') }}"
       warehouse: "{{ env_var('CI_SNOWFLAKE_DBT_WAREHOUSE') }}"
-      schema: netsuite_integration_tests
+      schema: netsuite_integration_tests_1
       threads: 8
     postgres:
       type: postgres
@@ -42,13 +42,13 @@ integration_tests:
       pass: "{{ env_var('CI_POSTGRES_DBT_PASS') }}"
       dbname: "{{ env_var('CI_POSTGRES_DBT_DBNAME') }}"
       port: 5432
-      schema: netsuite_integration_tests
+      schema: netsuite_integration_tests_1
       threads: 8
     databricks:
-      catalog: null
+      catalog: "{{ env_var('CI_DATABRICKS_DBT_CATALOG') }}"
       host: "{{ env_var('CI_DATABRICKS_DBT_HOST') }}"
       http_path: "{{ env_var('CI_DATABRICKS_DBT_HTTP_PATH') }}"
-      schema: netsuite_integration_tests
+      schema: netsuite_integration_tests_1
       threads: 2
       token: "{{ env_var('CI_DATABRICKS_DBT_TOKEN') }}"
       type: databricks

integration_tests/dbt_project.yml

@@ -1,11 +1,11 @@
 name: 'netsuite_integration_tests'
-version: '0.7.1'
+version: '0.8.0'
 profile: 'integration_tests'
 config-version: 2
 models:
   +materialized: table
 vars:
-  netsuite_schema: netsuite_integration_tests
+  netsuite_schema: netsuite_integration_tests_1
   netsuite_data_model_override: netsuite
   netsuite_source:
     # Netsuite Seed Data

models/netsuite.yml

@@ -48,7 +48,22 @@ models:
       either Retained Earnings or Net Income.  The Cumulative Translation
       Adjustment total, which in most cases does not have transactions
       associated with it, is calculated manually in the second part of the query.
+    tests:
+      - dbt_utils.unique_combination_of_columns:
+          combination_of_columns:
+            - transaction_line_id
+            - transaction_id
+            - accounting_period_id
+            - account_name
     columns:
+    - name: transaction_id
+      description: Netsuite internal transaction ID.
+      tests:
+        - not_null
+    - name: transaction_line_id
+      description: Netsuite internal transaction line ID.
+      tests:
+        - not_null
     - name: accounting_period_id
       description: The unique identifier of the accounting period.
     - name: accounting_period_ending
@@ -81,7 +96,22 @@ models:
       The income statement query uses the transactions_with_converted_amount transformation 
       to recreate all lines necessary for the income statement.  It also joins in class, 
       department and location information for enhanced reporting. 
+    tests:
+      - dbt_utils.unique_combination_of_columns:
+          combination_of_columns:
+            - transaction_line_id
+            - transaction_id
+            - accounting_period_id
+            - account_name
     columns:
+    - name: transaction_id
+      description: Netsuite internal transaction ID.
+      tests:
+        - not_null
+    - name: transaction_line_id
+      description: Netsuite internal transaction line ID.
+      tests:
+        - not_null
     - name: accounting_period_id
       description: The unique identifier of the accounting period.
     - name: accounting_period_ending
@@ -171,6 +201,11 @@ models:
       description: Name of the account.
     - name: account_type_name
       description: The accounts type name.
+      tests:
+        - accepted_values:
+            values: ['Accounts Receivable', 'Bank', 'Deferred Expense', 'Fixed Asset', 'Other Asset', 'Other Current Asset', 'Unbilled Receivable', 'Prepaid Expense', 'Cost of Goods Sold', 'Expense', 'Other Expense', 'Income', 'Other Income', 'Accounts Payable', 'Credit Card', 'Deferred Revenue', 'Long Term Liability', 'Other Current Liability', 'Equity', 'Retained Earnings', 'Net Income', 'Non Posting', 'Statistical']
+            config:
+              severity: warn
     - name: account_id
       description: Unique identifier of the account.
     - name: account_number

models/netsuite/intermediate/int_netsuite__transactions_with_converted_amounts.sql

@@ -52,16 +52,17 @@ transactions_with_converted_amounts as (
       else false 
         end as is_income_statement,
     case
-      when lower(accounts.type_name) in ('accounts receivable', 'bank', 'deferred expense', 'fixed asset', 'other asset', 'other current asset', 'unbilled receivable') then 'Asset'
+      when lower(accounts.type_name) in ('accounts receivable', 'bank', 'deferred expense', 'fixed asset', 'other asset', 'other current asset', 'unbilled receivable', 'prepaid expense') then 'Asset'
       when lower(accounts.type_name) in ('cost of goods sold', 'expense', 'other expense') then 'Expense'
       when lower(accounts.type_name) in ('income', 'other income') then 'Income'
       when lower(accounts.type_name) in ('accounts payable', 'credit card', 'deferred revenue', 'long term liability', 'other current liability') then 'Liability'
       when lower(accounts.type_name) in ('equity', 'retained earnings', 'net income') then 'Equity'
+      when lower(accounts.type_name) in ('non posting', 'statistical') then 'Other'
       else null 
         end as account_category
   from transactions_in_every_calculation_period_w_exchange_rates
 
-  join accounts 
+  left join accounts 
     on accounts.account_id = transactions_in_every_calculation_period_w_exchange_rates.account_id 
 )
 

models/netsuite/netsuite__balance_sheet.sql

@@ -30,6 +30,8 @@ subsidiaries as (
 
 balance_sheet as ( 
   select
+    transactions_with_converted_amounts.transaction_id,
+    transactions_with_converted_amounts.transaction_line_id,
     reporting_accounting_periods.accounting_period_id as accounting_period_id,
     reporting_accounting_periods.ending_at as accounting_period_ending,
     reporting_accounting_periods.full_name as accounting_period_full_name,
@@ -96,19 +98,19 @@ balance_sheet as (
 
   --Below is only used if balance sheet transaction detail columns are specified dbt_project.yml file.
   {% if var('balance_sheet_transaction_detail_columns') != []%}
-  join transaction_details
+  left join transaction_details
     on transaction_details.transaction_id = transactions_with_converted_amounts.transaction_id
       and transaction_details.transaction_line_id = transactions_with_converted_amounts.transaction_line_id
   {% endif %}
 
 
-  join accounts 
+  left join accounts 
     on accounts.account_id = transactions_with_converted_amounts.account_id
 
-  join accounting_periods as reporting_accounting_periods 
+  left join accounting_periods as reporting_accounting_periods 
     on reporting_accounting_periods.accounting_period_id = transactions_with_converted_amounts.reporting_accounting_period_id
 
-  join accounting_periods as transaction_accounting_periods 
+  left join accounting_periods as transaction_accounting_periods 
     on transaction_accounting_periods.accounting_period_id = transactions_with_converted_amounts.transaction_accounting_period_id
 
   where reporting_accounting_periods.fiscal_calendar_id = (select fiscal_calendar_id from subsidiaries where parent_id is null)
@@ -119,6 +121,8 @@ balance_sheet as (
   union all
 
   select
+    transactions_with_converted_amounts.transaction_id,
+    transactions_with_converted_amounts.transaction_line_id,
     reporting_accounting_periods.accounting_period_id as accounting_period_id,
     reporting_accounting_periods.ending_at as accounting_period_ending,
     reporting_accounting_periods.full_name as accounting_period_full_name,
@@ -154,15 +158,15 @@ balance_sheet as (
 
   --Below is only used if balance sheet transaction detail columns are specified dbt_project.yml file.
   {% if var('balance_sheet_transaction_detail_columns') != []%}
-  join transaction_details
+  left join transaction_details
     on transaction_details.transaction_id = transactions_with_converted_amounts.transaction_id
       and transaction_details.transaction_line_id = transactions_with_converted_amounts.transaction_line_id
   {% endif %}
 
-  join accounts
+  left join accounts
     on accounts.account_id = transactions_with_converted_amounts.account_id
 
-  join accounting_periods as reporting_accounting_periods 
+  left join accounting_periods as reporting_accounting_periods 
     on reporting_accounting_periods.accounting_period_id = transactions_with_converted_amounts.reporting_accounting_period_id
 
   where reporting_accounting_periods.fiscal_calendar_id = (select fiscal_calendar_id from subsidiaries where parent_id is null)

models/netsuite/netsuite__income_statement.sql

@@ -50,6 +50,8 @@ departments as (
 
 income_statement as (
     select
+        transactions_with_converted_amounts.transaction_id,
+        transactions_with_converted_amounts.transaction_line_id,
         reporting_accounting_periods.accounting_period_id as accounting_period_id,
         reporting_accounting_periods.ending_at as accounting_period_ending,
         reporting_accounting_periods.full_name as accounting_period_full_name,

models/netsuite2.yml

@@ -195,6 +195,11 @@ models:
       description: Name of the account.
     - name: account_type_name
       description: The accounts type name.
+      tests:
+        - accepted_values:
+            values: ['Accounts Receivable', 'Bank', 'Deferred Expense', 'Fixed Asset', 'Other Asset', 'Other Current Asset', 'Unbilled Receivable', 'Prepaid Expense', 'Cost of Goods Sold', 'Expense', 'Other Expense', 'Income', 'Other Income', 'Accounts Payable', 'Credit Card', 'Deferred Revenue', 'Long Term Liability', 'Other Current Liability', 'Equity', 'Retained Earnings', 'Net Income', 'Non Posting', 'Statistical']
+            config:
+              severity: warn
     - name: account_id
       description: Unique identifier of the account.
     - name: account_number

models/netsuite2/intermediate/int_netsuite2__tran_with_converted_amounts.sql

@@ -28,15 +28,15 @@ transactions_in_every_calculation_period_w_exchange_rates as (
     exchange_transaction_period.exchange_rate as exchange_rate_transaction_period
   from transaction_lines_w_accounting_period
 
-  join transaction_and_reporting_periods 
+  left join transaction_and_reporting_periods 
     on transaction_and_reporting_periods.accounting_period_id = transaction_lines_w_accounting_period.transaction_accounting_period_id 
 
-  join accountxperiod_exchange_rate_map as exchange_reporting_period
+  left join accountxperiod_exchange_rate_map as exchange_reporting_period
     on exchange_reporting_period.accounting_period_id = transaction_and_reporting_periods.reporting_accounting_period_id
       and exchange_reporting_period.account_id = transaction_lines_w_accounting_period.account_id
       and exchange_reporting_period.from_subsidiary_id = transaction_lines_w_accounting_period.subsidiary_id
 
-  join accountxperiod_exchange_rate_map as exchange_transaction_period
+  left join accountxperiod_exchange_rate_map as exchange_transaction_period
     on exchange_transaction_period.accounting_period_id = transaction_and_reporting_periods.accounting_period_id
       and exchange_transaction_period.account_id = transaction_lines_w_accounting_period.account_id
       and exchange_transaction_period.from_subsidiary_id = transaction_lines_w_accounting_period.subsidiary_id
@@ -52,16 +52,17 @@ transactions_with_converted_amounts as (
       else false 
         end as is_income_statement,
     case
-      when lower(accounts.type_name) in ('accounts receivable', 'bank', 'deferred expense', 'fixed asset', 'other asset', 'other current asset', 'unbilled receivable') then 'Asset'
+      when lower(accounts.type_name) in ('accounts receivable', 'bank', 'deferred expense', 'fixed asset', 'other asset', 'other current asset', 'unbilled receivable', 'prepaid expense') then 'Asset'
       when lower(accounts.type_name) in ('cost of goods sold', 'expense', 'other expense') then 'Expense'
       when lower(accounts.type_name) in ('income', 'other income') then 'Income'
       when lower(accounts.type_name) in ('accounts payable', 'credit card', 'deferred revenue', 'long term liability', 'other current liability') then 'Liability'
       when lower(accounts.type_name) in ('equity', 'retained earnings', 'net income') then 'Equity'
+      when lower(accounts.type_name) in ('non posting', 'statistical') then 'Other'
       else null 
         end as account_category
   from transactions_in_every_calculation_period_w_exchange_rates
 
-  join accounts 
+  left join accounts
     on accounts.account_id = transactions_in_every_calculation_period_w_exchange_rates.account_id 
 )