Merge pull request #83 from fivetran/feature/standardized-line-item-m…

…odel

add line item standardized model

CHANGELOG.md

@@ -1,3 +1,13 @@
+# dbt_shopify v0.13.0
+[PR #83](https://github.com/fivetran/dbt_shopify/pull/83) includes the following changes:
+
+## Features
+- Addition of the `shopify__line_item_enhanced` model. This model constructs a comprehensive, denormalized analytical table that enables reporting on key revenue, customer, and product metrics from your billing platform. It’s designed to align with the schema of the `*__line_item_enhanced` model found in Shopify, Recharge, Stripe, Zuora, and Recurly, offering standardized reporting across various billing platforms. To see the kinds of insights this model can generate, explore example visualizations in the [Fivetran Billing Model Streamlit App](https://fivetran-billing-model.streamlit.app/). Visit the app for more details.
+  - This model is currently disabled by default. You may enable it by setting the `shopify__standardized_billing_model_enabled` as `true` in your `dbt_project.yml`.
+
+## Under the Hood:
+- Added consistency test within integration_tests for the `stripe__line_item_enhanced` model.
+
 # dbt_shopify v0.12.2
 
 [PR #84](https://github.com/fivetran/dbt_shopify/pull/84) includes the following changes:

README.md

@@ -38,6 +38,17 @@ The following table provides a detailed list of all models materialized within t
 | [shopify__daily_shop](https://fivetran.github.io/dbt_shopify/#!/model/model.shopify.shopify__daily_shop)     | Each record represents a day of activity for each of your shops, conveyed by a suite of daily metrics about customers, orders, abandoned checkouts, fulfillment events, and more.                        |
 | [shopify__discounts](https://fivetran.github.io/dbt_shopify/#!/model/model.shopify.shopify__discounts)     | Each record represents a unique discount, enriched with information about its associated `price_rule`and metrics regarding orders and abandoned checkouts.                        |
 | [shopify__inventory_levels](https://fivetran.github.io/dbt_shopify/#!/model/model.shopify.shopify__inventory_levels)     | Each record represents an inventory level (unique pairing of inventory items and locations), enriched with information about its products, orders, and fulfillments.                        |
+| [shopify__line_item_enhanced](https://fivetran.github.io/dbt_shopify/#!/model/model.shopify.shopify__line_item_enhanced)       | This model constructs a comprehensive, denormalized analytical table that enables reporting on key revenue, customer, and product metrics from your billing platform. It’s designed to align with the schema of the `*__line_item_enhanced` model found in Shopify, Recharge, Stripe, Zuora, and Recurly, offering standardized reporting across various billing platforms. To see the kinds of insights this model can generate, explore example visualizations in the [Fivetran Billing Model Streamlit App](https://fivetran-billing-model.streamlit.app/). Visit the app for more details.  |
+
+## Example Visualizations
+Curious what these models can do? Check out example visualizations from the [shopify__line_item_enhanced](https://fivetran.github.io/dbt_shopify/#!/model/model.shopify.shopify__line_item_enhanced) model in the [Fivetran Billing Model Streamlit App](https://fivetran-billing-model.streamlit.app/), and see how you can use these models in your own reporting. Below is a screenshot of an example report—explore the app for more.
+
+<p align="center">
+  <a href="https://fivetran-billing-model.streamlit.app/">
+    <img src="https://raw.githubusercontent.com/fivetran/dbt_shopify/main/images/streamlit_example.png" alt="Streamlit Billing Model App" width="75%">
+  </a>
+</p>
+
 <!--section-end-->
 
 # 🎯 How do I use the dbt package?
@@ -59,7 +70,7 @@ If you are **not** using the [Shopify Holistic reporting package](https://github
 ```yml
 packages:
   - package: fivetran/shopify
-    version: [">=0.12.0", "<0.13.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: [">=0.13.0", "<0.14.0"] # we recommend using ranges to capture non-breaking changes automatically
 ```
 
 Do **NOT** include the `shopify_source` package in this file. The transformation package itself has a dependency on it and will install the source package as well. 
@@ -124,6 +135,14 @@ vars:
 
 ## (Optional) Step 6: Additional configurations
 <details open><summary>Expand/Collapse details</summary>
+
+### Enabling Standardized Billing Model
+This package contains the `shopify__line_item_enhanced` model which constructs a comprehensive, denormalized analytical table that enables reporting on key revenue, subscription, customer, and product metrics from your billing platform. It’s designed to align with the schema of the `*__line_item_enhanced` model found in Recurly, Recharge, Stripe, Shopify, and Zuora, offering standardized reporting across various billing platforms. To see the kinds of insights this model can generate, explore example visualizations in the [Fivetran Billing Model Streamlit App](https://fivetran-billing-model.streamlit.app/). For the time being, this model is disabled by default. If you would like to enable this model you will need to adjust the `shopify__standardized_billing_model_enabled` variable to be `true` within your `dbt_project.yml`:
+
+```yml
+vars:
+  shopify__standardized_billing_model_enabled: true # false by default.
+```
 
 ### Passing Through Additional Fields
 This package includes all source columns defined in the macros folder. You can add more columns using our pass-through column variables. These variables allow for the pass-through fields to be aliased (`alias`) and casted (`transform_sql`) if desired, but not required. Datatype casting is configured via a sql snippet within the `transform_sql` key. You may add the desired sql while omitting the `as field_name` at the end and your custom pass-though fields will be casted accordingly. Use the below format for declaring the respective pass-through variables:

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'shopify'
-version: '0.12.2'
+version: '0.13.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 models:
@@ -22,7 +22,6 @@ vars:
     shopify_transaction: "{{ ref('stg_shopify__transaction') }}"
     shopify_refund: "{{ ref('stg_shopify__refund') }}"
     shopify_order_adjustment: "{{ ref('stg_shopify__order_adjustment') }}"
-
     shopify_abandoned_checkout: "{{ ref('stg_shopify__abandoned_checkout') }}"
     shopify_collection_product: "{{ ref('stg_shopify__collection_product') }}"
     shopify_collection: "{{ ref('stg_shopify__collection') }}"

integration_tests/ci/sample.profiles.yml

@@ -16,13 +16,13 @@ integration_tests:
       pass: "{{ env_var('CI_REDSHIFT_DBT_PASS') }}"
       dbname: "{{ env_var('CI_REDSHIFT_DBT_DBNAME') }}"
       port: 5439
-      schema: shopify_integration_tests_8
+      schema: shopify_integration_tests_10
       threads: 8
     bigquery:
       type: bigquery
       method: service-account-json
       project: 'dbt-package-testing'
-      schema: shopify_integration_tests_8
+      schema: shopify_integration_tests_10
       threads: 8
       keyfile_json: "{{ env_var('GCLOUD_SERVICE_KEY') | as_native }}"
     snowflake:
@@ -33,7 +33,7 @@ integration_tests:
       role: "{{ env_var('CI_SNOWFLAKE_DBT_ROLE') }}"
       database: "{{ env_var('CI_SNOWFLAKE_DBT_DATABASE') }}"
       warehouse: "{{ env_var('CI_SNOWFLAKE_DBT_WAREHOUSE') }}"
-      schema: shopify_integration_tests_8
+      schema: shopify_integration_tests_10
       threads: 8
     postgres:
       type: postgres
@@ -42,13 +42,13 @@ integration_tests:
       pass: "{{ env_var('CI_POSTGRES_DBT_PASS') }}"
       dbname: "{{ env_var('CI_POSTGRES_DBT_DBNAME') }}"
       port: 5432
-      schema: shopify_integration_tests_8
+      schema: shopify_integration_tests_10
       threads: 8
     databricks:
       catalog: "{{ env_var('CI_DATABRICKS_DBT_CATALOG') }}"
       host: "{{ env_var('CI_DATABRICKS_DBT_HOST') }}"
       http_path: "{{ env_var('CI_DATABRICKS_DBT_HTTP_PATH') }}"
-      schema: shopify_integration_tests_8
+      schema: shopify_integration_tests_10
       threads: 8
       token: "{{ env_var('CI_DATABRICKS_DBT_TOKEN') }}"
       type: databricks

integration_tests/dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'shopify_integration_tests'
-version: '0.12.2'
+version: '0.13.0'
 profile: 'integration_tests'
 config-version: 2
 
@@ -11,7 +11,7 @@ clean-targets:         # directories to be removed by `dbt clean`
 vars:
   # shopify_using_fulfillment_event: true # set to true when regenerating docs
   # shopify_using_all_metafields: true # set to true when regenerating docs
-  shopify_schema: shopify_integration_tests_8
+  shopify_schema: shopify_integration_tests_10
   shopify_source:
     shopify_customer_identifier: "shopify_customer_data"
     shopify_order_line_refund_identifier: "shopify_order_line_refund_data"
@@ -53,8 +53,7 @@ dispatch:
     search_order: ['spark_utils', 'dbt_utils']
 
 models:
-  shopify:
-    +schema: "{{ 'shopify_integrations_tests_sqlw' if target.name == 'databricks-sql' else 'shopify' }}"
+  +schema: "{{ 'shopify_integrations_tests_sqlw' if target.name == 'databricks-sql' else 'shopify' }}"
   # +schema: "shopify_{{ var('directed_schema','dev') }}"
 
 seeds:

integration_tests/tests/consistency/consistency_line_item_enhanced.sql

@@ -0,0 +1,31 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+with prod as (
+    select *
+    from {{ target.schema }}_shopify_prod.shopify__line_item_enhanced
+),
+
+dev as (
+    select *
+    from {{ target.schema }}_shopify_dev.shopify__line_item_enhanced
+), 
+
+final as (
+    -- test will fail if any rows from prod are not found in dev
+    (select * from prod
+    except distinct
+    select * from dev)
+
+    union all -- union since we only care if rows are produced
+
+    -- test will fail if any rows from dev are not found in prod
+    (select * from dev
+    except distinct
+    select * from prod)
+    )
+
+select *
+from final

integration_tests/tests/consistency/consistency_line_item_enhanced_count.sql

@@ -0,0 +1,21 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+-- this test is to make sure the rows counts are the same between versions
+with prod as (
+    select count(*) as prod_rows
+    from {{ target.schema }}_shopify_prod.shopify__line_item_enhanced
+),
+
+dev as (
+    select count(*) as dev_rows
+    from {{ target.schema }}_shopify_dev.shopify__line_item_enhanced
+)
+
+-- test will return values and fail if the row counts don't match
+select *
+from prod
+join dev
+    on prod.prod_rows != dev.dev_rows

integration_tests/tests/integrity/vertical_integrity_line_item_enhanced.sql

@@ -0,0 +1,36 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+-- this test is to make sure there is no fanout between the staging order_line_table and the line_item_enhanced model.
+with stg_order_line as (
+    select
+        1 as join_key,
+        count(*) as order_line_count,
+        count(distinct order_id) as order_count
+    from {{ ref('stg_shopify__order_line') }}
+),
+
+line_item_enhanced as (
+    select
+        1 as join_key,
+        count(*) as line_item_enhanced_count
+    from {{ ref('shopify__line_item_enhanced') }}
+),
+
+-- test will return values and fail if the row counts don't match
+
+final as (
+    select 
+        stg_order_line.join_key,
+        stg_order_line.order_line_count + stg_order_line.order_count as total_line_and_order_count,
+        line_item_enhanced.line_item_enhanced_count
+    from stg_order_line
+    join line_item_enhanced
+        on stg_order_line.join_key = line_item_enhanced.join_key
+) 
+
+select *
+from final
+where total_line_and_order_count != line_item_enhanced_count