Code

Datalake V1#905

Code

Draft

ShadeWyrm wants to merge 35 commits intomaincds-snc/forms-terraform:mainfrom feat/datalake-v1cds-snc/forms-terraform:feat/datalake-v1Copy head branch name to clipboard

+1,398-1

Commits

Commits on Nov 26, 2024

• * Draft datalake

  ShadeWyrm

  committed
  1a077ef

  Copy full SHA for 1a077ef

• * Swap depcrated storage object type to new one.

  ShadeWyrm

  committed
  ad99fcf

  Copy full SHA for ad99fcf

• * Add glue service to localstack.

  ShadeWyrm

  committed
  3342118

  Copy full SHA for 3342118

Commits on Nov 27, 2024

• * add glue to missing local-provider.tf

  ShadeWyrm

  committed
  cb93b7a

  Copy full SHA for cb93b7a

• * update endpoint for localstack vs aws

  ShadeWyrm

  committed
  3f8cf66

  Copy full SHA for 3f8cf66

• * pass s3/vs/localstack to rds_etl script.

  ShadeWyrm

  committed
  de7d785

  Copy full SHA for de7d785

• * Sorta working ETL!~

  ShadeWyrm

  committed
  8e3a920

  Copy full SHA for 8e3a920

Commits on Nov 28, 2024

• * working post-crawler

  ShadeWyrm

  committed
  31e50d3

  Copy full SHA for 31e50d3

• * Force update if script changed.

  ShadeWyrm

  committed
  37d1fa4

  Copy full SHA for 37d1fa4

Commits on Dec 3, 2024

• * Add athena.

  ShadeWyrm

  committed
  f7faf1f

  Copy full SHA for f7faf1f

• Merge branch 'main' into feat/datalake-v1

  ShadeWyrm

  authored
  b01fc40

  Copy full SHA for b01fc40

• * Update v ref to checksum

  ShadeWyrm

  committed
  52d1d08

  Copy full SHA for 52d1d08

• Update ETL script -> Need to use Java timestamp format for dates.

  ShadeWyrm

  committed
  c492d8d

  Copy full SHA for c492d8d

• * add athena

  ShadeWyrm

  committed
  0175b99

  Copy full SHA for 0175b99

Commits on Dec 6, 2024

• * add timestamp for partitioning data.

  ShadeWyrm

  committed
  8050f9c

  Copy full SHA for 8050f9c

• Merge branch 'main' into feat/datalake-v1

  ShadeWyrm

  authored
  5d25c44

  Copy full SHA for 5d25c44

Commits on Dec 9, 2024

• * Base historical job.

  ShadeWyrm

  committed
  58f36d8

  Copy full SHA for 58f36d8

• * add historical crawler

  ShadeWyrm

  committed
  926ef6a

  Copy full SHA for 926ef6a

Commits on Dec 11, 2024

• Merge branch 'main' into feat/datalake-v1

  ShadeWyrm

  authored
  86fedf9

  Copy full SHA for 86fedf9

Commits on Dec 12, 2024

• * draft of prepared query for metrics

  ShadeWyrm

  committed
  a1363e9

  Copy full SHA for a1363e9

Commits on Dec 16, 2024

• * Update athena IAM

  ShadeWyrm

  committed
  3ee7022

  Copy full SHA for 3ee7022

Commits on Dec 19, 2024

• Merge branch 'main' into feat/datalake-v1

  ShadeWyrm

  authored
  d4e63d3

  Copy full SHA for d4e63d3

Commits on Dec 27, 2024

• * add test script, add glue job to encryption

  ShadeWyrm

  committed
  5f0695a

  Copy full SHA for 5f0695a

• * update log to store for a year.

  ShadeWyrm

  committed
  d36ece6

  Copy full SHA for d36ece6

• * add kms key

  ShadeWyrm

  committed
  446f0f1

  Copy full SHA for 446f0f1

Commits on Jan 6, 2025

• * Add file size to test gen script.

  ShadeWyrm

  committed
  75bd7d2

  Copy full SHA for 75bd7d2

• * fix derp

  ShadeWyrm

  committed
  99415b4

  Copy full SHA for 99415b4

Commits on Jan 7, 2025

• * Add jsonparse to etl.

  ShadeWyrm

  committed
  cf3bb34

  Copy full SHA for cf3bb34

• * Swap to typed etl json grabbing.

  ShadeWyrm

  committed
  3c2ce75

  Copy full SHA for 3c2ce75

Commits on Jan 9, 2025

• * add meta info on field counts

  ShadeWyrm

  committed
  ad2d62a

  Copy full SHA for ad2d62a

• * remove null count.

  ShadeWyrm

  committed
  3010e4f

  Copy full SHA for 3010e4f

• * add comments to explain etl job more.

  ShadeWyrm

  committed
  cf129db

  Copy full SHA for cf129db

• * add delivery option.

  ShadeWyrm

  committed
  d957edb

  Copy full SHA for d957edb

• * add api as deliveryoption.

  ShadeWyrm

  committed
  7c2955a

  Copy full SHA for 7c2955a

Commits on Jan 14, 2025

• * Add user_id to template

  ShadeWyrm

  committed
  0cd5d4d

  Copy full SHA for 0cd5d4d