Full LENZ ingest

[richardmatthewsdev]

As an API user I want access to all LENZ data so I can build out demo experiences on the NZL website

Acceptance criteria

• Ensure that security controls and steps are followed as per protocol document
• Confirm appearance of S3 bucket that holds copy of LENZ source data
• Consider the best way to store this source data for ongoing access
• Consider the best way to store PDFs for ongoing access
• NZL website demo confirms the fields needed for their search and access experiences
• Mapping rules from source fields to API schema is confirmed with the Tech Lead before processing
• All required data is available in the API

Notes

• Suggest that time is spent on confirming data requirements for the demo website to limit the need for frequent ingestion
• See the structure of stored data public data https://www.legislation.govt.nz/subscribe/
• Transfer protocol is documented at https://nzpco.sharepoint.com/:w:/s/WebDevDataAggregation/EWbhMvIvoV5JoEAtKCSMZXQB25BomLHMyWPo7H9Sgt7tgw?e=DxMzHw

Acts SoD 79.4GB 3,625,900 Files, 39,346 Folders
Bills SoD 4.51GB 239,702 Files, 7,145 Folders
Regulations SoD 24.4GB 803,340 Files, 54,437 Folders
SOPs SoD 876MB 47,652 Files, 4,934 Folders

Plus Agency, Deemed Regs, etc, which are much smaller.