Create CMC data Pipeline

[franTarkenton]

Currently there are a number of bat scripts on drain/sewer server that are scheduled thought windows server. These scripts get the inputs that are then run through the CMC forecast Excel data prep macros to create input data to be used by the CLEVER model.

This epic describes the tasks related to creating a data pipeline for all steps related to preparing the CMC forecast data for input into the Clever model.

The end result will be event triggered by creating a process that listens to the federal governments Advanced Message Queue. When data becomes available the pipeline will be triggered creating the input data for the Clever model.

Dependencies

Epic

• for CMC data / processing put in a check to see if the process has already run #6
• Evaluate implementing Sarencia / AMQP for datamart data aquisitions  nr-rfc-admin#21
• Implement Event Listener - Trigger #8
• Add new hourly timestamps to GRIB process / Download script #13
• Event Listener - Containerize / GHA to complete containerization #15
• Event Listener - create deployment nr-rfc-admin#28
• Event Listener - Trigger Github action #16
• Update sarracenia Docs #17
• feat: implement connection recovery #19
• Get idempotency key from message #20
• Automated Deployment - gha->helm #22
• Events collected, but not triggering next process #23
• Reduce memory footprint #29