cb_ceiling app, middleware implementation for DieOff plots

[gopa-noaa]

Evaluate to see if any significant performance improvements to be had using middl-tier approach, and if so implement SQL templates and MATScommon->matsMiddle classes for the same.

[gopa-noaa]

Working on this issue on MATS & MATScoomon branches:
cb_ceiling_middle_tier_DieOff

[gopa-noaa]

DieOff-DieOff plot is working and match original (non-MiddleTier) implementation (for 3 stations for 1 month)
Execution time improvement:
Original: 180 seconds
matsMiddle: 12 Seconds
Moving on to testing all-Colorado and all-stations for one month.

[gopa-noaa]

DieOff->DieOff all Colorado for 1 month in 36 seconds, I haven't run the original timing, but may be about an hour is my guess.

[gopa-noaa]

Need to refactor to avoid excessive memory usage. In progress ...

[gopa-noaa]

I20230620-16:13:33.658(0)? sizes (MB), obs:40.18738079071045,model:67.99200248718262,ctc:428.41769886016846
I20230620-16:13:35.982(0)? imfve:600/721 idx: 6 in 394656 ms.
I20230620-16:13:35.984(0)? {
I20230620-16:13:35.984(0)? rss: 3970609152,
I20230620-16:13:35.985(0)? heapTotal: 3497046016,
I20230620-16:13:35.985(0)? heapUsed: 3353643640,
I20230620-16:13:35.985(0)? external: 25295002,
I20230620-16:13:35.985(0)? arrayBuffers: 23071383
I20230620-16:13:35.985(0)? }
sizes (MB), obs:40.18738079071045,model:81.70757389068604,ctc:428.41769886016846
I20230620-16:13:44.957(0)? imfve:0/721 idx: 0 in 67649 ms.

The CTC JSON sub_data is consuming a lot of space,
"N0": 40345,
"N_times": 49,
"sub_data": [
"1668272400;0;0;0;1",
"1668272400;0;0;0;1",
....
Would like to brainstorm possibilities for reducing this, or feed this, as @randytpierce suggested, stream-like to the stats generator.

[gopa-noaa]

@mollybsmith-noaa Molly's idea worked like a charm, CTC JSON size for all stations is now just 0.42 MB instead of 500 MB!
DieOff for all Colorado stations for 1 month in 30 seconds
DieOff for all stations for 1 month in 8 minutes!

[gopa-noaa]

CTC JSON is looking like this now

{
"fcst_lead": 0,
"hit": 57030,
"miss": 31656,
"fa": 35514,
"cn": 1301786,
"N0": 1425986,
"N_times": 3,
"sub_data": [
"1668276000;41;25;31;2007",
"1668279600;29;18;37;2039",
"1668283200;18;10;35;2060",
"1668286800;16;15;30;2063",
"1668290400;10;22;23;2071",
"1668294000;13;18;22;2069",
"1668297600;6;10;29;2073",
"1668301200;9;10;24;2075",
"1668304800;16;12;18;2068",
"1668308400;19;11;24;2046",

[mollybsmith-noaa]

Excellent! We can do this in all of the station plot types, it should really save some memory!

…

On Jun 22, 2023, at 10:29 AM, Gopa ***@***.***> wrote: CTC JSON is looking like this now { "fcst_lead": 0, "hit": 57030, "miss": 31656, "fa": 35514, "cn": 1301786, "N0": 1425986, "N_times": 3, "sub_data": [ "1668276000;41;25;31;2007", "1668279600;29;18;37;2039", "1668283200;18;10;35;2060", "1668286800;16;15;30;2063", "1668290400;10;22;23;2071", "1668294000;13;18;22;2069", "1668297600;6;10;29;2073", "1668301200;9;10;24;2075", "1668304800;16;12;18;2068", "1668308400;19;11;24;2046", — Reply to this email directly, view it on GitHub <#966 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHBWGZLYIM45JGF45N3BXYLXMRXILANCNFSM6AAAAAAX5JVATE>. You are receiving this because you were mentioned.

[gopa-noaa]

Yes, will plan to do that, I will create an issue for Time Series since that is the only one so far that has middleware implemented.

[gopa-noaa]

DieOff => Dieoff change is now tested and checked-in. Planning to create a pull request tomorrow to merge to development.