You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
At the moment RedshiftExporter immediately uploads all data to Redshift (although I think it would work with any PostgreSQL-compatible database). However, Redshift prefers data to be uploaded in large chunks, and the current Scoville upload size is clearly much smaller than the smallest Redshift allocation chunk, leading to a "table bloat" of approximately 100x.
This bloat can be fixed by compacting the table (by selecting all the data into a new table and swapping the two), but that requires a regular maintenance job to be run, which complicates the system.
Another approach would be for Scoville itself to buffer around 100 readings in memory (120 would be two hours worth at the default 1 minute setting) and upload them in larger chunks. This would make Scoville results less timely, but since Scoville data is used entirely for batch reporting, this isn't very important.
The text was updated successfully, but these errors were encountered:
At the moment
RedshiftExporter
immediately uploads all data to Redshift (although I think it would work with any PostgreSQL-compatible database). However, Redshift prefers data to be uploaded in large chunks, and the current Scoville upload size is clearly much smaller than the smallest Redshift allocation chunk, leading to a "table bloat" of approximately 100x.This bloat can be fixed by compacting the table (by selecting all the data into a new table and swapping the two), but that requires a regular maintenance job to be run, which complicates the system.
Another approach would be for Scoville itself to buffer around 100 readings in memory (120 would be two hours worth at the default 1 minute setting) and upload them in larger chunks. This would make Scoville results less timely, but since Scoville data is used entirely for batch reporting, this isn't very important.
The text was updated successfully, but these errors were encountered: