Can I use Quirrel to clean data and add to DB?

[spidertwin2]

Hello,

In my blitz application:
User uploads .csv data file --> upload complete --> user closes browser

When upload is complete, a background process should start:
--> document is parsed
--> .csv data is preprocessed
--> cleaned data is added to postgres

In addition, when the user visits the web application again after some time, the most updated status of the background process should be displayed i.e. how much data is parsed, what data has been successfully uploaded, what tasks are pending, etc. The background process should only stop after all required data is pushed to postgres.

Can Quirrel help me create a background process for these tasks?

Thanks 🐼

[Skn0tt]

Hi @spidertwin2! That's a good question :) I think it depends on the nature of these .csv files. If they're not huge, meaning they can be parsed, preprocessed, and pushed into postgres in a couple of seconds, I don't immediately see the need for a background process / tasks / ...

If the files are big enough that that's not possible, then Quirrel could be a good part of the solution. Here's how it could work:

1. User uploads CSV.
2. A "task ID" is generated.
3. CSV is stored into some database, accessible under the task ID
4. A job with that task ID is enqueued into Quirrel.

You could then obtain a list of "what tasks are pending" using Quirrel.get().

One thing you'll need to think about is your execution environment. Are you running on a Serverless deployment provider? They mostly have a cap of 15secs or 60secs execution time. If you're running on something like Heroku or Render, chances are it'll be a little faster.

[spidertwin2]

This is really helpful! Thank you 🐨