Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automate the NKH CSV export. #57

Open
charliehoffs opened this issue Apr 22, 2021 · 2 comments
Open

Automate the NKH CSV export. #57

charliehoffs opened this issue Apr 22, 2021 · 2 comments
Assignees
Labels
Airtable Working with Airtable low-priority To complete after top and medium priority tasks

Comments

@charliehoffs
Copy link
Collaborator

charliehoffs commented Apr 22, 2021

Motivation/Context: BAC is the most accurate, up-to-date map of school meal sites in the Bay Area! Recognizing the quality of our data, the national organization No Kid Hungry (NKH) has been sourcing the data for the Bay Area regions of their Free Meal Find MAp and Texting Hotline (https://www.nokidhungry.org/find-free-meals) for the past year, which is linked on school districts', local governments', and food assistance websites across the region.

9 volunteers manually update our data every two weeks by checking school district websites and social media pages. 2 of our 9 volunteers - our School Meals Team Leads - conduct monthly data quality audits on the entire >1000 school database.

Currently, to get this updated data to NKH, @mudkins downloads a CSV from Airtable, does the following procedure transforming the CSV's data structure to the NKH template (https://github.com/dcl-covid-19/mega-map/tree/master/Tutorial:%20Airtable%20to%20NKH), then we download the emerging Google Sheet and email it to NKH.

To further automate this process and allow NKH to download new data whenever they please from our updated database, and remove the necessity of our manually sending it, we want to create a "View" in Airtable, with the data pre-transformed to NKH's structure, from which they can download a CSV anytime.

Achieving this goal will allow us to scale this process to other partners, with whom we hope to build similar data-sending relationships, so local organizations can easily take in (per their own data structure) BAC's high-quality updated data.

Estimate of time: TBD

Definition of Done:

  • There should exist an "NKH School Meals Data View" in our Airtable, comprising all school meal sites in our database, with their data transformed into the NKH template.

Suggested next steps:

  • Using a duplicate of our real database (so as not to do data engineering tests in our real database!), figure out how to have the data of one View populate another view, but with its variables getting transformed between them
  • Implement this, transforming the information from "School Meals View CSV" to the data structure of the NKH template (https://docs.google.com/spreadsheets/d/1INYFlE7gIjQCXbouNK9STncdXLVgSDNcMj6mXjxrahY/edit?usp=sharing), and having that data live in a new "NKH School Meals Data View". Put another way, we would like the data of our School Meals View to automatically, whenever updated, go live in the NKH School Meals Data View, pre-transformed into NKH's structure.
@charliehoffs charliehoffs added low-priority To complete after top and medium priority tasks Airtable Working with Airtable labels Apr 22, 2021
@mudkins
Copy link
Collaborator

mudkins commented May 7, 2021

i'll do this!

@charliehoffs
Copy link
Collaborator Author

AWESOME! Thank you @mudkins !! If helpful, I can imagine that other team members might have some clues about how we could make this work too, so hopefully folks can be helpful on the Monday night call and in the #bac-data-engineering channel, as needed!!

@charliehoffs charliehoffs assigned mudkins and unassigned mudkins May 9, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Airtable Working with Airtable low-priority To complete after top and medium priority tasks
Projects
None yet
Development

No branches or pull requests

2 participants