Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implementation of batch processing #48

Open
wants to merge 22 commits into
base: main
Choose a base branch
from
Open

Implementation of batch processing #48

wants to merge 22 commits into from

Commits on Oct 2, 2022

  1. Initial commit of IMPLEMENTATION.md

    @tgvashworth
    tgvashworth committed Oct 2, 2022
    Configuration menu
    Copy the full SHA
    c568142 View commit details
    Browse the repository at this point in the history

  2. Initial commit of code structure

    @tgvashworth
    tgvashworth committed Oct 2, 2022
    Configuration menu
    Copy the full SHA
    7ee244a View commit details
    Browse the repository at this point in the history

  3. Image downloads working

    @tgvashworth
    tgvashworth committed Oct 2, 2022
    Configuration menu
    Copy the full SHA
    1ba385e View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2022

  1. Add imagemagick with build, run & test in container 

    This commit does quite a bit:

- Introduces Makefile as the commands are getting long
- Adds installation of ImageMagick to the Dockerfile
- Adds imagick bindings
- Instatiates the bindings in main.go to check they work
    @tgvashworth
    tgvashworth committed Oct 3, 2022
    Configuration menu
    Copy the full SHA
    aa06bd5 View commit details
    Browse the repository at this point in the history

  2. Add make develop for developing in Docker

    @tgvashworth
    tgvashworth committed Oct 3, 2022
    Configuration menu
    Copy the full SHA
    d6aaec5 View commit details
    Browse the repository at this point in the history

  3. Small bash history hack

    @tgvashworth
    tgvashworth committed Oct 3, 2022
    Configuration menu
    Copy the full SHA
    7f94598 View commit details
    Browse the repository at this point in the history

  4. Convert images to grayscale

    @tgvashworth
    tgvashworth committed Oct 3, 2022
    Configuration menu
    Copy the full SHA
    5a0d78f View commit details
    Browse the repository at this point in the history

  5. Add a note about developing locally

    @tgvashworth
    tgvashworth committed Oct 3, 2022
    Configuration menu
    Copy the full SHA
    5cdbd0e View commit details
    Browse the repository at this point in the history

  6. Initial commit of uploading to S3

    @tgvashworth
    tgvashworth committed Oct 3, 2022
    Configuration menu
    Copy the full SHA
    281ee4d View commit details
    Browse the repository at this point in the history

  7. Test and fix bad HTTP status codes

    @tgvashworth
    tgvashworth committed Oct 3, 2022
    Configuration menu
    Copy the full SHA
    d4d08b2 View commit details
    Browse the repository at this point in the history

  8. Small implementation note tweaks

    @tgvashworth
    tgvashworth committed Oct 3, 2022
    Configuration menu
    Copy the full SHA
    bc89b11 View commit details
    Browse the repository at this point in the history

  9. Write output to CSV

    @tgvashworth
    tgvashworth committed Oct 3, 2022
    Configuration menu
    Copy the full SHA
    4b2141c View commit details
    Browse the repository at this point in the history

  10. Handle partial failure

    @tgvashworth
    tgvashworth committed Oct 3, 2022
    Configuration menu
    Copy the full SHA
    137e059 View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2022

  1. Fix run target entrypoint

    @tgvashworth
    tgvashworth committed Oct 4, 2022
    Configuration menu
    Copy the full SHA
    37c05c3 View commit details
    Browse the repository at this point in the history

  2. Basic tests for loading rows

    @tgvashworth
    tgvashworth committed Oct 4, 2022
    Configuration menu
    Copy the full SHA
    71a8f79 View commit details
    Browse the repository at this point in the history

  3. Add outputs to scripts and mount output dir

    @tgvashworth
    tgvashworth committed Oct 4, 2022
    Configuration menu
    Copy the full SHA
    418b0bd View commit details
    Browse the repository at this point in the history

  4. Small fixes

    @tgvashworth
    tgvashworth committed Oct 4, 2022
    Configuration menu
    Copy the full SHA
    ee37aac View commit details
    Browse the repository at this point in the history

  5. URN -> ARN

    @tgvashworth
    tgvashworth committed Oct 4, 2022
    Configuration menu
    Copy the full SHA
    2bc88ed View commit details
    Browse the repository at this point in the history

  6. Use -out suffix on output files

    @tgvashworth
    tgvashworth committed Oct 4, 2022
    Configuration menu
    Copy the full SHA
    6e10f3e View commit details
    Browse the repository at this point in the history

  7. Drop the upload context

    @tgvashworth
    tgvashworth committed Oct 4, 2022
    Configuration menu
    Copy the full SHA
    72b2b7f View commit details
    Browse the repository at this point in the history

  8. Extract core processing loop to function 

    - The `defer` calls now clean up immediately
- New `Config` and `Row` structs clean up the flow
    @tgvashworth
    tgvashworth committed Oct 4, 2022
    Configuration menu
    Copy the full SHA
    022f41a View commit details
    Browse the repository at this point in the history

  9. Clean up dockerfile

    @tgvashworth
    tgvashworth committed Oct 4, 2022
    Configuration menu
    Copy the full SHA
    e205c2e View commit details
    Browse the repository at this point in the history