Recipe: File processing

[holyjak]

The basic idea is you can drop a list of files or a directory. It scans it recursively for every file then computes a SHA for each file, and sends through through a pipeline (see drop-processor). If the code decides the file “matches”, then it can upload it…but the upload is designed to use AWS s3 presigned URLs (to avoid file size limits) so there is a server-side resolver that you can ask (via an ident) if a specific SHA already exists. It responds with exists, along with a presigned URL if it doesn’t thus the upload can short-circuit if that SHA is already in the store.

Here are the basics for file processing:

https://gist.github.com/awkay/3cf6d550986d4e25feefdebb8ff00671

The server-side resolver looks basically like this:

(pcm/defresolver signed-url-resolver
  "Get a signed URL for download/upload of the given file SHA. Requires a logged in user."
  [env {:file-store/keys [sha] :as input}]
  {::pc/input  #{:file-store/sha}
   ::pc/output [:file-store/signed-url
                :file-store/over-quota?
                :file-store/exists?]
   :check      logged-in}
  (let [{:keys [filename]} (:query-params env)
        storage-in-use 0                                    ; TODO: get current user usage...
        over-quota?    (> storage-in-use LIMIT)]
    (if (and sha (> (count sha) 12))
      (let [exists? (and sha (s3/object-exists? sha))]
        (cond-> {:file-store/sha sha :file-store/exists? exists?}
          over-quota? (assoc :file-store/over-quota? true)
          (not over-quota?) (assoc :file-store/exists? exists?)
          (and (not over-quota?) (not exists?)) (assoc :file-store/signed-url
                                                  (.toString (s3/put-url sha)))))
      (log/error "Invalid or missing SHA in upload URL request"))))