Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix uploading datapackages from iTAK iOS client #90

Draft
wants to merge 9 commits into
base: main
Choose a base branch
from
68 changes: 68 additions & 0 deletions .github/workflows/build-docker.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
on:
pull_request:
push:
branches:
- 'main'
Comment on lines +2 to +5
Copy link

@deemoowoor deemoowoor May 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can change these to:

on:
  workflow_dispatch:

and this workflow will only be activated by manual triggering from the Actions menu

https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_dispatch


env:
PLATFORMS: linux/amd64, linux/arm64, linux/armv7

jobs:
build_images:
runs-on: ubuntu-latest
permissions:
contents: read
id-token: write

steps:
- name: Checkout
uses: actions/checkout@v2

- name: Cache Docker layers
uses: actions/cache@v2
with:
path: /tmp/.buildx-cache
key: ${{ runner.os }}-buildx-${{ inputs.image-name }}
restore-keys: |
${{ runner.os }}-buildx-

- name: Set up QEMU
# Needed as part of https://github.com/docker/buildx/issues/495#issuecomment-1043341496
id: qemu
uses: docker/setup-qemu-action@v1
with:
image: tonistiigi/binfmt:latest
platforms: all

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2

- name: Login to GitHub Container Registry
uses: docker/login-action@v2
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Docker meta for taky
id: taky-meta
uses: docker/metadata-action@v4
with:
images: ghcr.io/${{ github.repository_owner }}/taky

- name: Build and push
uses: docker/build-push-action@v3
with:
context: .
push: true
platforms: ${{ env.PLATFORMS }}
pull: true
build-args: VERSION=${{ github.ref_name }}
tags: ${{ steps.taky-meta.outputs.tags }}
cache-from: type=local,src=/tmp/.buildx-cache
cache-to: type=local,dest=/tmp/.buildx-cache-new

- name: Move cache
run: |
rm -rf /tmp/.buildx-cache
mv /tmp/.buildx-cache-new /tmp/.buildx-cache
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,4 @@ htmlcov
*.pem
*.p12
*.zip
.venv
31 changes: 31 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# First stage: builder
FROM python:3.11 as builder

ENV TAKY_VERSION=0.9
ENV PUBLIC_IP=192.168.0.60

WORKDIR /build

RUN git clone --depth 1 https://github.com/tkuester/taky.git -b ${TAKY_VERSION}

WORKDIR /build/taky

RUN python3 -m pip install --upgrade pip && \
python3 -m pip install -r requirements.txt && \
python3 setup.py install && \
find /usr/local -name '*.pyc' -delete && \
find /usr/local -name '__pycache__' -type d -exec rm -rf {} +

RUN takyctl setup --public-ip=${PUBLIC_IP} /etc/taky

# Second stage: runtime
FROM python:3.11-slim as runtime

WORKDIR /

RUN mkdir /var/taky

COPY --from=builder /usr/local /usr/local
COPY --from=builder /etc/taky /etc/taky

ENTRYPOINT [ "taky", "-c", "/etc/taky/taky.conf" ]
53 changes: 53 additions & 0 deletions taky/dps/views/datapackage.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
import os
import json
import hashlib
import magic
from datetime import datetime as dt

from flask import request, send_file
Expand Down Expand Up @@ -99,6 +101,57 @@ def datapackage_get():

return send_file(name, as_attachment=True, download_name=meta["Name"])

#Experimental reverse-engineered endpoint for iTAK Datapackage upload
@app.route("/Marti/sync/upload", methods=["POST"])
@requires_auth
def datapackage_upload_itak():
mime = magic.Magic(mime=True)
try:
name = request.args["name"]
uid = request.args["uid"]
creator_uid = request.args["CreatorUid"]
f_hash = hashlib.sha256(request.data)
keywords = request.args["keywords"]
except KeyError:
return "Invalid arguments", 400

filename = secure_filename(f"{creator_uid}_{name}")

meta = get_meta(f_name=filename)
if meta.get("Hash") != f_hash:
old_meta_hash_path = os.path.join(
app.config["UPLOAD_PATH"], "meta", f'{meta.get("Hash")}.json'
)
try:
os.unlink(old_meta_hash_path)
except: # pylint: disable=bare-except
pass

# Save the uploaded file
file_path = os.path.join(app.config["UPLOAD_PATH"], filename)
with open(file_path, "wb") as binary_file:
binary_file.write(request.data)

sub_user = request.headers.get("X-USER", "Anonymous")
meta = {
"UID": uid, # What the file will be saved as
"Name": name, # File name on the server
"Hash": f_hash.hexdigest(), # SHA-256, checked
"PrimaryKey": 1, # Not used, must be >= 0
"SubmissionDateTime": dt.utcnow().isoformat() + "Z",
"SubmissionUser": sub_user,
"CreatorUid": creator_uid,
"Keywords": f"{keywords}",
"MIMEType": f"{mime.from_buffer(request.data)}",
"Size": os.path.getsize(file_path), # Checked, do not fake
"Visibility": "public",
}

put_meta(meta)

# src/main/java/com/atakmap/android/missionpackage/http/MissionPackageDownloader.java:539
# This is needed for client-to-client data package transmission
return url_for(f_hash.hexdigest())

@app.route("/Marti/sync/missionupload", methods=["POST"])
@requires_auth
Expand Down