Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ona custom changes #2

Open
wants to merge 65 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
9ac763c
Update /mvt query to work with configured TABLE_NAME and TABLE_COLUMN
FrankApiyo Nov 16, 2022
0017195
[WIP]Add authentication middleware
FrankApiyo Nov 17, 2022
ba5053d
Add id to submission properties in mvt tile
FrankApiyo Dec 5, 2022
094cbf7
Add submission json to mvt properties
FrankApiyo Dec 5, 2022
93fab2c
Add authentication middleware
FrankApiyo Dec 7, 2022
6585085
Ensure CORS is configured correctly using env and onadata intergratio…
FrankApiyo Dec 7, 2022
f266c0d
Add .env to .dockerignore
FrankApiyo Dec 7, 2022
3947bb2
Add a healthcheck endpoint
FrankApiyo Dec 7, 2022
d4cc051
[WIP]Add filter for field_name and field_value
FrankApiyo Dec 11, 2022
a2868e9
Add dockerhub build and push github action
FrankApiyo Dec 14, 2022
8ec0889
Add on push workflow trigger for ona-custom-changes
FrankApiyo Dec 14, 2022
9ac5039
Build docker image when a tag is pushed
FrankApiyo Dec 14, 2022
db113da
Add push: true to build and push github action
FrankApiyo Dec 14, 2022
0013eaf
Allow users who have no temp tokens to access public datasets
FrankApiyo Dec 19, 2022
8bd36bb
Add ternary operator to check if CORS_ORIGIN is string and remove con…
FrankApiyo May 11, 2023
91e4743
Update base image (node) version
FrankApiyo Dec 19, 2023
5bbb949
Merge pull request #3 from onaio/update-base-image-version
FrankApiyo Dec 19, 2023
0cac1e6
Merge branch 'master' into ona-custom-changes
KipSigei Mar 21, 2024
4e5644b
Use comma separated list of origins
KipSigei Mar 21, 2024
6d3247d
Update base image to use latest alpine
KipSigei Mar 21, 2024
e73472c
Merge pull request #4 from onaio/update-base-image
KipSigei Mar 21, 2024
d9529c0
Fix dependencies bug
KipSigei Mar 26, 2024
888ae3f
Merge pull request #5 from onaio/fixes
KipSigei Mar 26, 2024
17720f5
Fix dependencies bug
KipSigei Mar 26, 2024
5970e40
Merge pull request #6 from onaio/fixes
KipSigei Mar 26, 2024
8bd6b62
Cleanup
KipSigei Mar 26, 2024
c3d0955
Merge pull request #7 from onaio/fixes
KipSigei Mar 26, 2024
baa45a1
Fix imports
KipSigei Mar 26, 2024
c84c1b6
Use node 20.12.2 and update nodejs package-lock.json
ukanga Apr 23, 2024
7826310
Upgrade alpine packages in Dockerfile
ukanga Apr 24, 2024
c7a2696
Optimize mvt query
FrankApiyo May 4, 2024
f14a777
Fix where clause error
FrankApiyo May 4, 2024
72487d2
Merge pull request #9 from onaio/optimize-mvt-query
FrankApiyo May 6, 2024
d63a926
Merge pull request #8 from onaio/april-updates-2024
KipSigei May 7, 2024
ddfe73b
Optimive mvt query: Remove unneeded SRID and extra ST_Transform calls
FrankApiyo May 12, 2024
56319fb
Merge pull request #10 from onaio/optimize-mvt-query
FrankApiyo May 13, 2024
f257308
Use node:20.13.1-alpine3.19
ukanga May 27, 2024
abafc0e
Merge pull request #12 from onaio/may-updates-2024
KipSigei Jun 10, 2024
9f02a8f
Update base image to node:20.14.0-alpine3.19
KipSigei Jun 27, 2024
3c0eb21
Update base image to node:20.15.0-alpine3.19
KipSigei Jun 27, 2024
7ca685c
Merge pull request #13 from onaio/june-updates-2024
KipSigei Jun 27, 2024
c94086d
Update base image to node:20.15.1-alpine3.20
ciremusyoka Jul 31, 2024
283e96b
Merge pull request #14 from onaio/update-base-image
ciremusyoka Jul 31, 2024
62c8baf
Add endpoint to get map bounds and handle filters and errors
ciremusyoka Aug 9, 2024
ecd9ef9
Add limit subquery
ciremusyoka Aug 19, 2024
9ddcea8
Fix schema type
ciremusyoka Aug 19, 2024
1fe5086
Fix error when temp token not supplied and escape empty rows
ciremusyoka Aug 19, 2024
b7aac83
Merge pull request #15 from onaio/get-map-bounds
ciremusyoka Aug 28, 2024
e121b96
Update axios to v1.7.5
ciremusyoka Aug 28, 2024
4c6788f
Merge pull request #16 from onaio/update-axios
ciremusyoka Aug 28, 2024
967f090
Remove fastify express.
ciremusyoka Sep 24, 2024
e5e39f5
Use node v20.17.0
ciremusyoka Sep 24, 2024
e164c74
Merge pull request #17 from onaio/remove-express-upgrade-node
ciremusyoka Sep 25, 2024
0ba5cba
Update find-my-way 8.0.0 -> 8.2.2
FrankApiyo Sep 25, 2024
7a1021d
Merge pull request #18 from onaio/update-find-my-way
FrankApiyo Sep 25, 2024
0bf11b9
Update base image node:20.17 -> node:20.18
FrankApiyo Oct 29, 2024
08f0c50
Update the cookie and find-my-way npm packages
FrankApiyo Oct 29, 2024
2b5cb0c
Merge pull request #19 from onaio/oct-30-2024-sec-updates
FrankApiyo Oct 30, 2024
b9d74cd
Update package-spawn dependency
FrankApiyo Nov 26, 2024
6e1fda3
Update cross-spawn
FrankApiyo Nov 27, 2024
1cdfd30
Merge pull request #20 from onaio/update-cross-spawn
FrankApiyo Nov 27, 2024
d852acc
Update package-lock file
FrankApiyo Nov 27, 2024
d4aac19
Merge pull request #21 from onaio/update-package-lock-file
FrankApiyo Nov 27, 2024
fbe2029
Change CORS_ORIGIN -> CORS_ORIGINS
FrankApiyo Dec 11, 2024
e53568a
Merge pull request #22 from onaio/fix-cors-origins-bug
FrankApiyo Dec 13, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions .github/workflows/docker-image-build.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
name: build and push image to dockerhub
on:
push:
tags:
- '[0-9]+\.[0-9]+\.[0-9]+'
jobs:
main:
runs-on: ubuntu-22.04
steps:
- name: Get the version
id: get-version-release
run: echo "version=${GITHUB_REF_NAME#refs/heads}" >> $GITHUB_ENV

- name: Checkout to version
uses: actions/checkout@v3
with:
ref: ${{ env.version }}

- name: Login to DockerHub
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKER_HUB_USERNAME }}
password: ${{ secrets.DOCKER_HUB_ACCESS_TOKEN }}

- name: Build and push
id: docker-build
uses: docker/build-push-action@v3
with:
context: .
file: Dockerfile
platforms: linux/amd64
push: true
tags: |
onaio/dirt-tile-server:${{ env.version }}

- name: Image digest
run: echo ${{ steps.docker-build.outputs.digest }}
4 changes: 3 additions & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
# adapted from https://nodejs.org/en/docs/guides/nodejs-docker-webapp/
FROM node:20-slim
FROM node:20.18.0-alpine3.20
RUN apk update && apk upgrade
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
RUN npm install
EXPOSE 3000
CMD [ "npm", "run", "start" ]
190 changes: 114 additions & 76 deletions index.js
Original file line number Diff line number Diff line change
Expand Up @@ -11,98 +11,136 @@ if ("SERVER_LOGGER" in process.env) {
}
}

const fastify = require("fastify")({ logger: logger })
const axios = require("axios");

// EXIT IF POSTGRES_CONNECTION ENV VARIABLE NOT SET
if (!("POSTGRES_CONNECTION" in process.env)) {
throw new Error("Required ENV variable POSTGRES_CONNECTION is not set. Please see README.md for more information.");
}

// POSTGRES CONNECTION
const postgresConfig = { connectionString: process.env.POSTGRES_CONNECTION }
async function build() {
const fastify = require("fastify")({ logger: logger });
const queryString = require("query-string");
fastify.addHook('onRequest', async (req, reply) => {
reqParams = req.url.split("?")[1];
const parsedReqParams = queryString.parse(reqParams);
const formId = parsedReqParams.form_id;
const tempToken = parsedReqParams.temp_token;
if (formId) {
axios
.get(`${process.env.FORMS_ENDPOINT}${formId}.json`, {
headers: tempToken && tempToken.length > 0 ? {
Authorization: `TempToken ${tempToken}`,
} : {},
})
.then((res) => {
if (res && res.status === 200) {
return;
} else {
reply.code(403).send("Forbidden");
}
})
.catch((error) => {
req.log.error(error)
reply.code((error?.status || 500)).send(error.message);
});
} else if ("/health-check" == req.url) {
return;
} else {
reply.code(401).send("Authentication Failure");
}
});

if (process.env.SSL_ROOT_CERT) {
postgresConfig.ssl = {
ca: process.env.SSL_ROOT_CERT
// EXIT IF POSTGRES_CONNECTION ENV VARIABLE NOT SET
if (!("POSTGRES_CONNECTION" in process.env)) {
throw new Error("Required ENV variable POSTGRES_CONNECTION is not set. Please see README.md for more information.");
}
} else if (process.env.SSL_ROOT_CERT_PATH) {
postgresConfig.ssl = {
ca: fs.readFileSync(process.env.SSL_ROOT_CERT_PATH).toString()

// POSTGRES CONNECTION
const postgresConfig = { connectionString: process.env.POSTGRES_CONNECTION }

if (process.env.SSL_ROOT_CERT) {
postgresConfig.ssl = {
ca: process.env.SSL_ROOT_CERT
}
} else if (process.env.SSL_ROOT_CERT_PATH) {
postgresConfig.ssl = {
ca: fs.readFileSync(process.env.SSL_ROOT_CERT_PATH).toString()
}
}
}

fastify.register(require('@fastify/postgres'), postgresConfig)
fastify.register(require('@fastify/postgres'), postgresConfig)

// COMPRESSION
// add x-protobuf
fastify.register(
require('@fastify/compress'),
{ customTypes: /x-protobuf$/ }
)
// COMPRESSION
// add x-protobuf
fastify.register(
require('@fastify/compress'),
{ customTypes: /x-protobuf$/ }
)

// CACHE SETTINGS
fastify.register(
require('@fastify/caching'), {
// CACHE SETTINGS
fastify.register(
require('@fastify/caching'), {
privacy: process.env.CACHE_PRIVACY || 'private',
expiresIn: process.env.CACHE_EXPIRESIN || 3600,
serverExpiresIn: process.env.CACHE_SERVERCACHE
})

// CORS
fastify.register(require('@fastify/cors'), { origin: typeof process.env.CORS_ORIGINS === 'string' ? process.env.CORS_ORIGINS.split(",") : process.env.CORS_ORIGINS })

// OPTIONAL RATE LIMITER
if ("RATE_MAX" in process.env) {
fastify.register(import('@fastify/rate-limit'), {
max: process.env.RATE_MAX,
timeWindow: '1 minute'
})
}
)

// CORS
fastify.register(require('@fastify/cors'))
// INITIALIZE SWAGGER
fastify.register(require('@fastify/swagger'), {
exposeRoute: true,
hideUntagged: true,
swagger: {
"basePath": process.env.BASE_PATH || "/",
"info": {
"title": "Dirt-Simple PostGIS HTTP API",
"description": "The Dirt-Simple PostGIS HTTP API is an easy way to expose geospatial functionality to your applications. It takes simple requests over HTTP and returns JSON, JSONP, or protobuf (Mapbox Vector Tile) to the requester. Although the focus of the project has generally been on exposing PostGIS functionality to web apps, you can use the framework to make an API to any database.",
"version": process.env.npm_package_version || ""
},
"externalDocs": {
"url": "https://github.com/tobinbradley/dirt-simple-postgis-http-api",
"description": "Source code on Github"
},
"tags": [{
"name": "api",
"description": "code related end-points"
}, {
"name": "feature",
"description": "features in common formats for direct mapping."
}, {
"name": "meta",
"description": "meta information for tables and views."
}]
}
})

// OPTIONAL RATE LIMITER
if ("RATE_MAX" in process.env) {
fastify.register(import('@fastify/rate-limit'), {
max: process.env.RATE_MAX,
timeWindow: '1 minute'
// ADD ROUTES
fastify.register(require('@fastify/autoload'), {
dir: path.join(__dirname, 'routes')
})
}

// INITIALIZE SWAGGER
fastify.register(require('@fastify/swagger'), {
exposeRoute: true,
hideUntagged: true,
swagger: {
"basePath": process.env.BASE_PATH || "/",
"info": {
"title": "Dirt-Simple PostGIS HTTP API",
"description": "The Dirt-Simple PostGIS HTTP API is an easy way to expose geospatial functionality to your applications. It takes simple requests over HTTP and returns JSON, JSONP, or protobuf (Mapbox Vector Tile) to the requester. Although the focus of the project has generally been on exposing PostGIS functionality to web apps, you can use the framework to make an API to any database.",
"version": process.env.npm_package_version || ""
},
"externalDocs": {
"url": "https://github.com/tobinbradley/dirt-simple-postgis-http-api",
"description": "Source code on Github"
},
"tags": [{
"name": "api",
"description": "code related end-points"
}, {
"name": "feature",
"description": "features in common formats for direct mapping."
}, {
"name": "meta",
"description": "meta information for tables and views."
}]
}
})
fastify.get('/health-check', { logLevel: 'warn' }, (request, reply) => {
reply.send("healthy");
})

// SWAGGER UI
fastify.register(require("@fastify/swagger-ui"), {
routePrefix: "/"
})

// ADD ROUTES
fastify.register(require('@fastify/autoload'), {
dir: path.join(__dirname, 'routes')
})
return fastify
}

// LAUNCH SERVER
fastify.listen({port: process.env.SERVER_PORT || 3000, host: process.env.SERVER_HOST || '0.0.0.0'}, (err, address) => {
if (err) {
console.log(err)
process.exit(1)
}
console.info(`Server listening on ${address}`)
})
build()
.then(fastify => // LAUNCH SERVER
fastify.listen({ port: process.env.SERVER_PORT || 3000, host: process.env.SERVER_HOST || '0.0.0.0' }, (err, address) => {
if (err) {
fastify.log.error(err)
process.exit(1)
}
fastify.log.info(`Server listening on ${address}`)
}))
.catch(console.log)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we maybe add the error to logs

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If t this error occurs, most likely we didn't import fastify successfully and so the logger has not been initialized

Loading