Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(ci): backend docker #232

Open
wants to merge 53 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
97ab7ac
Revise backend build and compose
DerekRoberts Dec 11, 2024
285d645
Format workflow include.
fergmac Dec 11, 2024
78821cb
Update django whitenoise.
fergmac Dec 11, 2024
af0000d
Update vue frontend to include /gwells/ in URL. Remove docker compose…
fergmac Dec 12, 2024
8f5b940
Update docker compose.
fergmac Dec 12, 2024
84f6512
Build context
DerekRoberts Dec 12, 2024
132cc00
Build file
DerekRoberts Dec 12, 2024
b2c554d
Set Compose container names
DerekRoberts Dec 12, 2024
e698c6d
Remove generated staticfiles folder.
fergmac Dec 12, 2024
c389ccb
Update deployment yaml.
fergmac Dec 12, 2024
d7f186a
Update database deployment.
fergmac Dec 12, 2024
41f4d4a
Var consistency
DerekRoberts Dec 12, 2024
b8cc9f5
Var consistency
DerekRoberts Dec 12, 2024
524d846
Temporarily beef up deployment image
DerekRoberts Dec 12, 2024
01afa6c
Resolve merge conflicts.
fergmac Dec 13, 2024
b41d5cc
Update report default value.
fergmac Dec 13, 2024
5010604
Change image tag param to tag.
fergmac Dec 13, 2024
ec283cb
Remove deploy paramters not referenced in database deployment.
fergmac Dec 13, 2024
7a5bc1b
Update db container name.
fergmac Dec 13, 2024
90ec0ae
Update django secret object name from init deployment.
fergmac Dec 13, 2024
f3609e0
🙃
fergmac Dec 13, 2024
1a14627
Update secret name convention.
fergmac Dec 13, 2024
e6aaf58
Uncomment parameters.
fergmac Dec 13, 2024
90bf822
Try minimized python manage command in backend dockerfile.
fergmac Dec 13, 2024
4bb49e5
Uncomment params.
fergmac Dec 13, 2024
68bdd74
Revert backend dockerfile.
fergmac Dec 13, 2024
f87ad6a
Comment out dockerfile env vars.
fergmac Dec 13, 2024
690dbf1
Add gwells service host and port.
fergmac Dec 13, 2024
52044ac
Revert env var update.
fergmac Dec 13, 2024
215238b
Comment out route.
fergmac Dec 13, 2024
49d19cf
Update dockerfile env vars.
fergmac Dec 13, 2024
f188df0
Comment further env vars.
fergmac Dec 13, 2024
78c3b3f
Remove reference to CUSTOM_GDAL_GEOS.
fergmac Dec 13, 2024
f2dd483
Remove env vars from Dockerfile.
fergmac Dec 13, 2024
1f38f94
Add CUSTOM_GDAL_GEOS to backend deployment.
fergmac Dec 13, 2024
38314bb
Add init container.
fergmac Dec 18, 2024
db0dd6b
Update database service selector name.
fergmac Dec 18, 2024
922b840
Remove backend deployment volume.
fergmac Dec 18, 2024
617a626
Comment out init deployment database service name.
fergmac Dec 18, 2024
7e01f32
Update deployment.
fergmac Dec 18, 2024
ec8874c
Test database host update.
fergmac Dec 19, 2024
5e9b5a6
Update env var setting.
fergmac Dec 20, 2024
74776fe
Update network policy name.
fergmac Dec 20, 2024
e1dfa94
Add static files to Git ignore file.
fergmac Jan 2, 2025
c43e0d2
🙃
fergmac Jan 2, 2025
f4c84a9
🙃
fergmac Jan 2, 2025
9980eb0
Move git ignore file location.
fergmac Jan 2, 2025
bc2d96b
Try git ignore again.
fergmac Jan 2, 2025
c09d7bb
Remove backend static files.
fergmac Jan 2, 2025
fce5c04
Update backend resources.
fergmac Jan 2, 2025
f9c1840
Update Django database config.
fergmac Jan 2, 2025
0872016
Add readiness and liveness probe to backend and update ports.
fergmac Jan 2, 2025
3276ef4
Add psql client to backend docker image.
fergmac Jan 3, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
6 changes: 6 additions & 0 deletions .github/workflows/pr-open.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,17 @@ jobs:
strategy:
matrix:
package: [database, backend, frontend, minio]
include:
- package: backend
build_context: ./
build_file: ./backend/Dockerfile
timeout-minutes: 20
steps:
- uses: bcgov-nr/[email protected]
with:
keep_versions: 50
build_context: ${{ matrix.build_context }}
build_file: ${{ matrix.build_file }}
package: ${{ matrix.package }}
tag: ${{ github.event.number }}
tag_fallback: latest
Expand Down
65 changes: 25 additions & 40 deletions backend/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,3 +1,22 @@
FROM node:14.21.3 AS frontend

Check failure

Code scanning / Trivy

Image user should not be 'root' High

Artifact: backend/Dockerfile
Type: dockerfile
Vulnerability DS002
Severity: HIGH
Message: Specify at least 1 USER command in Dockerfile with non-root user as argument
Link: DS002
Dismissed Show dismissed Hide dismissed

ENV LOG_LEVEL="info"

RUN apt update && \
apt install -y git build-essential python3 python3-pip make g++ && \
apt autoremove -y && \
apt clean

WORKDIR /app

COPY ./frontend/package*.json ./

RUN npm install

COPY ./frontend/ /app

RUN npm run build

FROM python:3.7-slim

# Envars
Expand All @@ -7,24 +26,24 @@
ENV CUSTOM_GDAL_GEOS="False"
ENV DATABASE_NAME=gwells
ENV DATABASE_USER="gwells"
ENV DATABASE_PASSWORD="test1"

Check warning on line 29 in backend/Dockerfile

View workflow job for this annotation

GitHub Actions / Builds (backend)

Sensitive data should not be used in the ARG or ENV commands

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ENV "DATABASE_PASSWORD") More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/
ENV DATABASE_SERVICE_NAME=gwells
ENV DJANGO_ADMIN_URL=admin
ENV DJANGO_DEBUG="true"
ENV DJANGO_SECRET_KEY=secret

Check warning on line 33 in backend/Dockerfile

View workflow job for this annotation

GitHub Actions / Builds (backend)

Sensitive data should not be used in the ARG or ENV commands

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ENV "DJANGO_SECRET_KEY") More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/
ENV ENABLE_ADDITIONAL_DOCUMENTS="true"
ENV ENABLE_AQUIFERS_SEARCH="true"
ENV GWELLS_SERVICE_HOST="db"
ENV GWELLS_SERVICE_PORT="5432"
ENV MINIO_ACCESS_KEY=minio

Check warning on line 38 in backend/Dockerfile

View workflow job for this annotation

GitHub Actions / Builds (backend)

Sensitive data should not be used in the ARG or ENV commands

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ENV "MINIO_ACCESS_KEY") More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/
ENV MINIO_SECRET_KEY=minio1234

Check warning on line 39 in backend/Dockerfile

View workflow job for this annotation

GitHub Actions / Builds (backend)

Sensitive data should not be used in the ARG or ENV commands

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ENV "MINIO_SECRET_KEY") More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/
ENV PYTHONUNBUFFERED="1"
ENV SESSION_COOKIE_SECURE="False"
ENV SSO_AUDIENCE=gwells-4121
ENV SSO_CLIENT=gwells-4121
ENV SSO_TEST_AUDIENCE=gwells-api-tests-4820
ENV SSO_TEST_CLIENT=gwells-api-tests-4820
ENV SSO_AUTH_HOST=https://test.loginproxy.gov.bc.ca/auth

Check warning on line 46 in backend/Dockerfile

View workflow job for this annotation

GitHub Actions / Builds (backend)

Sensitive data should not be used in the ARG or ENV commands

SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ENV "SSO_AUTH_HOST") More info: https://docs.docker.com/go/dockerfile/rule/secrets-used-in-arg-or-env/
ENV SSO_IDP_HINT="undefined"
ENV SSO_PORT=0
ENV SSO_REALM=standard
Expand Down Expand Up @@ -60,9 +79,12 @@
RUN python3 -m pip install ptvsd
RUN python3 -m pip install 'setuptools<58.0'

COPY . /app
COPY ./backend /app
# COPY ./backend-command-script.sh /backend-command-script.sh
COPY ./requirements.txt /requirements.txt
COPY ./backend/requirements.txt /app/requirements.txt

COPY --from=frontend ./app/dist ./gwells/static
COPY --from=frontend ./app/dist/index.html ./gwells/templates/

# RUN chmod +x load_fixtures.sh works when i pull the dockerfile into backend but not when dockerfile is with other docker files
RUN chmod +x /app
Expand All @@ -71,48 +93,11 @@

RUN python3 -m pip install -r requirements.txt


# TODO: move to entrypoint in deployment template?
# chmod -R 777 /app && \
CMD sh -c "python3 manage.py migrate --noinput && \

Check warning on line 98 in backend/Dockerfile

View workflow job for this annotation

GitHub Actions / Builds (backend)

JSON arguments recommended for ENTRYPOINT/CMD to prevent unintended behavior related to OS signals

JSONArgsRecommended: JSON arguments recommended for CMD to prevent unintended behavior related to OS signals More info: https://docs.docker.com/go/dockerfile/rule/json-args-recommended/
./load_fixtures.sh all && \
python3 manage.py createinitialrevisions && \
# python3 manage.py collectstatic --noinput && \
python3 manage.py collectstatic --noinput && \
# python3 manage.py export --cleanup=1 --upload=1 && \
python3 manage.py runserver 0.0.0.0:8000"

# RUN mkdir -p /app/staticfiles/admin/css && \
# ln -s /app/staticfiles /tmp/staticfiles && \
# chmod -R 777 /app/staticfiles

# make script executable
# RUN chmod +x /backend/backend-command-script.sh

# production image step
# FROM python:3.7-slim

# # TODO: need to confirm version of libgdal-dev used with FROM python version
# RUN apt-get -y update && \
# apt-get -y install libgdal32 \
# gdal-bin

# ENV PATH="/usr/bin/python3:${PATH}"

# WORKDIR /app

# RUN python3 -m pip install 'setuptools<58.0'
# RUN python3 -m pip install --upgrade pip
# RUN python3 -m pip install ptvsd

# COPY --from=build /app /app
# COPY --from=build /backend-command-script.sh /backend-command-script.sh
# COPY --from=build /requirements.txt /requirements.txt

# RUN chmod +x /app

# RUN python3 -m pip install -r requirements.txt

# make script executable
# RUN chmod +x /backend/backend-command-script.sh

# CMD ["python3", "manage.py", "runserver", "0.0.0.0:8000"]
26 changes: 13 additions & 13 deletions backend/gwells/settings/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -100,13 +100,13 @@

# Application definition
INSTALLED_APPS = (
# 'whitenoise.runserver_nostatic',
'whitenoise.runserver_nostatic',
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
# 'django.contrib.staticfiles',
'django.contrib.staticfiles',
'debug_toolbar',
'django.contrib.postgres',
'rest_framework',
Expand All @@ -129,7 +129,7 @@
MIDDLEWARE = (
'corsheaders.middleware.CorsMiddleware',
'django.middleware.security.SecurityMiddleware',
# 'whitenoise.middleware.WhiteNoiseMiddleware',
'whitenoise.middleware.WhiteNoiseMiddleware',
'django.middleware.gzip.GZipMiddleware',
'debug_toolbar.middleware.DebugToolbarMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
Expand Down Expand Up @@ -198,17 +198,17 @@
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/1.8/howto/static-files/

# if APP_CONTEXT_ROOT:
# STATIC_URL = '/' + APP_CONTEXT_ROOT + '/'
# else:
# STATIC_URL = '/'
if APP_CONTEXT_ROOT:
STATIC_URL = '/' + APP_CONTEXT_ROOT + '/'
else:
STATIC_URL = '/'

# STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')

# STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'
# STATICFILES_DIR = (
# os.path.join(BASE_DIR, 'staticfiles')
# )
STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'
STATICFILES_DIR = (
os.path.join(BASE_DIR, 'staticfiles')
)

LOGGING = {
'version': 1,
Expand Down Expand Up @@ -344,5 +344,5 @@ def __getitem__(self, item):
if get_env_variable('DISABLE_MIGRATIONS', None, strict=False, warn=False) == 'DISABLE_MIGRATIONS':
MIGRATION_MODULES = DisableMigrations()

# WHITENOISE_INDEX_FILE = True
WHITENOISE_INDEX_FILE = True
APPEND_SLASH = True
6 changes: 3 additions & 3 deletions backend/gwells/urls.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
from django.shortcuts import redirect
from django.views.decorators.cache import never_cache

from gwells.views import SurveyListCreateView, SurveyUpdateDeleteView, HealthView, api
from gwells.views import SurveyListCreateView, SurveyUpdateDeleteView, HealthView, index, api
from gwells.views.bulk import BulkWellAquiferCorrelation, BulkVerticalAquiferExtents
from gwells.views.admin import *
from gwells.settings.base import get_env_variable
Expand Down Expand Up @@ -101,6 +101,6 @@ def api_path_prefix():
# Catch all other cases to api/ and 404 them
re_path(r'' + app_root_slash + api_path_prefix() + '/*', api.api_404, name='api-404'),

# # Catch all other cases and push it to the SPA
# re_path(r'' + app_root_slash + '*', index, name='spa'),
# Catch all other cases and push it to the SPA
re_path(r'' + app_root_slash + '*', index, name='spa'),
]
10 changes: 5 additions & 5 deletions backend/gwells/views/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -103,8 +103,8 @@ def get_queryset(self):
return super().get_queryset()


# @require_http_methods(["GET"])
# def index(request, **kwargs):
# # Serve up the index.html generated by vue as a template.
# # This is not ideal, consider switching serving up of Vue.js entirely to caddy.
# return render(request, 'index.html')
@require_http_methods(["GET"])
def index(request, **kwargs):
# Serve up the index.html generated by vue as a template.
# This is not ideal, consider switching serving up of Vue.js entirely to caddy.
return render(request, 'index.html')
2 changes: 1 addition & 1 deletion backend/requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ django-crispy-forms==2.0
django-formtools==2.4.1
django-filter==23.5
requests==2.27.1
# whitenoise==6.5.0
whitenoise==6.5.0
django-cors-headers==4.1.0
django-extensions==3.2.3
django-rest-multiple-models==2.1.3
Expand Down
Loading
Loading