Compose dev

.dockerignore

@@ -1,3 +1,52 @@
+# Files to ignore by Docker when transfering context
+
 **/.git
 /docs/
 /notebooks/
+
+**/.DS_Store
+
+# Back-up files
+**/*~
+**/*.swp
+
+# Generic auto-generated build files
+**/*.pyc
+**/*.pyo
+**/.ipynb_checkpoints/
+
+# Specific auto-generated build files
+/.eggs
+/.tox
+**/__pycache__
+/api_client/python/build/
+/cli_client/python/build/
+/importer_client/python/build/
+**/dependencies/
+/*.egg-info
+
+# Ignore frontend build related files
+**/node_modules
+/timesketch/static/dist
+**/package-lock.json
+**/yarn.lock
+
+# Test files
+**/.coverage
+**/tests-coverage.txt
+
+# Exclude Vagrant runtime files
+/vagrant/.vagrant/
+/vagrant/*.log
+
+# Exclude .venv folder
+/.venv/
+
+# Exclude Visual Studio Code files
+/.vscode/*
+
+# Exclude JetBrains IDE files
+/.idea/
+
+# Exclude Compose environment file
+/docker/dev/.env

.gitattributes

@@ -0,0 +1,5 @@
+*                               text=auto
+docker/**                       text=auto  eol=lf
+requirements.txt                text=auto  eol=lf
+requirements-dev.txt            text=auto  eol=lf
+

.gitignore

@@ -11,9 +11,6 @@
 *.pyo
 **/.ipynb_checkpoints/
 
-# Ignore back-up files.
-*~
-
 # Specific auto-generated build files
 /.eggs
 /.tox
@@ -27,6 +24,8 @@ importer_client/python/build/
 # Ignore frontend build related files
 node_modules
 timesketch/static/dist
+package-lock.json
+yarn.lock
 
 # Frontend Log files
 npm-debug.log*
@@ -48,4 +47,12 @@ vagrant/*.log
 
 # Exclude .venv folder
 .venv/
+
+# Exclude Visual Studio Code files
 .vscode/*
+
+# Exclude JetBrains IDE files
+.idea/
+
+# Exclude Compose environment file
+docker/dev/.env

docker/dev/.env.template

@@ -0,0 +1,15 @@
+GIFT_PPA_TRACK="stable"
+GIFT_PPA_URL="https://ppa.launchpadcontent.net/gift/${GIFT_PPA_TRACK}/ubuntu"
+NODE_VERSION="18.x"
+NODE_PPA_URL="https://deb.nodesource.com/node_${NODE_VERSION}"
+NODE_NPMRC=""
+PYTHON_PIP_CONF=""
+
+TIMESKETCH_BASE_IMAGE="ubuntu:22.04"
+TIMESKETCH_CONF_DIR="/etc/timesketch"
+TIMESKETCH_SECRET_KEY="L4np0jV3yAdAFdbVzWRMaBqiFMV8FKYd+Je1WKE40o8="
+TIMESKETCH_USER="dev"
+TIMESKETCH_PASSWORD="dev"
+
+POSTGRES_USER="timesketch"
+POSTGRES_PASSWORD="password"

docker/dev/README.md

@@ -5,44 +5,60 @@ Make sure to follow the docker [post-install](https://docs.docker.com/engine/ins
 
 NOTE: It is not recommended to try to run on a system with less than 8 GB of RAM.
 
-### Start a developer version of docker containers in this directory
+### Prepare a .env file
+
+Compose requires a `.env` file with top level environment variables to be set.
+To create it, just copy the `.env.template` file as a base.
 
+```bash
+cp .env.template .env
 ```
+
+Note the `.env` is ignored by Git: you can safely write sensitive data in it.
+
+You can optionally edit the `.env` file.
+This is useful if you need to build images with some company restrictions (accessing
+remote Ubuntu, PyPI or Node repositories).
+
+### Start a developer version of docker containers in this directory
+
+```bash
 docker compose up -d
 ```
 
 The provided container definition runs Timesketch in development mode as a volume from your cloned repo. Any changes you make will appear in Timesketch automatically.
 
 If you see the following message you can continue
 
-```
+```text
 Timesketch development server is ready!
 ```
-### Find out container ID for the timesketch container
 
-```
-CONTAINER_ID="$(docker container list -f name=timesketch-dev -q)"
-```
-In the output look for CONTAINER ID for the timesketch container
+### Start a celery container shell
 
-To write the ID to a variable, use:
-```
-export CONTAINER_ID="$(docker container list -f name=timesketch-dev -q)"
-```
-and test with
-```
-echo $CONTAINER_ID
-```
+Start the container in foreground (add `-d` to run in background):
 
-### Start a celery container shell
-```
-docker exec -it $CONTAINER_ID celery -A timesketch.lib.tasks worker --loglevel info
+```bash
+docker compose exec timesketch \
+  celery \
+  -A timesketch.lib.tasks \
+  worker \
+  --loglevel info
 ```
 
 ### Start development webserver (and metrics server)
 
-```
-docker exec -it $CONTAINER_ID gunicorn --reload -b 0.0.0.0:5000 --log-file - --timeout 600 -c /usr/local/src/timesketch/data/gunicorn_config.py timesketch.wsgi:application
+Start the container in foreground (add `-d` to run in background):
+
+```bash
+docker compose exec timesketch \
+  gunicorn \
+  --reload \
+  -b 0.0.0.0:5000 \
+  --log-file - \
+  --timeout 600 \
+  -c /usr/local/src/timesketch/data/gunicorn_config.py \
+  timesketch.wsgi:application
 ```
 
 You now can access your development version at http://127.0.0.1:5000/
@@ -53,17 +69,30 @@ You can also access a metrics dashboard at http://127.0.0.1:3000/
 
 ### Non-interactive
 
-Running the following as a script after `docker compose up -d` will bring up the development environment in the background for you.
+A script applies the previous commands in background for you.
+
+```bash
+docker compose up -d
+./start-frontend-ng-no-dev.sh
 ```
-export CONTAINER_ID="$(docker container list -f name=timesketch-dev -q)"
-docker exec $CONTAINER_ID celery -A timesketch.lib.tasks worker --loglevel info
-docker exec $CONTAINER_ID gunicorn --reload -b 0.0.0.0:5000 --log-file - --timeout 120 timesketch.wsgi:application
+
+A second script starts an additional development server for the frontend
+(http://127.0.0.1:5001/).
+You need to wait a few seconds before accessing it.
+
+```bash
+docker compose up -d
+./start-frontend-ng-dev.sh
 ```
 
 ### Run tests
 
-```
-docker exec -w /usr/local/src/timesketch -it $CONTAINER_ID python3 run_tests.py --coverage
+```bash
+docker compose exec \
+    -w /usr/local/src/timesketch \
+    -it \
+    timesketch \
+    python3 run_tests.py --coverage
 ```
 
 That will run all tests in your docker container. It is recommended to run all tests at least before creating a pull request.
@@ -74,33 +103,34 @@ To access a Jupyter notebook that has access to the Timesketch development
 environment start a browser and visit http://localhost:8844/ . The password to
 gain access is "timesketch".
 
-By default the /tmp directory is mapped as the data directory to store all
+By default, the /tmp directory is mapped as the data directory to store all
 notebooks. To change that, modify the line:
 
-```
+```yaml
       - /tmp/:/usr/local/src/picadata/
 ```
 
-in the docker-compose.yml file to point to a directory of your choosing.
+in the _compose.yaml_ file to point to a directory of your choosing.
 In order for the jupyter notebook to be able to make use of that folder it has
 to have read and write permission for the user with the UID 1000.
 
-By default the latest checked in code of the timesketch API client and
+By default, the latest checked in code of the timesketch API client and
 timesketch import client are installed. In order to install a new version, if
 you are modifying the clients you'll need to make sure that the timesketch
-source code on your machine is readable by the user with the UID 1000 and gid
-1000. If that is done, then the code is mapped into the
-      /usr/local/src/timesketch folder on the docker container.
+source code on your machine is readable by the user with the UID 1000 and
+gid 1000.
+If that is done, then the code is mapped into the `/usr/local/src/timesketch`
+folder on the docker container.
 
 New versions of timesketch api client can then be installed using:
 
-```python
+```bash
 !pip install -e /usr/local/src/timesketch/api_client/python/
 ```
 
 And the importer client:
 
-```python
+```bash
 !pip install -e /usr/local/src/timesketch/importer_client/python
 ```
 
@@ -109,6 +139,6 @@ active.
 
 To update the docker image run:
 
-```shell
+```bash
 $ sudo docker image pull us-docker.pkg.dev/osdfir-registry/timesketch/notebook:latest
 ```