Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compose dev #3226

Open
wants to merge 13 commits into
base: master
Choose a base branch
from
Open

Compose dev #3226

wants to merge 13 commits into from

Conversation

jbaptperez
Copy link

@jbaptperez jbaptperez commented Nov 21, 2024

IMPORTANT: All Pull Requests should be connected to an issue, if you don't
have an issue, please start by creating an issue and link it to the PR.

Please provide enough information so that others can review your pull request:

  • What existing problem does this PR solve?
  • What new feature is being introduced with this PR?
    • Add to the development set-up the possibility to customise image builds (among other) using a .env file,
    • Update of the Compose file syntax to the Compose specification,
    • Other minor enhancements related to the development Compose set-up,
    • Use a Python virtual environment for Timesketch requirements.
  • Overview of changes to existing functions if required.
    • The current behaviour remain the same, but the enhancement of the development set-up leads to different paths to the Dockefiles and simplified commands to manage the development container.

Checks

  • All tests succeed.
  • Unit tests added.
  • e2e tests added.
  • Documentation updated.

Closing issues

Closes #3225.

Copy link

google-cla bot commented Nov 21, 2024

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@jbaptperez jbaptperez force-pushed the compose-dev branch 2 times, most recently from f8d06cb to fb811f1 Compare November 22, 2024 10:24
@jbaptperez jbaptperez marked this pull request as ready for review November 22, 2024 10:34
@jbaptperez jbaptperez force-pushed the compose-dev branch 12 times, most recently from 62afd82 to c310703 Compare November 28, 2024 21:57
@jbaptperez
Copy link
Author

As I had time to update the PR, I also optimized some things, like using a Python virtual environment in the Docker image.
There is a lot of another improvements to be done, but maybe in another PR?

I thought about:

  • Adding an environment variable to allow a custom place for the configuration files (with a default value to be compatible), this is necessary for the next steps,
  • Put every component into its own container,
  • Using Compose profiles to start everything or only a set of services,
  • Allow, with port mapping to localhost, using our IDE to start a process in true debug mode (gunicorn or flask, celery),
  • Use Compose development features for an optimized synchronization with the current source code.

@jbaptperez jbaptperez force-pushed the compose-dev branch 2 times, most recently from 212f1bb to 08e8de9 Compare December 3, 2024 17:08
@jkppr jkppr mentioned this pull request Dec 12, 2024
4 tasks
@jkppr
Copy link
Collaborator

jkppr commented Dec 12, 2024

Thanks for the suggested changes to the development setup @jbaptperez. I appreciate you taking the time to propose these updates to improve the developer experience, especially for those in restricted environments. The changes are quite extensive and touch many core parts of our development setup, so we'll need some time to thoroughly review and test them.

While flexibility is important, we aim to maintain a simple and easily reproducible development environment that works out-of-the-box on a typical Linux machine. We generally encourage developers to manage their own custom configurations for specific environments, such as those found in corporate settings. Therefore, we may incorporate some of your proposed changes while declining others to ensure we keep the core setup as straightforward as possible.

@jbaptperez
Copy link
Author

@jkppr, actually, the apparent change is not so big: Everything run into a single container with docker compose exec commands instead of docker exec ones.

I paid special attention to keeping default settings so that the behaviour without changes is the same as before.

A big change reported by Git is the deletion of the recently added yarn.lock file as it freezes a package source repository URL, which is indeed an issue for the PR.

The rest is close to the previous behaviour, with a single real difference: I duplicated the timesketch.conf file so that the development Compose set-up have its own; This simplifies the configuration from a single .env file but this is a kind of duplication, which can be discussed.

However, I understand your message and I am open to suggestions.

The yarn.lock file speed-ups an initial set-up but freezes the registry
URL, which can be different in the ~/.npmrc file when developing in a
company.
Markdown formatting, typos.
Makes the repository handle file line endings.
This helps to make it cross-platform, asserting some files are Unix-ended.
Uses the prefeed name of the Compose specification.
Changes:
- Removes the deprecated "version" field,
- Adds a toplevel "name" field (prefix of container names nad network),
- Adds a toplevel "network" field, with a common "timesketch-dev"
  network,
- Removes container names (depends on and toplevel name and service
  names),
- Do not bind to the 127.0.0.1 interface only (0.0.0.0),
- Removes useless "links" (common network),
- Refactors environment variables to don't use a YAML array,
- Removes "restart" fields to detect undesired crashes in development,
- Binds ports of other services to the host (opensearch, redis).
Adds .gitignore files.
Dramatically improves an image build in a development context.
Adapts the list to the latest version of the repository:
- Fixes moved ones,
- Deletes removed ones.
Changes are:
- Allows Docker image builds in a restricted company context (limited
  access to remote Ubuntu, Python or Node repositories) using variables,
- Centralizes variables in a .env file (not versioned),
- Adds a .env.template file as .env template with predefined variables,
- Use a distinct directory for every service dependencies,
- Use named volumes to avoid anonymous ones (PostgreSQL, OpenSearch,
  Redis and Prometheus),
- Use a per-service environment file,
- Simplifies how development configuration files are transferred to
  Timesketch,
- Simplifies manipulation of containers using Compose CLI instead of
  the Docker one,
- Simplify and optimizes the Timesketch entrypoint,
- Updates the Bash scripts to start frontend-ng,
- Updates related documentation.

Use named volumes in Compose development

PostgreSQL, Redis and Prometheus declare volumes in their Dockerfile.
This leads to anonymous volume creations if they are not declared in
Compose.
Avoids using the system environment.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Adapt the development Compose set-up to allow restricted company contexts
2 participants