Merge branch 'main' into gfranxman/345-ensure-latest-version-of-rsync

CHANGELOG.md

@@ -1,5 +1,21 @@
 # Changelog
 
+## [1.17.2](https://github.com/sixfeetup/scaf/compare/v1.17.1...v1.17.2) (2024-10-30)
+
+### Bug Fixes
+
+* Do not change the ownership of the bind mounted directory ([#414](https://github.com/sixfeetup/scaf/issues/414)) ([2b4469e](https://github.com/sixfeetup/scaf/commit/2b4469e669ffefdf0370d688c727db14ae7f995e)), closes [#413](https://github.com/sixfeetup/scaf/issues/413)
+
+### Documentation
+
+* change readme structure ([#369](https://github.com/sixfeetup/scaf/issues/369)) ([abd3b64](https://github.com/sixfeetup/scaf/commit/abd3b649852e7bccced5d5fbc38147e4a3f4280a)), closes [#365](https://github.com/sixfeetup/scaf/issues/365)
+
+## [1.17.1](https://github.com/sixfeetup/scaf/compare/v1.17.0...v1.17.1) (2024-10-11)
+
+### Bug Fixes
+
+* updating tests in install.sh to detect missing preferred bin folder. ([#404](https://github.com/sixfeetup/scaf/issues/404)) ([e694307](https://github.com/sixfeetup/scaf/commit/e694307e18357461e7bacffe157916573e8eac48)), closes [#403](https://github.com/sixfeetup/scaf/issues/403)
+
 ## [1.17.0](https://github.com/sixfeetup/scaf/compare/v1.16.1...v1.17.0) (2024-10-05)
 
 ### Features

Dockerfile

@@ -1,11 +1,12 @@
-FROM python:3.10-alpine
+FROM python:3.12-alpine
 
 RUN set -eux; \
   apk add git shadow su-exec; \
   PIP_ROOT_USER_ACTION=ignore pip install uv; \
   uv pip install --system black isort cookiecutter
 
 COPY ./entrypoint.sh /usr/local/bin/entrypoint.sh
+RUN chmod +x /usr/local/bin/entrypoint.sh
 
 RUN addgroup -S scaf && adduser -s /bin/ash -S scaf -G scaf
 WORKDIR /home/scaf/out

entrypoint.sh

@@ -72,8 +72,6 @@ else
     echo "scaf already has the UID $HOST_UID."
 fi
 
-chown scaf:scaf /home/scaf/out
-
 echo "Finalizing: Dropping privileges and executing command..."
 # Drop privileges and execute the next container command, or 'sh' if not specified
 if [ "$#" -gt 0 ]; then

install.sh

@@ -22,8 +22,8 @@ ensure_bin_folder() {
       echo "Or set PREFERRED_BIN_FOLDER to a writable directory."
       exit 1
   fi
-  # verify if PREFERRED_BIN_FOLDER is in PATH or exit with advice.
-  if ! [ -n "$PATH" ] && [ -n "$HOME" ] && echo "$PATH" | grep -q ":$HOME/.local/bin:"; then
+
+  if [ -z "$PATH" ] || [ -z "$HOME" ] || ! echo $PATH | grep -qE ":?$PREFERRED_BIN_FOLDER:?"; then
       echo "Your PATH is missing $PREFERRED_BIN_FOLDER."
       echo "Please add the following line to your shell profile file (e.g. ~/.bashrc, ~/.zshrc, etc.)"
       echo "export PATH=\$PATH:$PREFERRED_BIN_FOLDER"

package.json

@@ -1,6 +1,6 @@
 {
   "name": "scaf",
-  "version": "1.17.0",
+  "version": "1.17.2",
   "devDependencies": {
     "@semantic-release/changelog": "^6.0.3",
     "@semantic-release/git": "^10.0.1",

{{cookiecutter.project_slug}}/README.md

@@ -1,244 +1,27 @@
 # {{ cookiecutter.project_name }}
-## How to set up your local environment for development
 
-### Requirements
+## :information_source: About the {{ cookiecutter.project_name }} project
 
-To work with Kubernetes, you need to install some additional software packages.
-Depending on your operating system, the installation instructions may vary.
+* [:telescope: Project overview](/docs/project-overview.md)
+* [:house: Architecture](/docs/architecture.md)
+<!-- * [:hospital: ER Diagram](/docs/erdiagram.md) -->
 
-The documentation and scripts in the repo are written to work with `kubectl`, `kind` and `Tilt`.
+## :technologist: How to set up your local environment for development
 
-Consult the links below if you prefer to use Minikube or Docker Desktop instead:
-* [minikube](https://minikube.sigs.k8s.io/docs/start/).
-* [Docker](https://docs.docker.com/get-docker/).
+See [docs/development.md](/docs/development.md)
 
-### Setup your environment
+## :package: How to deploy
 
-1. Get the repository
+See [docs/deployment.md](/docs/deployment.md)
 
-       $ git clone {{ cookiecutter.repo_url }}
-       $ cd {{ cookiecutter.project_slug }}
+## :shushing_face: How to manage passwords and sensitive values
 
-2. Prepare the environment variables. Edit the `.envrc` file to work for your environment.
+See [docs/secrets.md](/docs/secrets.md)
 
-### Run the kubernetes cluster and the {{ cookiecutter.project_slug }} app to develop the code
+## :bug: How to debug the application
 
-First load the environment variables, then run:
+See [docs/debug.md](/docs/debug.md)
 
-      $ make setup
-      $ tilt up
+## :microscope: How to monitor the application
 
-:information_source: It may take a little bit of time for all the services to start up, and it's possible for
-the first run to fail because of timing conflicts. If you do see messages indicating there
-were errors during the first run, stop all the containers using Ctrl-C, and then try it again.
-
-You are now ready to edit the code.
-The app will be automatically reloaded when its files change.
-
-To delete resources created by Tilt once you are done working:
-
-       $ tilt down
-
-This will not delete persistent volumes created by Tilt, and you should be able to start Tilt again with your data intact.
-
-To remove the cluster entirely:
-
-       $ kind delete cluster --name {{ cookiecutter.project_slug }}
-
-To switch between different Scaf project contexts:
-
-      $ tilt down    # inside the codebase of the previous project
-      $ make setup   # inside the codebase of the project you want to work on
-      $ tilt up
-
-### Update dependencies
-
-To update the backend app dependencies, you must edit the `backend/requirements/*.in` files.
-Once you have made your changes, you need to regenerate the `backend/requirements/*.txt` files using:
-
-       $ make compile
-
-
-## Next steps
-
-Creating a superuser account in the backend is useful so you have access to
-Django Admin that will be accesible at [http://localhost:8009/admin](http://localhost:8009/admin)
-
-To create a superuser use the following commands:
-
-    $ make shell-backend
-    $ ./manage.py createsuperuser
-{% if cookiecutter.create_nextjs_frontend == 'y' %}
-This project has a NextJS frontend configured. You can access it at [http://localhost:3000/](http://localhost:3000/).
-{% endif %}
-## Infrastructure provisioning
-
-Terraform can be used to provision AWS resources for your project deployment.
-Read `terraform/README.md` for more information and steps for provisioning
-resources.
-
-## Application deployment
-
-Use ArgoCD and Kubernetes to automate the deployment of your application to
-your infrastructure. ArgoCD monitors changes within your repository, promptly
-applying the relevant Kubernetes manifests. Read `bootstrap-cluster/README.md`
-for more details.
-
-## How to manage passwords and sensitive values
-
-### Using SealedSecrets
-
-SealedSecrets can be used to encrypt passwords for the values to be safely checked in.
-Creating a new secret involves encrypting the secret using kubeseal. [Installing kubeseal](https://github.com/bitnami-labs/sealed-secrets#kubeseal).
-
-Configure kubernetes to your current project config and context, making sure you are in the correct prod/sandbox environment
-
-    $ export KUBECONFIG=~/.kube/config:~/.kube/{{ cookiecutter.project_slug }}.ec2.config
-    $ kubectl config use-context {{ cookiecutter.project_slug }}-ec2-cluster
-
-Add the secrets to your manifest using the secrets template file, and run kubeseal on the unencrypted values. The makefile target `sandbox-secrets` will replace the variables in `./k8s/templates/secrets.yaml.template` with the encoded variables from the environment, and copy the manifest with the encrypted values to `.k8s/overlays/sandbox/secrets.yaml`. The same can be done for the prod environment using the `prod-secrets` target
-
-    $ make sandbox-secrets
-
-    $ make prod-secrets
-
-The `k8s/*/secrets.yaml` file can now be safely checked in. The passwords will be unencrypted by SealedSecrets in the cluster.
-When a secret is added/removed update the `k8s/templates` files, update the environment variables in .envrc and rerun the make commands.
-
-The decrypted values can be retrieved running:
-
-    $ kubectl get secret secrets-config -n {{ cookiecutter.project_dash }} -o yaml > unsealed_secrets.yaml
-
-### Using .envrc file
-
-To ease managing your passwords and secrets you can store the values in 1Password.
-You will need to install and configure [1Password cli](https://developer.1password.com/docs/cli/get-started/)
-
-You can automatically source from the `.envrc` file using [direnv](https://direnv.net/docs/installation.html)
-
-You can also manually export the variables to your environment using `source .envrc`
-
-
-
-### Debugging
-
-The steps below describe how to set up interactive debugging with PyCharm.
-
-#### PyCharm Debugging Setup
-Update `k8s/base/app.configmap.yaml` with `data` field `DEBUGGER_IDE: "pycharm"`
-
-In PyCharm:
-
-1. Go to 'Run' in the toolbar
-2. Click on 'Edit Configurations'
-3. Click on '+' in the top left of the dialog
-4. Select 'Python Debug Server'
-5. Set the host to 0.0.0.0 and the port to 6400, and the name as you see fit.
-6. Click 'Ok'
-
-#### Debugging in development
-Before the code you want to debug, add the following lines:
-
-```python
-from {{ cookiecutter.project_slug }}.utils.debugger import connect_debugger
-connect_debugger()
-``` 
-You can then set break points in your IDE and call the code as usual to hit them.
-
-When the debugger is first connected, you will see a screen pop up about mapping - Click 'auto-detect' path mapping settings and choose the file that matches `backend/{{ cookiecutter.project_slug }}/utils/debugger.py` 
-
-#### Troubleshooting
-If the debugger is connected early on, such as in `manage.py`, standard django functionality such the admin interface may break. For that reason connecting in proximity to the code you want to test is recommended.
-
-## How to monitor the application
-
-{% if cookiecutter.use_sentry == 'y' %}
-### How to monitor errors
-
-Sentry can be used for error reporting at the application level. Sentry is included as a dependency in the project requirements, and the SENTRY_DSN configuration variable is included in the Django config map.
-Next, one needs to add the project to Sentry by following the steps below:
-
-Note: The values for both tokens can be empty if you don't wish to use Sentry.
-
-1. Create two projects in your organisation's Sentry instance, e.g. https://sixfeetup.sentry.io/projects/
-   1. One project for the backend 
-   2. One project for the frontend 
-2. Configure Slack notifications
-3. Add team members in Sentry 
-4. Update `k8s/base/app.configmap.yaml` `SENTRY_DSN_BACKEND`, `VITE_SENTRY_DSN_FRONTEND` with the DSNs appropriate for the relevant Sentry projects. 
-
-For more detailed steps view `./docs/sentry.md`
-
-{% endif %}
-### How to monitor logs and the deployed application
-
-Install Loki for log aggregation and the Kube Prometheus Stack with Grafana Dashboards for monitoring.
-
-#### Setup AWS credentials
-
-First export the credentials to your environment variables. Change the values accordingly:
-
-```
-export AWS_ACCESS_KEY_ID='ABC123456'
-export AWS_SECRET_ACCESS_KEY='ABC123456'
-```
-
-Then create a secret in the monitoring namespace:
-
-```
-kubectl create secret generic iam-loki-s3 --from-literal=AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID --from-literal=AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -n monitoring
-```
-
-#### Install monitoring
-
-Before installing the monitoring tools, you will need to export the GRAFANA_ADMIN_PASSWORD environment variable. This will be used to set the Grafana admin password. Change the value accordingly:
-
-```
-export GRAFANA_ADMIN_PASSWORD='ABC123456'
-```
-
-Now install the loki-stack and kube-prometheus-stack helm charts:
-
-```
-make monitoring-up
-```
-
-If you want to store the logs in an S3 bucket, you will need to include the yaml values `k8s/_monitoring/loki-stack-values.yaml` file for the `helm install loki` command in the Makefile:
-
-```
-helm install loki grafana/loki-stack --values k8s/_monitoring/loki-stack-values.yaml --namespace monitoring --create-namespace
-```
-
-#### Connect to Grafana dashboard
-
-You can connect to Grafana through local port forwarding using the steps below. Alternatively, you can set up ingress to point to Grafana.
-
-```
-make monitoring-port-forward
-```
-
-And open http://localhost:8080 on your browser
-
-Login with admin / prom-operator that are the default values. To see these values, run
-
-```
-make monitoring-login
-```
-
-Login to Grafana. Hit the `Explore` button and this gets you to the place with existing data sources. Select the newly added Loki data source.
-
-You are also able to change the password for the Grafana admin user. To do this, run the following command:
-
-```
-kubectl exec --namespace monitoring -c grafana -it $(kubectl get pods --namespace monitoring -l "app.kubernetes.io/name=grafana" -o jsonpath="{.items[0].metadata.name}") -- grafana-cli admin reset-admin-password newpassword
-```
-
-By default, you are on the code view, and you can hit the 'label browser' option on the left side and make a selection based on a number of items - eg select namespace and the namespace that interests you. Hit the `Live` mode on the right side of the screen to see logs in real time - a good check that things are setup as expected!
-
-#### Create a dashboard
-
-There is a predefined django logs table dashboard that can be created with the following command:
-
-```
-make monitoring-dashboard
-```
+See [docs/monitoring.md](/docs/monitoring.md)

{{cookiecutter.project_slug}}/docs/architecture.md

@@ -0,0 +1,3 @@
+# :house: Architecture
+
+TODO: Use lots of diagrams to document the architecture of the project.

{{cookiecutter.project_slug}}/docs/debug.md

@@ -0,0 +1,30 @@
+# :bug: How to debug the application
+
+The steps below describe how to set up interactive debugging with PyCharm.
+
+## PyCharm Debugging Setup
+Update `k8s/base/app.configmap.yaml` with `data` field `DEBUGGER_IDE: "pycharm"`
+
+In PyCharm:
+
+1. Go to 'Run' in the toolbar
+2. Click on 'Edit Configurations'
+3. Click on '+' in the top left of the dialog
+4. Select 'Python Debug Server'
+5. Set the host to 0.0.0.0 and the port to 6400, and the name as you see fit.
+6. Click 'Ok'
+
+## Debugging in development
+Before the code you want to debug, add the following lines:
+
+```python
+from {{ cookiecutter.project_slug }}.utils.debugger import connect_debugger
+connect_debugger()
+```
+
+You can then set break points in your IDE and call the code as usual to hit them.
+
+When the debugger is first connected, you will see a screen pop up about mapping - Click 'auto-detect' path mapping settings and choose the file that matches [backend/{{ cookiecutter.project_slug }}/utils/debugger.py](/backend/{{ cookiecutter.project_slug }}/utils/debugger.py)
+
+## Troubleshooting
+If the debugger is connected early on, such as in `manage.py`, standard django functionality such the admin interface may break. For that reason connecting in proximity to the code you want to test is recommended.

{{cookiecutter.project_slug}}/docs/deployment.md

@@ -0,0 +1,10 @@
+# :package: How to deploy
+
+## Infrastructure provisioning
+
+Terraform can be used to provision AWS resources for your project deployment.
+Read [terraform/README.md](/terraform/README.md) for more information and steps for provisioning resources.
+
+## Application deployment
+
+Use ArgoCD and Kubernetes to automate the deployment of your application to your infrastructure. ArgoCD monitors changes within your repository, promptly applying the relevant Kubernetes manifests. Read [bootstrap-cluster/README.md](/bootstrap-cluster/README.md) for more details.