DevOps processes and Continuous Delivery - Moving Forward

[WadeBarnes]

This ticket is meant to be a place where we can start the discussion and design of our DevOps and Continuous Delivery processes moving forward. We have a significant amount of technical debit that has built up over the years. While our processes have served us well for a very long time, we're overdue for an update.

Requirements:

• Update CI/CD pipelines - Many of our projects are still relying on Jenkins pipelines. We want to migrate them all to newer platforms. Whether that is utilizing GitHub Actions, Tekton, ArgoCD, Helm, or the like is up for discussion. We're utilizing GitHub Actions and Helm in a couple projects, such as bcgov/traction, and bcgov/vc-authn-oidc, and have also started utilizing GitHub Actions to replace Jenkins in a couple other projects such as bcgov/von-bc-registries-audit, and bcgov/dts-endorser-service.
• Standardize on standard Kubernetes resources. Where possible, migrate to using generic Kubernetes resources and avoid using platform specific (OCP specific) resources. For example, migrate away from using DeploymentConfig triggers to monitor image stream tags for managing image deployments.
• Periodic image rebuilds. Rebuild and redeploy application pods at least once a month in order to pick updates and patches performed to the base image(s). This will require updates to the ACA-Py builds to perform the same periodic image rebuild, since the majority of our projects rely on the ACA-Py images.
• Devise a strategy to review and address ACS vulnerability reports on a regular basis. Reviews were started here; Review ACS Reports #16
• Document the plans and procedures in the bcgov/DITP-DevOps repo.

Considerations:

• Helm - Can be a fairly heavyweight at times. We may want to consider a hybrid approach combining Helm and our existing templates in the short to medium term.

Short to Mid Term action items:

• Address deployments impacted by platform service's new auto-scale-down jobs; Investigate the impact of the auto-scale-down jobs #157
• Use cron jobs to trigger existing CI/CD pipelines to address periodic image rebuilds in the short term.

Mid to Long Term action items:

• Items go here

Design Docs:

• ArgoCD Implementation

  • https://hackmd.io/@i5okie/r1i_7Pqha

• HashiCorp Vault Integration - Inventory of Credential Types in the DITP Environments

  • https://hackmd.io/tLjGdALtTNC--W3Z4ZdhGA

Related Tickets:

• Investigate the impact of the auto-scale-down jobs #157
• Review ACS Reports #16

[WadeBarnes]

cc @jleach, not assigning to you, but we're interested in your input.

[WadeBarnes]

ArgoCD Implementation - Design Doc

• https://hackmd.io/@i5okie/r1i_7Pqha

[WadeBarnes]

HashiCorp Vault Integration - Inventory of Credentials in the DITP Environment

• https://hackmd.io/tLjGdALtTNC--W3Z4ZdhGA

[WadeBarnes]

Finished, from my perspective, filling out the above document.

[WadeBarnes]

We had a productive meeting with reps from platform services and Red Hat yesterday to go over a few different approaches using Helm with ArgoCD. A couple of the approaches work around some restrictions imposed by the instance maintained by platform services.

Resources from the meeting:

• https://developers.redhat.com/articles/2023/05/25/3-patterns-deploying-helm-charts-argocd
• https://argo-cd.readthedocs.io/en/stable/user-guide/multiple_sources/
• https://github.com/gitops-examples/helm-patterns
• https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Security/
• https://developers.redhat.com/blog/2024/03/19/whats-new-red-hat-openshift-gitops-112#new_in_version_1_12

[i5okie]

Progress update:

• The ArgoCD Implementation doc has been updated with current implementation details.
• Grafana Agent has been deployed within bc0192, e79518, and ca7f8f namespaces with ArgoCD.
• traction-database -- dev, test, and prod crunchy clusters have been deployed with ArgoCD
• The traction and vc-authn-oidc Helm charts have been updated to be compatible with ArgoCD workflow.
• Validated Helm chart updates by deploying vc-authn-oidc to dev and test environments with ArgoCD.

Next steps:

• Deploy traction to dev. test, and prod via ArgoCD
• Update traction GitHub Actions workflows to trigger automatic deployment via ArgoCD to dev, and test
• Deploy vc-authn-oidc to prod via ArgoCD
• Update vc-authn-oidc GitHub Actions workflows to trigger automatic deployment via ArgoCD to dev, and test
• Document the workflow changes, and manual promotion of services to production

[i5okie]

Progress update:

• "Update CI/CD pipelines - Many of our projects are still relying on Jenkins pipelines. We want to migrate them all to newer platforms. Whether that is utilizing GitHub Actions, Tekton, ArgoCD, Helm, or the like is up for discussion...."

Pipelines for the traction and vc-authn-oidc have been updated and migrated to take advantage of GitOps automation with ArgoCD.
Deployment of dev environments will likely have to remain deployable directly from GitHub Actions workflows for both Traction and VC-AuthnN-OIDC projects.

• "Standardize on standard Kubernetes resources. Where possible, migrate to using generic Kubernetes resources and avoid using platform specific (OCP specific) resources.... "

Resources for both Traction and VC-AuthN-OIDC projects use platform-agnostic technologies such as Helm charts, and plain kubernetes manifests. These resources can be deployed by anyone, on almost any Kubernetes cluster.

---

Updated the implementation documentation https://hackmd.io/hirbZlbkSQmp-UK_D7FbrQ

[i5okie]

Standardize on standard Kubernetes resources. Where possible, migrate to using generic Kubernetes resources and avoid using platform specific (OCP specific) resources. For example, migrate away from using DeploymentConfig triggers to monitor image stream tags for managing image deployments.

Created issues for the following repositories to Migrate OCP templates to using generic Kubernetes resources
bcgov/DITP-DevOps#200
bcgov/openshift-aries-mediator-service#43
bcgov/dts-backup-configurations#22
bcgov/orgbook-configurations#144
bcgov/von-bc-registries-audit#42
bcgov/dts-endorser-service#50
bcgov/von-bc-registries-agent-configurations#81

[i5okie]

Update CI/CD pipelines - Many of our projects are still relying on Jenkins pipelines. We want to migrate them all to newer platforms. Whether that is utilizing GitHub Actions, Tekton, ArgoCD, Helm, or the like is up for discussion. We're utilizing GitHub Actions and Helm in a couple projects, such as bcgov/traction, and bcgov/vc-authn-oidc, and have also started utilizing GitHub Actions to replace Jenkins in a couple other projects such as bcgov/von-bc-registries-audit, and bcgov/dts-endorser-service.

Created tasks for migrating Jenkins pipelines to GitHub Actions workflows:
bcgov/openshift-aries-mediator-service#44
bcgov/orgbook-configurations#147
bcgov/von-bc-registries-agent-configurations#82
bcgov/dts-endorser-service#56