Kamal deploy azure (#1117)

* Remove some extra routes leading to 500

* Kamal init

* Add production credentials -- still need real master key though

* Shift to use the Dockerfile suggested by Rails

* Allow secrets to be included

* Silence storage lookup and fix up DB creds

* Enable public_file_server to handle fetching assets

* WIP

* Bust

* Stop trying to use azure storage for now

* Skip over the need for Azure to be defined when doing precomppile

* Add basic comment on whyu

* Add missing docker-entrypoint

* Update secrets

* Remove extra key

* Fix invalid encryption for credentials

* Fix indentation

* Does not like this

* Add healthcheck place

Up

WIP

WIP

Try to make healthcheck work

WIP

Does this help

Use scss to ensure compiling happens

Up

Fix registry

WIP

Fx

Prolly cheating here

Build

Up

Silence

WIP

W

WI

WIP

P

Use propshaft because it is easier

WIP

WHat

Now it is working

WIP

eXclude one placements

Up

Clean up

* Remove code we do not need

* Fix up to use the new credentials space

* Update

* Adding notes

* Starting documentation

* Ok

* ok

* Ok'

* WIP

* test

* test deploy

* change domain

* change name

* fix

* ssh

* clean up

delete

* update dockerhub

* dockerhub

space

* standard

---------

Co-authored-by: Edwin Mak <[email protected]>

.gitignore

@@ -51,3 +51,5 @@
 
 # OS
 .DS_Store
+
+/config/credentials/production.key

.kamal/hooks/docker-setup.sample

@@ -0,0 +1,3 @@
+#!/bin/sh
+
+echo "Docker set up on $KAMAL_HOSTS..."

.kamal/hooks/post-deploy.sample

@@ -0,0 +1,14 @@
+#!/bin/sh
+
+# A sample post-deploy hook
+#
+# These environment variables are available:
+# KAMAL_RECORDED_AT
+# KAMAL_PERFORMER
+# KAMAL_VERSION
+# KAMAL_HOSTS
+# KAMAL_ROLE (if set)
+# KAMAL_DESTINATION (if set)
+# KAMAL_RUNTIME
+
+echo "$KAMAL_PERFORMER deployed $KAMAL_VERSION to $KAMAL_DESTINATION in $KAMAL_RUNTIME seconds"

.kamal/hooks/post-proxy-reboot.sample

@@ -0,0 +1,3 @@
+#!/bin/sh
+
+echo "Rebooted kamal-proxy on $KAMAL_HOSTS"

.kamal/hooks/pre-build.sample

@@ -0,0 +1,51 @@
+#!/bin/sh
+
+# A sample pre-build hook
+#
+# Checks:
+# 1. We have a clean checkout
+# 2. A remote is configured
+# 3. The branch has been pushed to the remote
+# 4. The version we are deploying matches the remote
+#
+# These environment variables are available:
+# KAMAL_RECORDED_AT
+# KAMAL_PERFORMER
+# KAMAL_VERSION
+# KAMAL_HOSTS
+# KAMAL_ROLE (if set)
+# KAMAL_DESTINATION (if set)
+
+if [ -n "$(git status --porcelain)" ]; then
+  echo "Git checkout is not clean, aborting..." >&2
+  git status --porcelain >&2
+  exit 1
+fi
+
+first_remote=$(git remote)
+
+if [ -z "$first_remote" ]; then
+  echo "No git remote set, aborting..." >&2
+  exit 1
+fi
+
+current_branch=$(git branch --show-current)
+
+if [ -z "$current_branch" ]; then
+  echo "Not on a git branch, aborting..." >&2
+  exit 1
+fi
+
+remote_head=$(git ls-remote $first_remote --tags $current_branch | cut -f1)
+
+if [ -z "$remote_head" ]; then
+  echo "Branch not pushed to remote, aborting..." >&2
+  exit 1
+fi
+
+if [ "$KAMAL_VERSION" != "$remote_head" ]; then
+  echo "Version ($KAMAL_VERSION) does not match remote HEAD ($remote_head), aborting..." >&2
+  exit 1
+fi
+
+exit 0

.kamal/hooks/pre-connect.sample

@@ -0,0 +1,47 @@
+#!/usr/bin/env ruby
+
+# A sample pre-connect check
+#
+# Warms DNS before connecting to hosts in parallel
+#
+# These environment variables are available:
+# KAMAL_RECORDED_AT
+# KAMAL_PERFORMER
+# KAMAL_VERSION
+# KAMAL_HOSTS
+# KAMAL_ROLE (if set)
+# KAMAL_DESTINATION (if set)
+# KAMAL_RUNTIME
+
+hosts = ENV["KAMAL_HOSTS"].split(",")
+results = nil
+max = 3
+
+elapsed = Benchmark.realtime do
+  results = hosts.map do |host|
+    Thread.new do
+      tries = 1
+
+      begin
+        Socket.getaddrinfo(host, 0, Socket::AF_UNSPEC, Socket::SOCK_STREAM, nil, Socket::AI_CANONNAME)
+      rescue SocketError
+        if tries < max
+          puts "Retrying DNS warmup: #{host}"
+          tries += 1
+          sleep rand
+          retry
+        else
+          puts "DNS warmup failed: #{host}"
+          host
+        end
+      end
+
+      tries
+    end
+  end.map(&:value)
+end
+
+retries = results.sum - hosts.size
+nopes = results.count { |r| r == max }
+
+puts "Prewarmed %d DNS lookups in %.2f sec: %d retries, %d failures" % [ hosts.size, elapsed, retries, nopes ]

.kamal/hooks/pre-deploy.sample

@@ -0,0 +1,109 @@
+#!/usr/bin/env ruby
+
+# A sample pre-deploy hook
+#
+# Checks the Github status of the build, waiting for a pending build to complete for up to 720 seconds.
+#
+# Fails unless the combined status is "success"
+#
+# These environment variables are available:
+# KAMAL_RECORDED_AT
+# KAMAL_PERFORMER
+# KAMAL_VERSION
+# KAMAL_HOSTS
+# KAMAL_COMMAND
+# KAMAL_SUBCOMMAND
+# KAMAL_ROLE (if set)
+# KAMAL_DESTINATION (if set)
+
+# Only check the build status for production deployments
+if ENV["KAMAL_COMMAND"] == "rollback" || ENV["KAMAL_DESTINATION"] != "production"
+  exit 0
+end
+
+require "bundler/inline"
+
+# true = install gems so this is fast on repeat invocations
+gemfile(true, quiet: true) do
+  source "https://rubygems.org"
+
+  gem "octokit"
+  gem "faraday-retry"
+end
+
+MAX_ATTEMPTS = 72
+ATTEMPTS_GAP = 10
+
+def exit_with_error(message)
+  $stderr.puts message
+  exit 1
+end
+
+class GithubStatusChecks
+  attr_reader :remote_url, :git_sha, :github_client, :combined_status
+
+  def initialize
+    @remote_url = `git config --get remote.origin.url`.strip.delete_prefix("https://github.com/")
+    @git_sha = `git rev-parse HEAD`.strip
+    @github_client = Octokit::Client.new(access_token: ENV["GITHUB_TOKEN"])
+    refresh!
+  end
+
+  def refresh!
+    @combined_status = github_client.combined_status(remote_url, git_sha)
+  end
+
+  def state
+    combined_status[:state]
+  end
+
+  def first_status_url
+    first_status = combined_status[:statuses].find { |status| status[:state] == state }
+    first_status && first_status[:target_url]
+  end
+
+  def complete_count
+    combined_status[:statuses].count { |status| status[:state] != "pending"}
+  end
+
+  def total_count
+    combined_status[:statuses].count
+  end
+
+  def current_status
+    if total_count > 0
+      "Completed #{complete_count}/#{total_count} checks, see #{first_status_url} ..."
+    else
+      "Build not started..."
+    end
+  end
+end
+
+
+$stdout.sync = true
+
+puts "Checking build status..."
+attempts = 0
+checks = GithubStatusChecks.new
+
+begin
+  loop do
+    case checks.state
+    when "success"
+      puts "Checks passed, see #{checks.first_status_url}"
+      exit 0
+    when "failure"
+      exit_with_error "Checks failed, see #{checks.first_status_url}"
+    when "pending"
+      attempts += 1
+    end
+
+    exit_with_error "Checks are still pending, gave up after #{MAX_ATTEMPTS * ATTEMPTS_GAP} seconds" if attempts == MAX_ATTEMPTS
+
+    puts checks.current_status
+    sleep(ATTEMPTS_GAP)
+    checks.refresh!
+  end
+rescue Octokit::NotFound
+  exit_with_error "Build status could not be found"
+end

.kamal/hooks/pre-proxy-reboot.sample

@@ -0,0 +1,3 @@
+#!/bin/sh
+
+echo "Rebooting kamal-proxy on $KAMAL_HOSTS..."

.kamal/secrets

@@ -0,0 +1,21 @@
+# Secrets defined here are available for reference under registry/password, env/secret, builder/secrets,
+# and accessories/*/env/secret in config/deploy.yml. All secrets should be pulled from either
+# password manager, ENV, or a file. DO NOT ENTER RAW CREDENTIALS HERE! This file needs to be safe for git.
+
+# Option 1: Read secrets from the environment
+#
+KAMAL_REGISTRY_PASSWORD=$(RAILS_ENV=production KEY=deploy,kamal_registry_password, bin/rails credentials:read)
+DB_HOST=$(RAILS_ENV=production KEY=deploy,db_host, bin/rails credentials:read)
+DB_USER=$(RAILS_ENV=production KEY=deploy,db_user, bin/rails credentials:read)
+DB_PASSWORD=$(RAILS_ENV=production KEY=deploy,db_password, bin/rails credentials:read)
+
+# Option 2: Read secrets via a command
+RAILS_MASTER_KEY=$(cat config/credentials/production.key)
+
+# Option 3: Read secrets via kamal secrets helpers
+# These will handle logging in and fetching the secrets in as few calls as possible
+# There are adapters for 1Password, LastPass + Bitwarden
+#
+# SECRETS=$(kamal secrets fetch --adapter 1password --account my-account --from MyVault/MyItem KAMAL_REGISTRY_PASSWORD RAILS_MASTER_KEY)
+# KAMAL_REGISTRY_PASSWORD=$(kamal secrets extract KAMAL_REGISTRY_PASSWORD $SECRETS)
+# RAILS_MASTER_KEY=$(kamal secrets extract RAILS_MASTER_KEY $SECRETS)

DEPLOYMENT.md

@@ -0,0 +1,55 @@
+# Deploying To Azure
+
+## Introduction
+
+This document aims to provide a step-by-step guide to deploying this
+application to Azure using Kamal 2.0. Since this application is using Rails 7.2 since
+the writing of this document, the codebase lacks some required configuration
+changes included in Rails 8 by default. Furthermore, Azure is a bit different
+and requires some considerations not explicitly mentioned in the documentation.
+
+This guide focuses just deploying the application to a single VM instance on Azure.
+It doesn't cover setting up the database on the same VM instance or anything
+else. For this guide we are going to setup a domain to point to a Azure
+VM running the application with SSL + a external database also hosted
+on Azure.
+
+## Prerequisites
+
+Here are some key things you need before starting this guide and why:
+
+- Dockerhub account: You need this to store the Docker image of the application between
+deployments
+- Azure Account: You need this to create the VM instance and the database.
+- `production.key`: Needed to decrypt the credentials in the production
+environment which contain the kamal secrets. Ask leads for this file
+if you don't have it.
+
+## Step 1: Initial Kamal Setup
+
+First, you need to setup Kamal on your local machine. You can follow the guide
+[here](https://kamal-deploy.org/docs/installation/) here. You should have the
+`kamal` command and all extra files needed for deployment. Notably, you'll be
+modifying just two files `config/deploy.yml` and `.kamal/secrets` in the project.
+
+### Step 1.1: Setup `config/deploy.yml`
+
+You will need to do a few modifications here to make sure the deployment works
+
+
+- Change the `service`, `app_name`, and `servers` to match the application's name
+```
+# Name of your application. Used to uniquely configure containers.
+service: homeward-tails # Name of the application.
+
+# Name of the container image.
+image: edwinthinks/homeward-tails
+
+# Deploy to these servers.
+servers:
+  web:
+    - tails.edwinmak.com
+```
+
+
+## Step 2: Setup Azure

Dockerfile

@@ -0,0 +1,74 @@
+# syntax = docker/dockerfile:1
+
+# This Dockerfile is designed for production, not development. Use with Kamal or build'n'run by hand:
+# docker build -t my-app .
+# docker run -d -p 80:80 -p 443:443 --name my-app -e RAILS_MASTER_KEY=<value from config/master.key> my-app
+
+# Make sure RUBY_VERSION matches the Ruby version in .ruby-version
+ARG RUBY_VERSION=3.3.0
+FROM docker.io/library/ruby:$RUBY_VERSION-slim AS base
+
+# Rails app lives here
+WORKDIR /rails
+
+RUN echo "Bust"
+
+# Install base packages
+RUN apt-get update -qq && \
+    apt-get install --no-install-recommends -y curl libjemalloc2 libvips postgresql-client && \
+    rm -rf /var/lib/apt/lists /var/cache/apt/archives
+
+# Install Node.js
+# [edwin] - this appears to be necessary to deploy our application even though we are using Dartsass
+RUN curl -fsSL https://deb.nodesource.com/setup_18.x | bash - && \
+    apt-get install -y nodejs && \
+    rm -rf /var/lib/apt/lists /var/cache/apt/archives
+
+# Set production environment
+ENV RAILS_ENV="production" \
+    BUNDLE_DEPLOYMENT="1" \
+    BUNDLE_PATH="/usr/local/bundle" \
+    BUNDLE_WITHOUT="development"
+
+# Throw-away build stage to reduce size of final image
+FROM base AS build
+
+# Install packages needed to build gems
+RUN apt-get update -qq && \
+    apt-get install --no-install-recommends -y build-essential git libpq-dev pkg-config && \
+    rm -rf /var/lib/apt/lists /var/cache/apt/archives
+
+# Install application gems
+COPY Gemfile Gemfile.lock ./
+RUN bundle install && \
+    rm -rf ~/.bundle/ "${BUNDLE_PATH}"/ruby/*/cache "${BUNDLE_PATH}"/ruby/*/bundler/gems/*/.git && \
+    bundle exec bootsnap precompile --gemfile
+
+# Copy application code
+COPY . .
+
+# Precompile bootsnap code for faster boot times
+RUN bundle exec bootsnap precompile app/ lib/
+
+# Precompiling assets for production without requiring secret RAILS_MASTER_KEY
+RUN SECRET_KEY_BASE_DUMMY=1 PRECOMPILING=1 ./bin/rails assets:precompile
+
+# Final stage for app image
+FROM base
+
+# Copy built artifacts: gems, application
+COPY --from=build "${BUNDLE_PATH}" "${BUNDLE_PATH}"
+COPY --from=build /rails /rails
+
+# Run and own only the runtime files as a non-root user for security
+RUN groupadd --system --gid 1000 rails && \
+    useradd rails --uid 1000 --gid 1000 --create-home --shell /bin/bash && \
+    chown -R rails:rails db log storage tmp
+USER 1000:1000
+
+# Entrypoint prepares the database.
+ENTRYPOINT ["/rails/bin/docker-entrypoint"]
+
+# Start the server by default, this can be overwritten at runtime
+EXPOSE 3000
+CMD ["./bin/rails", "server"]