101_Vault Server Setup

Preface

About

This page provides instructions for configuring a High Availability Cluster for HashiCorp Vault. This entails setting up three Vault instances on three separate VPSs, each with renewable TLS provided by LetsEncrypt Certbot. Each step includes an identity that indicates which application should be used for the instructions that proceed. Examples are also provided for steps that have varying parameters depending on the context (server names, usernames, passwords, etc).

Note: Even though each new step indicates shelling into the server, it is not necessary often as you can continue on a previously open terminal that is already shelled into the server. The goal is to provide ease of understanding if a specific portion needs to be configured or changed in contrast from going through start to finish.

Example Values

The following values will be used for the examples:

Parameter	Value
User on VPS servers	luke
Active Vault Server	ovh12.jmaconsulting.biz
Standby Vault Server 1	b11.jmaconsulting.biz
Standby Vault Server 2	b12.jmaconsulting.biz

Prerequisites

• Three VPSs configured on three different bare metal servers

Main Documentation

1. Install Apache [source]

Identity: Local Machine, Terminal

# shell into Active Vault Server
$ ssh [user]@[host]

# ===== example =====
$ ssh [email protected]

Identity: Active Vault Server, Terminal

# install apache
$ sudo apt update
$ sudo apt upgrade -y
$ sudo apt install apache2 -y

# check apache status
$ sudo systemctl status apache2

# verify installation
$ apache2 -version

# configure firewall
$ sudo ufw app list
$ sudo ufw allow ‘Apache’

# check ufw status and enable if inactive
$ sudo ufw status
$ ​​sudo ufw enable

# IMPORTANT: YOU MUST ALLOW SSH OR YOU CANNOT SSH INTO THE SERVER
$ sudo ufw allow ssh

# allow https
$ sudo ufw allow https

# allow ports for vault
$ sudo ufw allow 8200
$ sudo ufw allow 8201

2. Setup TLS Using Certbot [source]

Identity: Local Machine, Terminal

# shell into Active Vault Server
$ ssh [user]@[host]

# ===== example =====
$ ssh [email protected]

Identity: Active Vault Server, Terminal

# install cerbot
$ sudo apt install snapd
$ sudo snap install core; sudo snap refresh core
$ sudo snap install --classic certbot
$ sudo ln -s /snap/bin/certbot /usr/bin/certbot

# get certificate for apache by following the prompts
$ sudo certbot --apache

# test auto renewal
$ sudo certbot renew --dry-run

3. Install Vault [source]

Identity: Local Machine, Terminal

# shell into Active Vault Server
$ ssh [user]@[host]

# ===== example =====
$ ssh [email protected]

Identity: Active Vault Server, Terminal

# install Vault
$ sudo apt update && sudo apt install gpg
$ wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor | sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg >/dev/null
$ gpg --no-default-keyring --keyring /usr/share/keyrings/hashicorp-archive-keyring.gpg --fingerprint
$ echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
$ sudo apt update && sudo apt install vault

# check that Vault has been installed properly
$ vault

4. Create Config File for Vault Server Instance

Identity: Local Machine, Terminal

# shell into Active Vault Server
$ ssh [user]@[host]

# ===== example =====
$ ssh [email protected]

Identity: Active Vault Server, Terminal

# make a data directory in the /var/lib directory
$ sudo mkdir /var/lib/vault
$ sudo mkdir /var/lib/vault/data

# change ownership of the folder
$ sudo chown -R vault:vault /var/lib/vault

# edit the configuration file
$ nano /etc/vault.d/vault.hcl

# add the following lines to the config file
# Note: node_ids need to be unique for each server ie. d1, d2, etc

Identity: config.hcl

listener "tcp" {
   address = "[IPv4_vault_server]:8200"
   tls_disable = "false"
   tls_cert_file="/etc/letsencrypt/live/[host].jmaconsulting.biz/fullchain.pem"
   tls_key_file="/etc/letsencrypt/live/[host].jmaconsulting.biz/privkey.pem"
}

storage "raft" {
  path    = "/var/lib/vault/data"
  node_id = "[host]"
}

api_addr = "https://[IPv4_vault_server]:8200"
cluster_addr = "https://[IPv4_vault_server]:8201"
ui = false

5. Configure SSL Reloading When Certificate and Key Renew [source]

Identity: Local Machine, Terminal

# shell into Active Vault server
# Note: you will need to repeat the following steps for all other Vault servers
$ ssh [user]@[host]

# ===== example =====
$ ssh [email protected]

Identity: Active Vault Server, Terminal

# create a bash script to copy over and reload tls cert and key whenever certbot renews
$ sudo nano /etc/letsencrypt/renewal-hooks/deploy/mysqld-deploy.sh
$ sudo chmod 700 /etc/letsencrypt/renewal-hooks/deploy/mysqld-deploy.sh

Identity: vault-deploy.sh

#!/bin/sh
kill -s 1 $(pidof vault)

6. Start the Server

Identity: Local Machine, Terminal

# shell into Active Vault Server
# Note: you will need to repeat these steps for the other vault servers
$ ssh [user]@[host]

# ===== example =====
$ ssh [email protected]

Identity: Active Vault Server, Terminal

# become root user to access TLS cert and key
$ sudo bash

# export the Vault address so that Vault knows where the Vault is
$ export VAULT_ADDR='https://[host].jmaconsulting.biz:8200'

# export the TLS cert so that Vault can use TLS
$ export VAULT_CACERT="/etc/letsencrypt/live/[host].jmaconsulting.biz/fullchain.pem"

# start the vault service
$ vault server -config=/etc/vault.d/vault.hcl &

# initialize the vault instance
$ vault operator init

# NOTE: Vault will output 5 keyshares and a root token. Store these in 1Password

# ===== example =====
$ sudo bash
$ export VAULT_ADDR='https://ovh12.jmaconsulting.biz:8200'
$ export VAULT_CACERT="/etc/letsencrypt/live/ovh12.jmaconsulting.biz/fullchain.pem"
$ vault operator init

Identity: Active Vault Server, Terminal

# unseal the Vault
$ vault operator unseal

# use the keyshares previously stored in 1Password to unseal the vault.
# Note: the command above will need to be used three times for three different keys.

# login as root with the initial root token provided when vault was initialized
$ vault login

7. Enable Secrets Engine and Configure Policy

Identity: Local Machine, Terminal

# shell into Active Vault Server
$ ssh [user]@[host]

# ===== example =====
$ ssh [email protected]

Identity: Active Vault Server, Terminal

# become root user to access TLS cert and key
$ sudo bash

# export the Vault address so that Vault knows where the Vault is
$ export VAULT_ADDR='https://[host].jmaconsulting.biz:8200'

# export the TLS cert so that Vault can use TLS
$ export VAULT_CACERT="/etc/letsencrypt/live/[host].jmaconsulting.biz/fullchain.pem"

# enable the secrets engine where the encryption key will be stored
$ vault secrets enable -version=2 kv

# create a policy that grants and restricts permissions
$ sudo nano /var/lib/vault/percona_policy.hcl

# ===== example =====
$ sudo bash
$ export VAULT_ADDR='https://ovh12.jmaconsulting.biz:8200'
$ export VAULT_CACERT="/etc/letsencrypt/live/ovh12.jmaconsulting.biz/fullchain.pem"
$ vault secrets enable -version=2 kv
$ sudo nano /var/lib/vault/percona_policy.hcl

Identity: policy.hcl

path "kv/*" {
    capabilities = ["read", "list"]
}

path "kv/data/dc1/*" {
    capabilities = ["create", "read", "delete", "update", "list"]
}

Identity: Active Vault Server, Terminal

# navigate to vault directory
$ cd /var/lib/vault
# write the policy
$ vault policy write percona_policy_1 percona_policy.hcl

8. Configuring Vault Audit Log [source]

Identity: Local Machine, Terminal

# shell into Primary Vault Server
$ ssh [user]@[host]

# ===== example =====
$ ssh [email protected]

Identity: Active Vault Server, Terminal

# set address and cert
$ sudo bash
$ export VAULT_ADDR='https://[host].jmaconsulting.biz:8200'
$ export VAULT_CACERT="/etc/letsencrypt/live/[host].jmaconsulting.biz/fullchain.pem"

# login with vault root token
$ vault login

# enable and set file location for audit log
$ vault audit enable file file_path=/var/lib/vault/vault-audit.log

# ===== example ===== 
$ sudo bash
$ export VAULT_ADDR='https://ovh12.jmaconsulting.biz:8200'
$ export VAULT_CACERT="/etc/letsencrypt/live/ovh12.jmaconsulting.biz/fullchain.pem"
$ vault login
$ vault audit enable file file_path=/var/lib/vault/vault-audit.log

9. Install Standby Vault Server

Repeat steps 1 to 5 on another vault server before continuing with the following instructions.

10. Start the Standby

Identity: Local Machine, Terminal

# shell into Standby Vault Server
$ ssh [user]@[host]

# ===== example =====
$ ssh [email protected]
$ ssh [email protected]

Identity: Standby Vault Server, Terminal

# become root user to access TLS cert and key
$ sudo bash

# export the Vault address so that Vault knows where the Vault is
$ export VAULT_ADDR='https://[host].jmaconsulting.biz:8200'

# export the TLS cert so that Vault can use TLS
$ export VAULT_CACERT="/etc/letsencrypt/live/[host].jmaconsulting.biz/fullchain.pem"

# start the vault server
$ vault server -config=/etc/vault.d/vault.hcl &

# join the vault cluster
$ vault operator raft join https://[active_vault_host].jmaconsulting.biz:8200

# ===== example =====
$ sudo bash
$ export VAULT_ADDR='https://b10.jmaconsulting.biz:8200'
$ export VAULT_CACERT="/etc/letsencrypt/live/b10.jmaconsulting.biz/fullchain.pem"
$ vault operator raft join https://ovh12.jmaconsulting.biz:8200

11. Unseal the Standby Vault Instance

Identity: Local Machine, Terminal

# shell into Standby Vault Server
# Note: you will have to repeate these steps again for the second standby Vault server
$ ssh [user]@[host]

# ===== example =====
$ ssh [email protected]
$ ssh [email protected]

Identity: Standby Vault Server, Terminal

# unseal the Vault

$ vault operator unseal

# use the keyshares previously stored in 1Password to unseal the vault.
# Note: these are the same keyshares used by the active vault server

Repeat steps 9 to 11 for the third and final vault server. By the end of this step you should have configured three vault servers, one active and two standby

12. Ensure that All Vault Servers are Connected to Cluster

Identity: Local Machine, Terminal

# shell into Primary Vault Server
$ ssh [user]@[host]

# ===== example =====
$ ssh [email protected]

Identity: Active Vault Server, Terminal

# set address and cert
$ sudo bash
$ export VAULT_ADDR='https://[host].jmaconsulting.biz:8200'
$ export VAULT_CACERT="/etc/letsencrypt/live/[host].jmaconsulting.biz/fullchain.pem"

# login with vault root token
$ vault login

# list all servers connected to the cluster
$ vault operator raft list-peers

# ===== example ===== 
$ sudo bash
$ export VAULT_ADDR='https://ovh12.jmaconsulting.biz:8200'
$ export VAULT_CACERT="/etc/letsencrypt/live/ovh12.jmaconsulting.biz/fullchain.pem"
$ vault login
$ vault operator raft list-peers