Troubleshoot_Common_Vault_Cluster_Issues.md

Troubleshoot Common Vault Cluster Issues

Search for underlying issues causing unhealthy Vault clusters. Check the Vault statefulset and various pod logs to determine what is impacting the health of the Vault.

Procedure

1. Verify the Vault statefulset.

   # kubectl -n vault get statefulset --show-labels
   NAME         READY   AGE   LABELS
   cray-vault   3/3     8d    app.kubernetes.io/name=vault,vault_cr=cray-vault

2. Check the pod logs for the bank-vaults container for Vault statefulset pods.

   # kubectl logs -n vault cray-vault-0 --tail=-1 --prefix -c bank-vaults
   # kubectl logs -n vault cray-vault-1 --tail=-1 --prefix -c bank-vaults
   # kubectl logs -n vault cray-vault-0 --tail=-1 --prefix -c bank-vaults

3. Check the Vault container logs within the pod.

   # kubectl logs -n vault cray-vault-0 --tail=-1 --prefix -c vault
   # kubectl logs -n vault cray-vault-1 --tail=-1 --prefix -c vault
   # kubectl logs -n vault cray-vault-2 --tail=-1 --prefix -c vault

4. Check the Vault operator pod logs using ephemerally named pods.

   # kubectl logs -n vault cray-vault-operator-7dbbdbb68b-zvg2g --tail=-1 --prefix