High Availability with PostgreSQL

QHx supports running SPIRE Server in high-availability (HA) mode with multiple replicas sharing an external PostgreSQL datastore. This guide covers both in-cluster PostgreSQL operators and managed services such as AWS RDS.

Note

For the architectural rationale behind SPIRE HA in QHx, see doc/arch/HA.md.

Prerequisites

• QHx installed in your cluster via Helm (see Helm-Based Installation)
• A StorageClass that supports ReadWriteOnce for SPIRE Server PVCs
• kubectl configured for your cluster
• Customer credentials for oci.messier42.com

---

Step 1 — Deploy a PostgreSQL Operator

AWS RDS

Amazon RDS for PostgreSQL is a managed option that removes the operational burden of running PostgreSQL inside the cluster.

Note

QHx connects to RDS using a standard PostgreSQL DSN. Native AWS features such as IAM database authentication and AWS KMS key management are not yet supported — use a database password for now.

Create the RDS instance (or use an existing one) and note the endpoint, e.g. spire.cxxx.us-east-1.rds.amazonaws.com.

Create the spire database and user:

CREATE DATABASE spire;
CREATE USER spire WITH PASSWORD '<strong-password>';
GRANT ALL PRIVILEGES ON DATABASE spire TO spire;

Ensure the RDS security group allows inbound TCP on port 5432 from the CIDR or security group of your cluster nodes.

The DSN to use in Step 2:

postgresql://spire:<password>@spire.cxxx.us-east-1.rds.amazonaws.com:5432/spire?sslmode=require

RDS uses a certificate from Amazon’s CA. If your cluster doesn’t have that CA in its trust store, add sslrootcert=/path/to/aws-rds-ca.pem to the DSN, or use sslmode=require without certificate verification (acceptable for most in-cluster use cases):

postgresql://spire:<password>@spire.cxxx.us-east-1.rds.amazonaws.com:5432/spire?sslmode=require

You do not need to deploy a PostgreSQL operator — skip straight to Step 2 once the RDS instance is ready.

CloudNativePG (recommended)

CloudNativePG is a CNCF project and the recommended option for production clusters.

kubectl apply --server-side -f \
  https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.24/releases/cnpg-1.24.0.yaml

Wait for the operator to be ready:

kubectl rollout status deployment/cnpg-controller-manager -n cnpg-system

Create a PostgreSQL cluster with a dedicated spire database:

spire-pg-cluster.yaml

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: spire-pg
  namespace: qhx-system
spec:
  instances: 3
  storage:
    size: 5Gi
  bootstrap:
    initdb:
      database: spire
      owner: spire
      secret:
        name: spire-pg-credentials

Create the password secret first:

kubectl create secret generic spire-pg-credentials \
  --namespace qhx-system \
  --from-literal=username=spire \
  --from-literal=password='<strong-password>'

Then apply the cluster manifest:

kubectl apply -f spire-pg-cluster.yaml

Wait for all instances to be ready:

kubectl wait cluster/spire-pg -n qhx-system \
  --for=condition=Ready --timeout=5m

The CloudNativePG operator creates a read-write service named spire-pg-rw.<namespace>.svc.cluster.local that SPIRE Server connects to.

Bitnami PostgreSQL

The Bitnami PostgreSQL Helm chart provides a simpler single-primary setup suitable for smaller clusters.

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

helm install spire-pg bitnami/postgresql \
  --namespace qhx-system \
  --set auth.database=spire \
  --set auth.username=spire \
  --set auth.password='<strong-password>' \
  --set primary.persistence.size=5Gi

The service is available at spire-pg-postgresql.qhx-system.svc.cluster.local.

---

Step 2 — Store the DSN in a Secret

SPIRE Server connects to PostgreSQL using the pkiDatastoreConnectionString field. Because this value contains credentials it must be stored in the manager Secret, not the ConfigMap.

The Secret must use the key qhx-manager-secret.yaml and contain valid YAML:

qhx-manager-secret.yaml (local file — do not commit)

pkiDatastoreConnectionString: "postgresql://spire:<password>@spire-pg-rw.qhx-system.svc.cluster.local:5432/spire?sslmode=require"

Caution

Use sslmode=require in production. For in-cluster deployments without TLS termination on the PostgreSQL service you may use sslmode=disable, but this is not recommended for sensitive workloads.

Create or update the Secret in the cluster:

kubectl create secret generic qhx-manager-secret \
  --namespace qhx-system \
  --from-file=qhx-manager-secret.yaml=./qhx-manager-secret.yaml \
  --dry-run=client -o yaml | kubectl apply -f -

Note

If you are using CloudNativePG you can reference the password directly from the spire-pg-credentials secret using an envFrom or valueFrom mount. The example above uses a literal value for brevity.

---

Step 3 — Configure the Manager Secret Reference

If you did not set managerSecretName during the initial install, upgrade the release to add it now. Use the same version and credentials you used when installing QHx:

Upgrade QHx to enable the manager Secret

$ helm upgrade qhx oci://oci.messier42.com/qhx/charts/qhx-core \
    --version <INSTALLED-VERSION> \
    --namespace qhx-system \
    --reuse-values \
    --set managerSecretName=qhx-manager-secret \
    --set managerSecretNamespace=qhx-system \
    --wait

Note

If you already set managerSecretName during the initial install (see Helm-Based Installation) you can skip this step.

---

Step 4 — Enable HA in the ConfigMap

Edit the qhx-manager ConfigMap to enable multiple SPIRE Server replicas and configure storage for the per-replica PVCs:

kubectl edit configmap qhx-manager -n qhx-system

Add or update these fields:

pkiServerReplicas: "2"          # number of SPIRE Server replicas (≥ 2 for HA)
pkiStorageClassName: "standard" # StorageClass that supports ReadWriteOnce
pkiStorageSize: "1Gi"           # size of each replica's PVC

The manager watches the ConfigMap and picks up changes without a restart. It will trigger a rolling update of the SPIRE Server StatefulSet automatically.

Caution

pkiServerReplicas > 1 requires pkiDatastoreConnectionString to be set. If the Secret is missing or the DSN is empty the manager will return an error and the StatefulSet will not be updated.

---

Step 5 — Verify the HA Deployment

Check that all SPIRE Server pods are running:

kubectl get pods -n qhx-system -l app.kubernetes.io/component=pki-server

Expected output (2-replica example):

NAME                   READY   STATUS    RESTARTS   AGE
qhx-spire-server-0     1/1     Running   0          2m
qhx-spire-server-1     1/1     Running   0          90s

Verify that one PVC was created per replica:

kubectl get pvc -n qhx-system -l app.kubernetes.io/component=pki-server

Confirm that SPIRE Server logs show the PostgreSQL datastore plugin:

kubectl logs -n qhx-system qhx-spire-server-0 | grep -i datastore

You should see lines similar to:

INFO Datastore connected  subsystem_name=catalog  type=sql  driver=postgres

---

Behaviour Notes

Aspect	Detail
Pod placement	The manager configures a soft (PreferredDuringScheduling) anti-affinity rule so replicas prefer different nodes.
Startup order	ParallelPodManagement is enabled — all pods start simultaneously rather than one-by-one.
Leader election	SPIRE Server leader election is automatically enabled when pkiServerReplicas > 1.
PVC lifecycle	Each replica gets its own PVC via VolumeClaimTemplates. Scaling down does not delete PVCs automatically.
ConfigMap / Secret reload	Both are watched live; no manager restart is required when you change the DSN or replica count.
StatefulSet VolumeClaimTemplates	These are immutable in Kubernetes. If you need to change pkiStorageClassName or pkiStorageSize after the StatefulSet exists, the manager will delete and recreate it automatically.

---

Rollback to Single-Replica Mode

To disable HA and revert to the embedded SQLite datastore:

1. Set pkiServerReplicas: "1" in the ConfigMap.
2. Remove or clear pkiDatastoreConnectionString from the Secret.
3. The manager will update the StatefulSet; the unused PVCs can be deleted manually.

Caution

Switching back to SQLite discards any state held in PostgreSQL (registered entries, bundles, etc.). Ensure your workloads re-attest after the transition.