Upgrade Policy

Overview

This page covers QHx’s version compatibility policy and safe upgrade practices for production clusters.

Compatibility Policy

QHx follows a “break on v1.0” policy:

Before v1.0 (current): any release may introduce breaking changes to the Helm chart schema, CRD fields, or component APIs. There are no backward-compatibility guarantees between minor versions.
From v1.0 onwards: breaking changes will only be introduced in a new major version. Minor and patch releases will be backward compatible.

Until v1.0, every upgrade should be treated as a potentially breaking change and tested before rolling out to production.

Supported Upgrade Path

Only single-version upgrades are tested and supported: upgrade from the immediately preceding release to the current one. Skipping versions is not tested and may fail.

Downgrading to a previous release is not supported. If a release causes issues, use the rollback procedure described below to restore the previous release.

Before You Upgrade

Check the release notes

Read the release notes for the target version before upgrading. Pay attention to:

CRD schema changes (new required fields, renamed fields, removed fields)
Helm values that have been renamed, moved, or removed
Changes to QHxPolicy or QHxClusterPolicy spec fields
Any migration steps that must be run manually

Back up your configuration

Before upgrading, save your current Helm values:

helm get values qhx-core -n qhx-system -o yaml > qhx-values-backup.yaml

This gives you the exact settings to reinstall if a rollback is needed.

Dry-run the upgrade

Use helm upgrade --dry-run to preview what would change without applying anything:

helm upgrade qhx-core ./qhx-core-chart \
  --namespace qhx-system \
  --values qhx-values.yaml \
  --dry-run

Review the diff carefully. Changes to SPIRE ConfigMap resources or PKI server StatefulSet specs will cause rolling restarts of identity-issuing components, which temporarily interrupts SVID issuance for new workloads.

Upgrade Steps

1. Upgrade CRDs first

Helm does not upgrade CRDs automatically when running helm upgrade. Apply the updated CRDs manually before upgrading the chart:

kubectl apply -f crds/

If a CRD has new required fields, existing custom resources will continue to work (Kubernetes does not re-validate existing objects on CRD upgrade), but new objects will require the new fields.

2. Apply the upgrade

helm upgrade qhx-core ./qhx-core-chart \
  --namespace qhx-system \
  --values qhx-values.yaml \
  --wait \
  --timeout 10m

--wait causes Helm to block until all Deployments, StatefulSets, and DaemonSets report ready. --timeout should be long enough to allow PKI server pods to restart and rejoin the trust bundle — 10 minutes is a safe default.

3. Verify the upgrade

After Helm reports success, confirm that all components are healthy:

# Check all QHx system pods are running
kubectl get pods -n qhx-system

# Confirm SPIRE StatefulSets are ready
kubectl get statefulset -n qhx-system -l qhx.dev/pki-instance-hash

# Confirm PKI agents are ready on all nodes
kubectl get daemonset -n qhx-system

# Check for errors in the manager
kubectl logs -n qhx-system deploy/manager --since=5m

If you use workload namespaces with QHxPolicy, verify that the namespace-to-SPIRE-instance mapping is still correct:

kubectl get configmap qhx-spire-ns-map -n qhx-system -o yaml

4. Smoke-test a workload

Deploy a test client-server pair using QHx proxies and confirm that mTLS connections succeed end-to-end. This verifies that SVID issuance, proxy configuration, and certificate chain validation are all working after the upgrade.

Staged Rollout

For large clusters or production environments, roll out upgrades incrementally:

Stage 1 — Non-production clusters: Apply the upgrade to dev and staging clusters first. Run your full integration test suite against the upgraded environment.
Stage 2 — Canary production namespace: Create a canary workload namespace with a QHxPolicy, deploy the proxy, and verify that SVIDs are issued and connections succeed.
Stage 3 — Full production rollout: Apply the upgrade to all remaining clusters.

Between stages, monitor:

SVID issuance latency (PKI agent metrics)
Connection error rates in proxy logs
Manager reconciliation errors (kubectl logs deploy/manager)

Rollback Procedure

Helm keeps a release history that supports one-step rollback:

# List available revisions
helm history qhx-core -n qhx-system

# Roll back to the previous revision
helm rollback qhx-core -n qhx-system --wait --timeout 10m

If Helm’s rollback fails or the release history is unavailable, reinstall from your saved values backup:

helm upgrade qhx-core ./qhx-core-chart-previous-version \
  --namespace qhx-system \
  --values qhx-values-backup.yaml \
  --wait \
  --timeout 10m

Note that rolling back a CRD schema change is not automatic. If the upgrade applied CRD changes, re-apply the previous CRD version manually:

kubectl apply -f crds-previous/

CRD Upgrade Considerations

QHxClusterPolicy and QHxPolicy are cluster-scoped and namespace-scoped CRDs respectively. When a CRD version bumps:

If a field is added as optional, existing resources continue to work unchanged.
If a field is added as required, existing resources are unaffected until they are next modified (at which point validation fires). Check the release notes for migration instructions.
If a field is renamed or removed, you must update all existing custom resources before or immediately after upgrading the CRD.

Always check kubectl get qhxclusterpolicy and kubectl get qhxpolicy -A for any resources that may be affected.