Federation Setup

Overview

Federation enables workloads in different QHx clusters to authenticate each other via SPIFFE federation. This allows secure communication across:

Multiple Kubernetes clusters
Different geographic regions
Separate trust domains
Cloud and on-premises environments

Use cases:

Multi-region deployments (US East + US West)
Cross-border operations (US + UK clusters)
Hybrid cloud (AWS + on-premises)
Disaster recovery (primary + DR site)

Time to complete: 1-2 hours
Prerequisites: Two or more QHx clusters deployed

Federation Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Cluster: US-EAST                          │
│  Trust Domain: east.qhx.dev                                 │
│                                                              │
│  ┌──────────────┐          ┌──────────────┐               │
│  │ PKI Server   │◄────────►│ LoadBalancer │               │
│  │ (Bundle EP)  │          │ (NLB/Public) │               │
│  └──────────────┘          └──────┬───────┘               │
│                                    │                        │
└────────────────────────────────────┼────────────────────────┘
                                     │
                                     │ HTTPS
                                     │ Port 8443
                                     │
┌────────────────────────────────────┼────────────────────────┐
│                                    │                        │
│  ┌──────────────┐          ┌──────▼───────┐               │
│  │ PKI Server   │◄────────►│ LoadBalancer │               │
│  │ (Bundle EP)  │          │ (NLB/Public) │               │
│  └──────────────┘          └──────────────┘               │
│                                                              │
│  Trust Domain: west.qhx.dev                                │
│                    Cluster: US-WEST                          │
└─────────────────────────────────────────────────────────────┘

Key components:

Bundle Endpoint - HTTPS endpoint exposing trust bundle
LoadBalancer - Exposes bundle endpoint externally
Trust Bundle Exchange - Automatic synchronization of CA certificates

Concepts

Trust Domain

A SPIFFE trust domain represents a boundary of trust. Each QHx cluster has its own trust domain:

Cluster: US-EAST  → Trust Domain: east.qhx.dev
Cluster: US-WEST  → Trust Domain: west.qhx.dev
Cluster: UK-PROD  → Trust Domain: uk.qhx.dev

Trust Bundle

A trust bundle contains CA certificates used to verify SVIDs. Federation allows clusters to trust each other’s CAs:

east.qhx.dev bundle contains:
  - east.qhx.dev CA certificate (ML-DSA-65)
  - west.qhx.dev CA certificate (federated)

west.qhx.dev bundle contains:
  - west.qhx.dev CA certificate (ML-DSA-65)
  - east.qhx.dev CA certificate (federated)

Bundle Endpoint

An HTTPS endpoint that serves the trust bundle:

https://pki-east-lb.example.com:8443
→ Returns east.qhx.dev trust bundle (JSON)

Step 1: Prerequisites

Verify QHx Installation

On both clusters:

# Set KUBECONFIG for each cluster
export KUBECONFIG_EAST=~/.kube/config-east
export KUBECONFIG_WEST=~/.kube/config-west

# Check QHx is running
KUBECONFIG=$KUBECONFIG_EAST kubectl -n qhx-system get pod
KUBECONFIG=$KUBECONFIG_WEST kubectl -n qhx-system get pod

# Verify PKI Servers are healthy
KUBECONFIG=$KUBECONFIG_EAST kubectl -n qhx-system exec pki-server-0 -c pki-server -- \
  /opt/spire/bin/spire-server healthcheck

KUBECONFIG=$KUBECONFIG_WEST kubectl -n qhx-system exec pki-server-0 -c pki-server -- \
  /opt/spire/bin/spire-server healthcheck

Network Connectivity

Ensure network connectivity between clusters:

# Test from EAST to WEST (after LoadBalancer created)
WEST_LB=$(KUBECONFIG=$KUBECONFIG_WEST kubectl -n qhx-system get svc pki-server-bundle-endpoint -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
curl -k https://$WEST_LB:8443

# Should return trust bundle JSON

Step 2: Create Bundle Endpoint LoadBalancers

AWS (Network Load Balancer)

For EAST cluster:

apiVersion: v1
kind: Service
metadata:
  name: pki-server-bundle-endpoint
  namespace: qhx-system
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing  # or internal
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
  labels:
    app: pki-server-bundle-endpoint
spec:
  type: LoadBalancer
  selector:
    app: qhx-pki-server
  ports:
  - name: bundle-endpoint
    port: 8443
    targetPort: 8443
    protocol: TCP
  sessionAffinity: None
  ipFamilyPolicy: SingleStack
  ipFamilies: [IPv4]

Apply to both clusters:

# EAST
KUBECONFIG=$KUBECONFIG_EAST kubectl apply -f pki-bundle-endpoint-east.yaml

# WEST
KUBECONFIG=$KUBECONFIG_WEST kubectl apply -f pki-bundle-endpoint-west.yaml

Azure (Standard Load Balancer)

apiVersion: v1
kind: Service
metadata:
  name: pki-server-bundle-endpoint
  namespace: qhx-system
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "false"
spec:
  type: LoadBalancer
  selector:
    app: qhx-pki-server
  ports:
  - port: 8443
    targetPort: 8443

GCP (Network Load Balancer)

apiVersion: v1
kind: Service
metadata:
  name: pki-server-bundle-endpoint
  namespace: qhx-system
  annotations:
    cloud.google.com/load-balancer-type: "External"
spec:
  type: LoadBalancer
  selector:
    app: qhx-pki-server
  ports:
  - port: 8443
    targetPort: 8443

Get LoadBalancer Hostnames

# Wait for LoadBalancers to be provisioned (can take 5-10 minutes)
KUBECONFIG=$KUBECONFIG_EAST kubectl -n qhx-system get svc pki-server-bundle-endpoint -w

# Get hostnames
EAST_LB=$(KUBECONFIG=$KUBECONFIG_EAST kubectl -n qhx-system get svc pki-server-bundle-endpoint -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
WEST_LB=$(KUBECONFIG=$KUBECONFIG_WEST kubectl -n qhx-system get svc pki-server-bundle-endpoint -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

echo "EAST LoadBalancer: $EAST_LB"
echo "WEST LoadBalancer: $WEST_LB"

Step 3: Configure Federation

Update PKI Server Configuration

For EAST cluster (federating with WEST):

# Get current config
KUBECONFIG=$KUBECONFIG_EAST kubectl -n qhx-system get cm pki-server -o yaml > pki-server-east.yaml

# Edit to add federation
cat >> pki-server-east.yaml <<EOF
        # BEGIN FEDERATION
        federates_with "west.qhx.dev" {
          bundle_endpoint_url = "https://${WEST_LB}:8443"
          bundle_endpoint_profile "https_spiffe" {
            endpoint_spiffe_id = "spiffe://west.qhx.dev/spire/server"
          }
        }
        # END FEDERATION
EOF

# Apply updated config
KUBECONFIG=$KUBECONFIG_EAST kubectl apply -f pki-server-east.yaml

For WEST cluster (federating with EAST):

# Get current config
KUBECONFIG=$KUBECONFIG_WEST kubectl -n qhx-system get cm pki-server -o yaml > pki-server-west.yaml

# Edit to add federation
cat >> pki-server-west.yaml <<EOF
        # BEGIN FEDERATION
        federates_with "east.qhx.dev" {
          bundle_endpoint_url = "https://${EAST_LB}:8443"
          bundle_endpoint_profile "https_spiffe" {
            endpoint_spiffe_id = "spiffe://east.qhx.dev/spire/server"
          }
        }
        # END FEDERATION
EOF

# Apply updated config
KUBECONFIG=$KUBECONFIG_WEST kubectl apply -f pki-server-west.yaml

Restart PKI Servers

# Restart EAST
KUBECONFIG=$KUBECONFIG_EAST kubectl -n qhx-system delete pod pki-server-0

# Wait for restart
KUBECONFIG=$KUBECONFIG_EAST kubectl -n qhx-system wait --for=condition=ready pod pki-server-0 --timeout=120s

# Restart WEST
KUBECONFIG=$KUBECONFIG_WEST kubectl -n qhx-system delete pod pki-server-0

# Wait for restart
KUBECONFIG=$KUBECONFIG_WEST kubectl -n qhx-system wait --for=condition=ready pod pki-server-0 --timeout=120s

Step 4: Exchange Trust Bundles

Export Trust Bundles

# Export EAST bundle
KUBECONFIG=$KUBECONFIG_EAST kubectl -n qhx-system exec pki-server-0 -c pki-server -- \
  /opt/spire/bin/spire-server bundle show -format spiffe \
  -socketPath /tmp/pki-server/private/api.sock \
  > east.bundle

# Export WEST bundle
KUBECONFIG=$KUBECONFIG_WEST kubectl -n qhx-system exec pki-server-0 -c pki-server -- \
  /opt/spire/bin/spire-server bundle show -format spiffe \
  -socketPath /tmp/pki-server/private/api.sock \
  > west.bundle

Import Trust Bundles

# Import WEST bundle into EAST
KUBECONFIG=$KUBECONFIG_EAST kubectl -n qhx-system exec -i pki-server-0 -c pki-server -- \
  /opt/spire/bin/spire-server bundle set -format spiffe \
  -socketPath /tmp/pki-server/private/api.sock \
  -id spiffe://west.qhx.dev \
  < west.bundle

# Import EAST bundle into WEST
KUBECONFIG=$KUBECONFIG_WEST kubectl -n qhx-system exec -i pki-server-0 -c pki-server -- \
  /opt/spire/bin/spire-server bundle set -format spiffe \
  -socketPath /tmp/pki-server/private/api.sock \
  -id spiffe://east.qhx.dev \
  < east.bundle

Step 5: Verify Federation

Check Federation Status

# On EAST, show federated bundles
KUBECONFIG=$KUBECONFIG_EAST kubectl -n qhx-system exec pki-server-0 -c pki-server -- \
  /opt/spire/bin/spire-server bundle show -format spiffe \
  -socketPath /tmp/pki-server/private/api.sock

# Should show:
# - east.qhx.dev bundle (local)
# - west.qhx.dev bundle (federated)

Test Cross-Cluster Authentication

Deploy test workload in EAST:

apiVersion: v1
kind: Pod
metadata:
  name: east-client
  namespace: demo
spec:
  containers:
  - name: client
    image: curlimages/curl:latest
    command: ["sleep", "3600"]

Deploy test workload in WEST:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: west-server
  namespace: demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: west-server
  template:
    metadata:
      labels:
        app: west-server
    spec:
      containers:
      - name: server
        image: hashicorp/http-echo:latest
        args: ["-text=Hello from WEST"]
---
apiVersion: v1
kind: Service
metadata:
  name: west-server
  namespace: demo
spec:
  selector:
    app: west-server
  ports:
  - port: 8080
    targetPort: 5678

Test cross-cluster request:

# From EAST client, call WEST server
KUBECONFIG=$KUBECONFIG_EAST kubectl -n demo exec east-client -- \
  curl http://west-server.west-cluster.svc.cluster.local:8080

# If federation works, request succeeds with SPIFFE identity verification

Cross-Domain Policies

Create Flowspec for Cross-Domain Access

On WEST cluster (allow EAST workloads):

apiVersion: qhx.dev/v1
kind: QHxFlowspec
metadata:
  name: allow-east-to-west
  namespace: demo
spec:
  action: allow
  match:
    source:
      trustDomain: east.qhx.dev
      namespace: demo
      serviceAccount: client
    destination:
      trustDomain: west.qhx.dev
      namespace: demo
      serviceAccount: west-server
  ports:
  - 8080

Apply:

KUBECONFIG=$KUBECONFIG_WEST kubectl apply -f allow-east-to-west.yaml

Multi-Region Deployment Patterns

Pattern 1: Active-Active

Both regions serve traffic independently with federation for cross-region calls.

        ┌──────────────┐
        │   Global LB  │
        └──────┬───────┘
               │
       ┌───────┴────────┐
       │                │
┌──────▼─────┐   ┌──────▼─────┐
│  US-EAST   │   │  US-WEST   │
│  Cluster   │◄─►│  Cluster   │
│            │   │            │
│ Active     │   │ Active     │
└────────────┘   └────────────┘

Pattern 2: Active-Passive (DR)

Primary site serves traffic, DR site for failover.

┌──────────────┐
│   Primary    │
│  US-EAST     │◄──┐
│  (Active)    │   │ Federation
└──────────────┘   │ (Bundle sync)
                   │
┌──────────────┐   │
│     DR       │   │
│  US-WEST     │◄──┘
│  (Standby)   │
└──────────────┘

Pattern 3: Hub-and-Spoke

Central cluster federates with regional clusters.

       ┌──────────────┐
       │   Hub        │
       │  Central DC  │
       └──────┬───────┘
              │
      ┌───────┼───────┐
      │       │       │
┌─────▼──┐ ┌──▼────┐ ┌▼─────┐
│ US-EAST│ │US-WEST│ │  UK  │
│ (Spoke)│ │(Spoke)│ │(Spoke)│
└────────┘ └───────┘ └──────┘

Security Considerations

Network Security

Restrict bundle endpoint access:

# AWS Security Group (example)
Ingress:
  - Protocol: TCP
    Port: 8443
    Source: <west-cluster-nat-gateway-ips>

Use internal LoadBalancers for private connectivity:

annotations:
  service.beta.kubernetes.io/aws-load-balancer-internal: "true"
  service.beta.kubernetes.io/aws-load-balancer-scheme: internal

TLS Verification

Bundle endpoints use SPIFFE authentication:

Client: EAST PKI Server
  ↓
  1. Connect to https://west-lb:8443
  2. Verify server presents SPIFFE ID: spiffe://west.qhx.dev/spire/server
  3. Verify certificate chains to west.qhx.dev CA
  4. Download trust bundle

Bundle Rotation

Trust bundles automatically update when CAs rotate:

# Check bundle update frequency (default: 5 minutes)
KUBECONFIG=$KUBECONFIG_EAST kubectl -n qhx-system logs pki-server-0 -c pki-server | grep "bundle updated"

Troubleshooting

Issue: Bundle endpoint not reachable

Symptom: Federation fails with connection timeout

Check:

# Test connectivity
curl -k https://$WEST_LB:8443

# Check LoadBalancer status
KUBECONFIG=$KUBECONFIG_WEST kubectl -n qhx-system describe svc pki-server-bundle-endpoint

Solutions:

Verify security groups allow port 8443
Check LoadBalancer provisioning (can take 5-10 minutes)
Verify DNS resolution of LoadBalancer hostname

Issue: Trust bundle not updating

Symptom: Federated bundles not appearing

Check:

# Check PKI Server logs
KUBECONFIG=$KUBECONFIG_EAST kubectl -n qhx-system logs pki-server-0 -c pki-server | grep federation

Solutions:

Verify federates_with config is correct
Restart PKI Server: kubectl -n qhx-system delete pod pki-server-0
Manually set bundle again (Step 4)

Issue: Cross-domain authentication fails

Symptom: Workloads cannot authenticate across clusters

Check:

# Verify both bundles present
KUBECONFIG=$KUBECONFIG_EAST kubectl -n qhx-system exec pki-server-0 -c pki-server -- \
  /opt/spire/bin/spire-server bundle show | grep "Trust domain"

# Should show both east.qhx.dev and west.qhx.dev

Solutions:

Verify flowspecs allow cross-domain traffic
Check network connectivity between clusters
Verify workloads have correct SPIFFE IDs

Cleanup

To remove federation:

# Remove federation config from ConfigMaps
# (Edit pki-server ConfigMap, remove federates_with block)

# Restart PKI Servers
KUBECONFIG=$KUBECONFIG_EAST kubectl -n qhx-system delete pod pki-server-0
KUBECONFIG=$KUBECONFIG_WEST kubectl -n qhx-system delete pod pki-server-0

# Delete LoadBalancers
KUBECONFIG=$KUBECONFIG_EAST kubectl -n qhx-system delete svc pki-server-bundle-endpoint
KUBECONFIG=$KUBECONFIG_WEST kubectl -n qhx-system delete svc pki-server-bundle-endpoint

Automation

Automated Federation Script

#!/bin/bash
set -eo pipefail

CLUSTER1=$1
CLUSTER2=$2
KUBECONFIG1=$3
KUBECONFIG2=$4

echo "Federating $CLUSTER1 with $CLUSTER2..."

# Create LoadBalancers
KUBECONFIG=$KUBECONFIG1 kubectl apply -f pki-bundle-endpoint.yaml
KUBECONFIG=$KUBECONFIG2 kubectl apply -f pki-bundle-endpoint.yaml

# Wait for LoadBalancers
echo "Waiting for LoadBalancers..."
sleep 60

# Get LoadBalancer hostnames
LB1=$(KUBECONFIG=$KUBECONFIG1 kubectl -n qhx-system get svc pki-server-bundle-endpoint -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
LB2=$(KUBECONFIG=$KUBECONFIG2 kubectl -n qhx-system get svc pki-server-bundle-endpoint -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')

# Update configs
# (Add federates_with blocks)

# Exchange bundles
# (Export and import as in Step 4)

echo "Federation complete!"

Quick Start - Initial QHx setup
EKS Deployment - Multi-cluster deployment
Flowspecs - Cross-domain policies
Architecture - Trust domain design

Summary

Federation enables:

Cross-cluster workload authentication
Multi-region deployments
Disaster recovery setups
Hybrid cloud architectures
Automatic trust bundle synchronization

Federation is essential for enterprise deployments spanning multiple sites or clouds.

Federation Setup

Overview

Federation Architecture

Concepts

Trust Domain

Trust Bundle

Bundle Endpoint

Step 1: Prerequisites

Verify QHx Installation

Network Connectivity

Step 2: Create Bundle Endpoint LoadBalancers

AWS (Network Load Balancer)

Azure (Standard Load Balancer)

GCP (Network Load Balancer)

Get LoadBalancer Hostnames

Step 3: Configure Federation

Update PKI Server Configuration

Restart PKI Servers

Step 4: Exchange Trust Bundles

Export Trust Bundles

Import Trust Bundles

Step 5: Verify Federation

Check Federation Status

Test Cross-Cluster Authentication

Cross-Domain Policies

Create Flowspec for Cross-Domain Access

Multi-Region Deployment Patterns

Pattern 1: Active-Active

Pattern 2: Active-Passive (DR)

Pattern 3: Hub-and-Spoke

Security Considerations

Network Security

TLS Verification

Bundle Rotation

Troubleshooting

Issue: Bundle endpoint not reachable

Issue: Trust bundle not updating

Issue: Cross-domain authentication fails

Cleanup

Automation

Automated Federation Script

Related Documentation

Summary