Federation Setup

Federation lets the selected QHx cluster acquire and maintain trust in a peer cluster’s authorities. The operator authorizes the relationship; the QHx Manager in the selected cluster retrieves and verifies the peer’s federation data. Configure each direction that your deployment needs.

Here, selected cluster means the cluster whose relationship you are configuring, identified by its kubeconfig and context. The peer cluster is the other participant. These are roles, not locations: either cluster can be remote from the host running the CLI.

Use the qhx federation reference for command options. Check qhx --version and qhx federation connect --help on the host you will use; available commands and help wording depend on the installed CLI.

Prerequisites

Before connecting clusters, confirm that:

• Both clusters have QHx installed, active authorities, and a published bootstrap package. The clusters must have distinct trust domains.
• You have a trusted way to obtain the peer cluster’s bootstrap reference or package, or authenticated Kubernetes access to both clusters.
• Your Kubernetes credentials permit the required reads and creation or update of QHxForeignCluster resources in the receiving cluster. The workflow with two kubeconfigs and package export also require Kubernetes port-forward access.
• The QHx Manager in the selected cluster can resolve and reach the peer’s M2M endpoint for online bootstrap and subsequent synchronization. For two-way federation, each manager needs access to the other cluster’s endpoint.

Inspect the QHx objects and M2M Service using the appropriate kubeconfig:

kubectl --kubeconfig ./selected.kubeconfig get qhxclusters,qhxauthorities
kubectl --kubeconfig ./selected.kubeconfig -n qhx-system get service m2m

The M2M Service listens on port 7500 within the cluster. An externally exposed NodePort can use a different port. qhx federation info discovers the NodePort endpoint by default; use --endpoint-host host:port to advertise an address reachable from the receiving manager when discovery is unsuitable. An address without a port assumes 7500. Restrict network access to the intended peers.

Federation establishes trust. It does not create cross-cluster application DNS, network routes, or application authorization rules. Configure those separately and test the actual protected application path after bootstrap completes.

Connect using a bootstrap reference

On a host with Kubernetes access to the peer cluster, print its reference:

qhx federation info --kubeconfig ./peer.kubeconfig \
  --endpoint-host peer-m2m.example.com:7500

Replace the endpoint with the peer cluster’s reachable address and port. The output includes the bootstrap reference, cluster trust domain, and package fingerprint. Transfer the reference through your organization’s trusted administrative channel and confirm which peer cluster it identifies.

On a host with Kubernetes access to the selected cluster, set PEER_REFERENCE to that complete qhx-bootstrap: value, then run:

qhx federation connect --kubeconfig ./selected.kubeconfig "$PEER_REFERENCE"

When given a bootstrap reference, the CLI records it in the selected cluster. The QHx Manager in that cluster then fetches the peer’s bootstrap package and verifies it against the reference. The operator’s host needs Kubernetes API access to the selected cluster; it does not need direct access to the peer’s M2M endpoint for this command.

This creates the selected cluster’s relationship with the peer cluster. To configure the reverse direction, exchange the selected cluster’s reference and repeat the operation with the peer cluster selected.

Connect using two kubeconfigs

If the operator has authenticated Kubernetes access to both clusters, pass the selected cluster’s kubeconfig first and the peer’s second:

qhx federation connect -K ./selected.kubeconfig -K ./peer.kubeconfig

The CLI reads and verifies both bootstrap packages through Kubernetes port-forwarding to obtain their fingerprints and trust domains. It discovers the clusters’ M2M NodePort addresses and records a reference to the other cluster on each side. The QHx Manager in each cluster then fetches and verifies its peer’s bootstrap package through the peer’s M2M endpoint.

For a relationship only from the selected cluster to the peer:

qhx federation connect -K ./selected.kubeconfig -K ./peer.kubeconfig \
  --unidirectional

This still requires Kubernetes API and port-forward access to both clusters. Only the selected cluster’s manager needs to reach the peer’s M2M endpoint for the relationship being created. If discovered NodePort addresses are unreachable, use the bootstrap-reference workflow with an explicit endpoint instead.

Transfer a bootstrap package offline

On a host with Kubernetes access to the peer cluster, export its package:

qhx federation export bootstrap-package -K ./peer.kubeconfig \
  --endpoint-host peer-m2m.example.com:7500 -f peer-bootstrap.cbor

The CLI reads the served package through Kubernetes port-forwarding and writes it to the file. The endpoint is an unsigned routing hint; it must point to the peer cluster’s M2M service for later synchronization.

Transfer the file through your approved administrative process. On the host where you will import it, inspect it before authorizing the relationship:

qhx federation inspect bootstrap-package -f peer-bootstrap.cbor

Inspection reports the package contents, fingerprint, and verification result. A valid self-signature alone does not establish that the package belongs to the intended peer cluster. Confirm the source of the file and its trust domain. If you obtained a fingerprint independently, set PEER_FINGERPRINT to that value and assert it during import:

qhx federation import bootstrap-package -K ./selected.kubeconfig \
  -f peer-bootstrap.cbor --expect-fingerprint "$PEER_FINGERPRINT"

--expect-fingerprint is optional; without it, the operator’s vetting of the file provides the external trust decision. Import verifies the package and creates or updates QHxForeignCluster in the selected cluster with its accepted bootstrap state. The QHx Manager in that cluster uses the endpoint hint for subsequent synchronization; --m2m-endpoint host:port can override it. Offline transfer does not provide ongoing synchronization while the peer’s endpoint is unreachable.

Verify the relationship

A successful connect command confirms that the CLI recorded the reference. The QHx Manager in the selected cluster completes bootstrap asynchronously. Inspect the resulting resources in each receiving cluster:

kubectl --kubeconfig ./selected.kubeconfig get qhxforeignclusters -o yaml

Check the relationship for the intended peer trust domain. Its conditions explain the current state:

Condition	Meaning
Accepted	The relationship specification is valid.
Bootstrapped	The manager holds accepted peer trust state with intact continuity.
FederationReconciled	The accepted state is consistent and ready for federation reconciliation.
Ready	The three conditions above are true.
Synchronized	The most recent fetch and verification cycle succeeded.
RetentionDegraded	When true, retained trust material needs attention, for example because a root is nearing expiry.

A temporary peer outage can make Synchronized false while Ready remains true. Review the condition messages and status.m2m.lastSuccessTime rather than treating one successful CLI invocation as an ongoing health check. Then test the application’s own network path and authorization policy.

Troubleshooting and recovery

• If the CLI cannot read cluster information, check the kubeconfig context, API reachability, and Kubernetes permissions. Export and two-kubeconfig connection also need permission to open port-forwards.
• If bootstrap is unreachable, check spec.m2mEndpoints, DNS, firewall rules, and the peer’s M2M Service from the selected cluster’s manager network. A working path from the operator’s laptop does not prove that the manager can reach it.
• If fingerprint or trust-domain verification fails, obtain a fresh reference through the trusted administrative channel and investigate the mismatch. Do not replace the trust reference merely to suppress a verification error.
• If the relationship reports broken, expired, or unverifiable continuity, vet a current package before using the import command’s --rebootstrap option. --force-rebootstrap replaces even a relationship whose continuity still works; use it only for an intentional replacement of peer trust.

Remove a relationship

Identify the resource by its peer trust domain, then delete that specific QHxForeignCluster with kubectl in the selected cluster. Removing this relationship does not remove the reverse relationship in the peer cluster. Coordinate both sides when removing two-way federation, and separately check application access and existing connections during the change.