#Global Cluster Disaster Recovery

This page is the disaster recovery entry point for a global cluster that runs on Immutable Infrastructure. Deploy-time disaster recovery configuration is part of the global cluster installation procedure.

#Deployment Procedure

Use Optional Disaster Recovery Deployment in the installation guide when you create the primary and standby global clusters.

That deployment procedure is the authoritative source for the installation-time DR configuration, including:

• Primary and standby clusters use the same Kubernetes API server encryption provider configuration.
• The KubeadmControlPlane etcd server certificate SAN list includes both the primary and standby control plane VIPs and the Platform Access Address. This list is separate from the platform console.cert SAN list.
• Huawei DCS deployments reference the shared encryption provider Secret from DCSCluster.spec.encryptionProviderConfigRef.
• VMware vSphere deployments write the same /etc/kubernetes/encryption-provider.conf file through KubeadmControlPlane.spec.kubeadmConfigSpec.files.
• Huawei Cloud Stack deployments write the same /etc/kubernetes/encryption-provider.conf file through KubeadmControlPlane.spec.kubeadmConfigSpec.files.
• Bare Metal deployments reference the shared encryption provider Secret from BaremetalCluster.spec.encryptionProviderConfigRef and use the same Kubernetes ServiceAccount signing key on both sides.
• Huawei DCS, VMware vSphere, and Huawei Cloud Stack create the dcs-import-extra-resources ConfigMap before installer import so the installation can import referenced credential Secrets and, where applicable, provider-specific infrastructure resources. The name keeps the dcs prefix for historical installer compatibility.
• Bare Metal creates the dcs-import-extra-resources ConfigMap before installer import so the installation can preserve the durable bare-metal and elemental owner resources required by handoff. It does not import Global plan Secrets or kubeadm bootstrap data Secrets; fresh handoff creates the initial Global plan Secrets after all target Global MachineInventory objects exist.
• The standby cluster installs global-etcd-sync with connection values that point to the active primary after both installations succeed.

#Operational Scope

After the primary and standby clusters are installed, operate DR as a separate lifecycle process. Keep the installation manifests aligned with the installation guide, then use an approved operations runbook for the following tasks:

• Verify etcd-sync health and replication lag.
• Verify that the standby cluster can decrypt Kubernetes Secrets created on the primary cluster.
• Validate the primary and standby control plane VIPs and the platform access path before a planned failover.
• Execute failover and failback with an approved operations procedure.
• Reconcile provider-specific resources after a failover.

#Bare Metal DR Model

Bare Metal DR adds host-level state that does not exist in VM-based providers. The global cluster does not create physical machines; it manages long-lived MachineInventory objects and writes plan Secrets that are consumed by elemental-system-agent on each host. Global-cluster machines therefore use a cluster-local identity, while non-global workload machines use a shared identity that follows the synchronized workload state. A standby global cluster must preserve its local Global identity while accepting the shared workload token from primary.

#Endpoint Rules

Use different system-agent endpoints for global hosts and workload-cluster hosts:

Global hosts always use their own side's direct kube-apiserver VIP; they do not follow the platform domain when active and standby switch roles. Workload hosts always use the platform domain so that their endpoint follows the active Global cluster.

During the bootstrap KIND phase, a Global host registration still uses the bootstrap endpoint, normally https://<bootstrap-host-ip>:12443. The bootstrap MachineRegistration must set the Global auth scope, but it must not set baremetal.cluster.io/system-agent-server-url; handoff has not happened yet.

metadata:
  annotations:
    baremetal.cluster.io/system-agent-auth-scope: global

Set handoffHook.controlPlaneVIP to the current side's VIP and set handoffHook.directAPIServer: true on both Bare Metal provider AppRelease objects. Handoff then rewrites only the installed Global hosts to https://<local-global-control-plane-vip>:6443, with no /kubernetes/global path. The agent uses the kube-apiserver CA from the local ServiceAccount token Secret. Workload hosts keep the platform ingress path.

For every post-handoff Global host registration, set the Global auth scope, the local side's direct API URL, and direct endpoint mode. A primary Global registration uses the primary VIP, and a standby Global registration uses the standby VIP.

apiVersion: elemental.cattle.io/v1beta1
kind: MachineRegistration
metadata:
  name: global-<purpose>
  namespace: cpaas-system
  annotations:
    baremetal.cluster.io/system-agent-auth-scope: global
    baremetal.cluster.io/system-agent-server-url: https://<local-global-control-plane-vip>:6443
    baremetal.cluster.io/system-agent-endpoint-mode: direct-apiserver

Use the explicit baremetal.cluster.io/system-agent-endpoint-mode: direct-apiserver annotation for every new Global registration.

#System-Agent Identity Requirements

Bare Metal DR uses two identity bundles:

• The local Global bundle consists of ServiceAccount, token Secret, Role, and RoleBinding named baremetal-global-system-agent (the Secret name has the -token suffix). Each side generates and retains its own bundle. Its Role contains only that side's Global plan Secrets.
• The workload shared bundle consists of the four baremetal-system-agent objects. It may be absent on both sides before the first shared-scope workload plan exists. Once primary creates it, etcd-sync copies the bundle exactly to standby. Its Role contains only explicitly shared-scope, non-global workload plan Secrets.

Primary and standby must still use the same /etc/kubernetes/pki/sa.key, /etc/kubernetes/pki/sa.pub, service-account-issuer, and api-audiences values so that the shared workload token is valid on both API servers. The two local Global tokens are intentionally different and must not be synchronized.

Set elemental.systemAgent.splitAuthEnabled: true on both provider AppRelease objects. This is the single source of truth used by both elemental-operator and handoff; do not configure a separate handoff auth-enable switch. With the split enabled, the active Global must set elemental.systemAgent.sharedAuthReadOnly: false so that elemental-operator reconciles the shared workload Role. The inactive Global must set it to true so that etcd-sync, not the local operator, owns the synchronized shared bundle. Both operators continue to reconcile their own Global-local bundles.

#etcd-sync Scope

Deploy etcd-sync on the standby cluster. The sync direction is one-way before failover:

source: primary global etcd
target: standby global etcd

Synchronize the workload-cluster resources that allow standby to continue reconciling existing Bare Metal clusters:

• Non-global CAPI resources: Cluster, Machine, MachineDeployment, KubeadmControlPlane, bootstrap templates, and related owner resources.
• Non-global Bare Metal provider resources: BaremetalCluster, BaremetalMachine, BaremetalMachineTemplate, and MachineInventoryPool.
• Non-global elemental resources: MachineInventory and MachineRegistration.
• Plan Secrets, bootstrap data Secrets, workload kubeconfig Secrets, and workload credential Secrets that are required by the synchronized resources.
• Provider referenced Secrets that must exist on the target side, including the encryption-provider Secret referenced by BaremetalCluster.spec.encryptionProviderConfigRef when Bare Metal DR is enabled.
• Exact shared auth bundle, once it exists: ServiceAccount/cpaas-system/baremetal-system-agent, Secret/cpaas-system/baremetal-system-agent-token, Role/cpaas-system/baremetal-system-agent, and RoleBinding/cpaas-system/baremetal-system-agent.

Do not use the synchronization list in this section to populate Step 7 dcs-import-extra-resources; the initial Global handoff excludes Global plan Secrets, KubeadmConfig objects, and kubeadm bootstrap data Secrets.

Do not synchronize:

• SeedImage.
• Standby Cluster/cpaas-system/global.
• Standby BaremetalCluster/cpaas-system/global.
• Standby BaremetalMachine, MachineInventoryPool, MachineInventory, and MachineRegistration objects that belong to the standby global cluster.
• ServiceAccount/cpaas-system/baremetal-global-system-agent, Secret/cpaas-system/baremetal-global-system-agent-token, Role/cpaas-system/baremetal-global-system-agent, and RoleBinding/cpaas-system/baremetal-global-system-agent from either side.
• ConfigMap/cpaas-system/baremetal-system-agent-handoff from either side.
• The standby-local token Secret selected by active_cluster_token_secret_ref and ServiceMonitor/cpaas-system/etcd-sync-monitor; MR !179 renders exact-ignore entries for both.
• Plan Secrets that belong to either side's local Cluster/global. Add their exact etcd keys to etcd_sync.additional_ignore_equal before starting synchronization.
• Secrets from the primary cluster that are not required by workload cluster reconciliation.

Before installing the ModuleInfo, follow the executable two-kubeconfig procedure in Install Primary and Standby Clusters to collect Role/baremetal-global-system-agent.rules[].resourceNames from both sides, merge and de-duplicate the names, and populate spec.config.additional_ignore_equal with exactly the corresponding /registry/secrets/cpaas-system/<plan-secret> keys. Do not duplicate static ignore keys in this dynamic list.

MR !179 adds a checksum of the rendered sync-key ConfigMap to both Deployment Pod templates. A fresh installation therefore starts with the final filter and does not need a manual rollout restart. Verify the dynamic and static entries, checksum annotations, Deployment readiness, and zero-difference monitor result with the installation guide's acceptance commands.

#Failover Procedure

Use this sequence for a planned or declared failover:

1. Fence the primary global write path. Stop or isolate the controllers and platform entry points that can continue writing workload cluster resources. Do this at the owning CR or platform-resource layer where possible; scaling a Deployment alone can be reverted by its owner.

2. Wait until primary-to-standby etcd-sync is caught up, then freeze or stop it. Do not allow old primary data to overwrite new standby writes after failover.

3. If your provider uses a sync gate, clear the gate after the sync has been stopped so standby controllers do not keep requeueing because they think synchronization is still in progress.

4. Switch the platform domain to the standby platform entrance.

5. Verify DNS and ingress:

   dig +short <platform-domain>
   curl -kI https://<platform-domain>/kubernetes/global/version
   curl -kI https://<standby-control-plane-vip>/kubernetes/global/version

   A 401 response is acceptable; it proves the request reached the platform ingress.

6. Confirm the standby cluster has the provider components needed to manage Bare Metal workload clusters: CAPI core, kubeadm provider, Bare Metal provider, elemental-operator, cluster-transformer, and alive-related delivery components.

7. Confirm existing workload hosts reconnect through the platform domain and can watch or patch their plan Secrets.

#Post-Failover Validation

After standby is active, validate in increasing risk order:

• Existing workload cluster remains Ready.
• All BaremetalMachine objects remain bound to the expected MachineInventory objects.
• Plan Secrets can be updated and the MachineInventory.status.plan feedback changes.
• Worker scale-out allocates an available inventory and updates Role/baremetal-system-agent.resourceNames with the new plan Secret.
• Worker scale-in runs a clean plan and releases inventory as designed.
• Control-plane replacement restores the desired KCP replica count and etcd membership.
• A new MachineRegistration and SeedImage created on standby can register a new host.
• A pre-failover workload ISO is usable only if its registration URL uses the platform domain.

#Bare Metal Troubleshooting

#Provider Notes

#See Also

For traditional-operating-system global cluster disaster recovery, see Global Cluster Disaster Recovery.

For installation and upgrade of the global cluster on Immutable Infrastructure, see:

• Installing the global Cluster
• Upgrading the global Cluster