Immutable Infrastructure

Installing the global Cluster

This document describes how to install the global cluster onto Immutable Infrastructure. The global cluster is the platform control plane and is provisioned through Cluster API. Use this path when the platform control plane must run on an immutable operating system such as Alauda OS.

TOC

When to Use This PathCommon PrerequisitesCompatibility and Version InputsProcedureStep 1 — Prepare Common VariablesStep 2 — Create the Bootstrap ClusterStep 3 — Upload and Install Provider PackagesStep 4 — Configure the Provider-Specific global ManifestStep 5 — Apply the global ManifestStep 6 — Wait for the Control PlaneStep 7 — Import Provider ResourcesStep 8 — Trigger the Platform InstallationStep 9 — Monitor the InstallationPhases You Will ObserveSignals During InstallationCommon Stalls and Where to LookOptional Disaster Recovery DeploymentPrepare Shared DR VariablesPrepare Shared ServiceAccount Signing KeyAdd DR etcd Server Certificate SANs to KubeadmControlPlaneAdd Provider-Specific DR FieldsInstall Primary and Standby ClustersVerificationDecommission the Bootstrap ClusterNext StepsWorked Example: Complete global Manifest for Huawei DCSBefore You Apply: Prepare These on DCSValues to Replace

When to Use This Path

Choose this installation path when all of the following conditions apply:

  • You want the global cluster to run on an immutable operating system. Alauda OS is the supported image today.
  • Your infrastructure is one of the documented providers: Huawei DCS, VMware vSphere, Huawei Cloud Stack, or Bare Metal.
  • You can run a temporary bootstrap host that has network access to the target IaaS platform.

For traditional operating systems such as Ubuntu or RHEL, use the standard installation path instead.

Common Prerequisites

The following prerequisites apply to every provider:

  • A bootstrap host that meets the minimum hardware and network requirements. See the Overview for sizing guidance.
  • The Core Package from the Customer Portal.
  • The Alauda Container Platform Kubeadm Provider package.
  • The infrastructure provider package for your target platform.
  • Network reachability between the bootstrap host and the target IaaS platform API endpoint.
  • IP and hostname planning for the global control plane and worker nodes. See Infrastructure Resources for the resource model used by each provider.
  • A stable Kubernetes API endpoint for the global cluster, such as a VIP or load balancer address.
  • A platform access address, registry address, and Pod and Service CIDR ranges.
  • For x86_64 nodes that use ACP-provided Alauda OS images, the underlying CPUs must support the x86-64-v2 ISA baseline. See OS Support Matrix.
Naming Convention (Required)

This rule applies to every infrastructure provider supported by this install path — Huawei DCS, Huawei Cloud Stack, VMware vSphere, and any provider added in the future. Every manifest you author in Step 4 must follow it. Misnaming these resources has two distinct failure modes, both detailed below; one breaks initial provisioning, the other only surfaces during disaster recovery.

  • The CAPI Cluster and the provider's infrastructure cluster resource (for example, DCSCluster for Huawei DCS or HCSCluster for Huawei Cloud Stack; each provider has its own equivalent) must be named exactly global. cpaas-installer looks them up by literal name, and the Huawei Cloud Stack provider only allocates the global ELB listener ports (11443 for the registry and console, 2379 for DR etcd-sync, 443 for web access) when the infra cluster is named global. A different name silently breaks registry pull, DR etcd-sync, and the web console.
  • Every other CAPI resource (KubeadmControlPlane, KubeadmConfigTemplate, MachineDeployment) and every other provider infrastructure resource (machine templates, IP/hostname pools, machine config pools, and any other per-provider resource) must use a name with the global- prefix. The DR (failover) mechanism uses this prefix to identify resources owned by the global cluster. A global cluster resource without the global- prefix is invisible to DR and causes the standby cluster's machines to be deleted at failover time — the cluster will provision and run normally, then lose nodes the first time DR is exercised. This is a hard requirement, not a stylistic convention.
  • Cluster.spec.controlPlaneRef.name and any other cross-references must match the prefixed names exactly.

Compatibility and Version Inputs

Before installation, record the supported version set for the delivery package:

InputPurpose
Core Package versionProvides the installer, local registry, and base platform payload.
Kubeadm provider chart versionMust match the Cluster API control plane resources used by the global manifest.
Infrastructure provider chart versionUse the VMware vSphere, DCS, HCS, or Bare Metal provider chart version delivered with the target release.
Alauda OS image or VM templateMust contain the Kubernetes version used by K8S_VERSION.
K8S_VERSIONUse v-prefixed semver that matches the target Alauda OS image, such as v<major>.<minor>.<patch>.

Procedure

Step 1 — Prepare Common Variables

Set the common variables on the bootstrap host.

export HOST_IP="<bootstrap-host-ip>"
export LOCAL_REGISTRY_ADDRESS="127.0.0.1:11443"
export BOOTSTRAP_REGISTRY_ADDRESS="172.18.0.1:11443"
export NODE_REGISTRY_ADDRESS="${HOST_IP}:11443"
export CONTROL_PLANE_VIP="<global-control-plane-vip>"
export PLATFORM_HOST="<platform-access-domain-or-vip>"
export REGISTRY_DOMAIN="<platform-registry-domain-or-vip>:11443"
export CLUSTER_CIDR="100.3.0.0/16"
export SERVICE_CIDR="100.4.0.0/16"
export KUBE_OVN_JOIN_CIDR="<kube-ovn-join-cidr>"
export K8S_VERSION="<target-kubernetes-version>"
export INGRESS_CLASS_NAME="global-alb2"
export PROVIDER_SECRET_NAME="global-secret"
# Use v-prefixed semver that matches the target Alauda OS image.

Use LOCAL_REGISTRY_ADDRESS when pushing packages from the bootstrap host. Use BOOTSTRAP_REGISTRY_ADDRESS in AppRelease chart repository values because provider Pods read the chart repository from inside the bootstrap cluster's network. Use NODE_REGISTRY_ADDRESS (the bootstrap host's registry, <bootstrap-host-ip>:11443) in the Cluster API registry annotations, because provisioned global nodes must pull images through an address reachable from their subnet during provisioning. This is a temporary value: after the global cluster's own registry comes up, the installer automatically rewrites the cpaas.io/registry-address annotation on the Cluster and DCSCluster to the permanent platform registry, so later reconciles pull from the global cluster instead of the bootstrap host.

Step 2 — Create the Bootstrap Cluster

Run the bootstrap script provided by the Core Package with Bash. This brings up a temporary KIND-based bootstrap cluster named minialauda on the bootstrap host — the temporary Cluster API management cluster used only to provision the global cluster. After it completes, make the matching kubectl client from the bootstrap control-plane container available on the bootstrap host and configure the exported kubeconfig.

mkdir -p /root/cpaas-install
tar -xvf <core-package> -C /root/cpaas-install
cd /root/cpaas-install/installer
bash setup.sh
mkdir -p "${HOME}/.local/bin" ~/.kube

if ! command -v kubectl >/dev/null 2>&1; then
  nerdctl cp minialauda-control-plane:/usr/bin/kubectl \
    "${HOME}/.local/bin/kubectl"
  chmod 0755 "${HOME}/.local/bin/kubectl"
  export PATH="${HOME}/.local/bin:${PATH}"
fi

cp /var/cpaas/data/alauda.kubeconfig ~/.kube/config
kubectl get nodes

The bootstrap script provisions an embedded registry, the Cluster API control plane, and the installer components that drive the global cluster installation.

Step 3 — Upload and Install Provider Packages

Upload the Kubeadm provider package and the infrastructure provider package to the local registry.

Why cluster.type is Baremetal for every provider

The AppRelease values in the tabs below all set global.cluster.type: Baremetal. This is a chart-internal classifier, not the IaaS provider name. Keep Baremetal for the Huawei DCS, VMware vSphere, Huawei Cloud Stack, and Bare Metal global installations. The value drives how the platform configures node-level components; it does not select the infrastructure provider.

Huawei DCS
VMware vSphere
Huawei Cloud Stack
Bare Metal

Set the provider package paths and chart versions.

export DCS_PROVIDER_PACK="/root/cluster-api-provider-dcs.amd64.<version>.tgz"
export KUBEADM_PROVIDER_PACK="/root/cluster-api-provider-kubeadm.amd64.<version>.tgz"
export DCS_PROVIDER_VERSION="<dcs-provider-chart-version>"
export KUBEADM_PROVIDER_VERSION="<kubeadm-provider-chart-version>"

Upload the packages.

/root/cpaas-install/installer/res/amd64/packtool pack push \
  -r "${LOCAL_REGISTRY_ADDRESS}" -c "${DCS_PROVIDER_PACK}"

/root/cpaas-install/installer/res/amd64/packtool pack push \
  -r "${LOCAL_REGISTRY_ADDRESS}" -c "${KUBEADM_PROVIDER_PACK}"

Create and apply the AppRelease resources for the Kubeadm provider and the DCS provider.

mkdir -p /root/yamls
export DCS_PROVIDER_APPRELEASES="/root/yamls/dcs-provider-appreleases.yaml"

cat > "${DCS_PROVIDER_APPRELEASES}" <<EOF
---
apiVersion: operator.alauda.io/v1alpha1
kind: AppRelease
metadata:
  annotations:
    auto-recycle: "true"
    interval-sync: "true"
  name: cluster-api-provider-kubeadm
  namespace: cpaas-system
spec:
  destination:
    cluster: ""
    namespace: ""
  source:
    chartPullSecret: global-registry-auth
    charts:
      - name: ait/chart-cluster-api-provider-kubeadm
        releaseName: cluster-api-provider-kubeadm
        targetRevision: ${KUBEADM_PROVIDER_VERSION}
    repoURL: ${BOOTSTRAP_REGISTRY_ADDRESS}
  timeout: 120
  values:
    global:
      albName: ${INGRESS_CLASS_NAME}
      auth:
        default_admin: admin@cpaas.io
      cluster:
        isGlobal: true
        name: global
        networkType: kube-ovn
        type: Baremetal
      host: ${PLATFORM_HOST}
      ingress:
        ingressClassName: ${INGRESS_CLASS_NAME}
      labelBaseDomain: cpaas.io
      namespace: cpaas-system
      platformUrl: https://${PLATFORM_HOST}
      protectSecretFiles:
        enabled: false
      region: global
      registry:
        address: ${BOOTSTRAP_REGISTRY_ADDRESS}
        imagePullSecrets:
          - global-registry-auth
      replicas: 1
      scheme: https
---
apiVersion: operator.alauda.io/v1alpha1
kind: AppRelease
metadata:
  annotations:
    auto-recycle: "true"
    interval-sync: "true"
  name: cluster-api-provider-dcs
  namespace: cpaas-system
spec:
  destination:
    cluster: ""
    namespace: ""
  source:
    chartPullSecret: global-registry-auth
    charts:
      - name: ait/chart-cluster-api-provider-dcs
        releaseName: cluster-api-provider-dcs
        targetRevision: ${DCS_PROVIDER_VERSION}
    repoURL: ${BOOTSTRAP_REGISTRY_ADDRESS}
  timeout: 120
  values:
    global:
      albName: ${INGRESS_CLASS_NAME}
      auth:
        default_admin: admin@cpaas.io
      cluster:
        isGlobal: true
        name: global
        networkType: kube-ovn
        type: Baremetal
      host: ${PLATFORM_HOST}
      ingress:
        ingressClassName: ${INGRESS_CLASS_NAME}
      labelBaseDomain: cpaas.io
      namespace: cpaas-system
      platformUrl: https://${PLATFORM_HOST}
      protectSecretFiles:
        enabled: false
      region: global
      registry:
        address: ${BOOTSTRAP_REGISTRY_ADDRESS}
        imagePullSecrets:
          - global-registry-auth
      replicas: 1
      scheme: https
EOF

kubectl apply -f "${DCS_PROVIDER_APPRELEASES}"

until kubectl get crd kubeadmcontrolplanes.controlplane.cluster.x-k8s.io --ignore-not-found 2>/dev/null | grep -q kubeadmcontrolplanes.controlplane.cluster.x-k8s.io; do
  sleep 10
done

until kubectl get crd dcsclusters.infrastructure.cluster.x-k8s.io --ignore-not-found 2>/dev/null | grep -q dcsclusters.infrastructure.cluster.x-k8s.io; do
  sleep 10
done

Step 4 — Configure the Provider-Specific global Manifest

Create one provider-specific manifest for the global cluster. The manifest uses the same provider resources as a workload cluster, but it must also include the global-specific labels, annotations, registry values, installer-compatible kubeadm settings, and persistent data paths required by the platform control plane.

Use the provider creation guides as the detailed resource reference:

Apply the naming convention from Common Prerequisites to every resource in the manifest you author below.

Set KubeadmControlPlane.spec.kubeadmConfigSpec.format to the value that the target provider accepts. The provider controllers enforce this:

ProviderBootstrap userdata format
Huawei DCSignition (provider-enforced; the DCS provider rejects any other format with invalid format, expected ignition, got <other>).
VMware vSpherecloud-init (provider default; setting ignition is not supported).
Huawei Cloud Stackcloud-init (provider-enforced; the HCS provider rejects ignition with ignition format is not supported).
Bare Metalcloud-init (the bare-metal provider consumes CAPI bootstrap data and renders elemental plans).
Huawei DCS
VMware vSphere
Huawei Cloud Stack
Bare Metal

Set the output path for the DCS global manifest before you render it.

export GLOBAL_DCS_YAML="/root/yamls/new-global.yaml"

The DCS global manifest must contain the following resources in the cpaas-system namespace:

ResourcePurpose
Secret with type: CloudCredentialStores authUser, authKey, endpoint, and site for DCS API access.
DCSIpHostnamePool for control plane nodesAssigns static IPs, hostnames, network settings, and any pool-managed persistent disks.
DCSMachineTemplate for control plane nodesDefines the DCS VM template, folder, CPU, memory, and template-local disks.
KubeadmControlPlaneBootstraps the Kubernetes control plane. Set spec.version to ${K8S_VERSION}.
DCSClusterDefines the DCS infrastructure cluster and control plane endpoint.
ClusterConnects the Cluster API Cluster to DCSCluster and KubeadmControlPlane.
DCSIpHostnamePool, DCSMachineTemplate, KubeadmConfigTemplate, and MachineDeployment for workersCreates worker nodes.

Use the DCS resource fields from Creating Clusters on Huawei DCS and Infrastructure Resources for Huawei DCS. For the global cluster, keep these additional requirements:

  • Set Cluster.metadata.name and DCSCluster.metadata.name to global (the infra cluster shares the CAPI Cluster name). Prefix every other CAPI resource and provider resource with global-; the wiring fragment below uses KubeadmControlPlane.metadata.name: global-kcp.
  • Set DCSCluster.spec.credentialSecretRef.name to ${PROVIDER_SECRET_NAME}. Step 7 imports this Secret into the final global cluster.
  • Add Cluster.metadata.labels.is-global: "true" and Cluster.metadata.labels.cluster-type: DCS.
  • Add Cluster.metadata.annotations["cpaas.io/registry-address"] with ${NODE_REGISTRY_ADDRESS}.
  • Set KubeadmControlPlane.spec.kubeadmConfigSpec.format: ignition for Alauda OS.
  • Keep the KubeadmControlPlane.spec.kubeadmConfigSpec.users entry with a non-empty sshAuthorizedKeys list (the boot user). The DCS ignition format rejects an empty SSH key list, so this field is required even for a global cluster you do not plan to access over SSH. See Resolving Placeholder Values for what to supply when no interactive key is needed.
  • Keep the non-encryption kubeadm files, kubelet patches, audit policy, and installer RBAC entries. The file contents (the PodSecurity admission config, the kubelet patch, and the audit policy), together with the full clusterConfiguration, preKubeadmCommands, postKubeadmCommands, and the init and join node-registration patches, are identical to a workload cluster. Copy them from the Complete KubeadmControlPlane Configuration appendix, or reference the dcs-kubernetes-<major.minor>-files Secret documented there. The wiring fragment below shows only the global-specific fields layered on top of that base.
  • For a normal non-DR deployment, do not set DCSCluster.spec.encryptionProviderConfigRef and do not add /etc/kubernetes/encryption-provider.conf to KubeadmControlPlane.spec.kubeadmConfigSpec.files.
  • Keep /var/cpaas as platform state. If you need the disk to survive rolling replacement, declare it in DCSIpHostnamePool.spec.pool[].persistentDisk; do not rely on DCSMachineTemplate template disks as preserved state.
  • Use concrete datastoreName values for DCS local storage unless you have verified that the selected datastore cluster can place volumes on hosts that can run the target VM.
Fragment Scope

The following YAML is a differential fragment, not a complete manifest that you can apply directly. Merge these global-specific changes into the manifest that you prepare from the DCS create-cluster references, then apply the complete manifest file. If you would rather start from a complete file, adapt the Worked Example: Complete global Manifest for Huawei DCS at the end of this page instead of assembling it from fragments.

The following fragment shows the global-specific Cluster API wiring. Fill the provider resource fields by using the DCS create-cluster references above.

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: global
  namespace: cpaas-system
  labels:
    cluster-type: DCS
    is-global: "true"
  annotations:
    capi.cpaas.io/resource-group-version: infrastructure.cluster.x-k8s.io/v1beta1
    capi.cpaas.io/resource-kind: DCSCluster
    cpaas.io/registry-address: "${NODE_REGISTRY_ADDRESS}"
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
        - ${CLUSTER_CIDR}
    services:
      cidrBlocks:
        - ${SERVICE_CIDR}
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: KubeadmControlPlane
    name: global-kcp
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: DCSCluster
    name: global
---
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
  name: global-kcp
  namespace: cpaas-system
  annotations:
    controlplane.cluster.x-k8s.io/skip-kube-proxy: ""
spec:
  replicas: 3
  version: ${K8S_VERSION}
  rolloutStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 0
  machineTemplate:
    nodeDrainTimeout: 1m
    nodeDeletionTimeout: 5m
    infrastructureRef:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: DCSMachineTemplate
      name: global-cp-template
  kubeadmConfigSpec:
    format: ignition
    clusterConfiguration:
      etcd:
        local:
          serverCertSANs:
            - "${CONTROL_PLANE_VIP}"
            - "${PLATFORM_HOST}"

Step 5 — Apply the global Manifest

Apply the provider-specific manifest to the bootstrap cluster.

Huawei DCS
VMware vSphere
Huawei Cloud Stack
Bare Metal
kubectl apply -f "${GLOBAL_DCS_YAML}"

Step 6 — Wait for the Control Plane

Wait for the Cluster API provider to provision the machines and bring up the Kubernetes control plane.

kubectl get clusters.cluster.x-k8s.io -n cpaas-system
kubectl get kubeadmcontrolplane -n cpaas-system
kubectl get machines -n cpaas-system

The control plane is ready when the KubeadmControlPlane reports Ready: True and the Cluster reports Phase: Provisioned.

Step 7 — Import Provider Resources

Before triggering the installer, create the dcs-import-extra-resources ConfigMap in the cpaas-system namespace for providers that require extra resource import. The ConfigMap name keeps the dcs prefix for historical installer compatibility, even when the provider is not Huawei DCS.

Every provider uses this ConfigMap to import the IaaS credential Secret into the new global cluster. For VMware vSphere, Huawei Cloud Stack, and Bare Metal it also imports the provider's Cluster API resources; for Huawei DCS those resources are migrated by the built-in flow, so the DCS ConfigMap only needs the credential Secret entry. VMware vSphere, Huawei Cloud Stack, and Bare Metal require it for both normal and disaster recovery global installations.

Huawei DCS
VMware vSphere
Huawei Cloud Stack
Bare Metal

The DCS provider's Cluster API resources are migrated by the built-in flow, so the ConfigMap only needs to import the credential Secret referenced by DCSCluster.spec.credentialSecretRef.name. Use the same Secret name you set in the global manifest.

mkdir -p /root/yamls
cat > /root/yamls/dcs-import-extra-resources.yaml <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: dcs-import-extra-resources
  namespace: cpaas-system
data:
  resources.yaml: |
    resources:
    - resource: "secrets"
      names: ["${PROVIDER_SECRET_NAME}"]
      method: kubectl
EOF

kubectl apply -f /root/yamls/dcs-import-extra-resources.yaml

Step 8 — Trigger the Platform Installation

Submit the platform installation request to the embedded installer REST API. The installer imports the Cluster API resources into the new global cluster, deploys the base operator, and installs the selected plugins.

export INSTALLER_IP=$(kubectl get pods -n cpaas-system -l service_name=cpaas-installer \
  -o jsonpath='{.items[0].status.podIP}')
Network Scope

INSTALLER_IP is the Pod IP of the embedded installer in the bootstrap cluster. The endpoint is used only during installation.

Create the provider-specific installer configuration JSON file on the current bootstrap host, then submit it to the installer endpoint. All providers in this install path use the same endpoint path, but their request bodies are different.

FieldHuawei DCSVMware vSphereHuawei Cloud StackBare Metal
Endpoint path/cpaas-installer/api/config/dcs/cpaas-installer/api/config/dcs/cpaas-installer/api/config/dcs/cpaas-installer/api/config/dcs
console.hostLocal global HA VIP listEmpty list, []Empty list, []Local global control-plane VIP list
console.globalHostPlatform access addressPlatform access addressPlatform access addressPlatform access address
cluster.clusterCIDR and cluster.serviceCIDRRequiredNot set; cluster CIDRs are declared in the VMware vSphere Cluster manifestNot setRequired
cluster.features.haRequired, points to the local HA VIP with isThirdParty: trueNot set; the control plane endpoint is declared in VSphereCluster.spec.controlPlaneEndpoint.hostNot set; HCS ELB is declared in HCSClusterRequired, points to the bare-metal control-plane VIP with isThirdParty: true
hostIPCurrent bootstrap host IPCurrent bootstrap host IPCurrent bootstrap host IPCurrent bootstrap host IP
Huawei DCS
VMware vSphere
Huawei Cloud Stack
Bare Metal

The DCS installer request includes the external HA VIP because DCS uses a third-party control plane VIP.

mkdir -p /root/yamls
export INSTALLER_CONFIG_JSON="/root/yamls/installer-config-dcs.json"

cat > "${INSTALLER_CONFIG_JSON}" <<EOF
{
  "basic": {
    "username": "admin@cpaas.io",
    "password": "<base64-platform-admin-password>"
  },
  "registry": {
    "domain": "${REGISTRY_DOMAIN}",
    "username": "<registry-username>",
    "password": "<base64-registry-password>"
  },
  "console": {
    "host": [
      "${CONTROL_PLANE_VIP}"
    ],
    "globalHost": "${PLATFORM_HOST}",
    "httpPort": 80,
    "httpsPort": 443,
    "cert": {
      "selfSigned": {}
    }
  },
  "cluster": {
    "clusterCIDR": "${CLUSTER_CIDR}",
    "serviceCIDR": "${SERVICE_CIDR}",
    "features": {
      "ha": {
        "vip": "${CONTROL_PLANE_VIP}",
        "vport": 6443,
        "isThirdParty": true
      }
    }
  },
  "product": [
    "base",
    "acp"
  ],
  "deployMode": "normal",
  "hostIP": "${HOST_IP}"
}
EOF

curl -k -X POST "http://${INSTALLER_IP}:8080/cpaas-installer/api/config/dcs" \
  -H 'Content-Type: application/json' \
  -d @"${INSTALLER_CONFIG_JSON}"

Set console.host and cluster.features.ha.vip to the local global HA VIP. Do not use the platform domain in console.host; use console.globalHost for the platform access address.

Third-Party Console Certificates

The examples use a self-signed console certificate. If the environment requires a third-party certificate, replace console.cert with a thirdParty block that contains the base64 full certificate chain, private key, and optional PKCS#12 values before you submit the installer request. This certificate is only for platform HTTPS ingress; it does not configure the kube-apiserver or etcd certificates generated from KubeadmControlPlane. The Global registry on port 11443 uses an independently managed global-registry-server certificate chain, not console.cert or dex.tls.

DR Certificate Requirement

For a primary/standby Bare Metal global DR deployment, do not let each side generate an unrelated self-signed certificate. Configure the same trusted thirdParty platform certificate on both sides. Its required SAN is the stable ${PLATFORM_HOST} domain. Do not add the primary or standby control-plane VIPs or internal Service names by default. Add a VIP SAN only when clients intentionally access platform HTTPS directly through that VIP. Registry access through port 11443 does not use this certificate; registry certificate management is independent and outside this certificate step. Global hosts handed off to https://<control-plane-vip>:6443 validate the separate kube-apiserver certificate and CA, not console.cert or dex.tls.

Step 9 — Monitor the Installation

After the installer accepts the request, the install runs through several phases that are observable from the bootstrap host. A typical immutable-OS global cluster takes 30–60 minutes; total time depends on IaaS provisioning speed, image pull time, and the number of plugins selected.

Phases You Will Observe

PhaseWhat is happeningFirst place to watch
BootstrapThe bootstrap cluster, embedded registry, and Cluster API providers are running on the bootstrap host. Completed in Step 2 and Step 3.bootstrap host terminal; kubectl get pods -n cpaas-system
Infrastructure provisioningThe Cluster API provider creates VMs from the Alauda OS template on the target IaaS platform.kubectl get machines -n cpaas-system
Control plane bootstrapKubeadmControlPlane bootstraps the first control plane node, etcd starts, and additional control plane nodes join.kubectl get kubeadmcontrolplane -n cpaas-system
Network and core add-onsThe CAPI provider reconciles Kube-OVN, CoreDNS, and kube-proxy on the new cluster.kubectl --kubeconfig <global-kubeconfig> get pods -n kube-system
Platform installationThe installer imports Cluster API resources into the new global cluster, deploys the base operator, and installs the selected plugins.Installer progress API; installer log
CompletionThe installer marks the request as Success and writes the final cluster state into ClusterModule/global.Installer progress API; kubectl --kubeconfig <global-kubeconfig> get clustermodule global

Signals During Installation

Watch the installer progress API and the installer log together. If one appears stalled, check the underlying Cluster API resources directly on the bootstrap host.

# Installer progress and live log
curl "http://${INSTALLER_IP}:8080/cpaas-installer/api/progress"
tail -f /var/cpaas/data/installer.log

# Cluster API resources on the bootstrap host
kubectl get clusters.cluster.x-k8s.io -A
kubectl get kubeadmcontrolplane -A
kubectl get machines -A

The installer log records every phase transition. Transient errors retry on a short interval; persistent errors stay visible in the log and surface in the progress API as a stalled stage.

Check the global cluster after the installer reports success.

kubectl --kubeconfig <global-kubeconfig> get nodes
kubectl --kubeconfig <global-kubeconfig> get pods -n cpaas-system
kubectl --kubeconfig <global-kubeconfig> get clustermodule global

Common Stalls and Where to Look

SymptomFirst place to lookWhat you are looking for
Machines stay in Pending or do not appearkubectl describe machine -n cpaas-system <machine>The provider-specific failure reason on the machine Bootstrap and Infrastructure conditions. IaaS quota, network, and credential issues surface here.
KubeadmControlPlane does not reach Readykubectl get nodes with the new cluster kubeconfig and kubectl describe kubeadmcontrolplane -n cpaas-systemetcd health on the first control plane node and join progress for the remaining nodes.
Pods in kube-system stay Pending or fail to pull imageskubectl --kubeconfig <global-kubeconfig> describe pod -n kube-system <pod>Image pull errors usually mean the node-facing registry address is not reachable from the new cluster's subnet.
Installer progress API shows a stalled stage/var/cpaas/data/installer.logThe most recent phase line and the most recent error message. Retried errors repeat on a short interval; persistent errors do not advance.
ClusterModule/global does not reach a healthy phasekubectl --kubeconfig <global-kubeconfig> describe clustermodule globalThe Status.conditions describe which module is blocking the cluster from completing.

Issues that are not listed here usually point to environment-specific causes. Capture the installer log, the progress API response, and the relevant kubectl describe output, then escalate.

Optional Disaster Recovery Deployment

Use this section when you deploy primary and standby global clusters for disaster recovery. Complete these additions before you apply the provider-specific manifest for each global cluster.

Bare Metal Fresh-install Scope

The Bare Metal split-auth procedure in this section covers only a fresh installation of the primary and standby global clusters. It does not define or validate an in-place upgrade or migration of an existing Bare Metal DR pair, or the repair or recovery of a failed environment. Use a separately validated operations runbook for those lifecycle tasks.

Primary and standby clusters must use the same encryption provider configuration. For Bare Metal, primary and standby must also use the same Kubernetes ServiceAccount signing key so that the workload-cluster baremetal-system-agent token created on the primary cluster is accepted by the standby API server after failover. The machines that form each global cluster use a separate cluster-local identity and do not share their token with the peer global cluster. For DCS and Bare Metal, the provider-specific cluster resource references a Secret that contains encryption-provider.conf; for HCS, normal non-DR deployments do not add /etc/kubernetes/encryption-provider.conf to KubeadmControlPlane.spec.kubeadmConfigSpec.files. VMware vSphere keeps the release manifest's /etc/kubernetes/encryption-provider.conf file entry.

Prepare Shared DR Variables

Set the same encryption key value on both the primary and standby installation environments.

export ENCRYPTION_PROVIDER_CONF="/root/yamls/encryption-provider.conf"
export ENCRYPTION_PROVIDER_SECRET_B64="<base64-shared-etcd-encryption-key>"
export PRIMARY_CLUSTER_VIP="<primary-ha-vip>"
export STANDBY_CLUSTER_VIP="<standby-ha-vip>"
export BAREMETAL_ENCRYPTION_PROVIDER_SECRET="global-encryption-provider-config"
export ETCD_SYNC_VERSION="<etcd-sync-version-containing-86c2ce5>"
export ETCD_SYNC_MODULEINFO="/root/yamls/global-etcd-sync-moduleinfo.json"
export SERVICE_ACCOUNT_ISSUER="https://kubernetes.default.svc.cluster.local"

Use the tested artifact produced from etcd-sync MR !179 head commit 86c2ce500c059ff5226dd2f5ebb5b6e908faf06d (86c2ce5). That revision contains the Global-local identity rules, plugin ServiceMonitor exact-ignore rule, dynamic Global plan-key input, and the local public-registry-credential exact-ignore rule required when each Global uses its own VIP Registry. It also removes the newly introduced legacy owner/cluster fallback: Global-local classification requires the explicit baremetal.cluster.io/system-agent-auth-scope: global annotation. An unannotated Elemental object is not protected as Global-local, remains eligible for the generic synchronization rules, and therefore fails fresh-install Global validation. Replace the placeholder with the published artifact version whose provenance proves it contains 86c2ce5 or an explicitly verified equivalent merge commit. Do not infer compatibility from a higher-looking version string alone.

Create the encryption provider configuration file on both installation environments.

mkdir -p "$(dirname "${ENCRYPTION_PROVIDER_CONF}")"
cat > "${ENCRYPTION_PROVIDER_CONF}" <<EOF_CONF
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aescbc:
      keys:
      - name: key1
        secret: ${ENCRYPTION_PROVIDER_SECRET_B64}
EOF_CONF

Prepare Shared ServiceAccount Signing Key

For Bare Metal DR, generate the ServiceAccount signing key once and use the same files in both the primary and standby KubeadmControlPlane manifests.

mkdir -p /root/global-dr-sa
openssl genrsa -out /root/global-dr-sa/sa.key 2048
openssl rsa -in /root/global-dr-sa/sa.key -pubout -out /root/global-dr-sa/sa.pub
chmod 0600 /root/global-dr-sa/sa.key
chmod 0644 /root/global-dr-sa/sa.pub

kubectl -n cpaas-system create secret generic global-sa-signing-key \
  --from-file=sa.key=/root/global-dr-sa/sa.key \
  --from-file=sa.pub=/root/global-dr-sa/sa.pub \
  --dry-run=client -o yaml | kubectl apply -f -

Add the following entries to the primary and standby KubeadmControlPlane.spec.kubeadmConfigSpec. The file content and the issuer/audience values must be identical on both sides.

files:
  - path: /etc/kubernetes/pki/sa.key
    owner: root:root
    permissions: "0600"
    contentFrom:
      secret:
        name: global-sa-signing-key
        key: sa.key
  - path: /etc/kubernetes/pki/sa.pub
    owner: root:root
    permissions: "0644"
    contentFrom:
      secret:
        name: global-sa-signing-key
        key: sa.pub
clusterConfiguration:
  apiServer:
    extraArgs:
      service-account-key-file: /etc/kubernetes/pki/sa.pub
      service-account-signing-key-file: /etc/kubernetes/pki/sa.key
      service-account-issuer: https://kubernetes.default.svc.cluster.local
      api-audiences: https://kubernetes.default.svc.cluster.local
  controllerManager:
    extraArgs:
      service-account-private-key-file: /etc/kubernetes/pki/sa.key

After the clusters are installed, verify the files and kubeadm static pod arguments on one control-plane node from each side.

sha256sum /etc/kubernetes/pki/sa.key /etc/kubernetes/pki/sa.pub
grep -E 'service-account-issuer|api-audiences|service-account-key-file|service-account-signing-key-file' \
  /etc/kubernetes/manifests/kube-apiserver.yaml
grep -E 'service-account-private-key-file' \
  /etc/kubernetes/manifests/kube-controller-manager.yaml

Add DR etcd Server Certificate SANs to KubeadmControlPlane

In the manifest generated in Step 4, include both the primary and standby control plane VIPs, the platform access address, and etcd.kube-system in KubeadmControlPlane.spec.kubeadmConfigSpec.clusterConfiguration.etcd.local.serverCertSANs. Use the same SAN list on both the primary and standby installation environments. These values configure the etcd server certificate generated by kubeadm; they are independent of the platform console.cert and must not be copied into its thirdParty SAN list.

serverCertSANs:
  - "${PRIMARY_CLUSTER_VIP}"
  - "${STANDBY_CLUSTER_VIP}"
  - "${PLATFORM_HOST}"
  - "etcd.kube-system"

Add Provider-Specific DR Fields

Huawei DCS
VMware vSphere
Huawei Cloud Stack
Bare Metal

Create the encryption provider Secret in the bootstrap cluster.

kubectl create secret generic encryption-provider-config \
  --from-file=encryption-provider.conf="${ENCRYPTION_PROVIDER_CONF}" \
  -n cpaas-system \
  --dry-run=client -o yaml | kubectl apply -f -

Add the Secret reference to DCSCluster.spec.

encryptionProviderConfigRef:
  name: encryption-provider-config

DCS uses DCSCluster.spec.encryptionProviderConfigRef to deliver the disaster recovery encryption provider configuration. Do not add /etc/kubernetes/encryption-provider.conf to KubeadmControlPlane.spec.kubeadmConfigSpec.files for the DCS DR path.

Create the DCS dcs-import-extra-resources ConfigMap from Step 7 on both installation environments. Set PROVIDER_SECRET_NAME to the same Secret name used by DCSCluster.spec.credentialSecretRef.name.

Install Primary and Standby Clusters

Run Steps 1 through 9 for both the primary and standby global clusters.

For Bare Metal DR, use two independent bootstrap hosts: one for the primary installation and one for the standby installation. Do not reuse the same bootstrap cluster for both sides. The bootstrap environment contains installer state, AppRelease objects, registry Secrets, MachineRegistration, SeedImage, and handoff state; sharing it can pollute the two global installations and can make handoff or cleanup affect the wrong side.

Use the provider-specific installer configuration differences for both sides:

ProviderPrimary installationStandby installation
Huawei DCSSet console.host and cluster.features.ha.vip to the primary HA VIP. Create the DCS dcs-import-extra-resources ConfigMap from Step 7 and keep PROVIDER_SECRET_NAME aligned with DCSCluster.spec.credentialSecretRef.name.Set console.host and cluster.features.ha.vip to the standby HA VIP. Create the DCS dcs-import-extra-resources ConfigMap from Step 7 and keep PROVIDER_SECRET_NAME aligned with DCSCluster.spec.credentialSecretRef.name.
VMware vSphereSet VSphereCluster.spec.controlPlaneEndpoint.host to the primary HA VIP used by the primary manifest. Create the VMware vSphere dcs-import-extra-resources ConfigMap from Step 7 and keep global-vsphere-credentials aligned with VSphereCluster.spec.identityRef.name.Set VSphereCluster.spec.controlPlaneEndpoint.host to the standby HA VIP used by the standby manifest. Create the VMware vSphere dcs-import-extra-resources ConfigMap from Step 7 and keep global-vsphere-credentials aligned with VSphereCluster.spec.identityRef.name.
Huawei Cloud StackKeep console.host: []; the primary VIP is managed by the HCS ELB. Create the HCS dcs-import-extra-resources ConfigMap from Step 7 and keep HCS_SECRET_NAME aligned with HCSCluster.spec.identityRef.name.Keep console.host: []; the standby VIP is managed by the HCS ELB. Create the HCS dcs-import-extra-resources ConfigMap from Step 7 and keep HCS_SECRET_NAME aligned with HCSCluster.spec.identityRef.name.
Bare MetalSet console.host, cluster.features.ha.vip, BaremetalCluster.spec.controlPlaneLoadBalancer.host, and handoffHook.controlPlaneVIP to the primary control-plane VIP. Set handoffHook.directAPIServer: true, elemental.systemAgent.splitAuthEnabled: true, and elemental.systemAgent.sharedAuthReadOnly: false. Create the Bare Metal dcs-import-extra-resources ConfigMap from Step 7.Set console.host, cluster.features.ha.vip, BaremetalCluster.spec.controlPlaneLoadBalancer.host, and handoffHook.controlPlaneVIP to the standby control-plane VIP. Set handoffHook.directAPIServer: true, elemental.systemAgent.splitAuthEnabled: true, and elemental.systemAgent.sharedAuthReadOnly: true. Create the Bare Metal dcs-import-extra-resources ConfigMap from Step 7.

For the primary cluster, make sure the platform domain resolves to the primary HA VIP. In Step 8, set hostIP to the primary bootstrap host IP. For DCS, set console.host and cluster.features.ha.vip to the primary HA VIP. For VMware vSphere, set the control plane endpoint in the primary manifest to the primary HA VIP. For HCS, keep console.host: [] because the VIP is owned by the HCS ELB. For Bare Metal, set both the manifest VIP and the installer VIP fields to the primary control-plane VIP.

After the primary cluster installation succeeds, switch the platform domain to the standby HA VIP as required by the DR procedure. Then install the standby cluster. This DNS switch before the standby installation is required because several platform resources are rendered with the platform domain and must resolve to the standby entrance while the standby installer runs. In Step 8 on the standby bootstrap host, set hostIP to the standby bootstrap host IP. For DCS, set console.host and cluster.features.ha.vip to the standby HA VIP. For VMware vSphere, set the control plane endpoint in the standby manifest to the standby HA VIP. For HCS, keep console.host: []. For Bare Metal, set both the manifest VIP and the installer VIP fields to the standby control-plane VIP, and set REGISTRY_DOMAIN to the standby ${CONTROL_PLANE_VIP}:11443. The primary installation must likewise use its own control-plane VIP. Do not change either Registry address when the platform domain switches. Get INSTALLER_IP from the cpaas-installer Pod on the standby bootstrap host; do not reuse the primary bootstrap host value.

For Bare Metal, verify the permanent Registry propagation on each final Global before installing etcd-sync. Keep ProductBase.spec.registry.preferPlatformURL at the installer-generated value; the Global-local Registry address is carried by spec.registry.address and the two Cluster annotations.

verify_baremetal_global_registry() {
  kubeconfig=$1
  expected_registry=$2

  test "$(kubectl --kubeconfig "${kubeconfig}" \
    get productbase.product.alauda.io base \
    -o jsonpath='{.spec.registry.address}')" = "${expected_registry}"
  test "$(kubectl --kubeconfig "${kubeconfig}" \
    get cluster.platform.tkestack.io global \
    -o jsonpath='{.metadata.annotations.cpaas\.io/registry-address}')" = \
    "${expected_registry}"
  test "$(kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
    get cluster.cluster.x-k8s.io global \
    -o jsonpath='{.metadata.annotations.cpaas\.io/registry-address}')" = \
    "${expected_registry}"
  test "$(kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
    get secret public-registry-credential -o jsonpath='{.data.registry}' | \
    base64 -d)" = "${expected_registry}"

  kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
    get secret global-registry-auth -o jsonpath='{.data.\.dockerconfigjson}' | \
    base64 -d | jq -e --arg registry "${expected_registry}" \
      '.auths[$registry] != null' >/dev/null

  kubectl --kubeconfig "${kubeconfig}" get apprelease -A -o json | \
    jq -e --arg registry "${expected_registry}" '
      all(.items[];
        .spec.source.repoURL == $registry and
        .spec.values.global.registry.address == $registry)
    ' >/dev/null
}

export PRIMARY_GLOBAL_REGISTRY_ADDRESS="<primary-control-plane-vip>:11443"
export STANDBY_GLOBAL_REGISTRY_ADDRESS="<standby-control-plane-vip>:11443"

verify_baremetal_global_registry \
  "${PRIMARY_GLOBAL_KUBECONFIG}" "${PRIMARY_GLOBAL_REGISTRY_ADDRESS}"
verify_baremetal_global_registry \
  "${STANDBY_GLOBAL_KUBECONFIG}" "${STANDBY_GLOBAL_REGISTRY_ADDRESS}"

For Bare Metal, run the complete handoff gate on both sides before decommissioning either bootstrap KIND cluster. GLOBAL_HOSTS is the space-separated management address list for that side's installed Global machines. Run this from a secured host that can use both Global kubeconfigs and SSH to the machines. Keep shell tracing disabled because the runtime connection file contains a bearer token.

set -euo pipefail
set +x

export PRIMARY_GLOBAL_KUBECONFIG="<path-to-primary-global-kubeconfig>"
export STANDBY_GLOBAL_KUBECONFIG="<path-to-standby-global-kubeconfig>"
export PRIMARY_BOOTSTRAP_KUBECONFIG="<path-to-primary-bootstrap-kubeconfig>"
export STANDBY_BOOTSTRAP_KUBECONFIG="<path-to-standby-bootstrap-kubeconfig>"
export PRIMARY_GLOBAL_HOSTS="<primary-global-host-1> <primary-global-host-2> <primary-global-host-3>"
export STANDBY_GLOBAL_HOSTS="<standby-global-host-1> <standby-global-host-2> <standby-global-host-3>"
export PRIMARY_GLOBAL_REGISTRATION_NAME="<primary-global-machine-registration-name>"
export STANDBY_GLOBAL_REGISTRATION_NAME="<standby-global-machine-registration-name>"

verify_global_auth_scope() {
  kubeconfig=$1
  registration_name=$2

  kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
    get machineregistration.elemental.cattle.io "${registration_name}" -o json | \
  jq -e '
    .metadata.annotations["baremetal.cluster.io/system-agent-auth-scope"] ==
      "global"
  ' >/dev/null

  global_inventory_names="$({
    kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
      get baremetalmachines.infrastructure.cluster.x-k8s.io -o json | \
    jq -c '[
      .items[]
      | select(
          .metadata.labels["cluster.x-k8s.io/cluster-name"] == "global"
        )
      | .status.machineInventoryRef.name? // empty
    ] | unique | sort'
  })"
  printf '%s\n' "${global_inventory_names}" | \
    jq -e 'length > 0' >/dev/null

  while IFS= read -r inventory_name; do
    kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
      get machineinventory.elemental.cattle.io "${inventory_name}" -o json | \
    jq -e '
      .metadata.annotations["baremetal.cluster.io/system-agent-auth-scope"] ==
        "global"
    ' >/dev/null
  done < <(printf '%s\n' "${global_inventory_names}" | jq -r '.[]')
}

verify_baremetal_handoff() {
  bootstrap_kubeconfig=$1
  kubeconfig=$2
  control_plane_vip=$3
  global_hosts=$4
  registration_name=$5
  expected_endpoint="https://${control_plane_vip}:6443"

  kubectl --kubeconfig "${bootstrap_kubeconfig}" -n cpaas-system \
    wait --for=condition=complete \
    job/baremetal-system-agent-handoff --timeout=30m

  kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
    get configmap baremetal-system-agent-handoff -o json | \
    jq -e '.data.ready == "true"' >/dev/null

  verify_global_auth_scope "${kubeconfig}" "${registration_name}"

  if kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
    get rolebinding baremetal-global-system-agent-handoff-bridge \
    >/dev/null 2>&1; then
    echo "temporary handoff bridge still exists" >&2
    return 1
  fi

  kubectl --kubeconfig "${kubeconfig}" -n cpaas-system get \
    serviceaccount/baremetal-global-system-agent \
    secret/baremetal-global-system-agent-token \
    role/baremetal-global-system-agent \
    rolebinding/baremetal-global-system-agent >/dev/null

  global_plan_names="$({
    kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
      get role baremetal-global-system-agent -o json | \
    jq -c '[
      .rules[]?
      | select(any(.apiGroups[]?; . == ""))
      | select(any(.resources[]?; . == "secrets"))
      | .resourceNames[]?
    ] | unique | sort'
  })"
  printf '%s\n' "${global_plan_names}" | jq -e 'length > 0' >/dev/null

  while IFS= read -r plan_secret; do
    test "$({
      kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
        get secret "${plan_secret}" -o jsonpath='{.type}'
    })" = "elemental.cattle.io/plan"
  done < <(printf '%s\n' "${global_plan_names}" | jq -r '.[]')

  expected_token_sha="$({
    kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
      get secret baremetal-global-system-agent-token \
      -o jsonpath='{.data.token}' | base64 -d | \
    sha256sum | awk '{print $1}'
  })"
  expected_ca_sha="$({
    kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
      get secret baremetal-global-system-agent-token \
      -o jsonpath='{.data.ca\.crt}' | base64 -d | \
    sha256sum | awk '{print $1}'
  })"

  for host in ${global_hosts}; do
    actual_endpoint="$({
      ssh "root@${host}" \
        'cat /var/lib/elemental/agent/elemental_connection.json' | \
      jq -r '.kubeConfig' | \
      awk '$1 == "server:" {print $2; exit}'
    })"
    test "${actual_endpoint}" = "${expected_endpoint}"

    actual_namespace="$({
      ssh "root@${host}" \
        'cat /var/lib/elemental/agent/elemental_connection.json' | \
      jq -r '.namespace'
    })"
    test "${actual_namespace}" = "cpaas-system"

    actual_plan_secret="$({
      ssh "root@${host}" \
        'cat /var/lib/elemental/agent/elemental_connection.json' | \
      jq -r '.secretName'
    })"
    printf '%s\n' "${global_plan_names}" | \
      jq -e --arg name "${actual_plan_secret}" 'index($name) != null' \
      >/dev/null

    actual_token_sha="$({
      ssh "root@${host}" \
        'cat /var/lib/elemental/agent/elemental_connection.json' | \
      jq -r '.kubeConfig' | \
      awk '$1 == "token:" {print $2; exit}' | \
      tr -d '\r\n' | sha256sum | awk '{print $1}'
    })"
    test "${actual_token_sha}" = "${expected_token_sha}"

    actual_ca_sha="$({
      ssh "root@${host}" \
        'cat /var/lib/elemental/agent/elemental_connection.json' | \
      jq -r '.kubeConfig' | \
      awk '$1 == "certificate-authority-data:" {print $2; exit}' | \
      base64 -d | sha256sum | awk '{print $1}'
    })"
    test "${actual_ca_sha}" = "${expected_ca_sha}"

    ssh "root@${host}" \
      "grep -Fq -- '${expected_endpoint}' /oem/elemental-system-agent.yaml"
    ssh "root@${host}" \
      'test "$(stat -c %a /oem/elemental-system-agent.yaml)" = 600'
    ssh "root@${host}" \
      'grep -Fqx -- "CATTLE_AGENT_STRICT_VERIFY=\"true\"" /etc/rancher/elemental/agent/envs'
    ssh "root@${host}" \
      'systemctl is-active --quiet elemental-system-agent.service'
  done
}

verify_baremetal_handoff \
  "${PRIMARY_BOOTSTRAP_KUBECONFIG}" "${PRIMARY_GLOBAL_KUBECONFIG}" \
  "${PRIMARY_CLUSTER_VIP}" \
  "${PRIMARY_GLOBAL_HOSTS}" "${PRIMARY_GLOBAL_REGISTRATION_NAME}"
verify_baremetal_handoff \
  "${STANDBY_BOOTSTRAP_KUBECONFIG}" "${STANDBY_GLOBAL_KUBECONFIG}" \
  "${STANDBY_CLUSTER_VIP}" \
  "${STANDBY_GLOBAL_HOSTS}" "${STANDBY_GLOBAL_REGISTRATION_NAME}"

Also verify the Global-local permission boundary on both sides. Before the first non-global workload plan exists, the shared baremetal-system-agent ServiceAccount, token Secret, Role, and RoleBinding may be absent on both clusters. Do not create or copy that bundle by hand merely to satisfy the Global installation gate; the active operator creates it when the first shared-scope registration is reconciled, and etcd-sync then copies it to standby.

verify_global_local_system_agent_rbac() {
  kubeconfig=$1

  kubectl --kubeconfig "${kubeconfig}" -n cpaas-system get \
    serviceaccount/baremetal-global-system-agent \
    secret/baremetal-global-system-agent-token \
    role/baremetal-global-system-agent \
    rolebinding/baremetal-global-system-agent >/dev/null

  local_plan_names="$({
    kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
      get role baremetal-global-system-agent -o json | \
    jq -c '[
      .rules[]?
      | select(any(.apiGroups[]?; . == ""))
      | select(any(.resources[]?; . == "secrets"))
      | .resourceNames[]?
    ] | unique | sort'
  })"
  printf '%s\n' "${local_plan_names}" | jq -e 'length > 0' >/dev/null

  expected_local_plan_names="$({
    kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
      get machineinventories.elemental.cattle.io -o json | \
    jq -c '[
      .items[]
      | select(
          .metadata.annotations["baremetal.cluster.io/system-agent-auth-scope"] ==
            "global"
        )
      | .status.plan.secretRef.name? // empty
    ] | unique | sort'
  })"
  test "${local_plan_names}" = "${expected_local_plan_names}"

  while IFS= read -r plan_secret; do
    test "$({
      kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
        get secret "${plan_secret}" -o jsonpath='{.type}'
    })" = "elemental.cattle.io/plan"

    test "$({
      kubectl --kubeconfig "${kubeconfig}" \
        --as=system:serviceaccount:cpaas-system:baremetal-global-system-agent \
        -n cpaas-system auth can-i get "secret/${plan_secret}"
    })" = yes
    test "$({
      kubectl --kubeconfig "${kubeconfig}" \
        --as=system:serviceaccount:cpaas-system:baremetal-global-system-agent \
        -n cpaas-system auth can-i patch "secret/${plan_secret}"
    })" = yes
    test "$({
      kubectl --kubeconfig "${kubeconfig}" \
        --as=system:serviceaccount:cpaas-system:baremetal-system-agent \
        -n cpaas-system auth can-i get "secret/${plan_secret}"
    })" = no
  done < <(printf '%s\n' "${local_plan_names}" | jq -r '.[]')

  for verb in list create delete; do
    test "$({
      kubectl --kubeconfig "${kubeconfig}" \
        --as=system:serviceaccount:cpaas-system:baremetal-global-system-agent \
        -n cpaas-system auth can-i "${verb}" secrets
    })" = no
  done

  test "$({
    kubectl --kubeconfig "${kubeconfig}" \
      --as=system:serviceaccount:cpaas-system:baremetal-global-system-agent \
      -n cpaas-system auth can-i get secret/global-registry-auth
  })" = no
}

verify_global_local_system_agent_rbac "${PRIMARY_GLOBAL_KUBECONFIG}"
verify_global_local_system_agent_rbac "${STANDBY_GLOBAL_KUBECONFIG}"

Do not reduce acceptance to the ready signal alone. A runtime token or CA mismatch, a stale bootstrap endpoint in the persistent OEM file, a failed plan probe, or an incorrect permission boundary means fresh handoff is incomplete even when data.ready is true.

After the standby installation succeeds, switch the platform domain back to the primary entrance before installing global-etcd-sync. Primary remains active during initial synchronization, and workload system-agent endpoints use the platform domain. Verify that DNS and https://<platform-domain>/kubernetes/global/version reach primary before continuing.

Before you create the global-etcd-sync ModuleInfo, build additional_ignore_equal exclusively from both sides' local Global plan Secret names. MR !179 already keeps the plugin's own ServiceMonitor/etcd-sync-monitor, the fixed Global-local objects, and public-registry-credential local, so do not duplicate those static keys in the dynamic list. Run these commands from a host that has jq and two kubeconfigs. Each kubeconfig must address its own Global API server; do not rely on the platform domain, because that domain points to only one side at a time. If one kubeconfig contains both clusters, use two distinct contexts in the equivalent kubectl --context commands.

set -euo pipefail

export PRIMARY_GLOBAL_KUBECONFIG="<path-to-primary-global-kubeconfig>"
export STANDBY_GLOBAL_KUBECONFIG="<path-to-standby-global-kubeconfig>"
export PRIMARY_GLOBAL_REGISTRATION_NAME="<primary-global-machine-registration-name>"
export STANDBY_GLOBAL_REGISTRATION_NAME="<standby-global-machine-registration-name>"
export PRIMARY_GLOBAL_CONTEXT="$(kubectl --kubeconfig "${PRIMARY_GLOBAL_KUBECONFIG}" config current-context)"
export STANDBY_GLOBAL_CONTEXT="$(kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" config current-context)"

PRIMARY_GLOBAL_SERVER="$(kubectl --kubeconfig "${PRIMARY_GLOBAL_KUBECONFIG}" \
  --context "${PRIMARY_GLOBAL_CONTEXT}" config view --minify \
  -o jsonpath='{.clusters[0].cluster.server}')"
STANDBY_GLOBAL_SERVER="$(kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \
  --context "${STANDBY_GLOBAL_CONTEXT}" config view --minify \
  -o jsonpath='{.clusters[0].cluster.server}')"

if [ "${PRIMARY_GLOBAL_SERVER}" = "${STANDBY_GLOBAL_SERVER}" ]; then
  echo "primary and standby kubeconfigs resolve to the same API server" >&2
  exit 1
fi

validate_global_auth_scope() {
  kubeconfig=$1
  context=$2
  registration_name=$3

  kubectl --kubeconfig "${kubeconfig}" --context "${context}" \
    -n cpaas-system get machineregistration.elemental.cattle.io \
    "${registration_name}" -o json | \
  jq -e '
    .metadata.annotations["baremetal.cluster.io/system-agent-auth-scope"] ==
      "global"
  ' >/dev/null

  global_inventory_names="$({
    kubectl --kubeconfig "${kubeconfig}" --context "${context}" \
      -n cpaas-system \
      get baremetalmachines.infrastructure.cluster.x-k8s.io -o json | \
    jq -c '[
      .items[]
      | select(
          .metadata.labels["cluster.x-k8s.io/cluster-name"] == "global"
        )
      | .status.machineInventoryRef.name? // empty
    ] | unique | sort'
  })"
  printf '%s\n' "${global_inventory_names}" | \
    jq -e 'length > 0' >/dev/null

  while IFS= read -r inventory_name; do
    kubectl --kubeconfig "${kubeconfig}" --context "${context}" \
      -n cpaas-system get machineinventory.elemental.cattle.io \
      "${inventory_name}" -o json | \
    jq -e '
      .metadata.annotations["baremetal.cluster.io/system-agent-auth-scope"] ==
        "global"
    ' >/dev/null
  done < <(printf '%s\n' "${global_inventory_names}" | jq -r '.[]')
}

validate_global_auth_scope \
  "${PRIMARY_GLOBAL_KUBECONFIG}" "${PRIMARY_GLOBAL_CONTEXT}" \
  "${PRIMARY_GLOBAL_REGISTRATION_NAME}"
validate_global_auth_scope \
  "${STANDBY_GLOBAL_KUBECONFIG}" "${STANDBY_GLOBAL_CONTEXT}" \
  "${STANDBY_GLOBAL_REGISTRATION_NAME}"

global_plan_secret_names() {
  kubeconfig=$1
  context=$2
  kubectl --kubeconfig "${kubeconfig}" --context "${context}" \
    -n cpaas-system get role baremetal-global-system-agent -o json | \
    jq -c '[
      .rules[]?
      | select(any(.apiGroups[]?; . == ""))
      | select(any(.resources[]?; . == "secrets"))
      | .resourceNames[]?
    ] | unique | sort'
}

PRIMARY_GLOBAL_PLAN_SECRET_NAMES="$(global_plan_secret_names \
  "${PRIMARY_GLOBAL_KUBECONFIG}" "${PRIMARY_GLOBAL_CONTEXT}")"
STANDBY_GLOBAL_PLAN_SECRET_NAMES="$(global_plan_secret_names \
  "${STANDBY_GLOBAL_KUBECONFIG}" "${STANDBY_GLOBAL_CONTEXT}")"

printf '%s\n' "${PRIMARY_GLOBAL_PLAN_SECRET_NAMES}" | \
  jq -e 'length > 0' >/dev/null || {
    echo "primary Role/baremetal-global-system-agent has no plan resourceNames" >&2
    exit 1
  }
printf '%s\n' "${STANDBY_GLOBAL_PLAN_SECRET_NAMES}" | \
  jq -e 'length > 0' >/dev/null || {
    echo "standby Role/baremetal-global-system-agent has no plan resourceNames" >&2
    exit 1
  }

export ETCD_SYNC_ADDITIONAL_IGNORE_EQUAL_JSON="$(
  jq -cn \
    --argjson primary "${PRIMARY_GLOBAL_PLAN_SECRET_NAMES}" \
    --argjson standby "${STANDBY_GLOBAL_PLAN_SECRET_NAMES}" \
    '(($primary + $standby)
      | unique
      | sort
      | map("/registry/secrets/cpaas-system/" + .))
     | unique
     | sort'
)"

printf '%s\n' "${ETCD_SYNC_ADDITIONAL_IGNORE_EQUAL_JSON}" | jq .

Do not install etcd-sync if either Global MachineRegistration or any Cluster/global MachineInventory lacks the explicit global scope, if either local Global Role has no plan resourceNames, or if the generated array is not exactly the de-duplicated union of both Roles converted to /registry/secrets/cpaas-system/<plan-secret>. An unannotated Elemental object remains eligible for generic synchronization, and the broad Secret sync prefix could overwrite a local Global plan Secret whose name happens to exist in the source or target.

Create a standby-local Secret that contains the primary cluster bearer token, then reference only its name from the plugin configuration. Do not put a token value in active_cluster_token for a fresh installation. The Secret must exist before the ModuleInfo is created because the chart's pre-install bootstrap Job mounts it.

set -euo pipefail
set +x
umask 077

export ETCD_SYNC_TOKEN_SECRET_NAME="etcd-sync-active-cluster-token"

PRIMARY_CLUSTER_TOKEN_B64="$({
  kubectl --kubeconfig "${PRIMARY_GLOBAL_KUBECONFIG}" \
    -n cpaas-system get secret k8sadmin -o jsonpath='{.data.token}'
})"
test -n "${PRIMARY_CLUSTER_TOKEN_B64}"

jq -cn \
  --arg name "${ETCD_SYNC_TOKEN_SECRET_NAME}" \
  --arg token "${PRIMARY_CLUSTER_TOKEN_B64}" '
  {
    apiVersion: "v1",
    kind: "Secret",
    metadata: {name: $name, namespace: "cpaas-system"},
    type: "Opaque",
    data: {token: $token}
  }
' | kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" apply -f -
unset PRIMARY_CLUSTER_TOKEN_B64

TOKEN_CHECK_FILE="$(mktemp)"
trap 'rm -f "${TOKEN_CHECK_FILE}"' EXIT
kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \
  -n cpaas-system get secret "${ETCD_SYNC_TOKEN_SECRET_NAME}" \
  -o jsonpath='{.data.token}' | base64 -d > "${TOKEN_CHECK_FILE}"
grep -Eq '^[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+$' \
  "${TOKEN_CHECK_FILE}"
rm -f "${TOKEN_CHECK_FILE}"
trap - EXIT

Get the standby k8sadmin token on a standby control-plane node. Use this decoded bearer token only to call the standby ModuleInfo API.

export STANDBY_CLUSTER_BEARER_TOKEN="$({
  kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \
    -n cpaas-system get secret k8sadmin -o jsonpath='{.data.token}' | \
  base64 -d
})"

Create the global-etcd-sync ModuleInfo payload for the standby cluster. active_cluster_vip points to primary, while active_cluster_token_secret_ref names the standby-local Secret prepared above.

umask 077
cat > "${ETCD_SYNC_MODULEINFO}" <<EOF
{
  "kind": "ModuleInfo",
  "apiVersion": "cluster.alauda.io/v1alpha1",
  "metadata": {
    "name": "global-etcd-sync",
    "labels": {
      "cpaas.io/cluster-name": "global",
      "cpaas.io/module-name": "etcd-sync",
      "cpaas.io/module-type": "plugin"
    }
  },
  "spec": {
    "version": "${ETCD_SYNC_VERSION}",
    "config": {
      "monitor_check_interval": 1,
      "detail": false,
      "active_cluster_vip": "${PRIMARY_CLUSTER_VIP}",
      "active_cluster_token_secret_ref": "${ETCD_SYNC_TOKEN_SECRET_NAME}",
      "additional_ignore_equal": ${ETCD_SYNC_ADDITIONAL_IGNORE_EQUAL_JSON}
    }
  }
}
EOF
chmod 0600 "${ETCD_SYNC_MODULEINFO}"

Install global-etcd-sync by calling the ModuleInfo API on the standby cluster. The platform can normalize the requested name to a generated global-<hash> name, so capture the actual object name from the response and use that name for every later patch or delete operation. The request and response both contain credentials; keep their files root-only and do not print them.

set -euo pipefail
set +x
umask 077

ETCD_SYNC_HEADERS="$(mktemp)"
ETCD_SYNC_RESPONSE="$(mktemp)"
trap 'rm -f "${ETCD_SYNC_HEADERS}" "${ETCD_SYNC_RESPONSE}"' EXIT

{
  printf 'Authorization: Bearer %s\n' "${STANDBY_CLUSTER_BEARER_TOKEN}"
  printf 'Content-Type: application/json\n'
} > "${ETCD_SYNC_HEADERS}"

HTTP_CODE="$({
  curl -sk -X POST \
    "https://${STANDBY_CLUSTER_VIP}/apis/cluster.alauda.io/v1alpha1/moduleinfoes" \
    -H @"${ETCD_SYNC_HEADERS}" \
    -d @"${ETCD_SYNC_MODULEINFO}" \
    -o "${ETCD_SYNC_RESPONSE}" \
    -w '%{http_code}'
})"

case "${HTTP_CODE}" in
  200|201) ;;
  *) echo "ModuleInfo API returned HTTP ${HTTP_CODE}" >&2; exit 1 ;;
esac

export ETCD_SYNC_MODULEINFO_NAME="$({
  jq -er '.metadata.name | select(type == "string" and length > 0)' \
    "${ETCD_SYNC_RESPONSE}"
})"
export ETCD_SYNC_MODULEINFO_NAME_FILE="/root/yamls/global-etcd-sync-moduleinfo.name"
printf '%s\n' "${ETCD_SYNC_MODULEINFO_NAME}" > \
  "${ETCD_SYNC_MODULEINFO_NAME_FILE}"
chmod 0600 "${ETCD_SYNC_MODULEINFO_NAME_FILE}"

rm -f "${ETCD_SYNC_HEADERS}" "${ETCD_SYNC_RESPONSE}"
trap - EXIT
rm -f "${ETCD_SYNC_MODULEINFO}"
unset STANDBY_CLUSTER_BEARER_TOKEN

Wait for the normalized ModuleInfo and its managed AppRelease/cpaas-system/etcd-sync. Do not declare the plugin installed until the AppRelease phase is Success, its installed revision matches ETCD_SYNC_VERSION, and both Deployments use an image built from etcd-sync MR !179 head commit 86c2ce5 or the explicitly verified equivalent merge commit.

deadline=$((SECONDS + 600))
until kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \
  get moduleinfoes.cluster.alauda.io "${ETCD_SYNC_MODULEINFO_NAME}" -o json | \
  jq -e --arg version "${ETCD_SYNC_VERSION}" \
    '.spec.version == $version' >/dev/null && \
  kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \
    -n cpaas-system get apprelease etcd-sync -o json | \
  jq -e --arg version "${ETCD_SYNC_VERSION}" '
    [.status.charts[]?]
    | any(.installedRevision == $version and .phase == "Success")
  ' >/dev/null; do
  if [ "${SECONDS}" -ge "${deadline}" ]; then
    echo "etcd-sync ModuleInfo/AppRelease did not reach Success" >&2
    exit 1
  fi
  sleep 10
done

export ETCD_SYNC_IMAGE_PROVENANCE_MARKER="86c2ce5"
for deployment in etcd-sync etcd-sync-monitor; do
  kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \
    -n cpaas-system get deployment "${deployment}" -o json | \
  jq -e --arg marker "${ETCD_SYNC_IMAGE_PROVENANCE_MARKER}" '
    all(.spec.template.spec.containers[]?; .image | contains($marker))
  ' >/dev/null
done

The marker is 86c2ce5 for the reviewed MR head. If the published image uses a merge-commit marker instead, set ETCD_SYNC_IMAGE_PROVENANCE_MARKER to that auditable marker only after recording that the merge contains 86c2ce500c059ff5226dd2f5ebb5b6e908faf06d; do not disable the provenance check.

Verify the Secret reference and the complete rendered filter before accepting the first synchronization. The dynamic list must contain only the two local Global Role plan-key union. The token Secret, Global-local bundle, local public-registry-credential, and plugin ServiceMonitor are static exact-ignore entries in MR !179 and must also be present. ConfigMap/baremetal-system-agent-handoff is already outside the configured synchronization ranges, so it remains local without an exact-ignore entry. The shared workload bundle must remain in the exact-sync list.

kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \
  get moduleinfoes.cluster.alauda.io "${ETCD_SYNC_MODULEINFO_NAME}" -o json | \
jq -e \
  --arg secret "${ETCD_SYNC_TOKEN_SECRET_NAME}" \
  --argjson expected_ignore "${ETCD_SYNC_ADDITIONAL_IGNORE_EQUAL_JSON}" '
  .spec.config.active_cluster_token_secret_ref == $secret and
  ((.spec.config.active_cluster_token // "") == "") and
  ((.spec.config.additional_ignore_equal // [] | sort) ==
    ($expected_ignore | sort))
' >/dev/null

kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \
  -n cpaas-system get apprelease etcd-sync -o json | \
jq -e \
  --arg secret "${ETCD_SYNC_TOKEN_SECRET_NAME}" \
  --argjson expected_ignore "${ETCD_SYNC_ADDITIONAL_IGNORE_EQUAL_JSON}" '
  .spec.values.etcd_sync.active_cluster_token_secret_ref == $secret and
  ((.spec.values.etcd_sync.active_cluster_token // "") == "") and
  ((.spec.values.etcd_sync.additional_ignore_equal // [] | sort) ==
    ($expected_ignore | sort))
' >/dev/null

kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \
  -n cpaas-system get deployment etcd-sync -o json | \
jq -e --arg secret "${ETCD_SYNC_TOKEN_SECRET_NAME}" '
  any(.spec.template.spec.volumes[]?;
    .name == "etcd-sync-token" and .secret.secretName == $secret)
' >/dev/null

if kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \
  -n cpaas-system get secret etcd-sync-token >/dev/null 2>&1; then
  echo "legacy Secret/etcd-sync-token must not be rendered for a fresh install" >&2
  exit 1
fi
if kubectl --kubeconfig "${PRIMARY_GLOBAL_KUBECONFIG}" \
  -n cpaas-system get secret "${ETCD_SYNC_TOKEN_SECRET_NAME}" \
  >/dev/null 2>&1; then
  echo "the etcd-sync active-cluster token Secret must remain standby-local" >&2
  exit 1
fi

IGNORE_EQUAL_TEXT="$({
  kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \
    -n cpaas-system get configmap etcd-sync-ignore-text \
    -o jsonpath='{.data.ignore-equal\.txt}'
})"
SYNC_EQUAL_TEXT="$({
  kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \
    -n cpaas-system get configmap etcd-sync-ignore-text \
    -o jsonpath='{.data.sync-equal-file\.txt}'
})"

printf '%s\n' "${ETCD_SYNC_ADDITIONAL_IGNORE_EQUAL_JSON}" | jq -r '.[]' | \
  while IFS= read -r key; do
    printf '%s\n' "${IGNORE_EQUAL_TEXT}" | grep -Fqx -- "${key}"
  done

for key in \
  "/registry/secrets/cpaas-system/${ETCD_SYNC_TOKEN_SECRET_NAME}" \
  /registry/serviceaccounts/cpaas-system/baremetal-global-system-agent \
  /registry/secrets/cpaas-system/baremetal-global-system-agent-token \
  /registry/roles/cpaas-system/baremetal-global-system-agent \
  /registry/rolebindings/cpaas-system/baremetal-global-system-agent \
  /registry/secrets/cpaas-system/public-registry-credential \
  /registry/monitoring.coreos.com/servicemonitors/cpaas-system/etcd-sync-monitor; do
  printf '%s\n' "${IGNORE_EQUAL_TEXT}" | grep -Fqx -- "${key}"
done

for key in \
  /registry/serviceaccounts/cpaas-system/baremetal-system-agent \
  /registry/secrets/cpaas-system/baremetal-system-agent-token \
  /registry/roles/cpaas-system/baremetal-system-agent \
  /registry/rolebindings/cpaas-system/baremetal-system-agent; do
  printf '%s\n' "${SYNC_EQUAL_TEXT}" | grep -Fqx -- "${key}"
done

for deployment in etcd-sync etcd-sync-monitor; do
  kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \
    -n cpaas-system get deployment "${deployment}" -o json | \
  jq -e '.spec.template.metadata.annotations["checksum/sync-key-config"] | length > 0' \
    >/dev/null
  kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \
    -n cpaas-system rollout status "deployment/${deployment}" --timeout=5m
done

The first installation renders the complete filter before the Pods start, so no manual rollout restart is required. The checksum annotation added by MR !179 triggers both Deployment rollouts if the rendered filter changes later.

Trigger an on-demand monitor comparison and require zero missed and zero surplus keys. Also require the filtered local and remote target-key counts to match. The /check endpoint returns HTTP 425 when the preceding check ran less than one minute ago; retry without treating that response as success.

set -euo pipefail
umask 077

PORT_FORWARD_LOG="$(mktemp)"
CHECK_RESPONSE="$(mktemp)"
kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" \
  -n cpaas-system port-forward service/etcd-sync-monitor 18080:80 \
  >"${PORT_FORWARD_LOG}" 2>&1 &
PORT_FORWARD_PID=$!
trap 'kill "${PORT_FORWARD_PID}" 2>/dev/null || true; rm -f "${PORT_FORWARD_LOG}" "${CHECK_RESPONSE}"' EXIT

until curl -fsS http://127.0.0.1:18080/healthz >/dev/null; do
  kill -0 "${PORT_FORWARD_PID}"
  sleep 1
done

while true; do
  HTTP_CODE="$({
    curl -sS -H 'Accept: application/json' \
      -o "${CHECK_RESPONSE}" -w '%{http_code}' \
      http://127.0.0.1:18080/check
  })"
  case "${HTTP_CODE}" in
    200) break ;;
    425) sleep 10 ;;
    *) echo "etcd-sync monitor check returned HTTP ${HTTP_CODE}" >&2; exit 1 ;;
  esac
done

jq -e '
  (.localMissedKeys | length) == 0 and
  (.localSurplusKeys | length) == 0
' "${CHECK_RESPONSE}" >/dev/null

METRICS="$(curl -fsS http://127.0.0.1:18080/metrics)"
LOCAL_TARGET_COUNT="$({
  printf '%s\n' "${METRICS}" | \
    awk '$1 ~ /^local_target_keys_count/ {value=$2} END {print value}'
})"
REMOTE_TARGET_COUNT="$({
  printf '%s\n' "${METRICS}" | \
    awk '$1 ~ /^remote_target_keys_count/ {value=$2} END {print value}'
})"
test -n "${LOCAL_TARGET_COUNT}"
test "${LOCAL_TARGET_COUNT}" = "${REMOTE_TARGET_COUNT}"

kill "${PORT_FORWARD_PID}"
wait "${PORT_FORWARD_PID}" 2>/dev/null || true
rm -f "${PORT_FORWARD_LOG}" "${CHECK_RESPONSE}"
trap - EXIT

After the first successful synchronization, evaluate the shared workload bundle as one atomic set. Before the first shared-scope workload plan exists, all four objects may be absent on both sides. Partial presence, or presence on only one side, is a failure. Once the bundle exists, validate the synchronized ServiceAccount UID and the token Secret's ServiceAccount UID annotation explicitly; a content comparison that discards the ServiceAccount UID is not sufficient to prove that the token authenticates on standby.

shared_bundle_count() {
  kubeconfig=$1
  count=0
  for resource in \
    serviceaccount/baremetal-system-agent \
    secret/baremetal-system-agent-token \
    role/baremetal-system-agent \
    rolebinding/baremetal-system-agent; do
    if kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
      get "${resource}" >/dev/null 2>&1; then
      count=$((count + 1))
    fi
  done
  printf '%s\n' "${count}"
}

shared_bundle_digest() {
  kubeconfig=$1
  resource=$2

  kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
    get "${resource}" -o json | \
  jq -S '
    del(
      .metadata.creationTimestamp,
      .metadata.generation,
      .metadata.managedFields,
      .metadata.resourceVersion,
      .metadata.uid
    )
  ' | sha256sum | awk '{print $1}'
}

validate_shared_bundle_cluster() {
  kubeconfig=$1

  sa_uid="$({
    kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
      get serviceaccount baremetal-system-agent -o jsonpath='{.metadata.uid}'
  })"
  test -n "${sa_uid}"

  kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
    get secret baremetal-system-agent-token -o json | \
  jq -e --arg uid "${sa_uid}" '
    .type == "kubernetes.io/service-account-token" and
    .metadata.annotations["kubernetes.io/service-account.name"] ==
      "baremetal-system-agent" and
    .metadata.annotations["kubernetes.io/service-account.uid"] == $uid and
    (.data.token | type == "string" and length > 0)
  ' >/dev/null

  kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
    get rolebinding baremetal-system-agent -o json | \
  jq -e '
    .roleRef == {
      apiGroup: "rbac.authorization.k8s.io",
      kind: "Role",
      name: "baremetal-system-agent"
    } and
    (.subjects | length) == 1 and
    .subjects[0] == {
      kind: "ServiceAccount",
      name: "baremetal-system-agent",
      namespace: "cpaas-system"
    }
  ' >/dev/null

  expected_shared_plans="$({
    kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
      get machineinventories.elemental.cattle.io -o json | \
    jq -c '[
      .items[]
      | select(
          (.metadata.annotations["baremetal.cluster.io/system-agent-auth-scope"] // "") ==
          "shared"
        )
      | .status.plan.secretRef.name? // empty
    ] | unique | sort'
  })"
  actual_shared_plans="$({
    kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \
      get role baremetal-system-agent -o json | \
    jq -c '[
      .rules[]?
      | select(any(.apiGroups[]?; . == ""))
      | select(any(.resources[]?; . == "secrets"))
      | .resourceNames[]?
    ] | unique | sort'
  })"
  test "${actual_shared_plans}" = "${expected_shared_plans}"
}

PRIMARY_SHARED_COUNT="$(shared_bundle_count "${PRIMARY_GLOBAL_KUBECONFIG}")"
STANDBY_SHARED_COUNT="$(shared_bundle_count "${STANDBY_GLOBAL_KUBECONFIG}")"

if [ "${PRIMARY_SHARED_COUNT}" = 0 ] && [ "${STANDBY_SHARED_COUNT}" = 0 ]; then
  echo "shared workload bundle is deferred until the first shared-scope workload plan"
elif [ "${PRIMARY_SHARED_COUNT}" = 4 ] && [ "${STANDBY_SHARED_COUNT}" = 4 ]; then
  validate_shared_bundle_cluster "${PRIMARY_GLOBAL_KUBECONFIG}"
  validate_shared_bundle_cluster "${STANDBY_GLOBAL_KUBECONFIG}"

  PRIMARY_SHARED_SA_UID="$({
    kubectl --kubeconfig "${PRIMARY_GLOBAL_KUBECONFIG}" -n cpaas-system \
      get serviceaccount baremetal-system-agent -o jsonpath='{.metadata.uid}'
  })"
  STANDBY_SHARED_SA_UID="$({
    kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" -n cpaas-system \
      get serviceaccount baremetal-system-agent -o jsonpath='{.metadata.uid}'
  })"
  test "${PRIMARY_SHARED_SA_UID}" = "${STANDBY_SHARED_SA_UID}"

  for resource in \
    serviceaccount/baremetal-system-agent \
    secret/baremetal-system-agent-token \
    role/baremetal-system-agent \
    rolebinding/baremetal-system-agent; do
    PRIMARY_DIGEST="$(shared_bundle_digest \
      "${PRIMARY_GLOBAL_KUBECONFIG}" "${resource}")"
    STANDBY_DIGEST="$(shared_bundle_digest \
      "${STANDBY_GLOBAL_KUBECONFIG}" "${resource}")"
    test "${PRIMARY_DIGEST}" = "${STANDBY_DIGEST}"
  done
else
  echo "shared workload bundle is only partially synchronized" >&2
  exit 1
fi

PRIMARY_LOCAL_TOKEN_SHA="$({
  kubectl --kubeconfig "${PRIMARY_GLOBAL_KUBECONFIG}" -n cpaas-system \
    get secret baremetal-global-system-agent-token \
    -o jsonpath='{.data.token}' | base64 -d | sha256sum | awk '{print $1}'
})"
STANDBY_LOCAL_TOKEN_SHA="$({
  kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" -n cpaas-system \
    get secret baremetal-global-system-agent-token \
    -o jsonpath='{.data.token}' | base64 -d | sha256sum | awk '{print $1}'
})"
test "${PRIMARY_LOCAL_TOKEN_SHA}" != "${STANDBY_LOCAL_TOKEN_SHA}"

If the shared bundle was absent, repeat this entire block after creating the first non-global workload plan. Fresh-install acceptance is complete only after the deferred check passes when shared workload provisioning is first exercised.

Restart the Pods that must reload DR and endpoint configuration. Run the same commands on a primary control plane node and on a standby control plane node.

sudo kubectl delete po -n cpaas-system -l 'service_name in (alertmanager,vmselect,vminsert)'
sudo kubectl delete po -n cpaas-system -l service_name=cpaas-elasticsearch
sudo kubectl delete po -n cpaas-system -l service_name=cluster-transformer

For the DR lifecycle after installation, see Global Cluster Disaster Recovery.

Verification

After the installer reports completion, verify that the global cluster is healthy.

kubectl --kubeconfig <global-kubeconfig> get nodes
kubectl --kubeconfig <global-kubeconfig> get clusters.platform.tkestack.io global \
  -o jsonpath='{.status.phase}'
kubectl --kubeconfig <global-kubeconfig> get pods -n cpaas-system
kubectl --kubeconfig <global-kubeconfig> get clustermodule global

The installation is successful when all of the following conditions are true:

  • The installer progress API reports status: Success and type: Complete.
  • All global cluster nodes are Ready.
  • Critical Pods in cpaas-system are Running or Completed.
  • ClusterModule/global reports the base module as healthy.

Decommission the Bootstrap Cluster

After the installer reports success, the global cluster runs its own Cluster API providers and no longer depends on the temporary bootstrap cluster (minialauda). Once verification passes, remove it from the bootstrap host — delete only the local bootstrap cluster (minialauda) and its KIND container network.

Verify the DCS credential Secret reached the global cluster

The DCS API credential Secret is copied to the global cluster during installation by the dcs-import-extra-resources ConfigMap you create in Step 7 — it imports the Secret named in your DCSCluster.spec.credentialSecretRef. Before you remove the bootstrap host, verify it is present: kubectl --kubeconfig <global-kubeconfig> get secret <name> -n cpaas-system. If it is missing, copy it over first — without it the global cluster's DCS provider has no DCS API credentials and cannot reconcile (for example, scale-out later fails).

Do not delete the Cluster API objects to clean up

Do not run kubectl delete cluster global, and do not delete the Cluster, KubeadmControlPlane, or provider infrastructure objects as a cleanup step. After installation these objects own the live global control plane machines, so deleting them cascades into deleting the control plane VMs and destroys the cluster you just installed. Decommissioning is limited to removing the local bootstrap cluster (its KIND container) on the bootstrap host; leave the Cluster API objects in place.

Next Steps

Worked Example: Complete global Manifest for Huawei DCS

This is a complete, single-file manifest for a three-replica control-plane global cluster on Huawei DCS. It is the same set of resources described in Step 4, already assembled so you do not have to merge fragments across pages. It uses documentation-only example values: replace every <placeholder>, and reuse the ${...} variables you exported in Step 1. Apply it in Step 5.

This example targets a non-DR cluster. To avoid maintaining two copies of it, the KubeadmControlPlane kubeadmConfigSpec body is not repeated here — it is identical to a workload cluster and is taken from the Complete KubeadmControlPlane Configuration appendix, with the two global / non-DR deltas noted inline in resource 4 below.

Before You Apply: Prepare These on DCS

The manifest references these but does not create them. Prepare them first, in order:

  1. DCS API access — the endpoint (https://<host>:7443), an administrator user and password, and the site ID. These fill the Secret and DCSCluster.spec.site. See Cloud Credentials.
  2. A VM template — upload the Alauda OS image and create a VM template from it; record its name for vmTemplateName. Use a 4.2.1+ template so the /var/cpaas persistent disk can be detached and reattached during node replacement. See Machine Templates.
  3. Compute cluster, distributed virtual switch, port group, and datastore — pick the target DCS compute cluster, a distributed virtual switch, a port group, and a datastore with enough free capacity. They fill resource, dvSwitchName, portGroupName, and datastoreName. See DCS Platform Capacity and Placement.
  4. IPs and the API load balancer — three free node IPs (with gateway and DNS) for the control plane, and a control-plane VIP / load balancer that balances at least port 6443 across the three nodes (the DCS provider does not create it). These fill the DCSIpHostnamePool and DCSCluster.spec.controlPlaneLoadBalancer / controlPlaneEndpoint. See Creating Clusters on Huawei DCS.
  5. Versions and IDs to read — ${K8S_VERSION} plus the CoreDNS and etcd image tags from the cpaas.io/dcs-vm-template ConfigMap (see Resolving Placeholder Values); the kube-ovn chart version from the OS Support Matrix; and the registry address, VIP, and CIDR values you exported in Step 1.

With those prepared, apply the manifest below.

---
# 1. DCS API credential. See Infrastructure Resources for the field sources.
apiVersion: v1
kind: Secret
metadata:
  name: <auth-secret-name>   # imported to the global cluster via the dcs-import-extra-resources ConfigMap in Step 7
  namespace: cpaas-system
  labels:
    cpaas.io/cluster-name: "global"
type: Opaque
stringData:
  authUser: "<dcs-api-user>"
  authKey: "<dcs-api-password>"
  endpoint: "https://<dcs-api-host>:7443"
  # userType: "interconnect"   # optional; "interconnect" (default) or "domain"
---
# 2. Control-plane IP / hostname pool (one entry per control-plane replica).
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DCSIpHostnamePool
metadata:
  name: global-cp-pool
  namespace: cpaas-system
  labels:
    cpaas.io/cluster-name: "global"
spec:
  pool:
  # /var/cpaas holds platform state and must survive node replacement, so it is
  # declared here as a persistentDisk bound to the IP slot (not as a
  # DCSMachineTemplate disk). Requires a DCS VM template 4.2.1+ and maxSurge: 0.
  - ip: "192.0.2.11"
    mask: "24"
    gateway: "192.0.2.1"
    dns: "192.0.2.2"
    hostname: "global-cp-1"
    machineName: "global-cp-1"
    persistentDisk:
    - {slot: 0, quantityGB: 100, datastoreName: <datastore-name>, path: /var/cpaas, format: xfs, mountOptions: [defaults]}
  - ip: "192.0.2.12"
    mask: "24"
    gateway: "192.0.2.1"
    dns: "192.0.2.2"
    hostname: "global-cp-2"
    machineName: "global-cp-2"
    persistentDisk:
    - {slot: 0, quantityGB: 100, datastoreName: <datastore-name>, path: /var/cpaas, format: xfs, mountOptions: [defaults]}
  - ip: "192.0.2.13"
    mask: "24"
    gateway: "192.0.2.1"
    dns: "192.0.2.2"
    hostname: "global-cp-3"
    machineName: "global-cp-3"
    persistentDisk:
    - {slot: 0, quantityGB: 100, datastoreName: <datastore-name>, path: /var/cpaas, format: xfs, mountOptions: [defaults]}
---
# 3. Control-plane VM spec.
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DCSMachineTemplate
metadata:
  name: global-cp-template
  namespace: cpaas-system
  labels:
    cpaas.io/cluster-name: "global"
spec:
  template:
    spec:
      vmTemplateName: <vm-template-name>
      # Places the cloned VMs in a DCS compute cluster.
      resource:
        type: cluster
        name: <dcs-cluster-name>
      vmConfig:
        dvSwitchName: <dvswitch-name>
        portGroupName: <port-group-name>
        dcsMachineCpuSpec: {quantity: 16}
        dcsMachineMemorySpec: {quantity: 32768}   # MB
        dcsMachineDiskSpec:
        - {quantity: 0,   datastoreName: <datastore-name>, systemVolume: true}
        - {quantity: 10,  datastoreName: <datastore-name>, path: /var/lib/etcd,       format: xfs}
        - {quantity: 100, datastoreName: <datastore-name>, path: /var/lib/kubelet,    format: xfs}
        - {quantity: 100, datastoreName: <datastore-name>, path: /var/lib/containerd, format: xfs}
        # /var/cpaas is intentionally NOT a template disk — it is declared as a
        # persistentDisk on the IP pool above so it survives node replacement.
      ipHostPoolRef:
        name: global-cp-pool
---
# 4. Control plane.
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
  name: global-kcp
  namespace: cpaas-system
  labels:
    cpaas.io/cluster-name: "global"
  annotations:
    controlplane.cluster.x-k8s.io/skip-kube-proxy: ""
spec:
  replicas: 3
  version: ${K8S_VERSION}
  rolloutStrategy:
    type: RollingUpdate
    rollingUpdate: {maxSurge: 0}
  machineTemplate:
    nodeDrainTimeout: 1m
    nodeDeletionTimeout: 5m
    infrastructureRef:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
      kind: DCSMachineTemplate
      name: global-cp-template
  kubeadmConfigSpec:
    format: ignition
    users:
    - name: boot
      sshAuthorizedKeys:
      - "ssh-ed25519 AAAA...replace-with-your-public-key... global-boot"
    # The rest of kubeadmConfigSpec — files, clusterConfiguration,
    # preKubeadmCommands, postKubeadmCommands, initConfiguration,
    # joinConfiguration — is identical to a workload cluster, so it is not
    # duplicated here. Take it verbatim from the Complete KubeadmControlPlane
    # Configuration appendix in the DCS create-cluster guide
    # (#complete-kubeadmcontrolplane-configuration). The three large files
    # (psa-config.yaml, control-plane-kubelet-patch.json, audit-policy.yaml) may
    # use contentFrom the dcs-kubernetes-<major.minor>-files Secret. Apply these
    # global / non-DR deltas to that body:
    #   1. Add clusterConfiguration.etcd.local.serverCertSANs:
    #        ["${CONTROL_PLANE_VIP}", "etcd.kube-system"]
    #   2. Non-DR: omit the /etc/kubernetes/encryption-provider.conf file AND the
    #        apiServer.extraArgs.encryption-provider-config argument (keep both for
    #        DR / at-rest encryption — see the note after this example).
---
# 5. DCS infrastructure cluster.
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DCSCluster
metadata:
  name: "global"
  namespace: cpaas-system
  labels:
    cpaas.io/cluster-name: "global"
  annotations:
    cpaas.io/registry-address: "${NODE_REGISTRY_ADDRESS}"
spec:
  controlPlaneLoadBalancer: {host: "${CONTROL_PLANE_VIP}", port: 6443, type: external}
  controlPlaneEndpoint: {host: "${CONTROL_PLANE_VIP}", port: 6443}
  credentialSecretRef: {name: <auth-secret-name>}
  networkType: kube-ovn
  site: <dcs-site-id>
---
# 6. Top-level CAPI Cluster: global wiring, labels, and annotations.
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: global
  namespace: cpaas-system
  labels:
    cpaas.io/cluster-name: "global"
    cluster-type: DCS
    is-global: "true"
  annotations:
    capi.cpaas.io/resource-group-version: infrastructure.cluster.x-k8s.io/v1beta1
    capi.cpaas.io/resource-kind: DCSCluster
    capi.cpaas.io/kubernetes: ${K8S_VERSION}   # same value as KubeadmControlPlane.spec.version
    cpaas.io/registry-address: "${NODE_REGISTRY_ADDRESS}"
    cpaas.io/nodes-mode: self-managed   # node lifecycle managed by CAPI + the DCS provider
    cpaas.io/kube-ovn-join-cidr: <kube-ovn-join-cidr>   # a /16 you choose; must not overlap the pod / service CIDRs or another cluster's join CIDR
    cpaas.io/kube-ovn-version: <kube-ovn-chart-version>
    cpaas.io/os-family: <os-family>   # OS family of the Alauda OS image, for example slemicro
spec:
  clusterNetwork:
    pods:     {cidrBlocks: ["${CLUSTER_CIDR}"]}
    services: {cidrBlocks: ["${SERVICE_CIDR}"]}
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: KubeadmControlPlane
    name: global-kcp
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: DCSCluster
    name: global

Values to Replace

Placeholder / variableWhere it comes from
${K8S_VERSION}, ${CONTROL_PLANE_VIP}, ${NODE_REGISTRY_ADDRESS}, ${CLUSTER_CIDR}, ${SERVICE_CIDR}Exported in Step 1.
<dcs-api-user> / <dcs-api-password> / <dcs-api-host> / <dcs-site-id>DCS platform credentials and site. See Cloud Credentials.
<vm-template-name>, <dns-image-tag>, <etcd-image-tag>The cpaas.io/dcs-vm-template ConfigMap. See Resolving Placeholder Values.
<dcs-cluster-name>, <dvswitch-name>, <port-group-name>, <datastore-name>DCS platform objects. Confirm with the DCS administrator.
192.0.2.x, <kube-ovn-join-cidr>, <os-family>, <kube-ovn-chart-version>Node IPs/gateway/DNS for your network; a /16 Kube-OVN join CIDR that does not overlap the pod / service CIDRs or another cluster's join CIDR; the OS family of the image; and the kube-ovn chart version from the OS Support Matrix.
ssh-ed25519 AAAA...A real OpenSSH public key. The ignition format rejects an empty list; supply any valid key even if you do not plan to SSH in.
Secret encryption and disaster recovery

This example does not enable etcd secret encryption-at-rest. To enable it, or to deploy a DR pair, add the /etc/kubernetes/encryption-provider.conf file and the apiServer.extraArgs.encryption-provider-config argument as described in Optional Disaster Recovery Deployment.