Skip to content

e2e-multicluster.md

Latest commit

 

History

History
225 lines (175 loc) · 7.42 KB

e2e-multicluster.md

File metadata and controls

225 lines (175 loc) · 7.42 KB
Raw

Running Multi-Cluster E2E Tests Locally

Run the multi-cluster (hub + spoke) E2E suite locally using kind. The automated wrapper is test/e2e-tests-multicluster.sh; the steps below are the manual equivalents.

Prerequisites

  • kind (>= v0.31.0), kubectl, ko, docker, envsubst (from gettext).
  • A working $KO_DOCKER_REPO (the script defaults to kind.local).
  • Go toolchain matching go.mod.
  • Host architecture: arm64 and amd64 are both supported.

All commands below are relative to the repository root.

1. Start hub and spoke kind clusters

export KIND_CLUSTER_NAME=kind
export SPOKE_CLUSTER_NAME=spoke
export SPOKE_KUBECONFIG=/tmp/spoke.kubeconfig
export SPOKE_HOST_KUBECONFIG=/tmp/spoke-host.kubeconfig

kind create cluster --name "${KIND_CLUSTER_NAME}" --wait 120s
kind create cluster --name "${SPOKE_CLUSTER_NAME}" \
  --kubeconfig "${SPOKE_HOST_KUBECONFIG}" --wait 120s

# Internal kubeconfig reachable from the hub operator pod via the docker bridge.
kind get kubeconfig --internal --name "${SPOKE_CLUSTER_NAME}" > "${SPOKE_KUBECONFIG}"
kind get kubeconfig          --name "${SPOKE_CLUSTER_NAME}" > "${SPOKE_HOST_KUBECONFIG}"

Both clusters should come up healthy:

kubectl                             get nodes
KUBECONFIG="${SPOKE_HOST_KUBECONFIG}" kubectl get nodes

2. Install the ClusterProfile CRD on the hub

: "${CLUSTER_INVENTORY_CRD_URL:=https://raw.githubusercontent.com/kubernetes-sigs/cluster-inventory-api/v0.1.3/config/crd/bases/multicluster.x-k8s.io_clusterprofiles.yaml}"
kubectl apply -f "${CLUSTER_INVENTORY_CRD_URL}"
kubectl wait --for=condition=Established --timeout=60s \
  crd/clusterprofiles.multicluster.x-k8s.io

3. Deploy the operator on the hub and wire up access provider config

Apply the operator from source:

ko apply -Rf config/

Generate a spoke bootstrap token, store it as knative-operator/${SPOKE_CLUSTER_NAME}, and mount the official secretreader image as the access provider. The helper script does this end to end:

source test/e2e-common.sh
install_access_provider_config   # installs the secretreader access provider
apply_cluster_profile default    # creates the ClusterProfile on the hub

A minimal provider config (written by the helper to /etc/cluster-inventory/config.json inside the operator pod) looks like:

{
  "providers": [
    {
      "name": "secretreader",
      "execConfig": {
        "apiVersion": "client.authentication.k8s.io/v1",
        "command": "/etc/cluster-inventory/plugin/bin/secretreader-plugin",
        "provideClusterInfo": true,
        "interactiveMode": "Never"
      }
    }
  ]
}

Point the operator at the config via the CLI flag. The helper patches the operator deployment; if you prefer to manage it yourself:

kubectl -n knative-operator patch deployment knative-operator \
  --type json \
  -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--clusterprofile-provider-file=/etc/cluster-inventory/config.json"}]'

(Optional) Tune the remote deployments poll interval for a large fleet simulation:

kubectl -n knative-operator patch deployment knative-operator \
  --type json \
  -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--remote-deployments-poll-interval=30s"}]'

4. Apply a hub CR targeting the spoke

Create the manifest in a temp file (the repository does not ship a hack/manual/ directory) and apply it:

kubectl create ns knative-serving
cat > /tmp/knativeserving-spoke.yaml <<'EOF'
apiVersion: operator.knative.dev/v1beta1
kind: KnativeServing
metadata:
  name: knative-serving
  namespace: knative-serving
spec:
  clusterProfileRef:
    name: spoke
    namespace: default
EOF
kubectl apply -f /tmp/knativeserving-spoke.yaml

Same pattern for Eventing:

kubectl create ns knative-eventing
cat > /tmp/knativeeventing-spoke.yaml <<'EOF'
apiVersion: operator.knative.dev/v1beta1
kind: KnativeEventing
metadata:
  name: knative-eventing
  namespace: knative-eventing
spec:
  clusterProfileRef:
    name: spoke
    namespace: default
EOF
kubectl apply -f /tmp/knativeeventing-spoke.yaml

Watch both status conditions and the spoke anchor:

kubectl get knativeserving -A -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.conditions[?(@.type=="TargetClusterResolved")].status}{"\t"}{.status.conditions[?(@.type=="InstallSucceeded")].status}{"\n"}{end}'

KUBECONFIG="${SPOKE_HOST_KUBECONFIG}" kubectl get cm -A \
  -l operator.knative.dev/cr-name=knative-serving

KUBECONFIG="${SPOKE_HOST_KUBECONFIG}" kubectl -n knative-serving rollout status deploy/activator

5. Tear down and verify

Delete the hub CRs in reverse order; the operator's finalizer cleans the spoke:

kubectl delete knativeeventing -n knative-eventing knative-eventing
kubectl delete knativeserving  -n knative-serving  knative-serving

# Anchors should be gone.
KUBECONFIG="${SPOKE_HOST_KUBECONFIG}" kubectl get cm -A \
  -l operator.knative.dev/cr-name 2>/dev/null

6. Running the Go E2E suite

The test package is gated by two build tags:

go test -v -tags 'e2e multicluster' -count=1 \
  -run '^TestMulticluster' ./test/e2e

Both tags are required: e2e enables the shared e2e bootstrap and multicluster enables only the spoke tests. Use a single space-separated string as shown above; go test does not accept comma-separated tag values or multiple -tags flags.

Environment variables read by the suite:

Variable Purpose Default
SPOKE_CLUSTER_NAME ClusterProfile.metadata.name used by the tests spoke
SPOKE_CLUSTER_NAMESPACE ClusterProfile.metadata.namespace default
KUBECONFIG Hub kubeconfig (standard Go client discovery) current context
SPOKE_HOST_KUBECONFIG Spoke kubeconfig from the host's perspective (required)

If you want the end-to-end bootstrap in a single step, use the wrapper script, which reuses the same helpers this guide calls manually:

./test/e2e-tests-multicluster.sh

7. Debugging tips

  • Hub operator logs: kubectl -n knative-operator logs deploy/knative-operator. Look for Remote deployments poll interval: after the first reconcile of a KnativeServing/KnativeEventing CR to confirm the flag value, and cluster provider closed during shutdown on pod restarts (see ClusterProviderClosed in multicluster.md).
  • Spoke state dump: KUBECONFIG="${SPOKE_HOST_KUBECONFIG}" kubectl get events -A --sort-by=.lastTimestamp.
  • If the TargetClusterResolved condition stays False with reason AccessProviderFailed, exec into the operator and run the plugin manually: 
    kubectl -n knative-operator exec deploy/knative-operator -- \
  env KUBERNETES_EXEC_INFO="{\"apiVersion\":\"client.authentication.k8s.io/v1\",\"kind\":\"ExecCredential\",\"spec\":{\"cluster\":{\"server\":\"https://debug.invalid\",\"config\":{\"clusterName\":\"${SPOKE_CLUSTER_NAME}\"}}}}" \
  /etc/cluster-inventory/plugin/bin/secretreader-plugin

8. Cleanup

kind delete cluster --name "${SPOKE_CLUSTER_NAME}"
kind delete cluster --name "${KIND_CLUSTER_NAME}"
rm -f "${SPOKE_KUBECONFIG}" "${SPOKE_HOST_KUBECONFIG}"

Why this matters before knative/infra#827

Until knative/infra#827 lands the Prow job, local kind is the only CI-like environment for exercising multi-cluster changes.