Cross-cluster access
In one line
Section titled “In one line”Reaching a service in another cluster is three separate problems: a network path for the packets, a way to find and expose the service, and an identity the server can verify. Pick one option per layer; they are mostly independent.
Why it exists
Section titled “Why it exists”A Kubernetes cluster is a closed world by design. Pod IPs, Service ClusterIPs and *.svc.cluster.local names mean something only inside it. That was fine while one cluster was the whole platform.
Then clusters multiplied: one per region for latency and failure isolation, one per environment, one per team or tenant, one per account for blast radius. A workload in region A now needs an API that lives in region B, and none of the in-cluster primitives reach across.
The fixes arrived layer by layer:
- Network. Cloud providers added VPC peering, then hub routers (AWS Transit Gateway, 2018), then service-level private connectivity (AWS PrivateLink, VPC Lattice) for cases where joining whole networks is too much.
- Discovery. At first everyone published an internal load balancer and a DNS record. Meshes and CNIs then added their own cross-cluster service models. Kubernetes SIG Multicluster wrote a vendor-neutral one, the Multi-Cluster Services API (KEP-1645), built on the idea of namespace sameness: a namespace with the same name in two clusters of a set is the same namespace.
- Identity. Once traffic crosses a network boundary, “it came from inside the cluster” stops being an argument. SPIFFE gave workloads cryptographic names that are valid across clusters, and cloud IAM gave a second option where the cloud is the common authority.
How it works
Section titled “How it works”Example: orders runs in EKS cluster A in eu-west-1 and must call inventory in cluster B in us-east-1.
Layer 1: network path
Section titled “Layer 1: network path”Something must carry packets from a pod in A to an address in B.
- Decide what must be routable. Pod to load balancer is enough for most designs: only VPC B’s load balancer subnets must be reachable. Pod to pod (flat networking) needs every pod CIDR routable both ways and non-overlapping. With the AWS VPC CNI, pods have real VPC addresses, so VPC routing already covers them.
- Pick the connection. Peering for two VPCs, a transit gateway when there are many, PrivateLink or VPC Lattice when you want to expose one service and not join networks, a public endpoint when no private path exists.
- Routes. Peering and transit gateway peering both need routes added by hand: a route in each VPC route table to the other CIDR, and for transit gateway peering a static route in the transit gateway route table (peering attachments do not propagate routes).
- Filters. Security groups on the target (load balancer, nodes or pods) must allow the source CIDR. Across regions you cannot reference a security group in the peer VPC, only CIDRs.
- MTU. Inter-region peering carries 8500 bytes, not the 9001 of same-region peering. Overlays and tunnels take more.
Layer 2: discovery and exposure
Section titled “Layer 2: discovery and exposure”The client needs a stable name, and cluster B must publish the service on something reachable.
- Expose in B. A
Serviceof typeLoadBalancerwith an internal Network Load Balancer, an internal Application Load Balancer through Ingress or Gateway API, or a mesh gateway. - Name it. A DNS record that cluster A can resolve: a Route 53 private hosted zone associated with both VPCs, often written by ExternalDNS. Or let a multi-cluster controller create the name inside cluster A.
- Multi-Cluster Services API. In B you create a
ServiceExportnamed like the Service. The controller creates aServiceImportin every cluster of the ClusterSet that has the same namespace. Clients useinventory.shop.svc.clusterset.local. Thecluster.localname keeps meaning “local only”, so going multi-cluster is an explicit choice by the client. - Mesh or CNI models do the same with their own objects: a Cilium global service is the same Service in both clusters with the annotation
service.cilium.io/global: "true"; Istio merges endpoints of same-named services across clusters; Linkerd mirrorsinventoryfrom clustereastasinventory-east.
Layer 3: identity and authorisation
Section titled “Layer 3: identity and authorisation”The network path says packets can arrive. It does not say who sent them.
- Transport identity (mTLS). Both sides present certificates. Across clusters this works only if each side trusts the other’s issuer: either both clusters’ certificate authorities chain to one root, or the two trust domains exchange trust bundles (SPIFFE federation). See mTLS and service mesh.
- Request identity (tokens or signatures). The client attaches proof to each request: an AWS Signature Version 4 signature made with its IAM role credentials, or a JWT such as a projected service account token with a custom audience. The server (or a proxy in front of it) verifies it.
- Authorisation. A policy maps the verified identity to allowed actions: a mesh authorization policy, a VPC Lattice auth policy, or the application’s own checks.
- Network controls as the second layer. Security groups and NetworkPolicy still limit which addresses can connect at all. They prove where a packet came from, not who sent it. Keep them, but do not treat them as identity.
A service account token from cluster A is verifiable by anyone who can fetch cluster A’s OIDC discovery document and keys. EKS publishes them at the cluster’s public issuer URL, which is the same mechanism IRSA uses. See EKS Pod Identity and IRSA.
Reaching the other cluster’s Kubernetes API
Section titled “Reaching the other cluster’s Kubernetes API”A controller in A that manages objects in B (a GitOps controller, a multi-cluster scheduler) needs the API server of B, which is its own instance of the three layers.
-
Network. The EKS endpoint can be public, public and private, or private only. A private-only endpoint is reachable from the cluster’s VPC or a connected network (peering, transit gateway, VPN). The endpoint name resolves through public DNS to private addresses. The cluster security group must allow port 443 from the caller’s CIDR.
-
Authentication. The caller presents a token derived from its IAM identity (
aws eks get-token). In cluster A the pod gets that IAM role from Pod Identity or IRSA. -
Authorisation. In cluster B an access entry maps the IAM principal to Kubernetes permissions, through an AWS-managed access policy or a Kubernetes group bound by RBAC. Access entries need the cluster authentication mode
APIorAPI_AND_CONFIG_MAP; they replace theaws-authConfigMap.Terminal window aws eks create-access-entry --cluster-name cluster-b \--principal-arn arn:aws:iam::111122223333:role/cluster-a-deployer \--type STANDARD --kubernetes-groups deployers -
Other account. The role in A’s account assumes a role in B’s account, and that second role holds the access entry.
Without IAM, the alternative is a service account token of cluster B stored as a Secret in cluster A. It works anywhere but is a long-lived credential to rotate.
Compare
Section titled “Compare”Layer 1: network path (AWS terms)
Section titled “Layer 1: network path (AWS terms)”| Option | What it is | Cost model | Complexity | Overlapping CIDRs | Failure domain | Identity it carries |
|---|---|---|---|---|---|---|
| VPC peering (inter-region) | one-to-one private link between two VPCs | no hourly charge, data transfer only | low for 2 VPCs. A full mesh of n VPCs needs n(n-1)/2 peerings, since peering is not transitive | not allowed, not even partly | per pair. No shared device | none. Source IP is preserved |
| Transit Gateway + inter-region peering | a regional router per region, peered with each other | per attachment per hour, plus per GB processed | medium: route tables, static routes on the peering attachment | cannot route between them | the transit gateway of each region, and its route tables, are shared by every VPC attached | none. Source IP is preserved |
| PrivateLink (endpoint service) | one NLB-fronted service appears as network interfaces in the consumer VPC. Cross-Region access is supported | per endpoint per AZ per hour, plus per GB | medium: endpoint service, allowed principals, endpoint per consumer VPC | tolerated. Networks are never joined | per service. One direction only, consumer to provider | none on the wire. The provider controls which AWS principals may create endpoints |
| VPC Lattice | managed application-layer network: services in a service network, with listeners, rules and target groups | per service per hour, per GB, per request | medium to high: new object model; in Kubernetes through the AWS Gateway API Controller | tolerated | the Lattice service in the region. A client in another region reaches a service network endpoint over peering or a transit gateway | IAM: auth policies on the service or the service network evaluate the caller’s SigV4 identity |
| Public endpoint with TLS | internet-facing load balancer in B, client leaves through NAT | NAT gateway and internet data transfer | lowest: no network joining | irrelevant | the internet path, the NAT gateway, the public load balancer | none unless you add it. Everything depends on TLS plus request authentication. Source IP is the NAT address |
| VPN or overlay (Site-to-Site VPN, WireGuard, Submariner) | encrypted tunnels between gateways | per connection per hour or your own instances | medium to high: gateways to run, MTU, throughput limits of a tunnel | only with NAT (Submariner Globalnet) | the gateway nodes | none. It encrypts, it does not identify workloads |
Notes: inter-region peering and inter-region transit gateway peering use the same AWS backbone and are encrypted at the network layer by AWS. That is link encryption between regions. It does not authenticate workloads.
Layer 2: discovery and exposure
Section titled “Layer 2: discovery and exposure”| Option | How the client finds the service | Network it needs | Overlapping pod CIDRs | Complexity | Failure domain | Identity it carries |
|---|---|---|---|---|---|---|
| Internal NLB or ALB + DNS | a DNS record for the load balancer in B | reachability to B’s load balancer subnets only | tolerated (only LB subnets matter) | low. Standard Kubernetes plus DNS | the load balancer and the DNS zone. No cluster-to-cluster control plane | none. Add TLS and request authentication |
| Gateway API | same as above; the Gateway is the portable, role-oriented way to configure the entry point in B (standard since v1.0, v1.7 in late 2026) | reachability to the Gateway’s address | tolerated | low to medium. Depends on the implementation | the gateway implementation | TLS termination or passthrough at the Gateway; client certificate validation depends on the implementation |
| Multi-Cluster Services API | ServiceExport in B, ServiceImport in A, name under clusterset.local |
whatever the implementation needs, usually pod-to-pod | implementation-dependent | medium. An API only; you need an implementation such as Cilium or Submariner Lighthouse | the implementation’s control plane | none by itself |
| Cilium ClusterMesh | global services (annotation) or MCS API; endpoints from all clusters are load-balanced in eBPF | node-to-node and, in native routing, pod-to-pod connectivity | not allowed: PodCIDRs must be unique across clusters | medium. Same datapath mode everywhere, Cilium versions within one minor, at most 255 clusters (511 with maxConnectedClusters) |
each cluster keeps its own control plane. If a remote cluster is unreachable, its last-known endpoints are kept | Cilium security identities for network policy across clusters. Not workload certificates |
| Istio multi-cluster | services with the same name and namespace are merged across clusters | single network: direct pod reachability, no overlapping addresses. Multi-network: only the east-west gateways must reach each other | tolerated in the multi-network model | high. Control plane model (multi-primary or primary-remote), remote secrets so control planes can read the other API server, common root CA | per control plane in multi-primary. A remote cluster depends on its primary | SPIFFE identities with mTLS end to end, through the east-west gateway without terminating |
| Linkerd multicluster | mirrored service inventory-east, or a federated service that unions same-named services |
gateway mode: only the gateway address. Flat and federated: pod-to-pod | tolerated in gateway mode | medium. A Link per cluster pair, a shared trust anchor |
each cluster has its own control plane; a link failing affects only mirrored services | mTLS across clusters. In flat mode the workload identity is preserved. Through the gateway the client identity is lost |
| Submariner | Lighthouse implements the MCS API on top of Submariner’s tunnels | it builds the L3 path itself between gateway nodes (IPsec with Libreswan by default, WireGuard, or unencrypted VXLAN) | handled by Globalnet, which gives each cluster a slice of a global CIDR | medium. Broker, gateway nodes (active/passive), route agents | the active gateway node of each cluster | none. Tunnel encryption only |
Things that are easy to get wrong:
- Do not put an Istio east-west gateway behind a Layer 7 load balancer. It relies on passing mTLS through by SNI, and an L7 load balancer terminates TLS.
- In Cilium 1.19 and later a network policy selects endpoints in the local cluster only, unless it names a cluster with the label
io.cilium.k8s.policy.cluster. Policies are not distributed between clusters; apply them in each. - With the MCS API there is no DNS name for “this service in that one cluster”. Export a differently named Service per region if you need to pin.
- A global or federated service fails over across regions by itself. That is the goal, and also a way to send all traffic across an expensive, slow link when local pods are unhealthy. Cilium has
service.cilium.io/affinity: localfor this.
Layer 3: identity and authorisation
Section titled “Layer 3: identity and authorisation”| Option | What the server verifies | Cross-cluster requirement | What it proves | Cost and complexity | Fails when |
|---|---|---|---|---|---|
| mTLS, shared trust root | the client certificate chains to a root both clusters trust; the SPIFFE ID or name in it | each cluster’s issuing CA is an intermediate under one root (Istio cacerts, Linkerd shared trust anchor) |
workload identity, per connection | one PKI to run and rotate. Compromise of the root affects all clusters | intermediates expire, clocks skew, trust domain names collide |
| mTLS, SPIFFE federation | the client certificate chains to a root in the other trust domain’s bundle | each side fetches the other’s trust bundle and keeps it fresh | workload identity, with separate roots per cluster or organisation | bundle exchange to operate. Istio has no built-in tooling for it; SPIRE does | a bundle goes stale after the other side rotates |
| IAM with SigV4 | AWS verifies the signature and evaluates IAM and resource policies | caller has an IAM role (Pod Identity or IRSA); the target accepts SigV4 (VPC Lattice auth policy, API Gateway, AWS services) | the IAM principal, per request. Works across accounts and regions | no PKI. The client must sign requests, so application or sidecar support is needed | credentials missing or expired, policy mismatch |
| OIDC token (service account token federation) | a JWT signed by cluster A’s issuer: signature, iss, aud, exp, sub |
B can fetch A’s JWKS (public for EKS), and the token is requested with an audience that names B | the service account, per request | low. Projected token volume with a custom audience, JWT validation in the server or proxy |
audience not checked (token replay to another service), key rotation not followed |
| Static shared secret or API key | the secret matches | distribute and rotate it in both clusters | possession only | lowest to start, worst to operate | leaked, never rotated |
| Security groups, NetworkPolicy, Cilium policy | source address, or a CNI identity derived from labels | CIDRs or cluster-aware selectors configured on both sides | network origin only | low | addresses are reused, NAT hides the source, anything on an allowed node can connect |
A common safe combination: private network path, internal load balancer with a TLS certificate, a request-level identity (SigV4 or JWT) checked by the server, and security groups limiting the source CIDR.
Kubernetes API of another cluster
Section titled “Kubernetes API of another cluster”| Approach | Network | Credential | Authorisation | Trade-off |
|---|---|---|---|---|
| IAM role + EKS access entry | public endpoint with an allowed CIDR list, or private endpoint over peering or transit gateway | short-lived token from the pod’s IAM role | access policy or Kubernetes group on the access entry | no stored secret. EKS only |
IAM role + aws-auth ConfigMap |
same | same | group mapping in the ConfigMap | legacy. One bad edit locks people out |
| Service account token of B stored in A | same | long-lived bearer token in a Secret | RBAC in B | works on any cluster. A secret to rotate and protect |
| OIDC: B trusts A’s issuer | same | A’s projected service account token, audience set for B | RBAC in B on the mapped username | no stored secret, portable. Needs API server authentication configuration in B |
| Agent in B dials out (pull model) | B to A only | B’s own identity | local to B | no inbound path to B’s API at all. Needs an agent per cluster |
What breaks
Section titled “What breaks”Test one layer at a time, from a pod in cluster A.
kubectl run net-debug --rm -it --image=nicolaka/netshoot -- bash
# layer 2: does the name resolve, and to whatdig +short inventory.internal.example.comdig +short inventory.shop.svc.clusterset.local
# layer 1: can a TCP connection opennc -vz -w 3 inventory.internal.example.com 443mtr -T -P 443 -c 5 -r inventory.internal.example.com
# layer 3: what certificate comes back, and does the request authenticateopenssl s_client -connect inventory.internal.example.com:443 -servername inventory.internal.example.com </dev/null 2>/dev/null | openssl x509 -noout -subject -issuer -ext subjectAltNamecurl -sv https://inventory.internal.example.com/healthz -o /dev/null| Symptom | Layer | Cause | Check |
|---|---|---|---|
| Name does not resolve in A | discovery | private hosted zone not associated with VPC A, or CoreDNS not configured for clusterset.local |
dig from the pod; kubectl -n kube-system get configmap coredns -o yaml |
ServiceImport missing in A |
discovery | namespace does not exist in A (namespace sameness), or the export has a conflict | kubectl get serviceexport,serviceimport -A; kubectl describe serviceexport inventory -n shop |
| Resolves, then connection times out | network | missing route, security group, or network ACL. Timeouts mean dropped packets | route tables on both sides; security group on the load balancer; VPC Reachability Analyzer |
| Works one way only | network | return route missing, or the transit gateway peering has a static route on one side only | transit gateway route tables in both regions |
| Peering cannot be created | network | overlapping CIDRs | re-address, or use PrivateLink, VPC Lattice or Submariner Globalnet |
| Small requests work, large responses hang | network | MTU: inter-region peering is 8500, tunnels are lower, and ICMP “fragmentation needed” is filtered | ping -M do -s 1472 TARGET, then raise the size |
connection refused |
exposure | load balancer has no healthy targets, or wrong port | target health; kubectl get endpointslices -n shop in B |
| TLS error: unknown CA | identity | the two clusters do not share a root, or a trust bundle is stale | compare the root certificates of both meshes |
| TLS handshake fails through the gateway | exposure | an L7 load balancer in front of a passthrough gateway | load balancer type and listener protocol |
403 from the mesh or from VPC Lattice |
authorisation | policy does not name the caller’s identity, trust domain or IAM role | the policy; the peer identity in the proxy access log |
| Cilium ClusterMesh pods cannot talk | network | PodCIDRs overlap, datapath modes differ, or node-to-node ports are closed | cilium clustermesh status; cilium status |
| Cross-cluster traffic denied after a Cilium upgrade | authorisation | policies now select the local cluster by default | add io.cilium.k8s.policy.cluster to the selector |
kubectl to cluster B: Unauthorized |
API access | no access entry for the role, wrong authentication mode, or the token is for another role | aws sts get-caller-identity; aws eks list-access-entries --cluster-name cluster-b |
kubectl to cluster B: timeout |
API access | private endpoint not reachable, or cluster security group blocks 443 | nc -vz to the endpoint name; cluster security group rules |
kubectl to cluster B: Forbidden |
API access | authenticated, but no access policy or RBAC binding | kubectl auth can-i --list |
Keywords
Section titled “Keywords”cross-cluster, multi-cluster, multi-region, VPC peering, inter-region peering, Transit Gateway, peering attachment, static route, PrivateLink, endpoint service, interface endpoint, VPC Lattice, service network, auth policy, SigV4, overlapping CIDR, NAT, MTU 8500, internal NLB, ALB, ExternalDNS, private hosted zone, Gateway API, Multi-Cluster Services API, MCS API, KEP-1645, ServiceExport, ServiceImport, ClusterSet, clusterset.local, namespace sameness, Cilium ClusterMesh, global service, service.cilium.io/global, clustermesh-apiserver, Istio multi-cluster, multi-primary, primary-remote, east-west gateway, remote secret, Linkerd multicluster, service mirror, Link, federated service, Submariner, Lighthouse, Globalnet, Broker, Libreswan, WireGuard, SPIFFE federation, trust bundle, trust domain, shared root CA, projected service account token, audience, OIDC issuer, JWKS, EKS access entry, authentication mode, aws-auth, cluster security group, private endpoint.
Sources
Section titled “Sources”- Amazon VPC: How VPC peering connections work (limitations, overlapping CIDRs, MTU, inter-region)
- AWS Transit Gateway: peering attachments (static routes, encryption between regions)
- AWS PrivateLink: Share your services (endpoint services, cross-Region access)
- Amazon VPC Lattice: What is VPC Lattice (components, auth policies, overlapping CIDRs, pricing dimensions)
- AWS Gateway API Controller
- Kubernetes Gateway API
- SIG Multicluster: Multi-Cluster Services API and KEP-1645
- Cilium: Setting up Cluster Mesh (requirements, 255 and 511 cluster limits)
- Cilium: Load-balancing and service discovery across clusters (global services, MCS API)
- Cilium: Network policy across clusters
- Istio: Deployment models (network, control plane and trust models)
- Istio: Install multi-primary on different networks (east-west gateway)
- Istio: Plug in CA certificates
- Linkerd: Multi-cluster communication and Federated services
- Submariner: Architecture (gateway engine, cable drivers, Lighthouse, Globalnet)
- SPIFFE: Concepts
- Amazon EKS: Grant IAM users and roles access to Kubernetes APIs (access entries)
- Amazon EKS: Cluster API server endpoint