mTLS and service mesh
In one line
Section titled “In one line”Mutual TLS gives every workload a certificate with its own name, so each side of a connection can prove who it is and the traffic between them is encrypted. A service mesh does this outside the application, with a proxy next to each pod or on each node.
Why it exists
Section titled “Why it exists”Inside a cluster, the old security model was the network perimeter: if a packet came from a pod IP, it was trusted. That breaks in three ways.
- IPs are not identities. Pod IPs are recycled within seconds. A rule that says “allow 10.0.3.17” allows whatever pod gets that address next. After NAT, or across clusters, the source address says even less.
- The network is shared. Many teams and many third-party images run on the same nodes. One compromised pod can connect to anything a flat network reaches, and can read plain traffic if it gets onto a node.
- Compliance asks for encryption in transit between services, not only at the edge.
“Zero trust” is the answer in one phrase: authenticate every connection by a cryptographic identity, not by where it comes from.
The pieces arrived in this order:
- TLS with client certificates existed all along (see mTLS in the TLS page). The hard part was never the handshake. It was giving thousands of short-lived workloads a certificate each, rotating them, and distributing trust roots.
- Service meshes automated that. Linkerd and Istio put a proxy beside every pod, issued certificates from the pod’s service account, and did mTLS without code changes.
- SPIFFE (Secure Production Identity Framework for Everyone) standardised the identity itself: a URI name, a document that proves it (the SVID), and an API to fetch it. SPIRE is its reference implementation.
- Sidecars turned out expensive: one proxy per pod to pay for, upgrade and debug. Istio’s ambient mode (stable in Istio 1.24, November 2024) moved the mTLS work into one proxy per node. Cilium went further down, into the kernel datapath.
How it works
Section titled “How it works”Identity: SPIFFE
Section titled “Identity: SPIFFE”| Term | Meaning |
|---|---|
| SPIFFE ID | a URI that names a workload: spiffe://<trust domain>/<path>. Istio uses spiffe://cluster.local/ns/<namespace>/sa/<service account> |
| Trust domain | the root of trust, one per organisation, environment or cluster. Workloads in it can be verified against its root keys |
| SVID | SPIFFE Verifiable Identity Document. An X.509-SVID is a certificate with the SPIFFE ID as a URI in the subject alternative name. A JWT-SVID is a signed token, meant for cases where a proxy sits between the two workloads |
| Trust bundle | the CA certificates (or public keys) used to verify SVIDs of a trust domain |
| Workload API | a local API from which a workload fetches its SVID, key and trust bundles, without holding any secret beforehand |
| Federation | two trust domains exchange bundles so each can verify the other’s SVIDs |
In Kubernetes the usual source of identity is the service account. That makes the service account the security boundary: two workloads that share one are the same principal to every mesh.
Mesh mTLS step by step
Section titled “Mesh mTLS step by step”Example: orders calls inventory, both meshed.
- Bootstrap. The proxy (or its agent) starts with the pod and reads the pod’s projected service account token.
- Key and CSR. It generates a private key in memory and sends a certificate signing request to the mesh certificate authority, with the token as proof.
- Issuance. The CA validates the token with the Kubernetes API, checks that the requested identity matches the token’s service account, and signs a short-lived certificate. The private key never leaves the proxy.
- Rotation. The proxy repeats this before expiry. Istio requests 24 hour certificates and renews at half the lifetime by default. Linkerd certificates last 24 hours.
- Outbound.
ordersopens a plain connection toinventory. The traffic is redirected to the local proxy. The control plane has told the proxy which endpoints exist and which identity each must present. - Handshake. The client proxy opens TLS to the server proxy. Both send certificates. Each side validates the chain against the trust bundle, and the client checks that the server’s identity is one allowed to run
inventory. Istio calls this check secure naming. - Authorisation. The server proxy now knows the caller’s identity and applies policy: allow
ordersto callGET /stock, deny the rest. mTLS answers “who”, and this step answers “may they”. - Delivery. The server proxy passes the request to the application over the pod’s loopback, in plain text.
The application sees none of this. It also means the application cannot tell an authenticated request from a local one unless the proxy passes the identity on in a header.
Two shapes of data plane
Section titled “Two shapes of data plane”Sidecar. A proxy container runs in every pod. Rules in the pod’s network namespace redirect all inbound and outbound traffic through it. Policy at L4 and L7, retries, routing and telemetry all live in that one proxy.
Istio ambient. Two layers:
- ztunnel: a per-node proxy written in Rust. It handles L4 only: mTLS, L4 authorization, TCP telemetry. It does not parse HTTP. It tunnels traffic with HBONE, which is HTTP/2
CONNECTover mTLS, on TCP port 15008. ztunnel holds one certificate per service account of the pods on its node and uses the right one for each connection. The CA refuses to give a ztunnel an identity that is not running on its node, so a compromised node cannot impersonate the whole mesh. - Waypoint: an optional Envoy deployment, per namespace or per service, that adds L7 policy, routing and telemetry. Traffic goes ztunnel to waypoint to ztunnel only when a waypoint is configured.
- Enrolment is a label,
istio.io/dataplane-mode=ambient, on a namespace or pod. No pod restart, no injection.
Kernel datapath (Cilium). No user-space proxy on the path for plain L3/L4. Encryption and authentication are separate features, see the next section.
Cilium: what is and is not mTLS
Section titled “Cilium: what is and is not mTLS”Cilium has three different things that are easy to confuse.
| Feature | What it does | Is it mTLS between workloads |
|---|---|---|
| Transparent encryption with WireGuard or IPsec | encrypts traffic between nodes (WireGuard tunnel on UDP port 51871, or IPsec ESP). Traffic between pods on the same node is not encrypted | no. The keys belong to nodes. It proves nothing about which pod sent a packet |
| Mutual authentication (beta in Cilium 1.20) | a network policy can say authentication.mode: "required". The Cilium agents of the two nodes then perform a mutual TLS handshake out of band, with SPIFFE identities issued by SPIRE, and the datapath allows the flow once it succeeded |
it authenticates with mTLS, but the application traffic does not travel inside that TLS session. It is not encrypted unless WireGuard or IPsec is also on. Authentication is cached per identity pair, not done per connection. Not compatible with Cluster Mesh |
| ztunnel transparent encryption (beta in Cilium 1.20) | Cilium runs a ztunnel per node and enrols namespaces labelled io.cilium/mtls-enabled=true. Pod traffic is redirected to ztunnel and carried over mutual TLS |
yes, L4 mTLS for the enrolled namespaces. Not compatible with Cluster Mesh |
So “Cilium with WireGuard” is encryption in transit, and “Cilium mutual authentication” is identity with a different trust model from a per-connection mTLS session. State which one you mean.
Doing it in the application
Section titled “Doing it in the application”Without a mesh, the application terminates TLS itself and needs a certificate, a key and a trust bundle, all rotated.
- cert-manager issues a certificate from a private CA
Issuerinto a Secret (tls.crt,tls.key,ca.crt), and renews it. The application mounts the Secret and must reload it on change. - cert-manager csi-driver-spiffe mounts a short-lived X.509-SVID straight into the pod: the key is generated on the node and never leaves it, the SPIFFE ID is derived from the pod’s service account (
spiffe://<trust domain>/ns/<namespace>/sa/<service account>), the default lifetime is 1 hour, and the files appear under/var/run/secrets/spiffe.io. - SPIRE runs a server (the CA for the trust domain) and an agent on every node. The agent proves the node’s identity to the server (node attestation), then identifies each calling process by asking the kernel and the kubelet about it (workload attestation), and hands out SVIDs over the Workload API. Applications use a SPIFFE library or a helper that writes files.
In all three the application (or its framework) must request client certificates, verify the peer’s name, and make its own authorisation decision.
Compare
Section titled “Compare”Ways to get mTLS between workloads
Section titled “Ways to get mTLS between workloads”| Istio, sidecar mode | Istio, ambient mode | Linkerd | Cilium | Consul service mesh | Application TLS with cert-manager | SPIFFE and SPIRE alone | |
|---|---|---|---|---|---|---|---|
| Data plane | Envoy sidecar in each pod | ztunnel per node (Rust, L4) plus optional Envoy waypoints (L7) | Rust micro-proxy sidecar in each pod | eBPF in the kernel; ztunnel per node in the ztunnel mode; Envoy per node for L7 | Envoy sidecar in each pod | none. The application’s TLS stack | none. The application, or a proxy you configure |
| Where TLS terminates | in the pod, in the sidecar | in ztunnel, logically inside the pod’s network namespace; in the waypoint for L7 | in the pod, in the sidecar | WireGuard or IPsec: at the node. ztunnel mode: in ztunnel. Mutual authentication: no data TLS session | in the pod, in the sidecar | in the application process | in the application process, or wherever you load the SVID |
| Plain hop left | loopback inside the pod | pod to ztunnel on the same node | loopback inside the pod | inside the node | loopback inside the pod | none | none |
| Identity source | service account, as a SPIFFE ID | service account, as a SPIFFE ID | service account, name like sa.ns.serviceaccount.identity.linkerd.cluster.local |
Cilium security identity (labels), with SPIFFE IDs from SPIRE for mutual authentication | Consul service identity | whatever names you put in the certificate; service account with csi-driver-spiffe | SPIFFE ID from registration entries and attestation |
| Certificate authority | istiod, or a plugged-in intermediate under your root | istiod | Linkerd identity service, under a trust anchor you supply or generate | SPIRE (mutual authentication); a CA Secret you generate (ztunnel mode) | built-in CA, configurable provider | cert-manager Issuer (private CA, Vault and others) | SPIRE server, or an upstream CA |
| Workload certificate lifetime | 24 hours requested by default, renewed at half life | same | 24 hours | set by SPIRE | set by the CA provider | what you set; 1 hour by default with csi-driver-spiffe | set by SPIRE |
| Is the application traffic inside mTLS | yes | yes (HBONE) | yes (TLS 1.3) | only in ztunnel mode. Otherwise encrypted by node keys, or authenticated out of band | yes | yes | yes, if you wire it |
| L7 policy | yes: AuthorizationPolicy on methods, paths, JWT claims |
only with a waypoint | yes: Server, HTTPRoute, AuthorizationPolicy, MeshTLSAuthentication |
yes: CiliumNetworkPolicy L7 rules through Envoy | yes: intentions | in application code | in application code |
| Default towards plain text | PERMISSIVE: accepts both until you set STRICT |
same | accepts plain text from unmeshed clients (all-unauthenticated) until a default-deny policy is set |
no authentication unless a policy requires it | depends on intentions and proxy mode | your server decides | your server decides |
| Overhead | a proxy per pod: memory and CPU scale with pod count, two extra proxy hops | a proxy per node; L7 cost only where a waypoint is used | a small proxy per pod | lowest for L3/L4; kernel encryption | a proxy per pod | TLS in process, no extra hop | TLS in process, plus an agent per node |
| Operational cost | injection, sidecar upgrades by restarting pods, proxy configuration size | no injection and no restarts; two components to understand, CNI integration | simplest mesh to run; trust anchor and issuer rotation is yours | you already run the CNI; SPIRE to run for mutual authentication | Consul servers to run, in or outside the cluster | every application must implement TLS and reload certificates | registration entries, attestation, library adoption in every application |
| Status (November 2026) | stable. Istio 1.32 current. CNCF graduated | stable since Istio 1.24 (November 2024) | CNCF graduated. Latest version 2.20 (June 2026). The open source project publishes edge releases only; stable release artifacts come from vendors since February 2024 (2.15) | WireGuard and IPsec stable. Mutual authentication beta. ztunnel encryption beta. Cilium 1.20 current. CNCF graduated | vendor product from HashiCorp | cert-manager is CNCF graduated | SPIFFE and SPIRE are CNCF graduated |
| Pick it when | you need the full feature set (traffic management, L7 policy everywhere) and accept the sidecar cost | you want mTLS for everything cheaply and L7 only in places | you want mTLS and golden metrics with the least machinery | you want encryption in transit and network policy from the CNI, and do not need per-connection workload mTLS today | you already run Consul, or span Kubernetes and VMs | a few services, or protocols a proxy handles badly | identity across platforms (Kubernetes, VMs, clouds) or as the CA under a mesh |
What each control proves
Section titled “What each control proves”| mTLS between workloads | One-way TLS | Network encryption (WireGuard, IPsec) | NetworkPolicy | |
|---|---|---|---|---|
| Confidentiality on the wire | yes | yes | yes, between nodes | no |
| Integrity on the wire | yes | yes | yes, between nodes | no |
| Server identity proven to client | yes, by certificate | yes, by certificate | no. Only “a node of this cluster” | no |
| Client identity proven to server | yes, by certificate | no | no | no. Source IP or label-derived identity is trusted, not proven |
| Granularity | workload (service account) | server name | node pair | pod labels, namespaces, CIDRs, ports |
| Same-node traffic | covered | covered | not encrypted | enforced |
| Stops a compromised pod connecting to a service | only with an authorisation policy on the verified identity | no | no | yes, if the policy excludes it |
| Stops a compromised node reading other pods’ traffic | traffic to pods on other nodes stays protected; keys on that node are exposed | same | no. The node holds the keys | no |
| Survives IP reuse and NAT | yes | yes | not relevant | no |
| Typical failure | certificate expiry, trust bundle mismatch, permissive mode left on | the same, without client checks | key rotation, MTU | rule gaps, default-allow when no policy selects a pod |
They stack. A reasonable baseline is default-deny NetworkPolicy, mTLS in strict mode, and authorisation policy on identities. Network encryption alone satisfies “encrypted in transit” but not “authenticated”.
Istio peer authentication modes
Section titled “Istio peer authentication modes”| Mode | Server accepts | Use |
|---|---|---|
PERMISSIVE (the mesh default when unset) |
both mTLS and plain text | migration. Clients with a proxy use mTLS automatically, others still work |
STRICT |
mTLS only | the target state. Set per namespace first, then mesh-wide in the root namespace |
DISABLE |
plain text only | exceptions |
UNSET |
inherits from the parent scope | workload inherits namespace, namespace inherits mesh |
apiVersion: security.istio.io/v1kind: PeerAuthenticationmetadata: name: default namespace: istio-systemspec: mtls: mode: STRICTIstio evaluates authorization policies in the order CUSTOM, DENY, ALLOW. A workload with no ALLOW policy allows everything; once one ALLOW policy selects it, anything not matched is denied. Istio uses TLS 1.2 as the minimum version.
What breaks
Section titled “What breaks”| Symptom | Cause | Check |
|---|---|---|
| “We have mTLS” but plain text still works | Istio in PERMISSIVE, or Linkerd default policy all-unauthenticated: unmeshed clients are accepted |
kubectl get peerauthentication -A; call the service from a pod without a proxy |
After switching to STRICT, some clients get curl: (56) Recv failure: Connection reset by peer |
those clients have no proxy (not injected, not enrolled, outside the cluster) | which pods lack the sidecar: kubectl get pods -n NS -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].name}{"\n"}{end}' |
503 with upstream connect error or disconnect/reset before headers between meshed pods |
client and server disagree about TLS: a DestinationRule turns TLS off or on against the server’s mode, or the server certificate is not trusted | kubectl get destinationrule -A -o yaml; istioctl proxy-config secret POD |
RBAC: access denied (HTTP 403) |
an authorization policy does not match the caller’s identity: wrong namespace or service account, or the request arrived as plain text so there is no identity | proxy log level rbac:debug: istioctl proxy-config log deploy/inventory --level rbac:debug |
| Handshake fails with certificate expired or not yet valid | workload certificate was not rotated (control plane or CA unreachable), or the node’s clock is wrong. Short-lived certificates leave little tolerance for clock skew | istioctl proxy-config secret POD (NOT AFTER, NOT BEFORE); date -u on the node; chronyc tracking |
| Whole mesh fails at once, a year after installation | the root or issuer certificate expired. Linkerd’s generated trust anchor and issuer both default to one year | linkerd check --proxy; for Istio, the cacerts or istio-ca-secret certificate dates |
| Cross-cluster calls fail with unknown CA | the clusters have different roots | compare root certificates. See Cross-cluster access |
| Ambient: traffic is plain | namespace not labelled, pod opted out, or the destination is outside the mesh | kubectl get ns -L istio.io/dataplane-mode; istioctl ztunnel-config workloads (PROTOCOL column shows HBONE or TCP) |
| Ambient: L7 rules ignored | no waypoint for that service, so only L4 policy is enforced | kubectl get gateway -n NS; label istio.io/use-waypoint |
| Linkerd: connection is not secured | one side is unmeshed, or the port is marked as a skip port | linkerd viz edges deployment -n NS (SECURED column); linkerd viz tap deploy/web -n NS shows tls=true or tls=no_tls_from_remote |
| Kubelet probes fail after enforcing mTLS | the kubelet is not in the mesh and has no identity | mesh-specific probe handling; in Linkerd the probe shows as tls=no_tls_from_remote |
| Cilium: policy with required authentication drops traffic | SPIRE server or agents unhealthy, or Cluster Mesh in use | kubectl get pods -n cilium-spire; cilium status |
| Cilium: encryption suspected off | WireGuard or IPsec not active on a node | kubectl -n kube-system exec ds/cilium -- cilium-dbg status (Encryption line); cilium-dbg encrypt status; tcpdump -n -i cilium_wg0 |
| Application-level mTLS: handshake fails after renewal | the application did not reload the new certificate, or the peer has an old CA bundle | certificate dates served on the port, see the openssl commands below |
| Server-first protocols (MySQL, SMTP) hang in a sidecar mesh | the proxy waits for the client to speak to detect the protocol | declare the port’s protocol explicitly, or exclude the port |
Look at certificates directly:
# Istio sidecar: the workload certificate and root the proxy holdsistioctl proxy-config secret deploy/orders -n shopistioctl proxy-config secret deploy/orders -n shop -o json \ | jq -r '[.dynamicActiveSecrets[] | select(.name == "default")][0].secret.tlsCertificate.certificateChain.inlineBytes' \ | base64 -d | openssl x509 -noout -subject -issuer -dates -ext subjectAltName# expect: URI:spiffe://cluster.local/ns/shop/sa/orders
# Istio ambient: what ztunnel knows and which certificates it holdsistioctl ztunnel-config workloadsistioctl ztunnel-config certificates ZTUNNEL_POD.istio-system
# is traffic on the wire encrypted (HBONE port, or the application port in clear)kubectl debug POD -it --image=nicolaka/netshoot -- tcpdump -nAi eth0 port 9080 or port 15008
# Linkerdlinkerd check --proxylinkerd viz edges deployment -n shoplinkerd viz tap deploy/orders -n shop
# any TLS server: what does it present, does it ask for a client certificateopenssl s_client -connect inventory.shop.svc:8443 -servername inventory.shop.svc </dev/null 2>/dev/null \ | openssl x509 -noout -subject -issuer -dates -ext subjectAltNameopenssl s_client -connect inventory.shop.svc:8443 </dev/null 2>&1 | grep -A3 'Acceptable client certificate CA names'
# present a client certificate by handopenssl s_client -connect inventory.shop.svc:8443 \ -cert /var/run/secrets/spiffe.io/tls.crt -key /var/run/secrets/spiffe.io/tls.key \ -CAfile /var/run/secrets/spiffe.io/ca.crt </dev/nullMigration order that avoids an outage: mesh every client first, watch for remaining plain connections in the mesh’s telemetry, set STRICT (or default-deny) one namespace at a time, then add authorisation policies. Going to strict before every client has a proxy is the classic self-inflicted incident.
Keywords
Section titled “Keywords”mTLS, mutual TLS, zero trust, workload identity, service account, SPIFFE, SPIFFE ID, SVID, X.509-SVID, JWT-SVID, trust domain, trust bundle, Workload API, SPIRE, node attestation, workload attestation, federation, service mesh, sidecar, Envoy, istiod, istio-agent, SDS, secure naming, PeerAuthentication, PERMISSIVE, STRICT, auto mTLS, DestinationRule, AuthorizationPolicy, ambient mode, ztunnel, waypoint, HBONE, port 15008, Linkerd, linkerd2-proxy, trust anchor, issuer certificate, MeshTLSAuthentication, default-inbound-policy, Cilium, mutual authentication, transparent encryption, WireGuard, IPsec, ESP, Consul, intentions, cert-manager, csi-driver-spiffe, trust-manager, NetworkPolicy, certificate rotation, clock skew, certificate expiry, istioctl proxy-config secret, istioctl ztunnel-config, linkerd viz edges, linkerd viz tap, openssl s_client.
Sources
Section titled “Sources”- SPIFFE: Concepts and SPIRE: Concepts
- Istio: Security concepts (identity, certificate provisioning, peer authentication modes, authorization order, minimum TLS version)
- Istio: pilot-agent reference (
SECRET_TTL24h,SECRET_GRACE_PERIOD_RATIO0.5,TRUST_DOMAIN) - Istio: Ambient mode overview, Ambient data plane, HBONE
- Istio blog: Ambient mode reaches general availability in 1.24 (7 November 2024)
- Istio: Verify mutual TLS is enabled (ambient) and Troubleshoot ztunnel
- Istio: Security problems and Mutual TLS migration
- Linkerd: Automatic mTLS (24 hour certificates, one year trust anchor and issuer, TLS 1.3)
- Linkerd: Validating your mTLS traffic
- Linkerd: Authorization policy and Server policy
- Linkerd: Releases and versions (edge and stable release model)
- Cilium: Mutual authentication (beta) (out-of-band handshake, limitations, roadmap status)
- Cilium: Transparent encryption, WireGuard, ztunnel (beta)
- Consul: Service mesh (Envoy sidecars, built-in certificate authority)
- cert-manager: csi-driver-spiffe
- Kubernetes: Network policies