Istio Security, mTLS, and Policy

Istio security combines workload identity, certificate distribution, mutual TLS, request authentication, and authorization policy. Keep those layers separate when designing or debugging access.

Security Layers

Layer Resource or Mechanism Purpose
Workload identity SPIFFE-style service identity Names the workload principal.
mTLS PeerAuthentication and auto mTLS Encrypts and authenticates service-to-service traffic.
Authorization AuthorizationPolicy Allows or denies requests based on source, operation, and conditions.
End-user authentication RequestAuthentication Validates JWTs and exposes claims to policy.
Gateway TLS Gateway listeners and credentials Presents edge certificates and controls termination or passthrough.

mTLS Modes

PeerAuthentication can be scoped mesh-wide, namespace-wide, workload-specific, or port-specific. Strict mode requires mTLS. Permissive mode accepts both plaintext and mTLS during migration. Disable mode turns mTLS off for the selected scope.

kubectl get peerauthentication --all-namespaces
istioctl authn tls-check <pod>.<namespace>
istioctl proxy-config secret <pod> -n <namespace>
istioctl proxy-config cluster <pod> -n <namespace> --fqdn <service>.<namespace>.svc.cluster.local

Most accidental mTLS outages are scope mismatches: one side requires mTLS while the other side sends plaintext or originates TLS incorrectly.

mTLS Handshake and Identity Path

sequenceDiagram
  participant Src as Source proxy / ztunnel
  participant CA as Istio CA
  participant Dst as Destination proxy / ztunnel

  Src->>CA: Request workload certificate
  CA-->>Src: SPIFFE-style cert and trust bundle
  Dst->>CA: Request workload certificate
  CA-->>Dst: SPIFFE-style cert and trust bundle
  Src->>Dst: TLS ClientHello and certificate
  Dst->>Src: Server certificate and trust validation
  Src->>Dst: Encrypted request with authenticated source identity
  Dst->>Dst: AuthorizationPolicy evaluates principal, namespace, method/path when available

Policy boundary matrix:

Enforcement Point Can See Cannot Reliably See
ztunnel L4 Source/destination workload identity, IP, port, mTLS state. HTTP path, method, headers.
Waypoint L7 HTTP host/path/method/headers and richer policy context. Traffic that bypasses the waypoint.
Sidecar Envoy Local workload traffic, L4/L7 attributes depending on protocol parsing. Traffic not captured by the sidecar.
Ingress gateway External TLS/SNI/HTTP boundary and route attachment. Internal workload identity unless traffic is reauthenticated downstream.

AuthorizationPolicy

AuthorizationPolicy evaluation depends on action, selector or target reference, source identity, namespace, path, method, host, port, and request attributes.

Common traps:

  • policy attaches to the wrong workload or namespace,
  • ambient L4 policy is expected to enforce HTTP fields without a waypoint,
  • path matching differs from application routing,
  • source principal is not what the operator expected,
  • deny policy has broader scope than intended,
  • JWT claims are referenced before RequestAuthentication validates them.
kubectl get authorizationpolicy --all-namespaces
kubectl describe authorizationpolicy <policy> -n <namespace>
istioctl analyze --all-namespaces
istioctl proxy-config listeners <pod> -n <namespace>

JWT and RequestAuthentication

RequestAuthentication validates tokens from configured issuers and JWKS locations. It authenticates the token; it does not by itself allow or deny traffic. Use AuthorizationPolicy to require principals, issuers, audiences, or claims.

Check:

  • issuer exactly matches the token,
  • JWKS URL is reachable by Istio,
  • audiences match the application expectation,
  • gateway versus workload attachment is intentional,
  • policy behavior for missing tokens is explicit.

Ambient Policy Split

In ambient mode, ztunnel handles L4 identity, mTLS, and L4 authorization. Waypoints handle L7 policy such as HTTP path, method, headers, and richer telemetry. If a policy uses L7 attributes, confirm traffic goes through a waypoint that owns that enforcement point.

istioctl ztunnel-config workloads -n <namespace>
istioctl waypoint list -A
kubectl get waypoint -A 2>/dev/null || kubectl get gateway -A

Security Debugging Runbook

  1. Identify the exact source workload, destination workload, and path through sidecar, ambient, waypoint, or gateway.
  2. Check PeerAuthentication scope and effective mTLS.
  3. Inspect proxy certificates and trust domain.
  4. List AuthorizationPolicy resources that can attach to the destination.
  5. Confirm whether JWT validation is required and whether missing tokens are denied.
  6. Use access logs and response flags to distinguish TLS failure, denied policy, and application 403.

Study Cards

Question

What does PeerAuthentication control?

Answer

The mTLS mode accepted by selected workloads, namespaces, or the mesh.

Question

What does AuthorizationPolicy decide?

Answer

Whether authenticated or unauthenticated requests are allowed or denied based on source, operation, and conditions.

Question

Why is RequestAuthentication not enough by itself?

Answer

It validates JWTs but does not require or authorize them unless AuthorizationPolicy uses the result.

Question

Why can ambient L7 policy need a waypoint?

Answer

ztunnel handles L4 policy; HTTP path, method, and header policy require an L7 enforcement point.

Question

What is a common mTLS mismatch?

Answer

One side requires mTLS while the other sends plaintext or uses an incompatible DestinationRule TLS mode.

References

Scenario Lab

Istio

Istio mTLS Policy Breakage

A service-to-service call starts returning proxy-generated 503 or 403 after a policy rollout.

Symptoms

  • Client Pods can resolve the Service, but requests fail at the proxy.
  • Access logs show mTLS, authorization, or upstream health response flags.
  • Direct app logs do not show the failed request reaching the handler.

Evidence

  • Compare PeerAuthentication, DestinationRule, AuthorizationPolicy, and RequestAuthentication scope.
  • Inspect Envoy clusters, routes, endpoints, and secrets for the source or gateway proxy.
  • Check workload identities and namespace labels before changing policy.

Command Examples

Command

istioctl analyze --all-namespaces

Example output

Warning [IST0101] (VirtualService payments.payments) Referenced host not found: payments-api

What it does: Finds mesh configuration conflicts before inspecting individual Envoy proxies.

Command

istioctl proxy-config clusters  -n 

Example output

SERVICE FQDN                  PORT     SUBSET     DIRECTION     TYPE
payments-api.payments.svc     8443     -          outbound      EDS

What it does: Shows whether the proxy received an outbound cluster for the target workload and port.

Command

istioctl proxy-config secret  -n 

Example output

RESOURCE NAME     TYPE           STATUS     VALID CERT     SERIAL NUMBER
default           Cert Chain     ACTIVE     true           1a2b3c

What it does: Verifies that the sidecar has active mTLS certificates and can participate in the mesh identity path.

Answer: Treat YAML as intent and proxy config as enforcement; prove whether the break is TLS mode, identity, authorization, endpoint readiness, or route attachment before relaxing policy.

Open related topic