Admission control and GitOps drift

In one line

Admission control is the stage where the API server may change or reject a write before storing it. A mutating policy changes the object on the way in, a GitOps controller then sees a live object that differs from Git and applies Git again, and the two repeat that forever unless one of them is told about the other.

Why it exists

Admission control. Authentication says who you are and authorization (RBAC) says whether you may create a Deployment. Neither can say “not with a latest tag”, “not as root”, or “add this label”. Kubernetes first solved that with admission plugins compiled into the API server (LimitRanger, ResourceQuota, PodSecurityPolicy). Every new rule needed a new release.

• Admission webhooks moved the logic out: the API server calls an HTTPS endpoint you run. Policy engines such as OPA Gatekeeper, Kyverno and Kubewarden are webhooks with a policy language on top.
• Webhooks cost a network hop on every write and can take the cluster’s write path down when they fail. So Kubernetes added policies that run inside the API server, written in the Common Expression Language (CEL): ValidatingAdmissionPolicy (stable in 1.30) and MutatingAdmissionPolicy (stable in 1.36).
• PodSecurityPolicy was removed in 1.25. Its replacement, Pod Security Admission, is a built-in plugin that only validates.

GitOps. Argo CD and Flux make Git the source of truth: a controller in the cluster compares what Git says with what is live and corrects the difference.

The conflict. Mutation makes the stored object differ from what was submitted. That is its purpose. GitOps treats any difference between submitted and stored as drift. Both are right by their own rules, and both are automatic, so they can disagree at machine speed.

How it works

The admission chain

Every write (create, update, patch, delete, connect) goes through these stages in order. Reads skip admission.

1. Authentication. Client certificate, bearer token, OIDC token. Result: a username and groups.
2. Authorization. RBAC (or another authorizer) answers allow or deny for verb, resource and namespace.
3. Mutating admission. Built-in mutating plugins, MutatingAdmissionPolicies and mutating webhooks run. Mutating webhooks are called one after another, and each sees the result of the previous one. They return a JSON patch. Defaults get applied here: a default StorageClass, a default ServiceAccount, injected sidecars, added labels.
4. Reinvocation. If a later mutation changed the object, plugins and webhooks with reinvocationPolicy: IfNeeded are called again so they can react. The default is Never. Because of this, a mutating webhook must be idempotent.
5. Schema validation. The final object is checked against the type’s schema (OpenAPI for built-in types, the CRD schema and its CEL rules for custom ones).
6. Validating admission. Built-in validating plugins, ValidatingAdmissionPolicies and validating webhooks run on the final object. Validating webhooks are called in parallel. They can only accept or reject. One rejection fails the request.
7. Persistence. The object is written to etcd. Controllers and the kubelet see it only now.

If any stage rejects, the request fails at once and nothing is stored.

Webhooks

A MutatingWebhookConfiguration or ValidatingWebhookConfiguration tells the API server when to call which endpoint.

Field	Default	What it does
rules	none	operations, API groups, versions and resources to intercept
namespaceSelector, objectSelector	match all	skip namespaces or objects by label, before any call is made
matchConditions	none	CEL expressions evaluated in the API server; up to 64 per webhook
failurePolicy	Fail	what to do when the call errors or times out: Fail rejects the request, Ignore lets it through
timeoutSeconds	10	allowed range 1 to 30
reinvocationPolicy (mutating only)	Never	IfNeeded allows a second call after other mutations
sideEffects	must be set	None or NoneOnDryRun; tells the API server whether dry-run requests are safe to send
matchPolicy	Equivalent	also match the same resource through other API versions

failurePolicy covers errors in calling the webhook: network errors, timeouts, a non-2xx or malformed response. An explicit, well-formed “denied” always rejects the request, whatever the policy says.

In-process policies (CEL)

	ValidatingAdmissionPolicy	MutatingAdmissionPolicy
Status	stable since Kubernetes 1.30	alpha in 1.32, beta in 1.34, stable since 1.36
API	admissionregistration.k8s.io/v1	admissionregistration.k8s.io/v1
Parts	the policy, an optional parameter resource, and a ValidatingAdmissionPolicyBinding	the policy, an optional parameter resource, and a MutatingAdmissionPolicyBinding
What it can do	reject, warn or audit: validationActions of Deny, Warn, Audit	change the object with an apply configuration (merged like server-side apply) or a JSON patch
Runs	inside the API server, no network call	inside the API server, no network call
Cannot do	call external systems, look at other objects beyond its parameters, scan existing objects	the same, and it cannot create other objects

A policy without a binding does nothing.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: replica-limit.example.com
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: ["apps"]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["deployments"]
  validations:
    - expression: "object.spec.replicas <= 5"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: replica-limit-binding.example.com
spec:
  policyName: replica-limit.example.com
  validationActions: [Deny]
  matchResources:
    namespaceSelector:
      matchLabels:
        environment: test

The mutate and out-of-sync loop

Example: Git holds a Deployment with image: nginx:1.27. A mutating policy rewrites every image to the internal mirror, registry.internal/nginx:1.27. Argo CD manages the application with automated sync and selfHeal: true.

1. Sync. Argo CD applies the manifest from Git.
2. Mutation. In the mutating admission stage the policy changes spec.template.spec.containers[0].image. The API server stores the mutated object.
3. Compare. Argo CD builds the desired object from Git and compares it with the live one. The image differs. The application is OutOfSync, seconds after a successful sync.
4. Self-heal. With selfHeal: true Argo CD syncs again after the self-heal timeout (5 seconds by default, --self-heal-timeout-seconds on the application controller). It applies nginx:1.27.
5. Mutation again. The same policy rewrites the same field. Go to 3.

Git never changes, so nothing ever ends the loop. Without self-heal the loop stops at step 3 and the application stays OutOfSync permanently, which hides real drift behind a false one.

The mutation happens after Argo CD's request and before storage, so Argo CD can only see it afterwards, as a difference. Every fix either removes the mutation, hides the field from the comparison, or lets the comparison include the mutation.

Not every mutation causes this. Roughly:

• A mutation that changes a field Git sets (image, resources, replicas, a security context value) always shows as a difference.
• A mutation that adds something Git does not mention (a label, an annotation) often does not show with Argo CD’s default three-way diff, because that diff looks at fields the manifest or the last applied configuration owns.
• A mutation inside a list that is compared as a whole (common in custom resources) shows even when it only adds an element.
• Pods are a special case: sidecar injection mutates Pods, and Argo CD manages the Deployment, so there is nothing to compare.

How Argo CD computes the difference matters, so know the three strategies:

Diff strategy	How it works	Sees mutations as
Legacy (default)	three-way diff of Git, live state and the last-applied-configuration annotation, computed in Argo CD	drift on fields Git sets
Structured-merge diff	used field ownership from managedFields. Discontinued	not applicable
Server-side diff (stable since Argo CD 3.1.0)	a server-side apply in dry-run mode for each resource; the API server’s answer is compared with live. Used automatically when the ServerSideApply sync option is on	by default mutating webhook changes are left out of the result. With IncludeMutationWebhook=true they are included, so the predicted object matches what the webhook will produce

Compare

Ways to resolve the loop

Fix	How	What you keep	What you give up
Put the mutation in Git	render the final value in the manifests (Helm values, Kustomize patches, a pre-commit or CI step that runs the same policy with the engine’s CLI)	Git is the full truth. No Argo CD configuration. The diff shows real drift only	the convenience of fixing things on admission. Every repository must produce the final value
Switch the policy from mutate to validate	the policy rejects objects that are not compliant instead of rewriting them. Usually paired with the row above	a clear failure at sync time with a message. No hidden changes	more failed syncs until teams fix manifests. Start in audit or warn mode
ignoreDifferences	on the Application: jsonPointers (RFC 6901 paths, / in a key written as ~1), jqPathExpressions (for list items selected by content), or managedFieldsManagers (ignore every field owned by a named field manager). System-wide in argocd-cm under resource.customizations.ignoreDifferences.<group>_<Kind> or .all	the mutation, and a green application	Argo CD no longer watches those fields at all. Real drift in them is invisible. One entry per mutated field, to maintain as policies change
ignoreDifferences + RespectIgnoreDifferences=true	the sync option makes the sync honour the ignore list too. Without it, ignored fields are still overwritten with the Git value on every sync, then mutated back	no write fight during syncs triggered by other changes	only effective once the resource exists. On creation the manifest is applied as-is
Server-side apply + server-side diff	sync option ServerSideApply=true, and annotation argocd.argoproj.io/compare-options: ServerSideDiff=true,IncludeMutationWebhook=true on the Application (or controller.diff.server.side: "true" in argocd-cmd-params-cm for all). The dry run passes through the mutating webhooks, so desired already contains the mutation	no per-field ignore rules. New mutating policies are covered automatically. Validating webhooks also answer during the diff	extra dry-run requests to the API server and to every webhook on each refresh. Not performed while the resource does not exist yet. Kyverno’s documentation recommends this combination
Field ownership by server-side apply alone	with ServerSideApply=true each manager owns the fields it set. A policy engine that mutates existing objects as its own field manager can then be ignored by name with managedFieldsManagers	a rule per manager, not per field	does not help for admission-time mutation: there the change is made inside Argo CD’s own request and is recorded under Argo CD’s manager
Exclude from the policy	narrow the policy’s match (namespace selector, labels, exclude rules) so it does not touch GitOps-managed objects	no loop, no Argo CD change	no enforcement where you excluded. Often the opposite of the intent
Turn off self-heal	selfHeal: false or manual sync for that application	the write loop stops	the application stays OutOfSync. Real drift is no longer corrected or even noticeable
Use Flux instead	Flux applies with server-side apply and computes drift with a dry run, which includes mutating webhooks	no special configuration, per the Kyverno documentation	a different tool. Dry runs still load the webhooks

Preference order in practice: validate instead of mutate where you can; mutate only for things Git should not know (cluster-specific defaults); for those, server-side diff with IncludeMutationWebhook=true; ignoreDifferences as the narrow fallback.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: shop
  namespace: argocd
  annotations:
    argocd.argoproj.io/compare-options: ServerSideDiff=true,IncludeMutationWebhook=true
spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - ServerSideApply=true
      - RespectIgnoreDifferences=true
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jqPathExpressions:
        - .spec.template.spec.containers[].image

The ignoreDifferences block and the server-side diff annotation are alternatives. Both are shown to give the exact spelling. A single resource can also carry its own rule in the annotation argocd.argoproj.io/ignore-differences.

Kyverno specifics

Kyverno 1.19 (late 2026) uses CEL-based policy types in the API group policies.kyverno.io/v1: ValidatingPolicy, MutatingPolicy, GeneratingPolicy, DeletingPolicy and ImageValidatingPolicy, each with a namespaced variant. The older ClusterPolicy and Policy kinds are deprecated in 1.19 and scheduled for removal in 1.20.

Kyverno feature	What it does	Interaction with Argo CD
Mutate on admission	changes the object inside the write request	the loop above. Fix with server-side diff plus IncludeMutationWebhook=true, or ignoreDifferences
Mutate existing (spec.evaluation.mutateExisting.enabled: true)	a background controller patches objects that already exist, asynchronously, after a trigger	drift appears some time after a clean sync, not at sync time. Self-heal reverts it, Kyverno patches again on the next trigger. Needs extra RBAC for Kyverno
Generate	creates or clones resources (a default NetworkPolicy, a copied Secret) when a trigger appears	the generated resource is not in Git. If it carries Argo CD’s tracking label it is treated as part of the application: OutOfSync, and deleted when prune is on
Generate with synchronize	Kyverno keeps the generated resource equal to its source and reverts edits	if Argo CD also manages that resource, two controllers write it in turn. Let one of them own it
MutatingPolicy autogen to MutatingAdmissionPolicy	Kyverno generates a built-in MutatingAdmissionPolicy so the API server runs the mutation in-process	same drift as any admission mutation, without the webhook hop
Kyverno installed by Argo CD	the Helm chart sets app.kubernetes.io/instance, which is also Argo CD’s label-tracking key	use annotation tracking. Use ServerSideApply=true, not Replace=true. Set config.preserve=false so cleanup works. Ignore aggregated ClusterRole rule changes with resource.compareoptions.ignoreAggregatedRoles: true

The tracking label problem. Argo CD decides which live objects belong to an application by a tracking mark. With label tracking that mark is app.kubernetes.io/instance: <app>. A clone made by Kyverno (or any tool that copies labels) inherits the label, so Argo CD believes the copy is part of the application, finds no manifest for it in Git, and wants to prune it. With annotation tracking, the default in Argo CD 3.x, the mark is argocd.argoproj.io/tracking-id: <app>:<group>/<kind>:<namespace>/<name>. A copy carries an ID that names the original, not itself, and Argo CD ignores such objects: no sync status, no pruning.

kubectl -n argocd get configmap argocd-cm -o jsonpath='{.data.application\.resourceTrackingMethod}'

Policy engines

	Kyverno	OPA Gatekeeper	Validating and Mutating AdmissionPolicy	Kubewarden	Pod Security Admission
Where it runs	webhook, own controllers	webhook, own controllers	inside the API server	webhook (PolicyServer)	inside the API server
Policy language	CEL in Kubernetes-style YAML	Rego, or CEL in a ConstraintTemplate	CEL	anything that compiles to WebAssembly (Rust, Go, Rego and others)	none. Three fixed levels
Validate	yes	yes	yes	yes	yes
Mutate	yes	yes: Assign, AssignMetadata, ModifySet (stable since 3.10)	yes (stable since 1.36)	yes	no
Generate resources	yes	no	no	no	no
Image signature checks	yes (ImageValidatingPolicy)	through external data providers	no	through policies	no
Existing objects	background scans and policy reports, mutate existing	audit of existing objects against constraints	no. Admission time only	audit scanner	audit and warn modes at admission; dry-run on namespace label change
Policy distribution	Kubernetes objects	ConstraintTemplate + Constraint objects	Kubernetes objects	OCI artifacts in a registry	namespace labels pod-security.kubernetes.io/<mode>: <level>
Default when the engine is down	Fail per policy	Ignore (fails open); audit catches up later	not applicable, no network call	a webhook like the others, see its documentation	not applicable
Uses built-in policies	can generate ValidatingAdmissionPolicy and MutatingAdmissionPolicy	can generate ValidatingAdmissionPolicy for CEL templates (beta, on by default since 3.20)	it is the built-in	no	it is a built-in
Project status	CNCF graduated (March 2026)	OPA is CNCF graduated	part of Kubernetes	CNCF sandbox	part of Kubernetes, stable since 1.25
Pick it when	you want validate, mutate, generate and image checks in one tool, in YAML	you already use Rego or OPA elsewhere, or need its constraint model	the rule is expressible in CEL on the object itself and you want no moving parts	you want policies as code in a general-purpose language, shipped as artifacts	baseline pod hardening per namespace and nothing custom
Cost	a webhook on the write path, several controllers, CRDs	a webhook, audit controller, Rego to learn	no audit of existing objects, no external data	a WebAssembly toolchain, a policy server	pods only, three levels, exemptions only in API server configuration

Pod Security Admission modes: enforce rejects the pod, audit adds an audit annotation, warn returns a warning to the client. Levels: privileged, baseline, restricted. enforce applies to Pods only, so a Deployment that violates the level is accepted and its pods then fail to be created. audit and warn also look at workload resources.

failurePolicy: Fail or Ignore

	Fail (the default, fail closed)	Ignore (fail open)
Webhook down or slow	every matching write is rejected	writes pass without the policy
Security	nothing bypasses the policy	an attacker, or bad luck, that takes the webhook down removes the control
Availability	the webhook is now part of the control plane’s write path	the cluster keeps working
Use for	validating policies that are real security boundaries, with exclusions for system namespaces and several replicas	mutating webhooks, per the Kubernetes guidance: fail open on mutation and validate the final state separately
Needs	high availability, PodDisruptionBudget, short timeout, namespace exclusions	a background audit to catch what slipped through

What breaks

Symptom	Cause	Check
Application OutOfSync seconds after a successful sync	a mutating webhook or controller changed a field Git sets	argocd app diff shop; the diff shows the field
Argo CD syncs the same application every few seconds	the loop with selfHeal: true	kubectl -n argocd get application shop -o jsonpath='{.status.operationState.finishedAt}{"\n"}{.status.sync.status}{"\n"}', repeated; controller logs
Who changed this field	field ownership is recorded per manager	kubectl get deploy web -o yaml --show-managed-fields
What would the API server store for this manifest	dry run through all mutating webhooks	kubectl apply --server-side --dry-run=server -f web.yaml -o yaml
Which webhook mutated the object	the API server records it in audit annotations	audit log entries mutation.webhook.admission.k8s.io/round_N_index_M and patch.webhook.admission.k8s.io/...
ignoreDifferences set, the field is still overwritten on each sync	the ignore list affects the diff only	add RespectIgnoreDifferences=true
jsonPointers entry has no effect	a / inside a key was not escaped, or the target is a list item	write ~1 for /; use jqPathExpressions for list items
Generated resources get pruned	label tracking plus copied labels	switch to annotation tracking; check resourceTrackingMethod
Every write fails: failed calling webhook ... context deadline exceeded	the webhook is down or unreachable and failurePolicy is Fail	kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations; the webhook’s pods and Service endpoints
The policy engine cannot restart: its own pods are rejected	the webhook matches its own namespace, and it is down	exclude the engine’s namespace and kube-system in the namespaceSelector. Kyverno does both by default
Cluster stuck after a node or zone outage: nothing can be scheduled or scaled	webhook pods were on the lost nodes, Fail blocks pod creation everywhere, including their own replacements	delete the webhook configurations to recover; then add replicas across zones and exclusions
Writes are slow, API server latency climbs	a webhook near its timeout on every request	API server metric apiserver_admission_webhook_admission_duration_seconds; lower timeoutSeconds
A second mutating webhook does not see the first one’s change	call order is not guaranteed and the default is no reinvocation	set reinvocationPolicy: IfNeeded; make the webhook idempotent
Two mutating webhooks, or a webhook and a controller, flip a field back and forth	competing mutators	audit log: the same field updated and reverted repeatedly
Deployment accepted, no pods appear	Pod Security Admission enforce rejects the pods, not the Deployment	kubectl describe replicaset; events show violates PodSecurity
ValidatingAdmissionPolicy has no effect	no binding, or the binding’s validationActions is Audit or Warn only	kubectl get validatingadmissionpolicybinding
Server-side diff shows drift only on new resources	server-side diff is not run on creation	expected. It settles after the first sync

Recover a cluster blocked by Kyverno’s webhooks (the engine recreates them when it is healthy again):

kubectl delete validatingwebhookconfiguration kyverno-resource-validating-webhook-cfg
kubectl delete mutatingwebhookconfiguration kyverno-resource-mutating-webhook-cfg

Inspect what intercepts a write:

kubectl get mutatingwebhookconfigurations,validatingwebhookconfigurations
kubectl get mutatingwebhookconfiguration NAME \
  -o jsonpath='{range .webhooks[*]}{.name}{"\t"}{.failurePolicy}{"\t"}{.timeoutSeconds}{"\t"}{.reinvocationPolicy}{"\n"}{end}'
kubectl get validatingadmissionpolicies,validatingadmissionpolicybindings
kubectl get mutatingadmissionpolicies,mutatingadmissionpolicybindings
kubectl get ns --show-labels | grep pod-security

Related: a crash-looping webhook pod is still a crash loop, see CrashLoopBackOff and pod states. The API server calls webhooks over TLS and checks the caBundle, see TLS.

Keywords

admission control, admission controller, authentication, authorization, RBAC, mutating admission, validating admission, MutatingWebhookConfiguration, ValidatingWebhookConfiguration, AdmissionReview, failurePolicy, Fail, Ignore, timeoutSeconds, reinvocationPolicy, IfNeeded, idempotent, namespaceSelector, objectSelector, matchConditions, sideEffects, CEL, ValidatingAdmissionPolicy, MutatingAdmissionPolicy, policy binding, Pod Security Admission, Pod Security Standards, Kyverno, ValidatingPolicy, MutatingPolicy, GeneratingPolicy, mutateExisting, synchronize, OPA Gatekeeper, ConstraintTemplate, Rego, Kubewarden, WebAssembly, GitOps, Argo CD, Flux, OutOfSync, drift, selfHeal, self-heal timeout, ignoreDifferences, jsonPointers, jqPathExpressions, managedFieldsManagers, RespectIgnoreDifferences, ServerSideApply, ServerSideDiff, IncludeMutationWebhook, compare-options, server-side apply, field manager, managedFields, dry run, resource tracking, tracking-id, app.kubernetes.io/instance, prune.

Sources

• Kubernetes: Admission control in Kubernetes (phases, serial and parallel webhook calls)
• Kubernetes: Dynamic admission control (failurePolicy, timeouts, reinvocation, matchConditions, audit annotations)
• Kubernetes: Admission webhook good practices (fail open and validate, loops, ordering, idempotence)
• Kubernetes: Validating admission policy
• Kubernetes: Mutating admission policy
• Kubernetes: Feature gates (MutatingAdmissionPolicy stable in 1.36, ValidatingAdmissionPolicy stable in 1.30)
• Kubernetes: Pod Security Admission
• Kubernetes: Server-side apply
• Argo CD: Diffing customization (ignoreDifferences, system-level configuration)
• Argo CD: Diff strategies (server-side diff, IncludeMutationWebhook)
• Argo CD: Sync options (ServerSideApply, RespectIgnoreDifferences)
• Argo CD: Automated sync policy (self-heal, 5 second timeout)
• Argo CD: Resource tracking
• Kyverno: Policy types overview
• Kyverno: MutatingPolicy (mutate existing, reinvocation, GitOps considerations)
• Kyverno: Platform notes, Argo CD
• Kyverno: Migrating to CEL policies (deprecation of ClusterPolicy)
• Kyverno: Troubleshooting, API server is blocked
• Gatekeeper: Mutation, Validating admission policy integration, Failing closed
• Kubewarden: Admission controller architecture
• CNCF: Kyverno project page