Containers, namespaces and cgroups

In one line

A container is an ordinary Linux process (or a few) with three things applied: namespaces limit what it can see, cgroups limit what it can use, and a reduced privilege set (capabilities, seccomp, a Linux Security Module, optionally a user namespace) limits what it can do. All containers on a host share one kernel.

Why it exists

Each step added one missing piece of isolation.

• 1979, chroot. Version 7 Unix added chroot(): a process gets a different root directory. File system view only, and root can escape it.
• 2000, FreeBSD jails. FreeBSD 4.0 extended the idea to processes, users and networking: the first thing that looks like a container.
• 2002, Linux mount namespaces. Linux 2.4.19 added the first namespace type. UTS and IPC followed in 2.6.19, PID and network in 2.6.24, and user namespaces were completed in 3.8 (2013).
• 2006 to 2008, cgroups. Engineers at Google started “process containers” in 2006 to account for and limit the resources of groups of processes. It was renamed control groups and merged in Linux 2.6.24 (January 2008).
• 2008, LXC. The first tool that combined namespaces and cgroups into containers on a mainline kernel.
• 2013, Docker. Docker did not add a kernel feature. It added the image: a layered, content-addressed file system bundle with a build file and a registry to push to and pull from. That made containers a packaging format.
• 2015, OCI. The Open Container Initiative standardised the image format and the runtime interface. Docker donated its runtime as runc. Any OCI runtime can now run any OCI image.
• 2016 onward, cgroup v2. Linux 4.5 made the redesigned single-hierarchy cgroup interface official. Kubernetes deprecated cgroup v1 in v1.35, and since then the kubelet refuses to start on a cgroup v1 node unless failCgroupV1 is set to false.

How it works

Starting one container

1. Unpack the image. The runtime stacks the image layers with overlayfs into one root file system and adds an empty writable layer on top.
2. Create a cgroup for the container and write its limits (cpu.max, memory.max, pids.max).
3. clone() with namespace flags. runc creates the first process with CLONE_NEWNS | CLONE_NEWPID | CLONE_NEWNET | CLONE_NEWIPC | CLONE_NEWUTS | CLONE_NEWCGROUP, or joins existing namespaces with setns(). In a Kubernetes pod the network, IPC and UTS namespaces belong to the pod sandbox (the pause container) and every container joins them.
4. Set up the view. Inside the new mount namespace: mount the root file system, /proc, /sys and /dev, mask sensitive paths, then pivot_root() so the old root is unreachable.
5. Drop privilege. Set the user and group, drop capabilities to the allowed set, set no_new_privs, apply the AppArmor or SELinux profile, and load the seccomp filter last.
6. execve() the entrypoint. It becomes PID 1 of the new PID namespace. See Processes and threads for what PID 1 must do about signals and zombies.

From the host, the result is a normal process: it appears in ps, it has a host PID, and its syscalls go straight to the host kernel.

Container syscalls go directly to the one shared host kernel, so a kernel bug is a path out of every container. A VM's syscalls stop at its own guest kernel, and only a narrow virtual hardware interface reaches the host.

Namespaces: what it can see

A namespace wraps one global resource so that processes inside see their own instance. Every process is in exactly one namespace of each type. /proc/<pid>/ns/ holds one symlink per type, and two processes with the same link target share that namespace.

Namespace	clone() flag	Isolates	Arrived in
Mount (mnt)	CLONE_NEWNS	the list of mount points, so the root file system	Linux 2.4.19 (2002)
UTS	CLONE_NEWUTS	hostname and NIS domain name	Linux 2.6.19
IPC	CLONE_NEWIPC	System V IPC objects and POSIX message queues	Linux 2.6.19
PID	CLONE_NEWPID	process ID numbers; the first process is PID 1	Linux 2.6.24
Network (net)	CLONE_NEWNET	network devices, addresses, routes, firewall rules, port space	Linux 2.6.24
User	CLONE_NEWUSER	user and group IDs and capabilities; UID 0 inside maps to an unprivileged UID outside	started in Linux 2.6.23, completed in 3.8
Cgroup	CLONE_NEWCGROUP	the cgroup root directory the process sees	Linux 4.6
Time	CLONE_NEWTIME	offsets of the boot and monotonic clocks	Linux 5.6

Not namespaced: the kernel itself, kernel modules, the system clock (wall time), most of /sys, and kernel keyrings. This is why the privilege limits below matter.

User namespaces are the strongest of these for security. Root in the container is an unprivileged user on the host, so a file system or capability mistake no longer means host root. In Kubernetes set hostUsers: false in the pod spec. The feature is stable since v1.36 and needs Linux 6.5 or later. A pod that uses it cannot also use hostNetwork, hostPID or hostIPC.

cgroups: what it can use

A control group is a directory under /sys/fs/cgroup. Processes are attached by writing their PID to cgroup.procs. Controllers enforce limits that are set by writing to files in the directory. Limits are hierarchical: a child can never exceed its parent.

	cgroup v1	cgroup v2
First release	Linux 2.6.24	official in Linux 4.5; cpu controller 4.15, cpuset 5.0, freezer 5.2
Hierarchies	one per controller (or set of controllers), each with its own tree	one unified tree for all controllers
Where processes live	any node	leaves only: a cgroup with controllers enabled for its children holds no processes (the “no internal processes” rule)
Granularity	per thread (tasks file)	per process; threaded mode since 4.14 for some controllers
Memory controls	hard limit, soft limit; separate kernel memory setting	memory.min, low, high, max; kernel memory included
Buffered write throttling	not attributed to the cgroup	io and memory cooperate, so page cache writeback is charged to the right cgroup
Pressure metrics	none	PSI files per cgroup
Delegation to unprivileged users	unsafe	designed for it
Kill everything in the group	signal each PID	cgroup.kill
Detect	stat -fc %T /sys/fs/cgroup prints tmpfs	prints cgroup2fs
Kubernetes	deprecated since v1.35	needs kernel 5.8 or later and the systemd cgroup driver

The controllers that matter for containers (cgroup v2 file names):

Controller	Files	What it does
cpu	cpu.weight (1 to 10000, default 100), cpu.max ($MAX $PERIOD, default max 100000), cpu.stat	weight shares CPU proportionally under contention; max is a hard ceiling per period
memory	memory.min, memory.low, memory.high, memory.max, memory.current, memory.events, memory.stat, memory.oom.group	protection, throttling and the hard limit
io	io.max (rbps, wbps, riops, wiops per device), io.weight	throughput and IOPS limits per block device
pids	pids.max, pids.current, pids.events	limits the number of tasks. It counts threads, not only processes. fork() and clone() fail with EAGAIN at the limit
cpuset	cpuset.cpus, cpuset.mems	pins the group to specific CPUs and NUMA nodes

The four memory thresholds, from soft to hard:

File	Default	Behaviour
memory.min	0	hard protection: this much is never reclaimed
memory.low	0	best-effort protection: reclaimed only when nothing else is left
memory.high	max	throttle: above it the group is slowed and reclaimed hard, never OOM-killed
memory.max	max	hard limit: if usage cannot be brought under it, the OOM killer runs inside the cgroup

How Kubernetes requests and limits become cgroup files

On a cgroup v2 node with the systemd driver, a container’s cgroup sits at a path like /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod<uid>.slice/cri-containerd-<id>.scope.

Pod spec	cgroup v2 file	Value
resources.requests.cpu	cpu.weight	the kubelet computes v1-style shares, milliCPU * 1024 / 1000, and the OCI runtime converts shares to a weight. See below
resources.limits.cpu	cpu.max	quota period, with period 100000 µs (100 ms, kubelet cpuCFSQuotaPeriod). 500m becomes 50000 100000. No limit leaves max 100000
resources.limits.memory	memory.max	the limit in bytes
resources.requests.memory	nothing by default	used for scheduling, eviction order and oom_score_adj. See Memory QoS below
pod PID limit (kubelet podPidsLimit, default -1, no limit)	pids.max on the pod cgroup	set per node, not per pod
CPU manager static policy, Guaranteed pod with whole CPUs	cpuset.cpus	exclusive cores

The shares to weight conversion changed. It lives in the OCI runtime, not in Kubernetes.

Request	v1 cpu.shares	Old weight: 1 + ((shares - 2) * 9999) / 262142	New weight (runc 1.3.2 and later, crun 1.23 and later)
100m	102	4	17
1 CPU	1024	39	102

The old linear formula gave a 1-CPU container a weight of 39, well under the default of 100 that system services outside Kubernetes get. The new quadratic formula maps 1024 shares to about 100. Monitoring that computes expected weights from requests needs to know which runtime version is on the node.

Memory QoS (feature gate MemoryQoS, beta and on by default since v1.37) can use the other memory files, but the default kubelet configuration sets none of them. Two kubelet settings turn it on:

• memoryThrottlingFactor (default unset): sets memory.high = requests + factor * (limits - requests) for Burstable and BestEffort containers. With 0.9, a 256 MiB request and a 1 GiB limit give about 947 MiB.
• memoryReservationPolicy: TieredReservation (default None): sets memory.min to the request for Guaranteed pods and memory.low to the request for Burstable pods.

Node-level OOM. If the node itself runs out of memory before the kubelet can evict pods, the kernel’s global OOM killer chooses a victim by score. The kubelet sets oom_score_adj by QoS class: Guaranteed -997, BestEffort 1000, Burstable between 2 and 999 (lower when the request is a larger share of node memory).

CPU throttling and OOM kill are different failures

• CPU is compressible. The quota works per period. With cpu.max = 50000 100000, the threads of the cgroup may run for 50 ms of CPU time in every 100 ms window. When the quota is used up, every thread in the group is stopped until the next window. The process is slowed, never killed. Eight busy threads with a 1-CPU limit burn the 100 ms quota in 12.5 ms and then sit idle for 87.5 ms. The symptom is tail latency, not errors. cpu.stat counts it: nr_periods, nr_throttled, throttled_usec.
• Memory is not compressible. When usage reaches memory.max the kernel first tries to reclaim page cache from the cgroup. If that is not enough, the cgroup OOM killer picks a process inside the cgroup and sends SIGKILL. On cgroup v2 the kubelet sets memory.oom.group, so all processes in the container are killed together (set kubelet singleProcessOOMKill: true for the old one-process behaviour). The container exits with code 137 (128 + 9) and reason OOMKilled.
• What counts as memory. Anonymous memory (heap, stacks), page cache for files the container read or wrote, tmpfs including emptyDir with medium: Memory, and kernel memory such as socket buffers and dentries. Page cache is reclaimable, so high usage alone is not a problem. Anonymous memory without swap is not.

PSI (pressure stall information, Linux 4.20) tells you how much time tasks lost waiting for a resource. /proc/pressure/{cpu,memory,io} covers the node, and cpu.pressure, memory.pressure, io.pressure in each cgroup v2 directory cover a container:

some avg10=1.25 avg60=0.40 avg300=0.10 total=4120331
full avg10=0.00 avg60=0.00 avg300=0.00 total=81223

some is the share of time at least one task was stalled. full is the share of time all non-idle tasks were stalled at once. Memory full above zero means the workload is thrashing before any OOM kill appears.

Privilege: what it can do

Mechanism	What it restricts	Kubernetes field
Capabilities	root’s power split into about 40 units. Runtimes keep a small default set (such as CHOWN, SETUID, NET_BIND_SERVICE, KILL) and drop the rest, including SYS_ADMIN, SYS_MODULE, SYS_PTRACE, NET_ADMIN, BPF	securityContext.capabilities.drop: ["ALL"]
seccomp	which syscalls the process may make, as a classic BPF filter run on every syscall	seccompProfile.type: RuntimeDefault
no_new_privs	setuid binaries and file capabilities cannot raise privilege after execve()	allowPrivilegeEscalation: false
LSM (AppArmor, SELinux)	mandatory access control on files, mounts, sockets and signals, whatever the UID	appArmorProfile, seLinuxOptions
User namespace	maps container root to an unprivileged host UID	hostUsers: false
Read-only root file system	no writes to the image layers; writable paths are explicit volumes	readOnlyRootFilesystem: true
Non-root user	the process does not start as UID 0	runAsNonRoot: true, runAsUser
Device cgroup	which device nodes may be opened; an eBPF program on cgroup v2	none by default; privileged: true removes it

privileged: true turns off nearly all of the above at once: all capabilities, all devices, no seccomp, unmasked /proc and /sys. What remains is only the namespaces. For the Pod Security Standards and for who should be allowed to hold CAP_BPF or CAP_SYS_ADMIN, see eBPF.

Image layers and overlayfs

An image is an ordered list of tar archives (layers), each identified by the SHA-256 digest of its content, plus a JSON config. Layers are shared between images and pulled once.

overlayfs (in mainline since Linux 3.18) merges them at run time:

• lowerdir: the image layers, read-only, stacked.
• upperdir: one writable directory per container.
• merged: the union the container sees as /.
• Copy-up: the first write to a file from a lower layer copies the whole file to the upper layer. Modifying one byte of a 1 GB file costs 1 GB.
• Whiteout: deleting a lower-layer file creates a marker in the upper layer. The data stays in the image, so rm in a later Dockerfile step does not make the image smaller.

The writable layer disappears with the container and counts against the node’s ephemeral storage. Data that must survive belongs in a volume.

Compare

The Kubernetes runtime stack, from the top down.

Layer	Examples	Job
kubelet		watches pod specs for its node, decides what should run, computes cgroup settings, runs probes
CRI (Container Runtime Interface)	gRPC over a Unix socket	the API the kubelet uses: RunPodSandbox, CreateContainer, PullImage. dockershim, the built-in Docker adapter, was removed in v1.24
High-level runtime	containerd, CRI-O	pulls and unpacks images, manages snapshots (overlayfs), sets up the pod sandbox, calls the CNI plugin for networking
Shim	containerd-shim-runc-v2, conmon (CRI-O)	one small process per pod or container that stays as its parent: holds stdio, collects the exit code, and lets the runtime daemon restart without killing containers
Low-level (OCI) runtime	runc (Go, the reference), crun (C, faster start, less memory), runsc (gVisor), kata-runtime	does the kernel work from an OCI config.json: namespaces, cgroups, mounts, capabilities, seccomp, then execve(). runc exits once the container is started
Kernel		enforces everything

The isolation spectrum.

	runc or crun container	gVisor (runsc)	Kata Containers	Firecracker microVM	Full VM (QEMU/KVM, cloud instance)
Isolation boundary	host kernel syscall interface, filtered by seccomp, capabilities and LSM	a user-space kernel (the Sentry, written in Go) handles the application’s syscalls and makes a small set of host syscalls itself	hardware virtualization: each pod runs in a small VM with its own guest kernel	hardware virtualization with a minimal device model (five emulated devices)	hardware virtualization with a full device model
Shares the host kernel	yes	host kernel is not directly reachable from the application	no	no	no
Overhead	near zero	low memory, but syscall-heavy and I/O-heavy workloads slow down noticeably	a guest kernel and agent per pod, tens of MiB	under 5 MiB of memory per microVM	hundreds of MiB, full OS
Start-up	tens to hundreds of ms	close to a container	around a second or less, depends on the hypervisor	under 125 ms to boot	seconds to tens of seconds
Compatibility	everything Linux runs	most applications; some syscalls and /proc or /sys details are missing	high; host device and hostPath use is restricted	Linux guests with virtio devices only; no GPU passthrough	everything, any OS
Kubernetes integration	default	RuntimeClass	RuntimeClass	through Kata or as the engine under a service	KubeVirt or separate nodes
Use when	you trust the workloads, or they are your own	untrusted code, defence in depth for multi-tenant services	untrusted or regulated tenants that need full Linux compatibility	many short-lived untrusted functions or sandboxes	different OS or kernel, strongest separation

What breaks

Symptom	Cause	Check
Container restarts, Reason: OOMKilled, Exit Code: 137	usage reached memory.max and could not be reclaimed	kubectl describe pod, memory.events, dmesg (below)
Exit code 137 without OOMKilled	SIGKILL from elsewhere: grace period expired, node-level OOM, or eviction	kubectl get events, dmesg | grep -i 'out of memory'; see CrashLoopBackOff
p99 latency high, CPU use looks well below the limit	CFS quota throttling: averages hide bursts inside 100 ms periods	cpu.stat: nr_throttled and throttled_usec rising
fork: retry: Resource temporarily unavailable, pthread_create: EAGAIN	pids.max reached (threads count)	cat pids.current pids.max pids.events
operation not permitted (EPERM) as root in the container	missing capability, seccomp filter, LSM denial, or root in a user namespace	grep -E 'Cap(Eff|Bnd)|Seccomp|NoNewPrivs' /proc/1/status, capsh --decode=<hex>, dmesg | grep -Ei 'apparmor|avc|audit'
read-only file system	readOnlyRootFilesystem: true and the app writes outside a volume	mount an emptyDir at that path
Pod evicted, ephemeral-storage	writes to the container’s writable layer or logs filled the node allowance	crictl stats, du on the upperdir
Slow first write to a large file	overlayfs copy-up	put the file on a volume
Application sees all host CPUs and memory	/proc/cpuinfo and /proc/meminfo are not namespaced; the runtime sizes thread pools and heaps from the host	make the runtime cgroup-aware (JVM container support, Go 1.25 GOMAXPROCS), or set sizes explicitly

Reading an OOM kill. Inside the container the cgroup is mounted at /sys/fs/cgroup:

cat /sys/fs/cgroup/memory.max /sys/fs/cgroup/memory.current
cat /sys/fs/cgroup/memory.events

low 0
high 0
max 1423
oom 2
oom_kill 2
oom_group_kill 1

max counts how often usage hit the limit and reclaim was forced. oom counts how often the cgroup ran out of options, and oom_kill counts processes killed. A growing max with zero oom_kill means the container lives on page cache reclaim and is close to the edge.

On the node:

dmesg -T | grep -E 'oom-kill|Memory cgroup out of memory|Killed process'

oom-kill:constraint=CONSTRAINT_MEMCG,...,oom_memcg=/kubepods.slice/kubepods-burstable.slice/...
Memory cgroup out of memory: Killed process 23817 (java) total-vm:8123456kB, anon-rss:1046512kB, file-rss:0kB, shmem-rss:0kB

CONSTRAINT_MEMCG means a cgroup limit caused it. CONSTRAINT_NONE means the whole node ran out.

Reading throttling:

cat /sys/fs/cgroup/cpu.max /sys/fs/cgroup/cpu.stat

50000 100000
usage_usec 912345678
user_usec 801234567
system_usec 111111111
nr_periods 48211
nr_throttled 9120
throttled_usec 412998113

Here 19% of periods were throttled (9120 of 48211) and the threads lost 413 seconds in total. Fixes: raise or remove the CPU limit, lower the number of busy threads to match the limit, or keep the request and drop the limit so the weight alone governs sharing.

Looking inside from the node:

lsns                                   # all namespaces, with the lowest PID in each
lsns -p 23817                          # the namespaces of one process
ls -l /proc/23817/ns                   # same, as symlinks; equal targets mean shared
nsenter -t 23817 -n ss -ltnp           # run the host's ss inside the container's network namespace
nsenter -t 23817 -a                    # enter all namespaces: a shell "in" the container
unshare --pid --fork --mount-proc bash # a new PID namespace by hand; ps shows bash as PID 1
unshare --user --map-root-user bash    # root inside a user namespace, unprivileged outside
cat /proc/23817/cgroup                 # which cgroup: 0::/kubepods.slice/...
systemd-cgls                           # the cgroup tree with processes
systemd-cgtop                          # live CPU, memory and I/O per cgroup
crictl ps                              # containers as the runtime sees them
crictl inspect <container-id>          # pid, cgroup path, mounts, capabilities, seccomp, namespaces
crictl inspectp <pod-sandbox-id>       # the pod sandbox and its network namespace
stat -fc %T /sys/fs/cgroup             # cgroup2fs or tmpfs

nsenter -t PID -n is the standard way to debug the network of a distroless container: the tools come from the host, the network view comes from the container.

Keywords

container, namespace, cgroup, control group, chroot, pivot_root, FreeBSD jail, LXC, Docker, OCI, runc, crun, containerd, CRI-O, CRI, shim, conmon, pause container, pod sandbox, mount namespace, PID namespace, network namespace, UTS namespace, IPC namespace, user namespace, cgroup namespace, time namespace, clone, unshare, setns, nsenter, lsns, hostUsers, cgroup v1, cgroup v2, unified hierarchy, no internal processes rule, cgroup.procs, cgroup.controllers, cgroup.subtree_control, cgroup.kill, cpu.weight, cpu.shares, cpu.max, cpu.stat, nr_throttled, throttled_usec, CFS quota, CPU throttling, memory.min, memory.low, memory.high, memory.max, memory.events, memory.oom.group, oom_kill, OOMKilled, exit code 137, oom_score_adj, QoS class, Guaranteed, Burstable, BestEffort, MemoryQoS, pids.max, podPidsLimit, cpuset, io.max, PSI, pressure stall information, capabilities, CAP_SYS_ADMIN, seccomp, RuntimeDefault, no_new_privs, AppArmor, SELinux, read-only root filesystem, privileged container, overlayfs, lowerdir, upperdir, copy-up, whiteout, image layer, gVisor, runsc, Sentry, Kata Containers, Firecracker, microVM, KVM, RuntimeClass, systemd-cgls, crictl inspect.

Sources

• namespaces(7), clone(2) (flag versions), pid_namespaces(7), user_namespaces(7)
• cgroups(7): v1 and v2 history, controller versions
• Control Group v2, kernel documentation: every interface file and default
• PSI, kernel documentation
• Overlay file system, kernel documentation
• capabilities(7), seccomp(2)
• Kubernetes: about cgroup v2: requirements, cgroup v1 deprecation in v1.35
• Kubernetes: resource management for pods and containers
• Kubernetes: Pod QoS classes and Memory QoS
• Kubernetes blog: new conversion from cgroup v1 CPU shares to v2 CPU weight
• Kubernetes: node-pressure eviction: oom_score_adj values
• Kubernetes: kubelet configuration reference: podPidsLimit, cpuCFSQuotaPeriod, singleProcessOOMKill
• Kubernetes: process ID limits and reservations
• Kubernetes: user namespaces and Linux kernel version requirements
• Kubernetes: container runtimes
• OCI runtime specification and image specification
• gVisor documentation, Kata Containers, Firecracker
• Kernel release notes: 2.6.24 (control groups, PID and network namespaces), 3.18 (overlayfs), 4.20 (PSI)