Processes and threads

In one line

A process is a running program with its own virtual address space, file descriptor table and credentials. A thread is one flow of execution inside a process: it shares the address space and file descriptors, and owns only its stack, registers and a few per-thread settings. On Linux both are the same kernel object, a task, created by clone() with different sharing flags.

Why it exists

• Processes came first. Early Unix ran one flow of control per address space. fork() copied the parent, exec() replaced the program. Isolation was the point: one program cannot scribble over the memory of another.
• Processes are expensive to coordinate. Two processes that must share data need pipes, sockets or shared memory. Servers that handle many clients wanted cheap parallel work on shared data, so threads appeared: several flows in one address space.
• Linux took a short cut. Instead of a separate thread object, the kernel got one task_struct and a clone() call that says what the child shares with the parent. The first thread library built on it (LinuxThreads) gave every thread its own process ID, which broke signals and ps. The CLONE_THREAD flag (Linux 2.4.0) and the NPTL library (Native POSIX Thread Library, Linux 2.6) fixed that with thread groups: many tasks, one process ID.
• Kernel threads are still not free. Each has a kernel stack, a user stack and a scheduler entry. A server with 100,000 mostly idle connections does not want 100,000 of them. That produced event loops (one thread, many sockets, epoll) and later user-space threads: goroutines in Go (2009) and virtual threads in Java (final in JDK 21, 2023).

How it works

What each one owns

	Process	Thread
Virtual address space (code, heap, mapped files)	own	shared with the other threads of the process
File descriptor table	own	shared
Credentials (UID, GID, capabilities)	own	per task in the kernel; the C library keeps them equal across threads
Signal handlers	own	shared
ID	PID	TID (thread ID)
Stack	own	own
CPU registers, program counter	own	own
Signal mask	own	own
Thread-local storage (TLS)	n/a	own
Scheduling policy, priority, CPU affinity	own	own

Green boxes are shared by every thread of process A, so a bad pointer in one thread corrupts all three. Processes B and C share nothing by default, so a crash in B leaves C running.

One kernel object, many flags

1. Everything is a task. The kernel schedules task_struct objects. It has no separate “thread” type. What user space calls a process is a thread group: one or more tasks that share an address space and a thread group ID.

2. clone() decides what is shared. Each flag makes the child share one resource with the caller instead of getting a copy.

   Flag	Child shares with the caller	Since
   CLONE_VM	the memory space	Linux 2.0
   CLONE_FILES	the file descriptor table	Linux 2.0
   CLONE_FS	root directory, current directory, umask	Linux 2.0
   CLONE_SIGHAND	the table of signal handlers (needs CLONE_VM since 2.6.0)	Linux 2.0
   CLONE_THREAD	the thread group, so the same PID (needs CLONE_SIGHAND since 2.5.35)	Linux 2.4.0
   CLONE_SETTLS	nothing: sets the new thread’s TLS descriptor	Linux 2.5.32
   CLONE_NEWNS, CLONE_NEWPID, CLONE_NEWNET …	the opposite: puts the child in a new namespace	see Containers

3. fork() is clone() with no sharing flags. The child gets a copy of the address space, the fd table and the signal handlers. Open files are still the same open file descriptions, so parent and child share file offsets.

4. Copy-on-write makes fork cheap. The kernel does not copy memory pages. It copies the page tables and marks the pages read-only in both processes. The first write to a page by either side triggers a page fault, and the kernel copies that one page. A process with 10 GB of heap forks in the time it takes to copy its page tables, and a child that calls exec() at once copies almost nothing.

5. exec() replaces the program, not the process. execve() loads a new binary into the same task: new code, heap and stack, same PID, same parent. File descriptors stay open unless marked close-on-exec (O_CLOEXEC). If a multi-threaded process calls execve(), all other threads are destroyed first.

6. A thread is clone() with everything shared. pthread_create() calls clone() with CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SIGHAND | CLONE_THREAD plus TLS and TID bookkeeping flags. The new task gets its own stack, allocated by the thread library with mmap(). The default size comes from the RLIMIT_STACK soft limit at program start (commonly 8 MiB), or 2 MB on most architectures when that limit is unlimited. It is virtual memory: pages are backed only when touched.

7. clone3() (Linux 5.3) is the extensible version of the call. It added, among other things, a way to get a pidfd for the child and to start the child directly in a target cgroup (Linux 5.7).

PID, TGID, TID

Name	What the kernel calls it	What it identifies	How to read it
TID	pid field of the task	one task (one thread)	gettid(), ps -L column LWP, entries in /proc/<pid>/task/
TGID	tgid field	the thread group, which is what user space calls the PID	getpid(), ps column PID, Tgid: in /proc/<pid>/status
PPID	parent’s TGID	who gets SIGCHLD and must wait()	getppid(), PPid: in status
PGID, SID	process group, session	job control, terminal signals	ps -o pgid,sid

The first thread of a process has TID equal to TGID and is the thread group leader. TIDs and PIDs come from one number space, capped by /proc/sys/kernel/pid_max. The default is 32768, and 64-bit systems can raise it to 2^22 (about 4 million).

Process states

The state letters in ps and top:

Letter	State	Meaning
R	running or runnable	on a CPU or on a run queue waiting for one
S	interruptible sleep	waiting for an event (a socket, a timer, a lock); a signal wakes it
D	uninterruptible sleep	waiting in the kernel, usually for disk or NFS I/O; signals do not wake it, and it counts towards load average
T	stopped	by a job control signal (SIGSTOP, SIGTSTP); SIGCONT resumes it
t	tracing stop	stopped by a debugger
Z	zombie (defunct)	exited, but the parent has not collected the exit status
I	idle	an idle kernel thread

Exit, zombies and orphans

1. A process exits. The kernel frees its memory and closes its files, but keeps the task entry with the exit status. That entry is a zombie. It uses no memory to speak of, only a slot in the process table and a PID.
2. The kernel sends SIGCHLD to the parent. The parent calls wait() or waitpid(), reads the status, and the zombie disappears. This is reaping.
3. If the parent never calls wait(), zombies pile up until the parent itself exits.
4. If the parent exits first, the children are orphans. The kernel re-parents each one to the nearest living ancestor that marked itself a subreaper (prctl(PR_SET_CHILD_SUBREAPER)), or else to the init process of its PID namespace, PID 1. PID 1 must therefore call wait() in a loop. systemd does. An application that was never written to be an init usually does not.

Context switches

A context switch saves the registers of the outgoing task, switches kernel stacks and loads the registers of the incoming task. What else it costs depends on what the two tasks share.

• Two threads of one process. Same address space, so the page tables stay loaded and the translation lookaside buffer (TLB, the CPU cache of virtual-to-physical address translations) stays valid.
• Two different processes. The kernel also switches page tables (on x86, it loads a new value into the CR3 register). TLB entries of the old process no longer apply. Modern x86 CPUs tag TLB entries with a process-context identifier (PCID), so the kernel can skip a full flush, but the new process still starts with few of its own translations cached.
• Both cases. The CPU data caches hold the old task’s data, so the first microseconds of the new task run on cache misses. This indirect cost is often larger than the switch itself.
• User-space threads. A goroutine or coroutine switch is a function call inside one process: save a few registers, change the stack pointer. No kernel entry, no page table change.

/proc/<pid>/status counts both kinds: voluntary_ctxt_switches (the task blocked) and nonvoluntary_ctxt_switches (the scheduler preempted it).

The scheduler

• Scheduling classes. Each task has a policy. SCHED_DEADLINE and the real-time policies SCHED_FIFO and SCHED_RR always run before normal tasks. Normal tasks use SCHED_OTHER (also called SCHED_NORMAL), SCHED_BATCH or SCHED_IDLE.
• CFS, 2007 to 2023. The Completely Fair Scheduler was merged in Linux 2.6.23. It tracked a virtual runtime per task and always ran the task that had received the least.
• EEVDF, since Linux 6.6. Earliest Eligible Virtual Deadline First replaced the CFS algorithm in Linux 6.6 (released 29 October 2023), and Linux 6.12 completed the move. The algorithm comes from a 1995 paper. Each task has a lag: the CPU time it is owed compared with its fair share. A task with lag of zero or more is eligible. Among eligible tasks the scheduler picks the one with the earliest virtual deadline, which depends on the task’s time slice. A task that asks for a shorter slice (through sched_setattr()) gets the CPU sooner but for less time, which helps latency-sensitive work without extra heuristics.
• Nice values (-20 to 19) still set the weight, the share of CPU a task gets relative to others. cgroup cpu.weight does the same for groups, see Containers.
• sched_ext, since Linux 6.12. A scheduling class whose policy is written as an eBPF program and loaded at run time. See eBPF.

Kernel threads

Kernel threads are tasks with no user address space. They run only kernel code. ps shows them in square brackets: [kthreadd] (PID 2, the parent of all of them), [ksoftirqd/0], [kswapd0], [kworker/3:1]. They cannot be killed with signals, and their CPU time shows as system time. High CPU in kswapd means memory pressure. High CPU in ksoftirqd usually means heavy network receive load.

In containers

A container is not a kernel object. It is one or more ordinary processes started with namespace flags and placed in a cgroup. Details are in Containers, namespaces and cgroups. Two process facts matter daily:

1. The entrypoint is PID 1 of a new PID namespace, and PID 1 is special in two ways.
   • Signals sent from inside the namespace reach PID 1 only if it installed a handler for them. The default action (terminate) does not apply. So SIGTERM to an entrypoint without a handler does nothing. SIGKILL and SIGSTOP from the parent namespace always work.
   • Orphans in the namespace are re-parented to it, so it must reap them. When PID 1 exits, the kernel kills every other process in the namespace with SIGKILL.
2. Stopping a container sends SIGTERM to PID 1, waits, then sends SIGKILL. Kubernetes waits terminationGracePeriodSeconds, 30 seconds by default. An entrypoint that ignores SIGTERM turns every rollout into a 30-second wait followed by exit code 137.

This is why tini exists. It is a minimal init: it starts the application as a child, forwards signals to it and reaps zombies. docker run --init puts docker-init, which is tini, in as PID 1. In Kubernetes, shareProcessNamespace: true makes the pod’s pause container PID 1 for all containers, and it reaps orphans.

The usual way to get this wrong is the shell form of ENTRYPOINT or CMD in a Dockerfile. It runs /bin/sh -c "app", so the shell is PID 1 and the application never sees SIGTERM. Use the exec form (ENTRYPOINT ["app"]) or exec app at the end of a wrapper script.

Compare

Units of execution.

	Process	Kernel thread (1:1)	Green thread, goroutine (M:N)	Coroutine, async task
Scheduled by	kernel	kernel	language runtime, on top of a few kernel threads	event loop in user space; the task yields at await
Memory isolation	yes, own address space	none inside the process	none	none
Creation cost	highest: new page tables, fd table, kernel structures	lower: new stack and task, shared address space	low: a small growable stack in the heap	lowest: one heap object
Switch cost	kernel entry plus address space switch and TLB effects	kernel entry, no address space switch	function call in user space	function call in user space
Uses several cores	yes	yes	yes, if the runtime runs several kernel threads	no, one loop is one thread; run several loops or processes
Preemption	yes	yes	depends on the runtime (Go preempts; others yield at blocking calls)	no, cooperative: one long computation blocks the loop
Crash blast radius	one process	the whole process	the whole process	the whole process
Communication	pipes, sockets, shared memory, files	shared memory with locks and atomics	channels or shared memory	plain shared state, safe between await points
Examples	PostgreSQL backends, nginx workers, Chrome tabs	pthreads, Java platform threads, Rust std::thread	Go goroutines, Java virtual threads, Erlang processes	Python asyncio, JavaScript promises, Rust async with tokio

Concurrency models in real software.

Software	Model	What it means in operation
nginx	one master process plus worker processes; each worker runs a single-threaded event loop over non-blocking sockets	worker_processes auto gives one worker per core. A worker handles thousands of connections. A blocking call inside a worker stalls all of them.
Apache httpd	pluggable: prefork (process per connection), worker (threads), event (threads plus an event loop for idle keep-alive connections)	prefork is the most isolated and the heaviest in memory.
PostgreSQL	process per connection: the postmaster starts one backend process for each client; backends share data through shared memory and semaphores	A connection costs a process, so thousands of clients need a pooler such as PgBouncer. A crashed backend makes the postmaster restart the others to protect shared memory.
Java	platform threads are 1:1 kernel threads. Virtual threads (final in JDK 21, JEP 444) are M:N: the JVM mounts them on a small pool of carrier threads and unmounts them at blocking calls	Since JDK 24 (JEP 491) a virtual thread no longer pins its carrier inside synchronized. Thread-per-request code scales without rewriting to async.
Go	goroutines, M:N, preemptive. The runtime runs up to GOMAXPROCS goroutines in parallel	Since Go 1.25 the default GOMAXPROCS on Linux respects the cgroup CPU limit, not only the number of host CPUs. Earlier versions in a container with a small CPU limit ran too many threads and were throttled.
Python (CPython)	kernel threads, but the global interpreter lock (GIL) lets one thread run Python bytecode at a time. Parallel CPU work needs multiprocessing or native extensions that release the GIL	The free-threaded build has no GIL. It was experimental in 3.13 (PEP 703) and is officially supported since 3.14 (PEP 779). In 3.15 it is still an optional separate build, and the GIL build is the default. Single-thread overhead is about 1% to 8%. An extension module that is not marked safe turns the GIL back on.
Node.js	one thread runs JavaScript on an event loop; a libuv thread pool (4 threads by default, UV_THREADPOOL_SIZE) does file I/O, DNS lookups and some crypto	worker_threads run JavaScript in parallel for CPU-heavy work and can share memory through SharedArrayBuffer. They do not help I/O-bound work.

What breaks

Symptom	Cause	Check
fork: Resource temporarily unavailable, pthread_create failed: EAGAIN, unable to create native thread	a task limit: the cgroup pids.max, the per-user RLIMIT_NPROC (it counts threads), or the system-wide threads-max or pid_max	cat /sys/fs/cgroup/pids.max pids.current, ulimit -u, cat /proc/sys/kernel/threads-max, ps -eLf | wc -l
Many <defunct> entries	the parent does not call wait()	ps -eo pid,ppid,stat,comm | awk '$3 ~ /^Z/'; fix or restart the parent (the PPID), killing the zombie does nothing
Zombies in a container	the application runs as PID 1 and does not reap orphans	add tini (docker run --init) or shareProcessNamespace: true
Container takes 30 s to stop, exit code 137	PID 1 has no SIGTERM handler, or a shell is PID 1 and does not forward it	kubectl exec POD -- cat /proc/1/cmdline | tr '\0' ' '; grep SigCgt /proc/1/status
High load average, low CPU use	tasks in D state waiting on I/O	ps -eo state,pid,comm | grep '^D', cat /proc/pressure/io
One process at 100% of one core, others idle	a single-threaded event loop or a GIL	top -H -p PID shows which thread is busy
Latency spikes under a CPU limit	more runnable threads than the cgroup quota allows, so the kernel throttles	cat /sys/fs/cgroup/cpu.stat (nr_throttled), thread count from ps -o nlwp
Process cannot be killed with kill -9	it is a zombie (already dead) or stuck in D state in the kernel	ps -o stat,wchan:32 -p PID

Commands worth knowing by heart:

ps -eLf                           # every thread: PID, PPID, LWP (TID), NLWP (thread count)
ps -o pid,tid,nlwp,stat,comm -p 1234 -L
top -H -p 1234                    # per-thread CPU
ls /proc/1234/task                # one directory per TID
grep -E 'Tgid|Pid|PPid|Threads|VmRSS|ctxt' /proc/1234/status
pstree -p -t 1234                 # tree with PIDs and thread names
strace -f -p 1234                 # trace syscalls of all threads and future children
strace -f -e trace=clone,clone3,execve,exit_group ./app
cat /proc/sys/kernel/threads-max  # system-wide task limit, sized from RAM at boot
cat /proc/sys/kernel/pid_max
ulimit -u                         # RLIMIT_NPROC for this user, counts threads

Output shape of the status fields:

Tgid:   1234
Pid:    1234
PPid:   1
Threads:        17
VmRSS:     84212 kB
voluntary_ctxt_switches:        5121
nonvoluntary_ctxt_switches:     388

VmRSS is the resident memory of the whole process. Threads share it, so do not add it up per thread.

Related pages: eBPF for tracing what a process does, CrashLoopBackOff for exit codes in Kubernetes.

Keywords

process, thread, task, task_struct, clone, clone3, fork, vfork, execve, copy-on-write (COW), CLONE_VM, CLONE_FILES, CLONE_FS, CLONE_SIGHAND, CLONE_THREAD, thread group, PID, TGID, TID, LWP, NLWP, PPID, pid_max, threads-max, RLIMIT_NPROC, pids.max, NPTL, pthread, thread-local storage (TLS), process state R S D T Z, uninterruptible sleep, zombie, defunct, orphan, reaping, wait, waitpid, SIGCHLD, subreaper, PID 1, init, tini, docker-init, SIGTERM, SIGKILL, exit code 137, context switch, TLB, PCID, CR3, voluntary_ctxt_switches, scheduler, CFS, EEVDF, lag, virtual deadline, nice, SCHED_FIFO, SCHED_DEADLINE, sched_ext, kernel thread, kthreadd, kworker, green thread, goroutine, GOMAXPROCS, M:N scheduling, virtual threads, carrier thread, coroutine, event loop, epoll, GIL, free-threaded Python, worker_threads, process per connection.

Sources

• clone(2): flags, versions, thread groups, clone3
• fork(2) and execve(2)
• pthread_create(3): default stack size
• ps(1): process state codes
• pid_namespaces(7): PID 1 signal rules, orphans
• PR_SET_CHILD_SUBREAPER(2const)
• getrlimit(2): RLIMIT_NPROC counts threads
• proc_sys_kernel(5): pid_max, and kernel sysctl documentation: threads-max
• sched(7): scheduling policies
• EEVDF scheduler, kernel documentation
• Linux 6.6 release notes (EEVDF replaces CFS) and Linux 6.12 release notes (EEVDF completed, sched_ext)
• docker container run: specify an init process
• Kubernetes: share process namespace between containers in a pod
• nginx beginner’s guide: master and worker processes
• PostgreSQL: how connections are established
• JEP 444: Virtual Threads (JDK 21) and JEP 491: Synchronize Virtual Threads without Pinning (JDK 24)
• Go 1.25 release notes: container-aware GOMAXPROCS
• Python free-threading HOWTO (3.15) and PEP 779
• Node.js worker_threads