Containers, VMs, namespaces and cgroups

Why it matters

Cloud platforms schedule containers and VMs, and much puzzling behavior inside them traces back to isolation mechanisms: a service with a CPU limit shows latency spikes at low average CPU, a pod is OOM-killed while its heap looks small, or co-located workloads slow each other down. If your coursework covered Kubernetes, resource allocation or interference, this topic supplies the kernel mechanisms behind those observations. Infrastructure, SRE and cloud-research interviews expect you to explain them and to compare containers with VMs and microVMs on isolation and overhead.

Core ideas

A container is a process. The Linux kernel has no container object. A runtime starts ordinary processes with four ingredients: namespaces (what the process can see), cgroups (what it can use), reduced privileges (capabilities, seccomp system-call filters, security modules) and its own root filesystem, usually image layers stacked with an overlay filesystem. All containers on a host share one kernel.

Namespaces give a process its own view of one kind of global resource: mount points, process IDs (the first process inside becomes PID 1), the network stack (interfaces, routes, ports), the hostname (UTS), System V IPC, user and group IDs (user namespaces map root inside to an unprivileged user outside), the cgroup tree and clocks.

cgroup v2 organizes processes in one hierarchy and attaches controllers:

• CPU: cpu.weight divides CPU proportionally, and only matters under contention. cpu.max sets a hard quota per period, for example 50 ms every 100 ms for half a CPU. A multithreaded process can burn its whole quota early in the period on several cores and is then throttled until the next period begins, which produces latency spikes even when average usage looks low; cpu.stat counts throttled periods and time. Kubernetes CPU limits map to this quota and CPU requests to weights.
• Memory: memory.max is a hard limit. Usage includes anonymous memory, page cache the group brought in and some kernel memory. Near the limit the kernel reclaims, clean page cache first; if it cannot reclaim enough, the OOM killer kills a process in that cgroup, visible as exit status 137 (SIGKILL) and in the oom_kill count in memory.events. memory.high throttles and reclaims earlier, before the hard limit.
• I/O and PIDs: bandwidth or IOPS limits and weights, and a cap on the number of processes that stops fork bombs. Pressure stall information (PSI) reports how long tasks wait for CPU, memory or I/O.

What isolation does not cover. cgroups meter CPU time and memory size, but co-located workloads still share last-level cache, memory bandwidth, kernel data structures and locks, and often network and disk queues. These are the usual sources of noisy-neighbor interference. Some processors can partition cache and memory bandwidth, but that is not configured by default.

Virtual machines. A hypervisor runs guest kernels on virtualized hardware. Virtualization extensions let guest code run directly until it executes a sensitive instruction, which traps to the hypervisor; nested page tables translate guest-physical to host-physical addresses in hardware; paravirtual virtio devices replace slow device emulation. The isolation boundary is a narrow, hardware-enforced interface rather than the full system-call surface, so VMs isolate better, at the cost of a guest kernel's memory, slower boot and lower density.

MicroVMs and sandboxes. Firecracker is a minimal virtual machine monitor built on KVM with a very small device model, designed to boot quickly with little per-VM memory overhead so multi-tenant serverless platforms can give every function VM-grade isolation. gVisor intercepts system calls in a user-space kernel, and Kata Containers runs containers inside lightweight VMs. Each trades some performance for a smaller shared attack surface.

Common misconceptions

• "A container is a lightweight VM." It is a set of processes sharing the host kernel; a kernel exploit can cross the boundary.
• "A CPU limit of 1 means one dedicated core." It is a time quota across all cores per period; use cpusets for dedicated cores.
• "Low average CPU means there is no CPU problem." Quota throttling within periods causes latency; check cpu.stat.
• "The memory limit only counts my heap." Page cache, tmpfs files and some kernel memory are charged too.
• "Root inside a container is harmless." Without user namespaces it is host root with fewer capabilities, and privileged containers or a mounted runtime socket can escape.
• "VMs fully isolate performance." They still share caches, memory bandwidth and storage with other tenants.

How to approach it

Run the experiments in the practice guidance on a Linux VM; seeing throttling counters and an OOM kill makes the explanations concrete. For interviews, be ready to draw the container recipe (namespaces, cgroups, privileges, root filesystem), explain throttling with a timeline of quota periods, and argue when you would choose a container, a VM or a microVM for untrusted code.