Virtual memory and memory management

Why it matters

Virtual memory is behind isolation between processes, cheap process creation, memory-mapped files and the gap between "allocated" and "actually used" memory. In practice it appears as questions like: why does a process show 30 GB of virtual memory but use 2 GB, why was the container OOM-killed when its heap was small, why does a forked snapshot double memory under write load, and why did huge pages speed one workload up and add latency spikes to another. Interviewers check that you can move between the textbook mechanism and these production symptoms.

Core ideas

Translation. A virtual address splits into a page number and an offset. The page table maps page numbers to physical frames plus permission bits (present, writable, user, executable) and status bits (accessed, dirty). With 4 KiB pages and 9-bit indexes per level, x86-64 walks four levels (five with the larger address-space extension) and RISC-V Sv39 walks three. Multi-level tables allocate lower levels only for regions in use, which is what makes huge, sparse address spaces affordable.

The TLB. Each translation could cost several memory accesses, so the CPU caches recent ones in the TLB. TLB reach, entries times page size, bounds how much memory you can touch without misses; 2 MiB and 1 GiB pages multiply it. Address-space identifiers (PCID on x86, ASIDs elsewhere) avoid flushing on every process switch. When the kernel unmaps or write-protects a page that other cores may have cached, it must interrupt them to invalidate their entries, a TLB shootdown, which is one reason munmap and mprotect are not free in multithreaded programs.

Page faults are normal. A fault means the hardware could not complete a translation, and the kernel decides what it means. A minor fault needs no I/O: first touch of a fresh anonymous page, a file page already in the page cache, a copy-on-write copy. A major fault must read from storage, from swap or a file. An access outside any valid mapping becomes a segmentation fault. Linux counts both kinds per process, visible for example through /usr/bin/time -v.

Lazy allocation and overcommit. A large malloc usually just maps address space; physical frames are assigned on first touch. Linux overcommits by default within a heuristic limit, so running out of memory often shows up later as the OOM killer choosing a victim rather than as an allocation returning NULL.

Copy-on-write. After fork(), parent and child share frames marked read-only. The first write to such a page faults; the kernel copies it and maps the copy writable. Frames now have several owners, so the kernel reference-counts them and frees a frame only when its last mapping disappears. A fork-based snapshot stays cheap only while few pages are written.

File-backed versus anonymous memory. The page cache and memory-mapped files hold pages that can be dropped if clean or written back if dirty. Anonymous memory (heap, stacks) can only be reclaimed by swapping; without swap, sustained pressure ends in OOM kills. Replacement policies approximate LRU with accessed bits, as in the clock algorithm, and when the working set exceeds memory the system thrashes.

Allocation and fragmentation. Internal fragmentation is waste inside an allocated block, from rounding up to a size class or a page. External fragmentation is free memory split into pieces too small for a request. Paging removes external fragmentation for ordinary page-sized mappings, but it returns for huge pages and physically contiguous buffers, which is why the kernel compacts memory. User-space allocators use size classes and per-thread caches to stay fast under concurrency, accepting some internal fragmentation and retained memory.

Common misconceptions

• "Virtual memory means swapping to disk." Its main jobs are translation and isolation; many servers run without swap.
• "If malloc succeeded, the memory is there." With lazy allocation and overcommit, frames are committed on touch, and failure can surface later as an OOM kill.
• "A page fault means a disk read." Most faults are minor and part of normal operation; only major faults wait for storage.
• "RSS is the memory my process uses." RSS counts shared pages in every process that maps them and excludes page cache the process caused; proportional set size and cgroup accounting answer different questions.
• "Huge pages are always faster." They raise TLB reach but can add internal fragmentation, compaction stalls and expensive copy-on-write of large pages, which is why some latency-sensitive services disable transparent huge pages.
• "Paging eliminates fragmentation." Only external fragmentation for page-granular mappings; internal fragmentation and contiguous-allocation failures remain.

How to approach it

Do one translation by hand for x86-64 or Sv39, splitting the address into indexes and offset and walking the levels, and compute how much page-table memory a small sparse process needs. Then make faults visible: touch a large array once and compare minor-fault counts, and fork a process with a big heap and write to it while watching memory grow. Graduate interviews focus on translation, TLBs and faults; mid-level ones on copy-on-write, overcommit and reading memory metrics; senior and infrastructure interviews on reclaim, OOM behavior under container limits, NUMA and huge pages.