Resource scheduling, fairness, interference and cold starts

Why it matters

Utilization is money: a cluster with low average utilization wastes most of its hardware, yet packing more work per machine risks interference that shows up as tail latency in the most important services. Cloud infrastructure roles and cloud research ask how schedulers balance this and how isolation is enforced. If you have measured interference in coursework, for example a latency-critical cache next to batch jobs, this topic gives you the vocabulary to explain it precisely.

Core ideas

Placement. Cluster schedulers filter machines by constraints and free requested resources, then score candidates (spreading for availability, packing for efficiency). Because tasks rarely use all they request, production schedulers overcommit and lend reserved but unused resources to lower-priority work, preempting or evicting it when the owner needs them back.

Fairness. With one resource, max-min fairness maximizes the smallest allocation. With several (CPU and memory), dominant resource fairness equalizes each user's dominant share, the largest fraction of any single resource the user holds, and keeps useful properties such as giving users no incentive to misreport demand.

Interference. Colocated workloads share last-level cache, memory bandwidth, hyperthread siblings, network and disk queues, and sometimes power and frequency headroom. A batch job thrashing the cache can multiply a neighbor's tail latency while average CPU utilization looks fine. Countermeasures include CPU quotas and shares (cgroups), dedicated or pinned cores, cache and memory-bandwidth partitioning on hardware that supports it, network traffic shaping, detecting antagonists from per-task hardware counters such as cycles per instruction, and feedback controllers that throttle best-effort work as the latency-critical service nears its target.

Isolation inside services. Multi-tenant services face the same problem one level up: one tenant's burst or poisonous request can starve the rest. Per-tenant quotas and admission control bound the damage, and shuffle sharding maps each tenant to a small random subset of workers so that one tenant's trouble touches few others.

Isolation versus overhead. Containers are dense and start quickly but share a kernel; VMs isolate more strongly at higher overhead; microVMs aim for VM-grade isolation with near-container density and startup time, which is why some multi-tenant serverless platforms use them.

Cold starts. An invocation without a warm instance must provision an isolated environment, fetch code or an image, start the runtime and run initialization. Mitigations (keeping instances warm, pre-warming from predicted demand, snapshotting initialized state, lazy image loading) all trade idle resources or complexity for latency.

Common misconceptions

• "CPU limits isolate tenants." They cap CPU time but not cache, memory-bandwidth or I/O contention.
• "Higher utilization is always better." Past some point, interference and queueing raise tail latency faster than cost falls.
• "Fair means equal shares of each resource." Workloads need different mixes; multi-resource fairness is defined differently.
• "Cold starts are a fixed platform cost." They depend on image size, runtime initialization and keep-alive policy, all of which you can influence.

How to approach it

Read the Borg paper for scheduling and reclamation, CPI2 for detecting and throttling noisy neighbors, the shuffle-sharding article for tenant isolation inside services, Firecracker for isolation versus density, and the Lambda container-loading paper for cold starts; connect each to Kubernetes requests, limits and QoS. Work DRF by hand from the definition above; the research-cloud topics on scheduling and interference go deeper into fairness and colocation papers. In interviews, explain one experiment or design you know well in terms of the shared resource, the metric affected and the isolation mechanism that would address it, without overstating what your measurements showed.