Elasticity, autoscaling and serverless cold starts

Why it matters

Elastic systems add and remove capacity as load changes. The hard parts are reacting quickly without oscillating, and choosing between paying for idle capacity and paying cold-start latency. Serverless platforms make that trade-off explicit through keep-alive and pre-warming policies, and microVMs such as Firecracker make fast, isolated startup possible. Research and infrastructure interviews ask you to reason about both the control loop and the cost of starting things.

Core ideas

Elasticity is a control loop with delays. Observe a metric, decide a target, act, wait for the effect. Each stage adds delay: metric collection and averaging, the controller's period, scheduling the new instance, provisioning a node when none has room (usually much slower than starting a pod on an existing node), pulling images, booting the sandbox, starting the runtime and initializing the application. Load that rises faster than this chain reacts lands in queues, so headroom, limits and load shedding remain necessary.

Kubernetes HPA. The controller periodically computes desired replicas = ceil(current replicas × current metric / target metric), skips changes within a tolerance band, treats unready pods and missing metrics conservatively, and smooths scale-down with a stabilization window; scaling policies bound how fast replicas change. CPU utilization is convenient but indirect; request rate, concurrency or queue length per replica track load more directly for latency-critical services (queue depth or KV-cache usage for LLM serving).

Oscillation. Aggressive scale-down with slow scale-up makes replicas flap. Asymmetric policies, stabilization windows and predictive scaling for known daily patterns reduce it.

Cold starts. A cold start pays for placement, sandbox creation, runtime start, code or model download and initialization code. Platforms shorten the sandbox part with lightweight VMMs, keep pools of pre-booted sandboxes (the Firecracker paper sizes such a pool with Little's law: pool size ≈ creation rate × creation latency) and restore from snapshots. Applications shorten their part by trimming dependencies and deferring work; for ML inference, loading weights can dominate.

Keep-alive and pre-warming. Keeping an instance warm after an invocation avoids the next cold start but holds memory. Shahrad et al. found in the Azure Functions workload that invocation rates span many orders of magnitude and that most applications are invoked rarely, so one fixed window fits badly: it wastes memory on rare applications and still misses their next call. Their hybrid histogram policy tracks each application's idle-time distribution and chooses a pre-warm delay and keep-alive window from its head and tail, falling back to a fixed window when data are scarce and to time-series forecasting when idle times exceed the histogram range. They evaluate cold-start percentage against wasted memory time.

Isolation and density. One microVM per function gives VM-level isolation; Firecracker argues that a minimal VMM keeps memory overhead and boot time low enough for this, and that soft allocation (oversubscription) makes idle instances cheap.

Geography and cost (awareness). Spreading capacity across regions lowers user latency and isolates failures but fragments pools (less statistical multiplexing), complicates data locality and changes prices, so autoscaling interacts with where users and data are.

Common misconceptions

• "Autoscaling removes the need for capacity planning." Reaction delays mean bursts hit queues first; headroom and limits are still required.
• "Cold starts are just image pulls." Sandbox boot, runtime start and application initialization often dominate, and weight loading dominates for ML services.
• "A longer keep-alive is a cheap fix." It helps frequently invoked applications and wastes memory on rare ones; per-application policies reach a better trade-off curve.
• "Scale to zero is free." It removes idle cost but guarantees cold starts after idle periods.
• "VMs are too slow for serverless." A specialized VMM combined with pools starts fast enough, at the cost of a deliberately limited device model.

How to approach it

Run the HPA walkthrough with a load step and timestamp each stage of scale-up. Simulate keep-alive policies on the Azure Functions trace and plot cold starts against idle memory. In interviews, decompose latency into the stages above and say which ones a proposed change shortens.