Model serving, monitoring and drift

Why it matters

A model that scores well offline only creates value if it serves predictions within latency and cost budgets, keeps working as data changes, and fails safely. Production-focused questions in ML and applied AI interviews target exactly this: how you would deploy, what you would monitor, and what happens when a model provider is slow or the input distribution shifts. Cloud and SRE knowledge transfers directly, with ML-specific additions.

Core ideas

Serving patterns. Batch inference precomputes predictions on a schedule: cheap and simple, but stale. Online inference answers each request within a latency budget. Streaming updates features or predictions from event streams. For LLM features the main choice is a hosted API (fast to start, but provider limits and data-handling terms apply) versus self-hosted open models (more control and possibly lower unit cost at scale, but you own GPUs, scaling and upgrades).

Latency and cost levers.

• Batching raises accelerator utilization and throughput at the cost of queueing delay; LLM servers use continuous batching, adding and removing sequences at every decoding step.
• Caching: exact-match result caches for repeated requests, provider prompt caching for long shared prefixes, and caches for embeddings and retrieval results. Semantic caches that match similar queries can return wrong answers, and every cache must be scoped by tenant.
• Model size: route easy requests to smaller models and escalate hard ones; quantization and distillation cut memory and cost with quality losses you must measure.
• Output length: each generated token is a sequential decoding step, so long outputs usually dominate LLM latency (very long prompts shift cost to prefill); constrain verbosity and stream tokens to improve perceived latency.

Training-serving skew. If features are computed differently online and offline, or joined against tables that have since changed, the served model is not the one you evaluated. Share feature code between training and serving, log the features actually served and train on those logs where possible, and compare online and offline feature distributions.

Drift. Covariate drift: the input distribution changes (a new customer segment, new phrasing). Label shift: class priors change (the fraud rate doubles), which breaks calibration and thresholds. Concept drift: the relationship between inputs and outputs changes (what counts as spam evolves). Detect drift by comparing feature and prediction distributions with a reference window (population stability index, two-sample tests, or embedding-distance statistics for text), and confirm impact with outcome metrics once labels arrive. Labels are often delayed or never arrive, so monitor proxies such as user corrections, escalation rates and acceptance of suggestions, and sample outputs for human review. Treat a drift alert as a hypothesis: rule out a pipeline bug that changed the inputs before retraining.

Monitoring layers. Service health (latency percentiles, errors, saturation); data quality (schema, missing values, ranges); model behavior (prediction distribution, confidence, refusal rate for LLMs); outcomes (quality and business metrics); and cost per request. Alert on the few signals that indicate user harm.

Reliability for model calls. Treat a model like any remote dependency that can be slow, rate-limited, unavailable or wrong. Derive timeouts from the latency budget; retry only idempotent calls, with exponential backoff, jitter and a retry budget; fall back to a smaller model, a cached answer, a rule-based default or a clear "unavailable" message; queue requests client-side to respect provider rate limits; and use circuit breakers to stop hammering a failing dependency. Validate model outputs before they reach downstream systems.

Releases. Version models, prompts, data and feature code together. Gate releases on offline evaluations, then run in shadow (scoring live traffic without serving it), canary to a small share of traffic, and A/B test where the metric allows; keep rollback one step away. Watch for feedback loops, where a model's own outputs shape its future training data.

Common misconceptions

• "Healthy latency and error rates mean the model is fine." Models fail silently; monitor inputs, predictions and outcomes.
• "Detected drift means retrain now." First rule out pipeline bugs and confirm a performance impact; retraining on broken data makes things worse.
• "Bigger batches are always better." They raise throughput but add queueing latency; size them to the latency budget.
• "Retries make LLM calls reliable." Unbounded retries amplify outages and can duplicate side effects; use budgets and idempotency.
• "Good offline evaluation guarantees online performance." Skew, drift and user behavior differ; use shadow and canary releases.
• "A semantic cache is a free speedup." Similar is not identical; it can return wrong or cross-tenant answers.

How to approach it

Reuse your cloud and SRE vocabulary and add the ML layers: skew, drift, delayed labels and evaluation gates. Deploy a small model behind an API with logging and a drift report, and write a failure-mode table for an LLM feature. In interviews, structure answers as serving pattern, budgets, monitoring layers, failure handling and release process.