Stream processing: event time, watermarks and replay

Why it matters

Product analytics, fraud detection, monitoring and feature pipelines for ML all compute over event streams. The hard part is rarely throughput; it is correctness. Events arrive late and out of order, machines fail mid-window, and a retry must not count a purchase twice. Interviewers probe whether you can say what a result means at the moment it is emitted and what happens when a late event arrives.

Core ideas

Two clocks. Event time is when something happened, carried in the event; processing time is when the system sees it. The gap varies with network delay, mobile clients reconnecting and backlogs after an outage. Grouping by processing time is simple but attributes events to the wrong period whenever the gap varies; event-time windows give results that do not depend on arrival timing.

Windows. Tumbling windows are fixed and non-overlapping (per minute); sliding windows overlap (the last 10 minutes, updated every minute); session windows are per key and close after a gap of inactivity. A windowed aggregation keeps keyed state for each open window.

Watermarks. A watermark at time t asserts that no more events with event time at or before t are expected. When the watermark passes a window's end, the window can fire. A perfect watermark needs complete knowledge of the source's ordering; most are heuristics, such as the largest event time seen minus a bound on out-of-orderness. A conservative watermark adds latency; an aggressive one emits incomplete results and produces more late data. One idle input partition can hold back the watermark of the whole job unless it is marked idle.

Late data. Events behind the watermark are late. You can drop them, keep window state for an allowed lateness and emit corrected results, or send them to a side output for separate repair. Corrections require consumers that can handle updated or retracted results.

Triggers and accumulation. The Dataflow model separates what is computed (the aggregation), where in event time (the windowing), when in processing time results are emitted (triggers: at the watermark, early speculative firings, late firings) and how successive results relate (discarding, accumulating, or accumulating with retractions). This vocabulary lets you state a pipeline's semantics precisely.

Fault tolerance. Stream processors checkpoint operator state periodically. Flink-style engines inject barriers into the streams; an operator snapshots its state once barriers from all its inputs have arrived, which yields a consistent global snapshot without stopping the pipeline. After a failure, state is restored from the last checkpoint and sources rewind to the offsets recorded with it, which requires a replayable source such as a durable log. This gives exactly-once effects on internal state. External effects need more: idempotent writes keyed by a deterministic ID, or transactional sinks that commit output together with the checkpoint. Otherwise replay produces duplicates downstream.

watermark = max_event_time_seen - 30s        # bounded out-of-orderness
window [12:00, 12:05) fires once watermark >= 12:05
event with time 12:03 arriving later: late -> update, side output or drop

Common misconceptions

• "Exactly-once means each event is processed once." After failures events are reprocessed; exactly-once describes the effect on state, and on outputs only with suitable sinks.
• "Watermarks guarantee completeness." Most are estimates, so late data must still be handled explicitly.
• "Processing-time windows are fine if the system is fast." Backlogs, retries and client delays still move events between windows.
• "Streaming replaces batch." Many pipelines pair a streaming path for freshness with batch recomputation for corrections; the Dataflow model treats batch as the bounded case of the same semantics.
• "More frequent checkpoints are always better." They shorten replay after failure but cost I/O and can add latency.

How to approach it

Start with Streaming 101 and 102 for intuition, then read the Dataflow paper's sections on windowing, triggers and refinements. Implement one windowed aggregation in a real engine (the Flink training's Hourly Tips exercise is a good size) with an explicit watermark strategy, inject late events and observe which outputs change. In interviews, pin down the metric's semantics first (which time, how late data can be, whether results may be revised), then choose windows, watermark and sink, and tell the failure story end to end.