Stream Processing: Taming Infinite Event Sequences in Real Time

Log as Storage

Each topic partition is an append-only log file. Producers append records; consumers read from any offset. Sequential disk I/O enables very high throughput — OS page cache makes reads fast without additional in-memory buffering.

Partitions for Parallelism

A topic is split into N partitions, each assigned to a broker. Producers choose a partition by key hash or round-robin. Each consumer group assigns partitions to consumers — N partitions = N parallel consumers in a group.

Consumer Offsets

Each consumer group tracks its read position (offset) per partition. Offset is committed back to Kafka (or stored externally). This enables replaying from any offset, independent consumer groups, and seamless consumer restarts.

Object Storage Tiering

Segments older than the hot retention period are tiered to S3/GCS. Consumers can still read from tiered storage — infinite retention at low cost. Kafka's Tiered Storage (since 3.x) makes this production-ready.

Event-time vs. Processing-time

Events have two timestamps: event time (when the event occurred, embedded in the event payload) and processing time (when the stream processor saw the event). Network delays, retries, and consumer lag cause events to arrive out of order relative to event time.

Approach: Use event time for windowing; track watermarks (estimated completion of a time window) to decide when to close and emit window results. Late-arriving events can be handled via allowed lateness or dedicated late data sinks.

Tumbling Windows

Fixed-size, non-overlapping time windows. Each event belongs to exactly one window. E.g., "aggregate all events in each 1-minute window."

Approach: Group events by floor(event_time / window_size). Emit result when watermark passes the window boundary.

Sliding Windows

Fixed-size windows that overlap. An event may belong to multiple windows. E.g., "count events in the past 5 minutes, updated every 1 minute."

Approach: More memory-intensive. Each event is assigned to multiple buckets. Flink's sliding window operator handles this natively.

Session Windows

Variable-length windows defined by periods of inactivity. A session ends after a gap in events exceeding a configured timeout. Used for user session analytics.

Approach: No fixed duration. Window state must be merged when late events arrive within the timeout gap of a closed window.

Stream-Table Join

Enrich a stream of events with data from a slowly-changing reference table (e.g., add product details to a stream of purchase events).

Approach: Load the table into a local hash map in each stream processor instance. Table updates are applied as they arrive (table is subscribed as a changelog stream).

Stream-Stream Join

Join two streams where events from both streams must be correlated within a time window. E.g., match ad impressions with subsequent ad clicks.

Approach: Buffer events from both streams in a per-key state store within the join window duration. When matching event arrives, emit joined record. Evict expired buffered events.

At-least-once

Messages may be processed multiple times (if consumer crashes before committing offset). Acceptable if processing is idempotent (applying it twice has the same effect as once). Used by most simple stream pipelines.

At-most-once

Messages may be skipped but never processed twice. Offset is committed before processing. Acceptable for lossy analytics (a dropped metric doesn't matter), never for financial transactions.

Exactly-once

Each message is processed exactly once, even if failures occur. Achieved via: (1) idempotent writes + offset-as-part-of-output, or (2) distributed transactions across consumer offset store and output sink. Flink uses periodic checkpointing + transactional sinks.

A Stream is an Unbounded Batch

The conceptual difference between batch and stream is purely the boundedness of the input. A batch job processes a finite dataset; a stream processor processes an infinite sequence of events. Many stream processing frameworks (Flink, Spark) unify both under the same API.

The Log is the Database

Kafka's log-based message broker is architecturally equivalent to a database's replication log. The current state of a database is the accumulated result of applying all past writes in order. The log IS the source of truth — the table is just a projection. This is the key insight behind event sourcing and CDC.

Dual-Write is a Race Condition

Writing the same data to two systems (e.g., database AND search index) introduces a race condition: the database write succeeds but the search write fails, or vice versa. CDC solves this: write only to the database; treat the replication log as the event stream that propagates changes to all other systems.

Consumer Lag is Not Always Bad

With log-based brokers, a slow consumer simply falls behind — it doesn't slow down producers or other consumers. Lag can be monitored and alerts set on it. Consumers can catch up during low-traffic periods. This property enables batch and stream processing to share the same message bus.

Watermarks Are Estimates, Not Guarantees

A watermark of "event time T" means the stream processor believes all events with timestamps ≤ T have been received. This is an estimate — late-arriving events can arrive after the watermark, requiring a decision: include in the window (recompute) or route to a late data sink. There is no perfect watermark in a real distributed system.

State is the Hard Part of Stream Processing

Stateless stream processing (filter, map, project) is simple. The hard problems are stateful: windowed aggregations, joins across streams, maintaining running totals. State must be persistent (survives processor restarts), correctly keyed (each key's state must be co-located), and queryable. Flink's RocksDB state backend is designed specifically for this.