Stream processing handles data continuously as it arrives. Instead of collecting data into batches and processing at intervals, stream processors analyze each event as it flows through the system. A credit card transaction is evaluated for fraud in milliseconds. A sensor reading triggers an alert when it crosses a threshold. A clickstream updates a recommendation model in real time. The processing happens in motion, not at rest.

The architecture differs from batch. Data arrives through message queues like Kafka or Kinesis. Stream processors like Flink, Spark Streaming, or Kafka Streams consume the events, apply transformations, and emit results. The system must handle late-arriving events, out-of-order data, and exactly-once semantics. Those are hard problems. Late events require watermarks to decide when a time window is complete. Out-of-order events require buffering and reordering. Exactly-once processing requires coordination between the source, the processor, and the sink. Getting these right is complex. When they are wrong, results are inconsistent or duplicated. Stream processing is powerful for use cases that need immediate insight. It is overkill for nightly reports. The choice between batch and stream depends on the latency requirement. If minutes or hours are acceptable, batch is simpler and cheaper. If seconds matter, stream is the only option.

Stream processing characteristics

• Continuous — processes events as they arrive
• Low latency — milliseconds to seconds
• Stateful — maintains state across events
• Complex — handles late and out-of-order data
• Scalable — distributes across many nodes

Stream processing is batch processing turned inside out. Instead of waiting for data to accumulate, it acts on each event as it appears.