Engineering notes

Total ordering across a whole stream requires a single point through which every event passes. That point becomes the throughput ceiling and the availability risk, and it buys a guarantee most applications never use: almost no business rule depends on the relative order of two unrelated accounts.

The requirement is nearly always per-subject. Deposits and withdrawals on one account must be applied in sequence. A device's state transitions must be applied in sequence. Across subjects, concurrency is not just acceptable — it is the reason the system scales.

The practical work is therefore choosing the key. A key that is too coarse (region, tenant) recreates the bottleneck; a key that is too fine (event id) destroys the ordering you needed. The right key is the entity your invariants are written about.

A processing step that looks up context from an external database on every event has state — it just lives elsewhere, with independent latency, independent availability and no coupling between the offset consumed and the value read.

When state sits next to the computation and is committed together with the offset that produced it, recovery has a defined meaning: resume from the checkpoint and both the position and the derived value agree. When it sits behind a network call, recovery becomes a reconciliation exercise.

The cost is that state must be owned. Partition ownership moves, state moves with it, and state size becomes an operational quantity you plan for per key. That is a trade we consider worth making explicit rather than hiding.

If input history is retained, a new derived view is not a migration project. It is a workload that starts at an earlier offset and catches up. Nobody has to backfill by hand, and nobody has to defend a one-off script.

The same property makes changes testable against reality. A new version of a computation can be run over recorded events and compared with the current one before it takes production traffic.

This only holds if replay is a first-class operation with an explicit start position, and if derived state is genuinely reproducible from the log rather than accumulated through side effects. Systems that mutate external records during processing lose the property quietly.