Skip to main content

Eight concepts. Then it is just data flow.

The model is deliberately small. If you understand keys and state ownership, the rest of the system follows from those two ideas.

01Vocabulary

The model, in the words we use for it

Stream
A named, partitioned sequence of events with a declared schema.
Key
The field that determines partition assignment, ordering scope and state ownership.
Operator
A unit of processing bound to a key, holding the state for the keys it owns.
State
Values retained per key between events, checkpointed against log offsets.
Window
A bounded grouping over event time, with an explicit policy for late arrivals.
View
The queryable projection of operator state, readable by key or range.
Subscription
An ordered feed of changes to a view for consumers that react rather than poll.
Workload
A deployed, versioned graph of operators with a defined start position.

02Working method

How building on it feels

A workload is defined as a graph: where it reads, what it computes per key, what state it keeps, and what it exposes. Deployment then becomes a versioning question rather than a migration.

Define

Declare the input streams and their keys, then the operators that transform them. Schema is declared at the boundary so malformed events fail at ingest, not in the middle of a pipeline.

Test against history

Because the log retains input, a new workload version can be run over past events before it takes production traffic.

Deploy

Ship the workload as a version with an explicit start offset — from now, from a timestamp, or from the beginning.

Observe

Follow per-partition progress, checkpoint state and state size per operator while it runs.

Evolve

Add a new derived view without disturbing existing consumers; it builds itself from the same log.

03Evaluation

What an evaluation involves

  1. 01

    Scope

    One real flow, chosen because it is currently painful.

  2. 02

    Design

    Keys, ordering requirement, state size and read pattern agreed in writing.

  3. 03

    Run

    The workload is built and run against representative data in your environment.

  4. 04

    Assess

    You measure the result on your own infrastructure — we do not ask you to accept ours.

Next step

Bring a real stream to the first conversation.

Volume profile, key cardinality, ordering requirement and read pattern are enough for us to be specific.