Skip to content

The logs already knew the order

The companion post, The test that knows what it is, made one promise it did not cash: it introduced the causation_id -- a correlation id threaded through every hop of a single action -- and said it was what lets an agent "stitch one logical action across the TABLE, PLAYER, and REFEREE logs into a single ordered story." But it stopped at one device. This post is about what that spine is for once the logs leave the device.

Here is the move. Phones, tablets, and TV browsers fail in places where you cannot attach a debugger -- a TV in someone's living room drops a WebSocket frame, and the bug is now three rooms away from any dev console. So the client keeps a bounded, redacted ring buffer of its own events and uploads it to the server after an error or on page close. Now the server holds the device's timeline next to its own. The naive next step -- merge the two and sort by timestamp -- is wrong, and wrong in a way that produces causally impossible orderings. The whole design is the alternative: don't sort by the clock, sort by the causal spine the logs were already carrying. Do that, and per-device swim-lane views and automatic dropped-message detection fall out nearly for free. That is the claim; the rest of this post is why it holds.

Why wall-clock sorting is a trap

Two devices, two clocks, and nobody synchronized them. A phone's clock and a TV's clock can disagree by hundreds of milliseconds to several seconds -- NTP drift, a paused background tab, a cheap TV SoC that never had the right time to begin with. None of that is exotic; it is the default state of a room full of consumer devices.

So consider an action: a PLAYER taps a card. The phone sends the move, the server applies it and broadcasts the new state, the TV receives it and renders. Causally, the send must precede the receive -- the TV cannot receive a message that was never sent. But if the phone's clock runs 800 ms ahead of the TV's, the receive event's timestamp can be numerically smaller than the send event's. Sort the merged log by timestamp and you get a timeline where the TV receives before the phone sends. That is not a rounding artifact you can squint past; it is a story that inverts cause and effect, and an agent (or a human) reading it to debug a dropped frame will chase a ghost.

The fix is to stop trusting the clocks for cross-device ordering and trust the server instead.

The spine: seq as a total order, causation_id as the join

The server is authoritative -- CLAUDE.md's architecture rules say so, and the log design leans on it. Every action the server applies gets a monotonic seq. That seq is an authoritative total order: it does not matter what any device's clock thinks, because the server stamped these events itself, in the order it actually processed them. Sorting cross-device events by server seq is sorting by the one clock everyone shares.

Within that order, the fan-out of a single action is held together by the causation_id. The server derives it (documented in the server docs) as:

client-msg-:server-seq-

with the client segment set to client-msg-none when there is no inbound client sequence (an AI move, a server-initiated transition). The same id is logged on the client's client.ws_message_sent, the server's action.apply, the server's room.broadcast_state, the outbound state_update payload, and every device's client.ws_message_received / client.room_state_delivered. One action, one id, every hop.

That gives you two keys that do all the work:

Key Role What it answers
server seq total order across all actions "which action came first?"
causation_id join key within one action's fan-out "which send, applies, broadcasts, and receives belong together?"

This is distributed tracing in miniature: causation_id is the trace/span id, client_session_id (the per-device session) is the lane. The system did not bolt tracing on after the fact -- the companion post's closed log vocabulary already threaded these fields through the round trip so the multi-device timeline specs could assert against them. Cross-device interleaving is the same fields read a second way.

Placing the events that have no seq

Not every event is part of a round trip. A route change, a wake-lock acquire, a local render -- these are purely client-local. They carry no server seq and no causation_id, because the server never saw them. You cannot join them to the spine. So where do they go on the timeline?

This is the one place the device clock has to come back into the picture -- but not as a source of truth, as a measured offset. The viewer estimates each device's clock offset from the round trips it already participated in. For a send/receive pair around a known server event, Cristian's algorithm gives the offset as roughly:

offset = server_time - (device_send_time + device_receive_time) / 2

with the error bounded by RTT / 2 (the round trip's half-width -- the message could have crossed the wire at either end of that interval). Each round trip the device did is one sample of its offset; average them and you have a per-device correction. Apply it to a local-only event's raw timestamp and you can drop that route change between the two server actions it actually happened between, even when the device clock is seconds off. The local events get placed on the server-time axis without ever being trusted to order across devices.

So the rule is layered: causal events are ordered by seq and grouped by causation_id (no clock involved at all), and local-only events are interpolated onto that same axis via an offset that was measured from the causal events, not assumed. The clock is an input to a correction, never the ordering authority.

The picture falls out: swim lanes and a round-trip DAG

Once events are grouped by causation_id and totally ordered by seq, the visualization is almost a transcription of the data shape rather than a new design.

  • Lanes. One lane per client_session_id, plus a server lane. Each device's events live in its own horizontal track.
  • Nodes. A causation group -- everything sharing one causation_id -- is a node, positioned by its server seq.
  • Edges. The fan-out of one action: PLAYER client.ws_message_sent -> server action.apply -> server room.broadcast_state -> each device's client.ws_message_received. That send-apply-broadcast-receive chain is the round-trip DAG.

You do not have to invent a layout, because the keys already encode it: seq is the x-axis, client_session_id is the lane assignment, causation_id is which nodes connect. This is what the companion post meant by the data shape making the picture nearly free -- the same property that made the test taxonomy mechanically queryable makes the timeline mechanically drawable.

Drawn out, with one action's fan-out expanded, the shape looks like this:

  lane \ server seq ->    41                  42
  ------------------------------------------------------------
  server                  apply, broadcast    apply, broadcast
  phone (sA)              sent, received      received
  tv    (sB)              MISSING receive     received
                          ( dropped frame )
  ------------------------------------------------------------
  seq 41  cid = client-msg-7:server-seq-41     (a PLAYER tap)
  seq 42  cid = client-msg-none:server-seq-42  (an AI / server move)

  one action's round trip (the seq-41 group) is a small DAG:

  phone (sA) : client.ws_message_sent
                     |
                     v
  server     : action.apply            (server stamps seq 41)
                     |
                     v
  server     : room.broadcast_state
                     |
            +--------+--------+          one broadcast,
            |                 |          fanned out down
            v                 v          every live lane
  phone (sA) :            tv (sB) :
  ws_message_received     MISSING ws_message_received  -> dropped frame

Lanes are client_session_ids plus a server lane; the x-axis is server seq; each causation group is one node placed at its seq; and the lane that should hold a receive but does not is the hole the next section is about.

The viewer that does this is real and read-only:

the private project wrapper renders the device-log interleave view

It reads the uploaded device JSONL logs from .runtime/client-logs/ and the backend events from .runtime/backend.log, groups by causation_id, orders by seq, and writes the swim-lane/DAG artifact to stdout (or --output <path>). It never rewrites its source logs -- the logs are inputs, the view is a derived artifact. The implementation lives in the project scripts.

The payoff: dropped messages detect themselves

Here is where the causal grouping stops being a nicer way to read logs and becomes a bug detector.

A causation group is a fan-out: one action, broadcast to every connected device. So the group has an expected shape -- every device lane that should have received this broadcast has a client.ws_message_received in the group. Now flip it around: a causation group where a known device lane is missing its receive is a dropped WebSocket message, made visible without a human reading a single log line. The TV that silently dropped the frame shows up as a hole in the fan-out, at a specific seq, on a specific lane.

That is exactly the failure class this project cares about most. TV browsers (Tizen, webOS, Android TV) drop WebSocket connections frequently -- it is called out in CLAUDE.md as a first-class quirk. The old way to find a dropped broadcast was to notice the TV looked wrong, then squint at two logs whose clocks disagreed. The causal interleave turns it into a structural check: a complete fan-out has a receive on every live lane; an incomplete one names the lane and the action where the message vanished. The detection is asserted by a test with a deliberately dropped receive (tests/unit/test_device_log_interleave.py), and the ingest side of the pipeline -- the upload endpoint and its redaction boundary -- is pinned by tests/integration/test_client_log_ingest.py.

The detector is only as honest as the upload, which is why the upload path is deliberately defensive rather than authenticated. TABLE and OBSERVER clients carry no strong credentials, so POST /api/diagnostics/client-log (uploaded by the private source tree) caps body size and event count, requires safe path segments, and re-runs the secret-redaction checks before a single event lands. A timeline you cannot trust to be free of leaked tokens is not a debugging aid; the companion post's point that "a log that leaks the private hand it was supposed to protect is the bug" applies just as hard when the log is being shipped off-device.

Where this stops, honestly

What exists today is a static artifact for one captured session: run the verb, read the swim lanes, find the hole. That is genuinely useful and it is shipped -- and so is the storage tier underneath it.

The uploaded logs land as newline-delimited JSON under .runtime/client-logs/<room_id>/<client_session_id>.jsonl -- append-friendly and portable, the right shape for small streaming uploads from many devices. When a session closes -- on game.session_summary at terminal state, or when its room is deleted -- the server compacts that now-immutable JSONL into a partitioned Parquet file under .runtime/client-log-archive/room_id=<room_id>/, and removes the source JSONL only after DuckDB reads the Parquet back and verifies the row count. A DuckDB view then UNIONs read_json_auto over the live JSONL with read_parquet over the archived Parquet and orders the union by seq, so a query never has to care which tier a row currently lives in. The split is deliberate: Parquet is the columnar analytical format but it is write-once in the way this project uses it, and appending every small upload straight to it would create a small-files problem -- so compaction waits for the session to be immutable. That two-tier store is shipped (the relevant server module). What is deliberately not built yet is the layer above it: richer browser or DuckDB dashboards and automated dropped-message summaries inside review artifacts. Those are future capabilities, not shipped ones, and this post is a draft describing a pipeline whose storage tiers have landed, not a finished observability stack.

What is load-bearing, and true today, is the design decision underneath all of it: the logs already carried the causal order, so the cross-device timeline never had to trust a clock. Sort by seq, join by causation_id, and the dropped frame on the TV three rooms away detects itself.

Or: subscribe to the newsletter for more posts on building this stuff.