The bus: nervous system and audit log

Supernova has one rule about communication: everything is an event on the bus. Kafka is not a source that feeds other systems — it is the substrate every system communicates through. Monitoring, workflows, agents, chat: all of it rides the same broker, and the durable record it leaves behind doubles as the platform’s audit log.

Why buy Kafka: the 479 MB bus.db

The predecessor platform, automatt, ran a bespoke Python event bus [live in automatt] — and it was the single most reliable source of unreliability. Documented failure modes: the Unix socket vanishing under a live service, the orchestrator crash-looping on the missing socket, oversized frames tripping LimitOverrunError, and a bus.db that bloated to 479 MB, with 1.6 GB memory peaks, forcing restarts every couple of hours. Matt’s ruling (2026-06-04):

“the EVENT BUS has consistently been pain and python will always be slow and kafka is like industry standard that does everything we could ever want and its performant and its right there.”

Moving to Kafka isn’t “off-the-shelf is easier”; it’s “the bespoke one keeps breaking and the proven one won’t.” This is the build-or-buy pattern at its cleanest: buy the broker, build only the conventions on top — plus the one genuinely bespoke layer.

Two layers, clearly delimited

1. The substrate (bought). Kafka provides durable pub/sub, topics, partitions, consumer groups, retention. The integration is deliberately thin: the bus crate must not reimplement storage, partitioning, or replication, and invariants ban domain logic in this layer.
2. The dispatch arm (built). The reactor layer — “do X when Y condition is met” — was once a standalone system and was folded into the bus (2026-06-20), because the dispatch arm is the consumer layer for the substrate. It exists because there is no native Kafka→Temporal trigger; someone has to own that seam.

The envelope

The bus owns the platform envelope: the mandatory field set every published event carries, enforced at publish time — a malformed envelope is rejected, never silently transported.

Delivery is at-least-once, never exactly-once; duplicates are expected and correct, and dedup rides message_id alone. The two IDs stay separate because merging them fails both ways: distinct actions in one chain would collide, and a retried event with a new correlation ID would double-fire (D21, 2026-06-18 — a round-two review caught the workflow system deriving IDs from correlation_id and reconciled it). The no-double-email guarantee rests on this rule. Baking correlation and initiator into the envelope from day one makes observability structural, not retroactive — automatt had the same fields, inconsistently populated, which is how you get recurring validation errors instead of traces.

Topic grammar and the registry

Every topic is event.<system>.<entity>.<verb> — first segment the owning system, verb in the past tense (“what happened”). No dynamic topic names, no wildcards; Matt (2026-06-04): “we don’t do dynamic topic names… I’d rather filter on params.” Variable data goes in the payload. The taxonomy stays fixed and enumerable, so it can live in a single canonical registry — roughly 90 topics across eighteen of the nineteen systems (core, the framework substrate, publishes none), each tagged owned or ownerless, mechanically diffed against every system’s declared topics by the wiki validator [engineered, not deployed]. A subscribed topic must have a registered producer; subscribers themselves are optional (Matt, 2026-06-23) — advisory wiring documentation, not a completeness gate. Naming disputes land in a reconciliation ledger with states registered | ruled | proposed | superseded that system docs must not silently re-litigate.

The two-write outbox

The bus-topics doc, rule 6 (ratified 2026-06-10): a state mutation and its event emission are two writes that “MUST NOT be able to diverge silently.” The outbox pattern is how: the event is recorded transactionally with the mutation — same store, same transaction — then relayed to the broker. If the broker is down, events queue locally and replay on reconnect in per-key order; a mutation whose event can never be recorded fails closed. publish() returns the generated message_id precisely so the producer can record it before relay. This is the mechanism that makes “every behavior traceable to root cause” hold exactly when things fail.

D81 (2026-07-03) narrowed the scope: the outbox is required only for load-bearing state-transition events from the five or six DB-backed systems. The discriminating test — “if this event is silently lost, does the system end up permanently wrong in a way that won’t self-correct?” Yes: outbox. No — config updates, wiki edits, log lines: publish directly, because losing one is a missed notification, not corruption.

flowchart LR
  P["producing system"] -->|"mutation + event, one txn"| O["outbox"]
  O -->|"relay, replay on reconnect"| K["Kafka topics"]
  K -->|"consume"| R["reactor dispatch arm"]
  R -->|"typed actions"| C["workflows, agents, chat, link"]
  C -->|"actions emit events"| K
  K -.->|"retention = durable history"| T["the trail"]
  M["monitor read-lens"] -.-> T

Note the cycle: subscribers’ actions are themselves events, emitted back onto the bus for others to react to — actions feed back into the trail rather than vanishing into logs.

The trail

The bus owns the trail: the durable event record plus its correlation_id causal chains. There is no separate audit store — “history of X” is a per-topic retention setting, not a new system. Matt (D46): “we kinda treat bus as audit and anything that cares should subscribe via metrics or reactor.” This one ruling dissolved a planned ops-audit store, the issue system’s audit list, project audit, and the monitor findings store into retention config. The monitor is a read-lens over the trail, not its owner; the trail also transports the attestation events that requirements evidence indexes. Even structured logging is just bus events on a registered log topic — instead of opaque text logs nobody can subscribe to.

The reactor dispatch arm

A reactor binding is a name, trigger topics, a declarative side-effect-free when predicate, and a typed action from a closed catalog — start_workflow, signal_workflow, chat_action, link_action, field_update, lib_call. No arbitrary callables; schema-invalid bindings are rejected at registration. Bindings are runtime registry state, loaded and unloaded dynamically, not static config.

The load-bearing case is the Kafka→Temporal bridge: the default primitive is signalWithStart, with the workflow ID derived from the trigger’s message_id — so a redelivered event converges on the one workflow instead of racing a double-start. Each binding keys idempotently on the message and binding name and fires once; bindings never suppress each other. Failure handling follows the platform failure classes: transient failures fail closed and let the uncommitted offset redeliver (the config schema deliberately has no retry knobs — blind retry is banned), deterministic failures auto-file a bug with an event-context snapshot, and denials are a designed branch kept out of the error stream.

Two details worth naming. First, the dispatch arm is never a side door to the outside world: there is exactly one external-action path, and it is the link system’s permission-and-approval-gated path. Second, the same find→attach→listen wakeup mechanism serves two consumers: it wakes sleeping agents and un-parks Temporal stage gates — one mechanism, two consumers.

The bus is also its own hardest case: broker-down means the loudness channel is down. The design enumerates its own exception — a bus-independent probe in ops (the one sanctioned violation of “everything rides the bus”) watches the broker and, on recovery, emits event.bus.broker.connected onto the freshly recovered bus, where a default reactor resumes parked workflows. The one recovery signal that cannot ride the bus to begin with is delivered by the only component allowed not to.

Where this stands

Build state, since “event bus” invites overclaiming:

• The Rust bus crate is real [built in-repo]: thirteen modules (~2,900 lines) with one-to-one test files covering the envelope, topic parser, outbox, reactor engine, and registry validation. All bus requirements are marked done on unit and integration evidence.
• The reactor pump demonstrably fires against a live project event spool — hundreds of observed dispatches driving pod spawns and ingest gates on the dev box.
• But the day-to-day transport today is a JSONL file spool, not Kafka. A compose Kafka broker is provisioned and answers, and a conditional live test exercises outbox queue-and-replay through it, but dual-write verification was still planned as of 2026-07-02. D80 (2026-07-03) is blunt about the interim file layer: the current file-outboxes are “the live two-write DIVERGENCE” — the very failure mode the pattern exists to prevent — and are scheduled for deletion as authoritative state moves to Postgres, Kafka, and Temporal.
• Retention tiers as operated config, the monitor’s OTel lens, and topic-browser surfaces are [supernova design] only. CloudEvents and W3C Trace Context were evaluated and deliberately deferred (2026-06-19) with a written non-breaking upgrade path — restraint over resume value.

What this costs

Kafka is heavy: a JVM broker under KRaft for a single-tenant platform on one box, serving ~90 topics — the wiki itself names NATS and Redpanda as lighter alternatives; Matt’s call was Kafka. At-least-once delivery pushes idempotency onto every consumer: each one must derive its idempotency key deterministically from message_id or double-fire, and ordering is per-partition-key only, which is easy to forget. The registry is mechanical to validate but manual to maintain, and it has lagged reality — a 2026-06-22 audit found an entire system section and four already-subscribed topics missing. Bus-as-audit couples audit depth to retention config: with a seven-day default, durable history is a knob, not an archival guarantee. D81’s outbox narrowing was an honest scope walk-back, but it means “cannot diverge silently” now protects only the DB-backed core, not everything. And per-requirement “done” here means design-plus-unit-proven, not production-soaked: a platform-wide assessment (D82, 2026-07-03) rated the end-to-end harness at roughly 12% real, and a sampled live envelope carried timestamp: 0 — the contract field existed; the honesty of its population did not. The design’s own case-law is the first to say so, which is rather the point.