Durable workflows and park-on-failure

Multi-stage agent work — decompose a feature, implement its wires, review, integrate, promote — takes hours to days and must survive process crashes, machine reboots, and broker outages without losing its place. Supernova puts that work on Temporal for durable execution, then inverts the default most workflow systems ship for the same moment: what happens when a step fails. It doesn’t retry. It parks.

Temporal, bought not built

The substrate ruling came early and flat. Matt (2026-06-04): “workflows are be temporal [sic]. straight up.” — and, in the same ruling, the philosophy the substrate serves: this project is “agents do what they want within very loosely structured workflows, with correctness enforced through disabling anything too sharp.” Workflows define step existence, ordering, and durability boundaries; correctness comes from the permission layer and gates, never from step-level validation of an agent’s choices.

The deployment posture [supernova design]: self-hosted Temporal on owned hardware with Postgres persistence — explicitly not Temporal Cloud — with workers and definitions written in Rust (the Temporal Rust authoring SDK, Public Preview as of May 2026). Scheduling is a verb under workflow, not a system: Temporal Schedules absorb all cron and interval work.

There is no native Kafka→Temporal trigger, so the bus reactor dispatch arm is the seam: reactors consume events and call signalWithStart with a deterministic workflow ID derived from the triggering event’s message_id — never its correlation id; the bus page owns that dedup rule (D21). All non-determinism lives in activities, every activity declares an explicit timeout (a missing timeout is a definition-validation error), and side-effecting activities are idempotent under deterministic tokens.

Park, don’t retry

The signature semantics, from a design note dated 2026-05-16: a failed workflow parks — alive, zero compute, awaiting a resume signal — while the system heals around it. An agent fixes the root cause, signals resume, and the run retries from the failed step, not from the beginning.

This forced a reckoning with the platform’s older “fail loud, no fallbacks” principle, resolved in the failure-classes doc (ratified 2026-06-10):

“Never proceed wrong, never die quiet, never kill the platform to make a point.”

Parking is failing loud, for contexts that have state worth keeping. Two axes decide everything. Context durability decides the loud-failure mode: durable Temporal steps park; ephemeral contexts — CLI invocations, handlers, reactors — crash with a nonzero exit. There is no third mode; “log a warning and continue” does not exist. Failure class decides the resume path:

Denials stay out of the error stream so the error stream stays meaningful. And blind retry is forbidden outright: Temporal’s max_attempts is conceptually always 1, and the retry-policy knobs are removed from config rather than defaulted — an operator setting max_attempts > 1 would silently re-enable the forbidden behavior. Retrying a deterministic bug with identical inputs can never succeed; it only burns compute and manufactures noise.

stateDiagram-v2
    [*] --> Running
    Running --> Completed: all steps green
    Running --> Parked: step fails, parked event emitted
    Parked --> Parked: system heals around it, zero compute
    Parked --> Running: resume signal, retry from failed step
    Completed --> [*]

Every park emits a lifecycle event carrying the workflow id, failed step, error, and failure class over the bus envelope, so parking is loud by construction. One refinement worth quoting — Matt, 2026-06-10: “I’m also not as militantly attached to ‘crash and fall over’ if the error visibility through the monitoring system is actually good.” So crashing is the bootstrap implementation of loudness, not the principle [supernova design]: a component class earns the right to stay standing only while synthetic canary errors demonstrably produce alerts within SLA; a failed canary reverts the class to crash-loud. The right to stay standing is continuously re-earned.

Composition: a shallow tree of durable runs

A feat entering ready starts exactly one named delivery workflow [supernova design] — the issue statuses and gates themselves belong to the delivery lifecycle. The tree is deliberately shallow: a delivery parent, one child wire workflow per wire, and a thin promotion sub-workflow that decides dev→main (“workflow DECIDES promote; vcs ACTUATES the ff-only merge”). Nesting caps at two levels. A unit gets its own child workflow only if it needs all three of: an independent park boundary, an addressable durable lifecycle, and independent failure isolation — one criterion means it’s an activity. Wanting a third durable level is the smell that a phase or an external-system call is being modeled as a workflow when it should be an activity or a parked-await. Review, E2E testing, and decomposition are accordingly not workflows: they are reactor-driven parked-awaits on other systems’ bus events.

The stage gates themselves collapse into one primitive: every gate is a single uniform parked-await over the evidence ledger — advance iff the issue type’s required evidence kinds are present, green, and fresh; otherwise park. A new evidence tier is a config row plus a producer, zero new workflow wiring.

The ephemeral executor seam

A wire workflow does not host its coder’s agent loop. It forks the coder from a warm repo base under a deterministic idempotency token (an activity retry never forks a duplicate), then observes rather than drives: the agent system’s turn-manager runs the turns, and the workflow watches the wire’s issue-state outcome. The ownership line, verbatim from the seam contract: “agent owns fork identity + teardown, workflow owns the binding + lifecycle triggers.” There is no crate dependency from the workflow system to the agent system; the trigger crosses the boundary as an in-process goal loop or a subprocess spawn.

The detail that makes park cheap: park reaps the coder. “Alive at zero compute” describes the durable Temporal run, not a held-open agent process. A parked wire workflow tears down its disposable fork; resume re-forks a fresh coder warm from the parent base with the failure context — never cold-boots, never resurrects the dead ephemeral. Durability lives in Temporal state, not in processes kept breathing.

Goal loops

The platform already is a control loop: requirements are the setpoint, tasks the actuator, evidence the sensor. A goal loop (sun agent goal pursue) scopes that loop to one agent and one target:

flowchart LR
    ACT["ACT: agent works the target"] --> EMIT["EMIT: real producers emit evidence"]
    EMIT --> CHECK{"obligation satisfied?"}
    CHECK -- yes --> DONE["goal closed"]
    CHECK -- no --> PARK["PARK on the bus signal that could change the answer"]
    PARK -- wake --> ACT

Two prohibitions carry the design. The agent does not self-attest its own success — evidence comes from the same independent producers as everywhere else, against an obligation declared up front by someone other than the doer. And the loop never busy-checks the ledger — react, not poll; it parks on the signal that would change the answer. Loops are bounded (iteration cap, wall-clock and token ceilings) and fail loud by parking-and-signaling, “the same primitive as success-parking, just with a different signal target.” A standing goal is a keep-satisfied loop, because closure is non-monotonic — done is rented, not owned.

Ratified 2026-06-24; the v1 loop runtime is implemented and unit-tested in the workspace [built in-repo] — closes-on-obligation, parks-not-polls, bounded-and-fail-loud are exercised in tests — while async parked-session wake and standing-goal hysteresis remain specced follow-ons [supernova design]. Deliberately, the top-down delivery stage gate and the agent-initiated goal loop are the same primitive at two altitudes.

Where this stands

The machinery that runs Matt’s operations today is the predecessor platform’s bespoke engine — reactors, a ship workflow, a batch orchestrator, APScheduler [live in automatt]. Supernova’s workflow system is the designed replacement. Its core is real Rust with real evidence: the parked-resume primitive is proven live against a self-hosted Temporal server on the dev box — a run queried while parked, sent a typed resume signal, completing — along with probe-scale schedule, cancel, and delivery-shaped runs with recorded run IDs [engineered, not deployed]. The canonical delivery workflow, live promotion gate, and critical-user-journey (CUJ) E2E on a real stack are explicitly held at draft [supernova design].

The project grades itself in public: D82 (2026-07-03) assessed the E2E workflow harness at roughly 12% real — stub probes returning fakes, an always-green CUJ workflow, a hardcoded promotion path. Making E2E real is now the prerequisite gate for closing anything more, and the first stub retirements landed the next day. That ledger of self-assessment lives in decisions as case-law.

What this costs

• Design far ahead of deployment. The most compelling machinery — the full delivery tree, the live promotion gate — is ratified prose and draft requirements, not a production system. Live evidence is probe-scale, and the E2E grade above is the project’s own number.
• Park needs a healthy healer. Deterministic parks resume only via explicit signal after a fix. If the bug-filing → fix → signal pipeline stalls, parked runs accumulate silently at zero compute; the model leans on monitoring and the canary discipline, which are among the less-proven parts.
• Closure rests on scope wording. Some requirements are marked done on carefully scoped slices while D82 simultaneously lists stubs in adjacent code paths; a skeptical reader should notice how much weight the scoping language carries.
• Temporal Rust SDK risk. The authoring SDK is Public Preview; the sanctioned fallback is a Python/TS workflow service behind a Kafka/HTTP boundary — a real architectural wart if it’s ever needed.
• Superseded along the way: per-project Temporal namespaces (rejected for one shared namespace with a project-id partition key), a dedicated rollback topic, the “ship workflow” naming, and correlation-id-as-dedup — each abandoned with the reasoning on record.