Detect, repair, learn

Supernova’s operational loop is where automation becomes self-management. It connects telemetry to the exact change and requirement involved, preserves enough context to repair the fault, and measures whether the repair actually improved the system.

Observe the whole causal chain

OpenTelemetry feeds Prometheus and Grafana while structured operational events enter the Kafka trail. Correlation identifiers connect an alert to the workflow, agent, tool call, approval, and change that preceded it.

The monitor system owns the current evidence ledger and the read models used to answer questions such as:

• Which requirements are no longer supported by fresh proof?
• Which project or workflow is blocked on a failing dependency?
• Did a recovery action improve the requested metric?
• Which reviewers or gates repeatedly miss defects?
• Is the fleet idle because there is no valuable work, or because generation failed?

The system audits itself

Monitor separates self-audit into two passes so intelligence is spent only where it helps.

1. Mechanical checks run cheaply and often. They produce structured artifacts for platform invariants, requirement-DAG coverage, bus wiring, provenance, feature adoption, external/CUJ delivery, store retention, staleness, recurring failures, cost, and audit health.
2. Agent-led audit reads those artifacts and applies judgment. It does not pay several agents to rediscover the same facts. The consumer decides whether a gap should become a specification, implementation wire, process change, or dismissal.

The audit catalog includes a meta-audit: did scheduled audits run, were their findings useful, did follow-up work close, how many false positives occurred, and are scoring or routing metrics being gamed? Repeated incidents can propose a new permanent mechanical check so the next occurrence costs zero model turns to detect.

Turn receipts, tool use, cache metrics, review round trips, escaped defects, and token/spend trends also feed the process audit. Sustained evidence can produce a focused, versioned proposal to improve a role’s prompt, maintained instructions, supplied context, checklist, or tools. When the right fix is an agent-definition change, sun agent def edit validates and atomically coordinates the role, model/provider, capabilities, isolation, and system prompt, then hot-reloads it. Monitor cannot apply that change itself: the proposal follows the ordinary review and approval path and keeps a rollback condition.

Facts first, judgment second

Mechanical checks may raise alerts, findings, and evidence changes. They do not automatically invent requirements or mutate issues. An authorized monitor agent investigates the context, applies existing precedent where appropriate, and decides whether to dismiss, escalate, amend the specification, or file bounded remediation work.

This prevents a noisy metric from turning directly into an unlimited autonomous backlog.

The repair loop

flowchart TD
    signal["Alert, failed proof, or audit finding"] --> context["Correlate trail + requirements"]
    context --> judge["Capability-scoped judgment"]
    judge --> work["File or amend bounded work"]
    work --> deliver["Agent + workflow delivery"]
    deliver --> verify["Before/after evidence"]
    verify -->|improved| close["Record outcome"]
    verify -->|not improved| context

A recovery action is not marked successful merely because a command ran. The proof contract requires matching before-and-after metrics and a broker-delivered receipt for the intended target and action.

Independent failure paths

Canaries exercise the alert-to-agent-to-workflow path. A separate watchdog checks the monitoring stack itself and can use an out-of-band provider when the normal bus or dashboard is unavailable. Long-lived field evidence and retention are treated as production obligations, not inferred from a short local test.

Learn from the process

The system records predicted value and later outcomes, review round trips, post-landing defects, reopened work, overrides, token use, and recurring failure classes. Audits can propose changes to work scoring, review criteria, role instructions, or system contracts.

Those proposals still pass through the same provenance, review, and approval mechanisms. The manager is improvable, but not exempt from management.

Build the successor, protect the current version

Supernova’s self-hosting boundary is intentionally conservative:

1. The stable running version treats its successor as an ordinary isolated project.
2. The candidate is built and tested without mutating the running floor.
3. State migration is rehearsed and checked under the current policy.
4. Cutover is bounded, observed, and approval-gated.
5. A tested rollback path remains available.

The candidate cannot approve itself or change the admission rules used to judge it.

What is working now

Alert lifecycle, evidence reconciliation, remediation issue creation, soak evaluation, canary and watchdog exercises, owned telemetry-stack tests, and selected recovery paths exist in the repository. Production long-window evidence, broader provider delivery, and concrete recovery adapters remain active work. See project state.