Concepts¶
How Opsmith thinks: the seven terms it uses, what each stage of the pipeline does, and how leads and tasks move through their lifecycles. Read this to understand what you see in the UI. If you want to change Opsmith itself, the internals are in development/architecture.md.
- Key concepts
- The pipeline, stage by stage
- Lifecycles
- From approval to pull request
- What never gets lost
Key concepts¶
Seven terms carry the whole system.
| Concept | What it is | Example |
|---|---|---|
| Signal | A normalized, redacted log event or metric point | ERROR upstream timeout after 30000ms |
| Bundle | A compact summary of one service over one time window: log templates and metric anomalies, with references to the raw data. Sized to fit a model's context | 80 timeout errors, p95 latency up 590% |
| Lead | An open question carried from one analysis run to the next until it is confirmed, refuted or expired | "Timeouts are recurring. Confirm next window." |
| Finding | A problem with evidence, from analysis (runtime) or from an audit or a check (static) | httpx.post without timeout at app/payments.py:10 |
| Task | One or more correlated findings, ranked P0 to P3, with an approach and acceptance criteria | "Add a timeout to the payment gateway call" |
| Review | A human decision on a task, recorded with identity and time | approved by Riley Reviewer |
| Execution run | One sandboxed attempt to implement an approved task and open a PR | branch opsmith/performance/…, PR #1 |
Everything is traceable in one direction:
PR → execution run → review → task → findings → evidence (bundles, raw events, audit results) → analysis runs → leads.
The task page in the UI and GET /api/v1/tasks/{id}/provenance show that chain.
The pipeline, stage by stage¶
1. Ingest¶
A connector fetches signals for a time window (Loki, VictoriaLogs, Prometheus, files) or receives them through the push API. Each record is normalized to a common schema, mapped to a service, and redacted before it is stored. Records that cannot be parsed are skipped and counted, not allowed to block the rest.
Bundles are then built per service:
- Log lines are grouped into templates (variable parts masked, similar lines clustered), each with counts, severities, sample messages, stack-trace locations and references to the raw events.
- Metrics get features: mean and percentiles, a baseline from a look-back window, the change against it, and a direction-aware anomaly flag (a latency rise and a success-rate drop are both bad).
Raw data stays in the object store. Bundles refer to it by key and line.
2. Analyze (chained runs)¶
On an interval per service (six hours by default), the configured model receives the new bundles and a summary of the open leads. It answers with:
- findings for problems the evidence supports,
- one update per lead: confirm, refute or carry,
- new leads for patterns that need another window to decide.
This is what gives the analysis memory without unbounded context: each run sees a bounded summary of open leads, never the history of earlier runs.
The model's output is untrusted. It must match a schema; updates for unknown leads are ignored; evidence references that were not in the input are dropped and the finding's confidence is reduced. Each run is stored as an immutable record with its inputs, prompt version, model, output and cost. A run that fails still records what it spent.
3. Audit¶
On a schedule per repository (nightly by default; a full audit at least weekly, changed files only in between), analyzers examine a checkout:
| Analyzer | Finds |
|---|---|
secrets |
Credentials and private keys in files (values are masked in the finding). A value that only refers to a secret is not one: ${DB_PASSWORD}, $PASS, $(cat ...), a {{ template }}, or a variable or call in program code (password = form.password). A literal still is, including the default in ${X:-...} and a value inside a string or a comment |
python-ast |
Bare except and swallowed exceptions, mutable default arguments, eval, yaml.load, subprocess with a shell, SQL built from strings, HTTP calls without a timeout, queries inside loops |
dependencies |
Unpinned dependencies; known vulnerabilities through OSV when enabled |
compliance |
A missing license or one outside your allowlist; your own policy rules |
coverage-gaps |
Modules without tests. Off by default: add coverage to the repository's audit_types |
ruff, semgrep, gitleaks, trivy, osv-scanner |
Used automatically when the tool is installed on the worker; recorded as unavailable otherwise |
A triage pass then filters likely false positives and calibrates confidence. A finding that triage drops is kept with the reason, never deleted.
Checks add reviews of your own. A check is an instruction in plain words ("review this code for missing authorization checks"), with the files it covers and a cron. A model carries it out, and every problem it reports must quote the file it names, or it is discarded. The problems that are kept become findings like the analyzers' and go on to aggregation. See Checks.
4. Aggregate¶
New findings are grouped, compared with existing tasks, and scored.
flowchart LR
LOG["Runtime finding<br/>80 timeout errors<br/>stack trace: app/payments.py:10"] --> MATCH{"Same repository and file, and<br/>same category nearby<br/>or the same line?"}
CODE["Static finding<br/>HTTP call without timeout<br/>app/payments.py:10"] --> MATCH
MATCH -- "yes" --> TASK["One task<br/>confidence 1 − (1−a)(1−b)<br/>ranked higher"]
MATCH -- "no" --> SEP["Separate tasks"]
| A new finding matches… | Result |
|---|---|
| an open task | Its evidence is added to that task; no new task |
| a task that is approved or executing | It waits. A task a human has approved is never changed underneath them; the evidence is attached once the execution is over |
| a task closed or merged recently (14 days by default) | Suppressed, with the reason recorded |
| a rejected task | Suppressed, unless the evidence materially changed: higher severity, a new kind of evidence, or two to three times stronger evidence, depending on why it was rejected |
| nothing | A new task, sent for review |
An audit re-reporting the same defect is the same evidence again, not more evidence. Every runtime finding is a new occurrence.
The score combines severity, confidence, evidence strength, how often and how recently it was seen, how many services are affected, whether runtime and static evidence agree, the category and the estimated effort. It maps to a priority from P0 to P3.
5. Review¶
Tasks wait in the review queue. A reviewer can:
| Decision | Effect |
|---|---|
| Approve | Queues the execution |
| Approve with edits | Changes title, approach, priority, paths, branch name or similar first, then approves |
| Reject (with a reason code) | Closes it and suppresses the same issue in future |
| Defer | Parks it; it can be resubmitted |
| Request investigation | Opens a lead with the reviewer's question for the next analysis run; the task returns to review with the answer |
A task that touches protected paths needs more: elevated paths need an admin who
explicitly acknowledges it; excluded paths cannot be approved at all.
6 to 8. Execute, open a pull request, verify¶
Covered in From approval to pull request. After a human
merges the PR, a verification job watches the task's original signals. If the log template or
metric anomaly does not come back, and a later audit no longer reports the static finding (for
a finding from a check, a later run of that check), the task becomes verified;
otherwise regressed, which returns it to review. A task merged again
after a regression is verified again. If no post-merge signals or audit arrive within 48 hours
of the delay, the verification job fails (a dead job, with a notification); retry it from the
Jobs page once data has arrived.
Lifecycles¶
Lead¶
stateDiagram-v2
[*] --> open: the model spots a pattern, or a reviewer asks
open --> carried: still uncertain
carried --> carried: still uncertain, age + 1
open --> confirmed: evidence establishes it
carried --> confirmed: evidence establishes it
open --> refuted: evidence contradicts it
carried --> refuted: evidence contradicts it
carried --> expired: carried too many times
confirmed --> [*]: becomes a finding
refuted --> [*]
expired --> [*]: escalated to a human
A lead the model does not mention is carried, not dropped. One carried more than
lead_max_carry times expires and is escalated with a notification.
Task¶
stateDiagram-v2
[*] --> proposed
proposed --> in_review: submitted to the queue
in_review --> approved: approve or approve with edits
in_review --> rejected: reject with a reason
in_review --> deferred: defer
in_review --> investigating: request investigation
deferred --> in_review: resubmitted
investigating --> in_review: investigation lead resolved
rejected --> in_review: evidence materially changed
approved --> executing
approved --> in_review: approval became void
executing --> pr_open: checks pass, PR opened
executing --> failed: agent, checks or a gate failed
failed --> in_review: back with an execution report
pr_open --> merged: a human merges
pr_open --> closed: PR closed
merged --> verified: signal stays gone
merged --> regressed: signal returns
regressed --> in_review
verified --> [*]
closed --> [*]
Every transition goes through one function and is written to the audit log with who did it.
From approval to pull request¶
sequenceDiagram
actor Reviewer
participant UI as Opsmith API / UI
participant Queue as Job queue
participant Exec as Executor
participant Box as Sandbox
participant Git as GitHub
Reviewer->>UI: Approve task
UI->>Queue: Record review with a digest of the task's scope, enqueue "execute"
Queue->>Exec: Run job
Exec->>Exec: Gate 1: human approval, scope unchanged, protected paths, sandbox policy
Exec->>Exec: Fresh checkout, no remote, no credentials. Snapshot git metadata outside it
Exec->>Box: Agent edits files and runs the configured checks
Exec->>Box: Opsmith re-runs build, lint and test itself
Exec->>Exec: Gate 2: changed files vs protected paths, added lines vs credential patterns
Exec->>Exec: Commit from the trusted snapshot (hooks off)
Exec->>Exec: Gate 3: approval re-checked, scope still unchanged
Exec->>Git: Push branch opsmith/…, open PR with evidence and provenance
Git-->>Exec: CI status (polled)
Reviewer->>Git: Review and merge (humans only)
Exec->>Exec: Post-merge verification: verified or regressed
Points worth knowing:
- The agent has six tools and nothing else: list files, read a file, write a file, replace
text in a file, run one of the configured checks by name, and finish. No shell, no network,
no credentials. Three agents ship, all behind that same tool set:
heuristic(deterministic fixes for a handful of rules, so the loop works offline),claude, andopenai(any model behind an OpenAI-compatible endpoint that supports tool calling). - The checks are the repository's own commands (
build_command,lint_command,test_command), set by an admin. They run only in the sandbox. - The gates run after the last sandboxed command, so what is checked is exactly what is committed.
- A failure of any kind (agent gave up, checks failed, a gate blocked, the sandbox is unavailable, the approval is void) returns the task to the review queue with a report and a notification. Up to two attempts are made when a failure looks transient.
- Concurrency is bounded per project, and tasks that touch the same files wait for each other.
What never gets lost¶
Every stage is a queued job that is retried when it fails. A failure always ends somewhere you can see it:
- a retry, with backoff;
- a dead job with a notification, once its retries are used up (Jobs in the UI; an admin can retry it);
- a task back in the review queue with a report, when an execution fails or is refused;
- a finding marked suppressed with the reason, when it is filtered out or is a duplicate.
Operations covers what to watch and what to do about each.