Architecture for contributors¶
How Opsmith is built, for people changing it. If you want to understand what it does from the outside, read Concepts first; this page assumes it. The reasons behind each design choice are in ARCHITECTURE_DECISIONS.md, and the rules that must never be broken are in AGENTS.md.
- Processes and storage
- The job pipeline
- Jobs, schedules and failure handling
- The model layer
- The execution path
- Where the code lives
- Extending Opsmith
Processes and storage¶
flowchart TB
USERS["Reviewers and admins<br/>browser or API"] --> API
PUSH["Collectors<br/>push API"] --> API
subgraph OPS["Opsmith processes"]
API["api<br/>REST API + review UI"]
SCHED["scheduler<br/>enqueues due jobs"]
WORKER["worker (1..n)<br/>runs pipeline jobs"]
end
DB[("PostgreSQL<br/>entities, job queue, audit log")]
OBJ[("Data directory<br/>raw signals")]
API <--> DB
SCHED --> DB
WORKER <--> DB
WORKER --> OBJ
WORKER --> SRC["Loki · Prometheus · VictoriaLogs · files"]
WORKER --> LLM["LLM gateway<br/>offline rules, Claude, or any<br/>OpenAI-compatible endpoint"]
WORKER --> SANDBOX["Sandbox containers<br/>no network"]
WORKER --> GH["GitHub<br/>branch push and PR only"]
Python 3.12, FastAPI, SQLAlchemy 2, Pydantic 2, uv. PostgreSQL in production; SQLite for development and tests. Every row carries a tenant id, and every query filters on it.
Model access is behind one gateway and a small provider interface. Three providers ship
(heuristic, anthropic, openai), and others can be added as plugins; see
Models.
platform.Platform is the composition root: API, worker, scheduler, CLI and tests all build
one, and it wires settings, storage, the gateway, SCM providers and every stage together.
The job pipeline¶
Stages never call each other. Each one writes its results and enqueues the next job in the same transaction, so a crash between two stages loses nothing.
flowchart LR
S["scheduler tick"] --> ING["ingest<br/>per connector"]
PUSHAPI["push API"] --> BB
ING --> BB["build_bundles<br/>per project"]
S --> INF["inference<br/>per service"]
BB -. "bundles wait for<br/>the next run" .-> INF
S --> AUD["audit<br/>per repository"]
S --> CHK["check<br/>per check and repository"]
INF --> AGG["aggregate<br/>per project"]
AUD --> AGG
CHK --> AGG
AGG --> N1["notify"]
REV["review (API/UI)"] --> EXE["execute<br/>per task"]
EXE --> N2["notify"]
S --> TRK["track_prs<br/>per project"]
TRK --> VER["verify<br/>per merged task"]
| Job kind | Handler | Enqueued by |
|---|---|---|
ingest |
etl.pipeline.IngestionService.run_connector |
scheduler, manual trigger |
build_bundles |
etl.pipeline.BundleBuilder.build_for_project |
ingest, push |
inference |
inference.orchestrator.InferenceOrchestrator.run |
scheduler, manual trigger |
audit |
audits.runner.AuditRunner.run |
scheduler, manual trigger |
check |
audits.checks.CheckRunner.run |
scheduler, manual trigger |
aggregate |
aggregation.aggregator.Aggregator.run |
inference, audit, check, manual trigger |
execute |
executor.service.ExecutionService.execute |
an approving review |
track_prs |
tracking.PRTracker.run |
scheduler, manual trigger |
verify |
tracking.Verifier.run |
PR tracking, after a merge |
notify |
notifications.handle_notify_job |
anything that calls notifications.notify |
The mapping lives in Platform.handlers().
Jobs, schedules and failure handling¶
Every stage is a job in a database-backed queue.
- Idempotent enqueue. Each job has a key derived from the work it stands for, so enqueueing twice yields one job. Follow-up work is enqueued in the same transaction as the state it depends on.
- Retries. A failing job is retried with exponential backoff. When its attempts are used up
it becomes
dead: audited, notified, visible in the UI, and retryable by an admin. - Leases. A worker holds a job under a lease that it renews while it is alive. A slow job is never handed to a second worker; a dead worker's job is picked up after the lease expires.
- Waiting is not failing. A handler can ask to be run later (for example while another task touches the same files) without using up an attempt.
- Schedules live in the database: an interval or a five-field cron expression per connector, service, repository, check and project. The slot is part of the job key, so several schedulers or repeated ticks never double-enqueue, and missed slots collapse into one run. A schedule that cannot be computed is disabled and audited rather than stopping the others.
Nothing is dropped silently. Every failure ends in one of four visible places: a retry, a dead job with a notification, a task back in review with a report, or a finding marked suppressed with a reason.
Implementation notes:
queue.enqueueis idempotent onidempotency_key(a unique column). Keys must be deterministic for the logical unit of work, for exampleexecute:<task>:<review>.- A handler raises
queue.RetryLaterto wait without using an attempt, andqueue.PermanentFailurewhen retrying cannot help. Worker._executeruns the handler with a heartbeat thread that renews the lease. Long work (clones, agents, checks) runs outside database transactions; seeExecutionService.- Work that must not overlap is serialized where the shared thing lives. One aggregation per
project and one analysis per service take a row lock (
db.SERIALIZE:FOR NO KEY UPDATEon PostgreSQL, which excludes another holder but not the inserts that reference the row). Audits and checks of one repository share one advisory file lock next to the repository's checkout (audits.runner.checkout_lock), because the shared thing there is a directory; a second run is deferred. - The API commits a request's transaction when its handler returns, before the response is
sent (
deps.SessionDepis a function-scoped dependency), so a failed commit is never answered with a 2xx. - State changes go through
audit_log.transition_task/transition_lead, which enforce the state machines and write the audit trail. A flush guard makes finishedInferenceRunandAuditRunrows immutable.
The model layer¶
Nothing in the pipeline talks to a model vendor directly.
flowchart LR
ST["a stage<br/>(inference, triage, check, task)"] --> GW["LLMGateway.structured"]
GW --> A1["allowlist<br/>process ∩ project"]
GW --> A2["redact variables"]
GW --> A3["render versioned prompt<br/>(untrusted data framed)"]
GW --> A4["budget check"]
GW --> P["provider.complete(request)"]
P --> H["heuristic"]
P --> AN["anthropic"]
P --> OA["openai-compatible"]
P --> V["validate against schema<br/>(one corrective retry)"]
V --> U["record usage and cost"]
- A provider implements
llm.base.LLMProvider: anameandcomplete(ProviderRequest) -> ProviderResponse. It receives the rendered prompt and the schema, and returns data plus token usage. A call that was billed but unusable (refused, truncated) raises anLLMErrorcarrying that usage, so the gateway can still record it. HeuristicProviderimplements every stage with rules and reads the structured variables instead of the prompt text. It is why the whole system runs offline.- Prompts live in
llm/prompts.pywith an id and a version that every run records. Bump the version when the text changes. - The coding agents (
executor/agents.py) do not go through the gateway, because they hold a conversation with tools rather than one structured call. They share one tool dispatcher (dispatch_tool) and one task brief (task_brief), and the execution service writes their usage to the same ledger.
tests/test_provider_parity.py runs the acceptance scenario through the OpenAI-compatible HTTP
path and requires the same outcome as the offline run. A change that breaks one provider's path
shows up there.
The execution path¶
executor.service.ExecutionService is the most security-sensitive code. Read
Concepts: From approval to pull request
for the sequence, then note how it is implemented:
assert_approvedis the single gate. It comparesreview.approval_scope_digestwith the digest stored on the approving review.sandbox_policy_errorrefuses thelocalbackend for non-local repositories.- After the checkout,
.gitis copied to a directory outside the workspace. Every latergitutil.run_git(..., git_dir=...)call uses that copy, with hooks and fsmonitor disabled. _diff_gatesruns after the platform's own check run.gitutil.worktree_changesandstaged_added_linesparse git's NUL-delimited output.scm.base.check_pushablerestricts pushes toopsmith/...branches.
Each of these has a test that tries to defeat it; Security lists them.
Where the code lives¶
Path under src/opsmith/ |
Responsibility |
|---|---|
platform.py |
Wires everything together and maps job kinds to handlers. Start reading here |
models.py, schemas.py |
Database entities; settings and model-output schemas |
migrations/ |
Schema migrations (Alembic), applied when a process starts |
connectors/, etl/ |
Signal sources (Loki, Prometheus, VictoriaLogs, files, push); normalize, redact, store, bundle |
redaction.py |
Secret and PII detectors, shared by ingestion, the gateway and the diff gate |
inference/orchestrator.py |
Chained analysis runs and the lead lifecycle |
audits/ |
Analyzers, external tool wrappers, the audit runner, and checks (checks.py, ADR-019) |
aggregation/ |
Correlation, dedup, suppression, scoring, task drafting |
review.py |
The human review gate and the approval scope digest |
executor/ |
Sandboxes, the confined workspace, coding agents, the execution service, PR text |
scm/ |
GitHub and local git providers. No merge operation, by design |
tracking.py |
PR and CI tracking; post-merge verification |
llm/ |
The gateway every model call goes through, prompts, pricing, and the providers (heuristic, anthropic, openai) |
queue.py, scheduler.py |
The job queue and schedules |
audit_log.py |
The audit trail, the task and lead state machines, run immutability |
localfs.py, config.py, protection.py |
Host-file confinement; process settings and credential references; protected paths |
api/, ui/ |
REST API (/api/v1) and the server-rendered UI (/ui) |
cli.py, demo.py |
The opsmith command; the offline end-to-end scenario |
gitutil.py |
The only place git is invoked: safe config, the trusted git dir, NUL-delimited parsing |
db.py |
Engine and sessions, utcnow, new_id, the SERIALIZE row lock |
security.py |
Tenants, users, roles and API keys (stored as SHA-256) |
notifications.py |
The notification outbox; delivery is a notify job |
provenance.py, dashboard.py |
A task's provenance chain; dashboard figures and the task funnel |
storage.py, observability.py, configfile.py |
The object store; JSON logs and /metrics; opsmith apply |
Extending Opsmith¶
Three kinds of plugin can be added without changing Opsmith, as a Python package that declares
an entry point. Built-in ones use the same mechanism through a @register decorator.
| What | Entry-point group | Implement |
|---|---|---|
| A signal source | opsmith.connectors |
A subclass of connectors.base.LogConnector or MetricConnector with fetch(start, end, cursor) -> FetchResult |
| An audit analyzer | opsmith.analyzers |
A subclass of audits.base.Analyzer (or audits.external.ExternalTool with a pure parse()), declaring name and audit_types |
| A model provider | opsmith.llm_providers |
A callable (settings) -> provider, where the provider has name and complete(request) |
# pyproject.toml of your plugin package
[project.entry-points."opsmith.connectors"]
datadog_logs = "my_package.datadog:DatadogLogConnector"
[project.entry-points."opsmith.llm_providers"]
my_gateway = "my_package.llm:make_provider"
Rules for each:
- Connector. Read-only. Return records in
[start, end); skip and count records you cannot parse (FetchResult.skipped) instead of raising. If you could not read the whole window, returncursor["last_end"]set to where you stopped. Validate URLs withconnectors.base.validate_base_url. Test withhttpx.MockTransport. - Analyzer. Return
RawFindings with a stablerule_id, a path and a line. Never execute repository code. Isolate failures per file. Mask secrets in snippets. - Provider. Send only what is in the request; it is already redacted. Report token usage.
Raise
LLMRefusal/LLMOutputErrorwith.usageset for billed-but-unusable responses.
Changes inside Opsmith:
- A new model stage: add an output schema to
schemas.py, a prompt tollm/prompts.py, and a matching branch inHeuristicProviderso the stage works offline. Call it throughLLMGateway.structured. - A new task state: extend
TASK_TRANSITIONS, the status chips inui/templates/macros.html,dashboard.FUNNEL, and add an ADR entry. - A new setting: add it to
schemas.ProjectSettingswith a bound, document it in Configuration, and add it toopsmith.example.toml. - Anything users can see:
make docsfails until it is documented. Environment variables go in Deployment and.env.example; API endpoints in API; notification events, connector types and audit types in Configuration; analyzers and task or lead states in Concepts; model providers and coding agents in Models; CLI commands in the README; job kinds in the table on this page. - A schema change: change
models.py, then write a revision against a scratch database that is at the newest revision:
export OPSMITH_DATABASE_URL=sqlite:////tmp/opsmith-migrations.db
uv run alembic upgrade head
uv run alembic revision --autogenerate --rev-id 0002 -m "add tasks.owner"
uv run alembic upgrade head && uv run alembic check # "No new upgrade operations detected."
Number revisions in sequence and read what was generated: autogenerate misses renames
(it drops and adds) and server defaults, and a new non-null column on an existing table
needs a default or a data step. Never edit a released revision. Run
tests/test_migrations.py on PostgreSQL as well (see Testing), and add an
Upgrade notes entry to the changelog. Details are in ADR-002.