AGENTS.md¶
Instructions for AI coding agents and for people working on this repository. This file is
canonical. CLAUDE.md imports it and adds only what is specific to Claude Code.
What this is¶
Opsmith is an engineering-ops platform. It ingests logs, metrics and code, runs chained LLM analysis with carried-over leads plus scheduled repo audits and checks, turns the results into ranked tasks, and, only after a recorded human approval, has a sandboxed subagent open a PR.
Where to read first:
| Need | Read |
|---|---|
| What the system does, from the outside | docs/guide/concepts.md |
| How it is built: job pipeline, model layer, execution path, plugins | docs/development/architecture.md |
| How to run and write tests | docs/development/testing.md |
| Why it is built this way | ARCHITECTURE_DECISIONS.md; read the relevant ADR before changing a subsystem |
| What is guaranteed, and which test proves it | docs/guide/security.md |
| The original intent | docs/development/specification.md |
| How past fixes and tool choices were decided | docs/DECISIONS.md |
| What the last security audit found and left open | docs/SECURITY-AUDIT.md |
Invariants: never break these¶
These are the spec §8 hard constraints, each covered by tests (docs/guide/security.md maps
every one to its tests). They override everything else in this file. If a task seems to need
one weakened, stop and ask a human.
- No code change without a recorded human approval. Execution goes through
executor.service.assert_approved(latest review is an approval by rolereviewer|admin, and the task still has the scope that review approved), checked before the run and again right before the push. Never add a bypass, flag or "auto approve" path. - An approval covers what the reviewer saw. Anything that changes what the agent is told to
do must be part of
review.approval_scope_digest. Nothing may modify a task that isapprovedorexecuting(the aggregator defers evidence for such tasks). - No merge capability. Do not add merge/auto-merge to
SCMProvideror call merge APIs. Pushes go only toopsmith/...branches (scm.base.check_pushable).LocalSCM.human_mergeexists only to simulate a human in tests and the demo. - Redact before any LLM call. Model calls go through
LLMGateway.structured(or the executor agent, which only sees the confined workspace). Never call a provider SDK directly from pipeline code. - Untrusted data stays data. Logs, code, file names and model output go inside
<untrusted_data>viallm.prompts.untrusted_json. Validate model output against schemas, and drop unknown IDs and fabricated evidence refs. - Sandboxed code never runs outside the sandbox. Repository commands run only through a
Sandbox. After the agent starts, the platform runs git only against the trusted snapshot (run_git(..., git_dir=...)), never against the checkout's own.git. The diff gates run after the last sandboxed command. Thelocalbackend is development-only and must stay refused for non-local repositories (sandbox_policy_error); never fall back to it. - Least privilege. Agents get no shell, network or credentials. Tools are a fixed list.
Tokens are resolved from
credentials_ref(env:/file:) throughconfig.resolve_tenant_credentialand never persisted. Paths from configuration that touch the worker's filesystem go throughlocalfs.confine. - Tenant isolation. Every query filters by
tenant_id. Useapi.deps.owned()to load by id, which returns 404 for other tenants. Read routes require a human role;servicekeys may only push. - Auditability. State changes go through
audit_log.transition_task/transition_leadoraudit_log.record. FinishedInferenceRun/AuditRunrows are immutable (a flush guard raises). - No silent drops. Failures must surface: retry,
deadjob with notification, task back toin_reviewwith anexecution_report, or a finding markedsuppressedwith astatus_reason.
How to work¶
Opsmith decides what code an AI agent may change on other people's repositories. Stability, security and correctness come before speed and before features. Every session, by a person or an agent, follows these rules from the first step to the last. Where they seem to conflict, the invariants win, then these rules, then the conventions below.
Start from a known state¶
- Read before you touch. Read this file, the docs for the area you will change (the table above), and the ADR behind it.
- Take a baseline. Run
make checkbefore you change anything and note the result. If it is not green, find out why before you add to it, and say so in your report. - Understand, then plan. Find the code the change touches, its callers and its tests. For
a change across several modules, write the design down first (an ADR, or an entry in
docs/DECISIONS.md) and check it against the invariants before you write code.
Change the code with care¶
- Do what was asked, completely, and nothing else. Something else you find goes into its own commit, or into your report; never fold it silently into the change.
- Correct, simple and readable beats clever. A reviewer must be able to see that it is right. Follow the conventions below and the code around you.
- Every change comes with tests: the happy path, the failure path and, for anything that guards safety, a negative test that shows the bad path is blocked.
- A bug fix starts with a failing test. Write the test that reproduces the bug, run it, and see it fail for the reason you expect. Then fix the code and see it pass. The test stays, as the regression test.
- Secure by default. Validate input at the edge and bound it to what stores it. Keep
untrusted data as data. Never add a secret, credential, real customer data or token to
code, tests, fixtures, logs or commits, and never add an unsafe pattern (
shell=True,eval,pickle, string-built SQL, unconfined paths, unbounded reads, calls without timeouts). - Never hide a failure: no bare
exceptthat swallows, no skipped test to get green, no retry that masks a bug (invariant 10). - Verify before you act. Read the code a page describes before you change either. Check a claim from a reviewer, a tool or another agent against the code before you apply it; they are often right and sometimes wrong. A number in the docs (tests, coverage, a version) comes from a run you saw, not from memory.
Prove it before you call it done¶
- Run the checks. Iterate with targeted runs (
uv run pytest tests/test_<area>.py -q), then runmake checkonce on the final tree. Also runmake sitewhen docs changed, the suite on PostgreSQL when you touched models or queries, anduv run pytest -m dockerwhen you touched the sandbox, the executor or the external scanners (audits/external.py) and Docker is available. - A green run counts only if you saw it. A skipped test is not a pass: read the skip
reasons (
uv run pytest -q -rs). Coverage and "all tests pass" come from a full run of the current tree. Report failures with their output, never describe a red run as green, and say what you did not run. Never weaken a test, a gate or a bound to make something pass; if a test is wrong, fix it on its own and say why. - Never merge, and never ask for a merge, with a regression: a test that passed before must pass after, coverage must stay above the floor, and the build must be clean.
Leave the docs true, for every change¶
- Update the Markdown docs the change affects: the pages under
docs/, the README, this file,CONTRIBUTING.md. A behaviour the docs do not describe, or describe wrongly, is a bug. The documentation site is built from the same pages: add a new page tonavinmkdocs.yml, and runmake siteto prove it builds with no warning. - Update
CHANGELOG.mdin the same commit, under Unreleased (Keep a Changelog: Added, Changed, Fixed, Security, with Upgrade notes for anything an operator must do). The site's changelog page is this file. - Record significant decisions in
docs/DECISIONS.md, with the reasoning and the options not taken: a choice between workable designs, a tool, something left open on purpose. Design decisions about the system itself also get an ADR inARCHITECTURE_DECISIONS.md.
Ship it¶
- Release every change that reaches
main. Whoever merged it releases it at once: move the entries under## <version> - <yyyy-mm-dd>(a patch version for fixes, a minor one for features), set that version inpyproject.tomlandsrc/opsmith/__init__.py, runuv lockso thatuv.lockrecords it, tagv<version>, and publish a GitHub Release whose notes are that changelog section. Do not let unreleased changes collect onmain. - Before every push, look at what leaves. Read
git diff origin/mainand the commit messages. The docs, the README and the changelog must already describe the change. Scan the outgoing commits for secrets (for examplegitleaks git --log-opts="origin/main..HEAD") and for details of the machine or deployment you worked on. Never print a token or key in a command's output, a log or a message. - After a push to
main, check that it landed. Whoever pushed checks that CI passed, that the Docs workflow deployed GitHub Pages, and that the live site shows the new version and changelog. A failed build or deploy is fixed before other work.
Keep the workspace safe, and know when to ask¶
- Leave things as you found them. Remove scratch files, containers and test databases
you started, and delete any credential you were handed once the task is done. Use only
credentials given for the task, never ones you find on the machine. On a shared or small
host, check free memory before a heavy run (the suite writes over 1 GB of temporary
files; point
--basetempat disk when/tmpis RAM-backed), and never run two full suites at once. - Stop and ask when a task seems to need an invariant changed, a destructive action on
shared state (a running deployment, another project's files, history on
main), or a decision only the maintainer can make: the license, the security policy, anything posted publicly in their name.
Every session ends with a short report: what changed, how it was verified (the commands and their results), what was not run, and anything left open.
Commands¶
uv sync --all-extras # install (Python 3.12+, uv 0.9+)
make check # lint, format, mypy, tests + coverage floor, docs (CI's main job)
make fmt # auto-fix lint and format (src, tests, scripts)
make audit # known vulnerabilities in the locked dependencies (CI runs it)
make site # build the documentation site, strict (make site-serve to preview)
uv run pytest tests/test_<area>.py -q # while iterating
uv run pytest -q -rs # the whole suite (4-6 min), listing what was skipped and why
uv run pytest tests/test_e2e.py # spec §11 acceptance criteria, end to end
uv run pytest -m docker # tests that start real containers (skipped without Docker)
uv run python scripts/check_docs.py # docs: links, anchors, cited tests, undocumented features
uv run mypy # strict-ish typing on src/
uv run opsmith demo --dir /tmp/demo # full offline loop: ingest -> ... -> PR -> verified
make docker-test # the same checks inside the dev image (no local Python needed)
On a host where /tmp is RAM-backed or small, keep the suite's temporary files on disk:
PYTEST_ADDOPTS="--basetemp=/var/tmp/opsmith-pytest" make check, then remove that directory.
The whole suite also runs on PostgreSQL. Point OPSMITH_TEST_DATABASE_URL at a server whose
user may create databases; each test then gets its own database:
docker run -d --rm --name pg -e POSTGRES_USER=opsmith -e POSTGRES_PASSWORD=opsmith -p 127.0.0.1:5432:5432 postgres:18.6-alpine3.24
OPSMITH_TEST_DATABASE_URL=postgresql+psycopg://opsmith:opsmith@localhost:5432/opsmith uv run pytest -q
docker stop pg
Repository map¶
Path (under src/opsmith/ unless it starts at the root) |
Responsibility |
|---|---|
platform.py |
Composition root and queue handler wiring. Start here. |
models.py, schemas.py |
ORM entities (all tenant-scoped) / pydantic contracts incl. settings and LLM stage I/O |
migrations/ |
Alembic revisions of the schema, applied by every process at start (ADR-002) |
queue.py, scheduler.py |
Durable idempotent jobs with leased, heartbeated claims; cron/interval schedules |
audit_log.py |
Append-only audit trail, task/lead state machines, run immutability |
db.py |
Engine and sessions; utcnow, new_id and the SERIALIZE row lock |
security.py |
Tenants, users, the four roles and API keys (stored as SHA-256) |
protection.py |
Protected paths: elevated needs an admin approval, excluded is never changed |
notifications.py |
Notification outbox; delivery is a notify job |
dashboard.py, provenance.py |
Dashboard figures and FUNNEL; a task's provenance chain |
cli.py, demo.py, configfile.py |
The opsmith command, the offline demo, opsmith apply |
config.py, localfs.py |
Process settings and credential references; confinement of host-file access |
connectors/, etl/ |
Signal sources; normalize → enrich → redact → store → bundle |
redaction.py |
Secret/PII detectors shared by ETL, the gateway and the diff gate |
llm/ |
LLMGateway (the only path to a model for pipeline stages; the coding agents in executor/agents.py call their model directly, inside the confined workspace), prompts, pricing, providers |
inference/orchestrator.py |
Chained runs, lead carry/confirm/refute/expire |
audits/ |
Analyzers (built-in + external tools), the audit runner, and checks (checks.py): admin-written instructions a model carries out on the code (ADR-019) |
aggregation/ |
Correlation, dedup/suppression, scoring, task drafting |
review.py |
Human review gate and the approval scope digest |
executor/ |
Sandboxes, workspace confinement, coding agents, execution service, PR body |
gitutil.py |
The only place git is invoked: safe config, trusted git dir, unambiguous parsing |
scm/ |
GitHub + local providers. No merge operation, by design. Repository addresses and branches are checked before git sees them (scm.base.check_repository) |
tracking.py |
PR/CI tracking, post-merge verification |
api/, ui/ |
FastAPI JSON API (/api/v1) and server-rendered UI (/ui) |
tests/ |
Unit/integration tests; conftest.py seeds a tenant, users, project, git repo. test_hardening.py holds the adversarial tests, test_runtime.py the process loops |
docs/, root *.md |
The documentation. docs/guide/ for people running Opsmith, docs/development/ for people changing it, docs/DECISIONS.md and docs/SECURITY-AUDIT.md for the record |
mkdocs.yml, site/, scripts/build_site.py |
The documentation site: every Markdown page, published to GitHub Pages by .github/workflows/docs.yml |
scripts/check_docs.py |
Fails the build when the docs and the code disagree (links, settings, endpoints, cited tests) |
Conventions¶
- Python 3.12, full type hints,
mypyclean,ruffclean (line length 100). No relative imports. - Use
db.utcnow()for time (aware UTC) anddb.new_id(prefix)for ids. - JSON columns are replaced, never mutated in place:
task.paths = [*task.paths, p], nottask.paths.append(p). SQLAlchemy does not track in-place changes. - Enqueue follow-up work with
queue.enqueue(..., idempotency_key=...)in the same transaction as the state it depends on. Keys must be deterministic for the logical unit of work. - Handlers raise
queue.RetryLaterto wait (no attempt consumed) andqueue.PermanentFailurewhen retrying cannot help. - Keep long-running work (clones, agents, sandboxed commands) outside DB transactions. See
ExecutionService. - Modules under
src/opsmith/api/avoidfrom __future__ import annotationsso FastAPI can resolve dependency annotations. - Bump a prompt's
versioninllm/prompts.pywhenever its text changes. Runs record it. - No model vendor lock-in. Pipeline code depends on
llm.base.LLMProviderand onexecutor.agents.CodingAgent, never on a vendor SDK. Anything a provider needs that is vendor-specific stays inside that provider. A change to the model path must keeptests/test_provider_parity.pygreen: the acceptance scenario has to end the same on every provider. - Default Claude model is
claude-opus-5-5. Claude model ids and prices live inllm/pricing.py; update it when adding one. Models of other providers are priced by the stage settings (input_price_per_mtok,output_price_per_mtok). - Every stage that calls a model must record its usage in
UsageRecord, including the coding agent, because that ledger is what the budgets read. - Claude API code (
llm/anthropic_provider.py,executor/agents.py): callmessages.create, nevermessages.parse; setoutput_config.formatwithanthropic_provider.output_format(schema)for structured stages, and alwaysoutput_config.effort. Passtimeout=anthropic_provider.request_timeout(max_tokens). Use a manual tool loop with strict tool schemas. Handle therefusalandmax_tokensstop reasons; a tool call cut off bymax_tokensgets an errortool_resultand is never run. Count input withanthropic_provider.input_tokens_of(usage), and append the fullresponse.contentto the conversation.CLAUDE.mdgives the reasons. - Invoke git only through
gitutil.run_git. Parse its NUL-delimited output (-z), never the line-based formats: file names are attacker-controlled. - Validate at the edge. Project settings are range-checked in
schemas.ProjectSettings; add a bound whenever you add a setting, because a bad value otherwise fails later inside a job. Add the setting todocs/guide/configuration.mdand toopsmith.example.tomltoo. - A model change that alters the schema needs a migration (ADR-002): a new revision in
src/opsmith/migrations/versions/, written withuv run alembic revision --autogenerate(seedocs/development/architecture.md). Never edit a released revision.tests/test_migrations.pyfails when the models and the revisions disagree. Call the change out under Upgrade notes inCHANGELOG.md.
Testing¶
- Tests must not touch the network. Use
httpx.MockTransportfor HTTP. For the Anthropic SDK, run the real SDK overhttpx.MockTransport(seetests/test_llm.pyandtests/test_executor.py); a hand-written fake accepts any arguments and hides mistakes. - Use the
platform,seedanddrainfixtures.drain()runs queued jobs to completion. Fixtures use thelocalsandbox and setlocal_rootto the test's temp dir. - Any behaviour change to the pipeline needs a test at the module level, and the spec §11
assertions in
tests/test_e2e.pymust keep passing. - Security-relevant code (gates, confinement, redaction, tenancy) needs a negative test that
proves the bad path is blocked. Put it in
tests/test_hardening.pyor next to its area, and list it indocs/guide/security.md. - Tests that need a real Docker daemon carry
@pytest.mark.dockerand skip themselves when Docker or the image is unavailable. - Code must work on SQLite and PostgreSQL. PostgreSQL enforces column lengths and types that SQLite ignores; run the suite against it (see Commands) when you touch models or queries.
Extending¶
Connectors, analyzers and model providers are plugins (opsmith.connectors,
opsmith.analyzers, opsmith.llm_providers entry points; built-ins use @register). The
rules for each, and for adding a model stage, a task state or a setting, are in
docs/development/architecture.md. In
short:
- Connector: read-only; skip and count records it cannot parse; test with
httpx.MockTransport. - Analyzer: never execute repository code; isolate failures per file; mask secrets.
- Model provider: implement
LLMProvider; report usage, including on refusals and truncation. - LLM stage: add the output schema, the prompt, and a
HeuristicProviderbranch so the stage works offline. Call it throughLLMGateway.structured. - Task state: extend
TASK_TRANSITIONS, the UI status chips,dashboard.FUNNEL, and add an ADR entry.
Definition of done¶
-
make checkpassed before the change and passes after it (ruff, format, mypy, the suite with the coverage floor, the docs check), with no test newly failing or skipped. - New behaviour is covered by tests, including failure and negative paths; a bug fix has a test that failed before the fix.
- The invariants are untouched, or a human approved the change and an ADR records it.
- The Markdown docs match the code, and
make sitebuilds the documentation site. -
CHANGELOG.mddescribes every user-visible change, in the commit that makes it; once onmain, the change is released (tag and GitHub Release) and the live site shows it. - Significant decisions are in
docs/DECISIONS.md(and design changes inARCHITECTURE_DECISIONS.md), with the reasoning. - No secrets, tokens or real customer data in code, tests, fixtures, logs or commits; the outgoing commits were scanned before the push.
- The session's report says what changed, how it was verified and what was not run.
Commits and PRs¶
Use imperative, scoped messages (executor: block secrets on added lines only), one topic
per commit. Work on a branch and open a pull request; keep PRs focused, describe the risk and
how you verified the change, and wait for CI to be green. Never commit generated data
(data/, *.db, demo output, build/).