Operations¶
Running Opsmith day to day: what to watch, how to upgrade, and what to do when something goes wrong.
Health and metrics¶
GET /healthzchecks that the API can reach the database. It stays at the root underOPSMITH_BASE_PATH, as does/metrics.GET /metricsis Prometheus text: stage latency, model calls with their cost and latency, ingested records, executions and HTTP timing.
Both are unauthenticated. Keep the API port private, or restrict those two paths at your proxy.
The Dashboard in the UI shows the same picture per project: what is waiting for review, open and escalated leads, dead jobs, cost in the last 24 hours, and cost and latency per stage.
Jobs and workers¶
Every stage is a queued job.
- A job that fails is retried with backoff. When its attempts are used up it becomes
dead, which is audited and sent as ajob.deadnotification. - See jobs in the UI under Jobs, or
GET /api/v1/jobs?status=dead. An admin can retry a dead job from either place. - Run as many workers as you need. A running job's lease is renewed while its worker is alive,
so a slow job is never handed to a second worker. If a worker dies, its job is picked up
again after
OPSMITH_WORKER_LEASE_SECONDS. - Stop a worker with
SIGTERM. It finishes the job it is running first if it is given the time: an execution can take up tomax_execution_seconds(30 minutes by default), anddocker compose stopkills after 10 seconds, so usedocker compose stop -t 1800 worker. A killed job is retried afterOPSMITH_WORKER_LEASE_SECONDS. - More than one scheduler is safe: a schedule slot is enqueued once however many ask.
- A repository is audited or checked by one job at a time. An audit or check that starts
while another one of the same repository is running is deferred by a minute (
another audit of this repository is running), because both would use the same checkout. - A
job.deadnotification goes to the channels of the project the job worked for, whatever the job's kind.
Things worth an alert:
| Signal | Means |
|---|---|
A job.dead notification |
A stage gave up after its retries. Read the job's error, fix the cause, retry it |
task.execution_failed |
An approved task did not become a pull request. The task is back in review with a report |
inference.failed |
A model call failed or returned unusable output. See Models |
budget.exceeded |
A token or cost cap stopped a stage. Raise the cap or wait for the next day |
check.failed |
A check failed or did nothing: no model can follow it (the offline rules cannot), no file matches its paths, or a model call failed. The reason is on the run |
lead.escalated |
The analysis could not decide about something for several runs. A person should look |
| An audit run that shows an error on the dashboard | An analyzer or scanner failed. The run still records what the others found, and the next audit re-checks everything |
Notifications¶
Configure a webhook per project to hear about all of the above (Configuration). Delivery is itself a queued job, so a failed delivery is retried.
Audit trail¶
Every state change and every decision is in the audit log, with who made it:
GET /api/v1/audit-log, or the bottom of any task page in the UI.
Backups¶
Back up the PostgreSQL database and the data directory (OPSMITH_DATA_DIR) together. The
database is the source of truth; the data directory holds the raw signals that evidence refers
to. With Compose:
docker compose exec -T postgres pg_dump -U opsmith -Fc opsmith > opsmith-db-$(date +%F).dump
docker run --rm -v opsmith_opsmith-data:/data:ro -v "$PWD":/backup postgres:18.6-alpine3.24 \
tar czf /backup/opsmith-data-$(date +%F).tgz -C /data .
To restore the database, start only postgres and run
docker compose exec -T postgres pg_restore -U opsmith -d opsmith --clean < opsmith-db-DATE.dump.
Upgrading¶
Opsmith migrates its database itself: every process brings the schema to the newest revision when it starts, so an upgrade needs no separate step (ADR-002). Releases that change the schema say so under Upgrade notes in the changelog. Processes that start together wait for each other, on PostgreSQL and on SQLite; one of them migrates.
Stop the processes before you upgrade, so that nothing old runs against the new schema, and take a backup (see Backups): the backup is how you go back to an older release.
git pull
docker compose stop -t 1800 worker # let a running execution finish
docker compose stop api scheduler # the database keeps running for the backup
# back up the database and the data directory here (see Backups)
docker compose up -d --build # the first process to start migrates; keeps the
# sandbox overlay only if COMPOSE_FILE is in .env
A process refuses to start, with a message that starts database schema:, when:
- the database was created by a release before 1.0.0. There is nothing to migrate it from; start from a fresh database, as the 1.0.0 upgrade notes say.
- the database was migrated by a newer release than the one starting. Run the newer release, or restore the backup taken before the upgrade.
A database created by 1.0.0, which had no migrations, is recognized and adopted when a newer release first opens it.
Troubleshooting¶
| Symptom | Cause and fix |
|---|---|
required variable POSTGRES_PASSWORD is missing a value |
Set it in .env. This is deliberate |
Task returns to review with sandbox unavailable |
The worker cannot use Docker. See Execution sandbox |
Task returns to review with sandbox policy |
OPSMITH_SANDBOX_BACKEND=local with a non-local repository. Use docker |
docker could not start the sandbox |
The image is missing or cannot be pulled, or OPSMITH_WORK_DIR is not shared with the host at the same path |
| Build, lint or test commands fail with download or DNS errors | The sandbox has no network. Bake dependencies into OPSMITH_SANDBOX_IMAGE |
task changed after it was approved |
The task's scope changed between approval and execution. Review it again |
... is outside OPSMITH_LOCAL_ROOT or reads the worker's filesystem, which is disabled |
Set OPSMITH_LOCAL_ROOT to the directory holding your local repositories and signal files |
... names a variable reserved for the platform |
A credentials_ref pointed at one of Opsmith's own secrets. Use a dedicated variable |
An analysis run is failed with a validation error |
The model did not return JSON matching the stage's schema, twice. Use a more capable model for that stage |
model endpoint unreachable or HTTP 4xx from the model endpoint |
Check OPSMITH_OPENAI_BASE_URL, the key, and the model name in the project settings. A 429 or 5xx is retried twice before it is reported |
An audit or execution fails with repository ...: a github repository needs an https:// clone URL (or default_branch must be a plain branch name) |
The repository was saved before 1.2.2 with a value git must not be given. Correct it with opsmith apply or the API |
diff introduces credential-like values naming a file the agent did not write |
A build, lint or test command left build output in the checkout (for example compiled bytecode holding a string from the source). Ignore those files in the repository's .gitignore |
An audit or check is deferred with another audit of this repository is running |
Expected: audits and checks of one repository share a checkout and run one at a time |
request body is over ... bytes (413) |
A request over 1 MiB, or a push over 32 MiB. Send pushes in smaller batches |
Permission denied on /var/run/docker.sock |
DOCKER_GID does not match the socket's group |
| UI form posts are rejected with 403 behind a proxy | The proxy must pass the browser's Host header through unchanged |