opsgentic

OpsGentic — Quickstart

Three ways to run opsgentic:

0) One-command demo — ./hack/demo-up.sh brings up the cluster, ArgoCD, Prometheus, the demo apps, and opsgentic. Start here.
A) Local dev — no cluster, fastest, for trying the graph and editing prompts.
B) Kubernetes (manual) — the full end-to-end path done by hand, for when you want to control each step.

End state (Path B): a failing demo workload fires an alert → Alertmanager calls opsgentic → opsgentic runs RCA → Validation → Remediation and opens a GitOps PR on your fork of the demo repo → you (or PR review) approve → ArgoCD applies the fix.

demo app (failing) --alert--> Alertmanager --webhook--> opsgentic API --queue--> worker
                                                                                    │
                                          GitHub PR on your fork  <--- remediation ─┘
                                                    │ merge
                                              ArgoCD sync --> fix applied

Prerequisites

For	You need
Local dev	Python 3.11+; a vLLM / OpenAI-compatible endpoint + key (optional — empty falls back to a canned response)
Kubernetes	A cluster + `kubectl`; `kustomize` (or `kubectl` ≥ 1.27 with `-k`); a container registry; kube-prometheus-stack (Prometheus Operator + Alertmanager); ArgoCD; a GitHub account
Both	`openssl`, `jq` (handy for the test commands)

0. One-command demo

The fastest way to see the whole loop. One script provisions everything and wires it together:

./hack/demo-up.sh

It walks you through, in order:

Cluster — uses your current kubectl context if a cluster is reachable; otherwise offers to spin up a local k3d / minikube / kind cluster (handy if you don’t want to install Kubernetes).
GitHub auth — prints the fine-grained PAT creation link (needs Contents, Pull requests, and Administration → Read & write), reads your token, and forks lehuannhatrang/demo-workload into your account automatically.
LLM — asks for an OpenAI-compatible endpoint; leave it blank to use the built-in canned-response fallback.
Helm installs — ArgoCD and kube-prometheus-stack (Prometheus + Alertmanager), with the Alertmanager selectors pre-configured so the demo’s cross-namespace routes are picked up (no manual step B5).
Demo workloads — creates one ArgoCD Application per app in apps/* pointing at your fork, with auto-sync. The apps start failing and firing alerts.
OpsGentic — generates bootstrap.env from your answers and runs ./bootstrap.sh.

The script is idempotent — re-run it any time. When it finishes it prints the access URLs (ArgoCD, Prometheus/Grafana, the opsgentic API + console) and a one-line curl to verify end-to-end. Any prompt can be pre-answered with an env var (e.g. GITHUB_TOKEN=… LLM_BASE_URL=… ./hack/demo-up.sh).

The generated bootstrap.env contains your PAT and is gitignored — never commit it.

Tear everything down (Helm releases, opsgentic, demo apps) with:

./hack/demo-down.sh

It leaves your cluster running and just prints the command to delete a local cluster if you made one.

For the manual, step-by-step version of this same flow, read on (paths A and B).

A. Local dev

python -m venv .venv && . .venv/bin/activate
pip install -e .
cp .env.example .env          # set LLM_BASE_URL / LLM_API_KEY / LLM_MODEL

Leave DATABASE_URL empty: the API then runs each request synchronously in-process (no queue/worker), and the CLI runs the graph directly.

# one-shot via the CLI
opsgentic --file examples/grafana_alert.json --source grafana --approve

# or the HTTP service (returns the full result synchronously when DATABASE_URL is empty)
opsgentic-api                 # :8080
curl -s localhost:8080/chat -XPOST -H 'content-type: application/json' \
  -d '{"title":"t","message":"payments-api high memory","labels":{"namespace":"payments","app":"payments-api"}}' | jq .

Locally, the MCP servers (cluster/repo reads) are not reachable, so RCA uses a stub context and remediation falls back to a proposal. For real RCA + manifest-editing remediation, use Path B.

B. Kubernetes (end-to-end)

B1. Fork the demo repo + ArgoCD

Fork github.com/lehuannhatrang/demo-workload to your account — opsgentic opens remediation PRs here.

In ArgoCD, create one Application per demo app pointing at your fork (or one ApplicationSet over apps/*):

App	Path	Scenario	opsgentic should fix
payments-api	`apps/payments-api`	memory near limit	raise `resources.limits.memory`
checkout-api	`apps/checkout-api`	CPU saturation	raise `resources.limits.cpu`
orders-api	`apps/orders-api`	crash-loop (early liveness probe)	raise `livenessProbe.initialDelaySeconds`
inventory-api	`apps/inventory-api`	OOMKilled	raise `resources.limits.memory`

Sync them — the workloads start failing and fire alerts. opsgentic discovers the owning repo from these ArgoCD Applications automatically.

B2. GitHub auth — pick one

opsgentic needs Git credentials to read your fork (via github-mcp) and open PRs.

Default — a fine-grained PAT (easiest). GitHub → Settings → Developer settings → Fine-grained tokens → Generate:

Repository access: only your demo-workload fork.
Permissions: Contents → Read & write, Pull requests → Read & write.
Put it in GITHUB_TOKEN. One token covers both PRs and repo reads (GITHUB_MCP_TOKEN defaults to it).

Optional — a GitHub App (production). Short-lived installation tokens instead of a long-lived PAT:

New GitHub App; Repository permissions Contents: RW + Pull requests: RW; webhooks off; generate a private key (.pem); note the App ID.
Install it on your fork; note the Installation ID (.../installations/<ID>).
Set GITHUB_APP_ID / GITHUB_APP_INSTALLATION_ID / GITHUB_APP_PRIVATE_KEY_PATH, and leave GITHUB_TOKEN empty.

B3. Build & push the image (Optional)

docker build -t docker.io/<you>/opsgentic:dev-1.0 .
docker push docker.io/<you>/opsgentic:dev-1.0

Set the same value in deploy/manifests/kustomization.yaml (images: → newName/newTag) — the bootstrap script does this for you.

B4. Bootstrap (deploy everything)

cp bootstrap.env.example bootstrap.env     # fill in: LLM, GITHUB_TOKEN (PAT) or App, image
./bootstrap.sh

bootstrap.sh (idempotent):

generates the opsgentic-secrets + postgres-secrets Secrets (deploy/manifests/secrets.yaml, gitignored) from bootstrap.env — the GitHub PAT (or App PEM if you chose the App) and a generated Postgres password;
sets the image in kustomization.yaml;
kubectl apply -k deploy/manifests/;
patches opsgentic-config (App IDs / model / auto-approve) and rolls out;
waits for Postgres, the MCP servers, the API, and the worker.

Check:

kubectl -n opsgentic get pods       # postgres, kubernetes-mcp, github-mcp, opsgentic, opsgentic-worker

Always deploy with kubectl apply -k (kustomize), never -f: the base relies on a configMapGenerator (agent skills) and images:.

B5. Wire Alertmanager → opsgentic

Each demo app ships an AlertmanagerConfig routing its alert to http://opsgentic.opsgentic.svc:80/webhook/grafana (sendResolved: false). For kube-prometheus-stack to pick up these cross-namespace configs, the Alertmanager CR must select them:

kubectl -n monitoring get alertmanager -o yaml \
  | grep -A4 -iE "alertmanagerConfigSelector|alertmanagerConfigNamespaceSelector"

You need alertmanagerConfigNamespaceSelector to include the app namespaces (or be empty = all) and alertmanagerConfigSelector to match label release: kube-prometheus-stack. (In the kube-prometheus-stack Helm values: alertmanager.alertmanagerSpec.alertmanagerConfigNamespaceSelector: {} and ...alertmanagerConfigSelector matching your release label.)

B6. Verify end-to-end

NODE=<any-node-ip>           # opsgentic Service is NodePort 31080
# fast path: synthetic chat trigger
TID=$(curl -s $NODE:31080/chat -XPOST -H 'content-type: application/json' \
  -d '{"title":"high memory","message":"payments-api high memory","labels":{"namespace":"payments","app":"payments-api"}}' | jq -r .thread_id)
curl -s $NODE:31080/runs/$TID | jq '{status, awaiting_approval, pr_url: .state.pr_url}'

Poll until status is applied (or awaiting_approval if AUTO_APPROVE=false → approve at /runs/$TID/approve or open /ui/$TID). A PR appears on your fork.

Real path: let an app’s alert fire (e.g. inventory-api OOMKilled). Alertmanager → webhook → opsgentic enqueues → the worker runs the graph → a PR opens on your fork. Re-fires of the same alert converge on the existing PR (a comment, or one incremental commit) instead of stacking divergent commits.

Editing agent skills

Agent instructions are editable markdown, not hard-coded prompts:

deploy/manifests/agent-skills/{sre,gitops,validation,code}.md

Each file has frontmatter (name, description, agents: [...]) + a body. The agents field wires a skill to agents (rca, context, remediation, resolver); the loader composes the bodies into each agent’s system prompt.

To change behavior (no image rebuild):

# edit deploy/manifests/agent-skills/gitops.md
kubectl apply -k deploy/manifests/                                  # updates the opsgentic-skills ConfigMap
kubectl -n opsgentic rollout restart deploy/opsgentic deploy/opsgentic-worker

(Skills are read once at startup, so the restart is required.) Add a brand-new skill by dropping agent-skills/<name>.md with agents: [...] and adding it to the configMapGenerator in kustomization.yaml.

Troubleshooting

Symptom	Check
`apply -k` errors on `kind: Kustomization` / `generateName`	You used `-f`. Use `kubectl apply -k deploy/manifests/`.
Trigger returns a result, not `202`	`DATABASE_URL` is empty → synchronous mode (no queue). Set it for async + worker.
Run stuck at `queued`	Worker not running: `kubectl -n opsgentic logs deploy/opsgentic-worker`.
Alert never reaches opsgentic	Alertmanager selectors (B5); confirm the alert is firing in Prometheus.
PR url is `stub://...`	GitHub App not configured (App ID / Installation ID / PEM) or no token.
`context_data.source = stub` / proposal-only PR	MCP servers unreachable or github-mcp auth — `kubectl -n opsgentic logs deploy/opsgentic-worker`; `python -m opsgentic.mcp.diagnose`.

Full env reference: see docs/USAGE.md → Configuration.

This site is open source. Improve this page.