opsgentic

OpsGentic — Architecture

How OpsGentic turns an alert into a reviewed GitOps pull request. For setup see the Quickstart; for day-to-day operation see USAGE.md.

Overview

A trigger (a Grafana/Alertmanager alert or a user chat message) hits the API, which enqueues the run on a Postgres-backed task queue and returns 202 { thread_id, status, poll_url }. A worker consumes the job and runs a stateful LangGraph graph across three core agents:

• RCA Agent — enriches context (via read-only MCP tools) and produces a root cause hypothesis.
• Validation Agent — verifies the hypothesis using deterministic Validation Skills, then drafts a remediation plan.
• Action Agent — opens a Git commit + pull request so a GitOps controller (ArgoCD/Flux) can apply the change.

Two tool layers feed the agents:

• MCP Servers (read-only) — reused open-source MCP servers (kubernetes-mcp-server for the cluster, github-mcp-server for the repo) for querying cluster/observability/repo state. No mutation happens here; writes only occur through GitOps.
• Validation Skills — plain-Python business logic the Validation Agent calls directly, independent of MCP.

Agent instructions are not hard-coded: each agent composes its system prompt from an editable agent-skill library. The Action Agent never opens a PR autonomously — the graph pauses at an interrupt_before gate for human approval (unless AUTO_APPROVE=true, where PR review/merge becomes the gate).

Async execution & the task queue

OpsGentic decouples the API from the (minutes-long) agent run:

• The API enqueues the run on a Postgres-backed task queue (Procrastinate) and returns 202 immediately.
• A separate worker Deployment consumes the queue and drives the graph, recording the lifecycle (queued → running → awaiting_approval → applied/failed) in an opsgentic_runs table.
• Clients poll GET /runs/{thread_id} for status + state.
• The queue reuses the same Postgres used for durable checkpoints — no extra broker, identical locally and on Kubernetes.
• Without DATABASE_URL (local/dev) there is no queue: the API runs the graph synchronously in-process and returns the full result, and the CLI runs it directly.

Machine state & graph flow

trigger -> [enqueue] -> worker:  RCA -> resolve_target -> Validation -> [interrupt_before] -> Action -> END
                                            |                                  ^
                                            +-- (fail / unresolved repo) ------+ (self-heal loop)

1. RCA reads alert_payload, gathers context_data, writes hypothesis.
2. resolve_target maps the alerting workload to its GitOps repo/path (see Multi-repo resolution).
3. Validation runs the Validation Skills. On pass (and a resolved repo) it drafts a remediation_plan and sets execution_status="awaiting_approval". On fail or unresolved repo it loops back to RCA up to MAX_RCA_ATTEMPTS, then escalates.
4. The graph pauses before action (interrupt_before=["action"]); state is persisted by the checkpointer keyed on thread_id. No PR exists yet.
5. A human approves (or AUTO_APPROVE=true); the run resumes and the Action Agent opens (or updates) the remediation PR.

State survives restarts via a durable PostgresSaver checkpointer (DATABASE_URL); without it, an in-memory checkpointer is used (single process, dev only).

Remediation: surgical edits, convergence on re-fire

The Action Agent reads the resolved repo via github-mcp-server and proposes surgical field edits (forced via structured output), which OpsGentic applies to the real manifest with comments/formatting preserved (ruamel). If no edits can be produced, it falls back to a remediations/<key>.md proposal.

A stable issue key (repo|file|alertname|namespace) drives a deterministic branch, so a re-fired alert does not open a duplicate PR. Instead OpsGentic converges (GitHub):

• It reads what the open PR already proposes (the base...branch diff) and asks the agent whether that already resolves the alert.
• Sufficient → it posts one comment, no new commit.
• Insufficient → it commits a single incremental edit on the branch.

This avoids the divergent-commit churn from regenerating a (non-deterministic) fix on every re-fire.

Agent skills

Agent instructions live in an editable markdown library instead of hard-coded prompts:

• deploy/manifests/agent-skills/{sre,gitops,validation,code}.md — each has YAML frontmatter (name, description, agents: [...]) plus a markdown body.
• A skill is wired to agents via its agents field; the loader (agent_skills.py) composes the bodies of all skills targeting an agent into its system prompt. Code-bound mechanics (the remediation edit schema, the resolver index-only answer) stay in code.
• Wiring: rca/context → sre; remediation → sre + gitops + validation + code; resolver → gitops.
• Delivery: shipped as the opsgentic-skills ConfigMap (mounted at /app/agent-skills), so prompts are tunable without an image rebuild (a baked-in copy is the fallback). See USAGE.md → Editing agent skills.

Multi-repo resolution

OpsGentic maps an alert to the GitOps repo/path that owns the affected workload, so one deployment serves many repos. Resolution chain (first match wins):

1. Explicit alert labels gitops_repo / gitops_path (override / fast path).
2. ArgoCD — the Application whose status.resources (or destination namespace) contains the workload → spec.source.repoURL + path + targetRevision.
3. Flux — the workload’s kustomize.toolkit.fluxcd.io/{name,namespace} labels → Kustomization → GitRepository URL + path.
4. Unresolved → escalate (OpsGentic does not guess a repo).

The workload identity is derived from alert labels (workload/deployment/app, or the pod name with its replicaset/pod suffix stripped). Discovery is deterministic — CRDs are read read-only through the kubernetes-mcp-server; on an equal-confidence tie the LLM picks among candidates.

The resolved host selects a provider from the registry (config/gitops.yaml): each git host maps to a type (github / gitea / gitlab), an API base, and the env var holding that provider’s token. Auth: a fine-grained PAT (GITHUB_TOKEN) is simplest; a GitHub App is preferred for production (short-lived installation tokens). See USAGE.md → Configuration.

Components

1. Orchestrator & agents (src/opsgentic/graph) — LangGraph StateGraph over a typed MachineState; interrupt_before approval gate.
2. Validation Skills (src/opsgentic/skills) — deterministic Python checks, decoupled from MCP.
3. Agent skills (deploy/manifests/agent-skills, src/opsgentic/agent_skills.py) — editable markdown prompt library composed per agent.
4. MCP servers & tooling (mcp-config/, deploy/manifests/mcp/, src/opsgentic/mcp) — the read-only gateway for cluster + repo reads (no direct k8s client, no kubectl).
5. GitOps (src/opsgentic/gitops) — alert→repo resolver, provider registry, PR create + re-fire convergence, remediator, YAML editing.
6. Task queue & worker (src/opsgentic/tasks.py, runs.py, worker.py) — Procrastinate over Postgres; the API enqueues, the worker executes.
7. Deployment manifests (deploy/manifests/) — namespace, read-only RBAC, MCP servers, Postgres, ConfigMap/Secret, API + worker Deployments, Service.

Project layout

src/opsgentic/
  config.py             # Settings (env-driven)
  agent_skills.py       # markdown agent-skill loader (frontmatter -> per-agent prompt)
  agents/llm.py         # ChatOpenAI factory pointing at vLLM
  graph/
    state.py            # MachineState (TypedDict)
    builder.py          # StateGraph wiring + interrupt_before
    nodes/              # rca / resolve_target / validation / action
  skills/               # deterministic Validation Skills (Python)
  mcp/                  # MCP loader + read-only context enrichment
  gitops/               # resolver, provider registry, PR create + convergence, remediator, yamledit
  triggers/normalize.py # Grafana + chat -> alert_payload
  runner.py             # enqueue (async) + execute (sync) + status tracking
  tasks.py              # Procrastinate app + tasks (run_alert / resume_run)
  runs.py               # opsgentic_runs status table
  worker.py             # queue worker entrypoint (opsgentic-worker)
  main.py               # FastAPI service (async, 202 + polling)
  cli.py                # local runner (synchronous)
mcp-config/             # MCP server config (servers.yaml)
config/gitops.yaml      # git provider registry (host -> provider/token)
deploy/manifests/       # K8s manifests + agent-skills/ (-> ConfigMap)
docs/                   # ARCHITECTURE.md, USAGE.md, figures/

This site is open source. Improve this page.