KubeTective — Kubernetes incident investigation engine

KubeTective investigates Kubernetes incidents end to end: it collects deterministic evidence (pod state, events, logs, deployment owner graph, HPA pressure, PVC, service, GitOps CRs), builds a timeline and evidence graph, scores candidate root causes with a calibrated, explainable model, and only then consults an LLM for narrative — never for scoring. Every investigation is recorded and replayable; every recommendation is evidence-linked; every action is human-gated.

Install

Homebrew

brew install gledilami/kubetective/kubetective

kubectl plugin

make install-plugin   # puts kubectl-investigate on your PATH
kubectl investigate deployment/checkout --since=30m

Binary / container

# GitHub release assets (linux/darwin, amd64/arm64, deb, rpm)
# container image:
docker pull ghcr.io/gledilami/kubetective:latest

Quickstart

# investigate a failing pod (or deployment, PVC, ...)
kubectl investigate pod/checkout-7f84c9
kubectl investigate deployment/checkout --since=2h

# run as a kubectl plugin
kubectl investigate deployment/checkout

# preflight your environment
kubetective doctor

# replay a recorded incident
kubetective replay <incident-id>

# search the incident store
kubetective incidents search --target checkout --since=7d --limit 10

Every command above works with zero configuration against your current kubeconfig. Optional sources (Prometheus, Loki, a git checkout) extend the evidence; an optional OpenAI-compatible LLM adds narrative on top — never authority.

Integrations

Investigate from an alert (PagerDuty, Grafana, Slack)

# Grafana webhook payload -> investigation
kubetective alert grafana --file=alert.json

# PagerDuty v2 webhook, or a Slack slash-command payload
echo '{...pagerduty...}' | kubetective alert pagerduty
echo '{"text":"deployment/checkout since=2h"}' | kubetective alert slack

Zero API keys: the payload is parsed locally, the engine uses its existing cluster access. Grafana kubernetes_* alert labels, PagerDuty incident titles, and Slack command text all name the target; payloads without one fail with a readable error instead of guessing.

Completion webhook (outbound)

# kubetective.yaml
webhook_url: https://ops.example.com/kubetective
webhook_secret: <shared-secret>

Every investigation POSTs a notification signed with HMAC-SHA256 (X-Kubetective-Signature). Receivers must verify the signature before parsing. Notification failure never fails the investigation.

Servers

kubetective serve exposes POST /v1/investigate, GET /v1/incidents[/{id}], /healthz and /metrics; kubetective mcp speaks the Model Context Protocol over stdio (read-only tools).

Scoring & calibration

Every score decomposes into the evidence lines that produced it, drawn from a documented six-band weight scale. A fitted temperature is adopted only when the benchmark can justify one: the suite must contain incorrect predictions, the fit must sit inside its search grid, and it must beat the default out-of-sample on two independent proper scoring rules (NLL and Brier) by a margin wider than noise. Adoption stayed refused until the suite carried enough real failures to support one — six live recordings later, at 71% measured accuracy, it does.

The decision deliberately does not run on expected calibration error. ECE is binned, so on a suite whose confidences sit inside two buckets a single scenario can swing the out-of-sample estimate by twenty points — and it did, flipping the adopt/refuse verdict on every second scenario added. ECE is still reported, because “displayed confidence is off by this much” is the number a human wants; nothing hangs on it.

Four gates run in CI — scenario, mutation (delete a verdict's evidence, the verdict must move), noise (bury it under 500 irrelevant observations, the verdict must hold), and false-positive. The evaluation report is a published, per-release artifact: reports/evaluation/latest.md.

Operations

Least-privilege RBAC: deploy/rbac.yaml — read-only collector role, namespaced write role for human-gated rollback/restart actions.
Structured logs: --log-format json|text; self-metrics via /metrics; pprof behind --pprof.
Records: JSONL incident store with a versioned schema — records written by a newer engine are rejected loudly, never silently mis-read.

Security

Incident records (which may carry log snippets) are owner-only on disk. The completion webhook is HMAC-signed. Container images are distroless and run as a non-root user. See SECURITY.md and the v1.0 security review.