Human in the Loop: Building a Stateful Kubernetes Operator with LLMs

Our previous stateful operator - a real-estate data ingestion controller - took nearly three months of dedicated engineering. For the next one, handling Postgres failover, PV claims, and multi-cluster reconciliation in a climate-tech DMRV pipeline, we tried to build it with heavy LLM assistance. The same team, comparable complexity, a different problem domain.

The first attempt failed. Generated code handled the happy path correctly and the failure paths incorrectly - reconcile logic that passed CI and produced state divergence only under concurrent reconciliation. Separately, the model kept importing public PyPI packages our internal mirrors had blocked or deprecated, breaking the build outright.

We abandoned the Copilot pattern and built a structured process: 23 bounded tasks, a context system injecting our real CRDs and internal registry manifests, and automated validation before any human review. This talk gives the error-funnel numbers - what automation caught, what reviewers caught, what escaped to staging - the review-hour cost, and the decisions an LLM must never own in stateful infrastructure code.