Science
Self-improving prompts: how we built our prompt platform
Why we built our own platform to version, evaluate, and continuously improve the prompts behind our product — and how a loop of automated evals and human-approved AI proposals replaced grading on gut feel.
Here's a confession that will sound familiar to anyone building with language models: for a long time, the way we knew a prompt was "good" was that someone read a handful of outputs and nodded. That's it. That was the test. A teammate would tweak an instruction, generate a few examples, squint at them, and say "yeah, that's better." And we'd ship it.
It worked, in the way that a lot of things work when you're small and moving fast. Until it didn't.
Prompts are product logic now
When you build an AI-native product, a surprising amount of your "source code" isn't code at all. It's prompts. The instructions we hand to models are where a huge share of our product's actual behavior lives — how an interview is conducted, how a follow-up question is phrased, how a summary is framed, how a synthetic respondent stays in character for twenty turns without breaking. These aren't config strings. They're the logic. Get them right and the product feels uncannily, almost suspiciously good. Get them subtly wrong and no amount of clean architecture underneath will save you.
The trouble is that prompts don't behave like the rest of your codebase. You can't unit-test a tone. You can't diff a vibe. A one-word change can ripple through thousands of generations in ways you won't see until a customer does. For a while, we treated prompts like ordinary code anyway — and then like ordinary content — and both framings quietly failed us. Two problems, specifically.
Problem one: a hundred prompts and no map
Our prompts didn't live buried in source files — we'd already moved them into our database, which was a real step up. They were centralized, they were queryable, they weren't hostage to a deploy. Better than most teams start out.
But "in a database" is not the same as "managed." Each prompt carried its own little world of configuration — which model it ran on, which provider, the temperature, the fallback behavior, the variables it expected — and that surface area kept growing. We crossed a hundred prompts without ever really deciding to. And the questions that should have been easy got hard. Which prompts run on which model? What changed in this one three weeks ago, and who changed it, and why? What's the exact version live in production right now versus the one we were experimenting with? The answers existed, scattered across rows and memories and the occasional Slack thread, but there was no single place that told the truth and kept the history. We had storage. We didn't have a control plane.
Problem two: we were grading on gut feel
The second problem was the uncomfortable one, and it's the "someone nodded" story from the top. Our entire quality bar for a prompt change was human intuition applied to a small, hand-picked sample. And intuition is a genuinely valuable signal — but as a process it has three fatal properties: it doesn't scale, it isn't consistent between people or across weeks, and it drifts without anyone noticing.
Picture the failure mode. You improve a prompt so it handles one kind of input beautifully. The three examples you check look great. You ship. What you can't see is the other kind of input — the edge case you didn't sample — where your change quietly made things worse. Nobody catches it, because nobody re-reads the thousand generations that prompt now touches every day. With a hundred-plus prompts each producing a steady stream of output, "read a few and trust your gut" had stopped being a strategy and started being a liability we'd simply gotten used to.
We didn't need better intuition. We needed a way to turn intuition into evidence — and to do the bulk of the measuring without a human in the loop for every single check.
The spark: Anthropic shipping evals into skill-creator
The idea that crystallized all of this came from watching how Anthropic rolled out evaluation in their skill-creator. Instead of authoring a skill and eyeballing whether it worked, they built a loop around it: create, evaluate, improve, benchmark. Every change gets measured — run against the version before it, scored on explicit criteria, with the specific failures fed back in as the raw material for the next iteration. It even splits its test set so it's improving against held-out cases rather than overfitting to the ones it already passed.
The clever part wasn't a clever prompt. It was a posture. Treat prompts as artifacts you measure and improve on a loop — the same way no serious team ships code without tests, and no serious team decides a feature is "done" because it looked fine on someone's screen once. That reframing was the permission we needed to stop polishing prompts by hand and start building the machine that would do it for us.
What we built
The platform we built is one place that owns every prompt, every version, and every configuration — and, layered on top of that, a loop that keeps improving them.
A note on the screenshots. The images throughout this post use mock data. The prompt names, contents, evaluation scores, and proposals shown are synthetic examples for illustration — not our real prompts.
A single source of truth. Every prompt now lives in one system, fully versioned, with complete history and change notes. You can see exactly what's deployed to each environment, lock a version so it can't drift out from under you, and roll changes out deliberately instead of hopefully. Versioning here isn't bureaucracy — it's the thing that makes the rest of the loop trustworthy. You can only know whether a change helped if you know, precisely and permanently, what changed.
Quality you can see, not sample. For each prompt we define explicit quality criteria — concrete scoring rubrics, not a vague "is this good?" An LLM acting as a judge scores outputs against those rubrics and shows its reasoning, so a score is something you can interrogate rather than just trust. And because every prompt reports the same kind of number, quality stops being a feeling you sample occasionally and becomes a signal you can watch move — per criterion, per version, and across the entire fleet at a glance. A prompt that's slipping shows up as at risk or failing before a customer finds it for you.
Evaluation that runs itself. The measuring doesn't wait for a human to remember to do it. Once a day, each prompt is scored against its criteria on a fresh, diverse sample of real recent usage — not a stale set of hand-picked examples — and the score is tracked over time against a pass threshold. When it drops below the line, an alert fires. This is the piece that finally kills the silent-regression failure mode: the edge case you didn't think to check gets checked anyway, every day, automatically.
The loop actually closes now
Everything above is measurement. Measurement tells you that a prompt is slipping; it doesn't fix it. The part we're most fond of — and the newest — is what happens next.
We run headless agents on a schedule. After the daily scoring, an agent looks at each prompt that needs attention and reads three things together: the eval signal (which criteria are weak, and by how much), the real observations behind those scores (what the prompt actually produced in the wild), and the prompt's own human-written knowledge file (what it's supposed to do). Then it decides whether a change is warranted — and if it is, it doesn't touch the prompt. It writes a proposal.
That distinction is the whole design. An agent that edits production directly is a liability; an agent that drafts a well-reasoned suggestion and hands it to a human is leverage. Every automated change — a reworded instruction, a new scoring criterion, an added edge-case test, an update to the knowledge file — lands in one place: a review inbox.
The proposals inbox: AI-drafted changes across every prompt, grouped and filterable by type, each tagged with the loop that produced it.
Open one and you get the agent's case, not just its conclusion: a plain-language explanation of why it's suggesting the change, and — for a prompt revision — the exact diff. You can reject it, accept it (which creates a new version but leaves it inactive), or accept and deploy it in one step. Nothing the agent proposes ever goes live on its own. A newer proposal of the same kind quietly supersedes the older one, so the inbox always reflects the latest thinking rather than piling up.
There's a subtle trap in a system like this, and it's worth naming because avoiding it shaped the design. If the same agent both writes the prompts and defines the criteria they're graded on, it can quietly start gaming its own test — rewriting the rubric to flatter the behavior it already produces. So the two jobs are split into two loops that run on staggered schedules, both anchored to real observations and to the human-authored knowledge file, and both gated behind human approval. The criteria loop isn't allowed to invent standards that merely rationalize the status quo. The human stays in the loop precisely where judgment is irreplaceable, and nowhere else.
One agent, everywhere
Alongside the scheduled loops, the same capability is available on demand. Every prompt has a prompt-aware assistant built into its page — one agent that can already see that prompt's evals, its runs, its history, and its knowledge file. You can ask it to tighten an instruction, run a test, score the result, or grow the eval suite, and it works against the real context instead of a blank chat window. It's the same machinery the nightly loops use, just with you driving.
Improvement stops being something a person does on a Tuesday, and becomes something the system does every day — with humans in the loop.
A living memory for every prompt
Threaded through all of this is a document that predates the agents and quietly makes them work: each prompt's knowledge file. It captures the prompt's purpose, what's been tried, what actually worked, and how to think about scoring it — in plain language, human-readable and human-editable. It's the source of truth for intent, and it's the anchor every agent reads before it acts and writes back to as it learns. It's also the guardrail that keeps the improvement loop honest: when an agent is deciding whether a change is warranted, "what is this prompt actually for" is a question the knowledge file answers, not one the agent gets to invent. The institutional memory that used to evaporate the moment a teammate moved on now accumulates in one place, where the next person — or the next agent — can pick it up.
We're not finished — the loop only gets tighter as it runs, with cleaner criteria and prompts that need less and less human nudging. But the fundamental shift has already happened. Our prompts are no longer loose text we manage by hand and grade on feel. They're versioned, measured, and — now, genuinely — getting better on their own, with us kept exactly where we add the most value: saying yes or no.