Build your own harness: prompts, rituals and quality checks

Step by step

1. 1. Start from the four layers, not from the prompt text

   The assistant is arranged as four layers. Capture collects what happened: meetings, documents, user feedback, calendar, chat, email. Harness turns that into a repeatable way of working: a library of prompts, agent instruction sets, day/week/month rituals, templates, company rules and quality scoring. The assistant uses the harness to come to you on its own and say what you missed, what to hand over and in which format. A separate check layer walks the product through the eyes of a novice. The harness is the second layer, and a prompt on its own is not the product: it is one component of the wrapper that makes prompts work. Before editing any text, decide which layer you are working on. If the assistant is silent, that is the assistant layer and the ritual schedule, not the prompt library. If the answer is right but shaped wrong, that is the prompt library.

2. 2. Read the contract of every prompt before editing it

   In the Harness screen, the Prompt library section lists prompts by key, task type and locale. Opening one shows four things: the instruction, what goes in, what comes out, and the quality criterion. Task types are the recurring jobs of a product manager — discovery, decision, handoff, retro, sizing, requirements analysis, learning and the daily digest. A prompt that already carries all four parts is meant to be launched as it is; if you need different behaviour, the change belongs in a new version of that instruction, not in a fresh prompt written from zero. The same library holds the body of the prompt, so the text you run and the text you read are the same text. Read it the way you would read a contract with a new colleague: what is expected, what is forbidden, and how the result is judged.

3. 3. Put a ritual on the schedule and check the log

   Rituals are regular procedures that run on a schedule: a daily digest, a weekly overview, a monthly retro. Each ritual gathers its own material and produces a result on its own, so the assistant’s activity does not depend on your discipline. The part that matters is what happens when a ritual does not run. The skip is recorded as a separate entry with a reason, and it appears in the journal — a ritual that vanished without a trace is a defect, because after a month nobody can tell whether there was nothing to report or the scheduler fell over. In the Rituals section you see the schedule and the next run. Choose the cadence that matches how the team actually goes over its work: a daily digest that nobody reads is noise, and a monthly retro that collects too much material is unreadable.

4. 4. Change a prompt by creating a new version

   A version is immutable: its body never changes afterwards. A new edit creates a new version with its own number, its own note and a hash of the body, so a quiet modification can be detected rather than argued about. Before a new version is applied it is run against the evaluation set, and the result is shown next to the previous one — before and after. Rollback switches the current version back and deletes nothing: the old version and the score it received stay in the history. Change one thing at a time. If the instruction, the input format and the output format are edited together, the evaluation result will not tell you which of the three changes helped, and you will be tempted to judge by eye. A new version is also the unit of meaning for the run log: each recorded run stores which version of which prompt produced it, so months later you can say which text produced which artifact.

5. 5. Build an evaluation set from your own tasks

   An evaluation set is a list of examples with an expected result. Each case carries an input, an expected behaviour, the fields the answer must contain, and the fields it must not contain. The forbidden fields are usually numbers: market size, revenue, ROI, effort in days, conversion rate. The model does not invent those, and a version that fills them in has failed the case even if the prose around them is fine. A prompt change without an evaluation is not accepted — “it reads better now” is not a result you can compare. The set lives in the repository next to the prompts, so the person checking it reads examples and prompt text together, and a new case is an ordinary change that goes through the same check. Start with the cases that hurt most: the answer you had to rewrite completely, the meeting you had to summarise by hand, the artifact that went out with the wrong tone.

6. 6. Read the quality dashboard as feedback, not as a score

   The dashboard answers three questions: where the assistant was wrong, where you rewrote its output, and where it missed something. Three metrics sit above them. Take rate is the share of proposals accepted without edits. Evaluation score is the share of matches with the reference on the evaluation set. Recoverability is the share of errors the assistant noticed and corrected on its own. Read the breakdown per prompt before touching any text: a prompt that gets rewritten often is a signal about the harness, not a mistake by the person who rewrote it, and the same holds for a proposal that is regularly ignored. Silence is a valid state and is not treated as a refusal to use the product. The dashboard collects signals from real runs, so it becomes useful only after the assistant has been used — an empty dashboard is not a verdict.

7. 7. Fix the boundaries: untrusted input and unconfirmed extraction

   Two rules keep the assistant trustworthy, and both belong to the harness rather than to your prompt text. The first: customer content is untrusted input. Documents, transcripts, letters and feedback reach the model as data and never as instructions to execute; a line inside a document that tells the assistant to ignore its rules is a fragment to analyse, not an order to follow. The second: extracted material is not context until a person confirms it. Everything the capture layer extracts arrives with the status “not verified”, and before confirmation it does not enter answers, is not cited as a fact and does not affect any calculation. The pipeline is: a channel event goes to extraction, extraction is marked “not verified”, a person confirms it, and only then does a context node with provenance appear. If you change a prompt, check that it does not weaken either rule, and if a rule is violated in practice, that is a defect to report rather than a wording problem to smooth over.

8. 8. Close the loop with a naive-user check and with learning

   The fourth layer of the assistant is the check. The product is walked through as a first-time user who knows nothing about its internals and is allowed to ask an obvious question, in the sandbox environment you specify; the check never writes to your production data. The result is a map of sticking points — where the naive user stopped, what they did not understand, which path they could not find — with steps that reproduce each one. The feedback is not softened: what broke, how much it interferes, and what to change. Formulas such as “on the whole, not bad” are not accepted. Findings you accept become backlog tasks with a priority, a link to the reproduction and an assessment of the impact. The same layer carries learning: the assistant explains a method on your own material, first your example and then the general principle, checks understanding with a short question and brings weak topics back for repetition. What you want to keep from a lesson becomes a decision and enters the context through the usual confirmation path.

If it does not work

The assistant’s answers sound generic and do not fit how our team writes.

Open the quality dashboard and read the rewrite breakdown per prompt. A prompt that is rewritten often does not fit your company. Create a new version with a narrower instruction, run the evaluation set, and compare the result before and after. Keep the change in the version history instead of editing the old text in place — that is what makes the next comparison possible.

I changed a prompt and the quality of the answers got worse.

Every new version is evaluated on the example set before it is applied, and the score is shown next to the previous one. Compare the two, then roll back to the earlier version. Rollback switches the current version and keeps the history, so nothing is lost and the failed attempt stays available as evidence.

The ritual did not run and I received nothing in the channel.

A skipped ritual is recorded with a reason instead of disappearing. Open the ritual journal, read the reason, fix the schedule or the underlying cause, and re-run the ritual. If there was genuinely nothing to report, that is a valid outcome — but it is shown as a completed run with an empty result rather than as silence.

The assistant cites a statement that nobody in the team confirmed.

That breaks the confirmation rule: extracted nodes carry the status “not verified” and stay out of answers until a person confirms them. Open the citation, read the source fragment, and either confirm the node or reject it. If such statements should appear in answers regularly, the supported path is to confirm them in the context — not to let the assistant treat them as settled.

My evaluation set does not cover the case I actually care about.

Add a case with an input, an expected result, the required fields and the forbidden fields. Cases live in the repository next to the prompts and are checked together with them, so this is an ordinary change. Without a matching case a prompt change cannot be measured: the score will not move, and the result will be judged by eye.