Iterate
Prompt Lab
Take low-scoring sessions, regenerate answers with a candidate prompt, compare metrics.
1Pick weak sessions
16 sessions below 70% pass
2Edit candidate prompt
apps/eval/prompts/candidates/<name>.md (NOT prod)
3Regenerate answers
gpt-4.1-nano · same Q + new prompt
4Compare metrics
OLD vs NEW · ship if better
Low-Scoring Sessions
click Run with new prompt to regenerate all questions in a session using the candidate prompt
| Session | Tier | Role | Q's | Pass | Worst metric | |
|---|---|---|---|---|---|---|
| best buy — Quick Practice | consumer | unknown | 13 | 0% | Context Precision: 0.18 | Run with new prompt |
| Quick Practice | consumer | unknown | 3 | 0% | Context Precision: 0.39 | Run with new prompt |
| Stripe — AI enginner (Copy 104) | enterprise | unknown | 4 | 0% | Opening: 0.27 | Run with new prompt |
| Stripe — AI enginner (Copy 96) | enterprise | unknown | 1 | 0% | Voice Authenticity: 0.25 | Run with new prompt |
| Stripe — AI enginner (Copy 88) | enterprise | unknown | 1 | 0% | Context Precision: 0.22 | Run with new prompt |
| Stripe — AI enginner (Copy 87) | enterprise | unknown | 1 | 0% | Opening: 0.49 | Run with new prompt |
| Stripe — AI enginner (Copy 15) | enterprise | unknown | 6 | 0% | Voice Authenticity: 0.28 | Run with new prompt |
| Stripe — AI enginner (Copy 12) | enterprise | unknown | 19 | 0% | Voice Authenticity: 0.33 | Run with new prompt |
| Stripe — AI enginner (Copy 3) | enterprise | unknown | 5 | 0% | Context Precision: 0.03 | Run with new prompt |
| Google — AI ml engineer | enterprise | unknown | 6 | 0% | Context Precision: 0.07 | Run with new prompt |
| Google — AI enginner (Copy 5) | enterprise | unknown | 2 | 0% | Faithfulness: 0.36 | Run with new prompt |
| Google — AI enginner (Copy 6) | enterprise | unknown | 25 | 0% | Faithfulness: 0.22 | Run with new prompt |
| Google — AI enginner (Copy) | enterprise | unknown | 1 | 0% | Context Precision: 0.26 | Run with new prompt |
| Google — AI enginner | enterprise | unknown | 8 | 0% | Context Recall: 0.01 | Run with new prompt |
| Google — AI enginner (Copy) | enterprise | unknown | 2 | 0% | Voice Authenticity: 0.44 | Run with new prompt |
| Google — AI enginner | enterprise | unknown | 3 | 0% | Voice Authenticity: 0.55 | Run with new prompt |
How regeneration works
- Copy prod prompt to a candidate —
copy apps\server\prompts\_base\VOICE_AND_AUTHENTICITY.md apps\eval\prompts\candidates\my-test.md. Edit the CANDIDATE file, NEVER touch the prod template directly. - Trigger the workflow — click "Run with new prompt" on a session above. Opens GH Actions
eval-run.yml. Pick mode "regenerate" and enter your candidate prompt path (relative to repo root) likeapps/eval/prompts/candidates/my-test.md - Pipeline runs — pulls each question from the session, calls
gpt-4.1-nanowith the new prompt, scores the new answers with the same 7-metric judge. - Compare + export — once the run finishes, the A/B Compare page shows OLD vs NEW side-by-side. Click Export Markdown to download a self-contained report you can hand to a prompt-review agent. The agent reviews the metrics + Q&A pairs and updates the prod template based on what worked.
See
apps/eval/README.md for the full CLI reference (regenerate command, A/B options, threshold tuning).