Why test a prompt
If a cook makes one good soup, we do not yet know if the recipe is good. Maybe the cook was lucky that day. A recipe is good when it works on many days, with many pots. We test a prompt the same way: on many inputs, not just one.
- A prompt can look perfect on the one example you tried and fail on the next. Testing means running the prompt on many realistic inputs and scoring the results.
- Anthropic's documentation on defining success criteria and building evaluations says that defining success criteria and then designing evaluations to measure performance against them is central to prompt engineering. Its flowchart of the process runs: test cases, preliminary prompt, iterative testing and refinement, final validation, ship.
- An eval (short for evaluation) is a test you run on a prompt: a set of inputs plus a way to score the answers.
- Anthropic's prompting guide says that where a technique names a specific model, it was measured on that model, and you should re-check it against your own evals before applying it to another model. So tests also protect you when the model changes.
- AI answers are not identical each time. Running the same prompt twice can give different results, so one good run is weak evidence.
- Testing is not only for programmers. A teacher, an office worker or a shop owner can run a test set by hand in a spreadsheet.
| Without testing | With testing |
|---|---|
| "It worked when I tried it." | "It passed 17 of 20 cases." |
| You change the prompt and hope. | You change one thing and see if the score moves. |
| A new model arrives; you do not know what broke. | You re-run your cases and see exactly what changed. |
- Pick one prompt you use at least weekly (for example the chain or JSON prompt from course 1.2.2).
- Write the job of the prompt in one sentence.
- Run it on one input and note the result.
- Run it again on the same input, in a fresh chat. Note whether the answer differs.
- Now run it on a very different input (a short one, a messy one, one with missing information). Note what happens.
- Write one line: "I do not yet know if this prompt works because ...".
- Check: you have seen at least one difference between runs or inputs, and you can explain why one try is not enough.
Videos by other creators on YouTube, for reference. They are not made by or affiliated with The Future Market.
Read the video script
[Host] "One good answer proves nothing. Maybe you were lucky."
[Screen: same prompt run on three different inputs, one fails] "Same prompt, three different inputs. Two work. One breaks."
[Text on screen: Prompt, inputs, score] "Testing is simple: a prompt, many inputs, and a score."
[Host] "Do that, and you stop hoping and start knowing."
Run one prompt of yours on three different inputs, twice each. Write down every difference you notice between runs.
ClaudeStuck? Copy a ready-made prompt for Claude
Copy this, open Claude, paste it, and Claude will walk you through the lesson step by step.
For a grown-up: Claude is for adults (18+). Copy this, paste it into Claude, and use the answer to explain the lesson to your child.
I'm learning "Prompt Testing: evaluate and improve your prompts" on The Future Market. Lesson 1: "Why test a prompt". What the lesson teaches: - A prompt can look perfect on the one example you tried and fail on the next. Testing means running the prompt on many realistic inputs and scoring the results. - Anthropic's documentation on defining success criteria and building evaluations says that defining success criteria and then designing evaluations to measure performance against them is central to prompt engineering. Its flowchart of the process runs: test cases, preliminary prompt, iterative testing and refinement, final validation, ship. - An eval (short for evaluation) is a test you run on a prompt: a set of inputs plus a way to score the answers. - Anthropic's prompting guide says that where a technique names a specific model, it was measured on that model, and you should re-check it against your own evals before applying it to another model. So tests also protect you when the model changes. - AI answers are not identical each time. Running the same prompt twice can give different results, so one good run is weak evidence. - Testing is not only for programmers. A teacher, an office worker or a shop owner can run a test set by hand in a spreadsheet. - Without testing - With testing The exercise I'm working on: Run one prompt of yours on three different inputs, twice each. Write down every difference you notice between runs. I'm stuck. Please: 1. Explain the main idea in plain, simple words with one everyday example. 2. Walk me through the exercise one small step at a time. Ask me what I've done so far before giving the next step. 3. Don't just give me the final answer. Help me do it myself, and check my work when I share it.
I'm a parent helping my child (about 8 years old) with a lesson called "Why test a prompt" from the course "Prompt Testing: evaluate and improve your prompts". Here's what the lesson says: If a cook makes one good soup, we do not yet know if the recipe is good. Maybe the cook was lucky that day. A recipe is good when it works on many days, with many pots. We test a prompt the same way: on many inputs, not just one. The activity: Run one prompt of yours on three different inputs, twice each. Write down every difference you notice between runs. Please explain this so an 8-year-old understands: short sentences, a fun everyday example, and no hard words. Then give me 3 simple questions I can ask my child to check they understood, and a way to do the activity together, one tiny step at a time.
Sources
- Anthropic, Define success criteria and build evaluations: platform.claude.com/docs/en/test-and-evaluate/develop-tests
- Anthropic, Prompting best practices (General principles): platform.claude.com/docs/en/build-with-claude/prompt-engineering/claud
Define what "good" means
This lesson is part of full access: every course and every level for one payment.
Build a test set
This lesson is part of full access: every course and every level for one payment.
Score answers with a rubric
This lesson is part of full access: every course and every level for one payment.
Compare prompt versions fairly
This lesson is part of full access: every course and every level for one payment.
Find and reduce errors
This lesson is part of full access: every course and every level for one payment.
Track changes and keep what helps
This lesson is part of full access: every course and every level for one payment.
Final project
This lesson is part of full access: every course and every level for one payment.
Quiz
Why is one good answer not enough to trust a prompt?
AI answers can vary between runs and inputs, so you need many realistic cases.
What is an eval?
A test for a prompt: a set of inputs plus a way to score the answers.
Name Anthropic's four qualities of good success criteria.
Specific, measurable, achievable and relevant.
Why include edge cases in a test set?
Real inputs include odd, missing, long or irrelevant ones, and the prompt must handle them.
What is a held-out test set?
Cases you do not use to write or tune the prompt, kept for a fair final check.
What is a golden answer?
The correct answer you decide on in advance for a test case.
Name the three grading methods in Anthropic's documentation.
Code-based, human and LLM-based grading.
What is a good rule about who grades with an AI?
Use a different model from the one that produced the answers, and spot-check the grades yourself.
Why change only one thing between versions?
So you know which change caused the result.
What is a regression?
A case that used to pass and now fails after a change.
What does permission to say "I don't know" do?
It can drastically reduce false information.
Do these techniques remove hallucinations completely?
No. They reduce them, so you still validate important outputs.
Final exam
This lesson is part of full access: every course and every level for one payment.
Certificate
This lesson is part of full access: every course and every level for one payment.