THE FUTURE MARKETACADEMY
Your classroomRoom 01 · AI Lab
School of AI

Prompt Testing: evaluate and improve your prompts

What you'll be able to do
  • Explain why one good answer does not prove a prompt works, and how testing fixes that.
  • Write success criteria that are specific and measurable.
  • Build a small test set that includes normal cases, edge cases and answers you already know.
  • Score answers with a rubric, using code-style checks, human checks and an AI grader where each fits.
  • Compare two versions of a prompt fairly and keep only the changes that help.
  • Reduce errors, and keep a log so you know what changed and why.

Before this course: Prompting for Complex Tasks: chains, roles and structured outputs (course 1.2.2). You already know how to split a job into a chain, label the parts of a prompt, ask for tables and JSON, add examples and self-checks, and debug a prompt on three inputs. Course 1.2.2 ended with a test log on 3 inputs. This course makes that testing serious: a proper test set, a scoring rubric, fair comparisons and a change log. No coding is required; where Anthropic's pages show code, we do the same thing by hand in a chat or a spreadsheet.

Words in this course 50Open the Vocabulary Hall
01

Why test a prompt

In simple words

If a cook makes one good soup, we do not yet know if the recipe is good. Maybe the cook was lucky that day. A recipe is good when it works on many days, with many pots. We test a prompt the same way: on many inputs, not just one.

Key facts
  • A prompt can look perfect on the one example you tried and fail on the next. Testing means running the prompt on many realistic inputs and scoring the results.
  • Anthropic's documentation on defining success criteria and building evaluations says that defining success criteria and then designing evaluations to measure performance against them is central to prompt engineering. Its flowchart of the process runs: test cases, preliminary prompt, iterative testing and refinement, final validation, ship.
  • An eval (short for evaluation) is a test you run on a prompt: a set of inputs plus a way to score the answers.
  • Anthropic's prompting guide says that where a technique names a specific model, it was measured on that model, and you should re-check it against your own evals before applying it to another model. So tests also protect you when the model changes.
  • AI answers are not identical each time. Running the same prompt twice can give different results, so one good run is weak evidence.
  • Testing is not only for programmers. A teacher, an office worker or a shop owner can run a test set by hand in a spreadsheet.
Without testingWith testing
"It worked when I tried it.""It passed 17 of 20 cases."
You change the prompt and hope.You change one thing and see if the score moves.
A new model arrives; you do not know what broke.You re-run your cases and see exactly what changed.
Step by step
  1. Pick one prompt you use at least weekly (for example the chain or JSON prompt from course 1.2.2).
  2. Write the job of the prompt in one sentence.
  3. Run it on one input and note the result.
  4. Run it again on the same input, in a fresh chat. Note whether the answer differs.
  5. Now run it on a very different input (a short one, a messy one, one with missing information). Note what happens.
  6. Write one line: "I do not yet know if this prompt works because ...".
  7. Check: you have seen at least one difference between runs or inputs, and you can explain why one try is not enough.
Watch on YouTube
Evaluate prompts in the Anthropic ConsoleAnthropic · YouTube · youtube.com

Videos by other creators on YouTube, for reference. They are not made by or affiliated with The Future Market.

LESSON 01Why test a prompt
LESSON 01 / ANIMATED LESSON / AI VOICE00:00 / 01:00
Read the video script
[Host] "One good answer proves nothing. Maybe you were lucky."
[Screen: same prompt run on three different inputs, one fails] "Same prompt, three different inputs. Two work. One breaks."
[Text on screen: Prompt, inputs, score] "Testing is simple: a prompt, many inputs, and a score."
[Host] "Do that, and you stop hoping and start knowing."
Try it now

Run one prompt of yours on three different inputs, twice each. Write down every difference you notice between runs.

ClaudeStuck? Copy a ready-made prompt for Claude

Copy this, open Claude, paste it, and Claude will walk you through the lesson step by step.

For a grown-up: Claude is for adults (18+). Copy this, paste it into Claude, and use the answer to explain the lesson to your child.

I'm learning "Prompt Testing: evaluate and improve your prompts" on The Future Market. Lesson 1: "Why test a prompt".

What the lesson teaches:
- A prompt can look perfect on the one example you tried and fail on the next. Testing means running the prompt on many realistic inputs and scoring the results.
- Anthropic's documentation on defining success criteria and building evaluations says that defining success criteria and then designing evaluations to measure performance against them is central to prompt engineering. Its flowchart of the process runs: test cases, preliminary prompt, iterative testing and refinement, final validation, ship.
- An eval (short for evaluation) is a test you run on a prompt: a set of inputs plus a way to score the answers.
- Anthropic's prompting guide says that where a technique names a specific model, it was measured on that model, and you should re-check it against your own evals before applying it to another model. So tests also protect you when the model changes.
- AI answers are not identical each time. Running the same prompt twice can give different results, so one good run is weak evidence.
- Testing is not only for programmers. A teacher, an office worker or a shop owner can run a test set by hand in a spreadsheet.
- Without testing - With testing

The exercise I'm working on:
Run one prompt of yours on three different inputs, twice each. Write down every difference you notice between runs.

I'm stuck. Please:
1. Explain the main idea in plain, simple words with one everyday example.
2. Walk me through the exercise one small step at a time. Ask me what I've done so far before giving the next step.
3. Don't just give me the final answer. Help me do it myself, and check my work when I share it.
I'm a parent helping my child (about 8 years old) with a lesson called "Why test a prompt" from the course "Prompt Testing: evaluate and improve your prompts".

Here's what the lesson says:
If a cook makes one good soup, we do not yet know if the recipe is good. Maybe the cook was lucky that day. A recipe is good when it works on many days, with many pots. We test a prompt the same way: on many inputs, not just one.

The activity:
Run one prompt of yours on three different inputs, twice each. Write down every difference you notice between runs.

Please explain this so an 8-year-old understands: short sentences, a fun everyday example, and no hard words.
Then give me 3 simple questions I can ask my child to check they understood, and a way to do the activity together, one tiny step at a time.
Open Claude
Sources
My notes
02

Define what "good" means

This lesson is part of full access: every course and every level for one payment.

03

Build a test set

This lesson is part of full access: every course and every level for one payment.

04

Score answers with a rubric

This lesson is part of full access: every course and every level for one payment.

05

Compare prompt versions fairly

This lesson is part of full access: every course and every level for one payment.

06

Find and reduce errors

This lesson is part of full access: every course and every level for one payment.

07

Track changes and keep what helps

This lesson is part of full access: every course and every level for one payment.

++

Final project

This lesson is part of full access: every course and every level for one payment.

??

Quiz

  1. Why is one good answer not enough to trust a prompt?

  2. What is an eval?

  3. Name Anthropic's four qualities of good success criteria.

  4. Why include edge cases in a test set?

  5. What is a held-out test set?

  6. What is a golden answer?

  7. Name the three grading methods in Anthropic's documentation.

  8. What is a good rule about who grades with an AI?

  9. Why change only one thing between versions?

  10. What is a regression?

  11. What does permission to say "I don't know" do?

  12. Do these techniques remove hallucinations completely?

EX

Final exam

This lesson is part of full access: every course and every level for one payment.

CE

Certificate

This lesson is part of full access: every course and every level for one payment.