Before you start, and from idea to testable claim
Saying "my lucky socks help my team win" is not a test. A test says: "I will wear them for 20 games and count wins, and I will compare with 20 games without them." You decide the rules first. Then you look. And you accept the result even if it is boring.
- Education only. Testing is how you find out when an idea fails. It never makes an idea safe.
- This course assumes you finished course 5.10.2: you can build a simple indicator by hand, you wrote a rule down before testing, and you split data into in-sample and out-of-sample parts.
- A hypothesis is a claim you can check. A null hypothesis is the boring claim: "the rule does nothing special; results are what luck would give." You try to see whether the data is hard to explain with luck alone.
- A testable claim has: the exact rule (numbers, not adjectives), the data (market, dates, timeframe), what you measure (for example the price change 20 candles later), the costs and the pass/fail line written before you look.
- Vague: "Price bounces at support." Testable: "On the daily chart of market X, from 2015 to 2024, when the close is within 1% of the lowest low of the last 50 days, record the change in close 10 days later. Null: the average change is the same as for all other days."
- Baseline: always compare with something simple, such as the average change on all days, or a random rule. A rule that "makes money" in a rising market might only be riding the rise.
- Writing the rule first matters because rules chosen after seeing results are a known problem in the research (Park and Irwin, 2007, covered in course 5.10.2).
- Pick one idea you have read about (for example "price bounces near a 50-day low"). You are not testing whether to trade it.
- Write the idea as one sentence with numbers (like the testable example above).
- Write the null hypothesis in one sentence.
- Write the data: market, dates, timeframe (names of buttons and tools can differ by site).
- Write what you will measure and the baseline you will compare with.
- Write the pass/fail line before you look: "I will say the idea showed something only if ___ and I will still call it unproven."
- Date the page and keep it. Do not edit it after you look at any data.
- Check: another person could run your test from the page alone, and the pass/fail line is dated before any data was seen.
Videos by other creators on YouTube, for reference. They are not made by or affiliated with The Future Market.
Read the video script
[Host] "A chart idea is not a test. It becomes a test when you write it down first."
[Text on screen: exact rule, data, measure, baseline, pass/fail line] "Five parts: the exact rule, the data, what you measure, a baseline and your pass or fail line."
[Text on screen: Null hypothesis = luck explains it] "Then ask the boring question: could luck alone explain this result?"
[Screen: a dated test plan page] "Write the plan, date it and do not change it after you look."
[Host] "Education only. Testing is how we find out where an idea breaks."
Write one dated test plan with the five parts and a null hypothesis. Do not look at any data yet and do not place any trade.
ClaudeStuck? Copy a ready-made prompt for Claude
Copy this, open Claude, paste it, and Claude will walk you through the lesson step by step.
For a grown-up: Claude is for adults (18+). Copy this, paste it into Claude, and use the answer to explain the lesson to your child.
I'm learning "Testing Chart Ideas: backtests and statistics (education only)" on The Future Market. Lesson 1: "Before you start, and from idea to testable claim". What the lesson teaches: - Education only. Testing is how you find out when an idea fails. It never makes an idea safe. - This course assumes you finished course 5.10.2: you can build a simple indicator by hand, you wrote a rule down before testing, and you split data into in-sample and out-of-sample parts. - A hypothesis is a claim you can check. A null hypothesis is the boring claim: "the rule does nothing special; results are what luck would give." You try to see whether the data is hard to explain with luck alone. - A testable claim has: the exact rule (numbers, not adjectives), the data (market, dates, timeframe), what you measure (for example the price change 20 candles later), the costs and the pass/fail line written before you look. - Vague: "Price bounces at support." Testable: "On the daily chart of market X, from 2015 to 2024, when the close is within 1% of the lowest low of the last 50 days, record the change in close 10 days later. Null: the average change is the same as for all other days." - Baseline: always compare with something simple, such as the average change on all days, or a random rule. A rule that "makes money" in a rising market might only be riding the rise. - Writing the rule first matters because rules chosen after seeing results are a known problem in the research (Park and Irwin, 2007, covered in course 5.10.2). The exercise I'm working on: Write one dated test plan with the five parts and a null hypothesis. Do not look at any data yet and do not place any trade. I'm stuck. Please: 1. Explain the main idea in plain, simple words with one everyday example. 2. Walk me through the exercise one small step at a time. Ask me what I've done so far before giving the next step. 3. Don't just give me the final answer. Help me do it myself, and check my work when I share it. 4. This is for learning only. Remind me to check with a licensed professional before I make a real decision about my own money, health or legal situation.
I'm a parent helping my child (about 8 years old) with a lesson called "Before you start, and from idea to testable claim" from the course "Testing Chart Ideas: backtests and statistics (education only)". Here's what the lesson says: Saying "my lucky socks help my team win" is not a test. A test says: "I will wear them for 20 games and count wins, and I will compare with 20 games without them." You decide the rules first. Then you look. And you accept the result even if it is boring. The activity: Write one dated test plan with the five parts and a null hypothesis. Do not look at any data yet and do not place any trade. Please explain this so an 8-year-old understands: short sentences, a fun everyday example, and no hard words. Then give me 3 simple questions I can ask my child to check they understood, and a way to do the activity together, one tiny step at a time. Keep it general and educational. This is not advice for a real situation.
Sources
- Park, C.-H. and Irwin, S. H. (2007), "What Do We Know About the Profitability of Technical Analysis?", Journal of Economic Surveys 21(4): econpapers.repec.org/RePEc:bla:jecsur:v:21:y:2007:i:4:p:786-826
- CME Group Education, Technical Analysis course: www.cmegroup.com/education/courses/technical-analysis.html
Backtest anatomy: win rate, expectancy, drawdown and costs
This lesson is part of full access: every course and every level for one payment.
Luck, sample size and p-values
This lesson is part of full access: every course and every level for one payment.
Many tries, false winners: data snooping and multiple testing
This lesson is part of full access: every course and every level for one payment.
Out-of-sample, walk-forward and why time order matters
This lesson is part of full access: every course and every level for one payment.
Failure modes gallery: survivorship, regime change, costs and drawdowns
This lesson is part of full access: every course and every level for one payment.
Capstone: a pre-written test plan and an honest report
This lesson is part of full access: every course and every level for one payment.
Final project
This lesson is part of full access: every course and every level for one payment.
Quiz
What is a null hypothesis?
The boring claim that the rule does nothing special and luck explains the results.
Why must the pass/fail line be written before looking at the data?
Otherwise you can move it to fit the result.
What is expectancy?
The average result per case: win rate x average win minus loss rate x average loss, minus costs.
A 25% fall needs what rise to recover?
About 33.3%.
What does a p-value tell you?
How surprising a result would be if luck alone were at work; it is not the chance the rule works.
What happens to the wobble of a win rate when you have four times as many cases?
It is cut in half.
If 20 independent tests each have a 5% false-positive chance, what is the chance of at least one false positive?
About 64%.
What is a trials log for?
To record every rule and setting tried so the number of tries is part of the result.
What is a walk-forward test?
Building on one block of data, testing on the next, and repeating in time order.
Why not shuffle time-series data for testing?
The test could learn from data after the moment it predicts, which flatters the result.
What is survivorship bias?
Testing only markets that still exist, so failures are missing and results look too good.
Which wording is careful for a result?
"The result was positive on this sample but is unproven."
Final exam
This lesson is part of full access: every course and every level for one payment.
Certificate
This lesson is part of full access: every course and every level for one payment.