How to Backtest a Trading Strategy Without Fooling Yourself

A backtest can prove almost anything. The real problem begins when the result looks so good that we stop asking why.

You have an idea for a strategy. Buy after a breakout. Enter when RSI falls below 30. Trade a moving-average crossover. Inside Day. NR7. Opening Range Breakout.

On a chart, it looks reasonable. A few trades line up almost perfectly. You define the rules, run a backtest, and get:

+187% in five years.

Great?

Not necessarily.

A good backtest is not supposed to answer:

“Can I find a set of rules that made money in the past?”

That is relatively easy.

The harder question is:

“Do I have a reason to believe the result is more than chance?”

That is where real research begins.

A beautiful equity curve is not evidence

Imagine a simple experiment.

We test a strategy on US stocks. It has several parameters:

• moving-average length,
• stop-loss level,
• entry time,
• minimum volume,
• volatility filter.

Each parameter has several possible values.

Very quickly, one strategy becomes dozens or hundreds of variants.

If we choose the version that made the most money historically, we are almost automatically selecting the variant that benefited the most from luck.

Think of it like flipping coins.

Flip one coin ten times and getting eight heads may look interesting.

Flip one thousand coins ten times each and you will almost certainly find one that produces nine or even ten heads.

That does not mean you discovered a magical coin.

Backtests behave in a similar way.

Rule one: separate the data used to search from the data used to verify

One of the simplest protections is to divide history into two periods.

The first is train — the data used to develop and select the idea.

The second is test — data that is not used while searching for the strategy.

For example:

2015–2022 → research / train
2023–2025 → test / out-of-sample

You can compare variants and explore rules on the first period.

But once you choose a winner, the rules are frozen.

Only then do you examine the second period.

If a strategy looks excellent on train data and falls apart immediately on the test period, you have learned something important:

the apparent historical edge may have been the result of fitting the past.

A negative out-of-sample result can be more valuable than another attractive equity curve.

Look-ahead bias: a strategy that knows the future

One of the most dangerous backtesting mistakes is look-ahead bias.

It happens when an algorithm uses information that was not actually available at the moment the decision was supposedly made.

A simple example:

A strategy generates a signal using Monday's closing bar.

The backtest also fills the trade at Monday's closing price.

The problem?

The final closing price is only known once the session has ended.

If the complete candle is needed to generate the signal, realistic execution may only be possible later — for example, at the next session's open.

The difference may look small.

Across thousands of trades, it can completely change the result.

Similar problems appear with fundamental data that was published later, future index membership, splits, delistings, or testing old markets using today's list of surviving companies.

A backtest can be mathematically correct while still being impossible to execute in the real world.

Transaction costs matter more than they appear to

This is especially brutal in short-term trading.

Assume a strategy trades frequently and has only a small average edge per trade.

Without costs, it looks excellent.

Then we add:

• commissions,
• spread,
• slippage.

Suddenly a strategy that was mildly profitable before costs becomes negative after costs.

That is not a cosmetic adjustment.

It can be the difference between:

“We found an effect”

and:

“We found a reliable way to pay the broker.”

For intraday research, this is particularly important. The shorter the timeframe and the more frequent the trading, the less room there is to ignore realistic execution costs.

A high win rate can still lose money

“The strategy wins 78% of the time.”

Sounds impressive.

But win rate by itself tells you very little.

You could have:

• 78% winning trades at +1%
• 22% losing trades at -6%

That strategy may have a great win rate and terrible expectancy.

What matters is not only how often you win, but also:

• how much you make on average,
• how much you lose on average,
• how deep the drawdowns are,
• how stable the results are,
• whether the effect survives outside the period where it was discovered.

One instrument can create the illusion of an edge

Suppose you test a strategy on ten stocks.

The aggregate result is positive.

It looks good.

Then a closer review shows that nearly all the profit came from one stock.

Remove that stock and the strategy loses money.

That changes the interpretation.

Maybe you did not discover a general market effect.

Maybe you found a pattern that happened to fit one instrument particularly well.

That is why robustness checks such as the following are useful:

What happens if you remove each instrument one at a time?

If excluding a single stock reverses the conclusion of the entire study, the apparent edge is much less stable than the aggregate result suggests.

We saw exactly this in one of our own tests

While researching simple patterns inspired by Tony Crabel's work, we tested NR4 among several fixed candidates.

NR4 describes a session whose range is narrower than each of the previous three sessions.

The first result looked interesting.

On the training period, NR4 was the best-performing candidate among the variants we tested.

Then we checked the holdout period.

The apparent edge did not confirm.

The result became even more interesting after improving the completeness and quality of the dataset.

The positive aggregate training result turned out to be heavily dependent on one stock — META.

Without META, the sign of the result changed.

That does not mean:

“NR4 does not work.”

But it certainly does not justify saying:

“NR4 works and can be traded profitably.”

The more defensible conclusion is much less exciting:

In this specific sample, we did not obtain sufficiently stable confirmation of the hypothesis.

That is exactly what research is for.

Not every analysis should end with a green equity curve.

What about AI?

There is another layer to the problem.

Models such as ChatGPT and Claude can be very good at:

• explaining results,
• identifying relationships,
• comparing experiments,
• discussing what to test next.

But they should not invent market data or backtest outputs that they do not actually have.

If you ask an AI model:

“Does NR7 work on SPY?”

it may offer sensible arguments, summarize known ideas, and suggest a testing framework.

But the answer to:

“What was the net result after costs on this exact dataset?”

should come from a deterministic research engine, not generated prose.

A useful division of labor looks like this:

user idea
   ↓
research engine
   ↓
data + backtest + metrics + provenance
   ↓
Claude / ChatGPT
   ↓
interpretation and follow-up questions

The engine computes. AI helps you understand.

That is safer than asking a language model to be the database, the backtester, and the analyst at the same time.

What should an honest backtest contain?

There is no single universal standard for every strategy, but before taking a result seriously, it is worth asking at least:

• Were the rules defined before seeing the final result?
• How many strategy variants were actually tested?
• Is there a separate out-of-sample period?
• Do signals use only information that was available at that time?
• Are commissions and slippage included?
• Does the result depend almost entirely on one instrument or one period?
• Is the assumed execution actually possible?
• Do we know the origin and quality of the data?
• Are negative results preserved, or only the best variant?
• Can the experiment be reproduced later?

If you cannot answer most of these questions, an attractive equity curve does not tell you very much.

A backtest should not confirm what you already believe

This may be the most important rule.

If you begin with:

“This strategy has to work”

it is very easy to find a way to make the backtest confirm it.

A slightly different parameter.

A shorter period.

Remove an inconvenient instrument.

Add one more filter.

Add one more condition.

Suddenly the result looks fantastic.

But you are no longer testing a hypothesis.

You are engineering the past.

A good research process must allow you to conclude:

“We did not find sufficient evidence.”

That is not failure.

If the research stops you from deploying a strategy built on a random historical pattern, it has already done its job.

Why we are building FactorSift

FactorSift is being built around this way of thinking.

Not as a generator of “guaranteed signals” and not as a bot that promises to make money automatically.

It is meant to be a workbench for testing market ideas.

The engine performs deterministic calculations, preserves the experiment context, and makes it possible to compare results.

Research Chat lets users request supported analyses in natural language.

And through API and MCP, structured results can be passed to Claude, ChatGPT, or another compatible AI client for further interpretation.

The goal is not to replace the investor.

The goal is to make it harder to fool yourself with a good-looking backtest.

Final thought

The most dangerous backtest is not the one that loses money.

It is the one that shows you exactly what you wanted to see.

So instead of asking:

“How much did this strategy make?”

start by asking:

“What would need to be true for this result to be credible?”

That is where research begins.