Source-aware field guide · 20 answers
Strategy Research and Testing
A research-method library for hypotheses, data splits, simulation assumptions and controls against misleading backtest results.
Published · reviewed for scope, source visibility and answer ownership
How to use this guide
A research-method library for hypotheses, data splits, simulation assumptions and controls against misleading backtest results. The guide is written for systematic traders, quant researchers, developers and investment-review teams. The collection explains how to make a strategy test inspectable. It does not validate a strategy or promise that a simulated result will persist.
Use every answer to resolve one operational question: Can the test be reproduced from frozen data and code, and which bias or implementation cost could overturn the conclusion? Historical simulation is sensitive to data, assumptions and researcher choices. Out-of-sample labels are meaningful only if the data stayed unseen.
The supporting set is MetaTrader 5 Help — Strategy testing; MetaTrader 5 — Automated trading; NIST CSRC Glossary — Reproducibility. These links provide standards, regulator material or official product documentation for the subject; they do not imply endorsement, worldwide applicability or a verified feature in a particular deployment. Check the current source, contract, configuration and qualified local advice before a production, trading or compliance decision.
How Do You Write a Trading Hypothesis?
A trading hypothesis is a falsifiable statement linking an observable condition to an expected outcome, population and horizon before results are examined.
- Use it to
- Define the signal, execution, benchmark, metric and rejection condition in advance.
- Check the boundary
- An explanation invented after viewing performance is not an independent hypothesis.
How Much Backtest Data Is Enough?
Backtest sample adequacy depends on independent observations, variability, effect size, regime coverage and the uncertainty acceptable for the decision.
- Use it to
- Report trade count, time span, clustering and confidence rather than one universal minimum.
- Check the boundary
- Many trades generated from the same market episode are not fully independent.
What Is In-Sample Testing?
In-sample testing uses data available for designing, fitting or selecting a strategy or its parameters.
- Use it to
- Tag every dataset touched during research and confine performance claims accordingly.
- Check the boundary
- Repeatedly consulting a holdout effectively moves it into the training process.
What Is Out-of-Sample Testing?
Out-of-sample testing evaluates a frozen strategy on data that was not used to design or select it.
- Use it to
- Lock code and parameters before running the holdout and preserve the run evidence.
- Check the boundary
- Choosing among many models after seeing holdout results contaminates the test.
What Is Walk-Forward Analysis?
Walk-forward analysis repeatedly fits or selects a model on an earlier window and evaluates it on the next unseen window under a fixed schedule.
- Use it to
- Define window lengths, parameter process, update timing and aggregation before execution.
- Check the boundary
- Overlapping tests and repeated design changes can still leak future information.
What Is Time-Series Cross-Validation?
Time-series cross-validation evaluates ordered train and validation windows while preserving chronology and, where needed, gaps between related observations.
- Use it to
- Use splits that reflect the live retraining and decision process.
- Check the boundary
- Random shuffling can leak later conditions into earlier training.
What Is Look-Ahead Bias?
Look-ahead bias occurs when a historical decision uses data, revisions or labels that would not have been available at that decision time.
- Use it to
- Timestamp each input by availability and test signals on the next executable event.
- Check the boundary
- Bar-close calculations, future pivots and revised datasets commonly create hidden leakage.
What Is Survivorship Bias?
Survivorship bias occurs when a historical universe excludes assets, entities or strategies that disappeared, failed or became unavailable.
- Use it to
- Use point-in-time membership and preserve delisted or inactive records.
- Check the boundary
- Testing only current constituents can omit many poor historical outcomes.
What Is Backtest Overfitting?
Backtest overfitting is adapting a strategy too closely to historical noise through excessive parameters, searches or discretionary choices.
- Use it to
- Track every experiment, reduce degrees of freedom and demand stable holdout behavior.
- Check the boundary
- One smooth equity curve can be the selected winner among many failed trials.
What Is Data Snooping Bias?
Data snooping bias arises when repeated tests on the same data increase the chance of finding an apparently strong result by luck.
- Use it to
- Count trials, preserve rejected variants and adjust the strength of evidence.
- Check the boundary
- Reporting only the final specification hides the size of the search.
How Should Slippage Be Modeled?
A slippage model estimates the difference between a strategy's reference price and plausible execution using order side, size, liquidity, delay and market state.
- Use it to
- Calibrate distributions from comparable fills and stress worse conditions.
- Check the boundary
- A fixed favorable assumption can turn a non-tradable idea into a profitable simulation.
How Should Commissions Be Modeled?
A commission model applies venue, broker, exchange, clearing or service charges according to the actual unit, tier and side of activity.
- Use it to
- Version fee schedules and calculate them at fill level where applicable.
- Check the boundary
- Ignoring minimums, rebates, currency conversion or volume tiers distorts net results.
How Should Spread Be Modeled?
A spread model reconstructs executable bid and ask prices rather than assuming every trade occurs at a mid or bar-close value.
- Use it to
- Use time-aligned two-sided data or a conservative state-dependent estimate.
- Check the boundary
- Average spread misses widening around news, rollover and illiquid periods.
How Should Latency Be Modeled?
A latency model delays decisions and orders through data, strategy, network, provider and venue stages before selecting an executable price.
- Use it to
- Measure each stage and replay orders against later market states.
- Check the boundary
- Using the signal timestamp as the fill timestamp creates impossible execution.
What Is Monte Carlo Analysis for Trading?
Monte Carlo analysis generates alternative modeled paths by resampling or simulating returns, trades or parameters under declared dependency assumptions.
- Use it to
- Inspect distributions of drawdown and outcome rather than one historical order.
- Check the boundary
- Independent reshuffling can destroy streaks, regimes and market-impact relationships.
What Is Parameter Sensitivity Testing?
Parameter sensitivity testing varies inputs around a selected setting to see whether conclusions depend on a narrow value.
- Use it to
- Publish a neighborhood grid and evaluate stability across metrics and periods.
- Check the boundary
- Selecting the single best parameter after the sensitivity test repeats the optimization problem.
What Is Regime Testing?
Regime testing compares strategy behavior across predeclared market states such as volatility, trend or liquidity conditions.
- Use it to
- Define states with information available at the time and report sample size per state.
- Check the boundary
- Regimes chosen after observing losses can become a storytelling device.
What Is Forward Testing?
Forward testing runs a frozen strategy on newly arriving data in real or simulated execution conditions without changing the evaluated rules.
- Use it to
- Log signals, orders, fills, outages and deviations from the specification.
- Check the boundary
- Changing parameters during the test requires a new version and evaluation period.
What Is Paper Trading?
Paper trading records simulated orders and positions without committing live capital, using a specified fill and market-data model.
- Use it to
- Test workflow, rules and integration while tracking differences from executable live conditions.
- Check the boundary
- Simulators may omit queue position, rejects, partial fills, impact and emotional pressure.
How Do You Make a Backtest Reproducible?
A reproducible backtest can be rerun from versioned code, parameters, point-in-time data and environment details to obtain the documented output.
- Use it to
- Archive hashes, dependencies, calendars, costs and random seeds with each result.
- Check the boundary
- Screenshots and aggregate metrics alone do not establish how the result was produced.
Primary and official references
These sources establish definitions, standards or official product behavior used across this guide. Follow the exact source and check its current version before a live implementation.