Sample Size: The Foundation of Reliable EA Evaluation

Discover why sample size is the critical foundation that determines whether your EA's performance metrics are trustworthy or misleading.

Statistical Reliability

Ensure metrics reflect true performance

Reduce Outliers

Smooth out random fluctuations

Detect Curve-Fitting

Identify artificially optimized systems

Minimum Recommendation

Sample Size ≥ 50–100 trades

For statistical significance and reduced outlier impact

Start Learning

12 min read

Beginner to Intermediate

9,800+ learners

Why Sample Size Is Essential for Forex EA Evaluation

When evaluating Forex Expert Advisors (EAs), traders often focus on metrics like profit factor or win rate, but these can be misleading without a sufficient sample size. Sample size—the number of trades used to assess performance—determines the statistical reliability of an EA's results. This lesson explains why sample size is the foundation of trustworthy EA evaluation.

What Is Sample Size?

Sample size refers to the number of trades an EA has executed in a given test or live trading period. A larger sample size provides more data points, increasing confidence that the EA's performance metrics reflect its true capabilities rather than random fluctuations.

Sample Size Consideration:

Minimum Recommended Sample Size = 50–100 trades

Why? To achieve statistical significance and reduce the impact of outliers in performance metrics like profit factor, win rate, or drawdown.

5 Reasons Sample Size Is Critical for EA Evaluation

1. Ensures Statistical Significance

Small sample sizes can produce skewed results, making an EA appear more profitable or reliable than it is. A larger sample size ensures that performance metrics are statistically significant and not due to luck.

Example: An EA with 10 trades and a 90% win rate may seem impressive, but with 100 trades, the win rate drops to 60%, revealing its true performance.

2. Reduces Impact of Outliers

A few large wins or losses can heavily distort metrics like profit factor or average return in small samples. A larger sample size smooths out these outliers, providing a more accurate picture of performance.

3. Tests Robustness Across Conditions

Forex markets are dynamic, with varying volatility and trends. A large sample size ensures an EA has been tested across different market conditions, revealing its adaptability and consistency.

Example: An EA with 20 trades in a trending market may show a high profit factor, but with 200 trades across ranging and trending markets, its performance may falter, exposing weaknesses.

4. Improves Metric Reliability

Metrics like drawdown, Sharpe ratio, or win rate require sufficient data to be meaningful. A small sample size can lead to over-optimistic or misleading conclusions, while a larger sample size validates these metrics.

5. Detects Curve-Fitting

EAs optimized for small datasets often fail in live trading due to curve-fitting. A large sample size, tested over diverse periods, helps identify whether an EA's performance is genuine or artificially tuned to historical data.

Determining Sufficient Sample Size for Your EA

Here's an MQL5 example to count trades and assess whether the sample size is sufficient:

MQL5 Example: CalculateSampleSize

int CalculateSampleSize()
{
   int totalTrades = OrdersHistoryTotal();
   int closedTrades = 0;
   
   // Count closed trades in history
   for(int i = 0; i < totalTrades; i++)
   {
      if(OrderSelect(i, SELECT_BY_POS, MODE_HISTORY))
      {
         if(OrderType() <= OP_SELL && OrderCloseTime() > 0)
         {
            closedTrades++;
         }
      }
   }
   
   return closedTrades;
}

void OnStart()
{
   // Calculate sample size
   int sampleSize = CalculateSampleSize();
   Print("EA Sample Size: ", sampleSize, " trades");
   
   // Interpret the sample size
   string interpretation = "";
   if(sampleSize < 50) interpretation = "Insufficient sample size";
   else if(sampleSize < 100) interpretation = "Marginally sufficient sample size";
   else if(sampleSize < 250) interpretation = "Good sample size";
   else interpretation = "Highly reliable sample size";
   
   Print("Sample size interpretation: ", interpretation);
}

Interpreting Sample Sizes

Sample Size Range	Interpretation	Recommendation
N < 50	Insufficient	Avoid; too unreliable
50 ≤ N < 100	Marginally sufficient	Use with caution
100 ≤ N < 250	Good	Consider for trading
N ≥ 250	Highly reliable	Strong candidate

For Forex EAs, aim for a sample size of at least 100 trades to ensure reasonable reliability. Sample sizes above 250 trades are ideal, providing high confidence in the EA's performance metrics.

Practical Application in EA Selection

When evaluating EAs, prioritize sample size with these steps:

Verify sample size - Ensure the EA has at least 100 trades, preferably more.
Test across periods - Collect trades from different market conditions (e.g., volatile vs. stable markets).
Compare sample sizes - Favor EAs with larger, more diverse trade histories.
Assess metric stability - Check if metrics like profit factor or win rate remain consistent as sample size grows.
Use alongside metrics - Combine sample size analysis with profit factor, drawdown, and other metrics for a complete evaluation.

Key Takeaways

Sample size determines the reliability of an EA's performance metrics
Aim for at least 100 trades, ideally 250 or more, for trustworthy results
Small sample sizes can lead to misleading conclusions due to outliers or luck
Large sample sizes reveal robustness across market conditions
Use sample size as the starting point before relying on any other performance metric.