Backtesting Analysis Skill

Purpose: Interpret backtest results, understand performance metrics, and detect overfitting or unreliable strategies.

Progressive Disclosure: This primer contains essentials only. Full details available via docs command.

---

When to Use This Skill

Load when:

• Evaluating backtest results (Phase 3)
• Detecting potential overfitting
• Understanding strategy-specific performance expectations
• Comparing multiple strategies or explaining results

---

Quick Reference: Key Metrics

Sharpe Ratio (Primary Metric)

Formula: (Return - Risk-Free Rate) / Volatility

Sharpe	Quality	Action
< 0.5	Poor	Abandon
0.5 - 0.7	Marginal	Consider optimization
0.7 - 1.0	Acceptable	Optimize
1.0 - 1.5	Good	Production-ready
1.5 - 2.0	Very Good	Validate thoroughly
> 3.0	SUSPICIOUS	Likely overfitting

Key Insight: QuantConnect reports annual Sharpe. Sharpe > 1.0 is production-ready for most strategies.

---

Maximum Drawdown

Formula: (Trough - Peak) / Peak

Drawdown	Quality	Action
< 20%	Excellent	Low risk
20% - 30%	Good	Acceptable for live trading
30% - 40%	Concerning	Needs strong Sharpe to justify
> 40%	Too High	Unacceptable for most traders

Key Insight: Drawdowns > 30% are hard to tolerate psychologically. Consider: "Could I stomach this loss in real money?"

---

Total Trades (Statistical Significance)

Trade Count	Reliability	Decision Impact
< 20	Unreliable	Abandon or escalate
20 - 30	Low	Minimum viable
30 - 50	Moderate	Acceptable
50 - 100	Good	Strong confidence
100+	Excellent	Highly reliable

Key Insight: Need 30+ trades for basic significance, 100+ for high confidence. Few trades = unreliable metrics.

---

Win Rate

Win Rate	Quality	Interpretation
< 40%	Low	Needs large winners (trend following)
40% - 55%	Average	Typical for most strategies
55% - 65%	Good	Strong edge
> 75%	SUSPICIOUS	Likely overfitting

Key Insight: Win rate alone is misleading. Must consider profit factor and average win/loss ratio.

---

Profit Factor

Formula: Gross Profit / Gross Loss

Profit Factor	Quality	Interpretation
< 1.3	Marginal	Transaction costs may kill it
1.3 - 1.5	Acceptable	Decent after costs
1.5 - 2.0	Good	Strong profitability
> 3.0	Exceptional	Outstanding (verify no overfitting)

Key Insight: Minimum 1.5 for live trading to cover slippage and commissions.

---

Overfitting Detection (Red Flags)

1. Too Perfect Sharpe (> 3.0) → ESCALATE_TO_HUMAN
2. Too High Win Rate (> 75%) → Check for look-ahead bias
3. Too Few Trades (< 20) → Unreliable metrics
4. Excessive Optimization Improvement (> 30%) → Lucky parameters
5. Severe Out-of-Sample Degradation (> 40%) → ABANDON_HYPOTHESIS
6. Equity Curve Too Smooth → Check unrealistic assumptions
7. Works Only in One Market Regime → Not robust

Remember: If it looks too good to be true, it probably is.

---

Strategy-Type Expectations

Momentum

• Sharpe: 0.8 - 1.5 | Drawdown: 20-35% | Win Rate: 40-55%

Mean Reversion

• Sharpe: 0.7 - 1.3 | Drawdown: 15-30% | Win Rate: 55-70%

Trend Following

• Sharpe: 0.5 - 1.0 | Drawdown: 25-40% | Win Rate: 30-50%

Breakout

• Sharpe: 0.6 - 1.2 | Drawdown: 20-35% | Win Rate: 40-55%

Use these to calibrate expectations - different strategies have different profiles.

---

Example Decisions

GOOD (Optimization Worthy)

Sharpe: 0.85, Drawdown: 22%, Trades: 67, Win Rate: 42%, PF: 1.8
→ PROCEED_TO_OPTIMIZATION (decent baseline, worth improving)

EXCELLENT (Production Ready)

Sharpe: 1.35, Drawdown: 18%, Trades: 142, Win Rate: 53%, PF: 2.1
→ PROCEED_TO_VALIDATION (already strong, skip optimization)

SUSPICIOUS (Overfitting)

Sharpe: 4.2, Drawdown: 5%, Trades: 25, Win Rate: 88%, PF: 5.8
→ ESCALATE_TO_HUMAN (too perfect, likely look-ahead bias or bug)

POOR (Abandon)

Sharpe: 0.3, Drawdown: 38%, Trades: 89, Win Rate: 35%, PF: 1.1
→ ABANDON_HYPOTHESIS (poor risk-adjusted returns)

---

Common Confusion Points

Q: "Strategy made 200% returns, but Sharpe is only 0.6 - is this good?" A: No. We prioritize risk-adjusted returns (Sharpe), not raw returns. High returns with high volatility = bad Sharpe = risky.

Q: "Sharpe 2.5 with 15 trades - should I proceed?" A: ESCALATE_TO_HUMAN. Too few trades (<20) for statistical significance. High Sharpe with few trades = luck, not skill.

Q: "Optimization improved Sharpe from 0.8 to 1.5 (87% improvement) - is this good?" A: ESCALATE_TO_HUMAN. 87% > 30% threshold = likely overfitting to in-sample period.

Q: "Win rate is 78%, Sharpe is 1.2 - why is this flagged?" A: Win rate > 75% is an overfitting signal. Real strategies rarely achieve such high win rates.

---

Key Principles

1. Sharpe ratio is king - Primary metric for risk-adjusted returns
2. Trade count matters - Need 30+ for reliability, 100+ for confidence
3. Beware overfitting - Too perfect results are suspicious
4. Context by strategy type - Different strategies have different expectations
5. Risk-adjusted, not raw returns - High returns with high volatility = bad

---

Reference Documentation (Progressive Disclosure)

Need detailed analysis? All reference documentation accessible via --help:

python SCRIPTS/backtesting_analysis.py --help

That's the only way to access complete reference documentation.

Topics covered in --help:

• Sharpe Ratio Deep Dive
• Maximum Drawdown Analysis
• Trade Count Statistical Significance
• Win Rate Analysis
• Profit Factor Analysis
• Complete Overfitting Detection Guide
• Strategy-Type Profiles (Momentum, Mean Reversion, Trend Following, Breakout)
• 10+ Annotated Example Backtests
• Common Confusion Points

The primer above covers 90% of use cases. Use --help for edge cases and detailed analysis.

---

Integration with Decision Framework

This skill complements the decision-framework skill:

• decision-framework: Provides thresholds and decision logic
• backtesting-analysis: Provides interpretation and context

Workflow:

1. Load decision-framework to apply thresholds
2. Load backtesting-analysis to understand what metrics mean
3. Combine insights to make informed decisions

---

Related Files

• .claude/skills/decision-framework/skill.md - Decision thresholds
• SCRIPTS/decision_logic.py - Decision implementation
• .claude/commands/qc-backtest.md - Backtest execution

---

Version: 2.0.0 (Progressive Disclosure) Last Updated: November 13, 2025 Lines: ~200 (was 555) Context Reduction: 64%