The short version
- Claude can write the Python and run the math, but it cannot tell you whether your test design is honest. Most retail backtests fail at design, not at code.
- The reproducible workflow is: define one question first, fix the data window before you look at results, let Claude generate and run the code, then stress-test the answer against four well-known biases.
- A defensible backtest answers a single, pre-registered question. A backtest that "tries every weight" is almost always overfit, regardless of how clean the chart looks.
An LLM that can read pandas docs, write the code, run it in a sandbox, and explain the output is a remarkable shift for individual investors. It is also a fast way to produce convincing-looking nonsense. The interesting question is no longer "how do I run a backtest of my ETF allocation" — Claude will do that in two prompts. The question is whether the result tells you anything you should act on.
This piece is about the second question. It is the workflow the editor uses when evaluating allocation ideas before they get anywhere near a real account, and it is the same skeleton that lives inside, the Streamlit tool the blog runs alongside. The point is not to gatekeep — it is to share the parts that get skipped when a chatbot is the engine.
Why retail backtests usually mislead
The standard retail tool — Portfolio Visualizer, a spreadsheet, or now an LLM — makes it trivial to compute a CAGR, a maximum drawdown, and a Sharpe ratio over the longest data window the user can find. The numbers look authoritative. The problem is that the data window almost always covers a single macro regime, and the user has typically iterated on weights until the chart looked good.
Consider what a 10-year window of US ETF data covers as of 2026: a long disinflationary expansion, a pandemic shock with the fastest recovery in modern history, a 2022 rate-driven drawdown, and the AI-led concentration trade. That is one bull market with two short interruptions. There is no 2008-style credit event in the window. There is no sustained 1970s-style inflationary regime. Any allocation that did poorly in those environments looks fine here.
The 10-year Treasury at 4.39% and headline CPI at 3.3% (FRED, asof dates above) are themselves a reminder: the rate environment investors are sizing risk in today does not look much like the post-2009 decade. A backtest that does not flag this is hiding it.
The workflow Claude is actually useful inside
Treating the LLM as a code-writing collaborator inside a disciplined process is very different from treating it as an oracle. The structure below is what produces defensible output:
| Stage | What you decide (before any code) | What Claude does |
|---|---|---|
| 1. Pre-register the question | One specific hypothesis. "Does adding 10% AVUV to a 60/40 lift risk-adjusted return on a 15Y window?" | Nothing yet. The model has not seen any results. |
| 2. Fix the data spec | Window, frequency, dividend handling, rebalancing rule, transaction-cost assumption. | Generates the yfinance + pandas code, including total-return adjustment. |
| 3. Compute primary metrics | The metric list is fixed in step 1. No discovery now. | CAGR, volatility, max drawdown, drawdown duration, Sharpe, rolling correlations. |
| 4. Stress the result | Out-of-sample slice, alternative window, rebalancing-band sensitivity. | Re-runs the same metrics on the held-out data and reports differences. |
| 5. Document and stop | Whether the answer is "yes," "no," or "inconclusive given the window." | Generates a clean methodology paragraph you can paste into your notes. |
The discipline that matters is between steps 1 and 3: you do not change the metric list or the window after you have seen the answer. That single rule eliminates most accidental data-mining.
Where Claude is genuinely strong
Code generation against a well-specified prompt is the LLM's home turf. Claude reliably produces correct yfinance pulls, handles total-return vs price-return correctly when asked, computes Sharpe and Sortino with the right risk-free convention, and writes readable matplotlib output. It is also good at explaining what a number means in plain language — useful when the gap between "I see the Sharpe is 0.6" and "I understand what 0.6 implies about return per unit of volatility" is wider than retail content usually admits.
It is also good at the meta-step most retail users skip: critiquing its own output. A prompt of the form "list the three biggest reasons this backtest result might be misleading and check each one against the code you just wrote" produces a more honest summary than the chart on its own. Use it. The editor uses it on every run.
Overfitting is not a clever mistake. It is the default outcome of any process that lets you change weights after you see the chart.
Where Claude will not save you
The model has no opinion about whether your test was honest. If you ask it to find the allocation that maximized Sharpe over the past ten years, it will. The result is by construction overfit to that specific window. Claude does not flag this unless you ask it to — and even when asked, it will report the issue as one possibility among several rather than as the dominant problem.
It also does not know what data it does not have. Survivorship bias in mutual-fund backtests, look-ahead bias in factor backtests built from currently-known factor definitions, and the live-versus-backtest gap that hit nearly every published smart-beta product after 2010 — these are problems of the universe of available data, not problems Claude can detect from inside the code. The user has to bring them to the conversation.
Four biases worth holding in mind on every run. They are the same four institutional research desks check, and they are not exotic:
- Single-regime risk. A 10-year window in 2026 is roughly one regime. Re-run on a 20Y window, even if data quality is lower, and on regime-split slices (rate-rising vs rate-falling).
- Survivorship bias. Today's ETF universe excludes funds that closed. For broad-index ETFs this is small. For narrow thematic or factor ETFs it is meaningful.
- Look-ahead bias. Factor definitions and index methodology that exist today were not knowable in 2010. A backtest using today's index is not what an investor in 2010 could have bought.
- Iteration overfit. Every time you tweak weights and re-run, you spend a degree of freedom. After a dozen iterations the result is calibrated to noise. The defense is pre-registration: you fix the question and the metric in writing before the first run.
A prompt scaffold that produces honest output
The version below is what the editor actually uses. It is verbose on purpose — the constraints are the load-bearing part, not the analysis request.
"You are helping me run a single, pre-registered backtest. Constraints: do not propose alternative weights, do not optimize, do not suggest improvements until I ask. Use yfinance for total-return data (auto_adjust=True). Window: 2010-01-01 to 2024-12-31, in-sample; 2025-01-01 to 2026-04-30 held out. Portfolio: [my specific allocation]. Rebalancing: annual, no bands. Transaction costs: ignore. Compute: CAGR, annualized volatility, max drawdown, drawdown duration in months, Sharpe (rf = 3M T-bill from FRED), rolling 36-month correlation between equity and bond sleeves. Output a single table plus a price-path chart. After you show the result, list the three most likely ways this test could be misleading me, then stop."
The constraints in the first sentence matter more than the metric list. They prevent the model from drifting into "and here's a better allocation I found" — which is exactly the iteration trap.
Reading the output: which numbers carry weight
Once the run finishes, three numbers do most of the actual work, and they are not the ones retail summaries lead with.
Drawdown duration — the time from peak to recovery — matters more than max drawdown for behavioral risk. A 35% drawdown that recovers in 14 months tests different patience than a 25% drawdown that takes four years. Most backtest writeups quote the depth and skip the duration.
Rolling correlation between the sleeves of the portfolio reveals when diversification is real and when it has quietly disappeared. The 2022 episode, in which the equity-bond correlation flipped positive, is in every 10-year backtest now and is a genuine warning rather than a curiosity.
Performance gap between in-sample and held-out windows is the single best check on whether the design was honest. A small gap is reassurance; a large gap is the test telling you the in-sample result was largely noise.
The companion piece VOO vs. MTUM vs. QUAL walks through what these numbers look like for a real factor comparison. The framework that piece uses is exactly the workflow above.
Frequently asked questions
Do I need an Anthropic API key, or is the consumer chat product enough?
For a one-portfolio test, the consumer Claude product with a code-execution surface is sufficient. The API matters when you want to script the same backtest across many candidates programmatically, which is also when you most need to worry about iteration overfit.
Should I use Monte Carlo simulation on top of the historical backtest?
Cautiously. Monte Carlo on resampled historical returns inherits all the regime limitations of the underlying window and adds a layer of false precision. It is useful for sequence-of-returns risk in a withdrawal context. It is not a substitute for testing on a longer or different window.
Why not just use Portfolio Visualizer or a spreadsheet?
Both work. The reason to move to a Claude-plus-Python workflow is reproducibility — you have the actual code, you can re-run it next year on a longer window, and you can extend it (rolling beta, factor regression, regime-conditional metrics) without being limited to what the GUI exposes.
How long a window do I need?
Long enough to include at least one credit-driven equity drawdown and one inflation regime distinct from today's. For US data that means at least 25 years; even then the sample of "regimes" is small. For factor backtests that go back to 1927 (Ken French's data library), the lesson is humility, not confidence.
Is it worth backtesting at all if the limits are this severe?
Yes — but the goal is calibration, not validation. A backtest is most useful for understanding how an allocation behaves across the regimes that are in the data, and what its worst observed behavior looked like. It is least useful as proof that future returns will resemble past ones.
Key takeaways
- The bottleneck for retail backtesting is no longer code; it is the discipline of pre-registering the question before running it.
- Claude is a strong code collaborator and a competent self-critic when prompted, but it does not detect biases that live in the data universe rather than the code.
- Drawdown duration, rolling sleeve correlation, and the in-sample-versus-held-out gap deserve more weight than the headline CAGR and max drawdown most reports lead with.
- One regime in the window is the default, not the exception. Any conclusion you draw from a 10-year backtest in 2026 should be treated as conditional on a continuation of that regime.
Editor's read
Backtesting with Claude has changed what individual investors can do, in the same way Excel changed what accountants could do thirty years ago. The risk is the same too: lower friction makes it easier to produce results that look professional without being honest. If forced to give one rule, the editor's would be the pre-registration step in stage one of the workflow above. Everything else is plumbing; that one habit separates a test that informs a decision from a test that flatters one already made. The work also pairs naturally with the broader evidence-based portfolio framework the blog has been building.
The editor does not hold any single-name security discussed in this article and is the maintainer of the open-source tool referenced above.
Methodology
Macro figures cited (10-year Treasury, VIX, CPI year-over-year) are from the St. Louis Fed FRED database, retrieved 2026-05-05; asof dates are noted with each value. The workflow described is the editor's own practice and the design pattern used inside an in-house portfolio review project. No proprietary data is used. The article cites no individual ETF performance numbers because none were the subject of this piece; companion comparison articles on this site cite issuer fact sheets and yfinance directly.
This article is for educational purposes and does not constitute personalized financial advice. See full Disclaimer.