The edge turned out to be dirty data. A painful lesson
My recent audit of gold trading data revealed a significant problem that invalidates a large portion of my previous backtesting results.
My recent audit of gold trading data revealed a significant problem that invalidates a large portion of my previous backtesting results. During a routine check of my gold datasets, I noticed extreme price behavior in the 2025 to 2026 period. The daily intraday range exceeded 5% on 12 separate days during just the first four months of 2026. For context, even during the 2020 COVID market crash, the frequency was only about 10 days per year. Furthermore, the price essentially doubled over a 14-month window. Because these intraday moves reached as high as 14%, I suspect the data for this period is corrupted rather than reflective of real market conditions. To see how my strategies perform under normal circumstances, I retested a standard Donchian channel strategy using a clean dataset spanning 2015 to 2024. The difference was stark.
| Metric | Full Period (with 2025-2026) | Clean Period (2015-2024) |
|---|---|---|
| Annual Return | 30% to 46% | 0.3% to 1% |
| PF | 2 to 3 | ~1 |
| Sample Size (n=20) | Positive | Negative |
| In other words, the high performance I previously observed was not a result of a robust edge, but rather a byproduct of the strategy riding a price anomaly in the corrupted data. My past backtests, including those involving parameter survivors, have likely been contaminated by this issue. | ||
This has become a major roadblock, and I have moved to address it by implementing a data cleaning layer. I have added btengine/dataquality.py to detect abnormal bars (such as price jumps or periods with zero volume) and set loader.load(clean=True) as the default for all future tests. | ||
| While this tool successfully catches individual outliers (which account for about 0.1% of historical data, mostly during the 2020 crash), the gold data issue is more insidious. Because the 2025-2026 gold price increase was a smooth, sustained move rather than a single jagged spike, it bypassed my initial detection filters. | ||
| This experience serves as a reminder that a system can fail not just through over-optimization, but through the quality of the data itself. From now on, verifying the health of the data is the non-negotiable first step of every verification process. |
Code to reproduce
You can reproduce this with the following scripts (see repo).
btengine/dataquality.pyscripts/scan_quality.py