How do I know my backtesting is sound? How can I verify a strategy is ready for live execution? I employ several techniques to ensure my backtesting results are sound and represent1 what would have actually occurred. Strategies won't be run in live mode with real money until I can check these off. A quick list:
1) Live test and observe
This is more of a general catch all. But running the strategy in live mode will find bugs you won't otherwise find. And sometimes, those could affect backtesting. Any math errors that the server also calculates will show up here.
2) Use debug statements
Dump stack traces during each state change. Be careful of information overload, but the more you dump the more you can verify.
3) Check for logic errors
Make sure your not "cheating" with any of your math. And also verify your boolean and conditionals are executing as you think they should
4) Manually step through data and match output with strategy
This one requires you to trade your own strategy. step by step, on the test dataset. This can reveal coding errors, or strategy errors on your part.
5) Re-run data from live mode in backtesting and compare results
This has to be my favorite. I consider it the ultimate test so to speak, when the strategy will give the same results on the same data, whether run in live mode or backtesting mode. Simply collect data from live mode execution, and rerun in backtesting mode.
1. To the extent that is possible -- backtesting will never account for runtime differences such as slippage, spreads, etc.
No comments:
Post a Comment