How do I know my backtesting is sound? How can I verify a strategy is ready for live execution? I employ several techniques to ensure my backtesting results are sound and represent1 what would have actually occurred. Strategies won't be run in live mode with real money until I can check these off. A quick list:
1) Live test and observe
This is more of a general catch all. But running the strategy in live mode will find bugs you won't otherwise find. And sometimes, those could affect backtesting. Any math errors that the server also calculates will show up here.
2) Use debug statements
Dump stack traces during each state change. Be careful of information overload, but the more you dump the more you can verify.
3) Check for logic errors
Make sure your not "cheating" with any of your math. And also verify your boolean and conditionals are executing as you think they should
4) Manually step through data and match output with strategy
This one requires you to trade your own strategy. step by step, on the test dataset. This can reveal coding errors, or strategy errors on your part.
5) Re-run data from live mode in backtesting and compare results
This has to be my favorite. I consider it the ultimate test so to speak, when the strategy will give the same results on the same data, whether run in live mode or backtesting mode. Simply collect data from live mode execution, and rerun in backtesting mode.
1. To the extent that is possible -- backtesting will never account for runtime differences such as slippage, spreads, etc.
Showing posts with label backtesting. Show all posts
Showing posts with label backtesting. Show all posts
3.26.2010
Framework Optimizations
During this downtime I'm working on coding the new strategies, and have found it beneficial and necessary in some cases to recode some of the old strategies for quicker backtesting. After all, when one can't be actually trading, what better to do than paper trade eh?
My typical strategy shell as it were has certainly evolved over the course of coding. It never seems to stay up to date, a I continue to add to the framework, and realign my thinking and strategy code. The latest set of optimizations are basic in thought, but slightly harder than first realized to implement. Up until this point laziness combined with several other factors has led to an abuse of running a calcPL() command1. Because of design overlap and poor coding, sometimes this was being calculated 2 or even 3 times per iteration. Obviously this is burning CPU cycles, but more importantly, it's taking valuable time away from strategy analysis and price feed updates. And of course, it's also why backtesting has slowed from "instantaneous" results to taking 10 minutes.
Despite knowing the issue, I decided to take the opportunity to profile my code for the first time. Profiling2 quickly pointed out the problem as well. Several hundred million calcPL() calls made for only a few hundred trades.
Rectifying this CPU waste has been a challenge. The ultimate goal is to completely remove any redundant calls to calcPL(), and instead allow the strategy to "know" when it needs to execute something. Obviously this means more upfront calculation and tracking, as well as storing the data. To the extent it makes sense, the objects have gained additional attributes that will store much of the data. Some of it will remain inside the client. Of course, this is only a first pass, and will likely see its own enhancements as time goes on.
The end result should create very fast iterations while running the strategy. Essentially the new processing order for each iteration will be as follows:
Get the exchange rate
Simple check to see if we need to act
Currently there is both a calcPL() layer and an analysis layer sandwiched in-between those steps. If your strategy can be re-written to fit the simple check model as above, even your smartphone could run it :-)
1. I also wanted to have realtime data of my PL. Instead of pursuing the proper solution, until now I've optimized the calcPL() call. To get this realtime PL data under the new model, there are several approaches one could take. Before coding much of the framework, I had a separate monitor process that also tracked PL. Separation is an easy and wise solution, as you can both monitor and perform additional analysis with it, and gain the separate process stability.
2. I used Devel::NYTProf and got a cool html graph output. CPAN makes things easy, so don't put off profiling for as long as I did.
My typical strategy shell as it were has certainly evolved over the course of coding. It never seems to stay up to date, a I continue to add to the framework, and realign my thinking and strategy code. The latest set of optimizations are basic in thought, but slightly harder than first realized to implement. Up until this point laziness combined with several other factors has led to an abuse of running a calcPL() command1. Because of design overlap and poor coding, sometimes this was being calculated 2 or even 3 times per iteration. Obviously this is burning CPU cycles, but more importantly, it's taking valuable time away from strategy analysis and price feed updates. And of course, it's also why backtesting has slowed from "instantaneous" results to taking 10 minutes.
Despite knowing the issue, I decided to take the opportunity to profile my code for the first time. Profiling2 quickly pointed out the problem as well. Several hundred million calcPL() calls made for only a few hundred trades.
Rectifying this CPU waste has been a challenge. The ultimate goal is to completely remove any redundant calls to calcPL(), and instead allow the strategy to "know" when it needs to execute something. Obviously this means more upfront calculation and tracking, as well as storing the data. To the extent it makes sense, the objects have gained additional attributes that will store much of the data. Some of it will remain inside the client. Of course, this is only a first pass, and will likely see its own enhancements as time goes on.
The end result should create very fast iterations while running the strategy. Essentially the new processing order for each iteration will be as follows:
Get the exchange rate
Simple check to see if we need to act
Currently there is both a calcPL() layer and an analysis layer sandwiched in-between those steps. If your strategy can be re-written to fit the simple check model as above, even your smartphone could run it :-)
1. I also wanted to have realtime data of my PL. Instead of pursuing the proper solution, until now I've optimized the calcPL() call. To get this realtime PL data under the new model, there are several approaches one could take. Before coding much of the framework, I had a separate monitor process that also tracked PL. Separation is an easy and wise solution, as you can both monitor and perform additional analysis with it, and gain the separate process stability.
2. I used Devel::NYTProf and got a cool html graph output. CPAN makes things easy, so don't put off profiling for as long as I did.
2.25.2010
Of Backtesting
The joys of backtesting. In my last post I described the framework I am utilizing for autotrading. Once the strategy is coded inside the framework, I can choose to run it in live mode, or I can choose to run it thru previous price quotes in order to get a historical perspective on it's performance. This is helpful to reveal bugs more than anything else. People will often "tweak" there strategies in order to gain good backwards testing performance. Unfortunately, this is also how many are scammed with the inevitable forex bots, and pretty graphs showing beautiful equity curves. Suffice to say, tweaking for the past is an exercise in futility, and borders on dangerous. While it is enjoyable to see a strategy perform on an unknown dataset, I have successfully run robots that don't backtest well. The reason of course is that the past is different from the future; which is of course different from today. The only thing that matters is how the robot is performing now, in realtime.
That's not to say backtesting has no place. As I said, it helps to reveal bugs, and can identify or disprove your thesis on the core of the strategy. For example, if I code a trend following system, I would want to backtest it on a dataset containing trends. This will help me to see if my idea on how to idenitfy a trend is correct (and correctly coded). The performance of the robot on this dataset is a secondary indicator, but again is most useful for determining proper code. If a robot wipes the account of otherwise experiences severe drawdown my strategy doesn't call for, it usually means there is a bug somewhere. Typically you can no more always be wrong, than always be right. Either one is a cause for investigation. Or you found the grail. Heh. My money is on a bug!
That's not to say backtesting has no place. As I said, it helps to reveal bugs, and can identify or disprove your thesis on the core of the strategy. For example, if I code a trend following system, I would want to backtest it on a dataset containing trends. This will help me to see if my idea on how to idenitfy a trend is correct (and correctly coded). The performance of the robot on this dataset is a secondary indicator, but again is most useful for determining proper code. If a robot wipes the account of otherwise experiences severe drawdown my strategy doesn't call for, it usually means there is a bug somewhere. Typically you can no more always be wrong, than always be right. Either one is a cause for investigation. Or you found the grail. Heh. My money is on a bug!
Subscribe to:
Posts (Atom)