Does long term trading really work? I struggled for a long time during my first trading attempts wondering this question. It seemed to me longer term trading was inherently more risky than short term1. After all, I can't know what will happen 5 mins from now, let alone 5 days or weeks. However, stats geeks would tell me I can predict with greater certainty what will happen in 5 mins than 5 weeks2. And thus it seemed to me that short term was the way to go.
I had a good run for some time manual trading short term; however automated trading short term setups seemed to be an on-again, off-again sort of feeling. One week, I would win 90%. The next week, I could lose 90%. Plus, the concept behind why I would be entering positions in such a short term way was non-existent and flawed. I was simply guessing, hoping perhaps, and I had a risk/reward ratio that required a high win rate to succeed. Needless to say, the effect on my account was a slow drain, with nice profit spikes during those weeks where everything just clicked.
My current philosophy I suppose is that long term trading can and does work. However, I believe my initial analysis about higher risk was correct. If I trade long term with a poor risk reward ratio and with a short term viewpoint, then I have simply increased my risk. What I was missing initially was an entirely different way of trading. I suppose the "happy" way to think about longer term trading is that it gives you more possibilities of taking profit. In theory, if we can wait long enough, everything eventually comes around to profit. Or as price over time approaches infinity my win ratio becomes 100%3.
1. Without going crazy on definitions, I'm simply going to say "long-term" is anything more than a couple days
2. And they would be wrong. People's actions aren't a Gaussian distribution.
3. If your buying that idea, let me tell you about a system I would sell you. You see if you simply double up your bet each time, eventually you win. It's called a martingale, and ...
3.31.2010
Framework Optimizations Complete
The framework optimizations I mentioned in my previous post are now complete. Backtesting my current in progress strategy has been reduced from about 5 mins per month of simulation to ~30 seconds. Quite an increase. I think anything to increase speed from here on out will most likely be minor tweaks. At this point, I can increase the polling frequency if I chose and still keep the processing demand extremely low.
I would now in fact be focusing on testing and vetting the strategy for use, except for the fact my main machine is down for the count. This fact wouldn't be a huge issue if my SVN server didn't also happen to be running on it during this interim period. Right, I know what your thinking. Git is SO much cooler than SVN. I agree. However, there is little reason to migrate at this point, given the other pressing issues. After all, I don't have easy access to the code; let alone migrate it.
I would now in fact be focusing on testing and vetting the strategy for use, except for the fact my main machine is down for the count. This fact wouldn't be a huge issue if my SVN server didn't also happen to be running on it during this interim period. Right, I know what your thinking. Git is SO much cooler than SVN. I agree. However, there is little reason to migrate at this point, given the other pressing issues. After all, I don't have easy access to the code; let alone migrate it.
3.28.2010
Profit and the Mob
Profiting from a trade doesn't mean you are/were right; don't buy into this fallacy. It simply means others also traded the same way as you, during the same time you were vested. This brings up an odd perspective on the manual trader. There are traders who put in long1 hours manually trading the news, or fundamentals. Many of these folks are your classic stock market gurus who form an opinion on a company or sector as a whole and trade that opinion in those instruments. Yet it appears their ability to generate a profit from these activities is simply because they think the same way as others do; aka they are part of the mob.
In other words, if we view an instrument in the market at any time, we can think of the price as a representation of a random2 slice of people's opinions. The best traders are the ones who simply align themselves closest to the opinions of there peers within that instrument. Make sense?
Your paradox for the day - How can following the mob make you a profitable trader, when 95% of the mob loses money? It would seem the key to success in manual trading is to follow the mob, but be a mob leader.
1. If you trade the non-farm payrolls announcement exclusively, you only work 1 hour a month!
2. This is perhaps a topic for another post. However, market participants are not random. They certainly are not constant, but I think statistics are valid here for who is participating. For this reason, we can gain insight into who is making up our market, and according to my theory presented above, adopt their style of thinking for profit.
In other words, if we view an instrument in the market at any time, we can think of the price as a representation of a random2 slice of people's opinions. The best traders are the ones who simply align themselves closest to the opinions of there peers within that instrument. Make sense?
Your paradox for the day - How can following the mob make you a profitable trader, when 95% of the mob loses money? It would seem the key to success in manual trading is to follow the mob, but be a mob leader.
1. If you trade the non-farm payrolls announcement exclusively, you only work 1 hour a month!
2. This is perhaps a topic for another post. However, market participants are not random. They certainly are not constant, but I think statistics are valid here for who is participating. For this reason, we can gain insight into who is making up our market, and according to my theory presented above, adopt their style of thinking for profit.
3.26.2010
Strategy Verification
How do I know my backtesting is sound? How can I verify a strategy is ready for live execution? I employ several techniques to ensure my backtesting results are sound and represent1 what would have actually occurred. Strategies won't be run in live mode with real money until I can check these off. A quick list:
1) Live test and observe
This is more of a general catch all. But running the strategy in live mode will find bugs you won't otherwise find. And sometimes, those could affect backtesting. Any math errors that the server also calculates will show up here.
2) Use debug statements
Dump stack traces during each state change. Be careful of information overload, but the more you dump the more you can verify.
3) Check for logic errors
Make sure your not "cheating" with any of your math. And also verify your boolean and conditionals are executing as you think they should
4) Manually step through data and match output with strategy
This one requires you to trade your own strategy. step by step, on the test dataset. This can reveal coding errors, or strategy errors on your part.
5) Re-run data from live mode in backtesting and compare results
This has to be my favorite. I consider it the ultimate test so to speak, when the strategy will give the same results on the same data, whether run in live mode or backtesting mode. Simply collect data from live mode execution, and rerun in backtesting mode.
1. To the extent that is possible -- backtesting will never account for runtime differences such as slippage, spreads, etc.
1) Live test and observe
This is more of a general catch all. But running the strategy in live mode will find bugs you won't otherwise find. And sometimes, those could affect backtesting. Any math errors that the server also calculates will show up here.
2) Use debug statements
Dump stack traces during each state change. Be careful of information overload, but the more you dump the more you can verify.
3) Check for logic errors
Make sure your not "cheating" with any of your math. And also verify your boolean and conditionals are executing as you think they should
4) Manually step through data and match output with strategy
This one requires you to trade your own strategy. step by step, on the test dataset. This can reveal coding errors, or strategy errors on your part.
5) Re-run data from live mode in backtesting and compare results
This has to be my favorite. I consider it the ultimate test so to speak, when the strategy will give the same results on the same data, whether run in live mode or backtesting mode. Simply collect data from live mode execution, and rerun in backtesting mode.
1. To the extent that is possible -- backtesting will never account for runtime differences such as slippage, spreads, etc.
Of Execution
Since I touched very briefly on process separation in my last post, I thought I would quickly mention what actually is running and it's purpose on my auto-trading sever. I suppose setup will have to wait for another day. The first three I mention are always running, while the fourth is manually triggered by myself.
Pricefeed
This process grabs and stores realtime forex price data from Oanda. Currently, I don't have the disk space nor huge desire for long term storage, so I'm writing to csv files on a weekly basis for now. Given the data is available from Oanda with a 2 month rolling delay, there's little need for collection when dealing with the major pair, outside of verifying strategy execution.
Monitor
I monitor all strategies to ensure they are running, and have alerts based upon there actions. I will be alerted if a strategy dies, or if the PL encounters a drastic change, etc. These alerts are sent via email and SMS to my cell phone.
Strategy
Finally, each strategy itself will be running inside it's own process.
Analysis
This I will run from time to time to perform analysis on any strategy log; whether running in live mode or backtesting mode. Essentially it does things like produce statistics of trades, and graphs things like balance, PL, trade timespan, etc.
Pricefeed
This process grabs and stores realtime forex price data from Oanda. Currently, I don't have the disk space nor huge desire for long term storage, so I'm writing to csv files on a weekly basis for now. Given the data is available from Oanda with a 2 month rolling delay, there's little need for collection when dealing with the major pair, outside of verifying strategy execution.
Monitor
I monitor all strategies to ensure they are running, and have alerts based upon there actions. I will be alerted if a strategy dies, or if the PL encounters a drastic change, etc. These alerts are sent via email and SMS to my cell phone.
Strategy
Finally, each strategy itself will be running inside it's own process.
Analysis
This I will run from time to time to perform analysis on any strategy log; whether running in live mode or backtesting mode. Essentially it does things like produce statistics of trades, and graphs things like balance, PL, trade timespan, etc.
Framework Optimizations
During this downtime I'm working on coding the new strategies, and have found it beneficial and necessary in some cases to recode some of the old strategies for quicker backtesting. After all, when one can't be actually trading, what better to do than paper trade eh?
My typical strategy shell as it were has certainly evolved over the course of coding. It never seems to stay up to date, a I continue to add to the framework, and realign my thinking and strategy code. The latest set of optimizations are basic in thought, but slightly harder than first realized to implement. Up until this point laziness combined with several other factors has led to an abuse of running a calcPL() command1. Because of design overlap and poor coding, sometimes this was being calculated 2 or even 3 times per iteration. Obviously this is burning CPU cycles, but more importantly, it's taking valuable time away from strategy analysis and price feed updates. And of course, it's also why backtesting has slowed from "instantaneous" results to taking 10 minutes.
Despite knowing the issue, I decided to take the opportunity to profile my code for the first time. Profiling2 quickly pointed out the problem as well. Several hundred million calcPL() calls made for only a few hundred trades.
Rectifying this CPU waste has been a challenge. The ultimate goal is to completely remove any redundant calls to calcPL(), and instead allow the strategy to "know" when it needs to execute something. Obviously this means more upfront calculation and tracking, as well as storing the data. To the extent it makes sense, the objects have gained additional attributes that will store much of the data. Some of it will remain inside the client. Of course, this is only a first pass, and will likely see its own enhancements as time goes on.
The end result should create very fast iterations while running the strategy. Essentially the new processing order for each iteration will be as follows:
Get the exchange rate
Simple check to see if we need to act
Currently there is both a calcPL() layer and an analysis layer sandwiched in-between those steps. If your strategy can be re-written to fit the simple check model as above, even your smartphone could run it :-)
1. I also wanted to have realtime data of my PL. Instead of pursuing the proper solution, until now I've optimized the calcPL() call. To get this realtime PL data under the new model, there are several approaches one could take. Before coding much of the framework, I had a separate monitor process that also tracked PL. Separation is an easy and wise solution, as you can both monitor and perform additional analysis with it, and gain the separate process stability.
2. I used Devel::NYTProf and got a cool html graph output. CPAN makes things easy, so don't put off profiling for as long as I did.
My typical strategy shell as it were has certainly evolved over the course of coding. It never seems to stay up to date, a I continue to add to the framework, and realign my thinking and strategy code. The latest set of optimizations are basic in thought, but slightly harder than first realized to implement. Up until this point laziness combined with several other factors has led to an abuse of running a calcPL() command1. Because of design overlap and poor coding, sometimes this was being calculated 2 or even 3 times per iteration. Obviously this is burning CPU cycles, but more importantly, it's taking valuable time away from strategy analysis and price feed updates. And of course, it's also why backtesting has slowed from "instantaneous" results to taking 10 minutes.
Despite knowing the issue, I decided to take the opportunity to profile my code for the first time. Profiling2 quickly pointed out the problem as well. Several hundred million calcPL() calls made for only a few hundred trades.
Rectifying this CPU waste has been a challenge. The ultimate goal is to completely remove any redundant calls to calcPL(), and instead allow the strategy to "know" when it needs to execute something. Obviously this means more upfront calculation and tracking, as well as storing the data. To the extent it makes sense, the objects have gained additional attributes that will store much of the data. Some of it will remain inside the client. Of course, this is only a first pass, and will likely see its own enhancements as time goes on.
The end result should create very fast iterations while running the strategy. Essentially the new processing order for each iteration will be as follows:
Get the exchange rate
Simple check to see if we need to act
Currently there is both a calcPL() layer and an analysis layer sandwiched in-between those steps. If your strategy can be re-written to fit the simple check model as above, even your smartphone could run it :-)
1. I also wanted to have realtime data of my PL. Instead of pursuing the proper solution, until now I've optimized the calcPL() call. To get this realtime PL data under the new model, there are several approaches one could take. Before coding much of the framework, I had a separate monitor process that also tracked PL. Separation is an easy and wise solution, as you can both monitor and perform additional analysis with it, and gain the separate process stability.
2. I used Devel::NYTProf and got a cool html graph output. CPAN makes things easy, so don't put off profiling for as long as I did.
3.20.2010
Behavioral Analysis Trading
An idea has popped into my head regarding the other factor at work in a market. Until now, I had thought of price only as a function of time. I failed to really appreciate the human side of the equation. Ultimately human emotion drives the market. Right? After all, the decision to buy or sell is made by a human, and collectively those decisions become price movements. I've never taken the time to study human behavior in markets. Perhaps, just perhaps, there is an edge to be gained in doing so.
I can admit, I am not the typical consumer. I've realized I wouldn't have done well in the entrepreneurial world - merely because of my inability to understand people. Pick on anything you like; myself having been in IT for the dot com boom, I thought I might have a chance at starting something. Code it and they will come! However having seen sites like Myspace and Facebook do well, I realize how out of touch I am. Who would post all there personal details on some random companies site? And why communicate inside such a black box? Evidently there is a reason, since so many are doing it2.
Which brings us back to trading. Have you ever thought about why someone would take the flip side of your trade? How does TraderX1 make his trading decisions? Even a machine must be programmed. How does he trade? And does it matter to me? His decision, just like my own, moves the market! Given my choice to be proactive or reactive in my trading I have often sought to be reactive. Try and let the price set itself, and attempt to profit accordingly. I think being proactive is possible, and thinking about behavior could help that.
If your convinced at this point that this could play a role in your trading, I'll attempt to leave you with something. When this idea popped into my head, I immediately thought of Google Trends. It's sometimes shocking to see what the latest thing is on peoples minds (at least viewed thru google). The top searches and hot topics list are ever changing as the news disseminates what people should be thinking about today. Presumably this critical mass googles the sound bites, and thus you can "rank" the news on importance to the general populace. I would bet you the marketed media of the day is always in the top 5. Of course if you take that bet, I wouldn't have to worry about trading anymore. I'd be rich! Now, we simply need the same polling of currency traders.
1. Ok, I couldn't resist a Nick Leeson reference.
2. I suppose marketing to "con"-vince enough people into using the site until you reach a critical mass was the game plan here. Now people signup and use social networks simply because everyone else is. Amazing.
I can admit, I am not the typical consumer. I've realized I wouldn't have done well in the entrepreneurial world - merely because of my inability to understand people. Pick on anything you like; myself having been in IT for the dot com boom, I thought I might have a chance at starting something. Code it and they will come! However having seen sites like Myspace and Facebook do well, I realize how out of touch I am. Who would post all there personal details on some random companies site? And why communicate inside such a black box? Evidently there is a reason, since so many are doing it2.
Which brings us back to trading. Have you ever thought about why someone would take the flip side of your trade? How does TraderX1 make his trading decisions? Even a machine must be programmed. How does he trade? And does it matter to me? His decision, just like my own, moves the market! Given my choice to be proactive or reactive in my trading I have often sought to be reactive. Try and let the price set itself, and attempt to profit accordingly. I think being proactive is possible, and thinking about behavior could help that.
If your convinced at this point that this could play a role in your trading, I'll attempt to leave you with something. When this idea popped into my head, I immediately thought of Google Trends. It's sometimes shocking to see what the latest thing is on peoples minds (at least viewed thru google). The top searches and hot topics list are ever changing as the news disseminates what people should be thinking about today. Presumably this critical mass googles the sound bites, and thus you can "rank" the news on importance to the general populace. I would bet you the marketed media of the day is always in the top 5. Of course if you take that bet, I wouldn't have to worry about trading anymore. I'd be rich! Now, we simply need the same polling of currency traders.
1. Ok, I couldn't resist a Nick Leeson reference.
2. I suppose marketing to "con"-vince enough people into using the site until you reach a critical mass was the game plan here. Now people signup and use social networks simply because everyone else is. Amazing.
Voidness of space and time
This time spent inbetween my previous residence and my new office has to yield something right? Is there a grandiose strategy lurking inside my brain? I hope so. Meanwhile, I am stuck waiting on my stuff to arrive. I feel like a writer without paper. Since I am both a visual and straightforward thinker, I am almost lost without my workspace. I had assumed to be setup this weekend, but that reality has slipped away.
3.18.2010
In Transition
As I've been transitioning to a more climate friendly location, trading and in many ways life has come to a standstill. The prospects of gathering and packing everything you own is a healthy exercise. Its a time for simplification, and of purging. A good friend of mine once noted that if he was rich, he would seek to own nothing. The lesson learned is stark. If money is no object, given our services based economies, simply pay for everything. Why pack? Simply buy clothes there (or pay someone to). Why not rent different houses and move at whim? Without stuff, you are free. One can go too far here and quickly become a pure consumer, which runs contrary to our purpose in life. See Step 2.
All this to say it's been the better part of the month with no access to my code, systems, or trading in general. Losing your internet connection can be like dying. However, nothing is forever. And so, tomorrow my stuff should finally arrive. I should be back in business for next weeks session. Albeit, with no dedicated internet, running bots in live mode will have to remain postponed.
All this to say it's been the better part of the month with no access to my code, systems, or trading in general. Losing your internet connection can be like dying. However, nothing is forever. And so, tomorrow my stuff should finally arrive. I should be back in business for next weeks session. Albeit, with no dedicated internet, running bots in live mode will have to remain postponed.
3.05.2010
Of Capital
A couple of manual trades closed successfully shows me up a modest 4% this past month. Seems hard to justify the cash sitting in the bank with that kind of return.
At this point I suspect most readers are crunching the numbers and seeing a >50% annualized return as unobtainable, risky and foolish. However, I will kindly remind them (and myself!) of a few insights they may not realize.
In this case, it may very well be said investing such capital into any market, let alone forex is a huge risk, and is only attempted to the investors own peril. One does well to listen and think for themselves after receiving such a warning. The concept that money (in this case US dollars) sitting in a bank is safer than vested in the forex market. After all, banks have guarantees, known returns, etc. This is all well and good. Indeed, those telling me such things would never dream of investing in anything, barring perhaps the stock market 1. To anyone in this position, I would challenge that they are invested, and in my opinion, invested poorly in the forex market! They are (and presumably have been for some time) sitting on a giant dollar long position. To anyone holding this position (myself included) the long term returns have been quite negative. Your getting poorer, even if the digits in the account are higher than 10 years ago. Trading in the forex market merely allows you to diversify and choose where to put your chips. Having such a huge (albeit unleveraged bet) is not a wise investment choice at all times.
Now to the original claim that a double digit return is risky, unobtainable and foolish. I am certainly not so foolish as to count on nor expect that kind of return. However, the market has given, and will likely continue to do so 2, opportunities to achieve such returns. I can make informed investments, manage risk, and do everything "right" 3 and still lose money. Oddly enough, the converse is also true. I can do everything "wrong" and still make money.
1. The stock market shall have to be dealt with in another post. For now, suffice to say there is reason why I call it the magical money tree. Not matter what the brochure the salesman shows you says, we're not all going to be millionaires in 30 years.
2. As long as humans are the source of the market, volatility will know no bounds.
3. The definitions of right and wrong here are dubious and relative. However, the arguments hold no matter the readers definition of proper investing.
At this point I suspect most readers are crunching the numbers and seeing a >50% annualized return as unobtainable, risky and foolish. However, I will kindly remind them (and myself!) of a few insights they may not realize.
In this case, it may very well be said investing such capital into any market, let alone forex is a huge risk, and is only attempted to the investors own peril. One does well to listen and think for themselves after receiving such a warning. The concept that money (in this case US dollars) sitting in a bank is safer than vested in the forex market. After all, banks have guarantees, known returns, etc. This is all well and good. Indeed, those telling me such things would never dream of investing in anything, barring perhaps the stock market 1. To anyone in this position, I would challenge that they are invested, and in my opinion, invested poorly in the forex market! They are (and presumably have been for some time) sitting on a giant dollar long position. To anyone holding this position (myself included) the long term returns have been quite negative. Your getting poorer, even if the digits in the account are higher than 10 years ago. Trading in the forex market merely allows you to diversify and choose where to put your chips. Having such a huge (albeit unleveraged bet) is not a wise investment choice at all times.
Now to the original claim that a double digit return is risky, unobtainable and foolish. I am certainly not so foolish as to count on nor expect that kind of return. However, the market has given, and will likely continue to do so 2, opportunities to achieve such returns. I can make informed investments, manage risk, and do everything "right" 3 and still lose money. Oddly enough, the converse is also true. I can do everything "wrong" and still make money.
1. The stock market shall have to be dealt with in another post. For now, suffice to say there is reason why I call it the magical money tree. Not matter what the brochure the salesman shows you says, we're not all going to be millionaires in 30 years.
2. As long as humans are the source of the market, volatility will know no bounds.
3. The definitions of right and wrong here are dubious and relative. However, the arguments hold no matter the readers definition of proper investing.
Subscribe to:
Posts (Atom)