The test summary leaves out some important information, highlighted below. You may wish to examine these factors in greater detail.
One simple ratio is the recovery factor (RF). RF is absolute value of the ratio of net profit to maximum intraday drawdown, and it measures how far you recovered from the depth of the drawdown. In Table 3.13 the recovery factor is approximately equal to 5.6 (155,675/27,881). This number should be greater than 2, and the higher the better. It tells whether the potential benefits over the long haul are worth the aggravations caused by the drawdowns.
Another useful value is the adjusted gross profit, in which the largest winning trade is deducted from the gross profit. To penalize the system, do not correspondingly deduct the largest losing trade. The rationale here is not to expect to get the periods with large profits, but that a period of losses comparable to the worst losses in the test period is likely. The profit factor is then recalculated to see if it is still greater than 1. For the data in Table 3.13, the adjusted gross profit is $155,675 – 40,769 = $114,906. The adjusted profit factor is then 114,906/111,244 = 1.03. This is a sharp reduction from the reported profit factor of 2.40. Thus, a more realistic assessment of this system is that it will produce a small net profit over time.
The summary also does not give a histogram of your trades. You may wish to export your data to a spreadsheet to look for the maximum favorable excursion and maximum adverse excursion. These quantities will be explained in chapter 4 with the 65sma-3cc system.
The summary does not give you any feel for the variation in test results. It does not give a standard deviation of trade profits and losses for all trades. The variability is another important item you should cal-culate, using a spreadsheet if needed. The variation tells you what you can expect for volatility of returns.
You cannot get an idea of how a typical trade evolves in time from the test summary. For example, it does not tell you the average profit or maximum profit or loss on a day-in-trade basis. It does not show what happened on day 1 in the trade, or day 10 in the trade. A typical trade template, by Chande and Kroll, as discussed in The New Technical Trader (see bibliography), would help you understand the time-price evolution of a typical trade.
In addition, the test summary does not give a realistic impact of slippage. The software provides fills in a manner that may not be representative of fills in the real world. It is safer to assume that you will experience greater slippage than the model. In some instances, the software will give you a fill that you could not have obtained in practice. If this happened to be a big winner, you may overestimate trade profitability. Hence, you are better off using the average trade numbers to assess system performance, since they have averaged out the effects over many trades.
The performance summary also does not give any idea of how many successive .v-month periods would have been profitable. For example, it is useful to know how many successive 6-month periods have been profitable over a 5-year period. You could use any time interval you like. This breakdown tells you how quickly you can expect to get out of drawdowns, and is a vital piece of information for your mental approach to trading the system.
The most important factor to recognize is that the test summary does not tell you how the system will perform in the future. Your test results are hostage to your data. You should look below the surface of the results to get a better understanding of your system tests.
Ideally, you should examine the results on a trade-by-trade basis on the charts to understand how your system rules worked. This will reinforce your trading beliefs, and give you a good feel for when the system does or does not work. A study of unprofitable trades often reveals flaws in your logic. Convince yourself that you want to follow this system because its rules make money under market conditions that are likely to repeat in the future. A trade-by-trade review may also strengthen your ability to use discretion in trade entries or exits.
A Reality Check
This section sounds a note of warning before you proceed: Test results are not what they seem. You should recognize that trading systems are designed with the benefit of hindsight. This is true because you know, a priority, what the market has done in the past. Any trading system you design or optimize reflects your view of past market action. You may state your understanding in a generalized way that avoids the dangers of curve-fitting. However, it is worth recognizing that the influence of hindsight is difficult to eliminate.
It is also important to recognize that past price patterns may not repeat in precisely the same way. Hence, because the exact future sequence of trades is unpredictable, your system may not achieve profits or losses similar to the hypothetical system. It should be easy to conclude that past results are not indicative of future results because neither market action nor trader reaction is predictable.
There is another key problem area with simulated trades. Hypothetical trades from a trading system design exercise have not been entered in the markets and do not represent actual trading. They do not accurately reflect the effects of market liquidity, slippage, bad fills, overnight trading, or fast markets. They also do not reflect a trader’s psychology accurately since each and every signal is assumed to be executed with identical simplifying assumptions.
You, the trader, are perhaps the most capricious variable in the trading system. Because system testing is performed in an emotional vacuum, there is no assurance that you will execute all signals from a trading system without deviation. Thus, the biggest slippage could occur not in the markets, but at the source if you fail to enter orders as required.
As you will see in chapter 8 on data scrambling, it is possible to encounter market conditions that generate a long string of losing trades or one huge loss. Just because the probability that an event occurs is very small, this does not mean that it will not occur. The usual distribution of trades from a typical trading system has “fat” tails. This simply means that the probability that unusual market conditions will occur is much greater than you might expect from a normal distribution. Hence, system testing results will often underestimate market risks.
Thus, when you design trading systems, be aware that your hypothetical results do not accurately predict system performance in the fu-ture. In general, you should view any trading system results with all due caution.