New Trading Systems and Methods, Fourth Edition

CHAPTER 21

System Testing

Numbers are like people; torture them enough and they will tell you anything.

—Anonymous (quoted by John Ehlers in Rocket Science for Traders)

System testing has become a fight between computer power and human reason. Before 1990, testing a trading system was a simpler concept. Conscientious traders and analysts applied their ideas to charts, marking the buy and sell signals manually and keeping a hand-written record of the trades. The process took a long time; therefore, they were careful not to begin unless they were reasonably sure that the method had a good chance of success. Ideas came from an understanding of the fundamentals or the awareness of market patterns, in addition to experience. The tabulation of results was verification that the idea was sound.

By the early 1990s, you were at the high end of technology if you had daily data that could be read by a computer, and a program such as TradeStation or MetaStock to give you charts and indicators that could be combined into a trading system. Best of all, these programs showed the historic profitability of the trading strategies and, in particular, the best strategy. Moving averages and the RSI were still sophisticated tools, but the ability to test for the best calculation period and combine indicators began a step down a path that has now become very complex and taken the industry farther from the simple solution.

That process has continued to advance, but not always for the better. Those who ...