HamScan
Market Scanning Utility
Backtesting and Optimization Guide

Contents:

Introduction

Backtesting provides a way for you to test your trade strategies on historical data. For example, if you have a strategy that enters a trade when the price crosses a 20-bar moving average, you can use Backtesting to test it. Backtesting will start as far back as you would like (10 years, for example) and test your trading strategy bar-by-bar. It will place trades when your strategy triggers an entry, and exit trades when your strategy triggers an exit. Backtesting does this on every bar of historical data, and then generates a report for you that tells you how profitable the strategy was, and also lists the exact trade entry and exit conditions (price and date). By reviewing the backtesting report, you can determine whether your strategy would have been profitable if you had been trading it for the last 10 years. This lets you adjust your strategies to find conditions that have worked well in the past. Of course, just because a strategy has worked in the past, does not mean that the strategy will continue to work.

Optimization takes backtesting further by performing a series of Backtests using different settings. After you have selected a strategy formula that you would like to use, you can use the Optimization feature to test the strategy with different parameters. For example, if your strategy is "enter a trade when the price crosses a Simple Moving Average", you might like to find out what SMA values work the best. Often, the 50-SMA daily is used as support or resistance for institutional investors. But you might want to determine whether the 20SMA or 30SMA or 40SMA or 50SMA gives you a better profit when backtesting. The Optimization will run a Backtest on each of these conditions, and then give you a summary report that you can use to evaluate which of them works best for your particular situation.


Backtesting

Determine your Strategy Entry and Exit goals

To set up a useful Backtest, you need to identify two things: Trade Entry conditions and Trade Exit conditions.

Trade Entry conditions include:

Trade Exit conditions include: Example (used below in the Wizard screen):

Create a new Backtest page

Using the above Example trade Entry and Exit conditions, here are the steps to have the Backtest Wizard in HamScan create the Backtest for you:

HamScan Backtest Wizard

Note that the upper half (for Long trades) and the lower half (for Short trades) are identical except for the "Trade Entry" labels for Stop and Limit. Here is a field-by-field description of all of the fields in the Backtest Wizard:

Changing your Entry and Exit Strategy settings

After creating a Backtest Page using the Backtest Wizard, you may want to make some changes to the settings. To do this, make sure that you have the BackTest Page selected in the Page drop-down, then select from the HamScan menu Backtest / Backtest Wizard. This will bring up the above Backtest Wizard window with your current settings. You can modify the settings, and click the Save and exit button to change them or click Cancel if you change your mind.

Selection of ticker(s) to backtest

Depending on the speed of your computer, the amount of memory available, the number of Optimization combinations, and the number of bars of history you select to test, a Backtest may take anywhere from a few seconds to several hours. Generally, if you are Backtesting a particular formula, you would like to have it work across multiple stocks or markets. If you test or Optimize a particular formula for a particular stock, you should be careful to not be overly optimistic of the results due to possible curve fitting. In testing hundreds of TradeStation Strategies over the last two years, I have found that careful selection of tickers for backtesting is essential. If you trade one particular sector, you might choose two or three of the most liquid (highest trading volume) stocks in that sector. Another good approach is to test the strategy on the sector index (SOX, BTK, etc.), or even on a broader index like the Dow 30, Nasdaq 100, or S&P 500.

In general, you should select a very small number of representative tickers for Backtesting when you first start testing a formula. Then as you improve the formula, you may want to drill down to more specific sectors or stocks that interest you.

You may use any of your TCNet/TC2000 watchlists as a starting point. From there, right click on tickers to "hide" them (Backtesting and Optimization only calculate and report on visible tickers in the watchlist). Narrow your list of tickers to one or a very small number. Note that you can also save the visible tickers of a watchlist as a new watchlist (right click again). The easiest way that I have found is to create a new small watchlist specifically designed for Backtesting. Also note that HamScan version 03.00.01 has a faster way of selecting watchlists than from the drop-down list. You may now select them from the hierarchical list from the HamScan menu Data / Choose WatchList.

Performing the Backtest

To start a Backtest after selecting a Backtesting Page and choosing a list of ticker symbols, select Backtest / Test Strategy from the HamScan menu. The Backtest Settings window will appear:
HamScan Backtest Settings
To do a Backtest, you may ignore the Optimization Variable Settings - they should all be left blank.

Determine how far back you would like to test. For example, if you would like to test the strategy over the past two years, set Number of bars back to test to 500 (at approximately 250 trading days per year). You may choose to use a shorter timeframe to test the strategy over a shorter term bear or bull market - or you may choose a much longer timeframe to test the strategy over a combination of markets.

Next, select whether you would like to test Long or Short trades - or both. You may select either Long or Short to test only trades in that direction. If you select both, then Long and Short trades are allowed to reverse each other; that is, a Short trade signal may cause an exit from a current Long trade (before any Stops or Profit exit is triggered).

Finally, you may change the name of the file to which the report will be written. If you are only viewing the report, you do not need to change this. If you are saving different reports for future examination, then change the Save results to file name (click Browse to change file names or folders). HamScan supports both HTML and Excel spreadsheet outputs. If you output to Excel, then each individual report is written to a separate WorkSheet, and each Optimization report also is placed on it's own WorkSheet.

Click the Start backtest button to have HamScan begin the Backtesting. The progress of the Backtest will be displayed in the Status Message box in the upper right corner of the main screen - and the Stop button will allow you to interrupt the Backtesting process in between individual tests (each ticker or optimization will complete, however, so the Stop signal may take several seconds before it halts the backtesting). When HamScan has finished the backtest, it will automatically display the web page containing the Backtest Report. This web page is saved in the file you selected with Save results to, so that you can go back later to examine it.

How to read the Backtest Report

There are three main sections to the Backtest Report: Trade Summary, Backtest Settings, and Trade Statistics. Examples of each of the three sections are shown below, with a descriptions of the contents.

Trade Summary

The ticker symbol and "Test #xx" are listed at the top of the Trade Summary. The "Test #" is a sequence number (1, 2, ...) when doing Backtest Optimizations. Following these is a table with each trade that was taken listed, one per line. The table contains a list of all of the trades that Backtesting took, earliest first.


Backtest Settings

This section lists the settings that were used to generate the list of Backtest in the Trade Summary. It contains several lines of settings, followed by a table that lists each of the Formula calculations used, as set up by the Backtest Wizard (or manually).

Summary:
Backtesting for 100 bars, calculations using an additional 169 bars
Trade date range: 2003/08/19 - 2004/01/12
Trade size: 100 shares

Optimization variables: X=0.00 Y=0.00 Z=0.00

Backtesting condition settings:
ConditionColumn labelFormula
Long SETUPL SetupC>Avg(C,20) AND C1<Avg(C,20,1)
Long ENTRY (STOP: at or above)L EntryC1+.1
Long Exit Above (Limit)L ExitAboveE*1.10
Long Exit Below (Stop)L ExitBelowMaxL(E*0.98,-E*(F>E*1.10))
Long Exit Time (#bars)L ExitTime10
Short SETUPS SetupC<Avg(C,20) AND C1>Avg(C,20,1)
Short ENTRY (LIMIT: at or above)S EntryC1+.1
Short Exit Above (Stop)S ExitAboveE*1.05
Short Exit Below (Limit)S ExitBelowE*0.90
Short Exit Time (#bars)S ExitTime10


Trade Statistics

The Trade Statistics section summarizes all of the trades for one complete Backtest of one ticker with one group of Optimization and Formula settings. The table is a sample Trade Statistics section. It is separated into three columns: Long Trades, Short Trades, and All Trades. The Trade Statistics displayed are:

Optimization

Adding variables to your formula for optimization

The HamScan Optimization feature allows you to test the same formula multiple times, with different parameter values, and then compare the results. To do this, replace the number or calculation in your Formula with one of the Optimization Variables (X, Y, or Z). For example, if you want to test the formula "C>Avg(C,20) AND C1<Avg(C,20,1)" (which looks for the price to cross up over the 20-period Simple Moving Average), but you would like to determine whether 20 is the best period setting to use, replace the "20" with "X": "C>Avg(C,X) AND C1<Avg(C,X,1)". In the next step, when you start the Backtest / Optimization, you will be given the opportunity to select the value(s) that X can take on.

Starting an Optimization for a series of Backtest strategies

After altering your Trade Setup formulas by adding Optimization Variables, you are ready to run an Optimization Backtest. To start an Optimization Backtest, select Backtest / Test Strategy from the HamScan menu. The Backtest Settings window will appear:
HamScan Backtest Settings
With the above example formula where the value "20" was replaced with "X" to test for the optimal value for an X-period Simple Moving Average, enter values for each variable used in the formulas. In this example, we would like to test the Simple Moving Average for periods of 10, 20, 30, 40, and 50. To do this, we set the value of "X" with a Start Value of 10 (the first test we want to try). We also set the End Value to 50 (the final test we want to perform). Finally, we set the Increment value to 10, which means that each successive test will add 10 to the previous value of X. So, X will start at 10, then be incremented to 20, 30, 40, and 50 - and then stop after the Backtest with a value of 50 is completed.

You can set ranges for each of three variables (X, Y, and Z). However, each Backtest may take one second to several minutes, depending on your settings, and how much historical data you have chosen to test. Therefore, the most efficient technique is to use larger increment values to narrow your Optimization down, and then retest with a smaller range. For example, if the 10 to 50 test (10 at a time) indicates that 30 looks promising, you may then want to run another Optimization from 25 to 35 with an increment of 2 (25, 27, 29, 31, 33, 35). Also, if you use more than one variable (X, Y, Z), multiply the number of choices for each variable to determine how many total tests there will be. If you vary X from 10 to 50 by 10's, (5 values total) *and* you vary Y from 1 to 8 by 1's (8 values total), HamScan Optimization will backtest 5 x 8 = 40 different combinations. HamScan has no problem doing extensive Backtests, but you should be aware that the larger number of combinations you select, the more time will be required to finish the Optimization report. Note that you may click the Stop button at any time during an Optimization, but the current ticker and X,Y,Z setting will be completed before the Optimization is stopped.

How to read the Optimization Report

The Optimization Report includes all of the Backtest reports for each of the X,Y,Z combinations. This allows examination of the detail of the conditions and trades for each of the Backtests. In addition, the Optimization Report includes a summary of all of the Backtest reports to help to quickly identify the most profitable variable combinations. If a particular combination works well, then you can narrow the search by selecting a smaller range (and smaller increment) for each variable. Finally, when the best combination has been identified, replace the X, Y, Z variables in the original formula with the values determined to give the best results. Below is an example Optimization Report which shows the settings and results for each Backtest, one per line:

HamScan Backtest Optimization Report for BIIB
XNumber of TradesNet Profit/LossPercent Profitable
10.0085-$367.1818.82%
20.0067-$222.6219.40%
30.0045-$939.3815.56%
40.0043-$857.0213.95%
50.0043$968.6227.91%

Curve Fitting

Strategy Backtesting and Optimization are very powerful features - allowing you to test multiple combinations of formulas and determine whether these formulas have performed well or poorly in the past (based on historical data). There are two major flaws in the whole idea of Backtesting and Optimization. They do not necessarily invalidate the results - but they must be taken into consideration when evaluating the results.

The first issue, as mentioned in the disclaimer of most trading strategies, is that "past results may not reflect future profitability" - that is, if a strategy has been successful in the past, there is no guarantee that it will continue to work. To help validate that a particular formula works (and to help strengthen the premise that it will continue to work), the best method generally is to broaden the scope of the test. If a particular strategy works on a single stock, there is not any reason to believe that it will work on another one. However, if a strategy works on a group of stocks, then you may feel more confidence that it will work for the stocks in that group. For example, if a strategy works on KLAC, it may or not work well on other Semiconductor stocks. However, if the strategy works for the SOX index, then you may be able to have more confidence using the strategy on individual stocks (particularly the larger component stocks of the index) in the Semiconductor Sector. If a strategy works on broader based indices (Dow Jones 30, Nasdaq 100, S&P 500, etc.), then you may have more confidence in using it on individual stocks. Of course, each stock should be backtested to verify that past historical data does not break your model. Again, Past performance does not indicate that the strategy will be guaranteed to work in the future - you're just looking for a good probability that the strategy generally reflects the market.

The second issue, which is often abused when doing Strategy Optimization, is called Curve Fitting. Curve fitting is a mathematical technique that allows you to take an arbitrary set of data and generate a formula that approximates that data. When doing strategy Optimization, it is important to *not* try to get too detailed in your formula description, because if you do, then you are probably identifying particular conditions in *past* historical data, and improving the formula for them - which does *not* help for any future usage of the calculation. Again, if you are able to define a more generic formula (one that works on multiple stocks, or broader markets or indices), then you are more likely to have a valid test. However, it you choose a Simple Moving Average period of 43, but testing of values 41, 42, 44, and 45 produce inferior results, then you should be very cautious about believing the results. In general, if a particular value (like SMA period) in your strategy seems to work well, then it's also likely that values near that will work reasonably - be VERY suspicious if a single value works but others near it do not.