Stock Selection - Market Proxy Analysis

In my first couple of backtests, I used the iShares Dow Jones US Total Market Index (IYY) component stocks to represent the behavior of the entire market.  Backtest solve much faster on this list of stocks because there are only about 1300 compared to about 7000 for the entire US market.  The question is whether or not this proxy group of stocks accurately represents the behavior of the entire US stock market.  In this post I will be evaluating this claim.
I decided to compare the performance of the following three groups of stocks:

• All US Stocks - 6820 component stocks
• VTI - Vanguard Total Market Viper - 2405 component stocks
• IYY - iShares Dow Jones US Total Market Index - 1322 component stocks

My goal was to select either IYY or VTI as my market proxy for my backtests.  I evaluated the three groups using a percent breakout criteria, with no breakouts in the past two days before the breakout day.  The stock would be bought on the open the day after the breakout and sold at the close the following day.  I looked at using a percent threshold (ex. >5%) as well as independent breakout levels (ex. 5-6%).  All backtests were performed using StockFinder 4.0 and occur during the time period starting on September 15, 1998 and ending on September 15, 2009.

I will begin by comparing the results for the breakout threshold.  Below are the results for all US stocks (6820 component stocks):

Below are the results for VTI (2405 component stocks):

Below are the results for IYY (1322 component stocks):

At first glance the three results sets appear to be similar, with the annual return and the average trade becoming negative around the >8% or >9% threshold.  The maximum average trade is 0.14% for all stocks, 0.14% for VTI, and 0.13% for IYY.  As the breakout threshold increases, all three groups see a decrease in the percent profitable, an increase in the average winning trade, and a decrease in the average losing trade.

Although the data sets appear similar in many ways, there are many differences.  Below is a chart comparing the average return per trade for each of the groups of stocks over all breakout thresholds.

There are two interesting points about this chart.  First, the maximum average trade occurs at a similar threshold (>3%) and a similar magnitude (0.14%) for the All Stocks group and VTI.  IYY has a maximum average trade of 0.13% which occurs at a breakout threshold of >5%.  Second, the average trade becomes very negative for the All Stocks and VTI groups at breakout thresholds >9%, >10%, and >15%.  While IYY shows a similar trend of the average trade decreasing with increasing breakout threshold, IYY does not track as closely to the All Stocks group as VTI.

To dissect the data more closely, I looked at the average return for both the winning trades and the losing trades.  The charts below compare the average return for the winning and losing trades for each group of stocks over all breakout thresholds.

All three groups experience the same trends.  The average return for winning trades increases for increasing breakout thresholds.  The average return for losing trades decreases for increasing breakout thresholds.  In both cases, VTI more closely represents the All Stocks group than IYY.  Future studies should look at a way to improve the exit criteria and try to reduce the downside risk while maintaining the upside gains, especially for the larger breakout thresholds.

The three groups also differ with respect to the percent profitable results.  Below is a chart comparing the percent profitable for each group of stocks over all breakout thresholds.

As you can see from the chart, the three groups show a decrease in the percent profitable with and increase in the breakout threshold.  All three trends are quite similar in form, but the magnitudes differ.  VTI and IYY have a larger percent profitable than the All Stocks grouped across all breakout thresholds, but VTI is closer to the All Stocks group in magnitude than IYY.

To make sure the percent profitable statistics were reasonable, I also compared the winning percentage for the Monte Carlo Statistics.  The win percentage includes the 0%-return trades while the percent profitable only accounts for the winning trades.  Below is a chart comparing the Monte Carlo winning percentage for each group of stocks over all breakout thresholds.

Once the 0%-return trades are accounted for, the results for all three groups are more similar.  The decreasing trend is still intact, but now IYY tracks closer to the All Stocks group than VTI for the higher breakout thresholds.  For lower breakout thresholds, VTI is marginally better than IYY.

I also analyzed the results by independent breakout levels.  Below are the charts summarizing the results ofr the independent breakout levels for all three groups.

Once again I compared the average return per trade for all three groups.  Below is the chart showing the comparison over all breakout levels.

The maximum average trade occurs at the same level (6-7%) for the All Stocks and VTI group, while the maximum average trade for the IYY group is similar at the 7-8% level.  The maximum average trade is 0.20% for All Stocks, 0.22% for VTI, and 0.18% for IYY.  The IYY results underestimates the All Stocks results for levels between 2% and 6%, while the VTI results are similar.  Once again we see a large deviation of the IYY results from the All Stocks and VTI groups for large breakout levels.

I would say that the VTI group is a better proxy than the IYY group for All Stocks.  The VTI group still shows some small deviations for some of the metrics, but the trends of the group are similar in almost all cases indicating that the VTI group is still capturing the market's behavior.  In future backtest analyses I will use the VTI component stocks to evaluate results.

ANALYSIS SUMMARY:

• VTI is a better proxy than IYY to represent all US stocks
• The average return per trade for breakout thresholds is better for VTI
• The average return per winning trade for breakout thresholds is better for VTI
• The average return per losing trade for breakout thresholds is better for VTI
• The percent profitable for breakout thresholds is better for VTI
• The Monte Carlo winning percentage is better for IYY
• The average return per trade for breakout levels is better for VTI

UPDATE - 3/22/10
I have noticed that the components for the VTI Vanguard Total Market Vipers changes over time.  This could cause problems when if I compare the results of analyses I perform in the future with analyses already completed because the watchlist containing the component stocks could be different.  To resolve the issue, I created a separate backtest watchlist of all the component stocks for VTI on March 22, 2010.  As of this date the VTI contains 2401 stocks compared to the previous 2405 stocks.  I will be using the separate backtesting watchlist from now on so that the component stocks will not present an unwanted variable to the analyses.