The system evaluation methodology is broken up into three sections that include system setup, optimization, and evaluation. Below is the list of steps that should be taken when evaluating the potential and robustness of a trading strategy:
System Setup:
- Select a basket of at least five different contracts that have a low correlation to use as trading instruments. Each instrument should be very liquid, have a small spread, and have enough volatility to generate a profit regularly. It would be best if the contracts were from different sectors. Any basic strategy or indicator should work on this basket of products. Correlation testing will have to be done using Excel and can be performed on end of day data, even though we will be day trading.
- Obtain the tick level historical data for each contract going back to 2007. This will insure that we are covering all major market behaviors (range bound, bull market, and bear market).
- Determine the average volume for each contract over the past few months on a 10 minute time frame. Identify the times when the volume is at least 30 contracts a minute. Use this information to determine the trading times for each contract. Periodically re-evaluate the volume analysis.
- Determine your trading time frame by deciding how often you want to receive trading signals each day. If you want signals once a day, try starting with an hourly time frame. If you want signals once an hour, try starting with a 5 minute time frame. If you want signals once every 10 minutes, try starting with a tick chart. This will just be a starting point for the analysis of your strategy.
- Determine your basic entry methodology. Make sure that entries occur during times when the volume of the contract meets your trading requirements. Consider avoiding major economic announcements and entering right before the market closes.
- Determine your basic exit methodology. Will you be exiting the trade at the end of the day just before the market closes? Make sure that the exits occur during times when the volume of the contract meets your trading requirements. Consider exiting just before major economic announcements.
- Add a basic filter to help reduce the number of bad entry signals and whipsaws.
- Use the NinjaTrader Strategy Wizard to create the basic strategy. Create input parameters for the different aspects of the strategy that you would like to optimize such as indicator period, stop loss size, profit target size, etc. For a fully symmetrical system (always in the market long or short), do not plot your exit orders on the chart.
- Use the NinjaTrader Strategy Analyzer to backtest the data on a long duration time span (>250 trades). This time span is used extensively during research and optimization phases of strategy development. Pay close attention to the total net profit, percentage profitable, profit factor, Sharpe Ratio, average maximum excursion, and maximum drawdown. Use the chart tab to evaluate if the trade is entering and exiting orders correctly. Use the cumulative profit graph to evaluate the equity curve. Use the drawdown graph to evaluate the severity and duration of strategy drawdowns. Use the daily net profit graph to determine the consistency of daily returns. Use the distribution graph to evaluate the the profitability of the strategy by time of day or day of the week. Use the trades tab to export all the trade data for analysis in Excel.
System Optimization:
- Use the NinjaTrader Strategy Analyzer to optimize the strategy over the long duration backtest period. Optimization metrics should include maximum profit factor (preferred), maximum win/loss ratio, maximum average profit, minimum drawdown, and maximum net profit.
- Optimize strategy inputs until the parameter results are in the middle of the optimization range specified for each input. Perform an optimization of the strategy inputs for each instrument. Calculate the mean, median, and mode of each optimized input parameter for all instruments reducing the input parameter combinations down to a set of three. Once again, backtest the three sets (mean, median, mode) of input parameters on the entire basket of instruments to determine which set is best.
- Repeat previous optimization procedure for many different time frames including ticks, minutes, and hours.
- Perform a walk forward optimization beginning with the oldest data in the long duration backtest period and moving forward up to the beginning of the out of sample backtest period.
System Evaluation:
- Use the NinjaTrader Strategy Analyzer to backtest the data on specific market condition time frames such as range bound, bull, or bear markets. Use the date ranges in the Strategy Analyzer to specify important periods.
- Test the strategy using the simulated dynamic data feed to evaluate its performance on range bound markets, up trends, and downtrends.
- Use the NinjaTrader Strategy Analyzer to backtest the data on the out of sample period. This time span is used to validate if the strategy performs consistently over the backtest periods. The out of sample backtest is performed on the most recent historical data.
- Export trade backtest results for all instruments to Excel and perform analysis of equity curve to determine position sizing (% risk per trade), spread/slippage cushion, initial capital account size, and money management. Compare equity curve analysis results to the same equity curve analysis results for a buy-and-hold strategy on SPY (S&P500).
- Monte Carlo analysis (more research required)
- Bootstrap analysis (more research required)
- Use the NinjaTrader Market Replay to backtest the strategy on a dynamic data feed to help evaluate entries and exits.
- Simulate the strategy on a real-time live data feed using a demo account to understand if a strategy is tradeable, trade signals are generated correctly, and if slippage/commissions seem reasonable. Once you are consistently profitable for four weeks in a row with 4 out of 5 days showing a profit, and only hitting your max daily loss once a week, then you can begin thinking about trading the system live with real equity.
- Stochastic process control theory analysis to determine when to stop using the strategy (more research required)
- Trade system live with real money and monitor performance.
So I was able to add a lot more detail to the system evaluation procedure, but there is still more analysis that needs to be completed. Specifically, I have ordered some more books on walk-forward, Monte Carlo, and bootstrap analyses. The methodology will continue to constantly evolve, but I have a good foundation here to begin testing systems. It will take a lot of work/time, but this will ensure that I am confident in the final product and that I am doing everything I can to manage my risk.
