Backtesters: The Good, the Bad, and the Money
You wake up one day and think to yourself, “Woah, I have money.” Maybe you got it from saving, maybe from the lottery, or maybe someone just handed it to you. Either way, you now have some spare cash most likely sitting in a bank account doing nothing. Probably getting eaten up by inflation.
You start to learn about how to invest your money to save for retirement, make a quick buck, or simply avoid the negative effects of inflation, and then “BOOM!” you come up with an idea for a investment strategy.
Then you think, “How can I test whether this strategy is any good?” Well unfortunately you can’t know for certain how well your strategy will do going forward without executing it, but you can certainly figure out how well it would have performed in the past, and that’s where a backtester comes in handy.
What Is a Backtester?
A backtester runs a trading strategy (e.g., using technical indicators) through historical market data to determine how well the strategy’s buying and selling of stocks, ETFs, or other assets would have performed in the past. This process gives us a viewpoint into the past and a better understanding of the effectiveness of our strategies.
While it’s true that past performance does not promise future returns, history often repeats itself and knowing the good and bad of any trading strategy requires an historical viewpoint. With that in mind, a backtester is an educational tool, not a crystal ball. While the results of a backtest might show that you’ve found something that’s going to make you rich, it’s likely that you’re guilty of overfitting (e.g., by tweaking and/or adding more parameters to a strategy to work perfectly over a specific time period).
But what makes a good backtester?
While a backtester demands attention to many subtle technical details, some important characteristics include:
- Accurately representing the past as it once occurred.
- Quickly iterating through large amounts of historical time series data.
- Running various types of trading strategies.
Being able to execute a trading strategy as it would have been executed in real life while using data that represents true past events is arguably the most important characteristic of any backtester. Without an accurate representation of past events, any further analysis performed is of little importance and will cause a misunderstanding of the strategy. Garbage in, garbage out.
Good backtesters contain multiple features that deliver realistic, event-driven results, including using clean market data from reliable sources, accounting for bid-ask spreads when buying and selling assets, and executing trades at the market open prices based on the preceding market close prices.
No one wants to wait hours, even minutes, for a backtest to run. Time is money. However, this can be difficult to avoid since computing potentially complex trading strategies against big data sets at scale requires a deep understanding of some of the most advanced technologies.
Good backtesters thrive in environments containing oceans of data and complex trading strategies. By leveraging key high-performance computing technologies, including cloud computing, vectorization, and highly optimized low-level code, trading strategies can be evaluated quickly at scale when using the right backtester.
There are literally an unlimited number of different trading strategies that exist in the world; from strategic asset allocation to technical indicator-based. Unfortunately, being forced to sit on either side of this spectrum can greatly reduce the number of possible trading strategies you can test. The freedom to choose is imperative.
Good backtesters deliver on flexibility by running strategies based on strategic asset allocation, technical indicators, and anything in-between. By removing restrictive barriers for those willing to more actively participate in today’s financial markets, the evolution of a more flexible backtester must emerge as a solution to democratize the process of evaluating various types of trading strategies.
It’s easy to abuse a backtester and fool yourself into thinking you’ll be the next Rockefeller. It’s important to realize the potential issues that could arise when using a backtester, especially overfitting.
Overfitting is arguably the biggest issue that comes up when performing a backtest mainly because you can trick yourself into thinking your strategy is an ATM machine but in reality it’s a money pit. No one wants to lose money, especially if it’s avoidable.
An example of overfitting could be as simple as someone following the process of:
- Running a backtest on some strategy only to see poor results.
- Adding more indicators and parameters to the existing strategy.
- Rerunning the backtester on the strategy to find excellent results.
- Executing the newly discovered strategy in real life and failing miserably
There are tools to help curb the potential of overfitting by testing for robustness (e.g., a strategy that works as designed in various market conditions), including scenario generators and walk forward analyzers.
After coming up with some sort of investment strategy, generating results from a good backtester, and feeling confident you’ve avoided overfitting, it’s now time to see if you can reach your financial goal through live trading or paper trading.
While backtesting results cannot guarantee future returns, the generated insights can assist us in making more confident financial decisions. And in a world of uncertainty, especially within the financial markets, we can use all the help we can get.