The challenge of backtesting and systematic trading and how to get past it

I’m going to make this observation a challenge of my brevity because this is a topic I could write a book on if I sat still long enough. So, this is by no means going to be an exhaustive observation on the topic of trading system design, testing, and operation.

There are two kinds of traders; discretionary and systematic. I’m using the word “trading” because it means both buying and selling. To me, an investor is someone who buys and may never sell. For example, our clients are investors because they invest in our investment program and I do the buying and selling (trading.)

Discretionary trading relies on the skill, knowledge, experience, control, emotional intelligence, and discipline of the portfolio manager. If we make a decision as a human instead of operating a computer program that systematizes the decision, it’s discretionary.

Systematic trading is operating methodically according to a fixed plan (algorithm) that is designed to achieve a profit by entering and exiting markets trading on an exchange. Systematic is rules-based, so it’s a “system” for decision making. As such, the portfolio manager doesn’t look at a price trend chart to figure out what to do next but instead runs the computer program to get the buy or sell signal.

Automated, by the way, is an even more systematic program where the program sends the signal to the trade desk or broker for execution automatically without any intervention or supervision from the portfolio manager.  So, automated is a more advanced operation of systematic.

Discretionary can be rules-based and systematic, but for this purpose, I draw the binary distinction between the two. A chartist is an example of discretionary and a systematic trader runs a computer program to get the buy and sell signals to execute without interference.

By nature, mathematical, quantitative, systematic, and automatic methods to trading advance themselves to computerized testing. When done correctly, backtesting can add enormous value to an overall trading strategy. A property tested quantitative system can validate a strategy to determine its probability for generating asymmetric risk/reward or even a profit. So, the scientific method of testing makes the system verifiable; would it have worked in the past, or not? I could go on, but I’ll leave it there. The bottom line is, the advantage of applying a scientific systems testing process is to verify what it would have done had we been operating it in the past.

Of course, this is necessarily always done with perfect hindsight! So, we must necessarily be realistic with ourselves. For example, I can create a custom investment program for investment advisors and high net worth families to match their risk/reward objectives. Below is an example of a quantitative trading system applied to a fixed list of securities for an investor whose objective is absolute returns. The blue line is the hypothetical system as tested over the period 2000 – 2007. It may not seem impressive by the minimal profit over the period. but it’s better than the orange line and didn’t decline nearly as much as a stock index from 2000 – 2003. At the time, an absolute return investor would have appreciated its risk management benefit.

asymmetric trading system to 2007

Next is the return stream through the end of 2009. The system’s tactical risk management methods worked by reducing the drawdown in comparison to the orange line, an alternative I’m comparing to just for illustration. Absolute return strategy isn’t relative return, so they have no benchmark to play the horse race with.

asymmetric trading system through bear market

Once again, over this period of nine years, I’m guessing some may be thinking; my objective is to earn a profit and this isn’t’ much profit at all!

Well, sometimes we do need to consider that everything is relative. To see what I mean, the next chart of the returns streams I made the orange line a U.S. stock index. You can see the drawdowns are much more for the stock index.

relative return strategy

To be sure, here is the same period, but a drawdown chart. The stock index declined twice -50% or more. The blue line, which is the system, had drawdowns of around -15% or less.

asymmetric trading drawdown

So, with the stock index, most investors probably tap out near the lows after they are down -40% or -50% and are afraid to ‘get back in’ until well after prices have trended back up.

I don’t know how to handle such a tap out situation. There is no right answer to deal with it; when do you ever get back in?

Do you feel better prices fall another -20%?

Or do you wait for a +20% advance?

If you tap out after fearing more losses, when do you ever reenter? After you fear missing out?


It isn’t a situation with a good answer, because we never know what’s going to happen next.

My answer is to avoid the drawdown in the first place.

I instead prefer to actively manage my risk and apply drawdown control systems designed to limit the drawdowns.

By the way, the scenarios I described are discretionary decisions. More than anything, discretionary trading must master themselves to develop skill and discipline. 

The inability to execute decisions in the heat of the battle is the basis for the failure of the discretionary trader. Imagine what it would be like in a losing streak when entry and exits result in losses over and over for months or years.

But, systematic isn’t immune. The inability to follow a strategy with discipline is the basis for the failure of the systematic trader.

Some quantitative systems traders mistakingly proclaim their systems and models “remove the emotion from the equation” which is a sign of inexperience or lack of the mindset of a skilled operator. I feel my feelings like everyone else, except I feel the right feeling at the right time. I programmed myself to embrace the emotions and make it an edge.

The blue line is the full return stream of our hypothetical trading system example.

asymmetric trading system risk reward

If you compare it to the orange system you probably feel disappointed and unimpressed with the return post-2013.

But the difference is you may have stayed with it with9ut tapping out at the low.

Mic drop … THUMP!

But right now you’re thinking; what have you done for me lately?

If so, you would have never achieved any of this performance as most investors don’t according to studies from agencies like Dalbar. Investors tend to do the wrong thing at the wrong time. If they are real with themselves, they see what they’ve done long term.

In this example, the same system that avoided the tap out level drawdowns is the same one that has obviously taken on less exposure to loss the past five years.

To achieve it’s long term results over these full market cycles, you’d have to stick to the system.

Otherwise, you’d find yourself sitting there in a panic trying to figure out what’s going to happen next, you’ll never know the answer, and you’ll try to figure out what to do.

So, systems trading isn’t necessarily any easier. Systems don’t always work as there are hostile conditions for every system or simply periods when they don’t do as well.

A hint; markets cycle through different types of regimes. What works well with today’s backtest is unlikely to work well into the future. If the future isn’t like the past, the results won’t be the same. What does work best in the next cycle is often what doesn’t’ test so well over the prior four or five years, so investment advisors who sell backtested performance are likely to constantly disappoint. Who would buy a mediocre backtest? The incentive is strong to produce the over-optimized backtests.

Here is an example that looked great during the 2008-2009 crash as it made money, but…

overfitted trading system symmetry

You can probably see why they don’t work out as expected.  An over-optimized backtest that’s overfitted to the past can be much worse than the charts above.

At the end of the day, we always create our own results. Even if you invest with an asset manager, you still create your own results. So, be honest with yourself about it.

You decide the outcome one way or another. Be sure you’re confident in what you’re doing. For me, it started with exhaustive quantitative testing and improved with two decades of doing.

Mike Shell is the Founder and Chief Investment Officer of Shell Capital Management, LLC, and the portfolio manager of ASYMMETRY® Global Tactical.

Mike Shell and Shell Capital Management, LLC is a registered investment advisor focused on asymmetric risk-reward and absolute return strategies and provides investment advice and portfolio management only to clients with a signed and executed investment management agreement. The observations shared on this website are for general information only and should not be construed as advice to buy or sell any security. Securities reflected are not intended to represent any client holdings or any recommendations made by the firm. Any opinions expressed may change as subsequent conditions change.  Do not make any investment decisions based on such information as it is subject to change. Investing involves risk, including the potential loss of principal an investor must be willing to bear. Past performance is no guarantee of future results. All information and data are deemed reliable but is not guaranteed and should be independently verified. The presence of this website on the Internet shall in no direct or indirect way raise an implication that Shell Capital Management, LLC is offering to sell or soliciting to sell advisory services to residents of any state in which the firm is not registered as an investment advisor. The views and opinions expressed in ASYMMETRY® Observations are those of the authors and do not necessarily reflect a position of  Shell Capital Management, LLC. The use of this website is subject to its terms and conditions.

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s