How to recognize false prophets

You have seen “too good to be true” system performance claims, which usually turn out to be just that, even with supporting data. Here is how data can be massaged.

Who hasn’t received a flyer like this in the mail (see example quote below)? Our better instincts should tell us the example quote looks — and therefore is — too good to be true. The best traders in the world will confess to being right not much more than 50% of the time. So what gives here?

Is the vendor outright lying?

Technically, maybe not. He or she might well be able to produce a formula that generates the exact performance seen in the touted time frame. Unfortunately, the results are derived through backward retrofitting that will have nothing to do with future results. All the vendor has created is a perfect picture of the past. 

One way to create such a scam is to start out with a basic system and then add rules that lessen or eliminate the worst losses in the data field. My “flyer” was meant to be an anti-teaching tool joke, but the figures within were real in that they were taken off a Trade Station
performance summary in the Russell 2000 (see “Basic Charlatan system,” below). This represented a single mini contract traded between Jan. 2, 2005 and Dec. 31, 2016. No slippage/commission was added. Again, it’s an actual summary — no photo-shopping or any other editing trickery was used. Whether or not it’s a legitimate summary is another matter.
You can decide for yourself after following me through the steps. 

The numbers are eye-popping, as is the objective of any too-good-to-be-true come-on. How is it done? How were those numbers produced and why should they not be accepted? Again — reverse engineering. We started with a basic system shown in “The four-day average close systems,” below). 

If the close is less than the four-day average close, enter a long on the next open plus 75% of the average three-day range on a stop. (These are day session daily bars — 8:30 am through 3:15 pm Central time — no Globex). If the close is greater than the four-day average close, sell short on the next opening minus 75% of the average three-day range on a stop. Exit at a $1,000 profit target or exit and reverse on a contrary signal. 

The fact that there are no stops should be your first inkling that this system wasn’t designed with the best interest of the end user in mind. Numbers-wise, this is the worst variation you’re going to see in this chapter. It’s also the most legitimate version; technically, a near bona fide system. You wouldn’t want to trade without stops or live through the drawdowns, but at least the concept is simple and applied honestly. From here, nothing will be honest. Each new version will show progressively better results and inversely proportionate statistical legitimacy.

Optimizing to sell, not earn

There are several ways to nudge figures higher. We’ll concentrate mainly on the win-loss ratio; normally the best selling point of any unscrupulous solicitation. 

We’ll focus on losses. Imagine that we’re scrolling through a chart containing the system. We can visually note what the daily bars looked like immediately preceding the entry signal of one of our more disastrous trades. There are many conditions we could apply that would eliminate it. True, we’ll likely lose other trades before and after, including wins.
It’s also possible that since a previous trade won’t exist, a new trade could pop up on a different date. Again, it could be a winner or loser, and there could be several of them. The whole trade history might look entirely different. Sometimes the tradeoff won’t be worth it, but we can persevere until we find one that is. In short — add a new filter, eliminate the monster loss, and incorporate the variation when our net result jumps significantly in our favor. 

“Optimizing to sell,” (below) shows the performance summary of our first filter. It acts only on the short side (legitimate system rules are the same for longs and shorts. Never trust a performance summary that looks better because of dissimilar long/short mandates, but remember, we’re being deliberately illegitimate here). 

The rule:

DO NOT take a stipulated short if the highest two-day high minus the lowest two-day low is at least 1.2 times the size of the average two-day range.

Quite a jump. We’ve gone from $7,955 to $34,030 net profit. We’ve cut the worst drawdown by almost 40%. Our average profit has increased nearly nine-fold. We’ve gone from 50% to 62% profitability. What’s not to love? I’d contend that our added rule doesn’t make much theoretical sense and therefore the improved performance probably doesn’t mean anything. That’s my educated opinion, but I could be wrong. Let’s try to get more blatantly ridiculous as we go along.

 “Overfitting for fun,” (below) uses a dual filter against the long side. 

The Rule: 

DO NOT take a mandated long if the last bar’s high minus its open is less than or equal to 10% of the bar’s total range or the close was under the previous bar’s low. 

We’ve increased the net profit even more, but let’s not focus on that figure. As we continue to weed out trades, we’re likely to be left with less profit. We’re focusing on the percent profit and its related better drawdowns. Once we get to the $27,000 return on a mere $2,140 risk with only six losers, we may feel perfectly safe trading two, three or five units. And shouldn’t we? Don’t performance summaries give us the whole truth and nothing but?

If you’re still not convinced that our added rules so far are nothing more than nonsense, our final adjustments should convince you. We’re simply going to eliminate a day of the week and month from each side. Think about it: If we looked at all five days and 12 months, shouldn’t at least one from each group have an especially huge negative impact on the total result? It probably exists somewhere in the field, right? What are the odds that today’s biggest negative outlier has some magic impact that will persist into the future? Extremely low. This is clearly data mining at its worst, but it sure gets the job done if our only goal is to come up with something we can sell (see “Data mining for dollars,” below). 

The Rules: 

DO NOT take a mandated long on Thursday or any time in April.

DO NOT take a mandated short on Wednesday or any time in March.

That’s all the filters applied. What about that 96% profit with the six losing trades out of 130? That was achieved via a common method for bumping up percent profitability: reduce the profit target. We’re dropping the original $1,000 down to $200. That converts many of the former losers into winners: Trades couldn’t survive the $1,000 distance (20 points), but profit under the less stringent $200 limit. 

Be honest, especially with yourself

As humans, we all desire frequent affirmation and our scam has certainly provided plenty of that. We’ll happily collect our many winners until we sit through one of those comparatively humongous losing trades. Suddenly, we’re in a deep hole that will take an awful lot of the $200 profits to bail us out. God forbid, we hit a second, third or fourth loss before we’re back to ground level.

There’s a twofold point to all of this. One is the obvious one: You now have an understanding of how unscrupulous vendors operate. Be skeptical of performance summaries that are too good to be true; they almost certainly are.  Remember that a charlatan’s primary focus is on the percent profitability. An unrealistically huge win-to-loss ratio plays into our wish to not only have an overall winning system, but also one that is likely to reward us on any given individual trade. That is an unobtainable pipe dream. The best traders in the world are correct around 50% of the time. Their success comes from effective money management, not magical prognostication. Regard any two-to-one win-loss claims or better with extreme skepticism.  

You can duplicate my exact results in TradeStation or merely accept that I’ve  demonstrated how simple it is to finesse numbers. It’s far easier than producing summaries portending something promising for the future.

The second point is that it may be you rather than a gullible public that falls victim to deceptive numbers in your own testing. In this case, it would be carelessness rather than malevolence that trips you up. You have to be on guard against letting wish fulfillment pollute your work. Your goal in creating mechanical systems is trading without emotion. As much as possible, you have to keep that human frailty out of the research lab as well.