The effect was never there... it was just a random pattern

This caught our attention and we thought it worth sharing: Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance wherein we learn, should we have ever doubted, that Phineas Taylor Barnum is alive and well in the form of excessive back testing. Excessive backtesting is easy to do given the computational power currently available to most researchers and when combined with an absence of disclosure of relevant backtesting parameters, it creates an in inability to distinguish valid from invalid results and often with the consequence of adverse investment performance. The abstract is excerpted in whole below the quotations, also from the paper.

...

"with four parameters I can fit an elephant, and with five I can make him wiggle his trunk"

 John Von Neumann

"another thing I must point out is that you cannot prove a vague theory wrong. [...] Also, if the process of computing the consequences is indefinite, then with a little skill any experimental result can be made to look like the expected consequences." 

 Richard Feynman [1964]

“the effect was never there; instead it was just a random pattern that gave rise to an overfitted trading rule”

the authors

...

Abstract

Recent computational advances allow investment managers to methodically search through thousands or even millions of potential options for a profitable investment strategy. In many instances, the resulting strategy involves a pseudo-mathematical argument, which is spuriously validated through a simulation of its historical performance (also called backtest).

We prove that high performance is easily achievable after backtesting a relatively small number of alternative strategy configurations, a practice we denote “backtest overfitting." The higher the number of configurations tried, the greater is the probability that the backtest is overfit. Because financial analysts rarely report the number of configurations tried for a given backtest, investors cannot evaluate the degree of overfitting in most investment claims and analysis.

The implication is that investors can be easily misled into allocating capital to strategies that appear to be mathematically sound and empirically supported by a backtest. This practice is particularly pernicious, because due to the nature of financial time series, backtest overfitting has a detrimental effect on the strategy's future performance.

...

Our take is quite simple. The best protection: do not invest in that which you do not understand. If you buy financial products on the basis of backtesting, you had better understand the mathematics behind it, and if you do not, you better have in advance a dis-interested and sophisticated statistical sherpa for your guide. Lastly, we wonder how many documents disclose overfit as a material risk?