Data Mining for Stocks

Recently, I linked to an NYT article about data mining and statistics. I wrote about how valuable these skills are becoming, but indicated that it's possible to make big mistakes if you don't know what you're doing. This is why it's a good idea to get trained by experts who use statistical analysis in their own work all the time (such as our faculty here at the University of Utah's David Eccles School of Business).

And as if by divine command the WSJ posts an article illustrating just this fact:

Data Mining Isn't a Good Bet For Stock-Market Predictions

Cool nugget in the article: Data mining techniques suggest that Bangaldeshi butter production is highly correlated with US stock returns. So, get the butter data and make a fortune in the stock market, right?

This, of course, is completely ridiculous. As the article reminds us, correlation does not imply causation, and you have be both smart and well-trained (see plug to DESB, above) to know the difference.

But there's more to the stock-market/data-mining interface than just this. To illustrate my point, note that we can probably summarize the WSJ article as follows:

Data mining with stock market data might give you spurious correlations.

But the deeper point (which isn't discussed in the WSJ article but should be) is this:

Any correlations you find doing data mining on stock market data are probably spurious correlations.

Why?

It's the profit motive together with the price mechanism.

Here's how it works: Suppose data mining uncovers a real, causal relationship between some variable and future equity prices. As an example, suppose that any time it snows more than 6 inches at Alta on Martin Luther King Jr. Day, then the S&P 500 goes up by 10% in February. Well, the profit motive means there are lots and lots of people out there looking for such patterns in the data. And once these patterns are discovered, people will start to trade using this information.

We'll end up with a lot of people watching the MLK-day Alta snowfall. When it snows, they'll all buy stock, hoping to cash in on that 10% return in February. And what will their purchases do to market prices? Drive prices up, of course. How far? Prices will be driven up the full 10% on the day right after MLK day. And this means that prices won't go up by 10% in February; because that increase has already been priced into the market.

The profit motive together with the price mechanism will tend to knock out any "real" data-mining/stock-market connections, pretty much immediately.

So the question to ask about data-mining for stocks is this: If the connection is real, then why haven't traders found it (and exploited it) already?

This isn't to say that one could never make money doing data-mining type stuff on stocks. I do know people who do just this, and they seem to make money (sometimes, at least). But these guys have really big computers, and really big data sets, and they're doing really complicated stuff, and on top of that they're always asking themselves why the correlations they've found haven't been found (and exploited) by others. It's an extremely, extremely competitive area. So you gotta be careful.