I’ve long had Schmarzo’s book on big data on my shelf. I read it with great interest a few years ago when was in the early stages of my PhD, and I’ve returned to it now when a student asked for a good book on big data analytics in practice.
As I leafed through it, I appreciated again the applicability of Schmarzo’s writing to business. The book is a great big data primer for business people, with good case studies and easily digestible chapters. Just for these reasons, I’d recommend either this title, or the newer “Big Data MBA: Driving Business Strategies with Data Science”, as well as following @schmarzo on Twitter.
But Schmarzo also touches on a subject now very close to my heart, which I glossed over upon the first reading of his book. Somewhere early on in the book he starts talking about gaming the metrics: “one of the challenges with metrics is that eventually folks learn how to game the metrics for their own advantage”, he says. He deploys the example of baseball and the fielding percentage metric which used to be gamed by players improve their rankings, and how now big data changed it and made it possible to “get a better idea as to actual game dynamics” rather than metrics gamed by players. He goes on to say that cameras deployed on pitches are beneficial for team managers as they give “a new set of metrics that are better predictors of players’ performance”.
It’s interesting that Schmarzo accepts that metrics can be gamed, but identifies big data analytics as a solution to it, rather than part of the problem. If metrics can be gamed, and big data analytics is just a new set of metrics, surely it can be gamed too? It may be more challenging to work it out, the gaming strategies may be more sophisticated, but ultimately isn’t every metric in the danger zone? We can, of course, assume that big data analytics is very different and so different that it can’t be gamed. But this assumption is dangerous and may lead to all this analytical work being not really any more efficient than the old metrics it was trying to replace.
In other words, I’d like to suggest that it’s more prudent and it makes more business sense to assume that big data analytics can be gamed, just like any other metric. Now, how can we design analytics systems that take this effect into account, so their usefulness doesn’t become limited the moment they hit the market? Well, I guess, this is part of my research agenda for the years to come…