About two decades ago I thought I had a handle on big data. I was doing some data warehousing work with a telephone utility that had about 100 million transactions. That was a lot of data, I said to myself. Then, about 10 years ago, I was doing a review of a firm that audited financial trading on one of the major stock markets and I asked its big data guy how many transactions the company processed. His initial answer was, “On a slow day we get about 2.5 billion transactions.” “How many do you have on a busy day?” I asked with an air of shock. “4 or 5 billion,” he responded. Now that was really a lot of data.
Jump ahead a decade or so, and on 24 July 2014, Facebook announces that it is currently processing 1 trillion transactions per day. Now ”that” is really, really big data. If you are a CEO, that is just one of the reasons why you should worry about having a big data strategy. Even if your organization isn’t a telecom utility or a financial institution, the amount of data you’re going to have to process is shooting up, what with all the smart (wireless) devices your customers and employees use heavily, plus the volumes of data beginning to flood the organization from all the IoT devices/systems that increasingly control any number of real-time systems.
Now, it is unlikely your organization will have a Facebook data problem any time soon. However, it is very likely your organization will be handling 10s or maybe 100s of billions of transactions in this decade. Equally important, this data will not be the traditional transaction data you’re used to. No, this data will be unstructured (e.g., text, pictures, pictures, diagrams, voice, video, etc.) and, if your organization is like most, it won’t have had much experience building, deploying, and maintaining such systems. Nor may it have had much experience with the external data that will be increasingly available from, for example, Facebook, Google, government agencies, and so on.
So, without being an alarmist, those in your company charged with addressing your big data strategy must begin to think in terms of both tactical and strategic opportunities and challenges associated with developing this kind of plan. If I were seeking someone to lead this effort, I would probably look in the current BI organization. If you don’t have a BI organization, you need to put one together, fast. If I were building a new BI organization, I would build around the people I know already are responsible for management reporting and I would bring in a big data consultant for the short term. At the same time, I would start to search for data scientists and/or begin to train bright young people within the organization for the job.
Finding experienced data scientists won’t be easy or cheap, though, because the market for experienced data scientists is very, very tight and the salaries are skyrocketing. So, given all these factors, I recommend that one of the key elements of your big data strategy include developing an inhouse training program.