The State of BI in 2010

 Posted by on Oct 19, 2010  Add comments
Oct 192010

It has been just over 50 years since preeminent IBM computer scientist Hans Peter Luhn coined the term “business intelligence.” And ever since then, BI has been viewed as getting information to the people who need it in a timely fashion and in a form that is easily consumed and acted upon (the right data to the right people at the right time). From those seemingly prehistoric days of data processing, when BI consisted primarily of monthly reports on green bar paper, to today’s splashy interactive graphics on wireless mobile devices, both the data that is available and the means with which to deliver it to the right people have changed dramatically.

But have these changes made a measurable impact in organizations, or have the results fallen short of the promises? Almost everyone has access to far more data in a much more timely fashion than they used to (well, almost everyone), but is it better, more actionable data? In a real sense, BI tools, and the data hubs, data warehouses, and operational data stores that feed them, expand our view of data in the same way that a more sensitive telescope can see deeper into the universe. We now have the ability to see things in detail that we could never even see before; we can see deeper into the data on which our enterprises function. The real question with BI today is, are we seeing the right things at the time we need to see them?

Over the last few years, I’ve worked with a number of clients using a variety of BI tools. In many of those organizations, the “enterprise-level” BI tools seem to be limited to a small number of users, predominantly in the area of financial services. I worked with one organization a few years ago that had spent several million dollars on a BI solution only to discover that, once implemented, it had only 20 users in the entire company, and, of those, only a dozen or so were regular users. BI tools seem to be most often used to monitor costs and other predictable key performance indicators (KPIs), such as inventory levels, repair times, and other largely cost-related metrics. Occasionally, they are used to monitor value chains, such as supply and fulfillment chains. Once again, their use seems to be mostly limited to predictable metrics around spotting bottlenecks and trouble spots in the processes that are being monitored. On the flip side of the cost equation, BI tools are occasionally used to actively monitor revenue and growth forecasts and alert people (and, rarely, other systems) when problems seem likely to occur. Again, in most cases, the actual number of users in the organization is very small.

For many years, BI vendors have sold the promise of the “executive dashboard” — a one-stop, all-encompassing view of the enterprise in which senior executives and decision makers are presented a graphic set of dials and charts they can use to instantly see and monitor the health of the organization. In this vision, any time an executive sees a red light appear, he or she can simply click on the widget to drill as deeply as necessary into operational numbers. The problem is, these dashboards seem to be few and far between in the real world. In fact, organizations have built executive dashboards in the past only to find that the executives don’t use them. And that seems understandable: although portions of the enterprise can be distilled down to meaningful dashboards with a few KPIs, a large multinational private enterprise (or large public sector entity) is far too vast and complex to be translated into a handful of gauges and dials colored “red,” “yellow,” or “green.” If it could, just about anyone could run it by simply managing the dials.

Perhaps, Kas Kasravi points out in his article in the June issue of Cutter IT Journal (“A Convergence in Business Intelligence“), the next generation of BI should be actually “driving” the KPIs. As Kasravi points out, we already do let some systems autonomously manage low-level systems, such as cruise control and autopilot systems. Some of today’s systems do the same thing in financial equity trading: the so-called high-speed trading systems spot emerging market trends and make automatic buy/sell decisions in milliseconds. (In fact, these systems are colocated as physically close to the markets as they can be to eliminate latency due to the speed of light.) As Kasravi’ points out, there are going to be ongoing trust issues with letting BI systems “drive,” as evidenced by the so-called flash crash that took place in May 2010. As you probably know, it seems that multiple high-speed trading networks all decided that a particular afternoon was an excellent time to sell, which of course led to other systems deciding it was time to sell, and so on. After the Dow Jones average dropped and subsequently recovered nearly 1,000 points in a few minutes, many were left wondering whether high-speed trading BI algorithms should be allowed to “drive.”

For many years, BI vendors have also been promising meaningful analysis and integration of unstructured data. The storage, categorization, and retrieval of documents, e-mails, and other unstructured data have spawned an entire industry of content management, which many would consider to be a subset of the more generic BI. And in this exploding era of social collaboration, organizations are starting to use even more unstructured data from social media to enhance and support their BI capabilities. Ranging from mashups, to Web 2.0/3.0, to on-demand/cloud offerings, these tools may improve organizations’ BI for better data analysis, forecasting, and collaboration. Yet while some initial results seem promising, the integration of unstructured data into BI systems remains elusive for most organizations.

Perhaps since the idea’s birth in the late 1950s, we’ve come to expect too much of BI. I am reminded of a conversation Ken Orr and I had with T. Capers Jones back in the 1980s. His research at the time had studied the growth of data processing and projected out the personnel needs for the next couple of decades. The numbers told an interesting tale. They suggested that by the year 2000 (some 15 to 20 years in the future from then), every person in IT would have to be a computer programmer, or there would simply not be enough resources to satisfy the projected demand for new systems. He then pointed out that AT&T had done similar research at the turn of the 20th century, and back then it appeared that everyone on the planet was going to have to become a telephone switchboard operator in order to support the rapid growth in demand for telephone services up to that time. The problem, of course, is that technology continues to change. In a very real sense, everyone today is a telephone switchboard operator when we directly dial someone we wish to speak with, just as everyone in IT is a programmer when we create and manipulate spreadsheets, produce presentations, build reports, and prepare complex documents.

It seems that compared to the late 1950s, much of the original vision of BI has been realized in most organizations, and we have been so immersed in the changes that have taken place that the everyday satisfaction of BI needs often doesn’t feel like true BI anymore. An analogy to warfare has been often used: when you are on the ground fighting a battle for the next hill, you don’t see the hill beyond it until you’ve overtaken the hill immediately in front of you. Only then can you see the territory beyond and realize that although the battle for this hill is over, there’s another, perhaps larger, hill looming in the distance. Our expectation that BI systems will be the be-all and end-all in providing everyone in the organization with the right data at the right time is really what’s been happening in small measures all along. The only reason we realize that there are unfulfilled goals of BI is that we are now standing on a different hill, looking to another distant goal.


Dave Higgins

Dave Higgins is a Senior Consultant with Cutter Consortium's Data Insight & Social BI practice. He has been a strategic management consultant since the late 1980s and an evangelist for high-quality software systems development methods since 1975.


  One Response to “The State of BI in 2010”

  1. Dave, there are so many pithy observations in this piece, I’m tickled pink. Two are worth emphasizing:

    1) Everyone is a programmer. Examination of the average IT setting suggests that this is nearly true. Spreadsheet, presentations, and documents are by no means the end point. Web sites, web applications, data collection, basic workflows, data analysis, and even complex analytics are all within reach of a determined individual. It’s getting harder and harder to distinguish the “software professional” from the dogged needy. At least, the average business or IT executive cannot distinguish between the two.

    2) Dashboards are too frequently unsuccessful. Consistent measurement is hard work. The Agilistas can certainly testify to this, as well. I hear the following refrain from many IT staffers considering dashboards: “isn’t that just a tool?” In our experience, viewing a dashboard as a tool problem is looking through the wrong end of the telescope. The real problem is getting access to operational data from real, operational processes that are supportive of meaningful measurement. Whether home spun, Cognoz/Business Objects/Actuate, Sharepoint, Excel, the reporting tool does not matter. What matters is getting the business practices to “throw off” the right data. This problem is closer to change management than BI.

    Thanks for the thoughtful piece.

 Leave a Reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>