For years now, I have made a good living by exploiting Geary Rummler and Alan Brache’s famous subtitle, “How to Manage the White Space in the Organization Chart” (Improving Performance: How to Manage the White Space in the Organization Chart, Jossey-Bass, 1995). What Rummler and Brache meant by white space in this case was all the activities for which no one was explicitly responsible. My experience, which mirrors theirs, is that those things that fall between the cracks often get overlooked and undermanaged. Much like “white space” is the physics notion of “dark matter.” Dark matter is that unobservable “stuff” that physicists have computed exists in the universe that causes known celestial bodies to behave the way they behave with respect to our most fundamental physical laws.
In both white space and dark matter, there is an understanding that something important is being overlooked that is only known indirectly through some observed effect on some other body. The same is true, I have concluded, in the way we understand the software systems and data that exist in the average large enterprise. CIOs and CTOs constantly refer to the major applications and databases known to them. However, if one examines the difficulties in replacing or even making major improvements or modifications to those systems, one has to reason that there are dozens of small (and sometimes not-so-small) undocumented systems (“dark systems,” if you will) that exist in the white space between the major systems.
Any knowledgeable systems manager, especially after a drink or two, will acknowledge that there are indeed large numbers of small applications that orbit around the acknowledged large systems as well as hundreds — sometimes thousands — of undocumented databases and interfaces that exist as well. In my EA classes, when I refer to some of these dark systems as “mission-critical spreadsheets” or “phantom ACCESS databases applications,” my students nod knowingly. Increasingly, we hear the term “spreadmart.”
A great many of these dark systems have been developed without the knowledge, much less the assistance, of the central IT organization. A fair number of the major dark systems in most large organizations include the business intelligence applications that exist to support marketing and/or financial functions and which have their own developers, employees, or contractors who are often not classified as programmers or developers but as management analysts or general consultants.
My experience also shows that large organizations rarely face up to the number of dark systems that exist in their systems’ white spaces (or see how critical they are) until there is: (1) some major failure, (2) an urgent desire to replace some core system, or (3) some key person leaves (or dies), and it is discovered that some heretofore unknown system exists that has little or no documentation. This is not surprising. Central IT usually has enough on its plate without looking for more work and more organizational fights. However, failing to adequately understand just how central many of these “off-book” systems are creates a biased picture of the organization’s true systems and database landscape. In a future Enterprise Architecture Advisor, I’ll address what needs to be done to bring all these dark systems into the light.