In a mutual boot-strapping beginning with the dawn of Homo sapiens, mankind and information have both exploded in variety, velocity, and volume. Our fates have been intertwined. We advance by harvesting, using, and sharing information. Along the way, information persists, mutates, and diffuses further. Turning the crank, we find — no, we create — even more information. In a race to compete with each other, we endlessly seek more information to help us do better. The mass of information grows larger still, revealing more secrets. From the primordial single bit of information at the dawn of the universe has sprung a thing that now feeds itself in a never-ending cycle of perpetual novelty. And the crank turns faster and faster ….
To deal with the enormous size of information, we have not only to create better containers to hold it, we have to devise even more complex and difficult ways of describing it. We create metadata. With metadata as a handle, we can grip large collections of data within one concept and move it around. Today, all of information is desperately seeking metadata. We are now developing means of automatically creating metadata and automatically moving larger crates of data around so we can sift, sort, shuffle, distill, and otherwise remix information, hoping to gain advantage.
Since the beginning of humankind, we have found advantage in information. We have learned that hiding information from competitors or, for that matter, reducing any threats to us, helps us survive. When competitors meet and one side has a decided information advantage, the results can be devastating. The inferior side is forced into a sometimes humiliating retreat. The other side then has the freedom to define more metadata favoring their information handlers and disadvantaging all others. History is not always rewritten by the victors but metadata is. And rewritten metadata results in rewritten culture. Think the Battle of Hastings in 1066.
Not all of information denial is found at the feet of conflict. Sometimes a smart person or group blows our minds with entirely new ways of conceptualizing things. They set the agenda for the next transformation of information and metadata as diffusion and adoption of the new model occurs spontaneously with far less conflict. Think hula-hoops and pet rocks.
Nonetheless, information was and still is power. These days, the insight into information is increasingly abstract, arcane, and highly technical; still requiring a near-priestly guild to decipher and interpret for the rest of us the strange collection of mathematical symbols and algorithms that reveal the new order. Once in a while, those who are threatened try to suppress these technocrats, fearing the implications to the current regime. Books are burned, programs shut down, research suppressed and defunded. But denial isn’t just an active process. More often than not, entire teams, cultures, companies, and nations, locked into their own mental model echo chamber, can’t even see what is about to supplant them. Those in denial are unaware of their denial. The gods of irony are smiling.
To twist a March Madness (college basketball) phrase, sometimes organizations fail to stop the house from burning because they feel to see that the house is burning. By failing to adopt the mindset of their competitors or critics or properly see the threat, some organizations inadvertently allow the competitor to redefine the market and start taking territory too fast for them to stop.
Our ability to process information in our own brains is limited. While we are wired to absorb information and use it for advantage, we are still not wired to see enough of it correctly. Neuroplasticity only goes so far. Even if we step back a bit and look beyond Big Data to more decision automation through Watson-like artificial intelligence technology, so long as humans feel the need to use information to compete, we will make errors in our judgments due to our inability to conceptualize the world or our desire to stop others from reconceptualizing the world for us. A big, colossal machine to help humans make decisions merely creates an opportunity for big, colossal human errors.
With Big Data will come even bigger denial.
Sooner rather than later, humankind will need to address this big gap. But the odds are against us. As a human process, denial is hardwired into our brains and reproduces itself spontaneously and reliably globally and across time. To borrow from the renowned organizational development guru Chris Argyris, we do this effortlessly and skillfully. We have one heck of a foe: ourselves. Diffusion of how to use new technologies, what we call best practices, has been instrumental in human development. And it is still critical today. What we need right now is a clear and commonly understood set of practices that help us use our Big Data systems.
Already organizations are rethinking their governance processes regarding information diffusion within the enterprise. Governance processes tend to shield individuals from direct introspection and evaluation and place responsibilities in groups, which is helpful. But information denial is a deeply personal process. Worse still, not all of information denial is problematic. In some cases, information denial helps people naively persist in eventually beneficial actions. Think about entrepreneurs who start with a combination of optimism and ignorance that enables them to approach difficult situations and eventually learn. Denial can lead to learning. Learning exists because we are in denial or unaware of something. The fates of denial and learning are also intertwined.
To safely use Big Data we need to learn to temper our instincts that lead to denial.
What I will leave us with is a list of questions that are not meant to improve governance processes themselves, but questions for all of us to think about. These questions include:
- Is my gut feeling reliable?
- Do I eagerly look for disconfirming evidence for my plans?
- Do I challenge my own assumptions and review my own reasoning processes?
- Am I being overly optimistic?
- Am I continuing to invest in what should be stopped?
- Am I being biased by recent or significant events in the past?
- Am I failing to reconsider information that might challenge my decision?
- Have I received enough diverse viewpoints on the subject?
- What are competing perspectives or philosophies I should consider?
- Am I dismissing the competition too easily?
- What would be some unintended consequences that I haven’t considered?
- Are my information processing methods sound?
- Am I relying on too few sources of information?
- What biases exist in my sources of information?
- How would my competitors view my decision?
- How would my family view my decision?
Big Data will lead to big denial. The endless yin and yang of learning and denial that is probably forever part of the human condition ought to create some big opportunities for enterprising folks who can help us tame our denial instinct.