Big data: big opportunity, or big headache?

Big data is the breaking news story in the IT world.

Here’s the picture. Sometime soon company managers will have an enormous amount of information at their fingertips. They will be able to see everything and optimise everything. What could be better?

Within the past month, two very senior, astute individuals contacted me about this. One is a senior IT industry analyst, and the other a senior editor of a major business publication. They both had the same question: If big data becomes available, what will be the consequences?

My answer: big data offers big opportunities, but it carries the  strong likelihood of creating really big headaches in three areas: (1) paving the cowpaths, (2) managing at the right level, and (3) driving without a roadmap. Let me explain.

Paving the cowpaths

Remember Radio Frequency Identification, or RFID?

A few years ago, RFID was all the rage. An RFID tag is a small passive label that emits a precise identifier when it is hit by an electromagnetic field. The idea was that it was becoming technically and economically possible to affix an RFID tag to every item moving through a supply chain. Armed with this capability, managers could know the identification and location of every item in their company, and even those flowing into and out of their companies.

I remember discussing this with a former student, who is now a senior operating executive of a Fortune 20 company. His reaction was similar to mine: “Whatever would you do with all that information?”

In fact, several years ago I co-authored a column, “Are You Aiming Too Low with RFID?” in Harvard Business School’s Working Knowledge. In the piece, I joined Sanjay Sarma, co-founder of MIT’s Auto-ID Lab, and John Wass, CEO of WaveMark, to argue that the biggest danger with the flood of RFID information was the almost-irresistible temptation to focus on “paving the cowpaths”.

Here’s how we put it. “One of the most exquisite challenges of living in Boston is navigating the labyrinthine maze of streets in the downtown area. This part of town is the oldest part, and the streets follow the original paths formed by settlers driving their cows to pasture. Traffic flows poorly because the city fathers simply paved the cowpaths, making the ineffective more efficient. It’s much easier to navigate Back Bay, a part of Boston with grid-like streets, built on landfill centuries later.”

In the article, we outlined a number of highly focused, high-value analytical uses for the “big data” that RFID could produce, and advised avoiding the large-scale applications that simply automated routine activities.

In the absence of this disciplined, strategic approach, managers are in grave danger of utilising big data to pave the cowpaths, further entrenching existing practices and rendering the possibility of developing sweeping paradigm-changing initiatives more and more difficult.

This will almost inevitably occur because in the capital budgeting process, the tactical payoffs from paving the cowpaths will be clear and easy to measure, while the payoffs from far-reaching strategic changes in the business will be hazy and unmeasurable.

Managing at the right level

I was recently reminded again of the big data question when I re-read Wired for War, a terrific book by P.W. Singer which traces the robotics revolution and the use of robots in 21st-century conflict.

In a particularly telling chapter, Singer describes how the real-time video feeds from drone aircraft – or, big data – led to the systematic leadership problems that I call “managing at the wrong level.” (See my column, Managing at the Right Level.)

Over many years, improved communications technology has increasingly enabled commanders to command at a distance from actual battles. This has led to a very effective management structure in which top commanders focus on strategy and personnel, mid-level commanders on operational initiatives, and local officers on tactical issues. This parallels the leadership structure of most effective companies.

However, the widespread availability of drone aircraft information feeds has led to serious and systematic command and leadership problems. The ability of top commanders to see battlefield video feeds in real time has rapidly increased the centralisation of command and led to an explosion of micromanagement.

Crack for generals

Singer relates: “Too frequently, generals at a distance are now using information technology to interpose themselves into matters that used to be handled by those on the scene and at ranks far below them. One battalion commander in Iraq told how he had twelve stars worth of generals (a four-star general, two three-star lieutenant generals, and a two-star major general) tell him where to position his units during a battle.”

Singer continues: “An army special operations forces captain even had a brigadier general (four levels of command up) radio him while his team was in the midst of hunting down an Iraqi insurgent who had escaped during a raid. The general, watching a live Predator video back at the command centre…ordered the captain where to deploy not merely his units, but his individual soldiers.”

“It’s like crack for generals,” says Chuck Kamps, a professor at the Air Command and Staff College. “It gives them unprecedented ability to meddle in mission commanders’ jobs.”

This direct meddling by military leaders has led to the rise of what Singer calls the “tactical general,” as the line between timely supervision and micromanagement has became blurred. Officers in the field lament what they call the “Mother, may I?” syndrome which has come with these new technologies.

Moreover, power struggles are common when the feeds are available to multiple command groups.  Singer notes: “At its worst, this pattern can lead to the battlefield versions of too many cooks spoiling the meal. A marine officer recalls, for example, that during an operation in Afghanistan, he was sent wildly diverging orders by three different senior commanders. One told him to seize a town 50 miles away. Another told him to seize the roadway just outside of town. And the third told him not to do anything beyond patrolling five miles around the base.”

But the biggest problem with top-level micromanagement in the military – just like in business – is the huge hidden opportunity cost of failing to manage at the right level: a leader ignoring the critical issues of high-level strategy and organisational capability because he or she is so caught up in real-time micromanagement. This causes two very big, related problems.

First, the top managers fail to plan for the future. For example, in business, vice-presidents should primarily be focused on defining and developing the company as it should be in three to five years, since that is the time it takes to develop a new set of capabilities. Their other critical responsibility: coaching and developing the next generation of leaders.

In the absence of this hierarchical discipline, the company is in grave danger of getting mired in the present, and falling further and further behind.

Second, when top managers – or generals – take over tactical decisions, the lower-level leaders cannot develop their skills. Instead, they must be empowered to act with initiative, even if it means making a few mistakes along the way. No – especially since it means making a few mistakes along the way, since false starts and errors are a natural and necessary part of doing anything significant and new.

The answer? Singer calls it “enlightened control”, a concept he credits to the great Prussian generals of the 19th century, whose ideal was that the best general gave his officers the objective and left it to them to figure out how best to achieve it. He cites the commanding general who so trusted his officers that the only order he supposedly issued on the eve of the Prussian invasion of the Danish province of Schleswig was, “On February 1^st, I want to sleep in Schleswig.”

The action question for managers is: Will big data be “crack for your vice-presidents,” or will they have the insight and discipline to double down on “enlightened control”?

Driving without a roadmap

One might ask: But won’t big data let a company’s managers optimise everything? After all, every manager will have, theoretically, the information needed to get everything right. And if a company’s managers optimise everything, won’t the company be great?

This question embodies one of the biggest false assumptions in thinking about big data.

The glib answer is that if it weren’t for the humans, this premise might actually happen. Let me explain.

When I think of big data, an analogy comes to mind. Imagine you were living decades ago, at the time of the invention of the automobile. Assume further that all of a sudden, all the dirt roads and tracks were paved. What would you do? Where would you go?

The obvious answer is that either you would stay local or you would be paralyzed in the face of the enormous number of opportunities. In fact, you would need a road map, so you could see how to get to different places. Beyond that, you would need to understand the nature of the destinations so you could decide where to go, since you couldn’t get everywhere in one lifetime.

Further, if you had a number of different drivers, it is likely that each would head in a different direction, since each would go after the goal that he or she thought best. If these drivers had to co-ordinate, like the managers in a company, what would happen? The result would be chaos.

The problem here is two-fold. First, a company can’t do everything, because it takes significant time and resources to manage the change required to harvest any IT-based initiative. And second, the initiatives have to be co-ordinated and focused on the right long-term strategic goals to be effective. If the availability of big data encourages a massive flock of independent tactical initiatives, it will do more harm than good.

The problem with low-hanging fruit

This raises an important problem. Managers have an almost overwhelming tendency to focus first on opportunities that are near to hand, and have quick, visible payoff. Sometimes these are called “low-hanging fruit”.

The problem is that these relatively small, parochial projects will absorb the organisation’s resources and capability to change, even while they give the illusion of progress. The huge opportunity cost is losing the opportunity and ability to focus on the really important initiatives with the really big long-term payoffs.

Think of it this way. The analogy breaks down because the big money is not in harvesting fruit more efficiently, but rather in changing the location of the orchard and the type of trees you plant.

This dissipation of effort, with its focus on a large number of small, incremental projects – rather than on the smallest number of game-changing, high-payoff initiatives – is the ultimate danger of big data.

Keys to success

Big data offers great opportunities – and huge dangers. How can you navigate toward the benefits while avoiding the hazards?

The key is management insight and discipline.

The true promise of big data is to make your company better – not just to make parts of it more efficient. To accomplish this, you need one part technology to nine parts vision and great management.