Big data, little innovation: Why big data is overhyped

IT infrastructure spending has had a rough patch over the past few years, with many large Australian businesses slowing down their investments in new technologies. Cloud services have made inroads into many enterprise and government organisations precisely because of this drying up of internal investment in skills and infrastructure.

‘Big data’ is one of the latest technology bandwagons. The Australian federal government has even conducted a round of consultation around the potential use of big data within federal agencies. Big data is essentially a philosophy, where an organisation systematically performs widespread continuous data collection and analysis, measuring and examining actions across an organisation.

Many technology firms and analysts are spruiking big data as a wonder technology that will make or break business fortunes. To describe big data as overhyped is understating the situation.

A media frenzy has already emerged, as it is easy for journalists to draw links between the use of big data within businesses, the now widespread collection and sharing of sensitive personal information on websites like Facebook, and the proliferation of smartphones that can surreptitiously measure and report almost every aspect of our lives.

So it should hardly be surprising that the Australian government feels the need to consult on this topic. There are existing legal protections for the privacy of individual citizens, and restrictions covering the sharing of information between government departments. The timing is, however, quite ironic, coming within days of technology consulting firm Gartner finally proclaiming that the big data technology emperor has no clothes.

Above: Big data marketing in 1966

While there are clearly some elements of new technology to be found amongst the latest big data hype, even the most casual look back through archives of computer marketing material from the 1950s and 1960s reveals a simple truth. Large scale data analysis (big data) has always been one of the primary selling points used to justify investment in business computers. Even the ill-fated high profile release of the Apple III in 1980 used data analysis as a primary selling point.

At many large businesses, the board and senior executive teams started investing heavily in these visionary big data-style analysis capabilities in the 1980s. A particularly interesting example of big data hype repeating itself is this 1986 marketing brochure for a pattern recognition computer that “scans whole databases” and “handles words and pictures as easily as numbers”. The specific technologies being spruiked may differ, but the claims are extremely similar to those being touted for the latest big data technologies in 2013 by vendors such as IBM, Oracle, and EMC.

These strategic concepts are not new at an executive level, and the business outcomes being touted by today’s big data spruikers promise little more than the technology visions of the 80s. At a board level, every time a CIO goes back to the piggy bank for a new investment to solve the same old problem, the focus immediately shifts to the CIO’s credibility.

Above: Big Data marketing in 1986

At a very fundamental level, all big data initiatives are supposed to uncover a stream of fact-based insights to inspire new efficiencies, improved business processes, or product innovations. In their most recent research into Australian business innovation, the Australian Bureau of Statistics found that only 16.4% of Australian businesses introduced new operational processes compared to the year before.

In the same period, 17.3% of businesses introduced new products, and a total of 39.1% of all Australian businesses claimed to have made an innovation or process change of any type.

The report found that the top Australian industries carrying out innovation were wholesale trade (51%), retail (42%), and professional services (40%). This should not be surprising, as these industries are all facing intense competitive pressures caused by the widespread adoption of eCommerce, and changes in the Australian dollar exchange rate.

Research and development is of course a separate issue, with the ABS reporting that private sector R&D is currently 1.3% of GDP. While the mining industry gets a lot of media attention, the largest sector for R&D investment has actually been manufacturing: 80% of all private sector R&D dollars spent in Australia came from businesses in manufacturing, mining, financial services, and professional services. The government also spends very significantly on direct R&D, with most of that R&D spend going directly to universities.

ABS: Proportion of Innovation Active Businesses reporting lack of funds as an impediment to innovation 2010-2011

By dealing with big data as a technology issue, CIOs and the organisations they work for are setting themselves up for failure.

When examining impediments to innovation, the most recent ABS report found that amongst companies that actively innovated, lack of funds was the primary roadblock to innovation, followed by lack of skilled labour.

Data insights need to be translated into actionable change that is implemented, resulting in modified behaviour and improved operations.

In an idealised scenario it becomes a continuous process of iterative innovation, sparked by data-based insights and experimentation. The differences between the 1950s and 2013 simply boil down to the sources of the data points, the level of automation, and the amount of information that can be realistically stored and analysed.

In my experience, most large organisations struggle to build and maintain a culture of innovation, primarily because they have complex supply chains, well-defined business processes, and a large hierarchy of stakeholders. Put bluntly, change usually takes a lot of time and leadership attention.

For an insight to be useful, it must be rapidly turned into change. Large organisations that successfully innovate often choose to focus on small discrete projects, which are limited within a single division, or focused on a vertical slice of workflow. By limiting the scope, risks can be better understood, more focus placed on customer needs, and decision-making processes sped up.

This is precisely why decades of data analytics projects have been focused in narrow areas of interest within organisations. It was not a technical limitation, but a consequence of the way people interact within large organisations.

Leaders need to consider how much capacity there is for innovation within their organisations. While the internal workplace culture of the organisation can have a huge impact on innovation, supplier contracts and technology infrastructure may also limit the ability to rapidly make meaningful changes.

The returns from big data projects can be expected to flow from the innovations and minor changes that are actually implemented. Fostering a culture of innovation within an organisation is a complex long-term project. If there is little opportunity to quickly adapt products, change workflows, or respond to a weekly flow of insights, then there will be little return from a big data investment.