Elephant vs rider: why research gets it wrong

Two companies. Two well-intended product changes. Two expensive and embarrassing examples of how failing to understand real-world behaviour can backfire.

Glad I’m not Milo

Milo and Glad Wrap have been in the news this year for the wrong reasons.

Glad introduced a new method of cutting their cling wrap, moving the cutter from an exposed blade on the box to a lid-mounted cutter.

 Nestle changed the recipe of the New Zealand version of their malted-chocolate drink Milo, removing vanilla and changing the mix of vitamins.

In both cases, people flipped out and called for boycotts, leaving the companies scrambling to placate consumers and shore-up their market share.

As a former product manager I felt their pain.

Making changes to a product is no easy thing, and would have followed extensive business justification and research.

A spokesperson for Glad Wrap had this to say: “Before making these changes, Glad completed rigorous and extensive in-home research in Australia. The results were overwhelmingly positive favouring the changes including the movement of the cutter bar to the lid, with the safety aspect of this front-of-mind. In fact, more than 60% of those Australians, who participated in the research, preferred the improved product overall.”

So why is there such a chasm between what businesses think their consumers want and what they actually desire?

It seems businesses are doing one of three things:

• getting the wrong kind of research
• ignoring what the research is really telling them
• cherry picking research to support a decision they want to make

The wrong kind of research

If you are relying on what consumers tell you they want, don’t. 

It doesn’t work for predicting the outcome of elections and it doesn’t work for product development. Examples of this ‘say vs do’ gap include:

• 60% of US voters claiming they’ll turn out to vote but only 40% do
• 50% of Victorians say they have a healthy diet but only 7% eat their veggies
• Americans claim they love Guinness but actually buy Bud Light

Asking people in focus groups, surveys or interviews to tell you why they did or would do something engages our System Two rationalising, slow-thinking, cold state brain (often known as the ‘Rider’). This will give you a well-intentioned justification of past behaviour or well-intentioned rationalisation of future behaviour. 

While these answers look great in a research report and can be used as handy stats to quote in PR, the problem is System Two thinking is unlikely to be the one making the decision to use or buy your product.

System One, our fast-thinking, impulsive, habitual, hot-state brain (also known as the ‘Elephant’) is the one that makes most of the everyday decisions.

Unfortunately you can’t easily ask System One what is likely to happen in the future because merely asking the question prompts System Two to take the wheel.

Behavioural Economics helps you resolve this by providing:

1. A framework of predictable factors that influence behaviour so you can anticipate what people are likely to do in a given set of circumstances.
2. A system of creating experiments in observable rather than self-reported behaviour so you don’t have to ask, you can watch.

Ignoring what the research says

When you are a product manager your role is to make the product better, and as the expert you spend a lot of time and energy thinking about it. In fact, more time than your consumer ever would.

And that’s a problem because you develop ideas about what changes you should make, then commission research about those changes and, thanks to confirmation bias, look for results that support rather than contradict your views.

Compounding this, the effort you have expended heightens your sense of ownership, making walking away mighty hard due to sunk cost – your desire to hold out for a return on the investment you’ve made. 

How do you stop yourself from ignoring what research is actually telling you? Structure the research to contradict your hypothesis, looking for reasons to not do what you want to do. Then make sure the decision not to proceed with a change is valid so you won’t feel the pain of wasted effort.

Cherry picking research

Where ignoring research can, in part, happen subconsciously thanks to confirmation bias, cherry picking is more deliberate. This is about trawling the results to find anything that justifies a decision already taken, and sadly for consumers, researchers and business culture, it’s all too common.  I’ve done it in the past and I’m not proud.

How to prevent cherry picking? Too big a topic to tackle here because it gets to the politics of an organisation and its culture – the social norms at play and people’s fear of failure. As a leader you can, at the very least, interrogate the research and ask to see the full story, reminding yourself that in today’s world of social media the claims you go to market with are subject to relentless scrutiny.

Who you need to understand best

To avoid the situation Milo and Glad Wrap found themselves in, you need to know two people best:

1. Know thy consumer
2. Know thyself

To ‘know thy consumer’ make sure your research includes a behavioural analysis, drawing on observed as well as (or instead of) self-reported behaviour. (See Why you should ignore what customers say they want).

And to ‘know thyself’, create safeguards to compensate for you and your team’s biases and decision-making heuristics, because you otherwise you will be setting yourself up for failure.

Say vs do gap examples:

• The US election polling stat
• Eating habits of Victorians stats
• Guinness vs Bud Light

Bri Williams runs People Patterns, a consultancy specialising in the application of behavioural economics to everyday business issues.

Follow SmartCompany on Facebook, LinkedIn and Twitter.