The geek shall inherit

Few companies have done more over the past 20 years – and frankly ever – to turn Big Data into insights that inspire evidence-based action than dunnhumby. Before they helped Tesco create its Clubcard and methodically predict customer shopping behaviour, Sainsbury’s was the UK’s biggest supermarket.

Within a few years, the insights developed by dunnhumby powered Tesco to become number one by a country mile – at one point nearly twice the size of Sainsbury’s, taking one in every seven pounds on Britain’s high streets. And though discount retailers and changing shopping patterns have seen Tesco going backwards in recent years, it’s fair to say that Sir Terry Leahy’s stellar achievements at the helm of Tesco were in no small part the result of its partnership with – and eventual acquisition of – dunnhumby.

To get the inside track on how business develops and deploys insight at industrial scale, I spoke recently to Giles Pavey, Chief Data Scientist at dunnhumby. Giles is responsible for insights, analytics and research at a firm now active in 75 markets and with 3,000 employees. “I’ve had lots of job titles over the years here. I’ve been head of Innovation and Retail Solutions, and started off in 1998 as Head of Analysis. My data science team is focused on extracting insights from big data, in real time, to predict future behaviour.”

“I’m very taken by the scientific method,” says Pavey. “I like formulating hypotheses that seem plausible from apparent patterns in the data. We then test these hypotheses to see whether our hunches really exist and, if we find that they do, we quantify the implications. This then enables you to put forward a business case that you couldn’t have done before you’d analysed the data.

“That’s how we first developed and quantified different behavioural groups within the customer base at Tesco – groups that sounded very believable, and people intuitively knew were there: the Price Sensitive, the Health Conscious, the Traditional, the Upmarket, the Convenience Shopper and the Mainstream. The insight wasn’t so much that these groups exist, it was how many there were in each group, for any given store, for any given product, for any given time of day. Tesco could then tailor very specific promotions based on a forensic understanding of these groups.”

So Pavey’s approach is very much more about quantifying the intuitive, the instinctual, the gut reactions that experienced business people have, rather than unearthing genuinely shocking or counter-intuitive insights. Instincts, intuitions and experience are great ways to generate hypotheses; insights provide the proof as to whether they were valid or not.

“Occasionally, it’s true, we come up with insights that carry so much weight that they can completely change the course of a business. We worked with major US DIY chain and we showed them that their business was three times more dependent on professional construction companies than they had thought, and that had a major impact on how they should sell to the trade. But transformational insights like don’t come along every day. More often, insights are good sense, quantified.”

Stacking the odds in your favour

In terms of increasing the likelihood that you’ll generate more and more meaningful insights, Pavey is a strong advocate of a human-tech approach, combining the best in big data analytics that modern computer and data science can offer, with the knowledge, experience and judgment of human analysts. He cites the fact that chess tournaments between grandmasters, regular players and a simple PC chess emulator, and mainframe computers playing by themselves are routinely won by the human-tech pairings.

“People are brilliant at identifying apparent patterns in data sets, developing hypotheses based on these patterns, and working with ambiguity. Combine these traits with a computer’s tireless willingness to crunch data, and you have a winning formula.”

That’s quite different from – for example – Google’s apparent approach, which suggests you should just throw better and better algorithms at more and more data and you’ll get better and better answers. For Pavey, it’s the unique combination of data science, computer science and mathematical science – all overlaid by a contextual understanding – that produces the most genuinely insightful results.

“I’m also a big believer in using experts to build the hypotheses you choose to test. It’s helpful to stack your team with individuals with subject-matter expertise, often developed from years of experience. They’re much better at sniffing out red herrings and knowing when a potential insight is inherently plausible or not.”

Empathy – seeing the world through the eyes of those whose behaviour you’re looking to predict and influence – is also an incredibly important characteristic in those looking to develop meaningful insights. This includes forced empathy where natural empathy does not exist.

“A lot of marketers definitely have a real job on their hands to have authentic empathy with customers who are not from the same demographic as they are. Genuine insights generated with and about the target audience have a key role to play for any organisation.” This applies to a supermarket looking to sell a new range of ready meals, a charity after donations, or a would-be lothario looking to ingratiate himself with a new partner. Understanding the motivations and drivers of those we seek to influence –understanding of how the context looks to them – stacks the cards in our favour.


One major caveat for Pavey is seeing meaning in apparent patterns in data where there is no pattern – what he calls “seeing castles in the clouds”. He explains: “Pattern identification is one of humankind’s greatest skills, but it’s also one of our greatest weaknesses. Pattern recognition and retrospective delusion are definitely things we need to be wary of and avoid. You need to be careful that there’s enough signal in the noise, and you’re not simply overstating a pattern that isn’t really there.”

Pavey also warns about taking a strictly “Einsteinian” approach to building business cases, without considering the proper impact of context. It may be true that doing the same thing again and again and expecting different results is a meaningful definition of madness. But just because an approach or a business model has failed in the past, that doesn’t mean it necessarily will in the future. Aldi and Kwiksave deployed essentially the same model – sell a limited range of house brand products and do so cheaply – and the former is thriving, while the latter is history. “It’s all about context and choosing the right moment in time, the right context for an approach to work. This is true of business or human behaviour, not an immutable set of laws for all time.”

The recent explosion in volumes of data – the Big Data Revolution – has not necessarily provided us with more insights, but rather the opportunity to find new insights. “There’s a critical signal to noise dimension here too,” says Pavey. “There are many data sets that have huge numbers of observations, but from which it’s incredibly hard to draw meaningful conclusions. Twitter is an obvious example – can you really be sure you’re extracting a changing view of a brand or political situation from an automated sentiment analysis? I’m not so sure.

“Also, the quality and speed of Google search sets very high expectations in people’s minds as to what they should expect from data. Google is brilliant, but it started with no legacy. So organisations with a history that predates the internet have to work out how to bring more insights into that organisation to meet the expectations of management, stakeholders and customers.”

Pavey is a valuable and rare commodity in the insights business. Instinctive and intuitive based on years of experience; insightful and able to read the runes thanks to his evidence-based, hypothesis-testing approach.

Giles’ insightful reading list

Daniel Kahneman, Thinking Fast and Slow

Ian Ayers, Super Crunchers: How Anything Can Be Predicted

Richard Thaler and Cass Sunstein, Nudge

Vijay Govinderarjun, The Other Side of Innovation

Paul Dolan, Happiness By Design: Finding Purpose & Pleasure in Everyday Life

A version of this interview will appear in Sam Knowles forthcoming book, How to be Insightful, in 2016.