NPS (11) – 11th in the series – The law of small numbers
Continuing the discussion on Daniel Kahneman’s System 1 (intuitive fast thinking) and System 2 (rational slow thinking), some curious things happen when people study surprising data. As I write this article in June 2019 there are media reports in France of exactly what I am about to describe, a surprising ‘cluster’ of cancer cases in a relatively small area. This just reinforces the message I wrote a couple of years ago (see below) about similar American and Swiss studies.
Kidney cancer in the United States
Daniel Kahneman asked his readers to consider the example of the data for the incidence of kidney cancer in the 3,141 counties of the United States. “The counties in which the incidence of kidney cancer is lowest are mostly rural, sparsely populated, and located in traditional Republican states in the Midwest, the South and the West. What do you make of this?” I have recently seen a similar type of study here in Switzerland. Our rational minds immediately get busy trying to analyze why the geography or being Republican might cause cancer. For the Swiss study, I worked out that it was probably the healthy rural lifestyles that provided the benefit. Fresh air, fresh vegetables, plenty of exercise. How could that not be the reason?
What about the worst places?
However, the counties with the highest incidence of kidney cancer had exactly the same profile. The somewhat similar Swiss study concentrated on this aspect. The TV news report interviewed a person who talked about the use of pesticides in agriculture, various treatments in the vineyards and so on. Seems perfectly logical to think that the rural population would be far more exposed to them. In addition, the rural populations are poorer and probably have more difficulty getting high-quality health care. There were discussions of ‘cancer clusters’ in areas with the highest rates.
The key words were…
In reality, the most important words in the description of the counties with high rates of kidney cancer were ‘sparsely populated’. Counties in the United States have populations that range from under 500 to over 10 million. According to www.kidneycancer.org, the lifetime likelihood of anyone contracting kidney cancer is 1 in 63. In a small county, it does not take many cases for the percentage to be unusually high. The general randomness of cancer over the entire country means that there will of course be some small counties, with quite small numbers of cancers, but with relatively high or relatively low incidence percentages. The extremes are what grab the headlines. Next time you see a story about something like scientists studying why people in some small village live such long lives, treat it with suspicion. A similar village just down the road may have a population with an unusually short average lifespan. That’s statistics!
Use care when presenting your data
Customer experience data commonly has small numbers. If, for example, you are presenting NPS scores for 40 countries, it is very likely that both the highest and lowest scores will be from countries with relatively small sample sizes. This has little to do with statistical significance. If you are looking at enough items, each with 90% confidence, one in ten is just going to be a relatively poor reflection of reality. I used to present NPS benchmark scores for 45 enterprise software companies each quarter. It was very common to see a new small company top the list for a single quarter. If trends from prior quarters did not support this, I told my audience we should wait until the following quarter to react. If the prior quarterly trends did support it (as was the case with ServiceNow’s gradual move from awful NPS scores to great ones), I proposed immediate action.
Nobel laureates by per capita
Another entertaining example and research question would be the following: Why do the Faroe Islands have the highest number of Nobel Prizes by capita? Let’s send out a research team to study their education system and copy it everywhere! In fact, the reason is that they have had one Nobel, and have a population of just over 48,000.
Regression to the mean
If you have loads of data and a consistent trend for something, such as transactional NPS scores for a call center, be careful how you react when you see a substantial short-term change from the trend. To give an analogy, behavioral psychologists generally agree that punishing children for unusually bad behavior does not work, and can even have negative long-term effects. (These psychologists believe that rewarding good behavior works far better.) However, most parents believe punishment does work, and works well. The reason for this belief is regression to the mean. The child has a certain average behavior, trending in a certain direction. They then do something that is unacceptable, and outside the prior average. The parent punishes them and the behavior returns to the prior average. That would have happened in any case, even without any intervention. That is, it will regress to the mean. The only time you have act in new ways is when the trend has actually changed (as it does in fact do if you reward good behavior over time). If you are constantly seen to reward or punish people for what are really single random deviations from their normal trend, you will lose credibility.
Looking forward
Next time I will discuss whether you should adjust or ‘weight’ your research results in any way before you report them. It’s a tricky subject without simple answers. I hope I can reduce the complexity for you. And that post will also include a ‘true or false’ learning test.
As is often the case, the above is a slightly-edited version of a chapter in one of our books; in this case Net Promoter – Implement the System All of our books are available in paperback and Kindle formats from Amazon stores worldwide, and from your better book retailers.
