NPS (12) – 12th in the series – Should I weight my survey results?

This is the 12th episode in my series on the Net Promoter Score and the Net Promoter System. There is a learning check / quiz at the end of this post.

Suppose you have performed an NPS survey and feel the data set is not representative of your customers. What should you do?

A common issue, rarely addressed

Throughout my customer experience career, I have never adjusted my data sets before or after analyzing them. This has always made me somewhat uncomfortable. Think about these examples:

• Your competitive benchmark survey provider uses a panel of people in countries around the world. You do 60% of your business in the USA, 15% in Canada, 10% in the UK, and smaller percentages in over thirty countries. The survey provider panel is 40% in the USA, 10% in Canada, 15% in the UK, 10% each in France, Germany and Australia, and 5% in another twenty countries. Should you take the survey provider data and re-weight it to reflect the profile of your company? At HP, we did not have any data for Japan the year of the earthquake and tsunami. That had a material impact on the worldwide averages, but we did not call it out in the reports. Was this some subconscious decision because leaving out the Japanese data made the overall results slightly better?
• You run project completion surveys for your consulting company. Customers pay vastly different prices for different types of projects, varying by a factor of 100:1. Should you weight the survey responses by revenue, giving 100 times the weight to a response for an expensive project?
• Your relationship survey covers two CxOs, three other decision-makers, five decision-influencers and ten end users. Should you weight the survey so the responses from CxOs count more?
• You run transactional surveys after calls are closed in your service center. You code calls by their level of severity, into critical, high, medium and low. Should critical calls have more weight in your survey? If 10% of your calls are in the critical category, but only 5% of your survey responses, should you double the weight of those survey responses?

No consensus, major consequences

There is no consensus on the topic. The reason is that it is difficult to take an unbiased decision when adjusting data sets. This is of course not just the case for customer experience data. The New York Times published an article on the subject in September 2016, while the Trump-Clinton contest was in its final stages. The Upshot gave 867 voter intention responses to four different highly respected pollsters and studied the data themselves. Each person or team studying the data adjusted it using their own methodology. Adjustments were made to reflect race, gender and age to reflect the population of Florida, source of the data. Further adjustments were made to reflect likely voting intention. For example, if Hispanics are less likely to vote than Whites, some adjusted the poll results to reflect it. Others considered whether the people polled said they had voted in prior elections. The variations in methodology produced a greater variation in results than would be expected only from sampling error. The difference in views varied by five points. One showed Trump winning by one point; another showed Clinton winning by three points. History tells us that Clinton won the popular vote by two points.

Conclusion

I am not against adjusting an analysis to force data to better match your customer profile. I am against doing such adjustments differently each time, or having different principles for different types of surveys. Either you adjust to reflect customer revenue profiles, geography and other factors, or you don’t. You need to be consistent. If you have multiple businesses in your company, the same principles should be used by all, all the time. If you use a benchmark survey provider that customizes a solution specifically for you, you should agree response quotas in your most important customer dimensions. Do it once, then avoid the temptation to change it for a reasonable period, say three years or so.

Learning check

Decide whether the following statements are true or false. Answers at the bottom of the page. The questions refer to topics that I have covered at some point between the third article in the series and this twelfth article.

1. If your competitive benchmark NPS number is better than that of your main competitor, you will take market share from them moving forward.
2. Good NPS scores start at 30, great scores at 40, and you are world-class at 50.
3. Consistent trends in numbers that are based on small sample sizes are always useful. You do not need sample sizes for all competitors to be the same, for example.
4. All customers are considered equal by benchmark survey suppliers, so customers who spend a lot have the same weight as those who do not spend much.

Looking forward

Next time I will start the general topic of survey design with an article about when and whether to survey.

 

As is often the case, the above is a slightly-edited version of a chapter in one of our books; in this case Net Promoter – Implement the System All of our books are available in paperback and Kindle formats from Amazon stores worldwide, and from your better book retailers.

Learning check answers

1. False. What matters is the trend relative to competition. If your number is higher and has stayed flat, but your competitor’s number has improved, they will take share from you in the future.
2. False. A good NPS score is one whose trend is better than the trend of your main competitor. A world-class score is one that is better than the competition and improving faster. The absolute number has no importance.
3. True. You may want to treat a single outlier result as suspicious if the sample size is smaller than for your other competitors or countries, for example. However, if you see similar numbers in sequential sampling periods, you should trust the trend.
4. True, though the survey vendors also supply demographic data so you can do your own weighting if you so wish. However, when vendors such as Temkin publish tables on their websites, there is no weighting.