CX

Survey dramatic interpretation

NPS (13) – 13th in the series – Whether and when to survey

/ , , , , , / , , , , , /

This is the 13th episode in my series on the Net Promoter Score and the Net Promoter System. This one is a bit longer than the average and covers some important topics, including whether you should bother surveying your customers at all. If you have missed some prior articles in the series you will find them on our blog page here.

The first step in designing surveys is being selective about what research you are going to do. Make sure you are going to provide customers with more value than you will extract from them.

Sanity check on whether you should gather feedback at all

Here are some considerations that should help you decide where to start:

  1. Are you going to use any new input you get? If you have already established your top five improvement priorities, have staffed them, are implementing, and can’t add a new project in the next six months, don’t do any surveys other than those that provide progress feedback on your improvement work.
  2. Taking a software example, if you are planning a new major release and want user input on what should be in it, ask them in a formal way. After 30 years in and out of the software business, as a leader and as a user, I have never once been asked for formal input about what should be in the next version. Product managers generally feel they know what users want, based on informal and anecdotal conversations, rather than research. This of course applies beyond software.
  3. Not every customer touchpoint is important. Use your overall brand or benchmark surveys to establish which touchpoints customers mention as competitive advantages or disadvantages, and don’t survey anyone about any of the others. To pick a B2B example, surveying people about the friendliness, competence and efficiency of the person who handles a query about the accuracy of an invoice is certain to be a waste of resources that could better be spent elsewhere.

Timing is everything

Survey timing is an important part of research reliability. Seeking feedback about things that are in the distant past is unlikely to be of much value for example. Here are some appropriate times to consider collecting customer input, by type of survey. What follows presupposes that you are going to use the feedback to make improvements for your customers. If not, there is no good timing.

Product surveys part 1 – Unboxing and installing your product

Here we are talking about an individual product, not a generic purchase or other touchpoint experience that applies to all your products. Some products are sophisticated (the marketing word for complex) to install. Others are not. If your product takes time and effort to unbox or download, install and use for the first time, it is worthwhile understanding what your customers think of the experience and how it can be improved. The research needs to be done immediately after the experience. Waiting a week or more will mean losing the positive and negative emotional impact.

If yours is a technology product, I suggest sending the survey request or phoning the customer within 48 hours of delivery. While this will be easy to do in a direct business that sells to consumers, it is more difficult in other circumstances. You want to survey the end user of the product, and they may not be the buyer in a B2B situation, so you might not know whom to contact.

If you use resellers, they may not want to provide the customer data, and you should offer to run the survey on their behalf, providing them the data for initial screening. If yours is a software product, the installation routine itself should generate the survey request 24 hours after the installation takes place. Requiring or at least suggesting product registration on a website will give you additional sources of end-user contact details.

Product surveys part 2 – Usage

In most situations, you won’t hear anything from a customer who does not have any problems. Unfortunately, the reason for not having problems could be that the customer no longer uses your product. Perhaps they never started using it in the first place. You need to understand the views of customers who have been using your product for some time, and I suggest six months after purchase as a reasonable target for your research. The purpose, as explained to the customer, should be to get feedback for improving the product itself, as well as future products. In my own experience of software product usage surveys, it is common to get suggestions for additional features that the product already has, but the customer has not been able to find. Usage surveys have the same challenge as installation surveys, meaning you may not know whom to contact.

SaaS (Cloud) software is a completely different in that you do know who is using your software and indeed how much they use it. If a customer who has been using your software suddenly stops, contact them immediately. SaaS software usually allows you to see what functions a customer uses. If you can see that they are not using something they really should be using, you can send them a link to a training video or send over someone who can talk them through it.

Service contracts

The ideal time to get feedback on a service contract is a little before the customer is considering renewing it. This is a bit more subtle than it might seem. If you have a large multi-year contract (for example to provide security services to a company) your competitors will probably know about it and will start to contact your customer well before the contract renewal date. Since your competitors will be saying how wonderful they are, and how awful you are, you need to have acted well before this point. I suggest an annual survey as a minimum for multi-year contracts, taking place half way through each year. If you have a single-year contract, I suggest doing the research about three months in, so that all improvement suggestions can be implemented before the contract comes up for renewal consideration, probably about nine months in.

Hybrid product / service

An airline flight is a hybrid product & service. This sort of offer should be researched within 24 hours of completion. Most airlines seem to miss the point. Some give you the survey while you are still in the air, not knowing essential things like whether your luggage will show up on the carousel. Others send you the survey after a week or two. Frequent fliers will have completely forgotten about the experience by then or will confuse it with another flight. This happened to me regularly, as I have been taking over 100 flights per year for nearly as long as I can remember. What did I think of my flight from Atlanta to Paris two weeks ago? Who knows? Since product and service surveys are not mentioned in The Ultimate Question 2.0, this may be a good time to reinforce the definition. An entire flight experience, from awareness, to reservation, to the actual trip itself, involves a sequence of touchpoints. Each touchpoint could be studied by a transactional survey. The airline offers many different products and services, the sum of which might be studied in their brand survey.

Relationship surveys and Brand surveys

Twice a year is a good frequency for your most important customer relationships and may work best for your brand surveys. You may like to run them both on a continuous basis so you can see the effects of things like changes in the economy and your financial and product announcements. This would mean, for example, sampling one sixth of your customers every month. Remember that you need to report results and improvement projects to customers as well as to your own team.

Transactional surveys

While it might seem obvious that they should happen shortly after the transaction takes place, it is surprising how rarely this happens. All too often, I get pop-up windows when I have just started an online transaction. I dismiss the window, and then don’t think about it anymore. For the simplest transactions, surveying just after they are complete seems appropriate. For more complex things, like telephone support for a technology product, it seems appropriate to wait for 24 hours, as it may take a little time for the customer to know whether the issue has been completely resolved.

Conclusion

I hope this helps you to decide whether you should survey customers at all, and if you decide to do so, when it should be done. It is tricky to get it right.

Looking forward

Next time I will cover response rates and statistical significance.

 

As is often the case, the above is a slightly-edited version of a chapter in one of our books; in this case Net Promoter – Implement the System All of our books are available in paperback and Kindle formats from Amazon stores worldwide, and from your better book retailers.

 

Scales for article on weighting research results

NPS (12) – 12th in the series – Should I weight my survey results?

/ , , , , , / , , , , , /

This is the 12th episode in my series on the Net Promoter Score and the Net Promoter System. There is a learning check / quiz at the end of this post.

Suppose you have performed an NPS survey and feel the data set is not representative of your customers. What should you do?

A common issue, rarely addressed

Throughout my customer experience career, I have never adjusted my data sets before or after analyzing them. This has always made me somewhat uncomfortable. Think about these examples:

  • Your competitive benchmark survey provider uses a panel of people in countries around the world. You do 60% of your business in the USA, 15% in Canada, 10% in the UK, and smaller percentages in over thirty countries. The survey provider panel is 40% in the USA, 10% in Canada, 15% in the UK, 10% each in France, Germany and Australia, and 5% in another twenty countries. Should you take the survey provider data and re-weight it to reflect the profile of your company? At HP, we did not have any data for Japan the year of the earthquake and tsunami. That had a material impact on the worldwide averages, but we did not call it out in the reports. Was this some subconscious decision because leaving out the Japanese data made the overall results slightly better?
  • You run project completion surveys for your consulting company. Customers pay vastly different prices for different types of projects, varying by a factor of 100:1. Should you weight the survey responses by revenue, giving 100 times the weight to a response for an expensive project?
  • Your relationship survey covers two CxOs, three other decision-makers, five decision-influencers and ten end users. Should you weight the survey so the responses from CxOs count more?
  • You run transactional surveys after calls are closed in your service center. You code calls by their level of severity, into critical, high, medium and low. Should critical calls have more weight in your survey? If 10% of your calls are in the critical category, but only 5% of your survey responses, should you double the weight of those survey responses?

No consensus, major consequences

There is no consensus on the topic. The reason is that it is difficult to take an unbiased decision when adjusting data sets. This is of course not just the case for customer experience data. The New York Times published an article on the subject in September 2016, while the Trump-Clinton contest was in its final stages. The Upshot gave 867 voter intention responses to four different highly respected pollsters and studied the data themselves. Each person or team studying the data adjusted it using their own methodology. Adjustments were made to reflect race, gender and age to reflect the population of Florida, source of the data. Further adjustments were made to reflect likely voting intention. For example, if Hispanics are less likely to vote than Whites, some adjusted the poll results to reflect it. Others considered whether the people polled said they had voted in prior elections. The variations in methodology produced a greater variation in results than would be expected only from sampling error. The difference in views varied by five points. One showed Trump winning by one point; another showed Clinton winning by three points. History tells us that Clinton won the popular vote by two points.

Conclusion

I am not against adjusting an analysis to force data to better match your customer profile. I am against doing such adjustments differently each time, or having different principles for different types of surveys. Either you adjust to reflect customer revenue profiles, geography and other factors, or you don’t. You need to be consistent. If you have multiple businesses in your company, the same principles should be used by all, all the time. If you use a benchmark survey provider that customizes a solution specifically for you, you should agree response quotas in your most important customer dimensions. Do it once, then avoid the temptation to change it for a reasonable period, say three years or so.

Learning check

Decide whether the following statements are true or false. Answers at the bottom of the page. The questions refer to topics that I have covered at some point between the third article in the series and this twelfth article.

  1. If your competitive benchmark NPS number is better than that of your main competitor, you will take market share from them moving forward.
  2. Good NPS scores start at 30, great scores at 40, and you are world-class at 50.
  3. Consistent trends in numbers that are based on small sample sizes are always useful. You do not need sample sizes for all competitors to be the same, for example.
  4. All customers are considered equal by benchmark survey suppliers, so customers who spend a lot have the same weight as those who do not spend much.

Looking forward

Next time I will start the general topic of survey design with an article about when and whether to survey.

 

As is often the case, the above is a slightly-edited version of a chapter in one of our books; in this case Net Promoter – Implement the System All of our books are available in paperback and Kindle formats from Amazon stores worldwide, and from your better book retailers.

Learning check answers

  1. False. What matters is the trend relative to competition. If your number is higher and has stayed flat, but your competitor’s number has improved, they will take share from you in the future.
  2. False. A good NPS score is one whose trend is better than the trend of your main competitor. A world-class score is one that is better than the competition and improving faster. The absolute number has no importance.
  3. True. You may want to treat a single outlier result as suspicious if the sample size is smaller than for your other competitors or countries, for example. However, if you see similar numbers in sequential sampling periods, you should trust the trend.
  4. True, though the survey vendors also supply demographic data so you can do your own weighting if you so wish. However, when vendors such as Temkin publish tables on their websites, there is no weighting.
Law of small numbers illustration for blog

NPS (11) – 11th in the series – The law of small numbers

/ , , , , , / , , , , , , /

Continuing the discussion on Daniel Kahneman’s System 1 (intuitive fast thinking) and System 2 (rational slow thinking), some curious things happen when people study surprising data. As I write this article in June 2019 there are media reports in France of exactly what I am about to describe, a surprising ‘cluster’ of cancer cases in a relatively small area. This just reinforces the message I wrote a couple of years ago (see below) about similar American and Swiss studies.

Kidney cancer in the United States

Daniel Kahneman asked his readers to consider the example of the data for the incidence of kidney cancer in the 3,141 counties of the United States. “The counties in which the incidence of kidney cancer is lowest are mostly rural, sparsely populated, and located in traditional Republican states in the Midwest, the South and the West. What do you make of this?” I have recently seen a similar type of study here in Switzerland. Our rational minds immediately get busy trying to analyze why the geography or being Republican might cause cancer. For the Swiss study, I worked out that it was probably the healthy rural lifestyles that provided the benefit. Fresh air, fresh vegetables, plenty of exercise. How could that not be the reason?

What about the worst places?

However, the counties with the highest incidence of kidney cancer had exactly the same profile. The somewhat similar Swiss study concentrated on this aspect. The TV news report interviewed a person who talked about the use of pesticides in agriculture, various treatments in the vineyards and so on. Seems perfectly logical to think that the rural population would be far more exposed to them. In addition, the rural populations are poorer and probably have more difficulty getting high-quality health care. There were discussions of ‘cancer clusters’ in areas with the highest rates.

The key words were…

In reality, the most important words in the description of the counties with high rates of kidney cancer were ‘sparsely populated’. Counties in the United States have populations that range from under 500 to over 10 million. According to www.kidneycancer.org, the lifetime likelihood of anyone contracting kidney cancer is 1 in 63. In a small county, it does not take many cases for the percentage to be unusually high. The general randomness of cancer over the entire country means that there will of course be some small counties, with quite small numbers of cancers, but with relatively high or relatively low incidence percentages. The extremes are what grab the headlines. Next time you see a story about something like scientists studying why people in some small village live such long lives, treat it with suspicion. A similar village just down the road may have a population with an unusually short average lifespan. That’s statistics!

Use care when presenting your data

Customer experience data commonly has small numbers. If, for example, you are presenting NPS scores for 40 countries, it is very likely that both the highest and lowest scores will be from countries with relatively small sample sizes. This has little to do with statistical significance. If you are looking at enough items, each with 90% confidence, one in ten is just going to be a relatively poor reflection of reality. I used to present NPS benchmark scores for 45 enterprise software companies each quarter. It was very common to see a new small company top the list for a single quarter. If trends from prior quarters did not support this, I told my audience we should wait until the following quarter to react. If the prior quarterly trends did support it (as was the case with ServiceNow’s gradual move from awful NPS scores to great ones), I proposed immediate action.

Nobel laureates by per capita

Another entertaining example and research question would be the following: Why do the Faroe Islands have the highest number of Nobel Prizes by capita? Let’s send out a research team to study their education system and copy it everywhere! In fact, the reason is that they have had one Nobel, and have a population of just over 48,000.

Regression to the mean

If you have loads of data and a consistent trend for something, such as transactional NPS scores for a call center, be careful how you react when you see a substantial short-term change from the trend. To give an analogy, behavioral psychologists generally agree that punishing children for unusually bad behavior does not work, and can even have negative long-term effects. (These psychologists believe that rewarding good behavior works far better.) However, most parents believe punishment does work, and works well. The reason for this belief is regression to the mean. The child has a certain average behavior, trending in a certain direction. They then do something that is unacceptable, and outside the prior average. The parent punishes them and the behavior returns to the prior average. That would have happened in any case, even without any intervention. That is, it will regress to the mean. The only time you have act in new ways is when the trend has actually changed (as it does in fact do if you reward good behavior over time). If you are constantly seen to reward or punish people for what are really single random deviations from their normal trend, you will lose credibility.

Looking forward

Next time I will discuss whether you should adjust or ‘weight’ your research results in any way before you report them. It’s a tricky subject without simple answers. I hope I can reduce the complexity for you. And that post will also include a ‘true or false’ learning test.

 

As is often the case, the above is a slightly-edited version of a chapter in one of our books; in this case Net Promoter – Implement the System All of our books are available in paperback and Kindle formats from Amazon stores worldwide, and from your better book retailers.

Feature image for No No No complaints blog

How and why you should make it as easy as possible for customers to complain

/ , , , , , / , , , , , /

Complaints are bad, right? Well… sort of. The complaints you know about are valuable feedback and give you an opportunity to recover and retain your customers. It seems logical that you should do everything possible to minimize the number of complaints you receive.

Are you hearing all complaints?

But what about the customers who have a complaint and don’t tell you about it? But they tell their families, friends and colleagues at work. You have no opportunity to recover because you don’t know anything about the complaint.

Maybe this is such a small problem that you should not worry about it. Or maybe it is huge. I would like to suggest the latter is true, based on Bain research. The key points are shown in a graph on page 111 of the hardcover version of The Ultimate Question 2.0 by Reichheld and Markey, and explained in the text. The message is that the vast majority of people who do not respond to your survey requests are either Detractors or Passives. If your survey response rates are less than 100% your Detractors miss out on the opportunity to tell you what is making them so unhappy. They probably think you have wasted enough of their time already and just leave you for one of your competitors.

That’s not good.

Make it easy to complain

In short, I believe you should make it as easy as possible for your customers to complain. I have recently come across a startup that is doing just this. Jaakko Timonen (with whom I have no commercial relationship of any kind) founded his company No No No with the objective of ensuring that consumers can complain easily and get a response. Customers can log a public complaint about any company that is active in the United States here. The complaints are routed to the companies concerned. Their incentive to act is the simple fact that customers have the opportunity to publicly rate and comment on the issue resolution.

Think of it as an online version of the ombudsman function that exists in certain industries. An ombudsman is a person or office that is supposed to be an agreed neutral party in a dispute. They are common in the insurance industry, for example, and their role is to facilitate an agreement between the two sides. No No No does the same thing, at scale. At first sight, the Better Business Bureau website seems to offer a similar service. The difference between the two approaches may seem subtle, but I find it significant. The Better Business Bureau site allows you to log a complaint about any business but does not provide any way for you to rate the resolution of your complaint. I feel that making customer views about the effectiveness of a resolution is important. How else can companies compare themselves with their competitors?

At a more personal level – what are some tips to ensure your own complaints about a company get resolved?

A fascinating side-benefit of Jaakko’s work is that he has been able to study the keywords in a complaint that seem to most stimulate rapid resolution. The two that seem to have the greatest impact are the use of the word ‘you’ in the subject line, and mentioning a competitor in the main text of the complaint. Both improve resolution rates by a factor of five.

The technology is also available to individual companies. Bearing in mind what I have written above, if you owned the Penn Hotel, would you prefer the upper or lower customer feedback solution below?

 

For more about the excellent work Jakko and No No No are doing, please have a look here.

 

Images above are from the No No No website and are included with permission.

If you like this style of writing, think about buying one of our books. All are available in paperback and Kindle formats from Amazon stores worldwide, and from your better book retailers.

Elephant to illustrate confirmation bias

NPS (10) – 10th in the series – How to avoid confirmation bias when analyzing research results

/ , , , , , / , , , , , , /

In his book Thinking, Fast and Slow [1], Daniel Kahneman describes two ways our brains function. He refers to them as ‘System 1’ and ‘System 2’. At their most superficial, you can think of System 1 as intuition, and System 2 as rational thought. He offers persuasive arguments that System 1 acts first, and that System 2 is lazy, and does its best to avoid work.

Relevance to NPS

For NPS to be a trusted metric, it must work for both System 1 and System 2. The Ultimate Question 2.0 concentrates on System 2. This book’s content on survey design, improving response rates and things like the automated analysis of verbatim responses all appeal to System 2. I consider the System 2 work to be necessary but insufficient for success. You also have to appeal to the fast-acting intuitive reactions of System 1 thinking. The following are some items for you to consider.

Sequence of presentation

Kahneman describes an experiment set up by Solomon Asch [2]: subjects were given descriptions of two people and asked for comments on their personalities. Here they are. What do you think of Alan and Ben?

• Alan: intelligent, industrious, impulsive, critical, stubborn, envious.
• Ben: envious, stubborn, critical, impulsive, industrious, intelligent.

Most people describe Alan far more positively than Ben. Because the first word used to describe Alan is ‘intelligent’, the more negative character traits are considered to be justified. However, the words used to describe Alan and Ben are identical. Only the sequence has changed. The first thing that is presented has a disproportionate effect on your message, and sets the tone for all the rest. For example, if the overall message you want to communicate is that your company is making good progress, don’t start with the only negative story. When you are going to talk about what you need to improve and what is going well, start with what is going well. Of course, if you have just taken over the customer experience responsibility, you may want to be Machiavellian and start with the bad news, implicitly blaming it on the prior leadership. You could then reverse the sequence for the same information a few weeks later, and people will think you are making progress. Naturally, if you do this blatantly or often, you will lose trust.

Jumping to conclusions

We all jump to conclusions without being aware of it. Kahneman offers the example, “Ann approached the bank” to illustrate this point. You almost certainly formed an image of a woman walking towards a bank, possibly to deposit money. However, the statement is ambiguous. If we had also been told that Ann was on a canoeing trip, our conclusion would be different and we would see her paddling to the side of the river.

A lot of customer experience information is ambiguous. People’s System 1 will jump to conclusions and System 2 will sleep, unless you force it into action. You have probably heard of the metaphor of blind people trying to identify what they are touching, when each is able to feel just one part of an elephant. Turn this on its head and you have confirmation bias. If your audience has been primed to think, for example, that customer service is awful, they will use whatever they can in your data to support that conclusion, and may not even realize they are doing so.

How to get System 2 to work

In other words, how do you get people to react totally rationally to the NPS numbers you present and the financial business case you may be making? The sad truth is that you can’t do much about it. Your best solution is to force your System 1 to look at what you are going to present. Think in terms of someone who is going to ignore most of the data. What are they going to see when they look at your slides? What are they going to hear when you speak? Their intuitive, emotional reaction will win out. System 2 will act only for brief periods. Plan on it.

There is some science that suggests that asking people to do a simple mathematical puzzle that requires System 2 to get the correct answer will help engage System 2, at least for a brief period. Priming people by giving them a puzzle to do in a break, or just having it on the screen before your presentation may help. There is also evidence that making something difficult to read forces System 2 to engage. In any case, you should kick off by using a real customer quote or example, appealing to your audience’s emotions before you start showing data.

[1] Daniel Kahneman: Thinking, Fast and Slow, published by Farrar, Straus and Giroux, 2011, ISBN 978-037427631.

[2] The full set of Asch’s experiments in this area is described in the article below. The example used is “Experiment VI” in the article.
http://www.all-about-psychology.com/solomon-asch.html.

Looking forward

The next article in this series will examine ‘the law of small numbers’, an important thing to consider when you have small sample sizes.

 

As is often the case, the above is a slightly-edited version of a chapter in one of our books; in this case Net Promoter – Implement the System All of our books are available in paperback and Kindle formats from Amazon stores worldwide, and from your better book retailers.

Verbatim answers to open NPS questions

NPS (9) – 9th in the series – How to use the answers to the open questions

/ , , , , , / , , , , , , , /

While our rational minds like to see scores and trends, the primary value of Net Promoter System surveys lies in the answers to the “Why?” and “What should we improve?” questions. Interpreting and using these answers well is quite challenging. Indeed, it is so challenging that many companies don’t really try. Communicating trends in numbers is far easier than finding and communicating trends in text. Avoiding our personal biases when looking at the answers is tricky too. We will cover software solutions for text analysis in the section on infrastructure. Here are some considerations on other points.

The answers do not change quickly

Once you have been gathering feedback for some time, you will see that the most common answers to the verbatim questions change more slowly than NPS scores. This makes it difficult to maintain management interest. The main way to make people want to listen is to focus on personalization. By this I mean using quotes from specific named customers as examples of each response theme you are presenting. I suppose you can think of it as though you were directing a reality TV show. The most extreme comments are the ones that will attract the most attention. Of course, many comments will be entertaining, but not actually useful. Care is needed to ensure they support the improvement initiatives you are proposing.

Some answers matter more than others

You will quickly learn that some things are only mentioned by Detractors, and others only by Promoters. Understanding which are which is critical to understanding the items that are hygiene factors, meaning they just need to be ‘good enough’. Psychologist Frederick Herzberg developed his two-factor theory of motivation. At its core, ‘motivators’ can only motivate people, and ‘hygiene factors’ can only demotivate them. Once hygiene factors are good enough, nobody cares about them. When communicating them, I suggest separating the groupings, presenting just one at a time.

  • ‘Just not good enough’ is the set of three to five things that Detractors most often give as reasons for being detractors. If Passives and Promoters don’t mention them in their answers to the “Why” question, they are clearly hygiene factors. Once you improve them to the point where they are good enough, you don’t need to spend any more.
  • ‘Potential loyalty drivers'” are the most common improvement suggestions made by Promoters that do not come up as much from the other categories.
  • ‘Communication opportunities’ are things that commonly come up as improvement suggestions, but which you have already implemented. To pick an example, this might be a suggestion for a feature that is already in your software program, but is not as easy to find as it might be, or may be missing from your manual or help file.

Competitive comparisons are the key

The key to competitive advantage lies of course in the comparison between what customers say about you and what they say about your competitors. Using hygiene factors as an example, competitive comparisons are the only way you can know whether something some customers complain about is ‘good enough’ or not. A hygiene factor is ‘good enough’ if customers don’t complain about you more than they do about your competitors. (If you have not seen the reference to hygiene factors in an earlier post in this series, a good example of a hygiene factor in retail is the credit card payment process. It can only make your customers unhappy if your process fails. If it works perfectly, nobody will even notice it, let alone recommend your brand because of it.)

Beyond hygiene factors, if the most common improvement suggestion for the leading competitor’s product is something your product already does, this fact needs to be in your sales people’s ‘battle cards’ for competitive bids, or clearly communicated as such to your resellers. Maintain interest by showing trends over time.

If, for example, you can see that comments on a competitor’s product quality have been improving markedly, while yours have been getting worse, that should grab your audience’s attention. Support the relative trend discussion by including direct customer quotes.

The difference between the Why and Improve responses

When a customer is angry with you, the Why and Improve responses are often about the same problem. “I had to wait 20 minutes in the checkout queue” is often followed by a suggestion to “Hire more people to shorten queues at peak times.” So, for Detractors, asking both questions can seem superfluous. This is not the case for Promoters. For the same store as the Detractor, a Promoter might give “Great variety of modern furniture that suits my budget” as the reason they like you, and still say “Hire more people to shorten checkout queues” as an improvement suggestion.

Using people to study text responses

If you decide not to use software to study verbatim responses, ask a variety of people to study the data and propose the top three themes in each category. This is a type of work where using an ‘outsider’ is counter-productive. Everyone you ask to help must understand the subject matter in some detail. You are looking for expert views. If you have responses in languages your experts do not speak, using Google or Microsoft’s automatic translation software works well enough, and is of course far better than discarding those answers.



Looking forward

The next article in this series will look at how to avoid ‘confirmation bias’ as you examine survey results.

 

As is often the case, the above is a slightly-edited version of a chapter in one of our books; in this case Net Promoter – Implement the System All of our books are available in paperback and Kindle formats from Amazon stores worldwide, and from your better book retailers.

Is your NPS number good or bad?

NPS (8) – 8th in the series – I have my NPS number. Is it good or bad?

/ , , , , , / , , , , , , , /

“OK, I have my NPS number. Is it good or bad?”

There are two parts to the answer. First, the score does not tell you much on its own. The reasons customers give for the scores and what they want you to improve are more important. Second, the absolute number is meaningless. What matters are how it compares to your competition and whether it is improving or getting worse. Let’s start by talking about the score.

Scores vary greatly by industry

Not alone are average NPS scores quite different by industry, they vary widely within an industry. The graph below shows the results of a Temkin benchmark survey, for example. The overall message you should take here is that a single number is meaningless in isolation. An NPS of 15 is horrible for a supermarket and approaches best-in-class for an ISP or cable TV provider. There is one subtlety in NPS theory that is worth noting here: you gain or lose share when your NPS trend is more or less favorable compared to the leading competitor. The Temkin table considers all competitors to be equal when calculating the average score. Some of the extreme scores below will be for companies that may not be relevant (unless they happen to be your own company).

Moving from the facts in the chart to personal opinion, the five industries at the bottom of the table seem to share some characteristics. They are mainly companies that either are monopolies or used to be monopolies. In many cities or neighborhoods, there used to be only one way of getting TV, which was via your local cable network provider. Internet Service Providers suffered from a similar lack of competition. It is somewhat unfortunate that the new alternatives to cable television are dependent on ISPs for delivery.

As noted at the bottom, the graph expresses the views of US consumers. I believe the relative positions of different industries do not change much by country. The scores do vary by country, as I will discussed in a later article.

 

Temkin NPS benchmarks

Validation using CSAT scores by industry

The variations by industry are of course not unique to NPS. The table below shows the American Customer Satisfaction Index scores by industry as of early 2019. The pattern resembles that for NPS scores. Since NPS numbers can vary from -100 to +100, and ACSI numbers only from 0 to 100, it is normal that there is far less variation in the ACSI scale. Remember that these are not NPS numbers.

ACSI by sector

Communicating the ‘goodness’ of your NPS number

Once you are confident you understand the range of scores for your industry and geographical coverage, I suggest communicating some round numbers in a consistent way. For example, if you are in the supermarket business in the USA, try saying “30 is good, 40 is great, 50 is world-class.” Make sure you understand where your most important competitor is within the range.

Recognizing ‘marketing’ NPS scores

I suppose I am being polite here. I have seen too many companies trumpeting things like “Our NPS is 92.” Without any qualifier, the only acceptable source you should use for such statements is a reputable double-blind benchmark survey. If you use anything else, you should at least provide the source. I have been reading double-blind benchmark survey results for many industries for many years. A score of 92 is rare, to say the least. CEOs increasingly refer to such unattributed NPS numbers in their annual reports and earnings calls. I hope the emergence of the Bain / Qualtrics NPS Prism benchmarking service and similar offerings will make it easier for CEOs to provide recognizable sources for their statements.



Looking forward

The next article in this series will cover the use of customer answers to the open ‘Why?’ and ‘What should we improve?’ questions.

 

As is often the case, the above is a slightly-edited version of a chapter in one of our books; in this case Net Promoter – Implement the System All of our books are available in paperback and Kindle formats from Amazon stores worldwide, and from your better book retailers.

Raining surveys wide

NPS (7) – 7th in the series – Transactional / Episode surveys – Best avoided? Use very selectively!

/ , , , , , / , , , , , /

Transactional surveys are by far the most common type of customer research. It is helpful to think of them as feedback about ‘episodes’ or ‘jobs to be done’. Customers interact with you to try to accomplish some individual thing as part of their overall experience with you. An overall season of a TV series, for example, consists of a series of episodes. A customer may phone your support center to try to resolve a problem. Resolving the problem is the job they want to get done. Your research should be about that entire job, and not about a sub-component. Customer feedback about the components is only useful in the context of the overall ‘job’ they need you to do.

Three main situations where such research is useful

Transactional / episode research is most appropriate in three situations:

  1. Use it to periodically validate that improvements in your customer-centric operational metrics do indeed improve customer satisfaction. For example, your customer complaint handling metrics may show that all complaints are fully resolved within 48 hours. Do your customers agree that this is the case? Perhaps your agents have incentives that push them to close cases that are not actually resolved.
  2. Obtaining customer feedback that cannot easily be derived from operational metrics. Let’s suppose you have done a metric validation survey and have discovered that there is a disconnect between your operational metrics and what customers think about complaint handling. You know there is a problem, but don’t have the information necessary to fix it. What exactly is going on, as perceived by customers? Resist the temptation to ask about the politeness of the person on the phone and a long list of items you might think of. Just use the NPS format and ask the customers why, and what should be improved.
  3. As quality indicators for subcontracting agreements. Let’s suppose you have contracted with a trucking company to deliver goods, or have contracted with a third party to process your employees’ travel expense claims. You may have specified certain quality or satisfaction levels in the contract, and need to check that the company you are using meets your standards.

You may be surprised that I do not list something like “100% validation that customers are happy with telephone support”. You will never get 100% of the customers to answer, and your operational metrics and responses to the scripts you use should tell you this in any case. Over the years, I have seen that many call center scripts do not actually ask customers whether their issue has been resolved. I find this odd and recommend always asking, if you don’t already know for certain. If you work in software, please resist the temptation to consider a software defect as resolved if it has been agreed that it will be fixed in the next version. The customer’s issue is only resolved when the software has been released and the customer has installed it. You should leave the case open until then, to ensure correct follow up. Customers love to be remembered, and will appreciate the call at the appropriate time.

NPS may not be the solution for everything

There are types of transactional surveys where I feel NPS is not the best solution. You may still like to use it, simply to avoid having to communicate an additional measurement system within your company. Willingness to recommend a company or product is mainly an overall brand metric. While an individual transaction can destroy your standing with a customer, it is really unlikely that an individual transaction can cause a customer to recommend you all on its own. NPS does not have predictive value for the revenue of telephone support businesses, for example. If you can deal with the communication challenges, you may be better off using Customer Effort Score for many types of transactional surveys. Customer Effort Score measures how hard customers found it to do certain things, like solve a problem. The authors of The Effortless Experience (see below) have demonstrated that it predicts revenue for support businesses.

NPS has additional questions for transactional surveys

In the current version of the Net Promoter System for transactional surveys, the usual Recommendation and Why questions are followed by additional questions: “To what extent has your latest experience made you more or less likely to recommend [company]?” followed by an additional Why question. While this makes the format longer, it provides information on the positive or negative impact individual transactions may have on your overall brand perception.

Difficulties for service center research

It is challenging to use NPS, or indeed any other research system to gain insights about telephone service centers. The reason is simple. As documented in The Effortless Experience[1] by Dixon, Toman and DeLisi, most customers who contact you by phone are simultaneously trying to answer their questions or resolve their issues on your website or elsewhere. It is therefore difficult to know precisely what resolution process their survey responses refer to. The customer could have found 90% of the information they needed from a third-party website, and just received confirmation from you by phone. Think of this as you choose the wording of your questionnaire.

Pop-ups

Personally, I find pop-up web surveys for transactions irritating. I suppose they must work for some companies, or we would not see them anymore. Best practice is to have at least the rating scale directly in the pop-up, like the Windows 10 satisfaction pop-up you may have seen if you are a Windows user. If you are not able to work out how to capture text feedback within the pop-up, clicking on one of the numbers should take you to a web page to answer the remaining questions.

Conclusion

Transactional surveys are over-used in my opinion. Customers are almost never informed about what you have learned or what you are doing with the results. Indeed, even where I have been responsible for the implementation of transactional surveys, I never included a process for summarizing what we had learned and informing customers, so shame on me too. In most cases, the surveys are used to generate metrics, not action. Surveys are sometimes implemented when neither people nor funding are available to work on the improvements that might be required. Yes, there may well be a ‘Hawthorne effect'[2], in that many studies have shown that the simple fact of demonstrating you are paying attention to something does indeed improve its performance. I liken that to the placebo effect of ‘alternative’ medicines. These effects are worthwhile, but are not a great substitute for intervening and fixing what ails you.


[1] Matthew Dixon, Nick Toman and Rick DeLisi: The Effortless Experience, Portfolio, ISBN1591845815
[2] The psychological experiments at the GE Hawthorne plant are described in the Wikipedia article at https://en.wikipedia.org/wiki/Hawthorne_effect



Looking forward

The next article in this series will cover will answer a frequently-asked question: what is a good score?’

 

As is often the case, the above is a slightly-edited version of a chapter in one of our books; in this case Net Promoter – Implement the System All of our books are available in paperback and Kindle formats from Amazon stores worldwide, and from your better book retailers.

Bain NPS emoticons

NPS (6) – Getting useful feedback about one of your products, services, or customer projects

/ , , , , , / , , , , , /

Unless your company only sells a single product or service, you need a separate process to gather feedback about each product or service you sell. While benchmark and relationship surveys cover a variety of people and job roles at the customer end, product and service feedback should come from the end users. To get reliable and credible information, you should ask for it when you are explicitly planning to use it. For a service contract that you are going to renegotiate (for example), get feedback from the end users before the negotiation starts and use it to present a more convincing case for renewal.

For products, tell the end users that you are starting a new design cycle, solicit their suggestions, and tell them which suggestions were most common and will be included in the next product or version. If you have customers who are references for your product, pay particular attention to what they say. Your most reliable feedback will come from customers who actively use the product or service. For this type of survey, I don’t think it matters whether they have purchased it recently or not. Indeed, an end user may have no idea when the purchase decision was made. If your survey is about something like software that has versions, you will need to ask the users what version they are using if you don’t already know. If so, make sure you tell the users how to determine the version number.

Project surveys

Assuming you want to improve the way you plan and deliver customer projects, a formal feedback process is good practice. I have never seen these done well and they are indeed tricky. I believe you should consider project surveys as the least reliable in your portfolio. The reasons are quite complex, and change somewhat between shorter (up to three months) and longer projects:

  • The people who will answer the survey are most often chosen when the project is complete (for shorter projects) or when a major milestone has been achieved. Since the project team normally chooses the people who should provide feedback, they have an understandable tendency to select only those who will give favorable answers. Ideally, this can be offset by agreeing the names or job titles of those who will respond in the project contract. Unfortunately, I have never seen this done, both because people want to use standard contracts, and because they do not want to “waste time negotiating details.”
  • Another possibility is to be clear, both in theory and in practice, that you are not going to measure the project team on the customer feedback. Emphasize that you will use the feedback to improve the processes and tools for all projects, as well as to plan the development of project team members. It is almost impossible to communicate this credibly. Are you really willing to discard customer feedback when putting project team members through their performance reviews? The only thing left is to have a person outside the project team agree the names of those who will provide feedback. This can be the executive sponsor for the customer, if you have one, or the sales leader assigned to the customer.
  • For longer projects, the members of the project team and the overall project manager may change over time. It is quite common to see a first team for a proof of concept, a second team for a pilot and a third team for a comprehensive rollout. In such situations, you should not have a single feedback process, but have at least one for each phase. Working purely by phase is deficient too, as lack of project management continuity across phases is a common customer complaint.

While remaining aware of the aforementioned defects, my recommendation is to track trends by manager of project managers. It seems reasonable to expect that a single project manager will introduce the same conscious or unconscious bias to all project surveys over time. Tracking at the manager level will smooth it out somewhat.

Projects involving Alliances

Most people think of alliances between companies as being two-way affairs. However, it commonly takes three or more companies to implement a customer project, and any research process must take that into account. If a customer receives a survey from only one of the companies involved in implementing a project, they are likely to be surprised. An example may help here.

When I was managing certain software alliances at Compaq, I got a surprise. I had Lotus as one of the services alliances I led at a worldwide level. Lotus had been acquired by IBM about 18 months previously. IBM encouraged them to continue to act as though they were independent. One consequence of this was their love of Compaq. 55% of all the Lotus Notes installations were being sold on Compaq ProLiant servers, and the proportion was increasing slowly, to the irritation of Lotus’s new masters. I was in their Cambridge Mass. HQ, trying to work out how to use services to grow our joint business. It turned out that neither Lotus nor Compaq wanted to be ‘Prime contractor’ for any Notes implementations. A simple two-way alliance was pointless.

Further discussion revealed that the largest Notes implementer on the planet was PricewaterhouseCoopers (PwC), and that this was particularly the case in EMEA. I met repeatedly with the head of the PwC Notes practice in Zurich, and we were successful in some three-way projects with Lotus. (Then HP tried and failed to acquire PwC, ruining the PwC/Compaq work, though this is not directly relevant here.) Two- and three-way alliances are quite common for customer projects. You need to agree a single approach to surveying the whole project and driving improvements together.

Training surveys

Since I have taken many training courses over the years, I have been surveyed many times. Surveys tend to be quite long and to involve rating the teachers. Oddly, I don’t recall ever being asked what any individual teacher could improve. I feel NPS can be used in a hierarchical way, starting with an overall course score, and the usual Why and Improve questions. This can be followed by the same three questions for each teacher. I do not believe you should ask about the course material, facilities, catering, accommodation and so on. If they matter, they will come up in the written comments. Unfortunately, trainers tend to want to train you on how to answer the surveys. They will typically show a slide that gives a sample score (always a 10), and sample comments, which are always positive. This is a misguided approach and means the trainers do not get many of the improvement suggestions they need.

 

Note that the Bain NPS emoticons at the top of this article are registered service marks and are used with permission.



Looking forward

The next article in this series will cover will be about how to handle transactional / episode research.

 

As is often the case, the above is a slightly-edited version of a chapter in one of our books; in this case Net Promoter – Implement the System All of our books are available in paperback and Kindle formats from Amazon stores worldwide, and from your better book retailers.

Track relationships with an organization chart

NPS (5) – Relationship research with large B2B customers

/ , , , , , / , , , , , /

A good relationship feedback process could be described as a non-anonymous mono-vendor benchmark survey. You usually only get detailed information about your own company. If you just have a small number of customers, say less than a hundred, you should simply interview them all face-to-face. If you have a large number of customers, you should segment the customers in some way, using increasing depth for your most important customers. Deeper surveys do not mean asking more questions. The Net Promoter System format remains intact. When you interview your most important customers, you go into far greater depth on the ‘Why?’ and ‘What should we improve?’ questions. Clarifying the input right away will make it far easier to prepare a worthwhile improvement plan for the customer to approve.

It is certainly possible to add questions about competitors to your relationship survey, though it would dilute the focus on your own company’s improvement opportunities. In cases where your product and your competitor’s are mutually exclusive, meaning nobody would buy from both companies, adding questions about competition would give you an early indication about whether customers are thinking of changing. For example, “Why do you buy from us instead of [insert key competitor name]?

Resellers and other partners

Companies that help you to sell or implement your products and services are a special case. Since their job is to recommend your products and services, asking them the recommendation question is not appropriate. I recommend asking “Please rate your overall satisfaction as an Acme partner”, followed by the usual “Why?” and “What should we improve?” Since your partners have the choice of working with other suppliers, they are a good source of competitive information. I therefore suggest adding the following questions, using a reseller example, and supposing that the main competitors are called Alpha and Beta:

  • Do you sell products from Alpha or Beta?
  • What can we at Acme do to make it easier for you to recommend us over Alpha and Beta?

Future-facing relationship segmentation

I believe it is more useful to segment your research and improvement process by your future revenue expectations for a customer, rather than by what you achieve now. This means that customers who spend a lot on your type of product or service deserve a lot of attention, no matter who they spend it with. I suggest a two-year time horizon. The segmentation is used to determine the type of feedback process you will use. The top tier should be interviewed face-to-face, the second tier by phone or Skype and the lowest tier by email / web. The depth of your loop-closing and improvement commitments will also vary by tier.

Publish strategic relationship NPS numbers and trends by individual, not by company

Account-level NPS metrics can be counter-productive. If you mainly publish relationship NPS scores for each individual customer and perhaps a table that compares the scores, your metric will never be reliable. Account teams who do not look good on the list will do everything possible to manipulate the process so they do better the next time. To ensure reliability, publish the scores and trends by individual person in the customer relationship map. Your overall relationship with a customer is largely the sum of a number of individual relationships. These are what need to be monitored and improved, or referenced. Improvements should be agreed with each person. Even at the level of the individual, the trend is more important than the absolute number. The chart at the top of this article is a suggestion about how to show the relevant information quite simply.



Looking forward

The next article in this series will cover will be about how to get actionable feedback for individual products, services, and projects.

 

As is often the case, the above is a slightly-edited version of a chapter in one of our books; in this case Net Promoter – Implement the System All of our books are available in paperback and Kindle formats from Amazon stores worldwide, and from your better book retailers.

< 1 2 3 4 5 >»

↑

Net Promoter™, Net Promoter System™, Net Promoter Score™, NPS™ and the NPS™-related emoticons are registered trademarks of Bain & Company, Inc., Fred Reichheld and Satmetrix Systems, Inc.
CustomerStrategy.net is owned by Maurice FitzGerald Consulting, Geneva, Switzerland • Site developed in conjunction with iCulture and iBusiness

Show Buttons
Hide Buttons