Net Promoter System

Upwards arrow

NPS (16) – Five more survey response rate improvement suggestions – 16th in the series on NPS

/ , , / , , , /

This is the 16th article in a series on the Net Promoter Score and Net Promoter System.

In addition to the direct marketing techniques mentioned in my last article, here are some additional ways of improving response rates to email-based surveys, in priority sequence:

  1. By far the most important thing you can do is allow respondents to respond to the survey directly in the email they receive. Unfortunately, this is technically difficult. The best you can do is to embed the first question in the email, usually the overall rating question. Use HTML widgets to do this. They do not get blocked by the common email clients at least at the time this is written. Embedding the first question improves response rates by a factor of 2.5x, in my personal experience. Promoter.io and other software providers have found that it triples response rates. My suspicion is that embedding the first question makes it far less likely that people believe the survey link could be a phishing attack or other security risk.
  2. Provide feedback to everyone, not just the respondents. When you have completed the survey and determined the actions, write back to the entire mailing list to let them know their voices have been heard and telling them what you are going to do. Those who did not respond are more likely to answer the next time and could also be given a way of adding their voices immediately if they do not feel represented.
  3. Ensure you have an email contact list that you maintain and use regularly for customer communications. That has two main benefits, one obvious and one less so. First, you will have far fewer emails ‘bounce’ due to unknown addresses. The second benefit is that customers who read your emails will almost certainly have authorized the displaying of ‘pictures’ for anything received from your email domain. Outlook and other email clients default to blocking pictures from unknown sources. You should minimize the use of pictures and other graphics in outbound emails in any case, as they present consistency challenges across phone, tablet and PC platforms.
  4. Test your outbound email with a set of internal users first, to detect problems. Send some additional test emails to internal users from a non-company email address, to see how they display if an email client blocks anything or automatically puts the email in a spam folder.
  5. While I have suggested not using strong incentives, simply thanking people in advance, promising to act and to inform them of the results should be a sufficient incentive for many, and should have a positive effect on response rates.

Risk of over-surveying

How do you know whether you are over-surveying? There are two main principles. First, you need to provide more value than you extract, which should push you towards short surveys. Second, you need to be planning to act on the feedback, including informing customers about your progress. Remember that over-surveying is about individuals, not companies. It does not matter if you send surveys to multiple people in a company. What you want to avoid is sending one individual multiple surveys in a short period of time. When thinking of this, bear in mind that your competitors may be sending surveys to the same person too.

Over-surveying may be over-rated as a problem

Let’s face it, if a customer has not responded to your survey request, they have not wasted any time, but you do not know what they are thinking. Most transactional surveys will give you a response rate between 5 and 15%. Product survey response rates can be a little higher. The people you need to avoid asking to respond again in a short period of time are those who already answered your survey. Most of your customers will not have answered. These should be the focus for the following research round, assuming it happens reasonably soon after the previous one. Write something that starts like this:

“Dear Mrs. Smith,

I realize you did not have time to respond to my request to help us improve our product when I wrote to you six weeks ago. The people who did answer helped us to determine the two items below as our top two improvement priorities. I really want to have a third project and would like to once again ask for your help. There are just three questions and this won’t take you more than about 3 minutes…”

Taking this approach improves the overall response rate, and therefore improves reliability. Make sure you don’t send these emails to people who have already answered.

More generally

If you have listened to customers, made improvements, then reported the progress back to customers, you cannot be accused of over-surveying if you start the process again. Customers will be able to see that you have provided at least as much value as you have extracted and will cooperate. It is important to communicate your progress to all customers, not just those who responded.

Looking forward

A big subject the next time: cultural differences. Do they affect survey outcomes? (Yes, they do.) How should you deal with this? It’s a deep subject and will be the longest article so far in the series. And I will include a learning check on survey design and response rate improvement techniques.

 

As is often the case, the above is a slightly-edited version of a chapter in one of our books; in this case Net Promoter – Implement the System All of our books are available in paperback and Kindle formats from Amazon stores worldwide, and from your better book retailers.

Raining surveys

NPS (15) – Use direct marketing techniques to improve survey response rates – 15th in the series on NPS

/ , , / , , , /

This is the 15th article in a series on the Net Promoter Score and Net Promoter System. The subject this time is the use of classical direct marketing techniques to improve response rates. By classical, well… yes, I do mean ‘old’.

Like most people, I receive lots of survey requests by email. Unlike most people, I don’t just delete them all. I want to learn. What I usually learn is that most survey designers don’t understand even the most basic direct marketing techniques. Here are some that you may want to consider.

Direct marketing reference source

My direct marketing guru is Drayton Bird, author of Commonsense Direct & Digital Marketing. Most of the techniques for getting people to do what you want them to do have not changed since direct marketing was done by sending people things by snail mail. Here are the most critical points:

Personalize

Make your invitation as personal as conceivably possible. Use the person’s first and last name. (It is better to err on the side of formality. There are cultures where using an adult person’s first name only will turn them off. France is a good example.) If you have additional information about them, like the city where they live, the industry in which they work and so on, use that too. Personalization applies at the outbound source as well. If you are doing an overall brand image/relationship survey that is being sent by email to smaller customers and partners, it should go out from the CEO’s mailbox. Product-specific surveys should go out from the product manager’s mailbox and should start with “Good morning Mr. Doe, I am the product manager for the DoubleBubble Jacuzzi you bought recently at Home Depot…”

Single action request

Only ask the person to take a single action. When I was taught direct marketing, I was taught to think of the recipient holding a stack of letters and standing over a waste bin. The first letter asks the person to do A. She puts it in an action stack on her desk. The second one asks her to do B. She puts it in the bin. The third one asks her to do A or B… She thinks “Wow, A or B, which should I do? I can’t decide.” So she drops it in the bin. Like most aspects of direct marketing, this has been studied using A / B testing; the technique where you change only one thing in a marketing piece and measure what happens. While intuition would tell you that giving a person two choices would double the overall response rate, the opposite actually happens. Your overall response rate halves if you give a person two choices rather than asking them to do a single thing. If you give them three choices, response rates drop to a quarter.

Herd instinct

Appeal to herd instinct. The UK government has a partially privatized unit called the ‘Behavioural Insights Team’, generally known as the ‘Nudge unit’. They have convincingly shown that writing to people who are late paying their taxes in a particular way works. Something like, “Almost every company here in the greater Homeville area has paid their taxes on time. In the last year, these taxes have paid for a new playground on Middle St. and the new recycling centre behind the farmers’ market. Please join your fellow citizens in making Homeville a better place to live.” Of course, all that has to be true.

Since you are unlikely to get half the recipients of an email to respond to a survey before sending a reminder, the reminder could say something like, “Acme customers from all over the greater Chicago area have already answered our three-minute survey and are helping us to improve our lawnmowers. Please add your views to theirs to ensure all of Chicago is represented.” Naturally, this requires knowing that that the person does indeed live in that area. Again, the more personal you can make it, the better.

The correct choice of fonts

For some reason, this one is controversial and when I talk about it, I am usually told I am wrong, at least by the people who are being polite. The others use stronger words. The font you use makes a big difference to response rates. You should always use a serif font for body text in your emails or paper forms. Serif fonts are the ones with little hooks on them and with bases on all the vertical strokes. These extra hooks and bases are the serifs.There is a reason 99.999% of all books are written with serif fonts. These fonts are much easier to read.

Most people will recognize Arial as a ‘sans-serif’ font and Times New Roman as a serif font. The serifs give the eye a natural line to follow. Comprehension is much better than with sans-serif fonts because your eye never accidentally skips to the previous or following line. This is not an issue with short headings. Look around you and you will see that most books and magazines have sans-serif fonts for the short headings, and serif fonts for body text. This has been tested repeatedly in direct marketing, especially by Drayton Bird. The common view that this is no longer the case because so many sans-serif fonts are used on web pages is simply wrong.

Place pictures carefully, if at all

Don’t break up your text with pictures. Drayton Bird’s research shows that a high proportion of people won’t read anything below your picture. If you have to have a picture, it should be at the top or bottom, or perhaps to one side. Note that if you are sending emails to a new distribution list, the customers’ email clients may block the display of images.

The image below is an example of a survey request I received from Thai Airways. My Outlook client blocked the images, though not the button image, while my iPad mail client displayed it correctly. I can’t resist noting that while they know that I took the flight, they don’t know my name. Did you know that airline slang for passengers is ‘Self-loading freight’? Should I be consoled by the fact that they don’t know the name of the EVP who sent the email either? I suppose I should complete my pickiness by adding that you should spell out dates to avoid confusion. Thai has used the US date format for a European customer.

Thai airways survey for blog

Reminders

Send a reminder after two days. When you send a survey invitation by email, you will typically get 90% of the responses within 24 hours. Almost nobody files this sort of thing away for later consideration. They either handle it when they see it, or not at all. Only send the reminder to people who have not already responded. If 100 people responded to the first email, my experience suggests that about 65 will answer a second email telling them there are only 24 hours left to provide input.

Incentives

I suppose I should add a word about incentives. Incentives can improve response rates, but should not be used. Way back in early 1994, I ran a project to launch a Digital-Microsoft software subscription service across Europe. It was a precursor to Microsoft’s launch of a volume licensing program. The main technique used was direct-response advertising. One country project leader (OK, it was Norway) ignored my advice and included an incentive. The incentive was to be entered in a draw for two tickets to the soccer World Cup in the USA that year. Norway had qualified, to their surprise, and the promotion was a spectacular success. Surprisingly, it almost got me fired. A certain Steve Ballmer was leading sales at Microsoft at the time, and he apparently became irate that we had taken 20% market share in Norway with the promotion. The incumbent Microsoft resellers were unhappy and complained bitterly. My boss’s boss’s boss (John Rando at the time) had signed the agreement with Microsoft, came to see me, determined that everything was OK and defended me. Whew! In any case, it proved that incentives work in direct marketing, and have consequences.

Incentives produce positive bias

While that is an extreme case of an incentive, there is a more general issue that has to do with human nature. Providing incentives seems to positively bias responses to surveys. Humans have difficulty believing they can win something in a random prize draw if they give very negative feedback. So, if your intent is just to improve the metrics but not to gain accurate insights, go ahead and use incentives.

A/B testing

Direct marketing introduced the concept of A/B testing. This is where you change just one thing in an email (for example) and compare the results randomly with emails that do not have the change. If you want to test the effect of using a person’s name on response rates, randomly pick half your email list and start with “Dear customer” while starting the other half with “Dear Mr. Doe”. Try another experiment the following time, like using the person’s name in the subject line of the email.

Conclusion

This is not a complete list of everything you can do to improve response rates. It is just the subset that has to do with direct marketing techniques. Feel free to disagree with the above, or add your own suggestions below.

Looking forward

The next article will be the last one on improving response rates. It includes a number of proven ideas that don’t fit into a single category. One of the five is the one that I have personally found to have the strongest positive effect.

 

As is often the case, the above is a slightly-edited version of a chapter in one of our books; in this case Net Promoter – Implement the System All of our books are available in paperback and Kindle formats from Amazon stores worldwide, and from your better book retailers.

Normal curve for NPS blog on statistical significance

NPS (14) – 14th in the series – Customer survey response rates and statistical significance

/ , , , , , / , , , , , , /

Welcome to the 14th article in my series about NPS and the Net Promoter System. This time I cover response rates, sample sizes and statistical significance. Perhaps the most important points are about how to communicate such technical concepts. (Hint: Don’t!)

All survey types are sensitive to response rates. The lower your response rate, the less accurate your survey. Bain analysis shows that surveys with low response rates give over-optimistic results, as the people who like you tend to respond more easily than those who do not. Getting high response rates is primarily a function of the nature of the relationship you have with the people you are surveying. If you are using email to try to survey people you have never met, and with whom you have never done business, you should expect a response rate of 3% or less. At the other extreme, if you have a practice of formally interviewing ten executives from your largest client face-to-face every six months, have been doing so for the last five years, demonstrating that you provide the improvements they request, a 100% response rate is a reasonable expectation.

Perverse effect of high response rates

High response rates drive lower scores on a constant scale. In my experience with HP, this meant that our deep face-to-face relationship surveys with 80%+ response rates usually produced Net Promoter Scores that were considerably lower than the scores we saw in benchmark surveys or indeed in any other type of survey. Avoid the temptation to compare scores from different types of survey, or at least from surveys with very different response rates.

Here is what Reichheld and Markey say

My own experience is not an isolated case and others have found similar trends. Based on their many years of experience, Fred and Rob provide some numbers, and I believe the numbers should surprise most people. They show a far greater impact of low response rates than expected. I don’t believe the phenomenon is specific to NPS. The principles should apply to all rating systems. Let’s suppose you have carried out a survey and the results are as shown below.

What about the non-respondents?

The question is, what do the people on the right think? The diagram below is what is shown in The Ultimate Question 2.0. It represents what would have happened had 100% of the recipients responded to your survey.

The theory is that Promoters are the ones most likely to answer your surveys. They like you and want to help. Passives and Detractors are more likely to think you have already wasted enough of their time. Extreme Detractors will still tend to answer. In a conversation with Rob Markey I asked him how they had determined the numbers on the right. He said it was done by observing the actual behaviors of the non-respondents in a large number of studies in a variety of industries.

Surprise!

This surprises most people. It is a very difficult subject to study as you can’t simultaneously get different response rates for the same survey and same audience. What I have seen is that the results from people who take the survey when they receive a follow-up email tend to be lower than for the initial respondents, though I have also seen exceptions. My brother (and this book’s illustrator) has a doctorate in Cognitive Psychology and has not been able to design an experiment that would prove the theory. Nonetheless, the psychology suggested by Fred and Rob makes sense to me. (Feel free to accuse me of cognitive bias!)

Statistical significance

Analysis and determination of statistical significance of results can be challenging. The following statement may seem counter-intuitive: If you want people to trust your metrics, don’t include any information about statistical significance in your main communication vehicles. There are two main reasons for this:

  • There is no single way to explain significance across all feedback systems. Correctly explaining the significance for each type of survey is long and complex and will make people believe you are manipulating the numbers.
  • There are survey types where intuitive significance rules do not apply. For example, if you have 20 customers that represent 60% of your revenue and you have high-quality relationship survey results from 15 of them, you are in great shape. If you try to explain the statistical significance range correctly, some amateur statistician in your audience is bound to say, “There are less than 30 companies in your sample so it can’t possibly be significant.” If you don’t mention significance, there will be fewer opportunities for such people to embarrass themselves. If the question is asked, well, good luck, and be diplomatic.

Improving response rates

Next, let’s look at the key techniques for improving response rates:

  • Application of classical direct marketing practices.
  • Use of technology when email is used to send survey invitations.
  • Work on the quality of your contact list.

Looking forward

Next time I will cover the use of classical direct marketing techniques to improve response rates.

 

As is often the case, the above is a slightly-edited version of a chapter in one of our books; in this case Net Promoter – Implement the System All of our books are available in paperback and Kindle formats from Amazon stores worldwide, and from your better book retailers.

 

Survey dramatic interpretation

NPS (13) – 13th in the series – Whether and when to survey

/ , , , , , / , , , , , /

This is the 13th episode in my series on the Net Promoter Score and the Net Promoter System. This one is a bit longer than the average and covers some important topics, including whether you should bother surveying your customers at all. If you have missed some prior articles in the series you will find them on our blog page here.

The first step in designing surveys is being selective about what research you are going to do. Make sure you are going to provide customers with more value than you will extract from them.

Sanity check on whether you should gather feedback at all

Here are some considerations that should help you decide where to start:

  1. Are you going to use any new input you get? If you have already established your top five improvement priorities, have staffed them, are implementing, and can’t add a new project in the next six months, don’t do any surveys other than those that provide progress feedback on your improvement work.
  2. Taking a software example, if you are planning a new major release and want user input on what should be in it, ask them in a formal way. After 30 years in and out of the software business, as a leader and as a user, I have never once been asked for formal input about what should be in the next version. Product managers generally feel they know what users want, based on informal and anecdotal conversations, rather than research. This of course applies beyond software.
  3. Not every customer touchpoint is important. Use your overall brand or benchmark surveys to establish which touchpoints customers mention as competitive advantages or disadvantages, and don’t survey anyone about any of the others. To pick a B2B example, surveying people about the friendliness, competence and efficiency of the person who handles a query about the accuracy of an invoice is certain to be a waste of resources that could better be spent elsewhere.

Timing is everything

Survey timing is an important part of research reliability. Seeking feedback about things that are in the distant past is unlikely to be of much value for example. Here are some appropriate times to consider collecting customer input, by type of survey. What follows presupposes that you are going to use the feedback to make improvements for your customers. If not, there is no good timing.

Product surveys part 1 – Unboxing and installing your product

Here we are talking about an individual product, not a generic purchase or other touchpoint experience that applies to all your products. Some products are sophisticated (the marketing word for complex) to install. Others are not. If your product takes time and effort to unbox or download, install and use for the first time, it is worthwhile understanding what your customers think of the experience and how it can be improved. The research needs to be done immediately after the experience. Waiting a week or more will mean losing the positive and negative emotional impact.

If yours is a technology product, I suggest sending the survey request or phoning the customer within 48 hours of delivery. While this will be easy to do in a direct business that sells to consumers, it is more difficult in other circumstances. You want to survey the end user of the product, and they may not be the buyer in a B2B situation, so you might not know whom to contact.

If you use resellers, they may not want to provide the customer data, and you should offer to run the survey on their behalf, providing them the data for initial screening. If yours is a software product, the installation routine itself should generate the survey request 24 hours after the installation takes place. Requiring or at least suggesting product registration on a website will give you additional sources of end-user contact details.

Product surveys part 2 – Usage

In most situations, you won’t hear anything from a customer who does not have any problems. Unfortunately, the reason for not having problems could be that the customer no longer uses your product. Perhaps they never started using it in the first place. You need to understand the views of customers who have been using your product for some time, and I suggest six months after purchase as a reasonable target for your research. The purpose, as explained to the customer, should be to get feedback for improving the product itself, as well as future products. In my own experience of software product usage surveys, it is common to get suggestions for additional features that the product already has, but the customer has not been able to find. Usage surveys have the same challenge as installation surveys, meaning you may not know whom to contact.

SaaS (Cloud) software is a completely different in that you do know who is using your software and indeed how much they use it. If a customer who has been using your software suddenly stops, contact them immediately. SaaS software usually allows you to see what functions a customer uses. If you can see that they are not using something they really should be using, you can send them a link to a training video or send over someone who can talk them through it.

Service contracts

The ideal time to get feedback on a service contract is a little before the customer is considering renewing it. This is a bit more subtle than it might seem. If you have a large multi-year contract (for example to provide security services to a company) your competitors will probably know about it and will start to contact your customer well before the contract renewal date. Since your competitors will be saying how wonderful they are, and how awful you are, you need to have acted well before this point. I suggest an annual survey as a minimum for multi-year contracts, taking place half way through each year. If you have a single-year contract, I suggest doing the research about three months in, so that all improvement suggestions can be implemented before the contract comes up for renewal consideration, probably about nine months in.

Hybrid product / service

An airline flight is a hybrid product & service. This sort of offer should be researched within 24 hours of completion. Most airlines seem to miss the point. Some give you the survey while you are still in the air, not knowing essential things like whether your luggage will show up on the carousel. Others send you the survey after a week or two. Frequent fliers will have completely forgotten about the experience by then or will confuse it with another flight. This happened to me regularly, as I have been taking over 100 flights per year for nearly as long as I can remember. What did I think of my flight from Atlanta to Paris two weeks ago? Who knows? Since product and service surveys are not mentioned in The Ultimate Question 2.0, this may be a good time to reinforce the definition. An entire flight experience, from awareness, to reservation, to the actual trip itself, involves a sequence of touchpoints. Each touchpoint could be studied by a transactional survey. The airline offers many different products and services, the sum of which might be studied in their brand survey.

Relationship surveys and Brand surveys

Twice a year is a good frequency for your most important customer relationships and may work best for your brand surveys. You may like to run them both on a continuous basis so you can see the effects of things like changes in the economy and your financial and product announcements. This would mean, for example, sampling one sixth of your customers every month. Remember that you need to report results and improvement projects to customers as well as to your own team.

Transactional surveys

While it might seem obvious that they should happen shortly after the transaction takes place, it is surprising how rarely this happens. All too often, I get pop-up windows when I have just started an online transaction. I dismiss the window, and then don’t think about it anymore. For the simplest transactions, surveying just after they are complete seems appropriate. For more complex things, like telephone support for a technology product, it seems appropriate to wait for 24 hours, as it may take a little time for the customer to know whether the issue has been completely resolved.

Conclusion

I hope this helps you to decide whether you should survey customers at all, and if you decide to do so, when it should be done. It is tricky to get it right.

Looking forward

Next time I will cover response rates and statistical significance.

 

As is often the case, the above is a slightly-edited version of a chapter in one of our books; in this case Net Promoter – Implement the System All of our books are available in paperback and Kindle formats from Amazon stores worldwide, and from your better book retailers.

 

Scales for article on weighting research results

NPS (12) – 12th in the series – Should I weight my survey results?

/ , , , , , / , , , , , /

This is the 12th episode in my series on the Net Promoter Score and the Net Promoter System. There is a learning check / quiz at the end of this post.

Suppose you have performed an NPS survey and feel the data set is not representative of your customers. What should you do?

A common issue, rarely addressed

Throughout my customer experience career, I have never adjusted my data sets before or after analyzing them. This has always made me somewhat uncomfortable. Think about these examples:

  • Your competitive benchmark survey provider uses a panel of people in countries around the world. You do 60% of your business in the USA, 15% in Canada, 10% in the UK, and smaller percentages in over thirty countries. The survey provider panel is 40% in the USA, 10% in Canada, 15% in the UK, 10% each in France, Germany and Australia, and 5% in another twenty countries. Should you take the survey provider data and re-weight it to reflect the profile of your company? At HP, we did not have any data for Japan the year of the earthquake and tsunami. That had a material impact on the worldwide averages, but we did not call it out in the reports. Was this some subconscious decision because leaving out the Japanese data made the overall results slightly better?
  • You run project completion surveys for your consulting company. Customers pay vastly different prices for different types of projects, varying by a factor of 100:1. Should you weight the survey responses by revenue, giving 100 times the weight to a response for an expensive project?
  • Your relationship survey covers two CxOs, three other decision-makers, five decision-influencers and ten end users. Should you weight the survey so the responses from CxOs count more?
  • You run transactional surveys after calls are closed in your service center. You code calls by their level of severity, into critical, high, medium and low. Should critical calls have more weight in your survey? If 10% of your calls are in the critical category, but only 5% of your survey responses, should you double the weight of those survey responses?

No consensus, major consequences

There is no consensus on the topic. The reason is that it is difficult to take an unbiased decision when adjusting data sets. This is of course not just the case for customer experience data. The New York Times published an article on the subject in September 2016, while the Trump-Clinton contest was in its final stages. The Upshot gave 867 voter intention responses to four different highly respected pollsters and studied the data themselves. Each person or team studying the data adjusted it using their own methodology. Adjustments were made to reflect race, gender and age to reflect the population of Florida, source of the data. Further adjustments were made to reflect likely voting intention. For example, if Hispanics are less likely to vote than Whites, some adjusted the poll results to reflect it. Others considered whether the people polled said they had voted in prior elections. The variations in methodology produced a greater variation in results than would be expected only from sampling error. The difference in views varied by five points. One showed Trump winning by one point; another showed Clinton winning by three points. History tells us that Clinton won the popular vote by two points.

Conclusion

I am not against adjusting an analysis to force data to better match your customer profile. I am against doing such adjustments differently each time, or having different principles for different types of surveys. Either you adjust to reflect customer revenue profiles, geography and other factors, or you don’t. You need to be consistent. If you have multiple businesses in your company, the same principles should be used by all, all the time. If you use a benchmark survey provider that customizes a solution specifically for you, you should agree response quotas in your most important customer dimensions. Do it once, then avoid the temptation to change it for a reasonable period, say three years or so.

Learning check

Decide whether the following statements are true or false. Answers at the bottom of the page. The questions refer to topics that I have covered at some point between the third article in the series and this twelfth article.

  1. If your competitive benchmark NPS number is better than that of your main competitor, you will take market share from them moving forward.
  2. Good NPS scores start at 30, great scores at 40, and you are world-class at 50.
  3. Consistent trends in numbers that are based on small sample sizes are always useful. You do not need sample sizes for all competitors to be the same, for example.
  4. All customers are considered equal by benchmark survey suppliers, so customers who spend a lot have the same weight as those who do not spend much.

Looking forward

Next time I will start the general topic of survey design with an article about when and whether to survey.

 

As is often the case, the above is a slightly-edited version of a chapter in one of our books; in this case Net Promoter – Implement the System All of our books are available in paperback and Kindle formats from Amazon stores worldwide, and from your better book retailers.

Learning check answers

  1. False. What matters is the trend relative to competition. If your number is higher and has stayed flat, but your competitor’s number has improved, they will take share from you in the future.
  2. False. A good NPS score is one whose trend is better than the trend of your main competitor. A world-class score is one that is better than the competition and improving faster. The absolute number has no importance.
  3. True. You may want to treat a single outlier result as suspicious if the sample size is smaller than for your other competitors or countries, for example. However, if you see similar numbers in sequential sampling periods, you should trust the trend.
  4. True, though the survey vendors also supply demographic data so you can do your own weighting if you so wish. However, when vendors such as Temkin publish tables on their websites, there is no weighting.
Law of small numbers illustration for blog

NPS (11) – 11th in the series – The law of small numbers

/ , , , , , / , , , , , , /

Continuing the discussion on Daniel Kahneman’s System 1 (intuitive fast thinking) and System 2 (rational slow thinking), some curious things happen when people study surprising data. As I write this article in June 2019 there are media reports in France of exactly what I am about to describe, a surprising ‘cluster’ of cancer cases in a relatively small area. This just reinforces the message I wrote a couple of years ago (see below) about similar American and Swiss studies.

Kidney cancer in the United States

Daniel Kahneman asked his readers to consider the example of the data for the incidence of kidney cancer in the 3,141 counties of the United States. “The counties in which the incidence of kidney cancer is lowest are mostly rural, sparsely populated, and located in traditional Republican states in the Midwest, the South and the West. What do you make of this?” I have recently seen a similar type of study here in Switzerland. Our rational minds immediately get busy trying to analyze why the geography or being Republican might cause cancer. For the Swiss study, I worked out that it was probably the healthy rural lifestyles that provided the benefit. Fresh air, fresh vegetables, plenty of exercise. How could that not be the reason?

What about the worst places?

However, the counties with the highest incidence of kidney cancer had exactly the same profile. The somewhat similar Swiss study concentrated on this aspect. The TV news report interviewed a person who talked about the use of pesticides in agriculture, various treatments in the vineyards and so on. Seems perfectly logical to think that the rural population would be far more exposed to them. In addition, the rural populations are poorer and probably have more difficulty getting high-quality health care. There were discussions of ‘cancer clusters’ in areas with the highest rates.

The key words were…

In reality, the most important words in the description of the counties with high rates of kidney cancer were ‘sparsely populated’. Counties in the United States have populations that range from under 500 to over 10 million. According to www.kidneycancer.org, the lifetime likelihood of anyone contracting kidney cancer is 1 in 63. In a small county, it does not take many cases for the percentage to be unusually high. The general randomness of cancer over the entire country means that there will of course be some small counties, with quite small numbers of cancers, but with relatively high or relatively low incidence percentages. The extremes are what grab the headlines. Next time you see a story about something like scientists studying why people in some small village live such long lives, treat it with suspicion. A similar village just down the road may have a population with an unusually short average lifespan. That’s statistics!

Use care when presenting your data

Customer experience data commonly has small numbers. If, for example, you are presenting NPS scores for 40 countries, it is very likely that both the highest and lowest scores will be from countries with relatively small sample sizes. This has little to do with statistical significance. If you are looking at enough items, each with 90% confidence, one in ten is just going to be a relatively poor reflection of reality. I used to present NPS benchmark scores for 45 enterprise software companies each quarter. It was very common to see a new small company top the list for a single quarter. If trends from prior quarters did not support this, I told my audience we should wait until the following quarter to react. If the prior quarterly trends did support it (as was the case with ServiceNow’s gradual move from awful NPS scores to great ones), I proposed immediate action.

Nobel laureates by per capita

Another entertaining example and research question would be the following: Why do the Faroe Islands have the highest number of Nobel Prizes by capita? Let’s send out a research team to study their education system and copy it everywhere! In fact, the reason is that they have had one Nobel, and have a population of just over 48,000.

Regression to the mean

If you have loads of data and a consistent trend for something, such as transactional NPS scores for a call center, be careful how you react when you see a substantial short-term change from the trend. To give an analogy, behavioral psychologists generally agree that punishing children for unusually bad behavior does not work, and can even have negative long-term effects. (These psychologists believe that rewarding good behavior works far better.) However, most parents believe punishment does work, and works well. The reason for this belief is regression to the mean. The child has a certain average behavior, trending in a certain direction. They then do something that is unacceptable, and outside the prior average. The parent punishes them and the behavior returns to the prior average. That would have happened in any case, even without any intervention. That is, it will regress to the mean. The only time you have act in new ways is when the trend has actually changed (as it does in fact do if you reward good behavior over time). If you are constantly seen to reward or punish people for what are really single random deviations from their normal trend, you will lose credibility.

Looking forward

Next time I will discuss whether you should adjust or ‘weight’ your research results in any way before you report them. It’s a tricky subject without simple answers. I hope I can reduce the complexity for you. And that post will also include a ‘true or false’ learning test.

 

As is often the case, the above is a slightly-edited version of a chapter in one of our books; in this case Net Promoter – Implement the System All of our books are available in paperback and Kindle formats from Amazon stores worldwide, and from your better book retailers.

Feature image for No No No complaints blog

How and why you should make it as easy as possible for customers to complain

/ , , , , , / , , , , , /

Complaints are bad, right? Well… sort of. The complaints you know about are valuable feedback and give you an opportunity to recover and retain your customers. It seems logical that you should do everything possible to minimize the number of complaints you receive.

Are you hearing all complaints?

But what about the customers who have a complaint and don’t tell you about it? But they tell their families, friends and colleagues at work. You have no opportunity to recover because you don’t know anything about the complaint.

Maybe this is such a small problem that you should not worry about it. Or maybe it is huge. I would like to suggest the latter is true, based on Bain research. The key points are shown in a graph on page 111 of the hardcover version of The Ultimate Question 2.0 by Reichheld and Markey, and explained in the text. The message is that the vast majority of people who do not respond to your survey requests are either Detractors or Passives. If your survey response rates are less than 100% your Detractors miss out on the opportunity to tell you what is making them so unhappy. They probably think you have wasted enough of their time already and just leave you for one of your competitors.

That’s not good.

Make it easy to complain

In short, I believe you should make it as easy as possible for your customers to complain. I have recently come across a startup that is doing just this. Jaakko Timonen (with whom I have no commercial relationship of any kind) founded his company No No No with the objective of ensuring that consumers can complain easily and get a response. Customers can log a public complaint about any company that is active in the United States here. The complaints are routed to the companies concerned. Their incentive to act is the simple fact that customers have the opportunity to publicly rate and comment on the issue resolution.

Think of it as an online version of the ombudsman function that exists in certain industries. An ombudsman is a person or office that is supposed to be an agreed neutral party in a dispute. They are common in the insurance industry, for example, and their role is to facilitate an agreement between the two sides. No No No does the same thing, at scale. At first sight, the Better Business Bureau website seems to offer a similar service. The difference between the two approaches may seem subtle, but I find it significant. The Better Business Bureau site allows you to log a complaint about any business but does not provide any way for you to rate the resolution of your complaint. I feel that making customer views about the effectiveness of a resolution is important. How else can companies compare themselves with their competitors?

At a more personal level – what are some tips to ensure your own complaints about a company get resolved?

A fascinating side-benefit of Jaakko’s work is that he has been able to study the keywords in a complaint that seem to most stimulate rapid resolution. The two that seem to have the greatest impact are the use of the word ‘you’ in the subject line, and mentioning a competitor in the main text of the complaint. Both improve resolution rates by a factor of five.

The technology is also available to individual companies. Bearing in mind what I have written above, if you owned the Penn Hotel, would you prefer the upper or lower customer feedback solution below?

 

For more about the excellent work Jakko and No No No are doing, please have a look here.

 

Images above are from the No No No website and are included with permission.

If you like this style of writing, think about buying one of our books. All are available in paperback and Kindle formats from Amazon stores worldwide, and from your better book retailers.

Elephant to illustrate confirmation bias

NPS (10) – 10th in the series – How to avoid confirmation bias when analyzing research results

/ , , , , , / , , , , , , /

In his book Thinking, Fast and Slow [1], Daniel Kahneman describes two ways our brains function. He refers to them as ‘System 1’ and ‘System 2’. At their most superficial, you can think of System 1 as intuition, and System 2 as rational thought. He offers persuasive arguments that System 1 acts first, and that System 2 is lazy, and does its best to avoid work.

Relevance to NPS

For NPS to be a trusted metric, it must work for both System 1 and System 2. The Ultimate Question 2.0 concentrates on System 2. This book’s content on survey design, improving response rates and things like the automated analysis of verbatim responses all appeal to System 2. I consider the System 2 work to be necessary but insufficient for success. You also have to appeal to the fast-acting intuitive reactions of System 1 thinking. The following are some items for you to consider.

Sequence of presentation

Kahneman describes an experiment set up by Solomon Asch [2]: subjects were given descriptions of two people and asked for comments on their personalities. Here they are. What do you think of Alan and Ben?

• Alan: intelligent, industrious, impulsive, critical, stubborn, envious.
• Ben: envious, stubborn, critical, impulsive, industrious, intelligent.

Most people describe Alan far more positively than Ben. Because the first word used to describe Alan is ‘intelligent’, the more negative character traits are considered to be justified. However, the words used to describe Alan and Ben are identical. Only the sequence has changed. The first thing that is presented has a disproportionate effect on your message, and sets the tone for all the rest. For example, if the overall message you want to communicate is that your company is making good progress, don’t start with the only negative story. When you are going to talk about what you need to improve and what is going well, start with what is going well. Of course, if you have just taken over the customer experience responsibility, you may want to be Machiavellian and start with the bad news, implicitly blaming it on the prior leadership. You could then reverse the sequence for the same information a few weeks later, and people will think you are making progress. Naturally, if you do this blatantly or often, you will lose trust.

Jumping to conclusions

We all jump to conclusions without being aware of it. Kahneman offers the example, “Ann approached the bank” to illustrate this point. You almost certainly formed an image of a woman walking towards a bank, possibly to deposit money. However, the statement is ambiguous. If we had also been told that Ann was on a canoeing trip, our conclusion would be different and we would see her paddling to the side of the river.

A lot of customer experience information is ambiguous. People’s System 1 will jump to conclusions and System 2 will sleep, unless you force it into action. You have probably heard of the metaphor of blind people trying to identify what they are touching, when each is able to feel just one part of an elephant. Turn this on its head and you have confirmation bias. If your audience has been primed to think, for example, that customer service is awful, they will use whatever they can in your data to support that conclusion, and may not even realize they are doing so.

How to get System 2 to work

In other words, how do you get people to react totally rationally to the NPS numbers you present and the financial business case you may be making? The sad truth is that you can’t do much about it. Your best solution is to force your System 1 to look at what you are going to present. Think in terms of someone who is going to ignore most of the data. What are they going to see when they look at your slides? What are they going to hear when you speak? Their intuitive, emotional reaction will win out. System 2 will act only for brief periods. Plan on it.

There is some science that suggests that asking people to do a simple mathematical puzzle that requires System 2 to get the correct answer will help engage System 2, at least for a brief period. Priming people by giving them a puzzle to do in a break, or just having it on the screen before your presentation may help. There is also evidence that making something difficult to read forces System 2 to engage. In any case, you should kick off by using a real customer quote or example, appealing to your audience’s emotions before you start showing data.

[1] Daniel Kahneman: Thinking, Fast and Slow, published by Farrar, Straus and Giroux, 2011, ISBN 978-037427631.

[2] The full set of Asch’s experiments in this area is described in the article below. The example used is “Experiment VI” in the article.
http://www.all-about-psychology.com/solomon-asch.html.

Looking forward

The next article in this series will examine ‘the law of small numbers’, an important thing to consider when you have small sample sizes.

 

As is often the case, the above is a slightly-edited version of a chapter in one of our books; in this case Net Promoter – Implement the System All of our books are available in paperback and Kindle formats from Amazon stores worldwide, and from your better book retailers.

Verbatim answers to open NPS questions

NPS (9) – 9th in the series – How to use the answers to the open questions

/ , , , , , / , , , , , , , /

While our rational minds like to see scores and trends, the primary value of Net Promoter System surveys lies in the answers to the “Why?” and “What should we improve?” questions. Interpreting and using these answers well is quite challenging. Indeed, it is so challenging that many companies don’t really try. Communicating trends in numbers is far easier than finding and communicating trends in text. Avoiding our personal biases when looking at the answers is tricky too. We will cover software solutions for text analysis in the section on infrastructure. Here are some considerations on other points.

The answers do not change quickly

Once you have been gathering feedback for some time, you will see that the most common answers to the verbatim questions change more slowly than NPS scores. This makes it difficult to maintain management interest. The main way to make people want to listen is to focus on personalization. By this I mean using quotes from specific named customers as examples of each response theme you are presenting. I suppose you can think of it as though you were directing a reality TV show. The most extreme comments are the ones that will attract the most attention. Of course, many comments will be entertaining, but not actually useful. Care is needed to ensure they support the improvement initiatives you are proposing.

Some answers matter more than others

You will quickly learn that some things are only mentioned by Detractors, and others only by Promoters. Understanding which are which is critical to understanding the items that are hygiene factors, meaning they just need to be ‘good enough’. Psychologist Frederick Herzberg developed his two-factor theory of motivation. At its core, ‘motivators’ can only motivate people, and ‘hygiene factors’ can only demotivate them. Once hygiene factors are good enough, nobody cares about them. When communicating them, I suggest separating the groupings, presenting just one at a time.

  • ‘Just not good enough’ is the set of three to five things that Detractors most often give as reasons for being detractors. If Passives and Promoters don’t mention them in their answers to the “Why” question, they are clearly hygiene factors. Once you improve them to the point where they are good enough, you don’t need to spend any more.
  • ‘Potential loyalty drivers'” are the most common improvement suggestions made by Promoters that do not come up as much from the other categories.
  • ‘Communication opportunities’ are things that commonly come up as improvement suggestions, but which you have already implemented. To pick an example, this might be a suggestion for a feature that is already in your software program, but is not as easy to find as it might be, or may be missing from your manual or help file.

Competitive comparisons are the key

The key to competitive advantage lies of course in the comparison between what customers say about you and what they say about your competitors. Using hygiene factors as an example, competitive comparisons are the only way you can know whether something some customers complain about is ‘good enough’ or not. A hygiene factor is ‘good enough’ if customers don’t complain about you more than they do about your competitors. (If you have not seen the reference to hygiene factors in an earlier post in this series, a good example of a hygiene factor in retail is the credit card payment process. It can only make your customers unhappy if your process fails. If it works perfectly, nobody will even notice it, let alone recommend your brand because of it.)

Beyond hygiene factors, if the most common improvement suggestion for the leading competitor’s product is something your product already does, this fact needs to be in your sales people’s ‘battle cards’ for competitive bids, or clearly communicated as such to your resellers. Maintain interest by showing trends over time.

If, for example, you can see that comments on a competitor’s product quality have been improving markedly, while yours have been getting worse, that should grab your audience’s attention. Support the relative trend discussion by including direct customer quotes.

The difference between the Why and Improve responses

When a customer is angry with you, the Why and Improve responses are often about the same problem. “I had to wait 20 minutes in the checkout queue” is often followed by a suggestion to “Hire more people to shorten queues at peak times.” So, for Detractors, asking both questions can seem superfluous. This is not the case for Promoters. For the same store as the Detractor, a Promoter might give “Great variety of modern furniture that suits my budget” as the reason they like you, and still say “Hire more people to shorten checkout queues” as an improvement suggestion.

Using people to study text responses

If you decide not to use software to study verbatim responses, ask a variety of people to study the data and propose the top three themes in each category. This is a type of work where using an ‘outsider’ is counter-productive. Everyone you ask to help must understand the subject matter in some detail. You are looking for expert views. If you have responses in languages your experts do not speak, using Google or Microsoft’s automatic translation software works well enough, and is of course far better than discarding those answers.



Looking forward

The next article in this series will look at how to avoid ‘confirmation bias’ as you examine survey results.

 

As is often the case, the above is a slightly-edited version of a chapter in one of our books; in this case Net Promoter – Implement the System All of our books are available in paperback and Kindle formats from Amazon stores worldwide, and from your better book retailers.

Is your NPS number good or bad?

NPS (8) – 8th in the series – I have my NPS number. Is it good or bad?

/ , , , , , / , , , , , , , /

“OK, I have my NPS number. Is it good or bad?”

There are two parts to the answer. First, the score does not tell you much on its own. The reasons customers give for the scores and what they want you to improve are more important. Second, the absolute number is meaningless. What matters are how it compares to your competition and whether it is improving or getting worse. Let’s start by talking about the score.

Scores vary greatly by industry

Not alone are average NPS scores quite different by industry, they vary widely within an industry. The graph below shows the results of a Temkin benchmark survey, for example. The overall message you should take here is that a single number is meaningless in isolation. An NPS of 15 is horrible for a supermarket and approaches best-in-class for an ISP or cable TV provider. There is one subtlety in NPS theory that is worth noting here: you gain or lose share when your NPS trend is more or less favorable compared to the leading competitor. The Temkin table considers all competitors to be equal when calculating the average score. Some of the extreme scores below will be for companies that may not be relevant (unless they happen to be your own company).

Moving from the facts in the chart to personal opinion, the five industries at the bottom of the table seem to share some characteristics. They are mainly companies that either are monopolies or used to be monopolies. In many cities or neighborhoods, there used to be only one way of getting TV, which was via your local cable network provider. Internet Service Providers suffered from a similar lack of competition. It is somewhat unfortunate that the new alternatives to cable television are dependent on ISPs for delivery.

As noted at the bottom, the graph expresses the views of US consumers. I believe the relative positions of different industries do not change much by country. The scores do vary by country, as I will discussed in a later article.

 

Temkin NPS benchmarks

Validation using CSAT scores by industry

The variations by industry are of course not unique to NPS. The table below shows the American Customer Satisfaction Index scores by industry as of early 2019. The pattern resembles that for NPS scores. Since NPS numbers can vary from -100 to +100, and ACSI numbers only from 0 to 100, it is normal that there is far less variation in the ACSI scale. Remember that these are not NPS numbers.

ACSI by sector

Communicating the ‘goodness’ of your NPS number

Once you are confident you understand the range of scores for your industry and geographical coverage, I suggest communicating some round numbers in a consistent way. For example, if you are in the supermarket business in the USA, try saying “30 is good, 40 is great, 50 is world-class.” Make sure you understand where your most important competitor is within the range.

Recognizing ‘marketing’ NPS scores

I suppose I am being polite here. I have seen too many companies trumpeting things like “Our NPS is 92.” Without any qualifier, the only acceptable source you should use for such statements is a reputable double-blind benchmark survey. If you use anything else, you should at least provide the source. I have been reading double-blind benchmark survey results for many industries for many years. A score of 92 is rare, to say the least. CEOs increasingly refer to such unattributed NPS numbers in their annual reports and earnings calls. I hope the emergence of the Bain / Qualtrics NPS Prism benchmarking service and similar offerings will make it easier for CEOs to provide recognizable sources for their statements.



Looking forward

The next article in this series will cover the use of customer answers to the open ‘Why?’ and ‘What should we improve?’ questions.

 

As is often the case, the above is a slightly-edited version of a chapter in one of our books; in this case Net Promoter – Implement the System All of our books are available in paperback and Kindle formats from Amazon stores worldwide, and from your better book retailers.

1 2 3 4

↑

Net Promoter™, Net Promoter System™, Net Promoter Score™, NPS™ and the NPS™-related emoticons are registered trademarks of Bain & Company, Inc., Fred Reichheld and Satmetrix Systems, Inc.
CustomerStrategy.net is owned by Maurice FitzGerald Consulting, Geneva, Switzerland • Site developed in conjunction with iCulture and iBusiness

Show Buttons
Hide Buttons