customer surveys

This is the 13th episode in my series on the Net Promoter Score and the Net Promoter System. This one is a bit longer than the average and covers some important topics, including whether you should bother surveying your customers at all. If you have missed some prior articles in the series you will find them on our blog page here.

The first step in designing surveys is being selective about what research you are going to do. Make sure you are going to provide customers with more value than you will extract from them.

Sanity check on whether you should gather feedback at all

Here are some considerations that should help you decide where to start:

1. Are you going to use any new input you get? If you have already established your top five improvement priorities, have staffed them, are implementing, and can’t add a new project in the next six months, don’t do any surveys other than those that provide progress feedback on your improvement work.
2. Taking a software example, if you are planning a new major release and want user input on what should be in it, ask them in a formal way. After 30 years in and out of the software business, as a leader and as a user, I have never once been asked for formal input about what should be in the next version. Product managers generally feel they know what users want, based on informal and anecdotal conversations, rather than research. This of course applies beyond software.
3. Not every customer touchpoint is important. Use your overall brand or benchmark surveys to establish which touchpoints customers mention as competitive advantages or disadvantages, and don’t survey anyone about any of the others. To pick a B2B example, surveying people about the friendliness, competence and efficiency of the person who handles a query about the accuracy of an invoice is certain to be a waste of resources that could better be spent elsewhere.

Timing is everything

Survey timing is an important part of research reliability. Seeking feedback about things that are in the distant past is unlikely to be of much value for example. Here are some appropriate times to consider collecting customer input, by type of survey. What follows presupposes that you are going to use the feedback to make improvements for your customers. If not, there is no good timing.

Product surveys part 1 – Unboxing and installing your product

Here we are talking about an individual product, not a generic purchase or other touchpoint experience that applies to all your products. Some products are sophisticated (the marketing word for complex) to install. Others are not. If your product takes time and effort to unbox or download, install and use for the first time, it is worthwhile understanding what your customers think of the experience and how it can be improved. The research needs to be done immediately after the experience. Waiting a week or more will mean losing the positive and negative emotional impact.

If yours is a technology product, I suggest sending the survey request or phoning the customer within 48 hours of delivery. While this will be easy to do in a direct business that sells to consumers, it is more difficult in other circumstances. You want to survey the end user of the product, and they may not be the buyer in a B2B situation, so you might not know whom to contact.

If you use resellers, they may not want to provide the customer data, and you should offer to run the survey on their behalf, providing them the data for initial screening. If yours is a software product, the installation routine itself should generate the survey request 24 hours after the installation takes place. Requiring or at least suggesting product registration on a website will give you additional sources of end-user contact details.

Product surveys part 2 – Usage

In most situations, you won’t hear anything from a customer who does not have any problems. Unfortunately, the reason for not having problems could be that the customer no longer uses your product. Perhaps they never started using it in the first place. You need to understand the views of customers who have been using your product for some time, and I suggest six months after purchase as a reasonable target for your research. The purpose, as explained to the customer, should be to get feedback for improving the product itself, as well as future products. In my own experience of software product usage surveys, it is common to get suggestions for additional features that the product already has, but the customer has not been able to find. Usage surveys have the same challenge as installation surveys, meaning you may not know whom to contact.

SaaS (Cloud) software is a completely different in that you do know who is using your software and indeed how much they use it. If a customer who has been using your software suddenly stops, contact them immediately. SaaS software usually allows you to see what functions a customer uses. If you can see that they are not using something they really should be using, you can send them a link to a training video or send over someone who can talk them through it.

Service contracts

The ideal time to get feedback on a service contract is a little before the customer is considering renewing it. This is a bit more subtle than it might seem. If you have a large multi-year contract (for example to provide security services to a company) your competitors will probably know about it and will start to contact your customer well before the contract renewal date. Since your competitors will be saying how wonderful they are, and how awful you are, you need to have acted well before this point. I suggest an annual survey as a minimum for multi-year contracts, taking place half way through each year. If you have a single-year contract, I suggest doing the research about three months in, so that all improvement suggestions can be implemented before the contract comes up for renewal consideration, probably about nine months in.

Hybrid product / service

An airline flight is a hybrid product & service. This sort of offer should be researched within 24 hours of completion. Most airlines seem to miss the point. Some give you the survey while you are still in the air, not knowing essential things like whether your luggage will show up on the carousel. Others send you the survey after a week or two. Frequent fliers will have completely forgotten about the experience by then or will confuse it with another flight. This happened to me regularly, as I have been taking over 100 flights per year for nearly as long as I can remember. What did I think of my flight from Atlanta to Paris two weeks ago? Who knows? Since product and service surveys are not mentioned in The Ultimate Question 2.0, this may be a good time to reinforce the definition. An entire flight experience, from awareness, to reservation, to the actual trip itself, involves a sequence of touchpoints. Each touchpoint could be studied by a transactional survey. The airline offers many different products and services, the sum of which might be studied in their brand survey.

Relationship surveys and Brand surveys

Twice a year is a good frequency for your most important customer relationships and may work best for your brand surveys. You may like to run them both on a continuous basis so you can see the effects of things like changes in the economy and your financial and product announcements. This would mean, for example, sampling one sixth of your customers every month. Remember that you need to report results and improvement projects to customers as well as to your own team.

Transactional surveys

While it might seem obvious that they should happen shortly after the transaction takes place, it is surprising how rarely this happens. All too often, I get pop-up windows when I have just started an online transaction. I dismiss the window, and then don’t think about it anymore. For the simplest transactions, surveying just after they are complete seems appropriate. For more complex things, like telephone support for a technology product, it seems appropriate to wait for 24 hours, as it may take a little time for the customer to know whether the issue has been completely resolved.

Conclusion

I hope this helps you to decide whether you should survey customers at all, and if you decide to do so, when it should be done. It is tricky to get it right.

Looking forward

Next time I will cover response rates and statistical significance.

As is often the case, the above is a slightly-edited version of a chapter in one of our books; in this case Net Promoter – Implement the System All of our books are available in paperback and Kindle formats from Amazon stores worldwide, and from your better book retailers.

This is the 12th episode in my series on the Net Promoter Score and the Net Promoter System. There is a learning check / quiz at the end of this post.

Suppose you have performed an NPS survey and feel the data set is not representative of your customers. What should you do?

A common issue, rarely addressed

Throughout my customer experience career, I have never adjusted my data sets before or after analyzing them. This has always made me somewhat uncomfortable. Think about these examples:

• Your competitive benchmark survey provider uses a panel of people in countries around the world. You do 60% of your business in the USA, 15% in Canada, 10% in the UK, and smaller percentages in over thirty countries. The survey provider panel is 40% in the USA, 10% in Canada, 15% in the UK, 10% each in France, Germany and Australia, and 5% in another twenty countries. Should you take the survey provider data and re-weight it to reflect the profile of your company? At HP, we did not have any data for Japan the year of the earthquake and tsunami. That had a material impact on the worldwide averages, but we did not call it out in the reports. Was this some subconscious decision because leaving out the Japanese data made the overall results slightly better?
• You run project completion surveys for your consulting company. Customers pay vastly different prices for different types of projects, varying by a factor of 100:1. Should you weight the survey responses by revenue, giving 100 times the weight to a response for an expensive project?
• Your relationship survey covers two CxOs, three other decision-makers, five decision-influencers and ten end users. Should you weight the survey so the responses from CxOs count more?
• You run transactional surveys after calls are closed in your service center. You code calls by their level of severity, into critical, high, medium and low. Should critical calls have more weight in your survey? If 10% of your calls are in the critical category, but only 5% of your survey responses, should you double the weight of those survey responses?

No consensus, major consequences

There is no consensus on the topic. The reason is that it is difficult to take an unbiased decision when adjusting data sets. This is of course not just the case for customer experience data. The New York Times published an article on the subject in September 2016, while the Trump-Clinton contest was in its final stages. The Upshot gave 867 voter intention responses to four different highly respected pollsters and studied the data themselves. Each person or team studying the data adjusted it using their own methodology. Adjustments were made to reflect race, gender and age to reflect the population of Florida, source of the data. Further adjustments were made to reflect likely voting intention. For example, if Hispanics are less likely to vote than Whites, some adjusted the poll results to reflect it. Others considered whether the people polled said they had voted in prior elections. The variations in methodology produced a greater variation in results than would be expected only from sampling error. The difference in views varied by five points. One showed Trump winning by one point; another showed Clinton winning by three points. History tells us that Clinton won the popular vote by two points.

Conclusion

I am not against adjusting an analysis to force data to better match your customer profile. I am against doing such adjustments differently each time, or having different principles for different types of surveys. Either you adjust to reflect customer revenue profiles, geography and other factors, or you don’t. You need to be consistent. If you have multiple businesses in your company, the same principles should be used by all, all the time. If you use a benchmark survey provider that customizes a solution specifically for you, you should agree response quotas in your most important customer dimensions. Do it once, then avoid the temptation to change it for a reasonable period, say three years or so.

Learning check

Decide whether the following statements are true or false. Answers at the bottom of the page. The questions refer to topics that I have covered at some point between the third article in the series and this twelfth article.

1. If your competitive benchmark NPS number is better than that of your main competitor, you will take market share from them moving forward.
2. Good NPS scores start at 30, great scores at 40, and you are world-class at 50.
3. Consistent trends in numbers that are based on small sample sizes are always useful. You do not need sample sizes for all competitors to be the same, for example.
4. All customers are considered equal by benchmark survey suppliers, so customers who spend a lot have the same weight as those who do not spend much.

Looking forward

Next time I will start the general topic of survey design with an article about when and whether to survey.

As is often the case, the above is a slightly-edited version of a chapter in one of our books; in this case Net Promoter – Implement the System All of our books are available in paperback and Kindle formats from Amazon stores worldwide, and from your better book retailers.

Learning check answers

1. False. What matters is the trend relative to competition. If your number is higher and has stayed flat, but your competitor’s number has improved, they will take share from you in the future.
2. False. A good NPS score is one whose trend is better than the trend of your main competitor. A world-class score is one that is better than the competition and improving faster. The absolute number has no importance.
3. True. You may want to treat a single outlier result as suspicious if the sample size is smaller than for your other competitors or countries, for example. However, if you see similar numbers in sequential sampling periods, you should trust the trend.
4. True, though the survey vendors also supply demographic data so you can do your own weighting if you so wish. However, when vendors such as Temkin publish tables on their websites, there is no weighting.