Choosing a benchmark research provider for NPS, CSAT or any other customer experience metrics
I am trying to form my thoughts about what criteria are reasonable when deciding on a benchmark research provider. My current thinking is below. I welcome your additions and corrections.
What is benchmark research?
According to the Merriam-Webster dictionary, a benchmark is ‘something that serves as a standard by which others may be measured or judged’. In the case of customer experience research, understanding your NPS, CSAT, ACSI metrics, trends, and reasons for those trends compared to those of your competitors would be a good example. The challenge lies in how to obtain numbers and other data, such as verbatim responses, for your competitors that are measured in the same way as the information for your own company. This is where the concept of double-blind benchmark research comes in. I don’t see how you could do it on your own, so using a third party seems inevitable.
Why can’t I do my own research?
It seems really unlikely that you could get an accurate contact list for your competitors’ customers. While you may have employees who used to work for your competitors, it is generally illegal for you to ask them for the competitors’ customer lists. Even if you had such a list, phoning or emailing them from your own address will necessarily bias the results. You need something totally neutral and unbiased.
What should you be looking for?
Here the three factors I feel are most important, based on years of using various companies for such research. All of this supposes that you cannot easily find benchmark studies that already cover your industry. For example, if you work for a large company selling to US consumers, the ACSI, Temkin Group, and others may already have what you need.
- The single most important factor is the quality of the contact lists for your competitors and for your own company. I feel you should prefer companies that recruit their own panels of respondents, rather than those who source them from third-party panel companies. The reason is quality control. You need to be able to interact directly with the company that recruits panel members. At one point in my work at HP, I noticed that some answers to verbatim questions were pure garbage. People just hit random keys. One even wrote ‘I am just doing this for the points’. There were also cases of NPS scores which were opposite to the sentiment expressed in the verbatim comments. The company with whom you contract has to be able to eliminate such panel members, without your intervention.
- A company that has already done this type of research, ideally in your industry. Industry experience helps vendors to correctly identify positions and people who are decision-makers, influencers, and simply end users of your products or services.
- Ability to do basic data processing and presentation decks without your intervention. Ideally, they should be able to identify surprises in the research results for you. Over the years I learned that if you have to do a lot of work just to understand your own results, you lose the ability or motivation to dig deeply into your competitors’ results. A vendor who can supply the data in a way that can be directly analyzed using Tableau (or similar software) and text analytics would be ideal.
Syndication is another possibility
Since many companies choose to publish misleading NPS and other metrics, you and your competitors may actually be motivated to cooperate on a common benchmark methodology, and share the cost of hiring a company to do it for you. You could choose to make this into ‘open’ research, meaning you allow additional companies to join the effort and share the cost. This is ‘syndicated research’.
Some vendors
Bain has started an NPS benchmark research certification program. Their certification criteria are not yet public. J.D. Power and ROI Consultancy Services were the first two companies to be certified. You can find a list of research vendors on this page. Note that they do not indicate which vendors have been certified, and that I have no commercial relationship with Bain or any of the research vendors.
Conclusion
The three points above reflect the most important things I have learned over the years. Other factors may matter to you too, and I welcome learning about them. Benchmark research is the only type of research that should be able to accurately predict your revenue trends relative to your competitors. According to Bain and other research, competitive NPS trends explain 20 to 60% of market share trends, for example. Fred Reichheld and Rob Markey’s thoughts on the subject are here. Decent analysis of answers to open questions like ‘Why?’ and ‘What should we improve’ should guide your own efforts and give you competitive insights.
Please let me know what you think below.
