NPS (20) – How not to do customer research – 20th article in the series on NPS

While I am sure it is not their intent, the way British Airways surveys both its general and special customers is the worst I have ever seen from any company, from a customer perspective. Perhaps paradoxically, it is one of the richest possible from an internal British Airways perspective. It provides them with a rich set of reports and metrics. While it includes part of the Net Promoter System, it has far too many screens and questions. They have never contacted me about my issues or improvement suggestions. What follows is a critique that includes screen captures from their September 2016 Executive Club survey. Please bear in mind that I am not entirely negative about it all. I just feel it works from an old paradigm, is internally focused, and designed primarily to fit customer experience strategy and reporting software supplied by KPMG Nunwood.

Summary

While the covering email from Nunwood says the survey will take 15 minutes, there are 134 screens to go through, and it is hard to see how it could be done in 15 minutes. The survey invitation is signed by James Hiller, with no job title. A Google search indicates he manages the Executive Club. Nothing is said about what they have improved based on past surveys.

As distinct from a pure survey about the Executive Club itself, competitors are also covered, as are BA and competing airlines. The huge amount of data collected probably fits a comprehensive internal reporting structure, and provides no particular value to customers. It violates the fundamental principle of customer research in that it fails to provide more value than it extracts. Here is how it starts:

Cher Monsieur Fitzgerald,

Given that you are an Executive Club Member, we would welcome your views on your experiences with British Airways and we would like to understand how they have influenced the way you feel about us.

Whether positive, neutral, or negative, your feedback is welcome and highly valued, as it will help us to shape our products and services in the future, and understand what our most valuable customers need from us.

The survey will take about 15 minutes to complete.

Will I be able to provide the input I want to provide?

Before starting to work through the many screens, I decided what I wanted to say. I wanted to make three points:

1. Transiting through Heathrow Terminal 5 for long-haul travel from Geneva is far more complex than transiting via Zurich, Munich, Frankfurt, Amsterdam and even Paris. This is because passengers have to go through a pointless duplicate security check inbound from the USA / Canada, and a pointless duplicate outbound check when arriving from countries that use the same security standards as the UK. While I did not know this at the time I filled out the survey, the UK Civil Aviation Authority requires this second check and does not accept screening done anywhere else. I suppose this may date from an era when it was too difficult to separate the ‘clean’ and ‘not quite so clean’ transit passengers from each other. Whatever the reasoning, it seems pointless today. The layout of Terminal 5 could easily be modified to create this separation should the law change. BA should apply the necessary pressure, in my opinion.
2. It is next to impossible to use the Executive Club website to reserve some long-haul destinations using miles. Entire months have no availability. I am not talking about high-demand routes like Christmas in Australia. My latest experience with trying to reserve Australian flights in Business Class was for October, seven weeks in advance, and there were no flights on any date, either to Sydney or Melbourne. December and January show no availability either, which is less surprising. During a Christmas vacation, I looked at availability for South Africa the following Christmas. Nothing.
3. I wanted the opportunity not just to complain, but also to make improvement suggestions. Among the suggestions, I wanted to be able to mention that BA should tell us what they are doing with the input, and should contact people who make suggestions.

The short answer is that I was indeed able to provide the input. The more complete answer is that I was forced to bury this critical information inside many unimportant and irrelevant (to me) answers. I am sure the 100+ irrelevant answers correspond to the way some staff inside British Airways are measured.

The start

I expected BA to provide a choice of languages for the survey, and they do. I suppose it is a bit surprising that Spanish is not among the languages. It is the second-most common first language in the world with about 400 million speakers, after Chinese, with 1.2 billion.

NPS questions

The recommendation question is next as shown below. Oddly the ‘overall satisfaction’ question is asked as well. After a question about emotions, the NPS “Why?” question is asked.

Logically, the following question should be about what BA should improve. That question is never asked about British Airways, though it is asked about the Executive Club, close to the end of the survey. When you do not ask for improvement suggestions, you tend not to get any from Promoters. Remember that Promoters are the customers who admire you most, and want you to succeed. They tend to provide the most valuable improvement suggestions.

Lots more questions

Questions about my feelings about the airline follow. I suppose these provide overall statistics and somebody is measured on the trend. It is hard to see how someone would work directly on improvements from the answers, as there is no opportunity to explain the input. The questions all use rating scales and include:

• How likely would you be to travel with British Airways in the future?
• To what extent do you consider British Airways a convenient option when choosing an airline to fly with?
• Taking everything into consideration, overall how satisfied are you with British Airways?
• How satisfied are you with the value for money British Airways offers?
• How would you say you feel about British Airways?

Information about competitors

A useful section follows as shown below. After asking what airlines I have used for long- and short-haul travel in the last 12 months, the survey goes on to ask which airlines are my top choices for such travel. It then asks why the first-choice airline is in first place. I consider this section to be the only thing that is potentially useful beyond the basic NPS questions.

Oddly, they sort of throw this away by asking you to choose from a list of possible responses for not flying with BA. None of the preselected reasons was my main one, so I filled in the Other box. I feel they should have asked this as an open text question.

Then you are asked again about your overall satisfaction with BA. You can’t see the answers to prior questions and my second answer to the same question was one category lower than the first. Maybe because the survey was making my opinion more negative. The wording of the question was not exactly the same as the first one, though I don’t recognize them as different questions.

Lots of rating questions

A seven-point scale is then used to gather answers on flying touchpoints, emotions associated with BA and opinions about their communication. Finally, the section on the Executive Club starts, at last. Remember that the email invitation came from the head of the Executive Club, not the head of British Airways. Overall satisfaction and recommendation questions are followed by another extensive batch of seven-point rating scales Each rating question is presented individually, and you can review your answers to each section, possibly making changes.

Finally finally

A breath of hope is provided by the question “And finally, thinking about your experiences of any frequent flyer programmes, how could British Airways Executive Club be improved?” I mistakenly thought this was the last question. Unfortunately, I was then asked seven questions about demographic information that BA should already have in my Executive Club profile. This is one of the disadvantages of subcontracting surveys. The confidentiality agreement you have with your customers probably forbids you passing their details to a third party. However, the way BA has handled this with Nunwood is not optimal. Since they already had my name, there would be no reason not to provide them with a unique code, that would then allow BA to match my name and demographic details with their own database. The name on its own might be a duplicate.

Speculation and conclusion

I recommend a visit to the KPMG Nunwood website at www.nunwood.com. Their offers include customer experience strategy, journey mapping, NPS support and other comprehensive approaches to improvement. One of the approaches, the ‘Six Pillars’ explains their satisfaction surveys. The excessive number of questions satisfies the needs of the Six Pillars model, and fully exploits their Fizz reporting software.

I speculate that the survey and software are part of a larger consulting engagement with British Airways that requires a ‘slice and dice’ approach to data for distribution to a variety of internal departments. I feel this is an internally-focused way of proceeding. I would love to know what their response rates are, and speculate that they are under 10% overall. I suppose that higher-status members will be more likely to give feedback, and their input is critical, since they generate the most revenue for the company. I believe an interesting opportunity would be lower-status members who have higher status on other airlines. My recommendation for BA is an A/B split test, asking just the basic few NPS questions as a ‘pulse’ survey in between the main ones. I believe they would get better response rates and clearer actions to prioritize. Of course, they would lose the ‘slice-and-dice’ metrics they probably use to measure people. And finally, I have of course lost all my status and privileges with BA since retiring, so don’t know whether they have changed the way they survey Executive Club members since then. Please feel free to reply below if you have new information.

 

As is often the case, the above is a slightly-edited version of a chapter in one of our books; in this case Net Promoter – Implement the System All of our books are available in paperback and Kindle formats from Amazon stores worldwide, and from your better book retailers.