Blog

March 16, 2017

How to evaluate customer survey text analysis software

Maurice FitzGerald / Blog, NPS, Strategy /

Customer research of all types can be divided into two categories: rating questions and open questions. You are dealing with rating questions when you are asked to provide a numerical answer to each question, against a rating scale of some sort. Typically the questions correspond to what the company doing the research believes is important. The list is generated without asking customers what they think is important. Rating surveys can be useful for measuring your own internal departments, but are not very useful for driving improvements. After all, you have not asked the customer why they gave the particular rating. This is where open questions, like ‘Why?’ come in.

The main challenge of the open text response format is the need to analyze what people have written, ideally without any human bias. I would like to cover some general principles about using text analytics software. The overall intent is to provide you a way of comparing and selecting software for your own use. There is no single universally-correct answer. Your answer will depend on factors like volumes, languages and budgets, among others.

General principles about text analysis software

Here is what I believe text mining and analysis software has to be able to do, with increasing levels of difficulty as summarized in the diagram at the top of this post. Software should be graded in each area:

Accept inputs in a variety of formats such as Excel, SurveyMonkey, SalesForce, .csv, video and audio files and so on. The more the better.
Recognize inputs in a variety of languages. Again, the more the better.
Identify concepts in a meaningful way. There are four levels:
- The first is simple word frequency, with no filtering. Words like ‘the’ or ‘very’ could come up.
- The second recognizes nouns and discards articles and conjunctions. ‘Heart’ and ‘Attack’ would be separate topics at this level.
- The third level is multi-word topics, such as ‘Heart Attack’ or ‘Cardiac Arrest’.
- Finally, words that mean the same thing need to be grouped into a single concept. For example, “support website” and “support webpage.” This avoids long lists of concepts.
Identify sentiment to determine whether respondents refer to a concept in a positive, negative or neutral way. Sentiment analysis determines which attributes a customer associates with the concept.
Identify themes by combining concepts with sentiment and/or linguistic analysis. ‘Fatal heart attack’ or ‘Disgusting undercooked food’ might be themes. ‘Bad food’ is a different theme to ‘good food’. This last example illustrates why simple word frequency software is not useful for survey analytics. Just getting the word ‘food’ a lot will not tell a restaurant owner much.
Group similar themes to avoid excessive granularity. ‘Serious heart attack’ and ‘Major cardiac arrest’ are grouped together in this step. It may involve use of industry-specific libraries or taxonomies. Some software is delivered with such libraries or lets you add your own. Other programs use data-driven approaches to determine similar words based on how your customers use them in a sentence. Google, for example, uses data-driven Natural Language Processing to make sense of your search queries.
Finally, my experimentation so far has shown that correct handling of pronouns turns out to be very difficult indeed for software, remaining easy for humans.

Overall objective of any survey analysis process

Unless you are only interested in tracking a metric, for example to reward people, the fundamental purpose of any survey analysis is to find out what your top improvement priorities should be. The winner of any comparison should therefore be the process or tool that answers that question. The advantage of software over humans is that the only bias is that caused by the programmers. I admit that such bias can be considerable, particularly when selecting the algorithms to be used to categorize text.

A hard historic lesson on decision criteria

In all scientific experiments, it is critical to set up the methodology before you have any data. This was made very evident by changes in US rules on drug approvals. Experimenters are required to provide details of the exact methodology and data analysis process before they start the clinical trials. The results have been dramatic. If you go back 40 years to when researchers could decide how to run the analysis after they had the data, over half the trials were successful. Now the success rate is just under 10%, as people cannot remove data or change their analysis after the fact because the results do not suit them.

The seven-country study farce that affects our lives today

The most famous study that suffered from post-hoc adjustments was the legendary and highly influential ‘Seven countries study’ by Ancel Keys on the relationship between diet and heart disease by country. The study was done in 1958 and continued for 50 years. It showed a directly linear relationship between saturated fat consumption and heart disease. In 2009, Robert Lustig found that Keys had cherry-picked the seven countries from a data set that included either 21 or 22 countries. He apparently discarded the data from the 15 countries that did not support his theory. A separate piece of work also showed that Keys’ results could be better explained by differences in sugar consumption, which Keys had not studied. There is a nasty history of scientists who took positions opposing Keys losing their research funding and even their jobs.

Conclusion

These are just some guidelines that will help your selection process. Text analysis is difficult to do well, and the software for doing it is evolving quickly. I suggest including the items above among your selection criteria. Above all, decide on your selection criteria before testing the software, to avoid bias.

As always, your comments and suggestions are welcome below. There is a lot more on this subject in our book on the Net Promoter System.

March 8, 2017

Cost reduction communication mistakes I have seen

Maurice FitzGerald / Blog, Cost /

Here are some examples of cost reduction communication mistakes I have seen.

Compaq reductions in Nordic countries

Compaq went through a rare bad patch around 1999 and we needed to put some cost controls in place. A corporate leadership team member went to Norway, Sweden, Denmark and Finland to explain how serious the situation was. He explained that teams all needed to work on reducing cost. He had a major problem communicating this. It became known before his arrival that he had leased a private jet to minimize the time it would take to get around the four countries. It is irrelevant whether that saved cost in some imaginary or real way. It destroyed his message, and nobody seemed to take any particular action after he left. We then had to work on the various initiatives in detail, country by country, facing a bit more resistance than usual because of the communication mistake.

Senior executives behave as though restrictions do not apply to them

The leased private jet example is part of a wider phenomenon. The most senior people sometimes do not change their own spending habits while making “difficult decisions” that only affect others. Personally I think of this as cowardly leadership. Issuing travel freezes for your business from the comfort of your Gulfstream as you cross the Pacific is perhaps the most egregious, though unfortunately common example of a complete disconnect between leadership and employees. A G550 costs about $45 million. This is the same as 300,000 advance-booked return flights from Chicago to Atlanta on United by people who now have to jump through hoops to bring in business or cut costs due to the travel freeze.

As a general rule, the easiest way large corporations can cut travel costs significantly is to temporarily ground or totally eliminate their corporate aviation fleet. The symbolic value of such a move is also high. Employee reaction will be “Wow, we really need to get serious about reducing cost.”

Avoid cultural references when communicating internationally

I remember being with a group of young English Autonomy software employees in Cambridge when a US-based HP colleague came to visit. One of the English audience interrupted him the second time he said “We are going to hit the ball out of the park!” My English colleague asked why anyone would want to hit the ball out of the park, rather than putting it into the opponent’s goal. Similarly, you should avoid any references to television shows that are popular in your country, but which others may never have heard of. TV show names are often totally different when translated.

Understand who is present

Former US Secretary of State Condolezza Rice was a guest speaker at an HP annual kickoff event in Las Vegas. Our event team did an outstanding job with all but one single aspect of the multi-day event. What happened was really not their fault. Ms. Rice’s team prevented the event organizers having any direct contact whatsoever with her. While the team was given extensive briefings, I suppose they decided they did not want to bother Ms. Rice with all the details. There were about 1,500 of us in the room. We had come from all over the planet, representing over 100 different countries.

Quite soon after starting, our star guest said, “As you know, I spent a lot of time with leaders of countries around the world while I was Secretary of State. HP is a great company and you are great leaders. I am sure we have Italian-Americans in the room, and Irish-Americans, and even Russian-Americans…” A buzz started around the room. She clearly had no idea who her audience was, and her entire speech was off-target. The first step in preparing any communication is to understand who your audience is, in detail. If you are going to talk to a new audience about cost cuts, do your utmost to understand what they have already been told, and what they have learned through the rumor mill. If they have already been told that 20% of the jobs in their location will be eliminated, you should skip the part of your speech about future growth, and get directly to the point.

March 2, 2017

Customer-centric Cost Reduction and ‘Hygiene Factors’

Maurice FitzGerald / Blog, Cost /

Today’s post is a follow-on to my introduction to the concept of customer-centric cost reduction, back in January. You may recall that it is about how you achieve necessary cost reductions without screwing things up for your customers. It is not easy.

Various theories about human motivation include the concept of ‘hygiene factors’. Read up on Herzberg’s theory in particular if you want to know more. A link to the Wikipedia article on the subject is here. Hygiene factors can be compared to taking a shower, where doing more than a certain amount brings no additional benefit. Some hygiene factors are as basic as air. If you can breathe, you don’t notice it. Giving you more air than you need to breath normally adds no value.

Hygiene factors and delighting customers

A common customer experience slogan is “We must delight customers at every possible interaction.” While it sounds nice, it is nonsensical. Not many interactions matter. Basic service center interactions are a good example. If you have resolved the customer’s issue, that is good enough. Doing more does not add anything and wastes money.

Hygiene factors and customer-centric cost reduction

The heart of customer-centric cost reduction is understanding which of your activities just have to be “good enough”. Once they are good enough, they are not worthy of further investment. If such “above and beyond” investments have already been made, they should be cut back. They serve no purpose. There is a challenge, and that is how to determine whether the work areas in question are indeed good enough. Here are some indicators:

Common sense applies. You probably have costs that are totally invisible to customers, and which have no conceivable impact on them. For example, your customers really don’t care how much you pay for office supplies. They probably don’t care about the cost of some building they never visit. Assuming the basics function, they don’t care about your telecommunications costs. These sorts of costs can be reduced without affecting customer loyalty and growth.

Use your existing customer survey data to find out what customers care about. If you have a complaints department, you have a good source of information about things that may not be good enough, even if they should be hygiene factors. If you believe your service center hold times are fine, but your customers are irritated by them and say your competitors do a better job, then you should probably not cut costs in a way that makes the hold times worse. Note that it would also be a waste of energy to identify and communicate fifty hygiene factors. I suggest keeping your list to three to five things that are easy for everyone to remember.

While customer surveys are a great source for data on what customers care about and how you compare to competitors, not all survey types are useful for this purpose. The useful ones include open questions that ask customers what they would like to see improved, or for their reasons for giving an overall satisfaction rating of some type. If you do not have this type of question in your research, you risk cutting expenses where it is not appropriate. Many survey types require customers to choose between a set of options that you have dreamed up, but which may not reflect their reality. Think about this as the only question a waiter might ask in a restaurant: “Would you like your steak rare, medium-rare, medium or well-done?” Note that you do not get the choice of having something other than steak or being a vegetarian. If you are the restaurant owner, ask the open question “What would you like to eat today?” If nobody ever asks for a vegetarian option, you can safely remove it from your menu.

More common sense applies, in the absence of survey data. It should be obvious that a number of transactional process just have to work. If you accept credit cards, the credit card approval/denial process has to be almost instantaneous. No normal customer could be “delighted” by their credit card being approved, and nobody would recommend your company on the basis of the approval. Similarly, if your service center solves a customer issue, most customers will consider that to be acceptable, but could not be “delighted” about it. After all, they did not expect or want to have an issue to resolve in the first place. Once you believe your issue resolution is as good as that of your competitors, you should do it as inexpensively as possible. The same applies to processes associated with renewing existing service contracts, for example.

Implications for cost-reduction goals

For the sake of simplicity, lets’ suppose half your people costs are for work on hygiene factors, and half on things that genuinely make customers happy, generate recommendations and drive loyalty. Customer-centric cost-reduction means that your savings should come entirely from the hygiene factor work, with one major condition. That work has to already be “good enough”. In the absence of a a precise definition of good enough, it needs to be as good as your main competitors.

Implications for your organization

If you do indeed respect these guidelines, it will become clear to your employees that there are areas that are good for career development and others that will be relatively depressing places to work, under constant cost pressure. Your hygiene factors will not be the same as those of companies specialized in that type of work. You should consider achieving the cost-savings by using Business Process Outsourcing (BPO) services. Your hygiene factors are their core business. Your people may be happier being transferred to the outsourcing company.

Negotiating a good BPO contract

Business Process Outsourcing contracts tend to be long-term in nature. Five to seven year terms are common, and I have even seen a twenty-year contract agreement with a Swiss bank. A typical contract is for a 20% cost-reduction, usually measured in cost per transaction. For example, if you outsource the processing of expense claims, the contract will be for a 20% cost reduction per claim within a defined period of time. Standard contracts include renegotiation clauses. Save yourself the hassle by negotiating further ongoing price reductions, for example 1% per year, directly into the contract when you sign it.

Conclusion

Keeping your customers with you as you reduce cost can be challenging. I doubt you can do it by consensus. It is easier for a CEO to say “We have made some bad decisions and now need to reduce our costs by 20%. I want each and every one of you to reduce your costs by 20%. Tell me when you have completed the task.” That is all too common, and leads to loss of customers, and a corporate death spiral, as the CEO is surprised that revenue continues to decline. The guidelines I have provided can help CEOs to be more constructive: “We have to reduce our costs by 20%. Here are the top things that customers care about, and where we may have to spend more. Here are the top five cost areas that are invisible to customers and where I want to target a 50% reduction. Tell me what is possible.”

February 23, 2017

Customer research confusion reduction – There are six types of customer surveys

Maurice FitzGerald / Blog, NPS, Strategy /

There is quite a lot of confusion about which types of customer research are useful for predicting revenue, and which serve more general purposes. No matter what type of research you want to start, there is an important principle to observe: You must provide more value than you extract! Customers must get more benefit from your process than you do. You have to tell them why you are asking questions, what you have learned, and what you then plan to do.

The communication part is simple, though most companies do not do it. Implementation is harder. If you do not have the people and money necessary to act on what you learn from customer research, you should not do it at all. Those resources need to be guaranteed before you start, not afterwards.

Six survey types

Here are the six types of general customer experience research. Each survey type has specific uses.

Benchmark surveys are used to compare your performance with that of your competitors. If the survey and measurement system are chosen correctly, the trend in your performance compared to that of your main competitor will predict your relative market share. Benchmarks can be done at the level of your overall brand, as well as for individual divisions, products and services. The best are ‘double-blind’. This means you don’t know who answers, and the people answering do not know who is funding the research. Most double-blind surveys are panel-based, and panel members receive points or cash-equivalents for responding. The clear disadvantage is that there is no way of following up with an unhappy customer, since you don’t know who has responded.
Brand surveys are those where you ask your customers, partners or employees for their views on your company overall, as distinct from any individual product or service. If you can avoid influencing the responders, brand survey results trends can give you an indication of future revenue trend. However, since your brand survey does not cover your competitors, you don’t really know whether you are improving relative to them, so can’t predict revenue accurately If you improve, but they improve more quickly, you lose share over time.
Relationship surveys are in-depth research with your most important customers. While they are a subset of brand surveys, they require a specific methodology. They are particularly useful if your company has a small number of large customers that provide the majority of your revenue. The best relationship surveys are done face-to-face by a customer experience professional from your own company. They are more directly useful than benchmark surveys in that the input can be worked on directly with your customers. When relationship surveys are done by a function that is independent of the people being measured, they can be good revenue predictors for that customer segment.
Product / service / project surveys get feedback from customers about one of your individual product or service offerings. Using the example of Apple, a benchmark survey would cover the overall brand, while a product survey would be about an individual Mac or iPhone model, for example. They are not good revenue predictors.
Transactional surveys measure customer experience with various customer contact points. Examples would be satisfaction with a price quotation, an ordering process or with telephone support. They do not predict revenue.
Supplier surveys are often neglected. Many large companies survey their own people about the experience they have with a list of suppliers. The company’s procurement people are then measured on improving supplier performance in whatever dimensions are felt to be defective. If your company is covered by such a survey run by one of your customers, you are in luck. As distinct from all other survey types, the customer has ‘skin in the game’ and will want to work with you on improvements. Syndicated supplier surveys also exist.

Feedback welcome below. I would be happy to hear contradictory views.

February 17, 2017

Behavioral Economics and Customer Experience – Part 2 – Price Gouging and Revenge

Maurice FitzGerald / Behavioral economics, Blog / Revenge /

Price gouging – a particularly insidious move

The New York Attorney General has launched an investigation into Mylan Pharmaceuticals, the maker of EpiPens. An EpiPen is used to provide life-saving injections to people in anaphylactic shock, due to allergies. The reason for the investigation is that Mylan have had a monopoly on the product and have increased its price in the USA by 471% since acquiring it in 2007. The CEO’s initial public attitude to this seems to be something like “Hey… I am just running a business here.”

Daniel Kahneman has written about price fairness and the consequences of unfair pricing in Thinking, Fast and Slow. He gives examples of a hardware store that puts up the prices of snow shovels by a third during a storm, and of an employer putting down the salary of an existing employee due to competition. Very few people consider these practices to be fair. So what? So… he goes on to cover the revenge customers are prepared to take when they see an unfair practice. Indeed, not only will customers take revenge, many other people who are not customers will happily join in to punish the unfair practice and they will all feel great about it. This is what is currently happening with Mylan. Go to their website at www.mylan.com. It purports to show how caring they are. The actions do not seem to match the words.

More about revenge

Dan Ariely had an appalling experience with Audi customer service when his car frighteningly stopped working in the fast lane on the Massachusetts Turnpike. They showed no empathy and treated him extremely badly. He decided to take revenge, among other things, by including the story in his excellent book The Upside of Irrationality. The question here is whether your customers will take revenge on you if you treat them poorly. The answer seems to be that they will, and it does not take much.

The Ariely Starbucks experiment

Ariely was self-aware enough to become interested in the nature of revenge and his desire for it. He used a Starbucks and an actor to perform his experiment on unwitting subjects. The actor looked for people who were alone and about to sit down with their coffee. He asked them whether they would agree to do a five-minute task in return for five dollars. He explained the task and gave them a sheet to sign, acknowledging that they had received the five dollars for the task. There was a small stack of dollar bills on the sheet which he asked them to count before signing the receipt. He left them alone while they finished the task, counted the money and signed the receipt, and went off to search for another coffee drinker. However, there were actually six, seven, eight or nine dollar bills in the stack. It was up to the coffee drinker to decide whether to keep the extra cash or return it. This was the control group for the experiment.

The phone interruption

While explaining the task to the other half of coffee drinkers, the actor pretended to receive a twelve-second call from a friend who wanted to have pizza with him that evening. He did not acknowledge the call in any way to the coffee drinker, and just continued with the explanation after hanging up. This was the experimental group. The question was whether they would take revenge for the interruption by keeping more of the excess dollar bills on the table. And the answer is yes, they did. Rather disappointingly, only 45 percent of the people in the control group returned the extra dollars. The amount they received extra had no impact on this. However, when the actor had the fake interruption, it had a huge impact, and the honesty level dropped to 14 percent. Revenge for the annoying interruption.

The power of apologizing

Some time later, Ariely decided to investigate whether apologizing made any difference. He repeated the experiment, and added a third group. To the third group, the actor said, “I’m sorry. I should not have taken that call.” just after hanging up. That one small step completely eliminated the revenge effect, giving results that were the same as for the control group.

Conclusion

You will sometimes screw up. You will sometimes upset your customers. Your products will sometimes not work. (Even a Rolls-Royce person is supposed to have said, “Our cars do not break down. They fail to proceed.”) In such circumstances, the most powerful words you can utter are, “I am sorry.”

As always, your comments are welcome below. Please sign up for our newsletter to ensure you know about new posts. The drawing at the top is by Peter FitzGerald. Thanks for reading.

February 14, 2017

Valentine’s Day blog – How to stay married – to your customers of course!

Maurice FitzGerald / Blog, NPS / Customer Experience, Relationships /

It was Valentine’s Day, and I was puzzled by the results of some HP customer experience research a few years ago. The data source and volume were impeccable. However, the results seemed to prove something that differed from everything that I had learned about customer improvement work. Putting the data and the date together let me learn something unexpected about how to keep your customers. While relevant to both, it applies more to business-to-business than business-to-consumer.

We were trying to determine whether NPS worked for us

We were about a year into the implementation of the Net Promoter System under Meg Whitman’s leadership as CEO of HP. The time had come to work out the relationship between our Net Promoter Scores® relative to our competitors, and our revenue growth or decline. We had eight years of benchmark NPS® data and were able to match it against corresponding revenue data for most businesses. We found that NPS was a great leading indicator of revenue. One HP business decided to investigate it in more detail. That business mainly had multi-year service contracts with its customers. The question they wanted to answer was “Which customers actually leave us?” They used their own relationship survey to understand the different behaviors of the NPS Promoters, Passives, and Detractors. What they found surprised us. I believe the findings apply to all businesses that have ongoing contractual relationships with customers.

From surprise to human relationships

Remember Valentine’s Day had come around when I was dealing with the surprise. Here is how I rationalized it in terms of human relationships:

The happy-happy couples

We all know couples who seem to be made for each other. I call them the happy-happy ones. They agree on everything. They have been together for years. We admire and envy them. In Net Promoter System terms, they are the Promoters. They are like the customers for whom everything is going well. All major project milestones are being achieved early. The customers are getting more than the financial benefits they expected. They have been working with the same account team for years. Life is wonderful. The customers talk positively about us to people they know, and bring us new business.

The couples who seem to fight a lot

Most of us also know couples who bicker constantly, needle each other, and occasionally provoke major arguments. In many cases, we are surprised to note that they have been together for years and show no signs of separating. Why? Because they are communicating! They stay together because they have invested a lot in the relationship and changing seems too scary. In Net Promoter Score terms they are the Detractors. They escalate to the CEO. They get lots of attention. They get price rebates because they are angry. And, in the words of HP’s former head of Global Account sales (Jan Zadak) “I love escalations! The customers always wind up buying more stuff.”

When there are ongoing service relationships or investments in technology, the cost of change can be high. Imagine you have built all of your business processes around the way that SAP software works, but you now have a dispute with SAP. You will probably try to work it out. One of Meg Whitman’s catchphrases as CEO of HP has been “Run to the fire. Fix it in 24 hours or escalate it in 48.” I suppose it is also a guy thing that comes to the forefront in male-dominated industries. Men want to solve problems, and are willing to devote all their time to fixing a customer escalation, no matter what else is happening in the relationship.

The couples in the deadly zone of mutual indifference

Have you ever been in one of these relationships? I have, in the distant past. You get home. You think to yourself, “She is not saying anything… everything must be fine.” If you had that sort of relationship, how did it work out for you? Communication is the lifeblood of couples and of businesses with contractual relationships. Think of it this way at the very least; if you are not communicating, your competition may well be doing so. In the absence of communication, someone else smiles at your partner or customer and offers them flowers. They leave you, and you are surprised. Why did they not complain about something? What happened? In NPS terms, these are the Passives. Across most industries, they are 35 to 40 percent of all your customers.

What did the HP data show?

This particular multi-billion-dollar HP division had about a 15-point NPS decline over three years. Their revenue declined with the expected time lag. However the surprise was where it declined. Revenue with Promoters rose slightly. Revenue with Detractors stayed flat. Revenue with Passives declined close to 20%. Someone else was smiling at our Passives and sending them flowers. For the sake of completeness, I have to add that the particular business has since more than recovered the decline. I believe this data and these conclusions are representative of businesses that depend on renewable single-year or multi-year contracts. It is not helpful for pure product businesses.

The message

It is simple. Communication matters. Customers want to be remembered. If the only times your customers have contact with you are when you want to sell them something or when something is wrong, they will leave you. This could help your personal life too.

Your customer experience listening and improvement system is a crucial part of maintaining ongoing communication with the customers who are not escalating, and who are not active references for you. Just as you would never consider delegating communicating with your partner to someone else, you should manage your listening and improvement process on your own. Don’t delegate it to a third party. That would communicate something you don’t want: “I am too busy to listen to you. Talk to this person instead. That is a good use of your time, but not of mine.”

Win some, lose some

A symptom of the strength of your relationship is whether you win or lose new deals with existing customers. Understanding why you win and lose will be the subject of a future blog post.

In the meantime, please comment below and sign up for our newsletter here to stay up to date. The artwork is by Peter FitzGerald, and you can reach me at mfg@customerstrategy.net.

February 9, 2017

‘Gaming’ survey results is common – Here is how to do it

Maurice FitzGerald / Blog, NPS / Customer Experience, Surveys /

When your employees are measured individually on the results of surveys, ‘gaming the results’ is common and easy. What follows may seem a bit cynical, but has its basis in reality. If you are measured on survey results and want to manipulate them, here is how:

How to game transactional surveys:

First ensure the person you are speaking to is happy, then ask them to take the survey. This is probably easiest where a transaction is taking place by phone and the script requires the employee to ask whether the customer is willing to take a survey. Simply omit to ask the unhappy ones. This can be risky for your service center people if asking is mandatory and all conversations are recorded “for training purposes”.
Sending system-generated emails with survey requests is common at the end of service center calls. If you have the authority to update customer records, and the customer is unhappy, simply change their email address before the survey goes out to something that is made up and will bounce.
If you really need a bump in scores, replace the customer’s email address with one that belongs to you. Answer the survey on the customer’s behalf, giving yourself a top rating.
Simply tell the customer that your pay and job security depend on them giving you the maximum rating. Car dealerships do this all the time, so it must work. Since Uber drivers and passengers both benefit from a top rating of five, ‘5 for 5’ bartering is common.

How to get whatever result you want from relationship surveys:

Only ask your friends at the customer or partner to take the survey.
Pay no attention to their organization chart to find out who actually matters, favoring the people you have met.
Decide who should be asked about a major project that has been completed only after it is complete. That way you can avoid surveying anyone who has negative views. Make certain to avoid specifying the people to be surveyed in the contract before you start.

How to game your company’s product survey system:

Only ask members of your user group or another enthusiastic club of people about your product.
Exclude anyone who has a product escalation or service event in process from your survey list. Justify it by saying that you really should not bother them.
Exclude anyone who bought a software product more than a year ago and has not updated to a more recent version.
Don’t do a product survey when you know there is any problem of any kind with the product. Wait for the problem to be fixed first. Use vacation periods in another country or a long weekend as an excuse to anyone who wants to know why.
Make sure the wording of the email or other request to take the survey does not use the customer’s name, comes from a generic mailbox, is not in the local language, and makes no commitment to improve things. That way response rates will be lower. Low response rates tend to produce better scores because the people who love your products want to help you and will answer anyway. People who are less happy feel you have wasted enough of their time already and don’t bother.
Provide an incentive like entry in a prize draw for a shiny gadget to everyone who answers the survey. People won’t believe they can win if they provide negative reviews.

Of course, if you succeed in cheating like this, nothing good will happen for customers, but at least good things will happen for your metrics, and perhaps your bonus.

Gaming of scores seems to be particularly common in the automobile industry. It is the rule, rather than the exception in the USA. The General Motors head of customer experience explained it in a pubic seminar attended by one of my HP colleagues. Sales people typically need to average over 95% satisfaction in the GM dealer surveys to be able to participate in their bonus scheme. One low score can require ten perfect scores to even it out. David Mingle, Executive Director, North American Customer Experience for GM said that he knows this and did not plan to do anything about it. He said he personally pays no attention to the scores, only to the verbatim comments, saying customers tend to be honest in their written comments. The sales people are only measured on the scores and don’t care much what the customer writes about the experience.

OK, all of the above may seem a bit cynical. I have seen all of these happen in practice. They are ways of avoiding finding out what your customers are thinking. The best way to avoid survey gaming is to ensure that no single individual who can directly influence the results is measured on those same results. Collective team metrics are usually fine. Ensure the purpose of your research is clear and that employees are motivated to help customers.

Comments welcome below. Please sign up for our newsletter here.

February 7, 2017

Behavioral economics and customer experience – Part 1

Maurice FitzGerald / Blog, Strategy / Behavioral economics /

Behavioral economics is a relatively new field. It combines economic and psychological theory and reaches new conclusions. Economic theory holds that people always behave rationally, optimizing their financial and other outcomes in a totally predictable way. (I have always found this theory odd, even when I studied economics. It was never clear to me that the stock market behaves rationally since half the people “rationally” believe they should buy what the other half are rationally selling. One half must be wrong, or at least sub-optimal.) Psychologists don’t care much for the economists’ view of the world, often proving that people behave irrationally. Behavioral economics combines the two world views, giving us new insights about the way people, including your customers, actually behave.

Key authors

The best-selling behavioral economics books are probably Predictably Irrational, by Dan Ariely, Thinking, Fast and Slow by Daniel Kahneman, and Nudge by Thaler and Sunstein. Ariely has written follow-on books that I found less innovative. Kahneman won the Nobel Prize in Economics, mainly for his work on Prospect Theory, which is covered in the book.

System 1 and System 2

Kahneman’s most relevant work concerns two different ways our brains function as we think: fast and slow. System 1 is the fast way and can be thought of as intuitive, jumping to conclusions on limited data. System 2 is the slow way, and requires rational thought. He and his colleagues performed numerous persuasive experiments that show that the rational System 2 is lazy, and prefers to do no work at all, especially when System 1 has already jumped to its own conclusion. I feel the existence of these two ways of thinking also explains most of what Ariely, Thaler and Sunstein have written. You may be familiar with a particular mathematical question that illustrates the difference between the two systems perfectly. It is called the bat and ball problem. Try to think quickly and answer it within two seconds.

A bat and a ball cost $1.10.
The bat costs one dollar more than the ball.
How much does the ball cost?

Well? No matter what your final conclusion, a number almost certainly popped into your mind right away. Ten cents. Right? That answer is of course wrong. If that ball cost ten cents, and the bat one dollar more, the total cost would be $1.20. Appallingly, in large and repeated tests, over half the students tested at Harvard, MIT and Princeton got this wrong. System 1 jumped to the wrong conclusion while System 2 had a nice nap.

Relevance to customer experience

Customer experience publications and websites are full of System 1 content. By this I mean seemingly rational but wrong conclusions about cause and effect. My blog post on the importance (or not) of employee happiness and engagement for customer experience is a great example. Most practitioners believe it is among the very top factors impacting customer experience in most businesses. It is not. We just jump to that conclusion because it seems so rational that we don’t require any proof. What follows below is one example of a behavioral economics experimental result that I found particularly relevant to customer experience. I will provide more examples in the second part of this series.

The Peak-end rule

What customers experience and what they remember are different. For our purposes, the most relevant cause of the difference is what happens at the end of any interaction the customer has with your company. Let’s consider an experiment that Daniel Kahneman and his colleagues designed and called the “cold hand situation”. Participants were asked to hold their hand up to the wrist in what was perceived as painfully cold water. The actual temperature was 14 degrees Celsius, 57 Fahrenheit. (If you don’t think this is cold, get a thermometer and try it for yourself. I just did. Ouch!) The subjects were told they would have three trials, but actually only had two, in random sequence. In one trial, they put one hand in the water for 60 seconds and took it out. The other trial lasted 90 seconds and used the other hand. The first 60 seconds were identical. During the last 30 seconds, warmer water was added, taking the temperature up by one degree. The subjects were not given any information about what was happening.

The third trial

Participants were then told that they had a choice between repeating either of the first two trials. 80 percent chose the 90-second version. This is totally irrational, as the first 60 seconds are identical in the two situations. 80 percent chose to have extra pain. This is not because they were all masochists. They were predictably irrational. What they experienced and what they remembered were different. In the cold hand situation, the slightly-better last 30 seconds had huge weight, and carried the day.

So therefore…

From a customer experience perspective, this means that customers will most remember whatever happens at the end of any interaction with your company, no matter how painful or lengthy the overall experience. Let’s suppose you sell a technical product and have just gone through a lengthy telephone diagnosis and resolution process with the customer. Let’s face it, the customer believes the product should not have had a problem, so their attitude to the whole situation is not great. Rather than just saying, “Whew! It’s running again. Thank you for your patience.”, make the call a bit longer. “Let’s have a quick look at what happened next to other customers who had the same issue. Ah yes, here is how to avoid it in the future. And I can see that a lot of customers who had this issue also had this other problem. Can we take a quick look at that to make sure it is all working well?” Yes, it makes the call longer, but the end of the call is far more pleasant than the rest. Go for it.

Next week, in part 2

Next week I will write about the importance of price, and how to avoid customers taking revenge on your company. I feel the latter is quite topical, given the way consumers have used social media and other means to extract revenge on pharmaceutical companies perceived as guilty of price gouging. Remember, behavioral economics experiments often produce surprising results, and what you will read next week will be no exception.

Any comments you have are welcome below. As always, if you like what you are reading, please share it with your colleagues and friends and sign up for our weekly newsletter here. If you would like to send suggestions for future topics, you can email me at mfg@customerstrategy.net.

February 2, 2017

Survey design considerations

Maurice FitzGerald / Blog, NPS /

Edited by Maurice FitzGerald – Drawing by Peter FitzGerald

There are too many surveys in the world. Be selective. You don’t have unlimited resources. Nor do your customers. Don’t waste their time or yours. Here are some considerations that should help.

Don’t ask what you already know

You will use your customer database to determine who to contact by phone or email. You already know who they are, so don’t ask them their names. You already know what company they work for and what size it is, so don’t ask. Use the information you have to demonstrate that you remember the customer. This is less common than you might expect. I have even been asked for my email address in a survey sent to me by email. Yes, I suppose you could get to a level of creepy stalking, and that should be avoided.

Challenging when you subcontract

When you ask a third party to run your feedback process for you, it is usual that the confidentiality agreements you have in place with your customers prevent you from transferring all customer data to the subcontractor. If you think this is simple, talk to your company lawyers. To be able to transfer confidential data to a third party, the third party has to be acting as your legal “agent”. This agency status automatically includes a number of other attributes, making it rare in practice. The consequence of all this is that the company executing the feedback process for you often has to ask the customers for information you already have, wasting the customers’ time. You should be able to avoid this by providing the subcontractor with a unique code for each client, then matching the code back to your full set of customer data once you have the results. Not easy, but doable.

Tricky vendor relationships

Survey service providers often work on a convincing pitch saying that customers will provide more honest answers to a “neutral” third party. There are no studies that support this point of view, and I can’t see why that would be the case. I strongly believe you should design and run your own surveys. A third party will never be able to understand the necessary product and service vocabulary as well as you and your customers. There is one exception, and it is competitive benchmark surveys, which must be run by a neutral third party.

Try different things

If, for example, you have thousands of customers for a particular product and want to solicit their feedback and improvement suggestions via email, randomly try different formulations of the email to see which gets the the best response rate. Carefully test and measure the effectiveness of different survey questions. Test any new customer experience process in one business before implementing it everywhere. Experiment, experiment, experiment.

Keep it short

Keeping surveys short affects two things in a positive way: response rates and completion rates.

Response rates

There are many different factors that affect response rates, and survey length is not at the top of the list. The main thing that affects response rates is your relationship with the people on your contact list. If you write an email in your name to customers you know personally, you will get a relatively high response rate, no matter what the length of the survey. If you are trying to survey people who use a competing product and have never bought anything from your company, response rates are likely to be low. They have already proven they they don’t care much about you, so why should they waste their time? There is no good, recent academic research on the relationship between the number of questions on a survey and the proportion of people who answer it, everything else being equal. It should be quite easy for a large company with a lot of customers to run an A/B test, asking half of a large mailing list to answer a different survey from the other half. For the time being, you will have to rely on common sense and just trust that shorter surveys help.

Completion rates

For email based surveys, SurveyMonkey has done an outstanding piece of work to determine the effect of adding a question on what are called drop-out rates. A drop-out is a person who responds to at least one question, but does not complete the survey. SurveyMonkey looked at 2,000 surveys with one question, 2,000 with two questions, and so on all the way up to 2,000 surveys with 50 questions. The graph below shows what they found. Personally, I was surprised that so few people failed to complete the longest surveys. Still, the message remains that the shorter the survey is, the better the result.

Fig. 1 – Result of SurveyMonkey research on relationship between survey length and dropout rates

It is tempting to keep surveys very short to improve completion rates and avoid wasting customers’ time. However, I have yet to see a survey where demographics did not make a difference. Sometimes the differences are surprising.

What we mean by Demographics

Demographic information is that which you use to categorize responses. Again, you should never ask people to provide the categories if you already have the information. For web-based surveys, the categories you don’t already know about are normally provided using pull-down menus. Examples are things like these:

For product surveys, responses may vary depending on whether the respondent is a purchasing decision-maker, an influencer or an end user. Self-categorization.
For employee surveys, whether the person works from home, the office or is always on the road may matter. Perhaps oddly, many employee surveys seem to suggest that people who work from home are happier.
For a survey about a software product, it is useful to know what version of the software the customer is using. People using older versions of software may provide improvement suggestions that have already been implemented in the more recent versions. It is then easy to provide instant feedback and make them happy.

The consistent demographic surprise

Even in B2B environments, women often provide different input to men. You should be able to do the demographic split well enough using people’s names. In HP’s research on consumer printers, women mentioned ease of use and overall quality twice as often as men as a reason for recommending the printer. Men mentioned the price of supplies, low ink consumption and value for money twice as often as women. In short, the survey showed that women care about different things than men. It changed the messaging and marketing we used, as well as the training given to people in retail stores.

An example from a student survey

A Swiss business school did a simple NPS survey among its students. Once again, women and men scored quite differently depending on where they lived. Women living at home were far happier than male students living at home. This was not expected. It was pure chance that the survey owner decided to look at gender as a factor.

Fig. 2 – Student survey example

I suppose one other entertaining factoid from this survey is worth mentioning. While the beauty of the campus, with amazing views of the Alps and Lake Geneva was in the top three reasons students liked the school, half as many Swiss people mentioned it as French students. Too blasé, I suppose.

Action orientation

If you don’t already have the demographic information but it is of no conceivable use in an action plan, please don’t ask. Let’s suppose you want to know what should be improved in your commercial air conditioning products. Asking the respondents what age they are is of no conceivable relevance.

An important factor in surveys is asking questions that let the customer provide feedback on what they think is important, not what you think is important.

The British Airways counter-example

As a frequent flyer, I have answered many airline surveys over the years. The British Airways survey process is the worst I have ever seen in any industry from any company. I am confident they believe it is wonderful, or they would have changed it by now. It certainly gives them plenty of metrics to use to beat up their teams. Here is a screen capture from one survey. These are the top-level summaries. The cabin crew section, for example, has sixteen questions.

British Airways survey example

The main issues with these types of surveys are:

If you are the company who is running the survey, you will tend to believe that each one of the 100+ questions is equally important.
No distinction is made between things that are hygiene factors and things that give you a competitive advantage. Hygiene factors just have to be “good enough”, and it is a waste of resources to make them better. While it is not an airline example, Rob Markey likes to use credit card approvals in stores as an example. You expect it to work. It cannot delight you. You won’t notice it at all unless something goes wrong.
There is no screen where you are directly asked what you want British Airways to improve. That is the only conceivable way to decide which of these many questions actually matter.
The number of questions means some people won’t start the survey at all, and there will be slightly more drop-outs than with a shorter survey.
The survey is run by a third party who guarantees anonymity, so there is no way for British Airways to follow up with you.
They do not provide any information about anything they have learned from past surveys or any improvements they have made. Putting aside the lack of possibility of individual follow-up, there is no commitment to tell those surveyed collectively what they have learned.

What should it look like?

If I were designing the survey, here is what it would probably look like:

First, the email, personally addressed to me, would read, “Dear Mr. FitzGerald, thank you for using British Airways for your flight from Geneva to London and onwards to Denver on October 22nd. We are trying to improve our long-haul service. We can see that you checked in online, had one bag in the hold that arrived with you, and that you visited our lounge in Heathrow Terminal 5 while you were in transit. Fortunately, your flights were all on time, and the crew did not report any issues with the in-flight entertainment system. We see that you had a special meal and hope you enjoyed it. We would now like to ask you just four questions about your trip on October 22nd. The link below will take you to the secure feedback form. The survey should take you no more than three minutes. I hereby commit to tell you what we have learned from your feedback and that of other long-haul customers by November 30th at the latest. Now, please help us to improve.” OK, this example may cross the ‘creepy’ line, but my point is simple: don’t ask things you should already know. An airline asking about a particular flight should already know whether it was on time or not, for example.
The survey questions would be:
- How likely are you to recommend British Airways to a colleague or friend? [On a 0 to 10 scale]
- Have your flights from Geneva to Denver on October 22nd made you more or less likely to recommend BA?
- Why?
- What should we improve?

I can’t think why they would find it useful to ask anything else. If I don’t mention the food and drink on the flight, they cannot have mattered much to me. Note that the email is highly personalized and uses data the airline already has. I think the reason airlines often ask for the data they already have is that confidentiality rules mean they cannot provide it to a third party. Another great reason to run your own surveys.

Your comments are welcome below. Please sign up for our newsletter to stay up to date with blog posts and other news.

January 31, 2017

Best and worst combinations of customer and employee satisfaction

Maurice FitzGerald / Blog, Strategy, Uncategorized /

Edited by Maurice FitzGerald – Drawing by Peter FitzGerald

Deeper dive into ACSI and Glassdoor data

Following on from last week’s post on the relationship between employee and customer satisfaction, I am now going to provide four different cuts of data, at least one of which should be a major surprise. The four cuts are:

Companies that are in the top 10% for both employee and customer satisfaction.
Companies in the bottom 10% for both.
Companies in the top 20% for customer satisfaction and the bottom 20% for employee satisfaction.
Companies that are in the top 20% for employee satisfaction and the bottom 20% for customer satisfaction.

There is a call to action after the data and surprises.

Data sources, defects and notes

As discussed in last week’s post, there are not many public sources of data on these subjects. The best information I can find on customer satisfaction for large companies selling to US consumers is ACSI. The American Customer Satisfaction Index team gets input from 70,000 consumers on over 300 companies. Some companies, such as Apple and Samsung, have more than one business and there are scores for each.

The only public source of employee satisfaction data I have found for the same companies is Glassdoor. Current and former employees of a company provide an overall rating for their employer. Some companies do try to game Glassdoor and some scores may therefore be overly optimistic. That does not seem likely for the companies at the bottom of the list. Glassdoor covers 600,000 companies, right down to tiny businesses. Very few of them are included in the ACSI research, which concentrates on large companies selling to US consumers only.

Wherever more than one choice was available for a business covered in the tables below, I have selected the most positive available data. I have triple-checked every number, and still welcome any error detection you may like to carry out.

Since the ACSI data is for all of 2016, I have generally used Glassdoor numbers I researched in mid-2016. There are some exceptions where the January 30^th 2017 number is better and I have noted those cases. There are also two companies for which the ACSI number for 2016 is not yet available, and I have used the 2015 numbers. Both are in the ‘best of the best’ table below.

Best publicly-available data

If I had a choice, I would far prefer to use customer and employee Net Promoter Scores, but there is no public source for such data at all for eNPS, and public NPS scores have limited company coverage. There is no standard for measuring employee engagement. Most people promoting an employee engagement index seem to be promoting a proprietary methodology, software or consulting services. Yes, they all seem to be able to prove that their index predicts customer satisfaction and financial results, and I believe that this should indeed be the case. An agreed open standard backed by research would be nice.

I suppose I should once again note that some of the companies in the table below do not sell directly to end consumers. Their customers are resellers, brokers or other intermediaries, for whom no satisfaction data is available.

Method

I have used the data in the Excel file that you can find on our resources page here. It is simply a copy of what was on Glassdoor and the ACSI site at the time the data was prepared. The steps I followed were:

Rank the 340 ACSI scores from highest to lowest.
Establish the ‘Top 20%’ or quintile marks by counting down 68 rows from the top and 68 rows up from the bottom.
Repeat for the top and bottom 10%.
Use the conditional sorting function in Excel to produce four tables, using various combinations of these ranges.

Here is what this gives as the ranges. Feel free to reproduce these steps to do your own analysis.

The best of the best

At the considerable risk of using ‘the D word’, here are the companies that delight both their customers and their employees. These companies are in the top 10% on both the ACSI and Glassdoor rankings.

Chick-fil-A’s ACSI position should surprise nobody. It is great to see that they are also in the top 10% of Glassdoor ratings. There is no Glassdoor rating available for Lincoln automobiles on their own, so I have used the overall Ford number. I have also combined Toyota and Lexus, as there is only a single Glassdoor rating for Toyota North America. As noted last week, consumers may be dealing directly with franchise operators, rather than Ford or Toyota employees. Apple has an ACSI score of 80 for its phones, and I have used the overall Glassdoor score for Apple, as no separate number for the PC employees is available. Wegmans and Trader Joe’s ACSI numbers are from 2015. I will update the table as soon as 2016 numbers are available. Subaru Glassdoor rating is for Subaru of America, as of mid-2016.

The bottom 10%

Let’s be optimistic: these are the companies with the greatest improvement opportunities! They are the only companies in the bottom 10% of the 340 businesses for which I have both ACSI and Glassdoor ratings. To be as clear as possible about this, there are 34 businesses in the bottom 10% in each rating system. These are the only ones that are in the lowest 10% in both. Mediacom has had the worst ACSI score in both 2015 and 2016. This positioning of cable TV and ISP providers confirms what Bruce Temkin has found in his research.

ACSI rates Windstream and Mediacom for two different businesses, while Glassdoor has a single overall score.

Loved by customers, but…

Here is the list of companies that are in the top 20% of the ACSI ratings, while being in the bottom 20% of the Glassdoor ratings. It is an interesting combination of four companies where cost-control is a constant concern, and two Internet investment firms where the employees have little contact with customers. Since Vanguard is known for strong customer satisfaction, I was surprised to see that employees do not seem to receive the same attention. I certainly accept that this could be a case where the employees are not very happy, but are engaged.

Employees love working there, but…

This is the category with the big surprises. These companies seem to have the unfortunate combination of happy employees and unhappy customers. Each is in the top 20% of Glassdoor ratings, while being in the bottom 20% of of the ACSI scores. I can find no evidence that any of these companies is gaming the Glassdoor scores. Glassdoor does not allow companies to remove negative reviews. Perhaps a little less happiness might motivate these employees to look at things from more of a customer perspective. Most of the companies seem to have monopolies in their chosen business segments or geographies. Perhaps that makes them care a little less. Other than the airlines, they are also companies where employees have little or no contact with customers, so there is little opportunity for their enthusiasm to become contagious.

Facebook had an ACSI score of 75 in 2015, so would not have been on this list then. I used Consolidated Edison New York for the Glassdoor score.

Conclusion and call to action

Medium- and long-term business success can only be achieved by engaged employees who improve customers’ overall experience with their employers. There is no common standard or public data for employee engagement, so I have used a ‘second-best’ approach. I accept that happy employees are not necessarily productive employees. Companies can do well on Glassdoor because they have great food in the canteen, on-site child care and attractive vacation policies. None of those contributes directly to positive customer outcomes. A happy employee is not necessarily doing positive things for customers.

Companies that have high employee satisfaction but low customer satisfaction will be forced to improve if they can see competitors taking share from them. Some companies have effective monopolies, and probably won’t change quickly. They should bear in mind Willie Pietersen’s words in a Columbia lecture I attended. “Somewhere in the world, somebody has just had an idea that can kill your business.”

As I argued in last week’s blog, if you were to take a clean sheet of paper and brainstorm all the things that can affect customer satisfaction, employee engagement and / or satisfaction would probably be on the list, but not at the top. A railroad company, for example, would be likely to find that on-time arrival of trains and the availability of seats at rush hour matter more to customers than their very occasional contact with an employee. Motivated employees are of course enablers of many other things, such as getting the trains to run on time. The most important thing companies can do is to listen to customers, internalise what they want to see improved, and engage their entire workforce to make it happen.

Thank you for reading. Please sign up for our newsletter here, and spread the word to your colleagues and friends.

Maurice FitzGerald mfg@customerstrategy.net

«< 13 14 15 16 17 >