June 21, 2005
The Mystery of Mystery Shopping
Mystery shopping provides a real cost saving opportunity for most organizations. Simply put, most organizations would be better off not spending any money on mystery shopping because the data they collect is of limited use and can be misleading. Too often, mystery shoppers are easy to identify, are not representative of your customers, and conduct so few shops that the data received from mystery shopping doesn’t allow for detailed trend analysis.
Yet companies continue to spend $15 to $25 per mystery shop to get data that is of limited use and don’t routinely collect data from customers. They put more stock in the evaluation of people who don’t have a relationship with them or significant experience with their products and services rather than asking real customers to identify opportunities for improvement that impact loyalty and future purchases. The focus on mystery shopping rather than collecting real customer data has always baffled me. Does it make sense to you?
If you are unconvinced that mystery shopping has limited usefulness, here are some facts from a real customer situation that illustrate the point . . .
A regional bank with about 50 branches, 1,000 employees (500 of these employees are customer contact employees in the branch) and 130,000 customers uses mystery shopping as their primary means of customer data collection. The mystery shop focuses on Tellers, Customer Service Representatives and Location Appearance. Each branch is shopped at least once a month; larger branches may get shopped twice a month.
Here are some numbers for you to consider:
· With 50 branches getting shopped once a month (one Teller and one CSR get evaluated per shop), the minimum number of shops per year is 1,200. If you assume half the branches get shopped twice a month, then the number of shops per year is 1,800.
· With 130,000 customers, if we assume one branch or telephone service encounters per month, that equals 1.56 million transactions per year.
· With 500 CSR and Tellers, each person is getting shopped about once every three months. Over those three months, each person will handle about 780 transactions.
· Mystery shops are 0.11538% of the banks total transaction volume. For this data to be reliable, a good rule of thumb is that you need to collect data on 5% to 10% of total transactions, or 50 to 100 times what this organization is doing!
Still not convinced? Here are some other limitations to mystery shopping:
· You can’t link the data you get to your customer segments.
· The sample size is not large enough to make strategic or tactical changes. The feedback that you give employees based on their shop scores is not representative of their overall performance.
· Your customers have a vested interest in your business success—do mystery shoppers have the same level of interest and do they reflect the expectations of your customer base?
· Mystery shoppers can’t tell you what changes can be made to get more of their business or how likely they are to remain as customers. Without this data, it is impossible to identify which performance expectations actually drive customer behavior.
Mystery shopping, in and of itself, is of limited value. If you have a mystery shopping program in place and you cannot get rid of it, we recommend that you scale it back to focus on evaluating tangibles like physical appearance and traffic flows. These items have less variability than delivering the customer experience.
In high transaction volume businesses it is risky to drive strategic business decisions based on data extrapolated from individuals who are not customers, may not be representative of your key customer segments, and represent a very small proportion of customer touches to drive strategic business decisions. We believe that mystery shopping dollars will have a greater impact if used to measure overall customer satisfaction and loyalty of key customer segments on a consistent basis and check satisfaction with customer transactions over time.
Poll Closed
Posted by Greg Robinson at June 21, 2005 11:13 AM
View Printer Friendly VersionComments
Greg here are my thoughts;
Mystery shopping is a valuable tool. Mystery shopping enables a company to easily measure and report the execution of established service standards out in the field. I see it as a quality assurance tool. I do not view mystery shopping as a source for customer attitudes or perceptions. Mystery shops should be categorized as a source of quantitative data not qualitative.
Mystery shopping should be combined with the following.......................................
***A company vision/mission/strategic statement which encompasses service. An emphasis on fanatical customer service must be part of the culture.
***Research of industry benchmarks/customer expectations.
***A focus on employee satisfaction which I will not elaborate on as this is a discussion on its own.
***Clear, concise and uncompromisable service standards that have to make sense to EVERYONE in the company.
***A properly designed service training curriculum with courses tailored to the company hierarchy. Yes that includes EMC.
***An ongoing method for qualitatively "pulse checking" your service by surveying your customers. There are many different options and Greg probably knows them all.
>>>After doing all of the above and establishing the "must do's of service at all your Point of Contacts/Service/Sales" then it is time to quality check and that is where Mystery Shopping becomes a very valuable tool.
Ramses
Posted by: at June 21, 2005 06:57 PM
I believe Mystery Shopping can be successfully used as a marketing & sales tool. First, you need to let your staff know in advance that each month 1 or 2 persons who walk through the doors are going to be shoppers who will be evaluating them and sending in a report. This raises the staff awareness of the company's strong desire to make sure each customer is handled as well as possible. And, by letting them know in advance, it demonstrates that the company is strongly interested in "catching its staff doing something right." This is especially successful when an award and recognition program is combined with the Mystery Shopping. Properly done, and with enough successes and training for those who didn't perform as well, it can raise the esprit de corp.
Posted by: at June 27, 2005 09:17 AM
Along with the reasons Ramses and RWM have given, it is a mistake to assume that mystery shoppers are not customers. With the way mystery shoppers are recruited, they are often shopping businesses in their local community with whom they do have a relationship, so they are not ignorant of the business or their processes. Also, surveying customers can be misleading. The ones who are motivated to answer are often the disgruntled customer, a relatively happy customer just does not care enough to take the time. They just want to do the transaction and get on with their day. There is also a percentage of respondents who will deliberatly falsify their comments.
Posted by: at June 27, 2005 11:03 AM
1) Mystery Shopping is not meant to be the sole method of evaluating customer service but rather used in conjunction with customer surveys and other methods.
2)The main advantage of Mystery Shopping, if done right, is that it can get much more detail (depth) than customer surveys because of the length of time per observation vs. a quick survey. What it lacks is the representativeness of a larger sample of observers under a wider variety of condidtions and time. (width)
3) What is the basis for the comment about it taking a 5% to 10% sample of all transactions to be reliable? National polling, typically done with samples of 2,000 to 4,000, represents less than 0.00002% of the adult population yet they can be pretty accurate.
Posted by: at June 27, 2005 11:26 AM
Ramses and RWM, thanks for your posts. Mystery shopping as a quality assurance method only works if you sample enough transactions to understand the source of the variability in performance. Too often, mystery shopping is not used to "catch someone doing something right" but to punish people for doing something wrong. I a recent conversation with a manager at Washington Mutual who was a number of branches reporting to him, he chuckled at the notion of anonymous mystery shoppers. Apparently word gets around pretty fast as to who is doing the shopping, thus they get treatment that is better than the norm. Is this a flaw with the concept of mystery shopping? Not necessarily. But this does tend to be the norm in execution.
Posted by: Greg at June 27, 2005 01:37 PM
JT, thanks for your post. Some people are doing a better job of recruiting shoppers--if you are using a mystery shopping company and they do not give you actual customers then you should not use them. The benefit of actual customers rather than mystery shoppers is that you can do detailed segment analysis that mystery shopping typically does not allow you to do.
Our experience with survey respondents is that you get happy, unhappy, new, and emotionally connected customers to respond. And the vast majority that respond are motivated to improve the companies they do business with so they tell you the truth. Given our belief that the whole point of research is to find out what the problems are that are negatively impacting customer satisfaction and loyalty (as opposed to get a score that makes you look good), those customer that respond and tell you their problems are worth their weight in gold.
Posted by: Greg at June 27, 2005 01:44 PM
David,
National polling and customer satisfaction research are not the same animals. Frankly, national poling is pretty much of a joke (one of our senior partners used to do poliical poling). The reason it seems to be accurate is that the polls only give you two choices.
With customer satisfaction research, there are so many more variables that impact buyer behavior that you need a much larger sample to get an accurate picture. If you don't have a least 100 cases (and at least 1% of total events) per segment you are interested in, move forward at your own risk.
If mystery shopping data is getting you a representative picture, great. Just make sure that you have enough real data from your most important customers before taking action.
Posted by: Greg at June 27, 2005 01:51 PM
Greg here are my thoughts;
Mystery shopping is a valuable tool. Mystery shopping enables a company to easily measure and report the execution of established service standards out in the field. I see it as a quality assurance tool. I do not view mystery shopping as a source for customer attitudes or perceptions. Mystery shops should be categorized as a source of quantitative data not qualitative.
Mystery shopping should be combined with the following.......................................
***A company vision/mission/strategic statement which encompasses service. An emphasis on fanatical customer service must be part of the culture.
***Research of industry benchmarks/customer expectations.
***A focus on employee satisfaction which I will not elaborate on as this is a discussion on its own.
***Clear, concise and uncompromisable service standards that have to make sense to EVERYONE in the company.
***A properly designed service training curriculum with courses tailored to the company hierarchy. Yes that includes EMC.
***An ongoing method for qualitatively "pulse checking" your service by surveying your customers. There are many different options and Greg probably knows them all.
>>>After doing all of the above and establishing the "must do's of service at all your Point of Contacts/Service/Sales" then it is time to quality check and that is where Mystery Shopping becomes a very valuable tool.
Ramses
I believe Mystery Shopping can be successfully used as a marketing & sales tool. First, you need to let your staff know in advance that each month 1 or 2 persons who walk through the doors are going to be shoppers who will be evaluating them and sending in a report. This raises the staff awareness of the company's strong desire to make sure each customer is handled as well as possible. And, by letting them know in advance, it demonstrates that the company is strongly interested in "catching its staff doing something right." This is especially successful when an award and recognition program is combined with the Mystery Shopping. Properly done, and with enough successes and training for those who didn't perform as well, it can raise the esprit de corp.
Along with the reasons Ramses and RWM have given, it is a mistake to assume that mystery shoppers are not customers. With the way mystery shoppers are recruited, they are often shopping businesses in their local community with whom they do have a relationship, so they are not ignorant of the business or their processes. Also, surveying customers can be misleading. The ones who are motivated to answer are often the disgruntled customer, a relatively happy customer just does not care enough to take the time. They just want to do the transaction and get on with their day. There is also a percentage of respondents who will deliberatly falsify their comments.
1) Mystery Shopping is not meant to be the sole method of evaluating customer service but rather used in conjunction with customer surveys and other methods.
2)The main advantage of Mystery Shopping, if done right, is that it can get much more detail (depth) than customer surveys because of the length of time per observation vs. a quick survey. What it lacks is the representativeness of a larger sample of observers under a wider variety of condidtions and time. (width)
3) What is the basis for the comment about it taking a 5% to 10% sample of all transactions to be reliable? National polling, typically done with samples of 2,000 to 4,000, represents less than 0.00002% of the adult population yet they can be pretty accurate.
Ramses and RWM, thanks for your posts. Mystery shopping as a quality assurance method only works if you sample enough transactions to understand the source of the variability in performance. Too often, mystery shopping is not used to "catch someone doing something right" but to punish people for doing something wrong. I a recent conversation with a manager at Washington Mutual who was a number of branches reporting to him, he chuckled at the notion of anonymous mystery shoppers. Apparently word gets around pretty fast as to who is doing the shopping, thus they get treatment that is better than the norm. Is this a flaw with the concept of mystery shopping? Not necessarily. But this does tend to be the norm in execution.
JT, thanks for your post. Some people are doing a better job of recruiting shoppers--if you are using a mystery shopping company and they do not give you actual customers then you should not use them. The benefit of actual customers rather than mystery shoppers is that you can do detailed segment analysis that mystery shopping typically does not allow you to do.
Our experience with survey respondents is that you get happy, unhappy, new, and emotionally connected customers to respond. And the vast majority that respond are motivated to improve the companies they do business with so they tell you the truth. Given our belief that the whole point of research is to find out what the problems are that are negatively impacting customer satisfaction and loyalty (as opposed to get a score that makes you look good), those customer that respond and tell you their problems are worth their weight in gold.
David,
National polling and customer satisfaction research are not the same animals. Frankly, national poling is pretty much of a joke (one of our senior partners used to do poliical poling). The reason it seems to be accurate is that the polls only give you two choices.
With customer satisfaction research, there are so many more variables that impact buyer behavior that you need a much larger sample to get an accurate picture. If you don't have a least 100 cases (and at least 1% of total events) per segment you are interested in, move forward at your own risk.
If mystery shopping data is getting you a representative picture, great. Just make sure that you have enough real data from your most important customers before taking action.
