8 November 2023 792 words, 4 min. read

Mystery calling: a faithful reflection of customer satisfaction?

By Pierre-Nicolas Schwab PhD in marketing, director of IntoTheMinds
Scientific research sheds light on the limitations of mystery calling/mystery shopping for measuring customer satisfaction.

Mystery shopping and mystery calling have become widely used techniques by market research agencies to assess customer satisfaction. This technique is mainly applied in a B2C context. While the methodological approach seems justified, it is based on several assumptions and limitations, which we decipher in this article.

Entrust us with your customer satisfaction measurement project

Mystery shopping and customer satisfaction: All you need to know in 30 seconds

  • The use of mystery shopping/mystery calling techniques assumes that measurements taken by professionals will be consistent with those taken by real customers.
  • This assumption has been challenged by scientific research in the Netherlands.
  • Mystery shoppers tend to assess the NPS (Net Promoter Score) more severely than real customers. The difference is 0.5 points.
  • Real customers have a significantly higher level of gratitude than mystery callers.
  • There are no significant differencesin satisfaction between real customers and mystery callers.

Mystery shopping, and its telephone form “mystery calling,” is a widely used market research technique. It aims to assess and improve customer service quality. The technique is used in various sectors, including the hotel and restaurant industry, financial services, transport, retail, and health services. This method enables companies to:

The mystery shopping/mystery-calling industry has grown steadily over the years. Today, it represents a $2 billion industry employing 1.5 million people worldwide.

mystery calling hypothesis

The hypothesis that calls into question the value of “mystery” approaches

The value of mystery shopping and mystery calling for measuring customer satisfaction is based on a dual hypothesis:

  1. mystery callers” and “mystery shoppers” would faithfully represent the behavior of real customers
  2. They would provide representative assessments of how consumers feel about their satisfaction.

Of course, this assumption needs to be clearly stated.

Companies that opt for mystery shopping/calling are, therefore, unaware of it. Beyond the customer, it’s a safe bet that the research firm itself isn’t even aware of it.

What if this hypothesis isn’t confirmed? What if mystery shoppers underestimate or overestimate customer satisfaction? This is what we propose to discuss today, based on scientific research conducted at the University of Groningen (Netherlands).

mystery calling market research

The advantages of mystery shopping as a market research technique

Mystery techniques make it possible to:

  • Assess customer contact experiences
  • Take advantage of participatory observation
  • Experience a service situation first-hand
  • Monitor how processes and procedures are applied

These are undeniable advantages.

But there are others:

  • Customers are often unable to remember details
  • Customers don’t often give feedback
  • Mystery customers are trained to be objective
  • Mystery callers can gather factual information during a call

mystery calling real customers

Do mystery callers resemble real customers?

The authors of the research conducted two studies in a call center. They, therefore, focus on mystery calling as a market research technique, leaving mystery shopping to one side.

The first research aimed to compare the results of customer satisfaction as assessed by real customers with the scores awarded by “mystery callers.” This assessment was conducted based on 3 criteria:

  • Overall customer satisfaction
  • Gratitude
  • Word-of-mouth assessment based on the Net Promoter Score.

The second research aimed to research a group of mystery callers and determine the origin of differences in judgment.

mystery calling real customers

There are significant differences between mystery callers and real customers

The first research delivered an unexpected result, uncovering significant differences between real customers and mystery callers regarding “gratitude” and “word-of-mouth.” Specifically, mystery callers systematically underestimated these factors. They were, therefore, harsher than real customers. Regarding word-of-mouth, the difference was relatively large (over ½ point on the NPS scale).

As far as measuring customer satisfaction is concerned, the research shows no significant differences. This means that mystery calling delivers results in line with what real customers think. This is good news.

Mystery callers differ from one to another

The 2nd research sought to understand whether the origin of the variations was within the mystery caller group itself. Here, too, the results were surprising. The following factors significantly influence the satisfaction scores given by the mystery callers:

  • Gender
  • Experience
  • Age
  • Whether the mystery callers were themselves customers of the company being assessed

When mystery callers were the company’s customers, they tended to be more positive in their judgment than their non-customer colleagues. If you’re running a mystery calling campaign, it’s a good idea to check this with the people making the calls.

mystery calling discretion

Use mystery calls with discretion

Results show that mystery callers are biased regarding word-of-mouth and gratitude. As a result, the overall customer satisfaction score may need to be more representative. Remember that around 70% of companies that measure customer satisfaction extensively use NPS. This research shows a difference of almost half a point between real customers and mystery callers.

Our advice is to consider the various limitations of mystery calling when interpreting the results.

Posted in Marketing, Research.

Post your opinion

Your email address will not be published. Required fields are marked *