Mind the gap: A case study about reported and measured behavior on mobile devices

Do you really know how many times you use your mobile per day?

If you think you can answer this question with 'yes', you might be one of the few! According to our most recent research, 82% of the people don't know how many times they check their phone per day.

Imagine you are in line at the supermarket and you have 20 seconds to spare. Odds are you will take a brief glance at your phone to catch up on your email, Facebook or favorite news site.

Changing consumer behavior results in changing requirements for market research

It’s not only the shift to mobile or the usage of multiple devices. Devices are seamlessly integrated into people's lives – they check them frequently, quickly and unconsciously which creates an increasing amount of micro-moments and data. This 24/7 availability of the digital world has led to a decrease in people’s attention as well as to an increase in distraction.

The growing amount of digital events makes it all the more important to understand consumer behavior, but at the same time more complicated than ever before. However, traditional surveys are still the main tool in market research even though they strongly rely on the human memory to report complex online behavior.

We executed the following research to examine if the recollection of these events is accurate and if this reflects true mobile behavior.

Research objective

Our research objective was to explore how consumers aged 20-35 act on their tablets and smartphones, and whether there is a difference between what people recall and report doing on their mobile devices and their actual, measured behavior.

Research execution

Panelists were recruited from Respondi's online panel in the UK. 301 panelists completed the screener. After three weeks of passively measured mobile app and mobile web usage, 178 respondents completed the survey on their mobile behavior in the past 2 weeks.

Research Results

The survey contained questions across 5 main categories. The analysis of the survey and the behavioral data delivered meaninful insights for each category.

Let's have a look at some big numbers first:

Results

Time for some fun facts:

Less generic insights were also generated using the behavioral data:

  • The most played games, in terms of seconds, were modern classics Candy Crush Saga and Farm Heroes Saga.
  • Good old Solitaire was the most opened game in count of occurrence.
  • One male participant was “trying to catch them all”, playing Pokémon Go for over 3 hours in one single session.

On average people use mobile gaming apps nearly every second day. At these days, they spend 45 minutes per day with mobile gaming.

Let's dive a bit deeper into the details:

Miscalculation in frequency

As mentioned above, 82% of the participating people don’t know how many times they use their mobile device per day. 34% of them miscalculate their usage by one interval e.g. instead of 41 to 60 times a day, they use it 61 to 80 times. This means, that there are still a lot of people left who have a bigger difference between their estimation and actual usage.

Under-reporting in variety

On average people use 100% more apps than they report. The average number reported is 6, but they actually use 12 different apps. This number is surprisingly low, which might have to do with the fact that people, for example, don't perceive the phone or alarm function as an app.

However, this fact might not be crucial since it doesn't apply for visited mobile sites. And here, the gap is even bigger: We measured 250% more visited mobile sites than reported (reported: 4, measured: 14) – even though in terms of general mobile device usage apps clearly dominate compared to mobile sites.

Misjudging the heavy-usage time

When giving people 6-hour intervals to estimate the time they use the device the most, 50% assign a wrong period. Almost two thirds of them think they use their device most frequently later in day than they actually do. Of these people, 94% reported their most frequent period of activity as 'evening' (06.00pm-11.59pm) while it was either 'morning' (24%, 6.00am-11.59am), 'afternoon' (64%, 12.00pm-05.59pm) or 'night' (12%, 12.00am-00.59am).

The difference in frequency and duration

We compared the frequency to the duration of device usage to see if there are differences and/or similarities in the estimations people make.

How much time do you spend using apps? duration legend

When asking people about the duration, a lot of people overestimate the time they spend using apps. Looking into more details across the categories, the accuracy of social app usage is a lot worse than for news or banking – we can assume that people use social apps more on the side than banking or news apps. But still for news apps, half of the people report a wrong interval.

How often do you use apps?

frequency legend

Compared to the duration, more people underestimate their usage in regards to the frequency. Again, the news category performs best. Across all surveyed categories, only 19% of the people are right in terms of frequency – compared to the duration (45% in the above figure), a lot worse.

While people tend to overestimate the duration they spend on their mobile device, they underestimate how often they use it. This supports the idea that devices are so fully integrated into the daily lives of consumers that they even don't notice how often they check them.

Misestimation in top apps and mobile sites

When asking people about the apps and mobile sites they use the most, it is almost alarming which small share of their most used apps and sites they are are able to report correctly.

Which are the 5 apps you use most often?

apps

No one can report all of the 5 most used apps. Even 10% of the people are not able to recall any of them. On average people can only report 1.96 out of their 5 most used apps correctly.

Taking this one step further, the analysis of the passive metering shows that unsurprisingly Facebook and WhatsApp are the most used apps. This almost goes without saying. But, if people are able to report these two apps only, what about the rest?

Which are the 5 mobile sites you use most often?

sites

Also in this regard the performance of mobile sites is worse. There is not a single person who knows 4 out of their 5 most used mobile sites. Almost one third don't know any. On average only 1 out of 5 mobile sites is reported correctly.

One Person, two different stories!

We'd like to demonstrate with an example what this could mean for the research results and how different data sources might even lead to different conclusions.

Example

The left device shows the reported apps of one participant using Android:

  • Messenger
  • Reddit
  • Instagram
  • Chrome
  • Facebook

On the right we see the most used apps that are passively measured from the same participant:

  • Netflix
  • Pokémon GO
  • Griddlers Plus
  • Phone
  • Youtube

This example shows two completely different pictures. There is no overlap at all. If you think that this is an outlier, remember that this applies for 10% of the people (see figure 'Which are the 5 apps you use most often?').

Mindthegap

Main findings

People can’t remember or estimate their online activities properly.

On average 64% of the participants' answers about their mobile behavior are incorrect.

65% of the respondents who answer wrong overestimate their usage.

New data sources open up new opportunities for market research

But what can we learn from all these insights? And what exactly does this mean for today's research designs? It's high time to look for new approaches to really understand what consumers do online. The good news is, that new types of data sources have become available that are able to collect the actual behavior of consumer across multiple devices.

Behavioral data can reveal what consumers do online, including the search terms they enter, which websites they visit and which apps they use. For how long and how often – and without the limitations in self-reporting and memory. However, behavioral data alone is not the secret to success.

We see that surveys aren't suitable for collecting digital behavior, but this doesn’t mean that surveys are dying. There is a fair chance for surveys when it comes to context, opinions and intentions to understand why people behave like they do. This can't be observed.

We simply can't create a complete picture of online consumers with one single data source anymore. This is where observed data (the 'What') meets opinion data (the 'Why'). Each data source has its own value when trying to capture and understand consumers online behavior.

WhyWhat

Benefits for the consumer insights industry

There are several ways in which adding behavioral data can improve the research results and reveal additional consumer insights:

  • Use behavioral data to put the measured information with a questionnaire into context.
  • Use behavioral data to select the survey sample to better target your relevant sample.
  • Use the insights from behavioral data to choose the right touchpoint to reach the right customers, at the right time, with the right message.

Combining behavioral data with opinion data leads to insights that can help brands to enhance the customer experience and truly set their brand apart from the crowd.

The real magic happens when brands create customized experiences based on real consumer insights.

What are you waiting for? Let's work together and face this exciting challenge!

You can download the handout of our case study here.