From Uber’s self-driving cars to Amazon’s warehouse robots,
artificial intelligence (AI) seems to be reaching human-level dexterity nearly
everywhere. You might be wondering like us: how are brands actually taking
advantage of next-gen automation innovation in market research today? Can AI
identify better insights more cheaply? Will computers and robots render human
researchers useless? As market researchers, who also happen to be human beings,
we’ve been exploring many of the same questions!
Danone, one of the world’s leading food companies, recently sought
to understand consumer consumption drivers for a new product category.
When William Serfaty, global strategy and insights manager, turned
to our team to provide him with qualitative insights to inform communication
claims within a very tight timeframe, we saw an opportunity to put these robots
to the test. With Danone on-board, we partnered with Voxpopme, an automated
video research platform, to study the extent to which automated solutions can
replace or enhance the practice of qualitative research.
Together, we launched “(Wo)man vs.
Machine”, a head-to-head competition between human researchers
and automation technology. Each team was tasked with analysing self-recorded
consumer videos using different research methodologies. One team had access to
automated tools, while the other team of SKIM researchers relied on traditional
human methods of analysis. In total, three reports were produced and judged by
William Serfarty at Danone:
- mostly automated;
- human-only, and
- combination human
and machine.
At the end of the project, William, summed up our findings well, “The
outcome was a nice surprise! Now we can get a report faster that provides the
level of detail you’d get from a traditional report” – and my team was
equally surprised but thrilled with this result!
What we learned
Here I’m sharing five tips based on our learnings on why you
should consider automation tools and how you can successfully incorporate AI
into your qualitative market research plans this year (the good news for us
is that humans still have a crucial role to play).
1:
Initial skepticism of AI will be inevitable; push past it to reap the rewards
of automation: While
there’s much industry buzz around AI and other next-gen automation
technologies, unfortunately, machines won’t offer a “magic bullet” to fulfil
your brand’s insights needs. However, while AI and automation outputs alone
don’t offer much value and lack sophistication, these tools are certainly
useful during the human qualitative analysis process. Our “Woman vs. Machine”
study resulted in a full research report, produced in half the time
using automation tools, vs. the full report created by my human team alone.
William, at Danone, was pleased to discover that the time and cost
benefits that we reaped, didn’t come at the expense of the quality insights. In
fact, in a blinded evaluation, he preferred the human-machine collaborative
report above the fully automated and fully human-generated versions. Instead of
replacing human insights and consultative expertise, we like to think of
automation as a “turbo boost” for traditional qualitative researchers.
2:
Don’t expect machines to provide the answers: We all agree that opportunities to automate
appear to fall more in line with quantitative research. Given the human nature
of qualitative research, we went into this experiment questioning whether
automation is even possible? Although we’ve seen developments in natural
language processing, we’re far from achieving total automation just yet.
When conducting qualitative analysis, there is currently limited
value in automated tools without human involvement. The outputs produced are
words and charts that hold little meaning on their own and with accuracy that’s
hit or miss. Machines can’t connect the dots, determine which of the insights
are truly key or identify the drivers. Even to create an initial topline
report, human analysis is required to review automated outputs, understand
their meaning, and narrow down which information is relevant. While in time it
is likely their intelligence will increase, for now at least, automation tools
can’t provide stand-alone answers. So, it’s important to understand how best to
use them to our advantage.
3:
Use AI outputs as the starting point for human analysis: While they don’t provide a magic bullet
solution, we learned that automation tools can empower qualitative researchers
to conduct analysis much faster. In contrast to our human analysis team which
had to spend a week reviewing all the video transcripts, the starting point for
our automated team was the machine outputs. By analysing these, rather than the
raw data, within just one day we were able to build up a picture of the overall
story and identify key learnings.
4:
Expect high-speed analysis to produce high-level findings: When turning to these tools it’s important to
have the right expectations. If internal time pressure means immediate answers
are required, this technology can help. However, the result of high-speed
analysis is a birds-eye view, meaning very high-level findings — not the deep
dive insights and strategic recommendations you’ve come to expect from
qualitative studies.
5:
Being strategic takes time; don’t cut this corner: In a blind evaluation, Danone’s William Serfaty
preferred a collaborative (AI + human researcher) report over the mostly
automated and fully human-generated versions. Reports that relied heavily on
automated outputs may be quicker, but speed comes at the cost of strategic and
actionable insights.
More time and deeper human analysis is therefore required to
explain and translate high-level information into clear guidelines and
recommendations. Nevertheless, this process takes qualitative researchers half
the time when armed with automated tools to help them. As a result, at SKIM
we’re understandably optimistic about the future potential for automation and
AI-enhanced qualitative research methodologies.
Collaborating with machines can indeed enhance efficiency, without
compromising the depth and quality of insights. To learn more about this project
listen to this on-demand webinar and/or read the related whitepaper.