Customer Analytics – a Double Edged Sword
The following is a guest post by Lorraine Chapman, Director Research at Macadamian. If you want to submit your own guest post, click here for more information.
With the rise of analytics and big data, there is a ton of information out there that can give you insight into how customers use your products and feel about your brand. For example, tools like Google Analytics, Flurry, and Preemptive Mobile give you more data than you know what to do with if you want to analyze usage patterns of your web and mobile devices.
Analytics solutions like Netbase allow you to parse through millions of tweets on Twitter and get a sense for how customers perceive your brand. It’s all very exciting, but these reams of data can also be highly misleading.
Have you really spotted a trend?
If you ask enough questions, and gather enough data, meaningless patterns will emerge. Simple example: if you ask enough people what month they were born in, and a hundred other research questions, you will “discover” things like people born in November drink a disproportionate amount of Budweiser compared to everyone else, or that 86% of people with July birthdays like driving foreign cars over North American cars.
You haven’t actually discovered anything except a statistical phenomenon known as clustering. In a large set of random data, you get clusters of the same type of information. Roll a pair of dice 100,000 times and you’ll get strings of snake eyes or 12’s and other unlikely sequences. It doesn’t reveal a strange meaningful pattern. There are just a lot more clustered patterns than non-clustered possibilities out there.
To find out if the trend has meaning, you would have to run a separate follow-up experiment. Automatically assuming that a cluster of data has meaning — and making design decisions based on that — can lead you astray.
For example, a client once asked us why their website was so popular on Tuesdays, as shown by their demographic data. It didn’t make sense, so we asked them to redo the test. In the second test, the pattern disappeared.
When gathering web or mobile analytics, survey data, or any kind of quantitative measure, your biggest asset is knowing the right questions to ask. Start with a theory that you want to prove or disprove, or use analytics and surveys to steer more detailed user research. For example, you could use survey data to get an overall preference for a feature, then use more qualitative methods like user interviews to understand that preference in context and detail.
Misunderstanding statistical significance
Conclusions drawn from large amounts of data (from social media, analytics, BI and even traditional methods like customer surveys) are often highly regarded because they are “statistically significant”. However, when it comes to testing how users actually interact with your product (and hence what new features, improvements and innovations would best help them), statistical significance is far from a sufficient criterion for meaningful or proper research on its own.
On the other hand, you can’t draw conclusions from interviewing only two or three users.
So how many users do you need to speak with or observe?
It depends on the range of user groups your product is targeted to, the scope of product interactions you want to observe, how the results will be used, and how many rounds of research you conduct.
If you want to test a software application for usability and opportunities to improve the design, focus on one or two primary user groups, and five to six users each time will detect most usability issues.
Usability testing
Jakob Nielsen has shown that approximately five to six users will likely detect 80% of usability problems for a specific use of a product. Keep in mind this “formula only holds for comparable users who will be using the product in fairly similar ways.”
Laura Faulkner tested Nielsen’s theory on a web-based employee timesheet. She ran tests with a group of 65 users, then selected random groups of five participants and compared the percentage of issues each group of five detected compared to issues detected by all participants.
Nielsen’s 80% rule held, but variability meant some groups of five identified as few as 55% of the issues. By increasing the number of users to ten, groups found an average of 95% of issues.
Ultimately, the number of users you need to draw on depends on what type of information you want to get from your users. For a product benchmark study, you’ll need to run with a much greater number of users than a usability test, and you have to make sure you test across all user groups. For customer interviews, the number can vary significantly.
In summary
Many important research goals can be satisfied with a limited amount of research from a limited number of participants. Product Managers can seek advice from User Experience researchers with a background in experimental practices to help determine the extent needs of a product.
As analytics solutions get more powerful and the Big Data movement takes off, there are more and more novel ways to glean insights from customers about a product and brand. But having loads of data in and of itself doesn’t guarantee anything. Product teams need to maintain a healthy skepticism and balance these new data trends with traditional user research methods like contextual interviews and usability testing in order to truly understand how users are interacting with products, and where the opportunities for improvement and innovation lie.
Lorraine
——
Tweet this: Customer Analytics – A Double Edged Sword – http://wp.me/pXBON-3id #prodmgmt #ux
No related posts.
Customer Analytics – A Double Edged Sword – http://t.co/cJVZc077 #prodmgmt #ux
Customer Analytics – A Double Edged Sword – http://t.co/cJVZc077 #prodmgmt #ux
Customer Analytics – A Double Edged Sword – http://t.co/cJVZc077 #prodmgmt #ux
Customer Analytics – A Double Edged Sword – http://t.co/cJVZc077 #prodmgmt #ux
Customer Analytics – A Double Edged Sword – http://t.co/cJVZc077 #prodmgmt #ux
Customer Analytics – A Double Edged Sword – http://t.co/cJVZc077 #prodmgmt #ux
Customer Analytics – A Double Edged Sword – http://t.co/cJVZc077 #prodmgmt #ux
Customer Analytics – A Double Edged Sword – http://t.co/cJVZc077 #prodmgmt #ux
Great post on why stats don't always tell the truth. RT @onpm Customer Analytics – A Double Edged Sword – http://t.co/5ptsu0pJ #prodmgmt #ux
Customer Analytics – a Double Edged Sword http://t.co/Wv9tXm6t
Customer Analytics – A Double Edged Sword – http://t.co/AfH3r1xI #prodmgmt #ux
Check out Lorraine's post! http://t.co/pKDiXbYL
RT @onpm: Customer Analytics; a Double Edged Sword by #Macadamian's research director http://t.co/topqMGIV
From @onpm: Customer Analytics; a Double Edged Sword http://t.co/topqMGIV
How Google Analytics can mislead you when analyzing customer behavior. Lorraine Chapman's guest blog on OPM: http://t.co/C3SjQcWM
#Macadamian's Lorraine Chapman is in On Product Management discussing The Double Edge Sword of Customer Analytics http://t.co/L0j2QQ3O
#Macadamian's Lorraine Chapman is in On Product Management discussing The Double Edge Sword of Customer Analytics http://t.co/L0j2QQ3O
Customer Analytics – A Double Edged Sword: An article by #Macadamian Director of Research http://t.co/Yy3y29vh
#Macadamian's Director of Research has an article in On Product Management about Customer Analytics http://t.co/AOL9djGJ
#Macadamian's Director of Research has an article in On Product Management about Customer Analytics http://t.co/AOL9djGJ
@macadamianlabs Director of Research has an article in On Product Management about Customer Analytics http://t.co/x0pMhQ0Y #prodmgmt
Customer analytics is a double edge sword. #Macadamian's Lorraine Chapman explains in this article http://t.co/EL1dZ88C
Customer analytics is a double edge sword. #Macadamian's Lorraine Chapman explains in this article http://t.co/EL1dZ88C
Customer analytics is a double edge sword. #Macadamian's Lorraine Chapman explains in this article http://t.co/EL1dZ88C
Customer Analytics – A Double Edged Sword – http://t.co/x2LW0BjL #prodmgmt #ux
Customer Analytics – a Double Edged Sword http://t.co/DpcFZ9LX via @onpm
The double edge sword of customer analytics, an article by #Macadamian's Director of Research, Lorraine Chapman http://t.co/ZtBvpuhP
Customer Analytics – A Double Edged Sword – http://t.co/cJVZc077 #prodmgmt #ux
The Double Edge Sword of Customer Analytics: an article in On Product Management by #Macadamian's Lorraine Chapman http://t.co/WtKwrhEH
#Macadamian's Director of Research tells us why Customer Analytics is a Double Edge Sword in On Product Management http://t.co/mG6kA1e0
On Product Management publishes #Macadamian's Lorrain Chapman's article titled Double Edge Sword of Customer Analytics http://t.co/4i2rdSUi
On Product Management features an article by #Macadamian's Lorraine Chapman about the double edge sword of analytics http://t.co/kFrOdPjg
The double edge sword of customer analytics http://t.co/5WvKuogK
The double edge sword of customer analytics http://t.co/5WvKuogK
The double edge sword of customer analytics – an article by #Macadamian's Director of User Research http://t.co/Tk3JM4fO
#Analytics & #bigdata can tell a lot – including things that aren't true. Cautions via @macadamianlabs http://t.co/2ocDoezu #smartgrid
Customer #Analytics – a Double Edged Sword, http://t.co/EFnB8xgz #retail
The Double Edge Sword of Customer Analytics is discussed in this article by #Macadamian's Lorraine Chapman http://t.co/ywWPmzEe
The Double Edge Sword of Customer Analytics is discussed in this article by #Macadamian's Lorraine Chapman http://t.co/si80SdEE
Lorraine Chapman discusses the double edge sword of customer analytics in this On Product Management article http://t.co/fUAD07na
#Macadamian's Lorraine Chapman addresses the double edge sword of customer analytics in this article http://t.co/n3wGIqw6
RT @onpm: Customer Analytics – a Double Edged Sword http://t.co/DHFhTX72
#Macadamian's Lorraine Chapman discusses the double edge sword of customer analytics in this guest blog post http://t.co/zsLn6Iz3
RT @macadamianlabs: The double edge sword of customer analytics http://t.co/NJaG8hrw #UX
#Macadamian's Lorraine Chapman discusses the double edge sword of customer analytics in this guest blog post http://t.co/HorrPsWR
Customer Analytics – a Double Edged Sword http://t.co/5fxfLVYV
The double edge sword of customer analytics by #Macadamian's Lorraine Chapman http://t.co/r2Mddp8l
The double edge sword of customer analytics as featured in On Product Management http://t.co/eOs1a0ud
Taking a look at the double edge sword of customer analytics http://t.co/4dv7vWFL