Ever had more than 4-6 concepts to test and weren’t sure how to get through 20 concepts in a single two-hour session?  Yes, I’ve had that happen more than once.  The most recent time, the concepts were all very good (i.e., clearly distinctive, well written ideas) and we really couldn’t cull down the number prior to the sessions starting.  After the first market, the client did streamline the concepts based on consumer feedback and all worked out well.  We used a more traditional grading approach, where abbreviated concepts were read and graded all at one time; we discussed the “best” performers in greater depth than the other concepts.

In the QRCA session I attended entitled “Unleashing the Power of Real People” by Justin Masterson and Jamie Johnson of Seek Research, they described an “auctioneering” approach:  they posted concept boards around the room and gave participants time to review reach one, making notes, quietly – as one might see people doing before an actual auction.  They actually employed an auctioneer to “call the items” (aka concepts) and participants bid on their preferred concepts.  Very quickly the better concepts emerged and were discussed in depth.  (An actual auctioneer isn’t really required if the moderator is comfortable, and I would be.  It sounds like great fun and a way to engage participants.)

Another option they’ve used is to have participants “invest” in concepts as though the concepts were products being sold on a stock exchange.  This approach I’m a bit more iffy about.  It moves the “bidding” into the area of what someone thinks will do well in the market vs. their personal reaction to an idea.  There have been quantitative approaches which use this “market knowledge” approach, and I’ve heard mixed reviews from clients about using them.

Yet, there are times when I would use the stock exchange idea to get to the meat of concept discussion.  For instance, if I were doing a group of “prosumers” (marketing or operations professionals from other industries as participants to solve a problem)  I think this approach makes sense; it would get these very left-brained participants to go with their gut reactions.

What are your thoughts about this idea?  Is it something you might use?

Truth is, I couldn’t tell the difference in the quality of recruits.  They were all excellent.

A positive aspect of online recruiting is that you can screen someone all the way through without termination points and it doesn’t impact costs and/or you can prevent the use of the “back button” in the survey so that people can’t  “beat the system” by changing/anticipating answers.

However, if online recruiting is to be done, qualitative researchers and facilities need to consider several  things:

• Is the target online?  It might be faster, more accurate, and less expensive to administer the screener online, but if the target isn’t represented online as it might be in the general telephone population (or even if they don’t check their email as often – a major problem I’ve noticed before), other options should be employed.  Some stats:
  • Only 79% of adults are online, according to a May ’10 Pew Research report; this doesn’t vary by urbanicity, although African Americans and Hispanics (English and Spanish speaking) are less likely to have Internet access (70% and 64%, respectively, Pew Jan ’10).
  • As expected, Internet access is lower among older adults and those less well educated (only 38% of those age 65+ have Internet access,  coupled with only 39% of those with less than a high school education and 63% of high school grads,  Pew Jan ’10.)
• Security screening as it’s been done in the past needs to be re-worked for an online environment.  (It probably should be re-worked anyway.)  It’s fairly easy to focus on a potential recruit’s employment, but harder to get at immediate family “conflicts.”  But it’s doable.
• Just as online quantitative surveys required alternative question types and scales to increase engagement, online self-administered qualitative screening questionnaires need to be written differently.  This applies to questions about a participant’s aptitude for creativity, problem-solving, influencing others, etc., as well.
• For quality control:
  • Qualitative researchers should be approving skip patterns and the like for the online survey versions vs. just passing off the screener in print form.  (After all, the quality of the questionnaire creates an initial impression for the project, just as a personal recruiter does.)
  • Personal follow-ups with new types of validation questions are needed to ensure the person recruited is who they say they are.
  • Moderators should be hyper-vigilant in their own re-screening to ensure the newer methods are working as they’d like.
  • The facilities need to build systems that verify respondents are answering the same way on key demographic questions from study-to-study – something that would increase the integrity of participants, however they’re recruited, for all studies.

Recruiting, like much of qualitative research, is both an art and a science.  Technology and the Internet affords us other options for recruitment.  As time goes on, other issues are likely to emerge.

For now, these are my thoughts.  I’d love to hear what other moderators and clients have to add!

With the introduction of online focus groups and bulletin boards, new mini-batteries were created to make sure that people were comfortable with the technology and/or comfortable communicating in written form.

In the Summer 2010 issue of QRCA’s Views magazine, there was an interesting piece entitled “Emergent Consumers and New Product Development” by Praveen Kopalle, Ph.D., Tuck School of Business Administration.  What caught my eye was the discussion of the value of Emergent (vs. Lead) users.

“Lead” users we’ve often heard described as early adopters.  They are also people who might know what their needs are going to be, often because of their high level of involvement in the category; as such, they’re often ahead of the curve of the mass marketplace in being able to articulate these needs.

“Emergent” consumers are described differently.  Dr. Kopalle’s studies lead his team to conclude that there were consumers who had an almost “instinctive” ability to understand new concepts – even if they weren’t highly involved in the category – and that findings using these types of consumers in new product development lead to products which were more likely to be adopted by the mainstream.  There are eight questions in this “emergent consumer” battery, items such as:

• “Even if I don’t see an immediate use for a new product or service, I like to imagine how people in general might use it in the future,” and
• “When I see a new product or service idea, it is easy to visualize how it might fit into the life of an average person in the future.”

While I find this battery very intriguing, I’m a bit concerned that the only people who could answer all these questions confidently/accurately are those with lots of new product/service experience.  Maybe this is a new variant of “Prosumers” that has been discussed over the past two years or so.   And maybe my concerns are unfounded.

Today I was watching a TED talk by Chris Anderson on “How web video powers global innovation.”  A very interesting talk about how “crowds accelerate innovation.”   And when he described the “crowd” aspect, he put up this image:

In a crowd, he said, all these various types of people exist.  You and I might be able to add to this list (e.g., wall-flowers, copy-cats), but to my way of thinking, this is spot-on and a great springboard for refining recruiting screening criteria.

• For social media/communications, I might want to focus on Commentators or Superspreaders.
• In website usability research, I might want a mix of Innovators, Trend-spotters and Skeptics.
• And Cheerleaders takes the idea of a loyal customer to another level.

For me, this adds another rich layer to a study design and recruit that is likely to lead to some very interesting sessions.  The questions aren’t written yet to get at these various groups, but conceptually I like the idea of where this, coupled with the Emergent consumer battery for new product/service concepts, could go.

Your thoughts?

Discourse analysis, sentiment analysis, text analytics:  whatever you call it, you’re looking at a huge volume of information.  Unstructured information.

When CRM started in the 1990s, the challenge was to link customers to behaviors and make recommendations to keep them happy/buying more.  Think of the Amazon “recommendation” feature:  “people who looked at this, bought this” or “if you liked this, may we recommend…”  They were able to “mine” purchase data to make better shopper suggestions.

Text mining is infinitely more complicated.  The longer a blog post, Facebook comment, or product review, the more difficult it is to categorize.  Twitter posts and other microblogs are theoretically easiest because they’re so short.

How It’s Done.  Comments are filtered based on keywords (e.g., brand name).  Next, as I understand it, using proprietary algorithms (depending on the software), comments are grouped into three buckets:  positive, negative, and neutral.

The longer the post, the more likely the comment is to be classified as neutral.  This isn’t surprising.  Think of the complexity in nuance, sarcasm, slang, humor and differences in word usage across different cultures speaking the same language.  (Anyone watch the Emmy’s Sunday night and see Ricky Gervais?  He was laughing at Bucky Gunts’ name and it’s taken me 10 minutes to figure out why – so bad I can’t repeat it here!)

I’ve seen statistics showing that between 65-80% of all comments are tagged as neutral.  The number is somewhat lower among the microblogs – maybe only 40%.  Nevertheless, it’s obvious that huge biases can be introduced using this approach.

Elsewhere I’ve read that humans aren’t very accurate either.  Using Mechanical Turk (which isn’t something I’d personally recommend), humans agree only 79% of time on how items should be coded.

We should be moving forward in this discipline. Tom Anderson, of Anderson Analytics, agrees.  In a correspondence with him several months back, Tom said that he was using three different software packages and gained confidence based on the intersection their joint findings.  I’m not sure if more data is successfully coded or if this just provides more confidence in what he reports as findings to his clients.  I think this is very responsible, given the accuracy of single-solution approaches today.

Currently, I’m leaning in the direction of relying more on Market Research Online Communities (or Insight Communities), which straddle the space between pure discourse analysis and more structured qualitative or quantitative marketing research.  I view them as a hybrid, where I can get out of the way and follow the discussion.  With only 200-500 members, keeping up and staying accurate is more manageable.  Further, when custom research is needed, the recruit is a snap.

And sometimes we don’t need to build communities.  One may already exist because the product or person built their following online.  In this case, the fan group may be the “go to” community because it’s larger, more representative and more cost effective than creating a new community.  Think Justin Bieber, who was “discovered” by his YouTube videos.  Those subscribing to his stream, particularly in the early days, would have been a great insight-mining resource.

I’ll be keeping a close eye on both these topics as they appeal to both my quant and qual sides.  And if you have any thoughts, I’d love to hear and learn!

Some reading:

The best of the articles I read, BrandSavant:  http://brandsavant.com/the-hidden-bias-of-social-media-sentiment-analysis/

August 2010 Quirks has several articles on social media research

Mashable:  http://mashable.com/2010/04/19/sentiment-analysis/

No wonder marketers are confused about how to use social media effectively!  The juxtaposition of the two article titles made me laugh.  (Actually, the FeverBee.com piece is rather good, offering some creative pointers on how to engage community members.)

Staying with the subject of online communities, this month’s “Quirk’s” piece on the 2009 Globalpark Market Research Software Survey reported that only 17% of companies worldwide manage at least one online community and that 56% have no plans for starting one.  (A bit more detail:  60% of small and medium companies say they have no plans, with 41% of large companies responding the same way).

Bottom Line:  From their actions, most marketers seem aware of the a fine line between staying attuned to customers and invading their privacy.  Sure, an online community is a major investment of both time and money.  But there are other ramifications that marketers are having to consider which may be slowing down this new research method’s adoption.