What Part of ‘Yes’ Don’t You Understand? How Google Consumer Surveys Can Go So Terribly Wrong

The Experts on Google Consumer Surveys - 05/10/12 07:00 AM

Imagine you are about to launch a new line of ice cream, with millions of dollars riding on it, and you have spent a fortune on package design, advertising and marketing. Now, all you need is a few consumers to validate what you already know — that you are brilliant. So, you put the following Google Consumer Surveys (#gsurveys) question in front of people, and for the life of you, you cannot figure out why the responses you are getting back are so overwhelmingly negative.

Now we know what you are thinking: that we Photoshopped this Google Consumer Surveys research question. That we airbrushed it. That we digitally altered it somehow.

No. There is no need for any of that.

Real life is always better than anything you can make up. This is exactly how this Do-It-Yourself #gsurveys question appeared when we encountered it in the wild. I can just see Gary Larson drawing this survey question into a Far Side cartoon, with a marketing intern pointing at the question and saying in the speech bubble to his boss, “Well here’s your problem, Mr. Z.”

So, let’s take this tragedy apart and see what brings us to this moment.

A lot of research companies have certain tried-and-true questions that they have used for years. They are comfortable with the questions and they know what the results should be. An example of such a question might be:

Would you buy x?

  • Definitely
  • Probably
  • Probably not
  • Definitely not

(In this case, x is a new kind of ice cream.)

So far, so good, right. Nothing complicated. Very straightforward.


Google’s original 7 #gsurveys questions do not permit you to have 4 answer choices if you want to show the respondent a picture. If you want to show the respondent a picture, you get 2 (and only 2) answer choices.  Limited to 2 answer choices, one possible construction might have been:

Would you buy x?

  • Yes
  • No

It gets slightly complicated here, because (as we have previously pointed out, and as we here maintain), any image you put into a #gsurveys image question is likely going to be illegible and/or invisible, and this one manages to be both. So we hesitate to rewrite the question as, “Looking at this ad, are you likely to buy this brand of ice cream? Yes/No” ……… because …….. well …….. no one can see the ad. We have tried. Under tungsten light, daylight, incandescent light, New Jersey tanning-bed light: we cannot read the tag line, we cannot see the package design, we cannot read what’s on the carton. Nothing. A complete waste.

So, the Do-It-Yourself researchers at IceCreamZero could have done a lot of things: they could have thrown away the ad, abandoned the “definitely/probably” construction, or both.

Instead, they chose the worst of both worlds: they included the picture and they included a random 2 answer choices from the 4-answer choice boiler-plate question. And that means that respondents like us could only say that we probably would not buy the ice cream, or we definitely would not buy the ice cream.

Those of you who by pure chance did not get the “Probably Not / Definitely Not” construction that we got (shown above), might have instead gotten this version (below):

And of course there were other versions: “Definitely Yes” paired up with “Probably Yes,” was offered to some respondents, (so you could not answer “no” to the question) and all the variations on that theme.

You might be thinking: well, luckily those Do-It-Yourselfers at IceCreamZero didn’t spend a lot of money on this research, because one of the great selling points of #gsurveys is: they are inexpensive.

Except, these examples are part of an entire, big, fat, whopping survey, with lots of questions, that must have cost (potentially many) thousands of dollars to complete. (Yes: confirming, no one respondent would have seen more than one of the questions, that’s central to the Google Consumer Surveys model. But we gathered about 7 or 8 of these one-off-questions in our fly catcher, and there’s no reason to think we stumbled across them all).

Consider this variation from the current IceCreamZero #gsurveys rotation, which we found sadistic: “Do you enjoy seeing the ice cream cone on the package?” Enjoy? Seeing? What ice cream cone!?

And after all the squinting, and the showing it to our spouses (“Honey, can you make this out?”), we discover that the only 2 Gary Larson Far Side answer choices are “no” and “indifferent.” How can you make stuff like this up?

What advice would you offer the fine folks at IceCreamZero? They get an E for effort. They tried to do-it-themselves, it appears. They really tried. But here they are with all these survey results. And all the data they get back from Google will be graphed, and arrayed, and all of the numbers will have a very, very, very precisely stated margin of error associated with them, and Google will go out of its way to even tell IceCreamZero about the response rate that the survey achieved.



Except there’s not a single number in results that will be worth anything. Even if the (so called) response rate was 100%. Even if the (so called) margin of error approached zero percentage points (because IceCreamZero was willing to pay to interview just about everyone on earth).

Throw everything you get back from Google Consumer Surveys on this project in the trash can, Mr. Zero.

We beg you: retake your research using a professional researcher. A good researcher will be able to fix you right up.

Note well: we are not saying that anything the rest of you get back from #gsurveys should be thrown in the trash can; we have lots of examples in our files of inspired research that we hope to post over the coming days. We are, however, saying that these results — this specific execution using Google Consumer Surveys — needs to be discarded by IceCreamZero, or IceCreamZero will risk spinning out of orbit onto who knows what tangent.

Your thoughts about what IceCreamZero should do next are most welcome.

Copyright 2014 SurveyUSA®, Clifton NJ, all rights reserved. Terms & Conditions.