I occasionally get surveys in my spam email. They come in two varieties. Legitimate surveys and surveys that are designed to feed me an endless number of “special offers” until I either die of old age or my screen fills up with so many pop-up windows that I go blind.
When I get a legitimate survey I sometimes take the time to answer it. After all, I want my opinion to be heard. And I think my opinion is better than anybody else’s anyway.
Last year ACNielsen approached my family about adding a monitor to our television so we could be one of those families that decided which shows were hot and which shows were not.
We were interviewed and everything was going fine until at some point they decided that they could not use our opinions because their set-top monitoring equipment would not work with the DISH network and the satellite receivers we had. In other words, their sample data, which is used to decide which shows stay on the air is crap. Why? Because they have pre-screened their sample already. Because they throw away data that isn’t compatible with their system. Anyone who has taken a science course with a laboratory knows that you just don’t throw away data and expect your results to be significant. (You might throw away data if your lab partner sneezed on the petri dish or you might fake data to get that lab done before finals week, but that’s another story.)
Recently I have come across a bunch of seemingly legitimate internet based surveys that ignore data unless it comes from the right browser and operating system. Yup, it seems that some of these surveys end right after they find out your hardware specifications. If you are a Macintosh user on Safari then that’s it. Your opinion of what is the best flu remedy just doesn’t count. It’s not like the survey is looking for a target market of certain types of computer users. These surveys are for everything under the sun; the car you might buy to the soap you use.
Of course you have to question the fundamental absurdity of using only internet users to find out about soap. It’s not like internet users have cornered the market on taking baths, is it? In fact any email regarding soap might be taken as a gentle hint to many internet users that I have met. Internet surveys are another form of pre-screening. I suppose a student of statistics out there might try to argue that the demographics of the internet match the demographics of the population at large, or that a large enough sample size would solve the issue. Bullshit. “There are lies, damn lies, and statistics”*. The problem is that you are trying to match a demographic to the way people think. Throwing away any sample data because you can’t deal with it might just be the significant variable you wanted in the first place.
Don’t even get me started on polls, especially the phone-in variety,
*attributed to Benjamin Disraeli, Alfred Marshall, Mark Twain and others.