How To Create A Survey That Makes A Problem Seem Bigger Than It Is: 'Do You Know Anyone…'
from the lying-with-stats dept
There are all sorts of ways to lie with stats and surveys. One of the more obnoxious ones is used in a new “study” about “mobile etiquette” which claims the problem is reaching epidemic levels mainly based on an online survey. The accuracy of online surveys are already suspect enough, but this form of survey is done in a sneaky way that makes the problem look potentially much worse than it is:
Nine out of ten people who responded to the online survey said they have seen people misuse their mobile devices, according to the survey.
Right. It didn’t ask people about their own activities, but just if they’ve seen other people. That’s almost meaningless. Taken to an extreme version, say there was one (highly visible) mobile jackass, and 99 other people surveyed saw that one guy. Well, based on this survey, you could claim that mobile etiquette is in a terrible state.
I’m not arguing that there isn’t necessarily a problem with how some people handle mobile etiquette. There could very well be just such a problem, but a study that sets out to investigate the issue with a survey like this one really tells you nothing.
Filed Under: statistics, surveys
Comments on “How To Create A Survey That Makes A Problem Seem Bigger Than It Is: 'Do You Know Anyone…'”
Maybe I don’t get out enough, but I’ve actually noticed an improvement in mobile device behavior compared to this time maybe a year ago.
Specific case in point: a restaurant I frequent on a weekly basis. Used to grit my teeth at least once every 2 visits in reaction to a loud or annoying ring and/or conversation (and one time 2 young kids watching a YouTube video at full volume). Past 6 months or so? Not a peep.
Standing in line at various stores was always a near guarantee of those one-sided conversations that are too loud – because they’re so far away, you see, gotta yell into the phone*. Lately? Can’t recall the last time it happened.
I don’t see too many folks walking and phoning/texting, but here is not very urban or crowded either.
People who drive like idiots…there are too many reasons to guess at, won’t jump to conclusions there.
Maybe increased awareness in my little neck of the woods?
*The spouse was often guilty of this until it was brought to his attention. Now he whispers when he calls from the store with a question or dilemma, lol.
Hm, I doubt most people who violate the basics of phone etiquette would implicate themselves in a survey. Am I wrong or do these people seem oblivious to their own behavior? Not sure they’re the right folks to be asking.
I see your point however, and agree the results of the survey are essentially bogus, but I can also see why they may have phrased the question in the way they did. Self-reflection isn’t most people’s strong suit.
I’d like to start a poll asking if anyone knows of a corrupt politician.
It isn’t just the question, it is often the slanted person reading the study or survey:
http://www.techdirt.com/articles/20110228/23515813305/new-study-70-people-find-piracy-socially-acceptable.shtml
Josh Smith is the new John Smith?
Common Practice
This is actually common practice in sociological survey’s conducted about sensitive material where the subject is likely to lie about their own involvement. An example would be “Do you know anyone who would refuse to have a homosexual roommate?” While it seems like it would skew the data (and to some measurable degree it does), it allows you to investigate otherwise unbreachable data sets. This is often done regarding taboo topics such as race / sexuality / crime.
Re: Common Practice
Is it the study of sociology that encourages this?
It’s horrible. Absolutely horrible. Even on subjects which are super-obscure, downright impossible if you spend two minutes thinking, or otherwise everyone should be answering ‘no’, you can get a majority back saying ‘yes’.
And as the subject becomes less obscure, the problem becomes worse, obviously. Because less obscure = more visibility.
The white-washing of your data into a nearly normal curve with heavy negative skew should’ve been the tip that perhaps you are measuring a pseudo-random variable instead of what you want to measure. Unrelated questions on surveys should NOT form any type of pattern reliably.
I don’t buy the idea about lying about their own involvement, there must be better ways to ask than that.
I have often thought about a poll question such as: do people speed in front of your house. They usually will answer in the affirmative with a !. This is not very conclusive but you would really see a pattern on a poll.
Can't resist the classic Yes Prime MInister.....
Sir Humphrey: “You know what happens: nice young lady comes up to you. Obviously you want to create a good impression, you don’t want to look a fool, do you? So she starts asking you some questions: Mr. Woolley, are you worried about the number of young people without jobs?”
Bernard Woolley: “Yes”
Sir Humphrey: “Are you worried about the rise in crime among teenagers?”
Bernard Woolley: “Yes”
Sir Humphrey: “Do you think there is a lack of discipline in our Comprehensive schools?”
Bernard Woolley: “Yes”
Sir Humphrey: “Do you think young people welcome some authority and leadership in their lives?”
Bernard Woolley: “Yes”
Sir Humphrey: “Do you think they respond to a challenge?”
Bernard Woolley: “Yes”
Sir Humphrey: “Would you be in favour of reintroducing National Service?”
Bernard Woolley: “Oh…well, I suppose I might be.”
Sir Humphrey: “Yes or no?”
Bernard Woolley: “Yes”
Sir Humphrey: “Of course you would, Bernard. After all you told you can’t say no to that. So they don’t mention the first five questions and they publish the last one.”
Bernard Woolley: “Is that really what they do?”
Sir Humphrey: “Well, not the reputable ones no, but there aren’t many of those. So alternatively the young lady can get the opposite result.”
Bernard Woolley: “How?”
Sir Humphrey: “Mr. Woolley, are you worried about the danger of war?”
Bernard Woolley: “Yes”
Sir Humphrey: “Are you worried about the growth of armaments?”
Bernard Woolley: “Yes”
Sir Humphrey: “Do you think there is a danger in giving young people guns and teaching them how to kill?”
Bernard Woolley: “Yes”
Sir Humphrey: “Do you think it is wrong to force people to take up arms against their will?”
Bernard Woolley: “Yes”
Sir Humphrey: “Would you oppose the reintroduction of National Service?”
Bernard Woolley: “Yes”
Sir Humphrey: “There you are, you see Bernard. The perfect balanced sample.”
Daft Surveys
In a discussion a couple of nights ago about use of the internet 3 out of five were pretty definite that youngsters spend all their time in front of their computers playing mindless games. Students lift info from the net and paste it into essay without having a clue what they are on about. And computer tech alters brain structures. 2 out of five disagreed very strongly.
So: 60% of kids are being harmed by their computers;
or, 60% of people believe that kids are being harmed;
or, 60% of people don’t know what they are talking about;
or, none of the above.
I think that sounds about right.
This just in:
9 out of 10 people believe that other people are worse than they are.
Seems strange they wouldn’t attempt some sort of control by asking these respondents if they had abused phone etiquette. My guess is that would fall way under 50%, which would invalidate the first claim.
It’s pretty easy to get someone to agree that someone else sucks.
Re: This just in:
Or indeed some questions to establish exactly what the person they were asking thought “phone etiquette” actually included. I don’t recall ever having seen a set of written rules or anything. Ask a few hundred people that and you’d get anything from people who thought getting your phone out in public was bad to people who thought anything short of bludgeoning the guy next to you to death with it was fine.
Re: Re: This just in:
Exactly.
Side note: It was much easier to bludgeon people to death with a cell phone back in the early 90s. Smaller components mean less effective bludgeoning, people.
Re: Re: Re: This just in:
Hmm good point. Some sort of stick attachment to make a mace-type arrangment would seem to be in order then? 🙂
Old hat
This is really nothing new, statistics by and large are ways of lying to people via mathematics. I’m paraphrasing but one of my favorite sayings is “Statistics are often used as a drunk man uses a lamp post; for stability rather than illumination”. Real non-biased statistics and surveys are rarely seen.
Overhyped
Stats are just ridiculous, I mean from http://www.nss.org.uk/about/stalking-facts-figures/, anyone that takes these “facts” at face value is a moron.
“Over 1.2 million women and 900,000 men are stalked every year according to The British Crime Survey 2004”
Ok, we’re quite a small country, we only have ~66 million people here, so that’s really quite a large number!
DAMN, and I finally thought mikee wrote something
Damn, I saw the title and thought “WOW little mikee finally wrote an article on something that he’s and expert on.”
Then after reading it, I see that little mikee is just bitching and moaning about someone ELSES work, instead of doing something himself.
But then do we REALLY need little mikee making up surveys, hell if he had his way the entire world would be free to him, and noone ever would deserve to earn anything for their work.
Re: DAMN, and I finally thought mikee wrote something
Rather than complaining about the lack of contribution by others, maybe you could concentrate on providing some yourself?
Re: DAMN, and I finally thought mikee wrote something
Who is “little mikee”?
Re: Re: DAMN, and I finally thought mikee wrote something
Who is “little mikee”?
I’ve always kinda thought he was referring to one of his own body parts.
Re: Re: DAMN, and I finally thought mikee wrote something
I’m more concerned with Mr./Mrs. Noone, who apparently are having trouble getting people to pay for whatever the hell it is that they make.
Re: DAMN, and I finally thought mikee wrote something
Or the entire world would be rich, finally paying attention to consumers.
>Is it the study of sociology that encourages this?
No. Quantitative studies in sociology are nothing like this, and sociologists are quite capable of creating a study which is significantly more objective. This “survey” is junk science, and no professional sociologist would ever draw the conclusions about behavior from the indicated results. All one could reasonably do would be to draw conclusions about people’s *impressions* of behavior. There is a big difference – obviously.
Unfortunately, roughly 2% of the news media is able to make that differentiation, and as a result, we’re constantly bombarded by horribly misleading and inflammatory information. Good for ratings. Bad for everyone.
Kudos for calling BS on this, and for all of the right reasons.
Re: Re:
Glad to hear that Sociology doesn’t really form conclusions from such data.
The way the other guy said it, he was a sociologist, or studied as one, and that was a common technique. Now I won’t have to track down a soc. prof here and badger him with questions. :p
Yeah, I studied pure math, (and some stats with that), and I’m the guy constantly complaining about survey questions in politics. The religion question on British surveys, for a recent and strong example.
Or, y’know, the conservatives up in Canada cutting the long-form census completely. I suppose it doesn’t hurt them any, since they mainly pull their stats from their . . . hats.
I don’t get it. What purpose is this “survey” supposed to serve? Usually, the results are followed by a “therefore we all need to do (whatever)”.
Survey
Survey only tell you what you want to hear.
It is very hard to write a Survey that is completely objectionable (non biased).
Case in point our upper management approved a survey to rate their own performance for the purpose of their own bonus. No matter how you answer the survey question you put upper management in sparkling light (and not of the type of golden showers they rain down on you). For example: Does making your clients happy make you happy?
If you really want to know the real answers you need to find a very good third party to masterfully construct the survey questions. Sadly many survey writers lack the required skills.
Just because you want to know X does not necessarily mean you can directly ask X. The skill is exact wording and associated questions that can tell you exact about X.
A bad survey is like a account saying…. “and what do you want those numbers to add up to?”
Re: Survey
“Just because you want to know X”
Do they really want to know X ?
Or are they simply looking for agreement with their view?
Typically, surveys have an angle and attempt to persuade via herd mentality.
One thing that bugs me is I take my kids to a dance studio, and I’m typically the only guy there. Most of the moms are sitting there with iPhones and iPads completely ignoring their kids finishing up practice or even just ignoring them as they wait for their class.
I always do everything I can to put the phone down and avoid it when my kids are present, their attention and my attention is vastly more important. I’m constantly asking my kids to tell me something unique about their day at school, and to engage them in conversation about everything and everything that might spark other conversation.
I also see kids with their heads in these devices, and I don’t really blame them for long cartrips, but I see them basically stay in cars when their family goes into a store. I guess I have a concern that these devices cull some level of social development that is part of growing up and may be useful later in life.
We limit our kids media to 2 hours a day on non-school days, and more if the media is “educationally based.”