Margin of Error

A joke from Joe Klein: "Prime Minister Ehud Olmert is testing the limits of the possible: in a recent poll by a local television station, he had a favorable rating of 3%. Given the poll's margin of error, it was possible Olmert had no support beyond his extended family." I've had that thought before when I see polls with extreme results. I assume that it's not literally true. But I don't understand the right way to interpret a result like that is. If your sample size gives you an MOE of four percent and Olmert has a 3 percent approval rating, how do you interpret the possibility of negative popularity?

There's also some substantive point here about the prospects for an Israel-Syria peace deal and the Bush administration's absolutely bizarre desire to prevent that from happening.

3/31/07 12:05AM del.icio.us this incoming links

Comments

I seem to recall a Russian poll from the late Yeltsin era that had Boris with a popularity rating less than the MoE, raising the same possibility of a negative popularity.

Posted by: FMguru on March 31, 2007 12:13 AM

how do you interpret the possibility of negative popularity?

You don't. The MOE plus/minus is basically a function of sample size. So common sense takes over.

Posted by: pseudonymous in nc on March 31, 2007 12:35 AM

I think the basic reason you see weird poll results like "3% approval with 4% margin of error" is that they're reporting the maximum margin of error expected from the sample size. The maximum MOE would happen if the percentage being polled (approval rating, in this case) was 50%, but if you get some extreme result where the approval is close to 0 or 100, the MOE will be smaller. As Wikipedia puts it in their "margin of error" entry:

"The margin of error for a particular individual percentage will usually be smaller than the maximum margin of error quoted for the survey. This maximum only applies when the observed percentage is 50%, and the margin of error shrinks as the percentage approaches the extremes of 0% or 100%."

Posted by: Haggai on March 31, 2007 12:37 AM

K, here's the thing about MoE's: They're proportional to the square root of p*(1-p), where p is the percentage (expressed as a number from 0-1). Generally when they report MoEs the pollsters are conservative and assume p is .5 (50%), but when p is far from .5 the MoE is much smaller.

Posted by: Matt Stevens on March 31, 2007 12:37 AM

The margin of error narrows for fractions near 0 or 1. I believe it is usually stated for poll results near 50%. For poll results near 3% it will be about a third as wide. The margin of error is computed assuming the distribution of statistical error is normal. If this gives a substantial probability of a negative result this is telling you that a normal distribution is a bad approximation in your case.

Posted by: James B. Shearer on March 31, 2007 12:39 AM

A good Statistics 101 rule of thumb for the margin of error of a Yes/No question is this:

Let p = true proportion of Yes's
1-p = true proportion of No's

Thus, modelling pollees as independent, identically binomially distributed variates, you'll find that if you poll N people then

Average # of Yes's = pN
Average # of No's = (1-p)N
Standard Deviation of Yes's or No's = [p(1-p)N]^{1/2}.

Dividing through by the number polled N, we see

Standard Deviation of Yes's or No's percentage-wise = [p(1-p)/N]^{1/2}

Thus, taking the margin of error as the 95% confidence interval (which is basically 2 standard deviations assuming N large enough that distribution by central limit theorem tends to Gaussian)

Margin of Error = 2[p(1-p)/N]^{1/2}

But note! Quoted margin of error statistics almost always essentially assume p = 0.5 (i.e., true proportion of yes's vs. no's is assumed to be essentially 50/50) and thus use the formula

Almost-always Quoted Margin of Error = [1/N]^{1/2}

Losing that p(1-p) in the numerator makes a big difference. For example, take a poll with 1000 pollees. Most people will usually quote a margin of error of about (1/1000)^{1/2} = 3.2% for such a poll (I say "about" since there's actually more complicated rules for margin of error depending on polling methodology. But that's a good ballpark figure.) However, if the finding is very skewed, like Matt's example of 3% approve vs. 97% disapprove, this 3.2% figure isn't the one to use. A far better figure to use is the formula above

2[0.03(0.97)/1000]^{1/2} = 1.1%

which has the advantage of never implying negative probabilities... though that's always good for a cheap laugh.

[Yes, deviations for rare findings actually have more complicated formulas, but we needn't go into that here.]

Posted by: Bill Kaminsky on March 31, 2007 12:41 AM

This is what happens, when people try to put a Bayesian question to a non-Bayesian method.

Posted by: arthegall on March 31, 2007 12:42 AM

I'm always impressed by the sheer information gathering powers of the blogosphere. In the wee hours of a Friday night/Saturday morning, when policy junkies should be starting to trickle back home somewhat drunk from various Friday shananigans, you can find within minutes multiple people able to comment correctly on margin of errors in polls with skewed findings.

It gives me faith in humanity... sniff. :)

Posted by: Bill Kaminsky on March 31, 2007 12:44 AM

I have a computer science degree, and I've taken a lot of stat classes. Basically, when you calculate a MoE, what you're doing is figuring out how likely it is that the poll will turn out the same if you did it exactly the same way again.

Think about it like a swimming pool filled with red and white balls. If you sampled 1,200 balls and it came up 60%, you could say with 95% confidence that it would come up between 57% and 63% if you MoE was 3%. Now, pulling balls out of a pool is much more reliable then phone polls, so the MoE in my example would actually be less then 3%, but it's still certain that a person cannot have a negative approval rating, so you can alway say something like that it's really like a -3, +5, rather then +/- 4%.

What actually happens is that you have a bell curve representing the probability distribution for the result of another poll. In the case that the predicted value is close to 0 or 100%, you get a 'clumping' of that distribution on one side or the other.

Posted by: Chad Okere on March 31, 2007 12:47 AM

Actually the term for "clumping" I described earlier is "skewness". You can read about it (and see graphs) on wikipedia.

Posted by: chad okere on March 31, 2007 12:50 AM

It's an example of how artificial the assumptions of normnal distribution are to the "statistics" of some systems. The randomness of human behavior or opinion is implicit in the math of MOE and so on. More intuitively, at least some people claimed to approve of Olmert and thus his support is a non-zero and positive (assuming: approve? yes or no as the poll).

Posted by: Michael Tanowitz on March 31, 2007 03:17 AM

If Olmert's family tilts the MOE quiotient .00001 higher than the square root of -1 it's not such a bad variable. He has a daughter who's no apologist for Israel and maybe if that's where he's gone for solace- he'll do the right thing and recognize the new Palestinian Govt. He seems clueless but stranger things have happened. Everyone knows people are dying for this to end, the Occupation- the killings on both sides- all of it. So why not Olmert? What does he have to lose?

Posted by: Trevor on March 31, 2007 03:21 AM

In fact, given Olmert's prominence and career, the relatively small size of Israel's voting population and a large sample size, I wouldn't be surprised to find a non-trivial percentage in any poll who are indebted to or connected to Olmert in some way. Small world, etc...

Posted by: MattF on March 31, 2007 06:46 AM

First, note that "margin of error" doesn't define the absolute limits of what is possible, given poll results. If Bush's approval rating was 46% (he should be so lucky!) with a 4% margin of error, that doesn't mean that his actual approval rating is certain to be between 42% and 50%. Let's say Bush's true approval rating of everyone in the US was X% (you will never know X). A 4% margin of error means that, 95% of the time, the method you used will be within 4% of X.

Second, the idea of "margin of error" idea as (plus or minus N percentage points) only works when an estimated figure is near 50%; for very high or very low estimates, the margin of error is much higher on one side than another side.

If 100 Israelis were polled, and 3 said they approve of Olmert, then his estimated approval rating is 3%, and the margin of error is (0.6% - 8.5%).

Posted by: Allan M on March 31, 2007 07:45 AM

Re Trevor

Mr. Trevor continues to harp on the same old theme, namely end the "occupation" and all will be well in Palestine. Unfortunately, the Palestinian concept of the occupation is that all of Palestine is occupied by the Zionists. Therefore, ending the occupation for the former means that the State of Israel must agree to go out of business.

Posted by: SLC on March 31, 2007 09:05 AM

It is much better to think of the square roots of probability. Then negative support levels are imaginary. See -- simple.

Posted by: Robert the Red on March 31, 2007 09:44 AM

Well, I may not know much about statistics, but I know Bush has a negative rating with me. If everybody felt the way I do, I think we would have to count that as a negative approval rating.

Posted by: serial catowner on March 31, 2007 10:14 AM

Mr. Trevor continues to harp on the same old theme

Mr SLC appears to be engaging in kitchenware chromatic similitude.

Posted by: pseudonymous in nc on March 31, 2007 12:09 PM

Re pseudonymous

1. Its Dr. SLC.

2. I assume that Mr. pseudonymous is trying to say that I am engaged in musical forks.

Posted by: SLC on March 31, 2007 12:30 PM

Hmm. I didn't realize that the MoE reported for political polls made an assumption of normality. Why don't they use something like the Chernoff bound, which makes no such assumption?

Posted by: Umar on March 31, 2007 12:34 PM

RE: Umar

The confidence intervals used for calculating MOE's don't assume normality. They assume a Simple Random Sample of interviewees. The normal approximation just helps calculate the binomial probabilities.

And as mentioned above, the MOE is conservative (bigger than necessary).

Posted by: DC on March 31, 2007 02:47 PM

On top of everything else, be sure to remember that margin of error is merely a statistical measure. It says nothing about other types of error: poorly worded questions, sloppy interviewers, inattentive respondents, etc. It's hard to get any poll results near 0% or 100% because someone's usually either fuckin' up or fuckin' around.

Posted by: moriarty on March 31, 2007 05:15 PM

2. I assume that Mr. pseudonymous is trying to say that I am engaged in musical forks.

It's Dr pseudonymous, if we're pulling rank here. And you know exactly what I meant.

Posted by: pseudonymous in nc on March 31, 2007 05:55 PM

Matthew: it was possible Olmert had no support beyond his extended family

At the time when Olmert was chosen as acting PM, a long article was written on Olmert and his immediate family (in Haaretz?).

Bottom Line: His wife and children are liberals and don't agree with any of his Likud/conservative positions. They have agreed to disagree (while they do argue over dinner).

So, I'd guess that it is his work subordinates in the 3%, not his extended family.

Posted by: JimPortlandOR on March 31, 2007 07:56 PM

Missy SLC is still stuck in Judea and Samaria first gear. If the pre-coma Sharon could call it as it is "An Occupation" what's holding you back besides plain old neurosis? Israel isn't going away so stop pretending like a fucking moron that it is if it stops crucifying the Palestinians. If I've got you in a choke hold and others implore me to release the hold- am I within my rights to say: "Why, that would be suicide."?

Posted by: Trevor on March 31, 2007 08:05 PM

Re Trevor (aka Norman Finkelstein)

Mr. Trevor states that the State of Israel is not going away. Unfortunately, the Palestinians insist that the State of Israel must go away. Removing the occupation of the West Bank will not convince them of the error of their ways. It should be noted that the Gaza Strip is no longer occupied and the Palestinians there are using the former settlements therein as launching pads for their Kassem rockets. The fact of the matter is that the Palestinians consider all of Palestine to be illegally occupied territory. Until such time as the Palestinians disabuse themselves of this notion they will continue to be treated in a beastly manner by the State of Israel because they consider nice to be a sign of weakness.

Posted by: SLC on April 1, 2007 08:40 PM

porno porn

Posted by: porno on August 24, 2007 11:49 AM

Redtube 89 Porno Sex Seks güzel sözler acı sözler sohbet chat muhabbet adult

Posted by: Samet on August 24, 2007 12:14 PM

Redtube Porno Sex Seks Güzel Sözler Acı Sözler Chat Sohbet Muhabbet mirc

Posted by: Redtube 89 on August 25, 2007 12:23 PM

GüzeL SözLer Acı SözLer iLginç itiraflar Çılgınca şeyler sohbet chat muhabbet

Posted by: GüzeL SözLer on August 25, 2007 12:28 PM

Arkadaş Dost Canan Sohbet Muhabbet porno adult redtube 89

Posted by: Arkadaş on August 25, 2007 12:30 PM

Sohbet chat muhabbet mirc mirç sohpet çet