Bias Survey: Read It And Weep

[djAdvocate]

graphics8.nytimes.com/images/2012/11/10/us/politics/fivethirtyeight-1110-accuracy2012-1/fivethirtyeight-1110-accuracy2012-1-blog480.png

a lot of polls did really really well. and several did very poorly. here are the "HOT FIVE":

IBD/TIPP
Grove
Mellman
We Ask America
Quinnipac

make a note of them. remember them for next time. they were all off by less than 0.5%

here are the "S(*TCAN FIVE":

Gallup
Rasmussen
American Research Group
Gravis Marketing
Washington Post/ABC News

make a note of them. ignore them in the future. they were all off by over 2.5%. the first three, in particular, were biased OUTSIDE OF POLLING ERROR. aka = they are methodologically flawed. not to be considered seriously unless they fix things.

the two most surprising ones are IBD (which is a really right wing publication that was basically spot on in their polling), and the Washington Post (which is a pretty prestigious paper that did some really crappy polling).

[workpublic]

you're surprised that a "right wing" source wasn't bias?
and also surprised that a "mainstream" source was?


that sounds bias ;D

[Deleted]

I just bought Nate Silver's book for FIL as a Christmas gift.

[Gardening Grandma]

Nov 17, 2012 13:51:58 GMT -5 @anne81 said:

I just bought Nate Silver's book for FIL as a Christmas gift.

I'm putting that on my Amazon wishlist....

[djAdvocate]

Nov 17, 2012 9:57:21 GMT -5 workpublic said:

you're surprised that a "right wing" source wasn't bias?
and also surprised that a "mainstream" source was?


that sounds bias ;D

yes- but i never claimed that i am without bias. that is why i tend to consider data rather than my own "feelings".

[djAdvocate]

Nov 17, 2012 13:51:58 GMT -5 @anne81 said:

I just bought Nate Silver's book for FIL as a Christmas gift.

Nate Silver was basically 100% right. the only major race he didn't call right was Montana Senate. he got all 50 states right.

[EVT1]

Statistics done right are hard to argue with- and he put it on the line calling 80%+ for Obama. He has earned some respect. Might read the book- not really my thing, but I liked moneyball and not really stats but freakonomics- the methodologies are interesting.

[Virgil Showlion]

Nate Silver was basically 100% right

A highly erroneous conclusion, I'm afraid. :-\

the first three, in particular, were biased OUTSIDE OF POLLING ERROR. aka = they are methodologically flawed.

You cannot make this conclusion based on a measurement falling outside the 95% confidence interval (or any confidence interval, for that matter). The best you can do is obtain what is called a "posterior distribution" for the estimation bias, which permits you to estimate the probability that bias exceeds a certain threshold, given certain assumptions about the poll itself.

[djAdvocate]

Nov 18, 2012 17:39:33 GMT -5 Virgil Showlion said:

Nate Silver was basically 100% right

A highly erroneous conclusion, I'm afraid. :-\

not really. if you bet every state that Silver predicted Obama would win, you would have won every bet. that was my point. nobody can predict what will happen with absolute certainty, Virgil. you are reducing the argument to semantics, but i stand by my point. Silver's numbers were basically correct, and i couldn't care less if you or anyone else agrees with that. it is a fact.

[djAdvocate]

Nov 18, 2012 17:39:33 GMT -5 Virgil Showlion said:

the first three, in particular, were biased OUTSIDE OF POLLING ERROR. aka = they are methodologically flawed.

You cannot make this conclusion based on a measurement falling outside the 95% confidence interval (or any confidence interval, for that matter). The best you can do is obtain what is called a "posterior distribution" for the estimation bias, which permits you to estimate the probability that bias exceeds a certain threshold, given certain assumptions about the poll itself.

i have no idea what you are talking about. but please don't explain it. i am on vacation. ;D

[Virgil Showlion]

you are reducing the argument to semantics, but i stand by my point. Silver's numbers were basically correct, and i couldn't care less if you or anyone else agrees with that. it is a fact.

Your definition of "correct" as being "only events assigned a probability of 50% or greater occurred" would be deemed absurd by any statistician.

If Mr. Silver is indeed a respecter of statistics, the last thing he needs is fanboys filling up message boards with claims that ignore the stochastic nature of his predictions.

Consider this one of my pet peeves. You hate it when people misuse terms in the English language. I hate it when people willfully ignore the non-determinism of a problem. I enjoyed Mr. Silver's 538 blog precisely because he wasn't afraid to embrace, quantify, and even chart the uncertainty in his predictions.

[djAdvocate]

Nov 18, 2012 18:17:51 GMT -5 Virgil Showlion said:

you are reducing the argument to semantics, but i stand by my point. Silver's numbers were basically correct, and i couldn't care less if you or anyone else agrees with that. it is a fact.

Your definition of "correct" as being "only events assigned a probability of 50% or greater occurred" would be deemed absurd by any statistician.

statisticians would find your insistence on 100% probability absurd, as well. so i guess we are good company for each other.

[djAdvocate]

Nov 18, 2012 18:17:51 GMT -5 Virgil Showlion said:

you are reducing the argument to semantics, but i stand by my point. Silver's numbers were basically correct, and i couldn't care less if you or anyone else agrees with that. it is a fact.

Your definition of "correct" as being "only events assigned a probability of 50% or greater occurred" would be deemed absurd by any statistician.

If Mr. Silver is indeed a respecter of statistics, the last thing he needs is fanboys filling up message boards with claims that ignore the stochastic nature of his predictions.

Consider this one of my pet peeves. You hate it when people misuse terms in the English language.

i hate very little. i will vigorously dispute the misuse of perfectly good and honorable terms for political means, however.

I hate it when people willfully ignore the non-determinism of a problem. I enjoyed Mr. Silver's 538 blog precisely because he wasn't afraid to embrace, quantify, and even chart the uncertainty in his predictions.

Virgil- i am sorry i said he was "basically 100% correct". that seems to have really bent your noodle. but you have to admit that Silver nailed it, everywhere. he predicted a fairly even outcome in Florida. it was an even outcome. he predicted that Obama would win in Colorado and Virginia fairly easily, even while people like me and VB debated it. he was right. there is a lot to argue about in terms of certainty, but Silver not only got the black and white of the election right, he got most of the grey right, as well.

[Virgil Showlion]

statisticians would find your insistence on 100% probability absurd, as well. so i guess we are good company for each other.

They would object to the use of the term "correct" in the context of a non-deterministic system.

If mmhmm showed us a fair die and asked us to predict the outcome, and you replied "a 1/6 probability of each integer from 1 thru 6", and I replied "a 100% probability of the number 1", and mmhmm rolled a '1', my prediction would not be more correct than yours.

If mmhmm showed us a fair coin and asked us to predict the outcome, and you replied "a 50% probability of each heads and tails", and I replied "a 100% probability of tails", your model would have no more worth than mine in determining future outcomes. This fact would not change from before the coin flip to after the coin flip, regardless of the outcome.

i hate very little. i will vigorously dispute the misuse of perfectly good and honorable terms for political means, however.

My point exactly.

I'm using "hate" here in the sense of "vigorously opposed to the misuse of", and you're vigorously opposing my misuse of the word 'hate'.

he predicted a fairly even outcome in Florida. it was an even outcome. he predicted that Obama would win in Colorado and Virginia fairly easily, even while people like me and VB debated it. he was right. there is a lot to argue about in terms of certainty, but Silver not only got the black and white of the election right, he got most of the grey right, as well.

He was the best of the lot in this round.

If we believe the uncertainty margins that he himself published on his blog, the fact that his modal (i.e. most likely) prediction coincided perfectly with reality was a fluke of random chance.

We must recognize this because Mr. Silver's future modal predictions may be significantly less "correct", but this does not mean that his model is incorrect, or flawed, or biased. It does not mean he's losing his edge. Conversely, if his modal predictions continue to coincide perfectly with reality, it means that all of the variance analysis he's put up on his blog is bunkum. You cannot have a stochastic system with non-zero entropy and be "correct" 100% of the time.

Given the variance analysis that Mr. Silver did put up on his blog, his "100% correctness" was indeed a rather large fluke. I can't recall the charts exactly, but I do recall the modal outcome had a < 10% chance of occurring.

[djAdvocate]

Nov 19, 2012 8:30:52 GMT -5 Virgil Showlion said:

statisticians would find your insistence on 100% probability absurd, as well. so i guess we are good company for each other.

They would object to the use of the term "correct" in the context of a non-deterministic system.

well, i think i made it clear what i meant by that. again, i apologize if it bothered you, but it was the simplest way i could put it.

[djAdvocate]

Nov 19, 2012 8:30:52 GMT -5 Virgil Showlion said:

If we believe the uncertainty margins that he himself published on his blog, the fact that his modal (i.e. most likely) prediction coincided perfectly with reality was a fluke of random chance.

We must recognize this because Mr. Silver's future modal predictions may be significantly less "correct", but this does not mean that his model is incorrect, or flawed, or biased. It does not mean he's losing his edge. Conversely, if his modal predictions continue to coincide perfectly with reality, it means that all of the variance analysis he's put up on his blog is bunkum. You cannot have a stochastic system with non-zero entropy and be "correct" 100% of the time.

a fact that Nate Silver points out on a regular basis.

Given the variance analysis that Mr. Silver did put up on his blog, his "100% correctness" was indeed a rather large fluke. I can't recall the charts exactly, but I do recall the modal outcome had a < 10% chance of occurring.

untrue. the 332 EV outcome was the most probable, at 20%. it is still at the website.

[Virgil Showlion]

Nov 19, 2012 18:01:38 GMT -5 djAdvocate said:

Nov 19, 2012 8:30:52 GMT -5 Virgil Showlion said:

If we believe the uncertainty margins that he himself published on his blog, the fact that his modal (i.e. most likely) prediction coincided perfectly with reality was a fluke of random chance.

We must recognize this because Mr. Silver's future modal predictions may be significantly less "correct", but this does not mean that his model is incorrect, or flawed, or biased. It does not mean he's losing his edge. Conversely, if his modal predictions continue to coincide perfectly with reality, it means that all of the variance analysis he's put up on his blog is bunkum. You cannot have a stochastic system with non-zero entropy and be "correct" 100% of the time.

a fact that Nate Silver points out on a regular basis.

Given the variance analysis that Mr. Silver did put up on his blog, his "100% correctness" was indeed a rather large fluke. I can't recall the charts exactly, but I do recall the modal outcome had a < 10% chance of occurring.

untrue. the 332 EV outcome was the most probable, at 20%. it is still at the website.

The last time I checked was a week before the election, and the distribution wasn't as tight as it is now (presumably the day before the election).

Still, a 20% chance of the modal distribution makes my point just the same.