Browsed by
Category: Polls

Numbers Need Context Too

Numbers Need Context Too

“There are three kinds of lies: lies, damned lies, and statistics.” — Mark Twain

But I can paraphrase what supporters of gun rights say: Numbers don’t lie. People lie.

People use numbers to lie.

Sometimes it’s unconscious. I can say, “Most people believe X.” But that word “most” is imprecise, and sounds like I may not have studied the subject enough. So “Nine out of ten people believe X” sounds like I’ve studied the people and counted them.

I want to just post a few notes and recommend a couple of books.

  1. Many (notice that I haven’t counted them) misrepresentations involving statistics result from not noticing the margin of error when sampling is involved. Words like “surge” or “plummet” are used about polls in newspapers when the changes are within that margin of error. The context here is in understanding the precise nature of the numbers themselves. There’s a difference between “there are five books on my table” and “the average American will have five books on their bedside table.”
  2. Ask how the numbers were generated. If it was a survey, what was the question? In professional surveys, you can trace the numbers back to the survey. Reliable, ethical researchers show their work. For example, if you ask a number of people how many would kill their own mother for a million dollars, you don’t know how many actually would. You know how many say they would. The context here is the underlying basis for the numbers. It’s the difference between “I would guess that about 5 in 10 people do x”, or “I asked them”, or “I had a hidden camera on them and watched them.” All three methods generate numbers, but the meaning is different.
  3. Numbers don’t generate predictions on their own. They represent a state of affairs defined by the way they are collected and presented. The context of a numeric prediction can be complex. The prediction is only as valuable as the theory that generates it from the numbers, even if it is represented in numbers.
  4. Probability is a complex field. I enjoy reading about it. Numbers used to express probability are often difficult to follow and seem unintuitive. For those of us who are not mathematicians, a key point to notice is whether the person involved could actually have the necessary information. If you cannot model in detail an entire process, you don’t know the probability. Let me illustrate from Star Trek–the original series. Spock and Kirk are on Organia, and are unaware of the nature of the Organians. Kirk asks spock to rate their chances, and he gives a number including decimals. This number is irrelevant, because Spock doesn’t actually know what he would need to know.
  5. In number comparisons, the context of all numbers compared is important, as is their relationship. The data in a comparison is not so much in the numbers themselves but in the theory or theories used to connect them.

In my opinion, most (note how I say this) news articles and popular presentations that involved numbers misrepresent their meaning in some way. In most (note again!) that error is not that relevant to the readers. But in many unfortunate cases it is. If you count headlines, the misrepresentation is worse.

So when reading numbers, look for the source, check the context, understand the theory. The numbers may be correct, but the theory and presentation may make them deceptive.

Herewith a couple of books that are quite readable on this subject.

How to Lie with Statistics. This is an older book, but provides the basics in a readable format.

Lies, Damn Lies, and Statistics. I found this more politically slanted, but the principles expressed are quite good, in my opinion. The bias I noted was in the examples. Note that I did not statistically check my impression of bias!

Statistics: A Spectator Sport. I haven’t read this book, but it’s on my reading list. The description notes that it uses examples largely from education. I suspect that could be valuable as it may be less controversial than using primarily political or media examples.

Accuracy of Predictions

Accuracy of Predictions

There’s very little checking done of the accuracy of the prediction of pundits, which is a major reason I avoid even hearing what they have to say.

One exception to this rule is Nate Silver and his crew at They’ve just published an analysis of as many of their predictions as possible. It’s worth reading, just for the demonstration of doing an analysis in the first place.

I suspect most readers/viewers of the news find probability hard to understand, and pundits generally don’t do that. People generally don’t want to hear probabilities; they’d prefer certainty. I have not done one, but I suspect a survey would show that people prefer a certain answer to an accurate-but-uncertain one.

So I like the headline: When We Say 70 Percent, It Really Means 70 Percent. Well, actually 71%. The things they gave a probability of 70% to happened 71% of the time. Not all percentages matched that accuracy, but it’s overall quite good.

More importantly, it tends to demonstrate the nature of prediction and the value of having evaluations. This makes me tend away from TV and radio as news sources and toward written sources in which I can check the sources. And, of course, toward written articles that actually cite sources.

Note: Check my source for this article!

Measuring a Liberal Bias in Psychology

Measuring a Liberal Bias in Psychology

As a self-professed passionate moderate (the liberal charismatic title was thrust upon me by an opponent), I’m very conscious of bias on both the liberal and conservative sides. To be human is to be biased. I have my moderate biases, including a bias toward considering anything from the left or the right obviously biased. You just can’t win with me!

A number of readers likely already know that is one of my favorite, of not my absolute favorite, news source. Besides their efforts to state their own biases, and the fact that I like numbers, this is a result of their efforts to cite their sources and show their work. If I question their rating of a pollster, for example, I can go look at what goes into that rating.

Before I get to the article I’m linking from them today, I want to emphasize something important. I like numbers, yes, but you have to be careful. The reason for this is that you have to understand how the numbers you’re liking were produced. Let me give an example. A friend asked me to read a book on the ancient world because I know the languages and he wanted an assessment of how much credence I should give it. In the book, someone gave measurements for the original size of the great pyramid in millimeters. There is no way the author could actually have that information. Numbers calculated in that way are designed to give the impression of precision even when such precision does not exist.

A more common way to produce a number is to assign it, such as asking people to rate something on a scale from 1 to 10. In order to know the question asked, how it’s asked, and who it’s asked of. After that you might consider asking what those people might know. For example, asking a random sample to rate the quality of cardiac care in this country on a scale from 1 to 10 produces information on how the sample views this, but might tell you as little as nothing regarding the actual state of such care, depending on who is being asked and what they could know.

So here’s the article, Psychologists Looked in the Mirror and Saw a Bunch of Liberals. (You need to read the article—the whole article. This material is useless without the reasoning behind it and the look for solutions.)

Someone noted the bias with a simple show of hands, and followed up with a study looking at the way in which results of studies were presented in journal abstracts. Here’s the generalization:

Sure enough, the abstracts more often explained their findings in terms of conservative ideas rather than liberal ones, and conservatives were described more negatively in the eyes of the raters.

The study authors tested for a bias in their raters and found that their liberal raters actually rated the abstracts as more negative regarding conservative views than did conservative raters. In a  separate test, they also note that a panel of psychologists surveyed for their expectation of bias expected the results to be more biased than the study showed they were. You should, in turn, read the note on the potential problem with the panel of psychologists surveyed.

Note to self: Doing a deep enough study on an issue to have a strong opinion is a lot of work and takes a lot of time!

One of the solutions suggested is studies done by “trans-ideogical teams,” i.e., have research done by people who expect different results and who then design a study based on what would change their mind on the topic. I like this idea quite a lot.

I’ll note that this has a great deal to do with the way I publish (my company). I look to create conversation between people of widely differing viewpoints. (This is not identical to creating a church congregation, where some identity is necessary. I also support diverse congregations, but the boundaries will be set up differently.) I believe that in learning, there is great value in hearing the opposing position from someone who actually supports it.

A conservative professor requiring readings from a liberal book and explaining liberal ideas is not as challenging as hearing from an actual liberal. Similarly, if you reverse liberal and conservative. I have lived and learned in situations dominated by conservatives and at other times in ones dominated by liberals. The result I see is the same: Complacency, laziness, and arrogance. One decides one doesn’t have to have support for an idea because “everybody knows that.” But this “everybody” is a very selected subset.

I don’t see any solution here except intentionally involving people who disagree. I have found for myself that I cannot truly express the support for an idea I don’t accept myself nearly as well as a person who truly does support it, even if I try diligently.

This article is encouraging to me because it attacks bias in two ways: 1) Identifying and quantifying it, and 2) Looking at ways to correct for it.

Changing Polls

Changing Polls

No, President Obama’s approval rating; the poll that I have in the right hand sidebar.  It has been there for more than 18 months, and surprisingly enough is still generating interest.  The last comment is dated December 19, 2009 and there have been quite a number.  You can see the results here.

 Who does God hate?  
Selection   Votes 
Everyone, because we are all sinners  3% 13 
Only unrepentant sinners  9% 40 
All non-Christians  2%
Those who have never said the sinner’s prayer  0%
Nobody, God loves everybody  77% 330 
Other (please comment)  9% 39 
431 votes total free polls

In case anyone wants to keep this poll alive even further past its sell-by date, I’m including the form as well:

Who does God hate?
Everyone, because we are all sinners
Only unrepentant sinners
All non-Christians
Those who have never said the sinner’s prayer
Nobody, God loves everybody
Other (please comment) free polls
New Poll: Who Does God Hate?

New Poll: Who Does God Hate?

With the number of posts on this topic, I’d like to get an idea of how readers of this blog think or feel about God and hate.

You’ll find the poll in the right sidebar. Please feel free to comment either here or on the poll page itself. I suspect this one needs a comment or so!

Bible Reading Poll

Bible Reading Poll

I think it’s about time to change polls, so here are the results of the last one:

What is your most common reason for reading the Bible (i.e. the one that causes you to spend the most time reading)?

Selection Percentage Votes
Devotional 23% 8
Guidance for a specific personal situation 6% 2
Ethical guidance 3% 1
Learning correct doctrine 14% 5
Hearing God speak 40% 14
Historical or technical interest 9% 3
I don’t read the Bible 0% 0
Other (please comment) 6% 2
35 votes total

There were three comments left:

From Gideon on March 17, 2008 at 1:34 am.
to find out how to do good. Not that I am
good, but I want to know what is
“biblically good”.

Joh 5:29 NETB … the ones who have done what is good to the resurrection resulting in life, and the ones who have done what is evil to the resurrection resulting in condemnation.

From Mark Thompson on March 4, 2008 at 2:14 am.
I ticked hearing God speak, but to be more correct I would add ‘as man understood at that time and place’

From Philo on February 19, 2008 at 11:09 am.
“What is your most common reason for reading the Bible (i.e. the one that causes you to spend the most time reading)?”

For its poetry.

I was interested in how many chose “hearing God speak” and I wonder how many of those would agree with the second comment made by Mark Thompson.