Before journalists can report the statistical findings of a poll, it is essential they understand the methodology involved in the development of the poll and the data collection. The first thing to determine is whether the poll is transparent about its methods. There are many factors that go into creating a poll, and each one can have a big impact on the poll’s results. Try to answer each of the questions below before writing about a poll.
Consider who are the respondents of the poll and how many there are. A poll of adults is different than a poll of registered voters and should not be compared. The number of respondents mainly determines the margin of error for the poll.
Also, who conducted the poll? Is it an experienced, reputable polling organization? Polling organizations with a reputation for accuracy are more likely to produce reliable results, although it is not always a sure thing.
Consider what questions were asked, whether they were worded simply and clearly, and whether they were asked without bias. Even subtle differences in the order and wording of questions can change the results. For example, the Quinnipiac University Poll found that respondents who were asked whether they supported or opposed “stricter gun control laws” opposed them, 51–46 percent. A different group of respondents in the same poll who were asked whether they supported or opposed “stricter gun laws” supported them, 52–45 percent. Not including the word “control” showed a notable difference in the results of the question, which is why specific and thoughtful wording of questions is so important.
Consider when the interviews were conducted in relation to the timeliness of a current event or topic. Was the poll conducted before or after a major event that could affect the respondents’ opinions? For example, if a presidential election poll was mainly conducted before a presidential debate but released after it, many voters who watched the debate could have changed their minds.
Consider why the poll was conducted. Does the organization who paid for it and/or conducted it stand to benefit from the results of the poll? The reason why the poll was conducted could bias the methodology or even the results themselves. For example, an organization may withhold data that reflects negatively on its goal and only release data that supports it.
Consider how the polling organization obtained the sample and reached the respondents. A random sampling method gives all people in a target population an equal chance of being surveyed. This is how to represent a specific population such as American adults or registered voters. Telephone surveys conducted by random digit-dialing are considered the best practice to obtain a random sample. Telephone surveys conducted with voter registration lists can be a good sampling method for the registered voter population. When samples are not chosen randomly, the results are not as accurately generalized for an entire population. Online surveys using internet-based groups of people are common for non-random sampling.
When it comes to telephone surveys, there is another factor that can have a major influence on the randomness of the sample, and therefore, the accuracy of the results. Telephone polls can be conducted by live interviewers or by robocallers. There are a few ways this can affect the results. The most important one is that live interviewers are allowed to call people on cell phones, while robocallers are not. Since more than half of the U.S. population is only reachable by cell phone, this has a significant effect on who can be surveyed by the polling organization.
It’s also important to determine how the surveys are weighted. Weighting adjusts poll data to correct any relevant ways the sample results might overrepresent or underrepresent certain groups in the population. Polls in the United States are commonly weighted according to Census population and demographic data, including but not limited to gender, age, race, region and education.
Again, be very skeptical if the above information is not made easily accessible.