Applied Research Northwest ::

MEASUREMENT:
What is a Response Rate?
There are many ways that errors in survey research lessen the accuracy or meaningfulness of social survey data. One of the most often ignored issues is that of the response rate. A response rate is simply an indication of how many people responded to your survey, given the number of people who were invited to participate. As a general rule, we know that the more people you invite who participate, the higher the accuracy of your findings.

Consider this example. A researcher wants to evaluate people’s opinions of an open house event. The program’s goal is to familiarize the community with plans for a new hospital building project. She attends the event and waits near the exit door to recruit people as they are leaving to complete a brief survey about the event. She takes home 100 completed surveys after attempting to recruit 500 of the people attending the event. Her response rate would be 100/500 or 20%.

At first glance this might seem like a successful project, and it does have some strengths. She has substantial amount of feedback from people who attended the event, and she got it quickly. If she used a method to help her randomize her selection of people to recruit, she undoubtedly sampled a representative group of attendees.

However, consider how the people who refused to do the survey or ignored her might differ from those she successfully recruited to participate. Perhaps they didn’t take the time because they are very busy. Or worse, suppose those that passed her by didn’t take the time because they didn’t like what they saw at the presentation. Research has consistently shown over the years that people who are positively oriented toward a given topic or experience tend to be the first to want to contribute to its review, as well as those who have more leisure time, less demanding schedules and fewer competing commitments.

When studies experience low response rates, they will tend to be biased toward the positive. They will overestimate people’s positive experience and miss the more critical ones. This is a danger for organizations engaged in program improvement since criticism is often some of the most useful information managers can get.

Methods for Managing Response Rates
Survey research organizations often report difficulties in getting high response rates. The proliferation of direct mail for marketing as well as telemarketing has certainly made potential respondents harder to reach. Rigorously executed mail surveys, with repeated reminders and telephone follow-ups, generally yield only a 40% response rate at best. Telephone-only surveys tend to have higher overall response rates, but only if certain methods are applied. ARN uses the following methods to obtain high response rates – we find the extra effort pays off in reliability and confidence in the data without contributing substantially to costs.

Method 1. Varied call times
Daytime calling tends to reach a disproportionate number of elderly, shift workers, stay-at-home parents and the unemployed. Nighttime calling is better for reaching those who work during the day, and weekends (especially Sunday afternoon and evening) tends to produce a very balanced mix of respondents. ARN ensures that every potential respondent is tried on at least three different days including one weekend call and one daytime call.

Method 2. Repeated attempts
Because researchers generally lack information about when respondents are likely to be home, each potential respondent must be attempted repeatedly until better information is available (for example a roommate or spouse provides good times to call). A minimum of 3 attempts should be made on any given research project. For healthcare-related research and program evaluation ARN recommends six attempts be made to reach each potential respondent.

Method 3. Using Calendar time
Repeated attempts and varied call times all exercised within just 2-3 days tends to yield poorer representation of the population. By distributing attempts over the course of a couple of weeks, researchers can allow for people who are traveling to come home, or those who are sick to heal, or those who are busy to have their schedules change.

What you can ask for:
Any survey research vendor should be able to provide two pieces of information to let you know if your response rate is low and how it might be affecting your survey information. First, ask for the outcome of every respondent that was attempted, including the number of attempts that were made. This should be presented in a simple table that shows what’s called the “final disposition” for the sample (see below). You can calculate your own response rates by dividing the total number completed by the number attempted that were in working order.

Table 1. Call Results Summary
	N
Complete	300
Refused	138
No Answer/Machine/Busy	86
Callback Scheduled	21
Total Valid Contact	545
Bad Number - Line out of Order	86
Bad Number - Business	8
Unable - Language	3
Unable - Physical/Mental	30
Household Not Qualified - Age	0
Maximum Attempts Reached - 6x	270
Total Invalid Contact	397
Total Telephone Numbers Attempted	942
Response Rate (completes/valid contacts)	55%

Second, ask for a comparison on known characteristics for those who were attempted and those who completed the survey. Even by comparing just the age or sex distribution of each group, you should be able to see if there are pockets of patients who are being missed by the researcher.

Health care research has the benefit of being extremely relevant to respondents – people are generally very motivated to provide feedback to researchers about their healthcare experience. As a result, ARN consistently achieves 60-70% response rates in its patient follow up work.