Survey Research and Acceptable Response Rates

The adequacy of response rates to online and paper surveys

Nulty, D. D. (2008). The adequacy of response rates to online and paper surveys: what can be done?. Assessment & Evaluation in Higher Education,33(3), 301-314. https://www.uaf.edu/files/uafgov/fsadmin-nulty5-19-10.pdf

Abstract:

This article is about differences between, and the adequacy of, response rates to online and paper-based course and teaching evaluation surveys. Its aim is to provide practical guidance on these matters. The first part of the article gives an overview of online surveying in general, a review of data relating to survey response rates and practical advice to help boost response rates. The second part of the article discusses when a response rate may be considered large enough for the survey data to provide adequate evidence for accountability and improvement purposes. The article ends with suggestions for improving the effectiveness of evaluation strategy. These suggestions are: to seek to obtain the highest response rates possible to all surveys; to take account of probable effects of survey design and methods on the feedback obtained when interpreting that feedback; and to enhance this action by making use of data derived from multiple methods of gathering feedback. 

When the more traditional and conservative conditions are set, the best reported response rate obtained for on-paper surveys (65%) is only adequate when the class size exceeds approximately 500 students. The best reported response rates for online surveys (47%) are only adequate for class sizes above 750 students. The 20% response rate achieved for online surveys by Griffith University would not be adequate even with class sizes of 2000 students.

------------------------------------------

Typical Response Rates for Common Survey Types
Surveys that you distribute internally (i.e. to employees) generally have a much higher response rate than those distributed to external audiences (i.e. customers).

Internal surveys will generally receive a 30-40% response rate (or more) on average, compared to an average 10-15% response rate for external surveys.


3 Ways to Improve Your Survey Response Rates

To help improve your survey response rate keep these key factors in mind:

1. Survey Design: Research has shown that surveys should take 5 minutes or less to complete. Although 6 – 10 minutes is acceptable, those that take longer than 11 minutes will likely result in lower response rates. On average, respondents can complete 5 closed-ended questions per minute and 2 open-ended questions per minute.

2. Provide Clear Value: Offer a copy of the final results to all those who complete the survey, and, if appropriate, consider offering an incentive. If you plan to take action based on the results of your survey, make those clear in your survey invitation. Remember, people will be more likely to respond if they understand how that time will be spent.

3. Send Reminders: As the close of your survey approaches, gently nudge those who haven’t finished yet. Limit yourself to no more than two reminder emails, changing the time of day and the day of the week that you send them out so that you can reach as many different respondents as possible.

------------------------------------------------------------------------

Survey Sample Size Calculator

http://fluidsurveys.com/survey-sample-size-calculator/

When it comes to probability surveying, creating a sample size should never be left to guessing or estimates. Instead, it should be based on three criteria:

The size of your target population: This refers to the total amount of people that are eligible to participate in your survey. For example, a study on Ontario citizens’ sleeping habits would have a population equivalent to that province’s population (13.5 million). In many studies it will be impossible to know how many people make up a population. If this is the case, it is accepted among researchers to use a fake population size of 20,000 or larger.
Your desired confidence level: Usually placed at a value of 95% in surveying, the confidence level describes how sure you can be that your results are correct. With a 95% confidence level, a researcher can be certain that the value of any sample will fall in the range of the margin of error 95% of the time.
Your allowed margin of error: Margin of error depicts the random sampling error that is possible in the study. This is important because it is impossible to know whether a sample’s results are identical with the true value of the population. The value allotted to the margin of error describes the range in value that the population may have based on the results in the study. This is always described as a plus or minus value.

For example, most people choose a margin of error 5+/- with a 95% confidence interval. If your results showed that 67% of people love rock music, you could say that you are 95% confident that 62-72% (known as the confidence interval) of your targeted population love rock music.