LDS Church Growth, Member Activity, and Convert Retention: Review and Analysis
III-03: Survey Methodology
When designing a survey, the first step is to ensure that the population being interviewed or surveyed accurately represents the overall group one wishes to research, and the second is to ensure that the questions asked are relevant and meaningful. Methodological issues must be carefully considered, because significant unintended error or bias can be introduced by study design even when researchers are conscientious and candid.
Appropriate randomization is necessary for sampling to be valid. Non-randomized studies or those based on self-selected respondents, such as internet surveys, are not scientific in design, because the characteristics of self-selected respondents are often different from those of the larger population.
With surveys based on random sampling, there is always the question of how closely survey respondents reflect the population as a whole. If the survey methodology is biased towards a sample that does not accurately represent the entire group, or if important segments of the larger group are not adequately sampled, conclusions based on the non-representative sample may be erroneous or misleading. Some survey methods, such as telephone interviews, may skew the respondent profile by excluding individuals without land-line phones and those who are not home when calls are made or who do not return survey phone calls.
The question of how the religious profile respondents may differ from that of non-respondents or those not reached by study methodology is often difficult to answer, but must be carefully considered both in study design and subsequent analysis. The errors in some cases may be small enough to be inconsequential, but in large errors can undermine the applicability of study results. Good researchers carefully consider the potential for such errors in their study design, and thoughtfully disclose any methodological limitations, including those deemed likely to be inconsequential.