LDS Church Growth, Member Activity, and Convert Retention: Review and Analysis
III-09: Outcome Reporting Bias
Outcome reporting bias is to researchers what social acceptability bias is to respondents, skewing reported outcomes towards those deemed favorable while ignoring contrary data. Often this bias is unintentional, resulting from flawed methodology or lack of awareness. In other cases, outcome reporting bias may be intentional, as when a conflict of interest exists or when statements are presented to serve public relations purposes rather than to objectively inform.
Outcome reporting bias has been pervasive in missiologic research and writing on LDS church growth. Of the numerous books and histories on the international LDS Church and in the many hundreds of articles in the LDS Church News and the Ensign, a common pattern is seen. Historians and journalists to describe in detail their experience with a handful of faithful members of whom they have direct knowledge. Overall church membership statistics for the nation are then presented. It is implied to the reader that the individuals interviewed or described are representative of church members in that nation, and the experience of the few is then extrapolated to total official membership.
Although the critical methodological flaws of such an approach should be immediately apparent to any trained researcher, it is easy for authors to commit this error without realizing it. This is because members attending congregations on Sunday represent a selected group out of total LDS membership. If one visits a congregation, one by definition meets only members who have had a positive outcome: the active ones. Inactive, disengaged, and hostile members are not present, and being out of sight, they are also out of mind. The average visitor has no idea of the total denominator of members on congregational membership rolls, or how many more are on the address unknown file. It is likely that journalists and historians also consider the stories of inactive or disaffiliated members to be less interesting or worthy; most seek inspirational success stories, rather than objectively seeking samples that are truly representative. There is a place for inspirational stories, yet such tactics can quickly become misleading when an exclusive focus on faithful active members is extrapolated to total membership numbers and is then purveyed as representing quality journalism or history.
Journalistic articles and histories also pay almost all focus on active members because most time and resources are directed toward active members. The overwhelming majority of church-related contact church members have is with other active members. Similarly, a Church leader who visits a nation may be greeted by thousands of dedicated Saints, whereas disengaged inactives are never seen and are noted only briefly when statistical reports are reviewed. Inactives and disaffiliated members are by definition more difficult to reach, and can pose difficulties even for researchers acutely aware of the problem.
To minimize or avoid these problems, research based upon interviews with active members must also present data regarding total activity and member participation, or must be carefully qualified to ensure that findings are not extrapolated to LDS membership as a whole. Special attention and diligence are needed to ensure that the issue of inactivity and the whole denominator, including members not present are kept in mind.