Cumorah.com News & Updates
View All News
Response to Mormon Studies ReviewMonday, 1st October, 2018 04:28 AM
PDF version at http://cumorah.com/Mormon_Studies_Review_Response.pdf
December 13, 2015
Dear Mormon Studies Review Staff:
We are deeply concerned by numerous breaches of the scholarly process in the review of our work, Reaching the Nations, in the current edition of Mormon Studies Review. Many of the criticisms involve alleged inaccuracies in which the reviewer is demonstrably in error, producing a highly misleading impression of our work. We wonder why the editors at MSR were not able to perform a basic reference check which would have found that your reviewer’s allegation that we had misquoted the title of an article was in error and that our citation was correct, or to check our work to ensure that it actually stated what a reviewer claims when it does not, or why the editors lacked the familiarity with the history or geography of the regions discussed which would have enabled them to catch a number of glaring reviewer misstatements. The pattern of errors in MSR's review of our work transcends mere sloppiness to sheer recklessness. It does not appear to us that the editorial process at MSR, as reflected in the review of our work, approaches the core academic standards of reputable scholarly journals.
Although we are disappointed with Mr. Decoo's inaccuracies and misstatements, the primary responsibility for allowing a defective product to go to press lies squarely with the MSR editorial staff. The editors took no discernable effort to avail themselves of the scholarly tools and processes that are necessary to ensure the accuracy of the printed comments, whether by conducting a fact-check of Mr. Decoo's statements, engaging an assistant to verify references, or perhaps even by checking key points with the authors if the reviewer lacked clarity or understanding of the work, to ensure that the reviewer comments could withstand some degree of scrutiny before printing them as if they were valid and authoritative. We find irony in the critique of our work regarding obscure items when reviewers critique our sources such as the LDS Church News and other reference works – which we have cited correctly – whereas the MSR reviewers apparently took no effort to check the accuracy of Mr. Decoo’s claims, many of which are demonstrably false. We have documented this in our attached response. Mr. Decoo’s diatribe in which he misquotes our text and presents his off-the-cuff ramblings as fact even when they contradict the consensus of geologists, historians, and reference sources, does not represent scholarship of any kind. There are more errors in Decoo’s claims which we have mercifully omitted, but would be pleased to submit if requested.
We are also disappointed with the lack of scholarly balance and impartiality of the MSR editorial staff as reflected by their decision to print long selections of Mr. Decoo’s review which were entirely negative and contain numerous factual errors, while failing to accurately represent his full review which also makes strong positive statements regarding the contribution of our work. The numerous errors and misstatements in MSR’s review are unfair to the authors, as well as to Mr. Decoo, who we could have handled more gently in our response if MSR had printed balanced remarks that were more representative of his overall review, or if a competent editor had caught and corrected at least some of Decoo’s many errors before going to press.
We are astonished at the arrogance and lack of self-insight with which MSR deigns to “review” our work while failing to follow the basic scholarly principles and processes that it ironically proclaims. We had concerns before the review regarding MSR and its predecessor, FARMS Review, due to what we perceived as a pattern of bias and inconsistently applied academic standards. We were also concerned with MSR’s lack of experience with any writing which would be comparable to our work, or even in the same discipline of scholarship, but hoped that MSR would step up to the task. We regret that this has not been the case. We wonder if the editors at MSR are really so careless, or if the current review was motivated by a non-scholarly agenda. We hope that these errors represent an unusual breach of the academic principles which MSR professes, and not a modus operandi founded in deeper issues of bias and lack of scholarly integrity.
Adherence to the principles of the scholarly process would have resulted in a review which was more factually accurate and more reflective of the authors' work, as well as saving MSR the embarrassment in having to subsequently address a litany of errors which slipped through their editorial process. We believe that scholarship is a search for truth based on the consistent application of objective standards and scientific processes. We believe that scholarly rigor should not be an empty polemical slogan for the editors and reviewers of MSR, but must be reflected in the standards and procedures of the journal itself.
Fortunately, the scholarly process offers us recourse on this matter. Consistent with the policy of reputable scholarly journals in printing authors’ responses and corrections when substantive errors of reviewers or editors are identified, we expect that you will uphold the standards of the academic community by publishing our attached response in the next print edition of MSR. Printing our response will hold the reviewer firmly accountable for his misstatements of fact, while affirming MSR’s commitment to academic integrity and to the scholarly process. Readers understand that everyone makes mistakes and even the most prestigious journals must periodically issue errata or print rebuttals to poorly-implemented pieces. However, a failure to rectify errors once identified would be a far more serious matter. Failing to allow us to respond in the same medium to documentable errors in your publication regarding our work would represent a major breach of academic integrity and objectivity on the side of MSR. We expect that you will be eager to remedy these issues promptly and in a matter consistent with accepted standards of scholarship.
We are also confident that our reply sheds considerable light on the study of LDS Church growth and makes a number of relevant points which will be of some interest to your readers. We are willing to redact or modify our response to meet MSR’s specifications, so long as standards of scholarship and academic integrity are not compromised. We are also willing to provide additional documentation on any points as deemed necessary.
MSR REVIEW REPLY
We appreciate the selection of our work, Reaching the Nations, for review in Mormon Studies Review. We value the role of careful scholarship and peer review in the development and evaluation of significant works in furthering the understanding of issues related to Mormonism and church growth. We applaud Mormon Studies Review for expanding its horizons to consider our work, which represents a broadening of horizons from the topics that have traditionally been presented and reviewed in MSR and its predecessor, FARMS Review.
The selection of a panel of twelve reviewers, all experts regarding the LDS Church in different world areas, was also an ambitious project for the MSR editorial staff, and it is evident that they took some pains to achieve balance in selecting reviewers with expertise representing most areas of the international Church. We appreciate the thoughtful and open-minded approach that eleven of the twelve reviewers took in regard to our work. We appreciate in the generally warm reception of our work and the recognition of its place in church growth studies by competent scholars.
However, we were disappointed by the comments of one reviewer, who in our view failed to adhere to academic standards. Wilfried Decoo has alleged a number of sloppy, inaccurate, and “muddled” misstatements of fact in our work. However, few of the points he cites as errors in our text stick when examined. On many of these items, Decoo is demonstrably wrong; on others, his statements involve fine hair splitting or his presentation of personal opinions as fact where we believe that different views may legitimately exist. Some of his comments also demonstrate a significant misunderstanding of our work. It is also unfortunate that the MSR editors chose to print unbalanced long and entirely negative selections from Decoo which do not constitute a balanced selection from his full review, which finds much positive in our work. We are also disappointed that the MSR editors apparently failed to take any effort to verify Decoo’s factual claims as would befit their ostensible scholarly standards.
The review process involves reciprocal rights and responsibilities to ensure accuracy, impartiality, integrity, and adherence to scholar standards by the authors, the reviewers, and the journal. Reviewers are welcome to share their own opinions and perspectives, but have a responsibility for factual accuracy and for adherence to academic standards. The authors have a responsibility to produce a work of scholarly merit, and the right to insist that the factual claims by reviewers be accurate. The editors have a responsibility to ensure that statements printed in their journal are factually correct and accurately referenced. The scholarly process requires that claims must be supportable, and individuals should be held accountable for their statements. In the event that this process is breached and a reviewer has made substantial misstatements of fact which have eluded the editors, the authors have the right to respond in the same venue with corrections and clarifications.
Decoo disputes our statement that “Belgium colonized the Congo in the nineteenth and twentieth centuries,” claiming that colonization occurred only “in part of the twentieth century,” but provides no source. Although the political entity called the Belgian Congo existed from 1908 to 1960, “Colonial rule in the Congo began in the late 19th century” with the personal colony of King Leopold of Belgium called the Congo Free State being organized in 1885 on the entire territory of the current Democratic Republic of Congo. Our statement is correct and Decoo’s attempt to restrict the period of Belgian colonization of the Congo to the existence of the political entity of the Belgian Congo is in error.
The brutal nineteenth century Belgian colonial exploitation and genocide in the Congo, which is blamed for up to ten million excess deaths – more than the Holocaust – and was brought to Western attention by Joseph Conrad’s novel Heart of Darkness, is according to Mr. Decoo, a figment of the imagination to be brushed away with hand-waving. To the Congolese, the murderous legacy of nineteenth century Belgian colonialism is very real indeed and continues to have a profound impact on the present. The claims of Mr. Decoo as printed in MSR in denying nineteenth century Belgian colonialism in the Congo and the associated atrocities are not unlike the claims of modern-day Holocaust deniers, and are no less absurd.
Mr. Decoo claims that we have “muddle[d] the title of the article,” “The Church in Belgium: Membership has international flavor in Church version of United Nations ,” suggesting that the phrase “in Church version of the United Nations” represents our fabrication. Yet a simple search demonstrates that we have correctly referenced the title as it appears in the LDS Church News Archive; it is Mr. Decoo and the MSR editors who are muddled.
Decoo has attributed to us the view that the Ardennes represent a distinct non-overlapping region from Wallonia; this is his inference which is not found in our text. Decoo claims that the Ardennes have no real “mountains,” whereas the sources we have consulted note acknowledge that the Ardennes result from the "geological features of the Ardennes mountain range and the Moselle and Meuse River basins." While we acknowledge that the Ardennes are low and quite unimpressive mountains by the standards of the Alps or the Rocky Mountains (although, by the accounting of the geologists, the Ardennes are more ancient than either of the latter, being formed in the Givetian stage of the Middle Devonian), we accept the authority of the geologists rather than Decoo’s subjective view that these mountains are not real. We do agree that Wallonia occupies not only middle Belgium, but most of middle and all of southern Belgium.
Decoo claims that cigarette consumption in Belgium is low, at least for Europe, and not high as we state. Approximately 19% of Belgians smoke cigarettes daily, which is on the low end for the European region. The American Cancer Society cites tobacco use in Belgium at 2353 cigarettes smoked annually per person over 15 in Belgium in 2014: one of the highest rates in Europe, compared to 1504 in Bulgaria, 1480 in Germany and 992 in France, according to one source. This discrepancy may be resolved in part by noting that smoking prevalence, or the percentage of individuals smoking daily, may be lower than in many other European countries, whereas smoking intensity, or the number of cigarettes smoked per smoker, is high. This leaves our assertion that “cigarette consumption rates are high” in Belgium as correct, even if a smaller percentage of the population may be responsible for greater per capita cigarette consumption.
We have followed the lead of many sources in citing the Spanish control of Belgium as beginning with the accession of Charles V (also called Carlos I of Spain) to the throne of the Holy Roman Empire as in 1519. We appreciate Decoo’s acknowledgement that this “error” is found in many English sources, although it is not clear to us that it is an error, and we reject Decoo’s statement that “the region became Spanish when it was inherited by Philip II of Spain in 1556” as simplistic and inaccurate. Philip II was Charles V’s son, and we find little basis for Decoo’s ostensible assertion that Philip II was a Spanish monarch whereas Charles himself was not. Charles grew up in Ghent, but spent much of his adult life and reign in Spain; his mother was Joana of Castile, daughter of Isabella I of Castille and Ferdinand II of Aragon. Charles inherited most of the Low Countries in 1506 as the heir of the house of Valois-Burgundy, and became the first monarch to unite the lands of Castile and Aragon in 1517 through inheritance from his mother’s line, thus being deemed the first monarch of a united Spain. Philip II was Charles’ son through Isabella of Portugal, daughter of Maria of Aragon. By heredity, Charles was half Spanish; by the criteria of the US Census Bureau, we would designate him as Hispanic; if we were to apply the Jewish standard of ethnicity through maternal lineage, we would designate Charles as Spanish. We are unable to ask Charles to describe his self-identified ethnicity, but we wonder if this statement widely attributed to him might not shed light on the matter: “I speak Spanish to God, Italian to women, French to men, and German to my horse,” although it is likely that his actual words were somewhat different.
We agree with Decoo that Belgium’s annexation to France (1795) occurred under the First Republic prior to Napoleon’s election as First Consul. Napoleon’s election in 1799 followed on the heels of the revolts against the French of 1798 which are often collectively referred to as the “Peasants’ War.” It is our understanding that unrest and discontent continued for some years after, and we regard the Decoo’s claim that Belgium “was already well integrated into France when Napoleon took over” in 1799 as an exaggeration.
Tobacco is widely listed in almanacs as a major crop of Belgium,  and is cited as a major crop in references dating back to at least the mid-nineteenth century. Although we are gladdened by Mr. Decoo’s statement that it is to disappear, that is not as yet the case. We are not able to comment on his claim that the “entry on ‘other commonly spoken languages’ is painfully inaccurate toward certain groups” as he provides no specifics.
We agree that many items could warrant a more nuanced and greatly expanded discussion, were there the luxury of space and time. We are aware that some errors have slipped through in the work of this magnitude, although far fewer than Decoo implies, in spite of a lengthy proofing process. However, of Decoo’s seven claims of error in our work which we can evaluate, only the one regarding Napoleon shows substantive error when examined. Decoo’s claim of numerous alleged errors on our part and then citing these as detracting from the credibility and rigor of our work is therefore highly misleading. We wonder why Decoo was so eager to discredit our work that he failed to adhere to scholarly principles in providing a careful and accurate critique, at times going so far as to put words in our mouth which do not reflect the text. We wonder how an individual wielding the torch of scholarly discipline could be so careless, especially when he has had the opportunity to cherry-pick examples which he feels demonstrate his claims. We also wonder why the editors of MSR did not take any discernable effort to check the factual claims of the reviewers, although much of this information is in Decoo’s own words “only a few clicks away.”
LIMITATIONS OF THE SCHOLARLY LITERATURE ON LDS CHURCH GROWTH
We also wish to address issues regarding the present scholarly literature on LDS Church Growth in journals such as Dialogue, Journal of Mormon History, and BYU Studies. Decoo expresses his viewpoint that “any serious approach” to Mormon membership development would refer to the works of a single scholar listed for each of six countries or regions, two for Japan, and three for Latin America and New Zealand, and that on the basis of our failure to engage these works, the “scholarly basis” of analysis of growth issues “is therefore extremely weak.”
We are obviously familiar with the larger scholarly corpus. David’s prior book The Law of the Harvest cites Decoo’s research ten times, as well as some authors mentioned by Decoo and numerous others not mentioned. We have made numerous additional references in online articles, and for years a church growth bibliography mentioning many of the authors cited by Decoo has resided on our site. Why then did we not more extensively engage the LDS scholarly literature? This is an important question which deserves exposition.
At the outset of the research for the current volume, the authors conducted a survey of the scholarly articles on LDS Church growth. This survey raised several concerns that let to our decision not to quote from this literature more extensively.
The first is the sparsity and inconsistency of the data. Although the 92 articles on international LDS history in the Journal of Mormon History sounds expansive, when one considers that our work covers over 200 country and regional profiles, there is not even one article for every two nations or territories. The scholarly references are uneven in both coverage and content. Some nations with a small LDS population, including Finland, are the source of multiple articles, while others with a large LDS population, including the Philippines, have very little treatment in the LDS scholarly literature. Some articles provide brief mention of multiple countries, but provide little data to interrogate, whereas other nations lack any substantive mention at all. This work cannot provide a reliable foundation for scholarship when we have a void for most countries and very little data on others.
The second is the lack of anything that would approach scientific study design, methodology, or controls is most articles published in in venues such as the Journal of Mormon History, Dialogue, or BYU Studies. The “peer-reviewed” literature on LDS church growth is found mostly in journals dedicated to a range of eclectic topics touching Mormonism, alongside poems, anecdotes, reflections, book reviews, historical notes, and widely-ranging essays. Most works consist of the exposition of data known to the author, woven together with anecdotes, opinions, and citation of other literature. Personal anecdotes and small-scale observation are frequently extrapolated to be representative of a larger picture, which may or may not be the case. We are unable to validate such claims without scientific study design, clear statements of methodology, and adequate sample size. The editors of these journals are rarely individuals with any specialized training that would qualify them to closely scrutinize research work on these topics, and tend to rely almost entirely on a review process which is outsourced to outside “experts.”
For an example of the nature of data presented in the published LDS literature, we view as Gooren’s “Analyzing LDS Growth in Guatemala: a Report from a Barrio“ as one of the better scholarly pieces on LDS growth in a particular country. Unlike most other authors, Gooren has a formal methodology consisting of surveys of two wards from his PhD study as well as participation-observation and informal interviews a ward in San Jose, Costa Rica, that constitutes a part of his master’s study. The survey of two wards had a response rate of 25%, which Gooren acknowledges to be “much too small to be more than barely suggestive of certain recurrent traits in these wards,” to say nothing of the larger situation in Guatemala. This research design may provide potentially more relevance, but we are still left with anecdotes, personal observations, and statements from interviews with locals as the foundation. We wonder if many of Gooren’s observations and notes may reflect larger trends in LDS growth in Guatemala, but as Gooren himself acknowledges, his methodology does not allow robust conclusions, but is “barely suggestive” of trends even within the wards studied. Nonetheless, Gooren’s piece stands out as a bright light in the field of peer-reviewed articles on LDS Church growth, as it includes both master and PhD level research. With such works ranking among the best, the published data in LDS scholarly journals to date does not provide a foundation from which to draw strong conclusions that we could cite with confidence.
Robust statistical data from the published LDS literature on church growth is thus sparse. Many of the references (including Decoo’s own work) couch statistical estimates of LDS member activity and convert retention in phrases such as “estimated” and “about,” typically without further documentation beyond the author’s own authority, or the citation of other authors using similar language. No amount of citation or peer review will transform such data into more than “estimated” and “about,” and when we sort through all of the layers, we typically have no more definitive authority for the data than the original author’s assertion.
Many of the published articles on LDS Church growth are focused on specific periods and events, such as Kim Östman’s article “The Mormon Espionage Scare and Its Coverage in Finland, 82-84” and Knowlton’s “Mormonism and Guerillas in Bolivia.” These essays provide fascinating analysis, yet the authors themselves demure from providing specific estimates regarding the impact of these events, and make little attempt to separate the direct impact of the topics being addressed – the negative press and sectarian agitation against the LDS Church in Finland during the specified period, and the impact of a Bolivian guerilla group that for a time specifically targeted the LDS Church, respectively – from a variety concurrent issues, including secularism, perceptions of the LDS Church as a foreign faith, the impact of mission policies which have changed over time, and more. From these articles, we can conclude that the events described likely had some impact on church growth in the specified periods, but the authors themselves give us little insight regarding how much impact. Further, the impact of these past issues for future church growth is unclear. Did these events result from one-time circumstances which created an environment which impacted church growth for a period, or do they still have an ongoing impact on the situation of the church in the respective countries today? What practical lessons can be drawn and what is to be done in similar situations going forward, regarding anti-LDS agitation, negative press, or acts of terror? The authors do not provide us with convincing statements on these matters. Many of the findings and conclusions of these authors warrant additional critique, but this is beyond the scope of this letter. These and many similar articles provide valuable historical scholarship regarding specific events and periods, and we applaud the authors for their contributions and the journals that have published their work. However, these and other articles lack evidence of adequate strength from which to draw robust analytical conclusions.
Exceptions might include large-scale practices like the “Baseball Baptism” period in the United Kingdom, the “Australian Pentecost,” and a similar period of rushed baptisms in Japan. We chose to tread lightly regarding such events, which can be prominently found in the LDS specialty literature for those desiring detailed histories. We have however noted key lessons from such events in general terms, observing the challenges posed by past (and in some cases, ongoing) policies and practices of rushed baptisms, which our sources confirm have extended far beyond the nations and areas where these practices have been identified in the antecedent scholarly literature, as well as their consequences of low convert retention and member activity rates.
No amount of scholarly analysis can transform evidence of one type into another. For instance, Decoo cites many charming anecdotes in his peer-reviewed publications. One of David’s favorites is Decoo’s story in which a visiting general authority denied permission to translate church literature outside of the official curriculum into local languages and stated that the scriptures should be enough for any of the saints even as his wife and daughter read books by Hugh Nibley and Jack Weyland in the foyer. Yet no matter how many times this story is printed or cited, no matter what scholarly process it is subjected to, it is not transformed into anything more than an anecdote, nor does it become more valid or authoritative than the anecdotes and experiences of millions of Mormon around the world. Terms such as “academic rigor” and “scholarship,” or an author’s own judgment of the strength of conclusions drawn from evidence of this nature, mean nothing in this context.
The tendency to impute a higher level of confidence to available scholarship than is supported by evidence in the absence of more robust data is not confined to Mr. Decoo. It seems to be a matter of great difficulty for authorities to acknowledge that evidence for a prevailing view is weak, or that data of adequate quality to provide more robust answers is lacking. It is our view that the scholarship on LDS Church growth done to date, while significant, would benefit from a large dose of humility.
BIAS AND POTENTIAL CONFLICTS OF INTEREST
We are concerned by bias and potential conflicts of interest when a scholarly community is very small. The small size of the LDS scholarly community relative to the number of areas of research makes arms-length relationships difficult, and blinding of reviewers as well as other steps typically considered essential to the academic integrity of the peer review process are not possible. In this same journal and its predecessor, we have often perceived fawning when a scholar reviews the work of a close colleague or the work of authors who present viewpoints that he substantially favors while overlooking substantial limitations that seem glaring to those at arms’ length, while an entirely different standard which takes criticism to the extreme of being unscholarly when evaluating the work of writers who do not belong to the "club." We see this same tendency in Mr. Decoo, and believe that he overestimates the rigor of the work of his favored authors while underappreciating our work. Scholarly rigor must be consistent or it ceases to be scholarly altogether.
We are acutely aware of the problem of inbreeding in LDS scholarship, as well as the fallacy self-reinforcement that follows. As Decoo has acknowledged, the LDS scholarly community is dominated by a relatively small number of scholars, with many countries having only a single prominent specialist on church growth. Many have none at all, and some are located in an intermediate range where we have a few casual comments, but no systematic study. We are concerned that the findings and opinions of some authors are often cited and perpetuated as if they were authoritative, when frequently the data has not been adequately scrutinized and interrogated, and the quality of evidence upon which conclusions are formulated lacks the level of rigor which its proponents wish to attribute to it. Substantial agreement with or engagement of one or two leading authorities is sometimes presented as a criterion for admission to the scholarly club, as Decoo has suggested in as many words: others may be deemed credible academicians only to the extent that they support or build upon the views of a field's dominant authority. We reject this notion as circular logic and a breach of the scholarly principles of independence, objectivity, and impartiality. This is precisely the fallacy that we wished to avoid and the reason we have insisted on an independent approach to our research and analysis on LDS Church growth. To the extent that the conclusions of these authors are valid, they ought to be able to stand on their own, and to be confirmed by research that is truly independent.
The lack of depth of the LDS scholarly “bench,” if we can term it such, is often painfully obvious to scholars who contribute to broader academic fields and are accustomed to assessing the merits of competing viewpoints advocated by proponents who are acknowledged as experts in their field and who can cite significant evidence in support of their views, but who still arrive at widely different conclusions. The result is lively scholarly exchange and impetus for additional study. We were concerned about being too highly influenced by existing works, and consequently chose to take a fresh look.
THE NEED FOR SYSTEMATIC RESEARCH
We desired an approach to LDS growth which was truly systematic and, as much as possible, minimized bias. An a la carte approach to LDS Church growth literature characterizes most of the corpus of church growth literature referred to by Decoo. Review which is not systematic risks overemphasizing certain developments at the expense of others, and missing other developments altogether. A common error in diagnostic radiology is satisfaction of search, in which a radiologist identified an initial, often obvious, abnormality and, satisfied with the search, ceases to closely scrutinize the study and misses additional pathology. We are concerned about similar errors in scholarship which entertains too high an opinion of the work that has already been done, and as a result fails to employ a systematic approach in its work and fails to evenly consider other possibilities.
The only valid study design that is capable of definitively answering important questions related to church growth is a study with systematic methodology meeting criteria of research design and providing a sufficient level and quality of evidence. The existing LDS church growth literature cited by Decoo cannot definitively answer these questions. This literature may present valuable historical insights, or offer suggestions and ideas for further study, but no matter how many times it is peer-reviewed, quoted, cited, or declared by strongly-worded opinion of certain scholars to be definitive, it remains limited by the level of evidence of the original research.
Questions involving church growth can be answered by an adequately robust research design. We were surprised that none of the reviewers mentioned the need for systematic study of church growth with a clear methodology. The leap from essays and storytelling to systematic research with study methodology, or from small-scale studies with inconsistent methodology and low response rates unevenly evaluating sporadically selected regions, to a systematic methodology applied to countries around the world, is substantial, and should be understood as such.
LEVELS OF EVIDENCE
Fortunately, we need not depend on the subjective appraisals of ostensible experts to assess the merit of scholarly literature. “Levels of Evidence” (LOE) is a system for ranking the robustness and findings of scientific research based on the study design, methodological rigor, and implementation. Levels of Evidence are printed along with original articles in the most prestigious journals,  and provide the reader with a view of the relative strength of the author’s research methodology and the potential robustness (or lack thereof) of the findings. Levels range from level 1 (the highest quality studies) to level 4 (low-quality studies) for research design, and for papers without scientific methodology, level 5 (expert opinion). This system is most widely used in medicine. However, the principles of Levels of Evidence can be applied to the analysis of a wide variety of kinds of research involving human populations, including the study of church growth.
Levels of Evidence provide an objective rating for the robustness of scholarly work, and can objectively determine which findings or conclusions have strong support, weak support, inconclusive evidence, or are refuted. The implementation of Levels of Evidence as a research standard has led to improved research design, more robust findings, improved professional training, and more consistent standards of patient care. Observational studies and cohort studies with historical controls can also be rated by levels of evidence. Additional statistical tools can assist in further evaluating this data. This system can be updated or modified to assist with answering specific research questions. Other types of data, including survey data, can be evaluated with statistical tools which can provide objective calculations of the robustness and potential error of survey data. Researchers are expected to discuss any limitations of methodology which they identify, which is subject to further critique by others. Any potential conflicts of interest by the authors must be disclosed, and potential sources of bias are evaluated.
We appreciate the clarity brought by Levels of Evidence (LOE). Application of Levels of Evidence give an objective estimation of the strength of conclusions, so that we are not left to the mercy of competing scholars’ conflicting opinions and all of the sources of bias that they entail. LOE brings clarity to the evaluation of data, and focuses attention on conducting research of higher quality and assists in identifying areas where more or better data are needed. The foundation of scholarship shifts from the pronouncements of experts or authorities in a field to the robustness of the data and methodological quality of the research, which we view as a great advance of the scholarly process. LOE deflates the rancorous debate of erstwhile scholars attempting to elevate their conclusions to a higher status than warranted by the quality of their work through strongly-worded statements of opinion, and makes attempts to gain currency for one’s conclusions through conflicts of interest, political means, or other non-scholarly mechanisms less viable.
It is often surprising to many not trained in research methodology to learn how many conclusions and widely-held beliefs that are conveyed by authorities as robust conclusions actually have weak or no support; similarly, some views that have been dismissed out of hand may require re-evaluation. In David’s field, many prior beliefs that were almost universally accepted have been thrown out after re-examination by the system of Levels of Evidence, whereas other findings which were not well-known have gained widespread acceptance.
Having reviewed works of the key authors referred to by Decoo before our writing, the level of evidence presented by most of these papers is the lowest level, level 5: expert opinion. Not all works of “expert opinion” are of similar weight. Some are better supported or well-reasoned than others, but work of this nature can never become more than the limitations of the author’s methodology allow. A very few studies can be qualified as level 4, and those only with considerable caveats due to low sample size, low response rates, non-representative sample, or other methodological flaws. None of these articles of which we are aware to date evaluate the robustness of the authors’ original research data by the application of standard statistical tools which are intended for that purpose.
METHODOLOGIES AND LIMITATIONS
Matthew described our methodology in his Miller-Eccles Lecture, available as a podcast at Dialogue, including a large cross-sectional survey design supplemented by numerous interviews with local members, current and returned missionaries, and present and past mission presidents. Our research included descriptive work consisting of cross-sectional surveys and qualitative research, and analytical research consisting of cross-sectional research and case studies, as well as a detailed literature review. This research has taken David to over 40 countries, and Matthew continues to be involved in daily information gathering. Additional information on methodology is provided in any of Matthew’s online church growth case studies.
To the extent that elements of our methodology are unclear to Mr. Decoo, further methodological information should be requested or, at best, it should be noted that that inadequate evidence is presented to discuss the scholarly basis for the findings, rather than rushing to judge that the data and conclusions cannot be supported. We believe that some of our differences with Mr. Decoo represent an interdisciplinary difference, much as the differences between linguists and anthropologists, or between sociologists and economists, can lead to each side claiming that the other's methodology is weak and conclusions lack credibility in key areas. We believe that this topic is best informed by multidisciplinary research involving a variety of approaches and perspectives.
The fact that numerous reviewers have collaborated that our statistical data is substantially correct, in spite of the lack of availability of such data in previously published sources, attests to the value of our research model in arriving at data that has not been available to many or most of the academic authors who Decoo views as setting the standard. The criticisms on this matter appear to center on some statistics being a couple of years out of date, which is to be expected in view of the time required to prepare and proof a work of this magnitude, but are also acknowledged to be substantially more current than currently available printed resources on most if not all of the items addressed by our reviewers. Regular updates include a number items touched on by reviewers are provided on Matthew’s church growth blog. The utility of our conclusions and the degree to which they support findings of other authors to date, as well as the additional insights and directions they offer, cannot be dismissed with hand-waving.
With a closer look, Decoo would have recognized areas where our conclusions closely coincide with those of the authors he commends. For example, our findings tend to strongly corroborate support a matter Decoo has written much about, as well as other authors including Jiro Numano and Marjorie Newton including the poor retention rates that are closely associated with missionary approaches that emphasize quick baptism, and the need for better preparation for baptism of prospective converts. Thoughtful readers can identify many other areas where our data lead to substantially similar conclusions to those of other authors of articles on church growth, as well as some differences, although a full exposition is beyond the scope of this letter.
Will all of our findings withstand the test of time? No. Our research methodology presents primarily level 4 and level 5 evidence, which although in many ways more robust than much published data in the current literature on church growth, as well as broader sample size involving more areas and regions. We have attempted a systematic approach to the study of church growth, although our research in some regions achieved a higher response rate than in others. We would like to see future work at higher levels of evidence and with greater robustness. We hope that other authors will surpass the bar that our work has set.
In our future work, we expect to implement the usage of statistical tools for the interrogation of survey and research data, which have been conspicuously absent from published independent research on LDS Church growth to date. We would also hope to see more robust research models, detailed disclosure of methodology, assignment of Levels of Evidence, discussion of research limitations, and the implementation of statistical tools for the analysis of research data in future articles on church growth published by other authors.
As one reviewer pointed out, research of this nature is very difficult to conduct without the formal cooperation of the LDS Church, which is unlikely for a number of reasons. Even if possible, such cooperation could introduce additional sources of bias, and we are well aware that authors of officially sanctioned church publications have not been free to address a number of issues we have covered that may paint other than a rosy public relations view of dynamic church growth with high member activity, or that may suggest shortcomings of “inspired” policies and practices. Real-world practicalities, including both hard constraints and soft constraints, lead us to strive for a constrained optimum which represents the best reasonably attainable solution in view of external limitations, internal constraints, and available resources. As constraints, resources, and opportunities can change over time, we expect that there will be opportunities for continued improvement. We appreciate the acknowledgement of some reviewers that in view of these and other limitations, our work represents a significant achievement. The scope and methodology of our work represent a broad leap forward in both scope and methodology from any research published to date in the scholarly literature on LDS Church growth.
Some critiques by various reviewers result from points where we have followed reference works with resources and budgets vastly in excess of ours. We are puzzled by the recommendations that we should not rely on the existing body of publically sourced literature typical of almanacs as a source of background information on the countries in our book, but are apparently to engage an army of researchers not merely to address the church growth related topics that are the central focus of our work, to compose each country background section as if written by a history professor for an academic journal, and to catch obscure errors in historical sources including the LDS Church News. However, upon reviewing Decoo’s critiques and finding what appear to us to represent a number of demonstrable errors in his factual claims, and others points where there appear to be room for real discussion and different views, it is not clear to us that this latter approach would have offered improvement.
We address numerous issues and challenges regarding church growth which are rarely if ever mentioned in official church publications, and have repeatedly cited organizational and policy challenges of the Church itself as impacting growth, often unfavorably. We were concerned that our work would represent too sharp a departure from the view presented in publications such as the LDS Church News and LDS Church Almanac to be palatable to much of our intended audience. We thus accept with some pride Decoo’s criticism that we have not been sufficiently critical of the church organization itself in evaluating factors hindering growth, as evidence that we have presented our findings in a constructive and often understated fashion, and have not been excessively critical or pessimistic as we had feared. David has addressed some internal issues of the church itself in more detail in a prior work. As other reviewers have acknowledged, our work takes a critical look at statistics and challenges impacting growth which depart from the sanitized histories and idealized accounts often presented in official church publications, even if much of the historical overview we have presented is informed by church publications rather than by critical literature. Further, the ability of local members, missionaries, and regional leaders to address what Decoo sees as internal limitations of the Church itself is nil. As our title and preface suggest, our work is directed towards practical issues related to church growth which we believe can offer insight and benefit to local members, missionaries, and leaders. While insights into local culture and evaluation of issues, challenges, and opportunities are matters of practical relevance, a critique of hindrances intrinsic to the church organization would have no discernable practical value for our readers, and could even be counterproductive. It is further not our role or purpose to critique church doctrine. We have chosen to leave such topics, we believe appropriately, for others to discuss in the other scholarly literature.
We take it as a compliment that some of our reviewers seemed to have nothing better than to suggest, for example, that we should have written Māori rather than Maori - a convention that, we are convinced, the majority of our readers are unfamiliar with, which we have rarely seen in informational works for a lay audience, and would have required additional explanation. We also appreciate the correction of some spelling items, such as Tshiluba instead of Tsihiluba, and Kasumbalesa instead of Kasambalesa.
Although we would have appreciated the opportunity to have Reaching the Nations reviewed in Mormon Studies Review had this process been conducted according to scholarly norms and standards, we are deeply disappointed by the numerous errors in your review and in MSR’s lack of academic integrity in refusing to correct gross errors even after they have been brought to your attention.
We hope that our comments will clarify a number of inaccuracies in the review comments, as well as providing additional perspective and clarity on the study of church growth. We hope that the coming years will see increased impetus for the implementation and publication of further systematic research on LDS Church growth that will present continuing advances in study design, methodology, and research quality.
 Miasnik, Chris. “The Church in Belgium: Membership has international flavor in Church version of United Nations,” LDS Church News, 7 December 1996. http://www.ldschurchnews.com/articles/28444/6000-members-living-in-Belgium-have-rich-history.html
 Miasnik, Chris. “The Church in Belgium: Membership has international flavor in Church version of United Nations,” LDS Church News, 7 December 1996. http://www.ldschurchnewsarchive.com/articles/28181/The-Church-in-Belgium--Membership-has-international-flavor-in-Church-version-of-United-Nations.html
 “Middle Rhine Highlands,” Encyclopedia Brittanica, http://www.britannica.com/place/Middle-Rhine-Highlands, accessed 7 January 2016, “Ardennes,” http://en.wikipedia.org/wiki/Ardennes. Accessed 4 December 2015
 “Tobacco Consumption Statistics,” Eurostat, http://ec.europa.eu/eurostat/statistics-explained/index.php/Tobacco_consumption_statistics. Accessed 2 December 2015
 Eriksen, M., et al. The Tobacco Atlas, 5th edition. American Cancer Society, 2015. http://3pk43x313ggr4cy0lh3tctjh.wpengine.netdna-cdn.com/wp-content/uploads/2015/03/TA5_2015_WEB.pdf. An interactive summary version is available at http://www.tobaccoatlas.org/topic/cigarette-use-globally , citing 2014 per capita cigarette consumption in individuals over 15 in Belgium at 2353. Accessed 6 December 2015.
 “Charles V, Holy Roman Emperor,” Encyclopedia Brittanica, http://www.britannica.com/biography/Charles-V-Holy-Roman-emperor, accessed 7 January 2016; also https://en.wikipedia.org/wiki/Charles_V,_Holy_Roman_Emperor
 Girolamo Fabrizi d'Acquapendente's 1601 work De Locutione states: “Unde solebat, ut audio, Carolus V Imperator dicere, Germanorum linguam esse militarem: Hispanorum amatoriam: Italorum oratoriam: Gallorum nobilem.” ("When Emperor Charles V used to say, as I hear, that the language of the Germans was military; that of the Spaniards pertained to love; that of the Italians was oratorical; that of the French was noble").
 Coleman, Henry. The Agriculture and Rural Economy of France, Belgium, Holland and Switzerland, London: John Petheram, 1848, p. 254-56.
 Stewart, David. The Law of the Harvest: Practical Principles of Effective Missionary Work. Cumorah Foundation, Henderson, NV, 2007. 461 p.
 An older bibliographic list of some key works, not fully redacted due to phasing out and revising the accompanying text, has resided on our website for years at http://www.cumorah.com/index.php?target=view_other_articles&story_id=29&cat_id=5. We maintain a more current list for personal use which is not posted online.
 A brief explanation of different potential study designs is found on the Center for Evidence-based Medicine website: http://www.cebm.net/study-designs . Most sociological studies are descriptive studies (cross sectional/surveys and qualitative studies) or observational analytic studies (cohort, cross sectional analytic, or occasionally case control). Missiologic studies can have experimental design.
 Gooren, Henri. “Analyzing LDS Growth in Guatemala: a Report from a Barrio.” Dialogue: A Journal of Mormon Thought 43 (Summer 2000): 97-115.
 Gooren, Henri. “Analyzing LDS Growth in Guatemala: a Report from a Barrio.” Dialogue: A Journal of Mormon Thought 43 (Summer 2000): 99.
 Decoo, Wilfried, “Issues in Writing European History and in Building the Church in Europe,” Journal of Mormon History, Spring 1997: 164.
 Östman, Kim B. “The Mormon Espionage Scare and Its Coverage in Finland, 1982—84.”Journal of Mormon History Vol. 34, No. 1 (Winter 2008), 82-117.
 Knowlton, David C. “Mormonism and Guerrillas in Bolivia.” Journal of Mormon History Vol. 32, No. 3 (Fall 2006), pp. 180-208.
 Quinn, D. Michael. "I-Thou Vs. I-It Conversions: The Mormon 'Baseball Baptism' Era." Sunstone 16:7/30 (Dec 93):32.
 Newton, Marjorie, "Towards 2000: Mormonism in Australia," Dialogue, 29/1 (Spring 1996): 193-206.
 Numano, Jiro. "Perseverance amid Paradox: The Struggle of the LDS Church in Japan Today." Dialogue 39:4 (Winter 2006): 138–155.
 Decoo, Wilfried, “Feeding the Fleeing Flock: Reflections on the Struggle to Retain Church Members in Europe,” Dialogue, 29/1 (Spring 1996): 118.
 Kim YW, Mansfield LT. Fool me twice: delayed diagnoses in radiology with emphasis on perpetuated errors. AJR Am J Roentgenol. 2014;202 (3): 465-70.
 One modification is found here: “Introducing Levels of Evidence to the Journal,” Journal of Bone and Joint Surgery 85A, No. 1 (January 2003):1-3. http://jbjs.org/content/jbjsam/85/1/1.full.pdf
 One example of this is the American Academy of Orthopedic Surgeons Clinical Practice Guidelines at http://www.aaos.org/Quality/Clinical_Practice_Guidelines/AAOS_Clinical_Practice_Guidelines/?ssopc=1
 Marx, RG, Wilson SM, Swiontkowski MF. “Updating the Assignment of Levels of Evidence.” Journal of Bone and Joint Surgery 97, No. 1 (January 2015): 1-2. http://jbjs.org/content/97/1/1
 Decoo, Wilfried, “Feeding the Fleeing Flock: Reflections on the Struggle to Retain Church Members in Europe,” Dialogue, 29/1 (Spring 1996): 118.
 Numano, Jiro. “Hasty Baptisms in Japan: The Early 1980s in the LDS Church.” Journal of Mormon History Vol. 36, No. 4 (Fall 2010), pp. 18-40.
 Newton, Marjorie, “Towards 2000: Mormonism in Australia,” Dialogue 29/1 (Spring 1996): 193–206.
 Stewart, David. The Law of the Harvest: Practical Principles of Effective Missionary Work. Cumorah Foundation, Henderson, NV, 2007. 460 p.