TY - JOUR

T1 - A serious flaw in nutrition epidemiology

T2 - A meta-analysis study

AU - Peace, Karl E.

AU - Yin, Jingjing

AU - Rochani, Haresh

AU - Pandeya, Sarbesh

AU - Young, Stanley

N1 - Publisher Copyright:
© 2018 Walter de Gruyter GmbH, Berlin/Boston.

PY - 2018

Y1 - 2018

N2 - Background: Many researchers have studied the relationship between diet and health. Specifically, there are papers showing an association between the consumption of sugar sweetened beverages and Type 2 diabetes. Many meta-analyses use individual studies that do not attempt to adjust for multiple testing or multiple modeling. Hence the claims reported in a meta-analysis paper may be unreliable as the base papers do not ensure unbiased statistics. Objective: Determine (i) the statistical reliability of 10 papers and (ii) indirectly the reliability of the metaanalysis study. Method: We obtained copies of each of the 10 papers used in a metaanalysis paper and counted the numbers of outcomes, predictors, and covariates. We estimate the size of the potential analysis search space available to the authors of these papers; i. e. the number of comparisons and models available. The potential analysis search space is the number of outcomes times the number of predictors times 2c, where c is the number of covariates. This formula was applied to information found in the abstracts (Space A) as well as the text (Space T) of each base paper. Results: The median and range of the number of comparisons possible across the base papers are 6.5 and (2 12,288), respectively for Space A, and 196,608 and (3072-117,117,952), respectively for Space T. It is noted that the median of 6.5 for Space A may be misleading as each study has 60-165 foods that could be predictors. Conclusion: Given that testing is at the 5% level and the number of comparisons is very large, nominal statistical significance is very weak support for a claim. The claims in these papers are not statistically supported and hence are unreliable so the meta-analysis paper is also unreliable.

AB - Background: Many researchers have studied the relationship between diet and health. Specifically, there are papers showing an association between the consumption of sugar sweetened beverages and Type 2 diabetes. Many meta-analyses use individual studies that do not attempt to adjust for multiple testing or multiple modeling. Hence the claims reported in a meta-analysis paper may be unreliable as the base papers do not ensure unbiased statistics. Objective: Determine (i) the statistical reliability of 10 papers and (ii) indirectly the reliability of the metaanalysis study. Method: We obtained copies of each of the 10 papers used in a metaanalysis paper and counted the numbers of outcomes, predictors, and covariates. We estimate the size of the potential analysis search space available to the authors of these papers; i. e. the number of comparisons and models available. The potential analysis search space is the number of outcomes times the number of predictors times 2c, where c is the number of covariates. This formula was applied to information found in the abstracts (Space A) as well as the text (Space T) of each base paper. Results: The median and range of the number of comparisons possible across the base papers are 6.5 and (2 12,288), respectively for Space A, and 196,608 and (3072-117,117,952), respectively for Space T. It is noted that the median of 6.5 for Space A may be misleading as each study has 60-165 foods that could be predictors. Conclusion: Given that testing is at the 5% level and the number of comparisons is very large, nominal statistical significance is very weak support for a claim. The claims in these papers are not statistically supported and hence are unreliable so the meta-analysis paper is also unreliable.

KW - Meta-analysis

KW - Multiple modeling

KW - Multiple testing

KW - Nutritional epidemiology

KW - Observational studies

KW - Reliability of claims

UR - http://www.scopus.com/inward/record.url?scp=85057570835&partnerID=8YFLogxK

U2 - 10.1515/ijb-2018-0079

DO - 10.1515/ijb-2018-0079

M3 - Systematic review

SN - 1557-4679

VL - 14

JO - International Journal of Biostatistics

JF - International Journal of Biostatistics

IS - 2

M1 - 20180079

ER -