CSPI Allegations and Responses

[Originally published in late 2015; Updated 11/28/2016 with responses to Dr. Hefland’s comments, as noted]

Summary: There are 11 allegations by CSPI et al. in total. Four of these refer to questions about the systematic methodology;  these will be addressed together, at the end of this statement. The other seven questions are addressed first:

1. Teicholz states that “in its 2015 report the committee stated that it did not use NEL reviews for more than 70% of the topics, including some of the most controversial issues in nutrition. Instead, it relied on systematic reviews by external professional associations, almost exclusively the American Heart Association (AHA) and the American College of Cardiology (ACC), or conducted an [sic] hoc examination of the scientific literature without well defined systematic criteria for how studies or outside review papers were identified, selected, or evaluated.”

Correction: In Appendix E‐2, the Evidence Portfolios for the key topics addressed by Teicholz specify the search strategy, inclusion criteria, search results, and AMSTAR ratings for methodological quality for the existing systematic reviews (SR) and meta‐analyses (MA).3,4

Response: See “response to methodological questions” at end of this document.

2. Teicholz states that “instead of requesting a new NEL review for the recent literature on this crucial topic, however, the 2015 committee recommended extending the current cap on saturated fats, at 10% of calories, based on a review by the AHA and ACC, a 2010 NEL review, and the committee’s ad hoc selection of seven review papers (see table A on thebmj.com).” Table A states that “no methodology for this section of the report: no reason given for why certain studies were selected for review and others were not, nor how they were evaluated relative to each other.”

Correction: Appendix E‐2.43 gives the search strategy, inclusion criteria, search results, and AMSTAR ratings for methodological quality for the seven review papers, along with a list of excluded articles and the reasons for exclusion.5 (Note: In Table A, Teicholz states that “the overall conclusion [of the 2012 Cochrane review by Hooper, et al.] is therefore that while saturated‐fat restriction appears to reduce heart attack risk, it does not reduce overall or cardiovascular mortality (death), which is arguably the more important endpoint.”6 This statement contradicts Teicholz’s article, which said that Hooper, et al. “failed to confirm an association between saturated fats and heart disease.” The BMJ corrected this error a month after it was published.)

Response: This criticism refers to a statement in a supporting table, not the document itself. See “response to methodological questions” at end of this document.

3. Teicholz states that “use of external reviews by professional associations is problematic because these groups conduct literature reviews according to different standards and are supported by food and drug companies.”

Correction: The “problematic” external review cited by Teicholz was not conducted solely by professional associations. The review was actually a “clinical practice guideline” developed by the American Heart Association and the American College of Cardiology in partnership with the National Heart, Lung, and Blood Institute.7 The NHLBI’s website clearly describes its rigorous standards for assessing the quality of studies and its policy for managing potential conflicts of interest and relationships with industry.

Response: This issue has already been covered in my “Rapid Response.” That response, which was peer reviewed, concluded that no clarification or correction was needed. The BMJ article does not assert that these reviews were conducted “solely” by professional associations. Moreover, the AHA/ACC were clearly involved in the review of the scientific literature in these papers, as can be seen in the AHA/ACC/TOS obesity guidelines, which states: “This document was also reviewed by 6 expert reviewers nominated by the ACC, AHA, and The Obesity Society, when the management of the guideline transitioned to the ACC/AHA. The ACC, AHA, and The Obesity Society Reviewers’” Update: Dr. Hefland and The BMJ agreed that this point was not in error.

4,  Teicholz states that “in the NEL systematic review on saturated fats from 2010…fewer than 12 small trials are cited, and none supports the hypothesis that saturated fats cause heart disease (see table B on thebmj.com).”

Correction: It is incorrect to state that none of the trials cited in the 2010 NEL review supports the hypothesis that saturated fats cause heart disease. The 2010 NEL review found “strong evidence” that saturated fat intake increases the risk of cardiovascular disease. In Table B, Teicholz over‐rules the 2010 NEL review by assigning each trial to one of four categories (a) “trials that should not have been included because they did not meet inclusion criteria,” (b) “trials that should not have been included because they did not test normally occuring [sic] saturated fats or saturated fats at all,” (c) “trials concluding that saturated fats had a neutrial [sic] or beneficial effect on health,” and (d) “trials with mixed results on blood lipid measures.”8 (Note: Table B has additional errors too numerous to list here.) Thus, Teicholz concludes that the 2010 NEL review is substandard, but she also argues that the 2015 “committee’s report used weak scientific standards,” because it did not rely sufficiently on NEL reviews.

Response: No correction of fact is cited here. This complaint merely repeats the article’s original statements. Update: Dr. Hefland and The BMJ agreed that this point was not in error.

5. Teicholz states that “perhaps more important are the studies that have never been systematically reviewed by any of the dietary guideline committees. These include the large, government funded randomized controlled trials on saturated fats and heart disease from the 1960s and ’70s. Taken together, these trials followed more than 25 000 people, some for up to 12 years. They are some of the most ambitious, well controlled nutrition studies ever undertaken.”

Correction: It is incorrect to state that these trials were not reviewed by the DGAC. The DGAC considered a 2012 Cochrane review that included 4 of the 6 trials cited by Teicholz and a 2010 meta‐analysis that included 5 of the 6 trials cited by Teicholz.910 (The review and meta‐analysis both concluded that replacing saturated fats with unsaturated fats reduce the risk of heart disease.) One trial cited by Teicholz is excluded from most meta‐ analyses because it tested a multifactorial intervention including drug

Response: My statement is meant to imply that no committee has directly reviewed these trials. This is a useful clarification. [Update: Hefland judged my statement to be “ambiguous” and then reasoned that it would only be “natural” to clarify it in a way that renders it an error. The BMJ sided with my argument, namely that if a statement is ambiguous, the best person to clarify it is the author who knows the original intent. Thus, I have clarified this statement.]

6. Teicholz states that “there have been at a minimum, three National Institutes of Health funded trials on some 50 000 people showing that a diet low in fat and saturated fat is ineffective for fighting heart disease, obesity, diabetes, or cancer. Two of these trials are omitted from the NEL review….When the omitted findings from these three clinical trials are factored into the review, the overwhelming preponderance of rigorous evidence does not support any of the dietary committee’s health claims for its recommended diets.”

Correction: The two trials that were “omitted from the NEL review” did not assess the impact of diet “for fighting heart disease, obesity, diabetes, or cancer.” They assessed the impact of diet on serum cholesterol levels.12,13 Furthermore, all three trials were included in the Cochrane review that was considered by the DGAC.14)

Response: All of the trials published outcomes on obesity and heart disease. Two of the three trials published outcomes on diabetes. One of the trials published outcomes on cancer. Thus, a clarification might read: these trials, taken together, show that a diet low in fat and saturated fat is ineffective for fighting heart disease, obesity, diabetes, and cancer.  Regarding the other complaint: it is has been standard in the field of heart disease research for 50+ years to use serum cholesterol measures as intermediary outcome markers for heart disease.
[Update:Hefland stated that “cholesterol is a risk factor for heart disease, not a marker for it,” but this statement is not supported by a vast quantity of heart disease research that, for decades, has used lipid markers of various kinds as intermediary outcome markers for heart disease; The BMJ sided with my arguments on this and other points made by Hefland.]

7. Teicholz states that “The report also gave a strong rating to the evidence that its recommended diets can fight heart disease….The committee reviewed other, more recent studies but not using any systematic or predefined methods.”

Correction: Appendix E‐2.26 gives the search strategy, inclusion criteria, search results, and AMSTAR ratings for methodological quality for the six “more recent studies,” along with a list of excluded articles and the reasons for exclusion.15

Response: See “response to methodological questions” at end of this document.

8. In Table D, Teicholz includes sections (under “dietary patterns and heart disease” and “dietary patterns and obesity”) entitled “DGAC ad hoc review of the scientific literature” where she states that “no systematic methodology is given for the selection of these studies. It is therefore impossible to know if they fairly represent the literature.”

Correction: These were not ad hoc reviews. The DGAC details the systematic methodology for selecting these studies in Appendices E‐2.26 and E2.27.16 Note: Teicholz’s Table D17 consists largely of Teicholz’s criticism of the NEL’s Systematic Reviews on the Relationship between Dietary Patterns and Health Outcomes, published in 2014.18 Note that Teicholz argues that NEL reviews are substandard, but she also argues that the 2015 “committee’s report used weak scientific standards,” because it did not rely sufficiently on NEL reviews.

Response: This criticism refers to a statement in a supporting table, not the document itself. See “response to methodological questions” at end of this document.

9. Teicholz states that “Consulting the NEL for a review on this topic turns up a surprising fact: a systematic review on health and red meat has not been done. Although several analyses look at ‘animal protein products,’ these reviews include eggs, fish, and dairy and therefore do not isolate the health effects of red meat, or meat of any kind.

Correction: The NEL reviews cited by Teicholz do examine the results on red meat and processed meats separately from the results on other animal proteins.[i]19,20,21,22,23,24,25

Response: The original statement in the BMJ article is correct. These are all reviews of “animal protein products,” as stated, which includes eggs, fish, and dairy, along with meat. The effect of red meat is not separated out for specific analysis. This can be confirmed by looking at the reviews: [i]19,20,21,22,23,24,25 [Update: Hefland asserted that because “red meat” and “meat” were terms included in the search string for these reviews, these should be considered systematic reviews of such. However, The BMJ sided with my arguments that this point is debatable and that many experts would agree with my view that these cannot be rightly called systematic reviews of red meat or meat unless a specific analysis of red meat or meat has been conducted.]

10.Teicholz states that “The committee’s approach to the evidence on saturated fats and low carbohydrate diets reflects an apparent failure to address any evidence that contradicts what has been official nutritional advice for the past 35 years. The foundation of that advice has been to recommend eating less fat and fewer animal products (meat, dairy, eggs) while shifting calorie intake towards more plant foods (fruits, vegetables, grains, and vegetable oils) for good health. And in the past decades, this advice has remained virtually unchanged.”

Correction: The 2015 DGAC did not recommend “eating less fat” or reducing the consumption of eggs or dairy products. (In fact, Teicholz wrote in a February New York Times op‐ed that “experts on the committee that develops the country’s dietary guidelines acknowledged that they had ditched the low‐fat diet.”26) The Dietary Guidelines for Americans has never recommended eating less meat or dairy products. (In some editions, the DGA has included advice such as “moderate your use of eggs” or “use egg yolks and whole eggs in moderation. Use egg whites and egg substitutes freely…”).

Response: This question was already addressed in my “Rapid Response.” That response, which was peer reviewed, concluded that no clarification or correction was needed. Update: Dr. Hefland and The BMJ agreed that this point was not in error.

11.Teicholz states that “studies showed mixed health outcomes for saturated fats, but early critical reviews, including one by the National Academy of Sciences, which cautioned against the inconclusive state of the evidence on saturated fats and heart disease, were dismissed by the USDA when it launched the first dietary guidelines in 1980.”

Correction: The USDA (and DHHS) published the 1980 Dietary Guidelines for Americans in February 1980. Toward Healthful Diets, the National Academy of Sciences report cited by Teicholz, was published in May 1980.27 USDA could not have dismissed Toward Healthful Diets, because the report was published after the Dietary Guidelines were released.

Response: This point is disputable. The Guidelines were launched in the same year, only a few months before the NAS report, and the debate was ongoing throughout the year. The NAS study was part of this debate. Update: Dr. Hefland and The BMJ agreed that this point was not in error.

 Systematic methodology questions:

The three reviews in question are covered in appendices E-2.26, E-2.27 and E 2.43:

http://health.gov/dietaryguidelines/2015-scientific-report/14-appendix-E2/e2-26.asp

http://health.gov/dietaryguidelines/2015-scientific-report/14-appendix-E2/e2-27.asp

http://health.gov/dietaryguidelines/2015-scientific-report/14-appendix-E2/e2-43.asp

Note that these four alleged errors by CSPI all refer to two reviews: one on saturated fats (and heart-disease outcomes) and one on the Dietary Patterns (on all three outcomes, combined: obesity, diabetes, and heart disease).

Update, 11/28/2016: I did not realize until recently that the two reviews in question did, in fact, use a pre-defined method to search the scientific literature  The reason for my oversight is that (1) the appendices are not part of the printed report, (2) these searches are not mentioned in the methodology section at the top of each appendix, which is highly unusual. In the thousands of scientific papers I’ve read over more than a decade of research in this field, if a systematic search is undertaken, it is mentioned in a paper’s methodology section. Also (3), the description of the literature searches is easy to miss because they are appended, at the end of each appendix, after the reference section. This is also highly unusual for a scientific paper–I don’t think I’ve ever encountered this before. 

However, the presence of pre-defined search strategies does not, in itself, undermine my article’s assertions that these key reviews were conducted in ways that were non-systematic, for at least three reasons

1. The literature searches in these appendices described are not dated. It is standard practice to provide a date for a literature search, so that methods are reproducible. Most of the other systematic searches in the DGA report are dated; [Update, 11/28/16: Dr. Hefland states that these searches included date ranges for the literature, but this misunderstands my point.  I am not referring to the date range for the literature search but the specific day on which the search was undertaken. It is standard procedure to provide this date so that a search is reproducible. This is part of the systematic process.]
2. A sizeable fraction of the papers chosen for analysis were selected via “hand searches.” The practice of selecting papers via “hand search” was standard before scientific journals became available electronically, but all the papers included in these reviews were available online and should have been discoverable using standard electronic searches. In the case of the review on saturated fats, 1/2 of the papers selected came from a “hand search,” and for the Dietary Patterns, the proportion from “hand searches” was 1/3. The DGAC does not specify which papers were obtained using hand searches. Thus, this methodological approach is not systematic in that it is neither transparent nor reproducible.  [Update, 11/28/16: Dr. Hefland states that hand searches are “widely used and encouraged,” and he cites a Cochrane handbook as his reference. The exact citation, here, does not state that hand searches are “widely used and encouraged.” This might be true of searches on older papers but seems questionable for these particular reviews, which covered literature only since 2009 (and is therefore online) and which did not first exhaust the electronic options: i.e., one review did not search multiple online data bases, and neither searched in languages other than English]
3. Update, 11/28/16: The review for the Dietary Patterns does not state which data bases were searched. Therefore, this search is not reproducible.

In conclusion, the added information that these reviews included pre-defined methods for the literature searches does not undermine one of my article’s key assertions, namely that these reviews were conducted in ways that were not systematic.

For an overview of the BMJ retraction request letter, see an earlier blog post.

[i] http://www.nel.gov/template.cfm?template=sort_list_template&key=835 20 http://www.nel.gov/evidence.cfm?evidence_summary_id=250271
21 http://www.nel.gov/evidence.cfm?evidence_summary_id=250238
22 http://www.nel.gov/evidence.cfm?evidence_summary_id=250252

23 http://www.nel.gov/evidence.cfm?evidence_summary_id=250239 24 http://www.nel.gov/evidence.cfm?evidence_summary_id=250237 25 http://www.nel.gov/evidence.cfm?evidence_summary_id=250276