Cognitive Biases and Statistical Analyses on Low Carb Diets
During a heated discussion about the merits (or detriments) of specific diets today, it is not uncommon for defendants of any eating pattern to quote certain “scientific studies” or “systematic trials” that allegedly prove a given point that they are trying to make. This is a prime example of how cognitive biases affect the nutritional paradigms to which we adhere. Proponents of veganism and plant-based diets will quickly point to several studies, including:
- This meta-analysis looks at the vegetarian diet’s effects on blood lipids,
- Or this study “proved” that a vegan diet lowered heart disease risk in children and adults suffering from obesity while lowering the children’s weight and the parents’ cholesterol and blood sugar levels.
On the other side of the debate, proponents of meat-eating will quickly point to several scientific studies showing that meat is one of the most efficient foods due to its high nutritional density and ability to provide a high concentration of nutrients with a relatively low amount of calories. They claim that studies show that meat is one of the most significant sources of complete proteins and that animal protein intake is a reliable indicator of the mass muscle index, the quantity of skeletal muscle. Eating meat regularly is a reliable predictor of longevity, according to one 2014 study.
Lord Kelvin, the British mathematician, and physicist, once famously remarked that “the grandest discoveries of science have been but the rewards of accurate measurement patient, long-continued labor in the minute sifting of numerical results.” While there is certainly truth in that sentiment, it is also true that many scientific inquiries are guided by the prevailing partialities of the people conducting the research in today’s society. As economist Joseph Stiglitz has said, “what you measure affects what you do.”
Zad Rafi is a researcher at NYU Langone, one of the nation’s premier academic medical centers devoted to patient care, education, and research. He also regularly writes about the issue of statistics in medicine at his website: http://lesslikely.com.
At NYU Langone, Rafi began his work as a research associate to help run a cluster randomized controlled trial for technology-based obesity intervention. This work initially focused on ensuring data was coming in correctly and being recorded and cleaned.
“I eventually developed an interest in evidence synthesis and appraisal, being highly invested in the philosophy of evidence-based medicine,” Rafi told the Paleo Foundation in a recent interview. “I spent a lot of time around medical librarians to learn how to do systematic reviews and meta-analyses of various topics to make sure I was rigorous and left little room for selection bias and it became a skill that became quite handy for several future projects.”
Despite this initial interest and excitement around systematic reviews and meta-analyses, Rafi mentions that this interest eventually waned once he advanced his training in designing and analyzing experiments. “Having seen novice mistakes by so-called EBM practitioners who were regularly conducting systematic reviews and meta-analyses, I realized (partially on my own, but also having read the critiques of luminaries in medical statistics and epidemiology) that a lot of the evidence that was out there simply could not be trusted due to inadequate design and analyses of primary studies (many of which were conducted by physicians and physician-epidemiologists),” he says.
Rafi found that many of the primary studies and meta-analyses suffered from the same cognitive biases they claimed to be free of. This finding drastically changed his research interests to statistical methods for various studies. “For the most part, I was put off by the skeptic and EBM community who I believed lacked the technical skills to properly conduct or even interpret studies, let alone design policies around them,” he said.
The Issue of Cognitive Biases and Misinterpreting Statistical Outputs in Certain Nutritional and Dietary Programs
More recently, Rafi has used his knowledge of cognitive biases and statistical analysis within the medical world to analyze some troubling claims made by proponents of different, popular dietary trends. “The issue of researchers misinterpreting statistical outputs is rampant in nearly every scientific discipline because there is nothing intuitive at all about probability and statistics,” Rafi tells us. “Even in statistics, statisticians and methodologists regularly make such mistakes and are rarely criticized because a certain philosophy of statistics has become so dominant and powerful that it has infiltrated nearly every discipline and is defended by the old guard.”
In the specific case of medical research, Rafi believes that similar mistakes have a high prevalence. He believes that several surveys have been done on physician researchers showing that they need help understanding vital concepts in statistics. “In nutrition, it is just as bad, if not worse, because not only is nutrition a more difficult subject to study due to how interconnected it is with everything else, but also because the stakes tend to be much lower than say a drug study with millions of potential users,” he believes.
The soaring popularity of low-carb diets, such as the Keto diet, is also susceptible to this statistical misinterpretation. One of the most commonly cited studies that promote the health benefits of the Keto diet concludes: “The present study shows the beneficial effects of a long-term ketogenic diet. It significantly reduced the body weight and body mass index of the patients. Furthermore, it decreased the level of triglycerides, LDL cholesterol, and blood glucose, and increased the level of HDL cholesterol.”
However, Rafi questions the statistical legitimacy of some prominent studies promoting low-carb diets. “I have no doubt in my mind that keto and low-carb studies are just as bad as all the other nutrition studies being conducted on various diets. Nutritional epidemiology is in particular, an egregious example of throwing complex statistical tools that users barely understand at very messy data and hoping to get an answer. But again, this isn’t limited to keto research, nutrition research, or just medical research, see the recent methodological debates around the “replication crisis” in sports nutrition/medicine and in psychology,” he says.
Exaggerated Claims and Cognitive Biases within the Low-Carb Community
In a recent Paleo Foundation interview with Dr. Paolo Raggi, a cardiologist from the University of Alberta, Raggi was highly critical of some of the claims made by advocates for the Keto diet, which he believed were made with a limited scientific basis. Rafi thinks that the general tendency to misinterpret statistical claims also affects medical and scientific proponents of the Keto Diet.
“My general opinion is that a ketogenic diet can be a very useful tool for certain groups of people for various health issues, and there is some good evidence for these outcomes,” Rafi says. “However, there is so much noise and quackery and ludicrous claims attributed to the diet, that it is not surprising to me that any health practitioner let alone a cardiologist or skeptic would be put off by it and highly aggressive against those who promote the diet.”
From Rafi’s analysis, the main issue is the need for more nuance in interpreting the existing data, which he believes is all over the place in terms of quality. “For those who are heavily invested in the ketogenic diet and have based their entire lives around it, I can only imagine that they will ignore any genuine criticisms of the diet or harms that could come from it for certain individuals as it has become their identity. The same goes for those who viciously attack it without carefully reading the literature, as very little can stop them from assuming that it is all pseudoscience or highly dangerous.”
As with most nutritional and general health topics, Rafi believes that much research still needs to be done on the ketogenic diet to lay a solid foundation for some of the supposed health claims. “One of the biggest shortcomings is simply the lack of long-term data from rigorously designed randomized trials for other health outcomes besides weight loss,” he believes.
What do the Existing Statistical Studies or RCTs on the Keto Diet Prove?
Many studies focus on essential issues such as weight loss or other imperative health-improving outcomes. A quick review of PubMed, the Web of Science, or Scopus websites will show dozens (if not hundreds) of studies showing low-carb diets’ benefits. In Rafi’s opinion, “some of these (studies) are likely real signals, while others were doomed to fail from the beginning because they weren’t designed and analyzed properly. The systematic reviews and meta-analyses that often compile these studies and analyze them are also often misleading because they fail to catch basic errors because they do not have sufficient knowledge on how to run these studies,” he believes. “Organizations that are highly respected by the EBM community like Cochrane even make these sorts of mistakes. This leads to overconfident reports and conclusions regarding the diet’s effects.”
On a more superficial level, the quick rise in popularity of the Keto Diet has explicitly led to several dubious claims by self-proclaimed diet gurus. In Rafi’s opinion, outlandish nutritional advice such as eating entire sticks of butter or gobbling down too much oil will not find scientific footing. Unfortunately, many people associate the Keto Diet with precisely these “Keto myths” that could potentially pose a problem to the health of people who follow these diets.
Lastly, Rafi believes some counterarguments, or “attacks” against the Keto Diet, are not based on sound, statistical input. As other medical professionals and researchers that The Paleo Foundation has interviewed have stated, the lack of focused long-term studies on health outcomes associated with the Keto Diet makes it challenging to accurately communicate health outcomes such as weight loss and cardiovascular disease. The statements made by some medical professionals that connect the Keto Diet to long-term health vulnerabilities are also similarly lacking in reliable, trustworthy, and long-term data.
Despite this deficiency in long-term studies, Rafi says he is interested in learning and discovering some of the effects of Keto and other low-carb diets on the gut microbiome. “For many individuals with gastrointestinal functional disorders, the diet seems to offer short-term relief, but there is serious uncertainty regarding how this will change the entire ecosystem, and whether it will lead to positive or negative outcomes,” he states. “I think there is much to study there, and it is worthwhile to study if done right.”
As with any nutritional program, objective statistical evidence analyzed from an unbiased, scientific perspective is essential in avoiding cognitive biases and determining the diet’s potential health benefits and detriments. In the specific case of the Keto Diet and other low-carb diets, strong opinions on both sides of the “debate” often make strong claims tainted by cognitive biases. As the Keto and low-carb diet community shows no sign of diminishing popularity, further scientific studies and unbiased statistical analyses are necessary to determine the health outcomes of this popular dietary trend.