Need for Further Statistical Analyses of Keto Diets

During a heated discussion about the merits (or detriments) of specific diets today, it is not uncommon for defendants of any eating pattern to quote certain “scientific studies” or “systematic trials” that allegedly prove a given point that they are trying to make. Proponents of veganism and plant-based diets will quickly point to several studies, including:

• This meta-analysis looking at the vegetarian diet’s effects on blood lipids,
• Or this study that “proved” that a vegan diet lowered heart disease risk in children and adults suffering from obesity while also lowering the children’s weight and the parents’ cholesterol and blood sugar levels.

On the other side of the debate, proponents of meat-eating will quickly point to several scientific studies showing that meat is one of the most efficient foods due to its very high nutritional density and its ability to provide a high concentration of nutrients with a relatively low amount of calories. They claim that studies show that meat is one of the most significant sources of complete proteins and that animal protein intake is a reliable indicator of the mass muscle index, the quantity of skeletal muscle. Eating meat regularly might even be a reliable predictor of longevity, according to one 2014 study.

Lord Kelvin, the British mathematician and physicist, once famously remarked that “the grandest discoveries of science have been but the rewards of accurate measurement patient, long-continued labor in the minute sifting of numerical results.” While there is certainly truth in that sentiment, it is also true that many scientific inquiries are guided by the prevailing partialities of the people conducting the research in today’s society. As economist Joseph Stiglitz has said, “what you measure affects what you do.”

Zad Rafi is a researcher at NYU Langone, one of the nation’s premier academic medical centers devoted to patient care, education, and research. He also regularly writes about the issue of statistics in medicine at his website: http://lesslikely.com and is a member of the Keto Certified Standards Team.

At NYU Langone, Rafi initially began his work as a research associate to help run a cluster randomized controlled trial for technology-based obesity intervention. This work initially was focused on ensuring that data was coming in correctly and being recorded and cleaned.

“I eventually developed an interest in evidence synthesis and appraisal, being highly invested in the philosophy of evidence-based medicine,” Zad Rafi told the Paleo Foundation in a recent interview. “I spent a lot of time around medical librarians to learn how to do systematic reviews and meta-analyses of various topics to make sure I was rigorous and left little room for selection bias and it became a skill that became quite handy for several future projects.”

Despite this initial interest and excitement around systematic reviews and meta-analyses, Rafi mentions that this interest eventually waned once he advanced his training in designing and analyzing experiments. “Having seen novice mistakes by so-called EBM practitioners who were regularly conducting systematic reviews and meta-analyses, I realized (partially on my own, but also having read the critiques of luminaries in medical statistics and epidemiology) that a lot of the evidence that was out there simply could not be trusted due to inadequate design and analyses of primary studies (many of which were conducted by physicians and physician-epidemiologists),” he says.

Rafi found that many of the primary studies and meta-analyses were suffering from the same cognitive biases that they claimed to be free of. This finding eventually drastically changed his research interests to statistical methods for various types of studies. “For the most part, I was put off by the skeptic and EBM community who I believed lacked the technical skills to properly conduct or even interpret studies, let alone design policies around them,” he said.

The Issue of Misinterpreting Statistical Outputs in Certain Nutritional and Dietary Programs 

More recently, Zad Rafi has used his knowledge about statistical analysis within the medical world to analyze some of the troubling claims made by proponents of different, popular dietary trends. “The issue of researchers misinterpreting statistical outputs is rampant in nearly every scientific discipline because there is nothing intuitive at all about probability and statistics,” Rafi tells us. “Even in statistics, statisticians and methodologists regularly make such mistakes and are rarely criticized because a certain philosophy of statistics has become so dominant and powerful that it has infiltrated nearly every discipline and is defended by the old guard.”

In the specific case of medical research, Rafi believes that similar mistakes have a high prevalence. He believes that several surveys have been done on physician researchers showing that they fail to understand vital concepts in statistics. “In nutrition, it is just as bad, if not worse, because not only is nutrition a more difficult subject to study due to how interconnected it is with everything else, but also because the stakes tend to be much lower, than say a drug study with millions of potential users,” he believes.

The soaring popularity of low-carb diets, such as the Keto diet, is also susceptible to this statistical misinterpretation. One of the most commonly cited studies that promotes the health benefits of the Keto diet concludes: “The present study shows the beneficial effects of a long-term ketogenic diet. It significantly reduced the body weight and body mass index of the patients. Furthermore, it decreased the level of triglycerides, LDL cholesterol and blood glucose, and increased the level of HDL cholesterol.”

However, Zad Rafi questions the statistical legitimacy of some prominent studies promoting Keto diets. “I have no doubt in my mind that keto and low-carb studies are just as bad as all the other nutrition studies being conducted on various diets. Nutritional epidemiology is in particular, an egregious example of throwing complex statistical tools that users barely understand at very messy data and hoping to get an answer. But again, this isn’t limited to keto research, nutrition research, or just medical research, see the recent methodological debates around the “replication crisis” in sports nutrition/medicine and in psychology,” he says.

Exaggerated Claims within the Low-Carb Community 

In a recent Paleo Foundation interview with Dr. Paolo Raggi, a cardiologist from the University of Alberta, Raggi was highly critical of some of the claims made by advocates for the Keto diet, which he believed were made with a limited scientific basis. Rafi believes that the general tendency to misinterpret statistical claims also affects medical and scientific proponents of the Keto Diet.

“My general opinion is that a ketogenic diet can be a very useful tool for certain groups of people for various health issues, and there is some good evidence for these outcomes,” Rafi says. “However, there is so much noise and quackery and ludicrous claims attributed to the diet, that it is not surprising to me that any health practitioner let alone a cardiologist or skeptic would be put off by it and highly aggressive against those who promote the diet.”

From Zad Rafi’s analysis, the main issue is the lack of nuance in interpreting the existing data, which he believes are all over the place in terms of quality. “For those who are heavily invested in the ketogenic diet and have based their entire lives around it, I can only imagine that they will ignore any genuine criticisms of the diet or harms that could come from it for certain individuals as it has become their identity. The same goes for those who viciously attack it without carefully reading the literature, as very little can stop them from assuming that it is all pseudoscience or highly dangerous.”

As with most nutritional and general health topics, Rafi believes that there is still much research that needs to be done on the ketogenic diet to lay a solid foundation on some of the supposed health claims. “One of the biggest shortcomings is simply the lack of long-term data from rigorously designed randomized trials for other health outcomes besides weight loss,” he believes.

What do the Existing Statistical Studies or RCTs on the Keto Diet Prove? 

A quick review of PubMed, the Web of Science, or Scopus websites will show dozens (if not hundreds) of different studies that show the benefits of low-carb diets. Many of these studies focus on important issues such as weight loss or other imperative health-improving outcomes. In Zad Rafi’s opinion, “some of these (studies) are likely real signals, while others were doomed to fail from the beginning because they weren’t designed and analyzed properly. The systematic reviews and meta-analyses that often compile these studies and analyze them are also often misleading because they fail to catch basic errors because they do not have sufficient knowledge on how to run these studies,” he believes. “Organizations that are highly respected by the EBM community like Cochrane even make these sorts of mistakes. This leads to overconfident reports and conclusions regarding the diet’s effects.”

On a more superficial level, the quick rise in popularity of the Keto Diet has explicitly led to several dubious claims by self-proclaimed diet gurus. In Rafi’s opinion, the outlandish nutritional advice such as eating entire sticks of butter or gobbling down too much oil is certainly not going to find scientific footing. Unfortunately, many people associate the Keto Diet with precisely these types of “Keto myths” that could potentially pose a problem to the health of people who follow these types of diets.

Lastly, Zad Rafi also believes that some of the counterarguments, or “attacks” against the Keto Diet, are not based on sound, statistical input. The statements made by some medical professionals that connect the Keto Diet to long-term health vulnerabilities are also similarly lacking in reliable, trustworthy, and long-term data. As other medical professionals and researchers that The Paleo Foundation has interviewed have stated, the lack of focused long-term studies on health outcomes associated with the Keto Diet make it difficult to make accurate statements on health outcomes such as weight loss and cardiovascular disease.

Despite this deficiency in long-term studies, Rafi says that he is interested in learning and discovering some of the effects that Keto and other low-carb diets might have on the gut microbiome. “For many individuals with gastrointestinal functional disorders, the diet seems to offer short-term relief, but there is serious uncertainty regarding how this will change the entire ecosystem, and whether it will lead to positive or negative outcomes,” he states. “I think there is much to study there, and it is worthwhile to study, if done right.”

As with any nutritional program, objective statistical evidence analyzed from an unbiased, scientific perspective is essential in determining the potential health benefits and detriments of that diet. In the specific case of the Keto Diet and other low-carb diets, strong opinions on both sides of the “debate” often make strong claims tainted by cognitive biases. As the Keto and low-carb diet community shows no sign of diminishing popularity, further scientific studies and unbiased statistical analyses are necessary to determine the health outcomes of this popular dietary trend.