‘Research says…’

How many articles have you seen on the newspaper and credible websites likes CNN.com that have a million studies referenced? How many authors fill their articles with ‘Research says…’ and link to PubMed? Sure makes the article seems legit, right? But tell me this – How many of those studies have you even read?

Here is the deal. You can write an article about anything under the sun and have studies backing up the claims! Why? Because such is the state of science today. Strongly influenced by politics and money! In this post, I’ll discuss some common mistakes with respect to understanding published literature and explain how you can save yourself from being fooled into believing that all ‘science’ is legit.

Don’t get me wrong. I am not saying that all studies are BS. I’m only saying that you need know how to differentiate between the good and bad studies and more importantly, you need to be able to gauge the relevance of the results to you, your body, your goals and your lifestyle. This is very important because, even for the most health conscious and well-meaning reader, studies and their conclusions can be very misleading if the he/she doesn’t posses a certain level of knowledge/understanding of the subject.

Case 1 – Insufficient understanding

Studies look at multiple (health) markers but researchers very often, in an effort to write attractive and tangible conclusions, decouple the results to a point of oversimplification that it becomes absolutely useless. You probably see many studies that say consumption of whole grains produce drastic improvements in health markers. But what you don’t understand is that when the volunteers were made to eat whole grains, they were also made to dump junk food and sugars, they were (in one way or the other) put in a calorie deficit and they were (in some way) made more active than they previously were! Basically, the control (i.e. the volunteer’s existing diet) is so unbelievably inferior that pretty much any half-decent diet will cause major health improvements. In addition to that, the calorie restriction results in weight loss which also improves health markers. The real deal is in the details and the 3 line conclusions or headlines you read don’t have them!

For example, distance running, you might have heard, is extremely beneficial for heart health. But this is all you hear and read (mostly because this is all you want to hear and read!). If you did take the time to dig deeper into the science behind this statement, it is not really the act of distance running that provides the benefit, but the elevated heart rate due to (any) activity that is responsible for the said benefit. But what is not taken into account is that distance running, in addition to elevating your heart rate, does other things like burning lean muscle and forcing overuse of certain joints. But then you, the running enthusiast, only register the headlines…

‘Research proves running reduces cardiovascular diseases by 40%’

… and jump into your shoes all pumped up for a long run. You invest a lot of time and effort on this because you are under the assumption that you are ‘leading a healthy lifestyle’. But eventually what happens is, due to the negatives associated with chronic distance running, you end up weak and skinny fat with achy joints. But by this time you have developed an emotional bond with running and you view running as the be-all-end-all of fitness that you are practically unable to look at your issues objectively! How many times have you complained of joint/ligament/muscle pain? And how many times have you said/thought… ‘Well, if you’re a runner, you will get injured! Its all part of the game!’. Once again… the real deal is in the details and headlines don’t have them!

Just to prove that I’m not randomly hating on distance running, let’s look at another example. I’m sure you’ve heard that coconut oil is magic and that it increases HDL, decreases LDL, increases metabolism, cures eczema and heals the gut. Let’s say you read the abstract and the conclusion says…

‘Test subjects who consumed 2 tablespoons of coconut oil every morning in an empty stomach lost more weight and had better blood lipids (HDL, LDL and Triglycerides) than the control group.’

Well, you think there is truth to this since you have read the study (only abstract of course) and this is perfect for you right now ‘cos your goal is weight loss and you don’t seem to have time to workout. So you drive to the nearest health store and buy yourself a can of extra virgin cold pressed coconut oil and start consuming 2 tablespoons every morning. A couple of months go by and you realize you’ve actually gained 3-5 lbs. WTF! How did that happen? You took the suggested dose per the advice in the paper! How? Holly hell! How? Well, like 99.99% of people, you didn’t read the paper and so you have no idea what else these test subjects did. You added 2 tablespoons of coconut oil (~ 250 cals) to your existing calorie intake but didn’t really remove anything else. What you didn’t do but the volunteers probably did was to manipulate energy intake (via exercise or food restriction) to ensure they were in a calorie deficit. The paper would have discussed their bodyweights, RMR, BMR, activity, maintenance calorie requirements and actual calorie intakes, but you didn’t read the paper! And once again… the real deal is in the details and conclusions don’t have them!

Case 2 – Neglecting the N=1 factor

Sure studies tell us whats good and whats bad in general and they can be useful in understanding a phenomenon or a concept. But that is really where it stops. The exact outcome of the study has nothing to do with you. A diet that produces 4 lb/week of fat loss on test subjects does not mean you will see the same amount of fat loss every week. There are differences between you and the people enrolled in the study – from lifestyle to body temperature to leptin sensitivity to receptor function to fat mobilization to mitochondrial activity!

And this is precisely why I find it funny when people cite a study and use that as a reason for doing or not doing something. I don’t care if there are 200 studies that say oatmeal is awesome. It absolutely messes with my gut and that is enough reason for me to stay the heck away from it! I don’t care if there are zero studies that prove elimination of grains improves sleep quality. I have solid n=1 evidence that I sleep better when my diet is grain free and that is worth a million studies! And I don’t care what you say ‘cos what I eat now is a result of extensive self-experimentation with respect to food quality, food quantity, meal timing, meal frequency etc. and you shouldn’t care about what I say if you have n=1 evidence that something works or doesn’t work for you! But, as obvious as it sounds, be very objective and true when experimenting with yourself. If a food or exercise doesn’t sit well, it doesn’t matter how much you love it, its got to go!

Case 3 – Assuming all studies are legit

Well, they’re not. They never were and they never will be.

Realize that it isn’t cheap to perform a study and every study that is done needs to be funded. Scraping just the surface – a team of surgeons/physicians/scientists/researchers/professors need to be hired, volunteers needs to be compensated, arrangements for the study (labs, rooms, food, testing facilities etc.) need to be made, animals need to be bred/bought/mobilized, benchmarks need to be set, tests need to be performed at various time intervals, analysis needs to be done, papers need to be written and publications need to be approved! And who do you think will fund all this? If you said the government, you get a $100 fine, you miss a turn and you don’t get to pass through Go! The correct answer is – people who have vested interest in the study. In other words, bigass food, medical and pharma companies!

Companies like Quaker will sponsor studies that prove oatmeal is good for health and companies like Monsanto will fund studies that say soy is a super food and Pharma companies will (abundantly) fund studies that prove statins save lives. So, now, it might not surprise you to know that most studies…

  • are initiated with the end result already decided.
  • are rigged by tweaking the study design in a way that the results, though unfavorable to the funding company, appear to be favorable.
  • have false conclusions that have absolutely nothing in common with the results obtained.
  • are short term studies which have no relevance towards long term health or are done on animals and have no carryover to human health.

So very clearly, one cannot make rigid dietary recommendations or choices purely based on scientific evidence. Then how does one know if a particular food/activity is good or bad?

In summary

1. Read the entire study in detail before you make changes or don’t make any changes.

2. If you don’t have the patience or time or capability or capacity to read and understand studies, talk to someone who does.

3. If you are convinced that the results of a study might benefit you, make the change only as long as it wont affect you adversely.

4. If you have made the change, listen to your body! And be objective about it. Always remember that studies don’t mean anything unless the change benefits YOU and if it doesn’t benefit you, throw it out.

And since people seem to understand pyramids better…

Read.

Understand.

Self-experiment.

Listen to your body.

Monitor and track progress.

Make decision to keep or dump change.

For more information on this topic be sure to read this post by J. Stanton on Gnolls.org and watch this video (Science For Smart People) by Tom Naughton.

Thats it for today folks! Have an awesome weekend and please share the post so your friends and family are in the know too!

Stay sane and safe ‘cos its a messed up world out there!

Peace out.

Advertisements

18 responses to “‘Research says…’

  1. Shashi July 2, 2011 at 12:25 am

    Perfect timing and well-written as usual! I have been pointing many friends to Tom Naughton’s Science for Smart People talk!

  2. Vizeet July 2, 2011 at 3:04 am

    We need to understand that our body is a complex system, there are hidden diseases in most of us and there are genetic differences (Though small) which is why healthy diet is quite individualistic. Self experiment is the key.

    This article covers “how to find out what might works for you” with solid reasoning.

    In almost every research there is conflict of interest (The funding company is the one which gets benefited) but that doesn’t mean you don’t read them. There are many vital information hidden in these research papers and that is what you should collect (Not the end result) and apply common sense.

    Modern research is pretty much a process of unlearning (what we have learned in ages).

  3. Mamatha July 2, 2011 at 6:23 am

    Well-written post, as always. The most important factor, n=1, is ignored by all these studies and the end result is often inconsequential for the individual. It’s ridiculous that cookie-cutter advice is given to people based on the outcome of a study/trial of a limited sample size.

  4. arv43 July 2, 2011 at 7:38 am

    Feels like you answered a personal request. Neat post. A tangent – I feel there are some cases where the “n=1” is overstated (not in studies, but in blogs).

  5. Vrishali July 4, 2011 at 1:00 pm

    Important topic , well presented. Being a health/nutrition junkie, scientific literature on these topics is all over the place. However, other than the factors you rightly point to, the human body is an extremely complex organism. It is very hard, if at all possible, to study all the interactions that lead to observable effects. Thus, we bound the phenomena and isolate parts of it that we want to study. Studies build on concepts and phenomena whose understanding depends upon the given concepts and instrumentation of the day. Different research circles allow different “leaps of interpretation” for missing gaps in the story. Science understands by controlling for the known endogenous/confounding factors, but the gaps and the endogenous/confounding factors are the phenomena. Thus, I feel we need other other hats for thinking too. Traditions of health and nutrition are one such “hat”. If they are not taken as dogma, but as material for inquiry and self experimentation, they broaden the limits of the experimental system to even beyond the human being to his/her environment. Not to pit science these against science, but they can be thinking hats in approaching self experimentation.
    What is surprising though, is how we, regardless of their understanding, feel comfortable citing scientific terms and second hand explanations for their health/nutritional choices. But we lack common sense in what we eat and how we cook. Perhaps not surprising. The terms, and the social pressure to know them, make us victims to the next health fad.

    • Vizeet July 5, 2011 at 7:13 am

      Look into the series “Food Reward: a Dominant Factor in Obesity” at http://wholehealthsource.blogspot.com/. If I agree with Whole Health Source then individual items may not give much info. Weight gain is complex, look into detailed plausible theories before agreeing.

    • RG July 6, 2011 at 10:24 am

      Hey Cameron – Welcome back! The study youre pointing to seems to have created quite a stir. A lot of nutritional researchers have expressed their views on the study. I, to be honest, havent had a chance to read the full study and hence am not qualified to say much. But again, this wasnt a controlled trial and too many factors come into play in such observational studies. So I wouldnt hang my hat on this or any other study that isnt controlled.

      • Cameron July 8, 2011 at 7:48 am

        Surely. Agreed. It’s more interesting to note in terms of correlational studies, i.e. if you’re going to find out ‘what works best for you’, it’s best to start with a good point of reference. Is it highly controlled? No, of course not. It’s next to impossible to have controls in place in any study for every single thing. This is more true for human studies simply because humans (and mammals in general) and their behaviors are very complex. The simplest way to counter that though is to look at trends, which is essentially what this study has done. Taking ‘snapshots’ in the form of two week trials with 10 people or so, I agree, isn’t very informative. However, having a huge sample size to draw from over a long period of time, as is available in this data set, allows for more incisive analysis, despite its correlative nature. Point being, that while their conclusions may not be ‘right’, it’s a pretty decent place to start for any personal exploration. This leads to an important distinction between a ‘conclusion’ and a ‘fact’. Conclusions within a paper or study are simply the authors interpretations of their data and should be nothing more. One should feel free to agree or disagree with any stated conclusion but should only do so based upon their own data or specific logical reasoning as to why said conclusion may not be the case. This differs from what is considered scientific fact in that a ‘fact’ is something that has been observed and replicated without major deviation by many different people over time. This by no means indicates something that should be static. Indeed, even scientific fact must be free to grow and evolve given new information. However, this does not preclude the initial observation from being basically correct (see gravity, DNA-RNA-protein dogma).

        On a side note, I’d like to comment about funding and scientists. Yes, what you say is correct about Big Food and Big Pharm funding some studies to essentially support new products that they have coming out. However, to suggest that this practice dominates the scientific literature is flawed. The type of studies that these entities would likely be funding are largely compartmentalized to certain fields of science, e.g. nutrition, human behavior, environmental impact, etc. Nearly all basic research is funded either by government grants through the NIH or similar agencies such as the CDC. What is left is typically covered either through intra-institutional funding or privately funded by principal investigators (PI – lab boss) that collaborate with disease oriented pharmaceutical companies. I make this distinction here because there are many smaller companies whose only interests are in curing/treating chronic human disease such as Alzheimer’s, CF, cancer, etc. and have no interest in such things as erectile dysfunction, appetite suppressors, etc. The vast majority of the searchable content on NCBI (http://www.ncbi.nlm.nih.gov/) is government funded, peer reviewed, basic research.

        For example, I work in a physiology lab at UT-Southwestern Medical Center in Dallas, TX. I am personally funded by a student fellowship grant from the university. Most of the people in the lab are either supported similarly or are part of grants from the NIH. Our PI also has direct collaborations with a local pharmaceutical company that allows him to supplement the lab’s funding with income from that venture. Again, they are far and away from Big Pharm. They employ perhaps 20-30 people and are interested principally in CF and diseases involving antioxidant response pathways such as cancer and ALS (Lou Gherig’s Disease), which is what I work on.

        I must say at this point that I’m not responding directly to your post per se. Rather, this is a general response to a trend of thought that I’ve encountered in the message boards and blogs of many different websites as it regards science and scientists. Most scientists aren’t jackasses, and most aren’t propped up by dirty money. Most of us are just curios geeks that are really good at tinkering in a very directed manner 🙂

  6. Pingback: 07/06/11 – Nutrition Challenge Day #10 – “Small” Hero WOD

  7. Ryan Kelly July 6, 2011 at 9:07 am

    I really do sympathize with a lot of your points to a great degree, but your description of scientific research leads me to believe that you are not following your own advice. I want to address a few of your points:

    Case 1: “researchers very often, in an effort to write attractive and tangible conclusions, decouple the results to a point of oversimplification that it becomes absolutely useless”
    You seem to think that a scientific paper is only the conclusions. For almost every scientific paper I’ve read, there is extensive discussion of the possible explanations for the results, and it is extremely difficult to publish a paper without these explanations. In fact, overreaching in your conclusions is a common way to have a paper rejected for publication. The section you’ve written here makes me think that you skip over the methods and results straight to the conclusions, or worse yet, you think that an abstract can actually prove something in a few hundred words. The purpose of science is to expand the fringe of knowledge, both for practical application and for other scientists to test and build on. Of course I believe people should try to read the studies if they can, but it requires patience and practice to be able to digest scientific papers. I also think it’s fine to not believe everything you read in the paper (many news outlets typically misreport the main scientific findings), to pin the blame on researchers is selling science extremely short. Everything you read and every source of information comes with some bias, but science is explicitly designed and practiced to minimize that bias, even though it is necessarily still present to some extent. In short, even though science too is flawed to some degree, it’s the best source of objective information we have, so it’s worth paying some attention.

    Case 2: “The exact outcome of the study has nothing to do with you”
    I think a better way to put this would be “The results of studies apply to a population of people, but the cross-individual effects may exceed the population effect.” It is true that you can basically always run more subjects and produce some significant result, since the statistical power of a study increases as N increases. So, as you say, the differences between people may dwarf the main effect in a study. And of course, if you have a particular problem with a food, than obviously it doesn’t work for you. But I think you are perhaps overestimating the average person’s ability to self experiment objectively – emotions and skewed perceptions are inherently tied to our reasoning, and while introspection and observation are critical, I’m sure everyone can come of with examples where their inherent biases led them astray from objectivity. You yourself even provide an example in “Case 1” where a runner who enjoys running already is no longer objective about the things happening to his/her joints. So I agree with you, very few people are not subject to this bias. I think it’s fine to encourage people to be as objective as possible, and to pay attention to their body, but N=1 is not always worth a million studies. If this was true, we’d all still be smoking.

    Case 3: “Most studies…are initiated with the end result already decided. are rigged by … have false conclusions…”
    This statement is just false. Science does certainly have the kinds of studies you describe, especially where the money is dirtier or more profit driven, like the food industry. This is why the funding sources are required to be disclosed in the reporting of results. There are certainly many high profile cases that support what you’re saying, but I don’t really see a justification for the claim that “most studies” are this way. Data falsification, if found, can completely ruin the rest of a scientists life, and is taken extremely seriously by journals and other scientists. In academic settings (not research within companies) scientists are rewarded by producing replicable results that advance the fringe of our understanding. They are can be rewarded strongly for debunking other research, and a large amount of back and forth is continuously going on within every field. I think the problem you have with science is the part of it that involves a conflict of interest. Despite your claim that the government won’t be funding research, most academic research is funded by the government, and most industry research is funded by industry. Obviously the industry, for-profit model will have more inherent bias, but academic research usually will have less. Of course, there are counter examples to this as well, such as the government being more likely to sponsor pro-milk research (the government is in the cheese business), but the goal is to minimize these effects, not to completely give up and ignore everything. I agree that knowing the funding source is important, and this also happens to be important to scientists. There was even a relatively recent Nature paper (Martinson et al. 2005) that attempted to explore this issue and the reasons for it, so clearly this is an ongoing issue. This is exactly why the current political debate about funding disclosure and publicly funded science is so important.

    I think your “In summary” section is quite good though – if this was all you had written I would completely agree. Healthy skepticism is exactly what powers science and I also think it can and should be a part of every individual. But I worry that many of your feelings above delve into “unhealthy skepticism” and many of your claims are overreaching and misleading, even a bit dangerous. The blanket blame over all science actually diminishes your point, which I think is mainly valid. I think this is a concise and reasonable description of the worries that you have:
    http://undsci.berkeley.edu/article/who_pays

    Ryan

    • RG July 6, 2011 at 10:44 am

      Ryan,
      First of all, thank you for taking the time to write down your thoughts on the subject and the blog post. Appreciate it.

      Case 1 – My point is exactly that. Read the study. Not the conclusions. If do spend enough time reading the entire study (including materials and methods and results) you will learn what the study exactly says. I am a trained researcher (in the healthcare industry) and have been doing this for many years.

      Case 2 – N=1 is worth a million studies if done right just like how a study is worth being applicable to a million people… if done right! Again, if I eat a food and end up with GI and still think its good for me, then I’m just stupid. I’m just believing what I want to believe. Let me give you an example. The paleo peeps love bacon. Everyone loves bacon. But the paleo folks OD on bacon and eating 1/2 to 1 lb of bacon in one sitting isnt rare by any standard. Why? Is this because bacon is good for you? Truth is, bacon is, at best, not bad for you in moderate quantities. But people continue to go nuts on it because of the flawed belief – Its paleo so its good for me!. Read Whole9’s post on bacon. Awesome is that!

      Case 3 – I should’ve said ‘Most studies that are published in mainstrem media’. We have taco bell advertising that they use quality ingredients and Lipitor advertising that 80% of ppl with heart attacks have high cholesterol. How much truth is there to these statements? Money plays a HUGE role here. And the new choco cherios… heart healthy? seriously? The thing is Ryan, I’m only concerned about the ‘studies’ that the lay person reads. Academic studies with less funding and hence small sample sizes dont really make the news and the lay person doesnt really go in search of these studies. I do agree with you about credibility and good studies. But I’d also ask you to look into research done to disprove the lipid hypothesis which has been ignored too many times.

      Again, thank you for taking the time.

  8. Pingback: The Gluten(free) Myth – Part 1 « Harder. Better. Faster. Stronger.

%d bloggers like this: