More Tips on How to Evaluate “New Study” Research

By AltMed1-Peggy  


Yesterday, we looked at how to evaluate “new study” research and we focused on the gold standard type of study, the placebo-controlled, double-blind, randomized multi-center trial.

The randomized controlled trial, or RCT for short, works well when the goal of the study is to find out how a new drug or new treatment stacks up when it is compared to a placebo or to an existing drug or treatment (i.e., something that is already being used to treat the same condition the new item is designed to treat).

But RCT is not the only type of study. Other types may be used at different stages of research to achieve different goals.

The starting point is an idea or finding in basic research.

Suppose, for example, there is a scientist somewhere whose lab work involves enzymes and one day he is thinking about how lactose enzyme replacement products make it possible for people who are lactose intolerant to eat cheese without suffering the bloating, gas, cramps, nausea, and diarrhea symptoms lactose intolerance can cause. And that makes him think of the guest who was sober when he arrived at the office holiday party but appeared drunk after just one beer…  Wouldn’t it be great if people like that guest whose bodies don’t metabolize alcohol normally could take an enzyme replacement like the lactose intolerant person can, the imaginary scientist thinks… If we could identify an enzyme that is different in the alcohol-challenged person’s body compared to a normal body, maybe that could lead to treatment for alcohol overdose or alcoholism…So our imaginary scientist thinks about which enzyme should be targeted for study and figures out a plan of action to test the idea by doing “test tube” research the lab.

His idea will go through several steps of research, first to see if it has enough merit to continue studying it, and then to see if it works, and then if it works – and how well it works – on real people.

There are seven primary types of study that begin with lab and animal studies and  move up through the seven levels of study that form the pyramid “hierarchy of evidence,” according to the Introduction to Evidence-Based Medicine website and similar resources.

  1. At the bottom of the pyramid are lab studies and animal research, the foundation on which other research is built.
  2. Case series and case reports come next, forming the sixth level.
  3. Level five is case control studies.
  4. The fourth level is cohort studies.
  5. On level five: randomized controlled trials.
  6. Next to the top is systematic review at level six.
  7. And the peak of the pyramid, or level seven, is meta-analysis.

Those levels are useful for scientists to understand and they’re helpful for us to know but often the key factors we need to consider when you and I look at medical studies to decide whether their findings apply to us or not are a little simpler.

Here are some of the keys I look for when I read about new studies:

  • Was it was the study done in the lab or on animals?  That level of research is interesting and is a fundamental step to build on but more research will be needed to find out whether it may apply to humans. Bottom line: It may be promising but it’s not something we can use now or soon.

  • If the study was done on people, how similar are we to the groups who were tested? For example, a study of marathon runners and people aged 18 to 30 who run at least 20 miles each week will be applicable to others in the same age group whose exercise levels are comparable but not so applicable to arthritic seniors in their 70s who can barely manage to walk 7 miles a week.

  • How big was the study?  And did it last long enough to draw the conclusions the researchers are making now? The larger the number of people tested, the better. There can be good reasons for limiting the number of participants in a study and the information small studies provide can still be useful, but as a general rule, bigger is better.

  • Was the study done at or by a qualified facility, ideally one that has expertise in doing this type of study, such as a large medical school or university hospital? 

  • Was the study funded by a company that makes the product that was tested or did the researchers disclose any affiliations with such a company? The best answer would be ‘no.’

  • Did the researchers look at the right data? For example, some reports say studies testing certain herbal remedies were doomed to show the remedy failed because the researchers tested the wrong dose, the wrong form, or the wrong method of administering the remedy instead of testing it using the form, dose, and method recommended by herbal medicine experts.

  • Was the information the study collected or examined or analyzed actually useful? Some of the comments posted on public forums responding to news of the recent study that found calcium may increase heart attack risk noted that the study did not identify what type of calcium supplement the study participants took (that one factor can make a big difference), and also didn’t have firm data on how much of the supplement people took how often for how long. The researchers did have people’s recollections of what they’d taken, but is memory reliable enough?

  • Did the study or the researchers seem to have an agenda? Does it look as though the study was done not to gain information but to confirm an expected outcome? And if it was set up that way, was that appropriate?

Do you have other questions you would add to this list? Please share them via the comment feature on this page.

To learn more about types of medical studies, take a look at these sites:

Study Design 101

Non-randomised controlled study designs

Understanding Medical Research

Tips for how patients can evaluate health research

SUNY Downstate Medical Center’s EBM Tutorial Guide to Research Methods


© 2012, All rights reserved.

Post a Comment

You must be logged in to post a comment.