Critique of AHRQ REPORT "Meditation Practices for Health: State of the Research"

SUMMARY

Background: “The Agency for Healthcare Research and Quality (AHRQ) is the lead Federal agency charged with improving the quality, safety, efficiency, and effectiveness of health care for all Americans. As one of 12 agencies within the Department of Health and Human Services. AHRQ supports health services research that will improve the quality of health care and promote evidence-based decision making.” http://www.ahrq.gov/about/ataglance.htm

In 2007 the National Center for Complementary and Alternative Medicine (NCAAM) requested and funded a review and evaluation of research on the health benefits of meditation by the AHRQ, which was conducted at the University of Alberta Evidence-based Practice Center. The review covered the Transcendental Meditation technique and other techniques under the rubrics of Mantra meditation, Mindfulness meditation, Yoga, Tai Chi, and Qi Gong.

Serious Problems. Serious problems with the report were found by experts in the field who were invited to participate in the study process as peer reviewers. These problems were largely ignored by both the study authors and the study's sponsors at AHRQ and NCCAM. Reviewers independently found the study had so many methodological flaws and mistakes that they recommended the report be withdrawn until it was corrected. Standard peer review, fact checking, and editors are usually effective at correcting misinformation. Unfortunately, these safeguards were not honored in this report, and misinformation is now positioned to guide public policy on the use of meditation techniques for healthcare.

The peer reviewers pointed out that the study's assessment of research quality was invalid because the study used inappropriate criteria for assessing research quality in the field of behavioral research. It did not heed the advice of reviewers and consultants to use correct criteria. It also omitted research in several important health areas such as tobacco and alcohol use, adolescents at risk for heart disease, and mortality. Moreover, it left out almost 100 studies from its review, although these had been provided to the reviewers in advance.

Conclusion of the Report. The report concluded that meditation research was of poor quality, not that meditation does not work, as was the interpretation given by much press coverage. The paper on the AHRQ report published in the Journal of Alternative and Complementary Medicine concluded: “Most clinical trials on meditation practices are generally characterized by poor methodological quality with significant threats to validity in every major quality domain assessed. Despite a statistically significant improvement in the methodological quality over time, it is imperative that future trials on meditation be rigorous in design, execution, analysis, and reporting of the results” (Ospina, et al., 2008).

Critique of the Report. The editors of the Journal of Alternative and Complementary Medicine invited me to review and to write a commentary on the AHRQ report (Orme-Johnson, 2008). Here are the main points of my commentary.

I. Wrong Criteria of  Meditation Research Quality Were Used. The AHRQ used a wrong standard for assessing research quality. The standard they used was double blinding, which means that neither the person learning the technique nor the teacher of the technique know which technique the subject is given. Double blinding is appropriate for drug studies, in which the identity of the drug or the placebo are easily disguised (the pills look identical). But it is impossible for the teachers of the Transcendental Meditation technique, Yoga, Tai Chi, or Qi Gong, etc. not to know what they are teaching and in most cases is it not possible for the student not to know.

II. Correct Criteria of Meditation Resarch Quality Were Not Used. The AHRQ report did not use any of the correct standards for assessing meditation research. The purpose of double blinding, as used in drug research, is to control for things like expectation, social influence, and bias. There are other ways besides double blinding that control for these factors and the AHRQ report did not score for any of them. Below is a list of factors that need to be controlled in behavioral research in general and in meditation research specifically, with examples of how the research on the Transcendental Meditation technique has controlled for them.

Controls for non-specific therapeutic attention—treatment and control groups should be made equivalent on:

1. Contact time with the teacher; there are TM studies in which the control subjects and TM subjects get the same amount of time with the teachers. Example of controls are health education classes or progressive muscle relaxation (PMR) used in blood pressure studies.
2.  Teaching environments; TM studies have made sure that controls are taught in the same environments as TM.
3. Schedules of teaching and follow-up; many TM studies have matched the schedules of controls to the TM schedule.
4. Schedule and duration of practice of the treatment and control procedures; controls are given the same schedule of home practice as the TM program, such as 20 minutes twice a day of progressive muscle relaxation.
5. Blinding of data collection personnel and the attending physicians (this also controls detection bias, below). In many TM studies the personnel collecting the data, such as measuring blood pressure, and the doctors attending the patients do not know which group the subjects are in.

Controls for expectation--treatment and control groups should be made equivalent on:

1. Scientific evidence for effectiveness in treating the condition under study; in many TM studies there is a body of real evidence that the control condition is an effective treatment for the condition being studied, like diet and exercise or PMR for reducing blood pressure.
2. Qualified teachers; TM studies have used teachers who are highly qualified and committed to their treatment (E.G., trained teachers of diet and exercise programs or PMR)
3. Use of equivalent payment of fees, which may be no fees, or equivalent reimbursement for all subjects. In grant-funded TM studies all subjects in all groups are reimbursed equally for their participation.

Controls for detection bias:

1. Blinding of data collection personnel and of the attending physicians as which group the subjects are in.
   
2. Open collaboration with researchers at universities and medical facilities independent of any organization associated with teaching the meditation technique. This has been done in most of the research on TM and health in the last couple of decades.

III. In the AHRQ Report Well-Designed Studies Were Not Properly Identified. The AHRQ report did not distinguish between meditation studies which used proper behavioral controls from those that did not. The National Institutes of Health research grant are highly competitive (only about 20% are funded), and they are peer reviewed by the leading scientists in the nation in the areas of specialization of the grant application. The National Institutes of Health have awarded over $24 million in funds for research on the Transcendental Meditation technique, indicating that these studies have been well designed. http://www.doctorsontm.com/national-institutes-of-health  Moreover, the TM research has been published in peer-reviewed professional journals in the field.

IV. The AHRQ report did not score on many other factors that are necessary for good meditation research. These include the use of qualified teachers, compliance of the subjects with the meditation schedule, checks that the meditators are doing the practice correctly, and use of valid measurement methods (e.g., measure blood pressure several times, not just once, to get a reliable measure).

V. A study could get a perfect score by the AHRQ report criteria, and if it failed on any one of these factors, the study would be totally invalid. For example, if the teachers were unqualified or the subjects did not learn the technique properly, or if they did not practice it regularly, then the study would be invalid, yet it could have a perfect score by the AHRQ report criteria.

VI. The AHRQ report did not use a proper peer view process. The essence of science is check and balances of the scientific process by feedback from other experts in the field. The AHRQ report did employ experts, and some of them pointed out many of the limitations outlined above, yet the AHRQ study authors ignored this feedback, which is a serious breach of the scientific process. When a reviewer (Dr. Ken Walton) inquired how the AHRQ resolved questions raised by the reviewers, they merely replied that they had their own policies and procedures. It would have increased the transparency of the report to know what AHRQ policies and procedures were used to resolve issues raised by reviewers, and what the justification was for not adhering to standard peer-review practices.

VII. The AHRQ report excluded studies on the effects of meditation techniques on tobacco, alcohol, and substance use, which are highly relevant for health. There are 19 on the TM technique (Alexander et al., 1994).

VIII. The AHRQ report excluded studies the effects of meditation on adolescents, notably on those at risk for heart disease, a highly important area for prevention.

IX. The AHRQ excluded well-controlled studies on mortality, the life extending effects of meditation, the bottom line of good health.

X. The AHRQ report excluded 98 research papers on meditation, including all of Collected Papers, volume 5, even though these papers were made available to the reviewers.

XI. The AHRQ report confounds meditation effects with exercise effects. It compared techniques in which the person sits down with eyes closed and does a mental technique, like the Transcendental Meditation technique and Mindfulness meditation with techniques that have a prominent exercise component, like Tai Chi, and Qi Gong.

XII. Subsequent Meta-analyses, which have corrected for deficiencies in the AHRQ report, have found that the TM technique is effective in reducing blood pressure. A meta-analysis of nine randomized controlled trials found the TM program lowered blood pressure on average by –4.7/–3.2 mm Hg compared with controls (Anderson, Liu, & Kryscio, 2008). Subjects ranged in age from adolescents to seniors, and included normotensive, pre-hypertensive and hypertensive subjects. Subgroup analyses of four hypertensive groups and three high-quality studies showed similar reductions.

Another meta-analysis of the effects of stress reduction programs on hypertension patients evaluated high-quality studies that used active controls, adequate baseline measurement, and blinded BP assessment. Meta-analysis was used to calculate BP changes. The results for blood pressure decreases with biofeedback, relaxation-assisted biofeedback, progressive muscle relaxation, and stress management training, were not statistically significant. However, the TM program was significant, –5.0/–2.8 mm Hg (Rainforth et al., 2007).

References
Alexander CN, Robinson P, Rainforth MV. Treating and preventing alcohol, nicotine, and drug abuse through Transcendental Meditation: a review and statistical meta-analysis. Alcoholism Treatment Quarterly 1994 11(1/2):13-87

Anderson JW, Liu C, Kryscio RJ. Blood pressure response to Transcendental Meditation: a meta-analysis. American Journal of Hypertension 2008 21(3):310-316

Ospina MB, Bond K, Karkhaneh M, et al. Trials of meditation practices in healthcare: An overview of their characteristics and quality. The Journal of Alternative and Complementary Medicine 2008; 14:1199-1213

Orme-Johnson DW. Commentary on the AHRQ report on research on meditation practices in health. The Journal of Alternative and Complementary Medicine 2008 14(10):1215-1221

Rainforth MV, Schneider RH, Nidich SI, Gaylord-King C, Salerno JW, Anderson JW. Stress reduction programs in patients with elevated blood pressure: a systematic review and meta-analysis. Current Hypertension Reports 2007 9(6):520-528

MISLEADING PRESS COVERAGE

Press coverage of the report, which appeared in HealthDay and Yahoo News, interpreted the report as showing that meditation "won't boost health." Lead author Maria Ospina commented to reviewer Dr. Ken Walton that it is unfortunate that some news media attempt to make their stories more interesting by exaggerating or even misrepresenting the results the authors have presented to them. She said that their report does not support the conclusion that meditation doesn't improve health, nor does it say that 'there's no evidence that meditation eases health problems.' She said they were careful to point out that some meditation practices do show apparent benefits, such as lowering blood pressure and reducing stress.

REVIEWERS FIND THE REPORT IS METHODOLOGICALLY FLAWED, INCOMPLETE, AND RECOMMEND THAT IT BE RETRACTED

The reviewers who found major flaws in the study were:

1. Dr. Harald Walach, Professor of Psychology of the University of Northampton and School of Social Sciences and the Samueli Institute for Information Biology in England, Editor in Chief of Spirituality and Health International, Editor in Chief, Forschende Komplementärmedizin/ Research in Complementary Medicine, member of the "Peer-Review Board" of the study;
2. Dr. Kenneth G. Walton, Research Associate Professor, Institute for Natural Medicine and Prevention, Maharishi University of Management, Fairfield Iowa, member of the "Peer-Review Board" of the study;
3. Dr. Vernon Barnes, Assistant Professor, Georgia Institute for Prevention of Human Diseases and Accidents, Department of Pediatrics, Medical College of Georgia, Augusta Georgia, member of the "Technical Expert Panel" of the study.

Click here for full text of Professor Walach's review.

Click here for Dr. Walton's review (PDF)

(For press release from these and other top researchers, see: Top Researchers Criticize New Meditation and Health Study as "Flawed, Incomplete, and Incorrect")

Some additional points from the reviewers not covered above.

Inadequate reporting. Reviewers of this report on meditation found that the authors of the study did not adequately describe the underlying assumptions and computational methods of the statistical analyses used, so that it was not possible to tell exactly how the report arrived at its results and conclusion.

Transcribing errors. Reviewers found basic errors in transcribing data, and some studies were counted twice.

Classification errors. Other studies were misclassified. For example, studies on pre-hypertensive subjects were included in the section on hypertension, a study was classified as Zen Buddhist meditation when in fact is was mainly progressive relaxation, and another study was classified as Transcendental Meditation when it was a newly invented meditation.

Mixed up of single and complex treatments. The report also has a fundamental problem in trying to compare single treatment meditation programs with complex or composite treatment programs, especially those that include a predominant exercise modality (which has by itself been shown to reduce BP, for example).

Methodological scoring errors. Moreover, reviewers found that another important aspect of research design—the method of random allocation of subjects to treatment—was incorrectly scored in some instances. Thus, some of the results and conclusions of the report were simply based on mistaken data entry.

Different results and conclusions when errors were corrected. Rainforth, Schneider, et al. (2007) found that in the case of the effects of meditation techniques on blood pressure, correcting the data entry mistakes in the report and redoing the meta-analysis resulted in different conclusions. The Transcendental Meditation technique was found to have a number of well-designed studies, more than other meditation techniques. It was found to produce clinically meaningful reductions in blood pressure, a mean reduction of -4.6 mm Hg systolic and -2.6 mm Hg diastolic, which is comparable with previously reported effects of aerobic exercise. Other stress-reduction treatments were not found to have clinically meaningful effects.

Inappropriate assessment of research quality. The reviewers also pointed out that the Jadad scale used to judge research quality was completely inappropriate for meditation research, because it weighs heavily on concealing from the subjects the identity of the meditation or control technique to which they were assigned. Such "double blinding" is appropriate for pharmaceutical trials, in which the identity of the pills are easily disguised, but in meditation research such practices are deceptive, unethical, and very difficult, if not impossible, to achieve. Moreover, they are counterproductive to the therapeutic relationship.

Important dimensions of research ignored. The Jadad scale is very narrow and does not measure many key elements of research validity, such as whether the meditation program was properly implemented, whether the techniques given were appropriate for treating the health condition under study, whether subjects were given the correct schedule ("dose") of meditation, or even such a basic things as level of compliance or whether the subjects even practiced the techniques at all. Using the Jadad scale, a study could be rated as having a high level of methodologic quality, when in fact the meditation technique might not be taught properly, or the subjects might not even practice it. The Jadad scale could have rated such a study as being of high quality, but in fact the information from it would have been completely useless.

No rating of whether the results generalized to other situations. The Jadad scale does not reflect "external validity." External validity refers to whether the technique works under different conditions—such as different patient populations, age groups, or settings. For example, the Transcendental Meditation technique has been found to effectively reduce blood pressure in different age groups (adolescents at risk for hypertension, hypertensive adults, and the elderly), in patients in different categories of risk factors (e.g., patients at different levels of psychosocial stress, obesity, alcohol use, physical inactivity, and sodium use), in different ethnic groups (e.g., African American and Caucasian), and in different nationalities (e.g., US, India, and New Zealand). The report did not take any of this kind of information into account.

No rating of objectivity of the measures. The Jadad scale also does not credit the studies for using non-fakeable, objective measures, as opposed to highly subjective measures, such as most psychological tests.

No triangulation of the data. Nor does the Jadad scale reflect cross-validation by triangulating from different domains of measurement. For example, the assertion that meditation techniques can reduce anxiety would be greatly strengthened if it could be demonstrated that they reduce psychological, biochemical, and physiological, as well as behavioral/social indications of anxiety. A case in point is the Transcendental Meditation technique, which has been found to reduce anxiety as measured by psychometric tests, to decrease biochemical correlates of anxiety, such as blood lactate levels and adrenaline metabolites, to reduce autonomic markers of anxiety, such as basal and spontaneous skin conductance, to reduce behavioral/pharmacological indicators of anxiety, such as reduced use of tranquilizers and sedatives, and to reduce systolic blood pressure reactivity to social stressors (an interview about a stressful event). Triangulation of different domains of data was not taken into account by the review.

Key studies omitted. Perhaps most significantly, apparently due to unnecessary constraints in the selection process, the report left out some of the most well-designed and important research in the field. In addition, many papers that met the authors' criteria for inclusion were omitted. These included two studies published in the American Journal of Cardiology in 2005, which demonstrated that individuals with high blood pressure who were randomly assigned to Transcendental Meditation groups had a 30% lower risk for mortality than controls.

The entire Volume 5 of Scientific Research on Maharishi’s Transcendental Meditation and TM-Sidhi Program (1990), which reprinted 75 important papers, was excluded, even though it was sent to the study authors to include in their review. Other studies which met the inclusion criteria, but which were incorrectly excluded, include nine studies on substance abuse, and numerous studies on the physiological effects of the Transcendental Meditation technique, including effects on cardiovascular functioning, pulmonary functioning, skin resistance/conductance, blood gas measurements, adrenocortical functioning, lipoprotein levels, EMG, carbohydrate metabolism, brain electrophysiological (EEG, EP. CNV), and numerous papers on cognitive/neuropsychological effects of meditation (1)attention, 2) memory, 3) perception, 4) other cognitive measures such as cognitive functioning, 5) reasoning, 6) sensorimotor functioning, 7) language, 8) creativity, 9) intelligence, and 10) spatial abilities). The report also incorrectly excluded dozens of studies on the effects of meditation on affective outcomes and mental health, such as depression, anxiety, and major meta-analyses on these outcomes (see Walton review for details).

Studies on children omitted. Reviewers also questioned the wisdom of the report sponsors' decision to include only research on adults over 18 years of age. Excluding research on children is not only a violation of NIH policy*, it goes against the national interest in early detection and prevention of health problems through behavioral interventions. If teaching children and teenagers to practice a meditation technique could prevent such problems as cardiovascular disease, drug abuse, and academic failure, that would be an ideal solution. Good research exists in these areas. For example, randomized controlled studies and other research designs have shown that the Transcendental Meditation technique reduces blood pressure and cardiovascular reactivity to stressors in pre-hypertensive high school students, and reduces drug, tobacco, and alcohol use, and increases field independence, intelligence, and creativity in adolescents.

*According to the NIH (NIH policy and guidelines on the inclusion of children as participants in research involving human subjects, March 6, 1998) "It is the policy of NIH that children (i.e., individuals under the age of 21) must be included in all human subjects research conducted or supported by the NIH, unless there are scientific and ethical reasons not to include them.... This policy applies to all NIH conducted or supported research involving human subjects...."

No constructive criticism. The report concludes that meditation studies are mostly of low quality yet provides little constructive guidance regarding what constitutes good methodology in the field of meditation research. It is disappointing that after a very extensive review of the meditation literature that the report only focuses on the problems in the field, only applies standards developed for pharmaceutical trials which it acknowledges are inappropriate for meditation research, and does not instead highlight progress that has been made in attempting to address the particular challenges of meditation research, such as studies that provide models of how to control for subject expectancy and attention from trainers. On the plus side, the finding that these "meditation techniques," or perhaps more accurately, "psychophysiological techniques," are each distinct in many ways from the other, with few or no characteristics that occur universally across the techniques, is a significant contribution and could be given greater emphasis.

THE SIGNIFICANCE OF PEER REVIEW:
EXCERPT OF LETTER FROM REVIEWER DR. KEN WALTON TO THE STUDY AUTHORS, JULY 15, 2007

"If either of you has ever applied for a competitive grant from the U.S. National Institutes of Health, you know the serious meaning of peer review. Our peers who review and score these grant applications, and thus are the main determinant of the awards in a field where the odds run from about one in four to as low as about one in ten, go over every part of the 100 pages or more of the minutely detailed application. These peers are experts in their fields and are intimately familiar with what "high quality research" means. Many of them have been in their fields for over 30 years, and are finely attuned to all the nuances of research quality. Likewise, the journals in which our best publications from these grant-supported studies have appeared are specialty journals, like Hypertension, in which the peer reviewers again are highly experienced in their fields. The studies published in these journals are reviewed as thoroughly by the top experts as are the grant applications that secured the funding in the first place. Obviously, not every study on the Transcendental Meditation technique is funded through this mechanism and not every one is published in the top specialty journals. This is one reason for the considerable range of research quality among the published papers on this technique of meditation and perhaps on other techniques as well. The point I make here, however, is that for you to make the strong claims you have made concerning poor research quality of the majority of studies and not to also point out that some of the papers are of the highest quality available and that these highest quality studies tend to show significant benefits of the technique is not a position that can be justified by your systematic approach. It is an injustice to lump the outcomes from high quality papers with those from low quality papers in any overall comparison or in any overall statement such as the ones you have used in your summary.

LETTER TO AHRQ, NCCAM, AND THE STUDY AUTHORS:
EXCERPTS FROM REVIEWERS DR. KG WALTON, DR. H. WALACH, AND DR. V. BARNES, V. (JULY 17, 2007)

"The purpose of this letter is to bring attention to problems and failures that render many of the report's conclusions questionable or overtly incorrect and to offer suggested remedies. Two of us (Walton and Barnes) are long-time acquaintances who discussed some of these points during the reviewing process. Dr. Walach is a recent acquaintance with whom Walton and Barnes have communicated only after the report was published. Because the three of us identified similar issues that were dealt with inadequately or not at all in the final report, we now feel obliged to address the report's sponsoring institutions and authors to ask for appropriate corrections.

"In the Preface to the report on page iii, AHRQ details its expectation that their "evidence reports and technology assessments will inform individual health plans, providers, and purchasers as well as the healthcare system as a whole by providing important information to help improve healthcare quality." The criticisms and solutions detailed in this letter are intended to help AHRQ realize these goals not only for this specific report but for future reports.

"The first and most central problem encountered, which may well be common to other AHRQ reports, is the spurious claim that the report was "peer reviewed." The absence of a normal peer review is probably the most obvious source of erroneous information in this report. And it was accomplished in a manner inappropriate to scientific discourse. The initial reference to peer review occurs at the end of the second paragraph of the Preface, on page iii of the front matter. In fact, the members of both the "review board" and the "technical expert panel" were led to believe, when being invited to participate, that the errors they found in their review of the draft report would be addressed in the standard manner of peer reviews, namely, that their comments and suggestions would be responded to by the authors in a document that the reviewers could either sign off on, or contest and work out to mutual satisfaction, prior to the final report. This is the norm in peer reviewed scientific articles, and it is a crucial step in carrying out good science. In actuality, however, the only opportunity to evaluate the degree to which our critiques were addressed was to examine the final report after it was published. As a result, many errors noted by the three of us, and presumably additional errors found by the other 15 members of the review and technical expert panels, were not corrected in the final report....

"First is the evaluation of methodological strength. Each of us objected in our reviews to the use of scores on the Jadad scale as the main measure of methodological strength of randomized controlled trials (RCTs). This scale is particularly unsuited for the evaluation of behavioral approaches such as meditation techniques and psychotherapy, where blinding of the subjects to the identity of the treatment they are receiving, if at all possible, is certainly unethical and is likely to interfere with treatment effectiveness. The Jadad scale is narrow, asking 5 questions relevant to internal validity and no questions pertinent to external validity. Some consensus exists, both within the behavioral medicine field and in the clinical trials literature, that the Jadad scale is inadequate even for the evaluation of internal validity....

"Thus, in both the choice of approach for evaluating methodological strength and in the application of the chosen, weak approach, the report is so seriously flawed that its conclusions could be diametrically opposed to the truth....

"Flaws also were serious in the actual performance of the quantitative meta-analysis of this case (the effect of Transcendental Meditation technique versus health education on blood pressure). For example, each of us detected that the 5 RCTs in the analysis were not all appropriate for inclusion.

"However, not only did the report's authors fail to contact the lead authors of the studies, they also failed to correct this overlap problem after each of us had included this information in our reviews.... Furthermore, if another error noted by the reviewers, namely, the use of incorrect means and standard deviations in one line of data shown in Figures 3 and 4, had been corrected by the authors as the reviewers asked, that alone would have changed the outcome of the analysis.

"This one example, therefore, contains several problems serious enough to produce erroneous results for both the evaluation of methodological quality and the quantitative evaluation of effects of the treatment. Unfortunately, every meta-analysis in the report that we examined in detail has similar problems, and there are other problems with the report than we cannot cover in this short letter."