Arthritis & Rheumatism, Volume 62,
November 2010 Abstract Supplement

Abstracts of the American College of
Rheumatology/Association of Rheumatology Health Professionals
Annual Scientific Meeting
Atlanta, Georgia November 6-11, 2010.


The Hawthorne Effect, Sponsored Trials, and the Overestimation of Treatment Effectiveness.

Wolfe1,  Frederick, Michaud2,  Kaleb D.

National Data Bank for Rheumatic Diseases, Wichita, KS
Univ of Nebraska Med Ctr, Omaha, NE

Objective:

Improvement in rheumatoid arthritis (RA) clinical trials is often greater than improvement seen in the clinic. But follow-up clinical trial studies by sponsors almost invariably show sustained improvement (durability of response). We investigated whether sponsored trial participation influenced study results to determine if the result of clinical trials are upwardly biased by the Hawthorne effect.

Methods:

We studied 264 RA patients who completed a commercially sponsored 3-month, open label, phase 4 trial of a Food and Drug Administration approved RA treatment. We evaluated changes in the Health Assessment Questionnaire disability index (HAQ) and VAS scales for pain, patient global, and fatigue during three periods: Pretreatment in the trial, on treatment at the close of the trial, and by a trial-unrelated survey 8 months after the close of the trial, but while the patients were receiving the same treatment. The study forms were the same for the trial and post-trial assessments.

Results:

The HAQ score (0–3) improved by 41.3% during the trial, but only by 16.5% when the end point was the post-trial result. Similar results for the other variables were patient global (0–10): 51.9% and 34.6%; pain (0–10) 51.7% and 39.7%; fatigue (0–10) 45.6% and 24.6%. Worsening between the trial end and the first survey assessment was: HAQ 0.29 units, pain 0.8 units, global 0.8 units, and fatigue 1.1 units. These values are documented in Table 1 and Figure 1.

Changes in Study Variables According to Study Time and Setting.

 Trial StartTrial endPost-trial follow-up
VariableMean (SD)Mean (SD)Positive ES (95% CI)Mean (SD)Negative ES (95% CI)
ACR20 (%) 63.1   
ACR50 (%) 43.4   
ACR70 (%) 20.2   
HAQ (0–3)1.21 (0.60)0.71 (0.61)0.83 (0.72, 0.95)1.01 (0.71)0.44 (0.60, 0.26)
Global (0–10)5.2 (2.3)2.5 (2.1)1.22 (1.07, 1.38)3.4 (2.5)0.38 (0.49, 0.25)
Pain (0–10)5.8 (2.3)2.8 (2.4)1.29 (1.12, 1.45)3.5 (2.7)0.30 (0.41, 0.19)
Fatigue (0–10)5.7 (2.7)3.1 (2.7)0.93 (0.78, 1.09)4.3 (2.9)0.41 (0.54, 0.27)
Trial start: pre-treatment; trial end (6 months later); Post-trial follow-up (0.8 years later).
Differences between trial end and trial follow-up are significant at p < 0.001.
Positive=clinical improvement, negative=clinical worsening.
ES=Effect Size

Conclusions:

Almost half of the improvement noted in the clinical trial HAQ score disappeared on entry to a non-sponsored follow-up study, and from 23%-44% of improvements in pain, patient global, and fatigue also disappeared. These changes can be attributed to the Hawthorne effect. The difference between the clinical trial HAQ and the community (clinic) HAQ is important because HAQ values are commonly used to map to utility scores and then in the calculation of cost effectiveness. If clinical trial results overstate HAQ scores, then the true effectiveness—the real functional status — is overstated. Based on our data, we hypothesize that the absolute values of RA outcome variables in clinical trials are upwardly biased, and that that treatment effect is less than observed. Our results suggest that 3rd party (non-sponsored) follow-up studies, removed from the trial assessment, can and should be used to estimate actual clinical improvement.

To cite this abstract, please use the following information:
Wolfe, Frederick, Michaud, Kaleb D.; The Hawthorne Effect, Sponsored Trials, and the Overestimation of Treatment Effectiveness. [abstract]. Arthritis Rheum 2010;62 Suppl 10 :1132
DOI: 10.1002/art.28899

Abstract Supplement

Meeting Menu

2010 ACR/ARHP