Arthritis & Rheumatism, Volume 63,
November 2011 Abstract Supplement

Abstracts of the American College of
Rheumatology/Association of Rheumatology Health Professionals
Annual Scientific Meeting
Chicago, Illinois November 4-9, 2011.

Development and Usability Testing of ANSWER: A Web-Based Methotrexate Decision Aid for Patients with Rheumatoid Arthritis.

Li1,  Linda C., Adam2,  Paul M., Townsend3,  Anne F., Lacaille3,  Diane, Yousefi3,  Charlene, Turnau1,  Shawn, Stacey4,  Dawn

University of British Columbia, Vancouver, BC
Mary Pack Arthritis Centre, Vancouver, BC
Arthritis Research Centre of Canada, Vancouver, BC
University of Ottawa, Ottawa, ON
Universtiy of Ottawa, Ottawa, ON
Institute of Population Hlth, Ottawa, ON
Vancouver & Ottawa


Decision aids are evidence-based tools designed to inform people of the potential benefits and harm of treatment options, clarify their preferences, and provide a structure for discussion at a clinic visit. For patients with rheumatoid arthritis (RA) who are considering methotrexate, we have developed an interactive web-based decision aid called the ANSWER. The current study aims to: 1) assess the usability of the ANSWER prototype; and 2) identify important components of usability testing from the patient's perspective.


Development of the ANSWER was consistent with the International Patient Decision Aid Standards.1 This decision aid consists of: 1) six animated patient stories and narrated information on the evidence of methotrexate for RA; and 2) interactive questionnaires to clarify patients' treatment preferences. Eligible participants for the usability test were patients with RA who had used methotrexate. We used the concurrent think-aloud method, whereby participants were asked to verbalise their thoughts while using the ANSWER. Sessions were audiotaped and field notes taken. Participants completed the System Usability Scale (SUS) 2;3 to assess the overall usability of the decision aid (range=0–100; higher=more user friendly). We conducted content analysis to identify major themes to understand the user experience.


15 patients participated in the usability testing; the majority were age 50 or over and university/college graduates (n=8, 53.4%). The median disease duration was 5 years (IRQ=0.83–10). Participants took an average of 56.1 minutes (SD=34.8) to complete the program. The mean SUS score was 81.2 (SD=13.5), indicating high usability. Content analysis of audiotapes and field notes revealed three themes that participants focused on while testing the ANSWER: 1) user engagement (i.e., the relevance of the decision aid design and content to the user); 2) information quality (i.e., clarity and credibility of information); and 3) user-tool interaction (i.e., consistency of the design; ease of use). Across these themes, participants commented extensively on the overall integration of the content and navigation (e.g., challenges of moving from one task to the next). We made revisions to the prototype based on the findings.


Although the SUS score indicated high usability, findings from the think-aloud sessions highlighted additional areas where further modifications were needed for the online ANSWER decision aid. Our results highlight the importance of direct observation methods in usability testing. With an increasing number of online and mobile programs being developed to improve arthritis knowledge and care, further research to advance the methodology of usability testing is warranted.

(1)Elwyn, G et al. BMJ 2006; 333(7565):417.

(2)Brooke, J. In: Jordan, PW, Thomas, B, Weerdmeester, BA & McClelland, IL, editors. Usability Evaluation in Industry. London. 1996. 189–194.

(3)Bangor, A et al. International Journal of Human-Computer Interaction 2008; 24(6):574–594.

To cite this abstract, please use the following information:
Li, Linda C., Adam, Paul M., Townsend, Anne F., Lacaille, Diane, Yousefi, Charlene, Turnau, Shawn, et al; Development and Usability Testing of ANSWER: A Web-Based Methotrexate Decision Aid for Patients with Rheumatoid Arthritis. [abstract]. Arthritis Rheum 2011;63 Suppl 10 :1669

Abstract Supplement

Meeting Menu