Arthritis & Rheumatism, Volume 60,
October 2009 Abstract Supplement

The 2009 ACR/ARHP Annual Scientific Meeting
Philadelphia October 16-21, 2009.

WCWL Rheumatology Priority Referral Score Reliability and Validity Testing

Fitzgerald1,  Avril, Spady1,  Barbara Conner, DeCoster1,  Carolyn, Naden2,  Ray, Hawker3,  Gillian A., Noseworthy1,  Thomas, the WCWL Investigators,  

University of Calgary, Calgary, AB
New Zealand Ministry of Health, Auckland, New Zealand
Women's College Hospital, Toronto, ON


Rheumatology consultation demand exceeds supply of rheumatologists in many countries. Western Canada Waiting List (WCWL) has developed a diagnosis independent, priority-referral score (PRS) for rating relative urgency for referral to rheumatologists by primary care providers (PCPs). This paper reports the reliability and validity of the PRS.


To develop WCWL PRS, a clinical panel of 5 rheumatologists and 5 PCPs, using anonymous case scenarios, engaged in a deliberative process leading to development of criteria and levels. Relative weights were generated using discrete choice experiment software (1000 Minds ®). Reliability and validity of PRS were tested independently with another 14 rheumatologists and 10 PCPs. These physicians clinically ranked, and then PRS scored, 16 case scenarios representative of the range of case-mix and urgency of referrals from PCPs to rheumatologists. Participants also rated relative urgency using visual analogue scale (VAS) and maximum acceptable waiting time (MAWT) for each case. Retesting was performed 6–12 weeks later, using altered case identifiers.


The inter- and intra-rater reliability coefficients (ICC) for PRS and clinical ranking are shown for rheumatologists, and inter-rater reliability coefficients for PCPs (retesting in progress).

Inter-rater Correlation Coefficients (ICC) for Urgency Scores

Time 1Time 2Time 1 
Clinical Rank0.810.780.60

Intra-rater Correlation Coefficients (ICC) for Urgency Scores


Three emergent cases were correctly identified by 14, 12 and 10 rheumatolgists initially, and 14, 13 and 13 on retesting. These emergent cases were correctly identified by 10, 8 and 7 of the 10 PCPs.


Average correlation between rheumatologists' clinical rankings and PRS-derived rankings is moderately strong at 0.71. Inter-rater and intra-rater ICCs for clinical ranking and PRS for rheumatologists are strong and consistent over six weeks. PCPs demonstrate higher inter-rater reliability with PRS than clinical ranking. Using PRS, PCPs have similarly high inter-rater reliability as rheumatologists. Emergent cases are identified by rheumatologists correctly, or score high on urgency. These results show acceptable performance of the PRS, albeit further testing of specific case mix groups is required. Implementation and pilot testing is in progress.

To cite this abstract, please use the following information:
Fitzgerald, Avril, Spady, Barbara Conner, DeCoster, Carolyn, Naden, Ray, Hawker, Gillian A., Noseworthy, Thomas, et al; WCWL Rheumatology Priority Referral Score Reliability and Validity Testing [abstract]. Arthritis Rheum 2009;60 Suppl 10 :54
DOI: 10.1002/art.25137

Abstract Supplement

Meeting Menu