|
Assessing the User Defined Validity of Components of Outcome Measures in Acute Psychiatry
Executive Summary
Project Reference: K/RED/18/35
Grantholders
Dr Robert Hunter, Consultant Psychiatrist and Director of Research and Development, Greater Glasgow Primary Care NHS Trust.
Mr Dave Peck, Area Psychologist, Highland Primary Care NHS Trust.
Dr Ian Pullen, Consultant Psychiatrist, Borders Primary Care NHS Trust.
Research Workers
Joanne McLean, Co-ordinator
Claire Quinn, Andy Greenfield, Bill McArthur, Research Assistants
Background
There is general agreement that the introduction of reliable and valid outcome indicators is an important prerequisite for the development of effective adult psychiatric services (Thornicroft and Tansella, 1996). The CRAG report on outcome measures (1996), recognised that any useful outcome measure would require reliability, validity, sensitivity to changes in health that are clinically important or meaningful to patients, and have utility in clinical practice. The Scottish Framework for Mental Health Services (1997) emphasised that outcome measures need to incorporate the perspective of people who use services and mental health professionals. In order to examine the validity of this approach, this study was commissioned by the Chief Scientist Office. The results of this study are timely: in England, a Department of Health Working Group on Health Outcome Indicators in Mental Health (1999) has recommended the introduction of HoNOS, a clinician rated outcome measure, as part of the proposed ‘Mental Health Minimum Data Set’. In Scotland, there has been considerable interest and support for outcome measures that reflect the views of service users, and the Outcomes Subgroup of the Mental Health Reference Group is due to make recommendations in this regard in May 2000.
Method
A random sample of 700 people who were using mental health services in four different regions of Scotland (Glasgow, Highland, Borders and Lothian), agreed to take part in the study. The validity of three different outcome measures was examined from the perspective of the service user. The measures used were the Health of the Nation Outcome Scale (HoNOS), devised and rated by clinicians, the Avon Mental Health Measure (AVON), which was designed for self-report by service users, and a pilot version of an individual goal setting indicator, originally described by CRAG in 1996, which was named Outcomes for People Using Services (OPUS). Keyworkers and service users were asked to complete each measure, and service users also gave their opinion on the relevance of HoNOS items. Demographic and socio-economic data were collected for all participants. After approximately 3 months, repeat assessment was carried out in a sub-sample of about 100 people, in order to assess the sensitivity to change of the three measures.
Results
There was little agreement between the HoNOS and AVON measures from the service user perspective. Needs or problems were more likely to be identified by AVON than HoNOS. Service users and keyworkers identified additional domains not covered by HoNOS or AVON such as stress, work, caring responsibilities, sexuality and medication issues. There was fair agreement between keyworker HoNOS ratings and service users’ views on the relevance of HoNOS items, although keyworkers were more likely to identify needs or problems using HoNOS than the service user. Comparison of AVON and OPUS proved difficult to analyse, but little association was found between them. All three measures demonstrated sensitivity to change and there was an overall reduction in scores for each measure. No significant difference was detected between subject and keyworker AVON total scores. Of the three measures studied, both subjects and keyworkers preferred AVON and nearly all felt that both service user and keyworker views were necessary to assess treatment outcomes.
Comment
According to the criteria set out in the CRAG report (1996), AVON appears a more useful outcome measure, than HoNOS or OPUS. This study provides strong support for the user-defined validity of AVON. There was consensus among clinicians and service users that both professional and client perspectives are needed in measuring outcomes, and that AVON was the preferred instrument. The findings also support the view that pragmatic outcome measures, that take account of service users’ views, could have considerable utility in general psychiatric services in Scotland.
Acknowledgements
The research team would like to thank the following people for their help with this study:
Firstly, this study would not have been possible without the help and co-operation of the 700 people who agreed to participate in the four study areas.
All of the keyworkers in Borders, Highland, Lothian and Glasgow who collected data for the project with great care and accuracy, and their service managers.
Alison Cox of Changing Minds, Bristol, for supporting the use of the Avon Mental Health Measure in this study.
The Chief Scientist Office, Edinburgh.
Dr Angus MacKay, Physician Superintendent and Clinical Director, Lomond and Argyll Primary Care NHS Trust and Chairman of the Scottish Health Technology Assessment Centre.
Glasgow Association for Mental Health.
Highland Users Group, Inverness
Royal College of Psychiatrists Research Unit, London
Robertson Centre for Biostatistics, University of Glasgow.
Audio-Visual Department for the use of teleconferencing facilities and the Nursing Research Initiative for Scotland, Glasgow Caledonian University.
Information and Statistics Division, Edinburgh.
Lastly, but not least, Brian Rae and staff at the Research and Development Directorate, Greater Glasgow Primary Care NHS Trust, Gartnavel Royal Hospital, Glasgow, for their expert support throughout.
| 