Developing core outcomes sets: methods for identifying and including patient-reported outcomes (PROs)

Outcome selection and reporting in randomised controlled trials (RCTs) is often problematic. Heterogeneity in outcomes measured across studies in the same disease or treatment may hamper effective evidence synthesis. A systematic review of oesophageal studies, for example, found 10 different measures for postoperative mortality which were often undefined [1]. In addition, selective reporting of outcomes puts trials at risk of outcome reporting bias and can mean treatment effects are exaggerated [2]. These issues may be further complicated for patient reported outcomes (PROs). PROs are typically assessed using questionnaires (patient reported outcome measures (PROMs)) and many validated questionnaires are available because PROMs have been developed by different groups and disciplines (for example, clinical versus psychological) or for differing purposes (for example, measurement of health in generic populations versus disease-specific patient groups). A single PROM can be made up of numerous scales and single items and generic and disease specific PROMs are often combined to assess a range of relevant health domains within an RCT. This means that different (and often ill-defined) outcomes may be reported and the multiplicity of items and scales may also allow selection of statistically significant rather than pre-determined a priori PRO endpoints to be reported, increasing the risk of outcome reporting bias. Problems are further accentuated for PROs because terminology of the scales and items across PROMs is not universally agreed meaning data synthesis across studies is difficult when different questionnaires are used, and while there is overlap in the issues that are measured there is also variation because PROMs have been developed by different methods and for different purposes. Potential solutions to these challenges are to develop and use core outcome sets.

Core outcome sets (COSs) are an agreed minimum set of outcome domains to be measured and reported in all trials of a particular treatment or condition [3]. The routine measurement of COSs has the potential to facilitate data synthesis and reduce outcome reporting bias by standardising the outcomes that are measured across studies and this has been emphasised by the COMET (Core Outcome Measures in Effectiveness Trials) initiative which supports the development and application of COSs for pragmatic (effectiveness) trials [4]. Pragmatic trials are designed to assess whether an intervention is effective for routine clinical practice and outcomes, therefore, need to be relevant and important to patients as well as clinicians and other key decision-makers [5]. In many cases these are the outcomes that are assessed with PROMs, particularly if the questionnaire has been developed with patient input [6] but the availability of so many different PROMs, however, means there are problems with selecting which of the measured health domains are ‘core’. The aim of this study, therefore, was to explore and report methods to identify PRO domains from the wealth of available PROMs and to use this approach to inform the development of a COS to use in pragmatic trials in a specific condition. Consensus on which outcomes to include in the final core set, and the methods to achieve this, are the focus of further research.

This study was undertaken within one disease site and treatment - radical treatment for oesophageal cancer, selected because the research team have clinical and PROM expertise in this area and have previously tried to summarise PRO evidence [7‐9]. There were three phases of work: (1) a systematic literature review to identify validated PROMs used in oesophageal cancer studies and the scope of these instruments; (2) a detailed content analysis to explore PROM diversity; and (3) categorisation of PROM content into health domains (Figure 1).

This study comprehensively analysed PROs from studies in radical treatment for oesophageal cancer. Some 116 scales and 32 single items were identified from 21 validated PROMs. As many as 94 different verbatim names were used to describe PRO scales and single items and although many names were similar, content examination revealed component questions did not always address comparable issues. In-depth examination and categorisation of PROM contents concluded that together they addressed 32 different health domains demonstrating the vast overlap between PROMs.

Our findings show how evidence synthesis of oesophageal cancer PROs may be hampered because of the range of PROMs used in trials and the multiple scales and single items within them, often with inconsistent and non-transparent terminology. Core outcome sets aim to reduce this problem by identifying and prioritising the important health domains to be measured in all studies. The development of core outcome sets in other clinical areas has been undertaken using a range of methods, in particular the approach to including PROs [86‐89]. In rheumatoid arthritis, for example, the initial American College of Rheumatology (ACR) core set was developed by a committee of experts (16 professionals in rheumatoid arthritis trials, health services research and biostatistics) who reviewed the literature on the validity of trial outcomes (for example, sensitivity to change or how well it predicted/correlated with a definite clinical change) and used a nominal group process to recommend and reach consensus on a list of core outcomes. The list was presented and finalised at a specialist international conference (OMERACT: Outcome Measures for Rheumatology in Clinical Trials) and contained both clinical and PROs, although patients were not involved in the consensus process. Outcomes were specific (for example, number of swollen joints) or more general domains (for example, functional status), with recommendations on how to measure the outcomes decided later [86]. Subsequent OMERACT conference discussions and workshops deliberately involved patients and led to the addition of fatigue in the ACR core set [87, 90], and continued work using interviews with patients, identified further important PROs [5]. This led to the development of a ‘patient core set’ of disease-specific and global outcome domains solely derived from patient opinion to complement the professional ACR core set [88]. Our current study methodology ensures that the patient perspective and relevant PROs inform the development of a core set of outcome domains from an early stage, because it examines the content of validated PROMs which are developed with significant patient involvement. The identified PRO domains will be prioritised using a Delphi method to reach consensus on the important to include in the core outcome set, alongside clinical outcomes [1], and is the focus of future work. Patients, surgeons and clinical nurse specialists will be surveyed to ensure the opinions of all key stakeholders are sought, a recommended approach by the COMET (Core Outcome Measures in Effectiveness Trials) initiative [4].

This study included a detailed systematic search to identify PROs measured in oesophageal cancer studies and used rigourous methodology to identify health domains, however, it does have weaknesses. The categorisation of question items into health domains was performed by two experts and independently checked by other members of the research team, including a patient advocate, but it is possible that others may have categorised items differently. Inter-rater reliability statistics could have been recorded to describe agreement between the experts when categorising items. Future work therefore is needed to standardised and validate this method. In addition, presentation of the methodology to a greater number of patients or patient representatives could strengthen the robustness and reliability of the categorisation process.

In summary, this study demonstrates there is diversity in the PROMs selected to evaluate radical treatment for oesophageal cancer. Within and between PROMs there is a lack of clarity between named scales and items and the underlying health domains being assessed meaning data synthesis is limited. A methodology for identifying important PRO health domains is proposed which can be used to inform the development of a core set of health domains. Following this it will be necessary to determine accurate and efficient ways to measure these core domains, drawing on items banks developed by initiatives such as PROMIS (Patient Reported Outcomes Measurement Information System) and COSMIN (Consensus-based Standards for the selection of health status Measurement Instruments) [91, 92].