September 2008: Designing and implementing the evaluation of the WSD pilots

Tim Ellis

Welcome to the first monthly update from the WSD Programme. Here, we examine how the evaluation of the three WSD Pilots - the ‘demonstrators’ - was designed and established and some of the challenges we have faced.

The three demonstrators were established to provide a better understanding of the level of benefit associated with telecare and telehealth, as well as to ‘fast-track’ change, identify implementation barriers and provide solutions for others to follow. The evaluation follows five themes:

  1. Service use
    Does the introduction of telehealth and/or telecare result in reduced use and costs of care (in line with the costs of the intervention)? If so, over what time period and in which participants?
  2. Participant-reported outcomes and clinical effectiveness
    Does the introduction of telehealth and/or telecare result in:
    • improvements in clinical outcomes, participant-reported quality of life, and psychological well being
    • reductions in carer burden and social isolation
    • increases in self-care behaviours, self-efficacy and empowerment
    • positive attitudes to and sustained use of telehealth/telecare?
  3. Cost and cost-effectiveness
    What are the economic consequences of introducing telecare and/or telehealth? How do these costs relate to other outcomes (in terms of clinical, quality of life and social care need measures)?
  4. Qualitative study
    What are the experiences of service users, carers and health and social care professionals using telehealth/telecare and their attitudes towards sustained use?
  5. Organisational Study
    What are the organisational factors that facilitate or impede the sustainable adoption and integration of telehealth/telecare?

The evaluation is based on a large randomised control trial - we believe the largest of telehealth and telecare anywhere in the world. The design seeks to examine 6,000 service users (2,000 in each demonstrator) over two years in a ’real-time’ study comparing ‘intervention groups’ with ‘control groups’ to assess the impact of Advanced Assistive Technology (AAT) in addition to ‘usual care’.

Establishing the trial has been a complex and challenging task. With an evaluation as large as this, there are always tensions between wanting to roll out innovative new services quickly while having to ‘hold back’ to ensure that all three sites are following the agreed evaluation methodology in a consistent manner. Planning for the roll-out of new technologies in each demonstrator site has therefore taken place alongside the development of the evaluation methodology.

The demonstrator sites, the numerous evaluators, and the Department of Health have learned a lot along the way. We believe, however, that the design we now have - approved by ethics and local R&D - is the most pragmatic solution. Nonetheless, as with all study designs, it uses assumptions that may be challenged by emerging evidence as the technology roll-out progresses within our demonstrator sites.

We monitor developments continuously within the three demonstrator sites and will keep you informed of progress as the Programme moves forward.

You may find the following presentation useful for reviewing the effectiveness of telecare and telehealth services in your area:

  • Evaluation of the Whole System Demonstrators – an overview of the key features (732 kb) [ppt]


Tim Ellis is the Programme Manager for the Whole System Demonstrators, Department of Health