This is a work in progress - more my notes and queries than a proper paper, stuff will change, references will be added. I wanted most to get this out there and to get your views, your inputs and your insights. Please comment, your thoughts are valued!
My recent research with Marilyn Deegan into the value and impact of digitised collections has shown that there is a serious lack of adequate means to assess impact in this sector and thus a lack of significant evidence beyond the anecdotal, number crunching from webometrics or evaluations of outputs rather than outcomes (http://www.kdcs.kcl.ac.uk/innovation/inspiring.html). In short, we need better evidence of impact. How has the digital resource delivered a positive change in a defined group of people’s lives or life opportunities?
In this Arcadia funded research, we are addressing some fundamental questions in assessing the impact of digitised resources on changing lives or life opportunities. We plan to report a synthesis of methods and techniques to resolve these into a cohesive, achievable methodology for impact assessment of digitised resources.To assist and clarify our thinking and research goals we would like to offer our description of impact.
Our conception of Impact for this research can thus be described as:
the measurable outcomes arising from the existence of a digital resource that demonstrate a change in the life or life opportunities of the community for which the resource is intended.
There is a well-established, professional and mature field of Impact Assessment (IA) in sectors not normally associated with memory institutions methods of evaluation such as those seen in Environmental, Health, Economic or Social Impact Assessment. These provide some scope and lessons for our distinctive view of impact.
Impact Assessment (IA) is often predictive in nature. Environmental IA, in particular, focuses upon identifying the future consequences of a current or proposed action. As such it becomes a technical tool to predict the likely change created by a specific planned intervention. The European Commission’s definition of IA relates to a process that prepares evidence for political decision-makers on the advantages and disadvantages of possible policy options by assessing their potential impacts. In this latter case, impact is often thought in both political and economic terms. Clearly the most important aspect of this mode of IA is to influence and inform decision makers on future interventions and potential policy pathways.
Other IA relates to measuring the change in a person’s well being through a specific intervention. Health IA generally considers a range of evidence about the health effects of a proposal using a structured framework. This can be used to determine population health outcomes or to defend policy decisions. The UK National Health Service uses a tool called the QALY system (Quality Adjusted Life Year). This system assesses not only how much longer the treatment will allow a person to live, but also how it improves the life of that person#. The QALY is a measure of the value of health outcomes and as such is somewhat more limited than other methods used in Health IA, particularly in palliative care. King’s College London has developed the Palliative care Outcome Scale (POS)#, a tool to measure patients' physical symptoms, psychological, emotional and spiritual needs, and provision of information and support at the end of life. POS is a validated instrument that can be used in clinical care, audit, research and training. These forms of IA are effective at measuring interventions but generally need clear baselines and large comparable populations to gain significance.
Other IA methods focus upon the wealth or level of economic activity in a given geographic area or zone of influence. They may be viewed in terms of: (1) business output, (2) value added, (3) wealth (including property values), (4) personal income (including wages), or (5) jobs#. Economic IA has the benefit of usually being able to identify baselines and significant indicators to measure improvement in the economic well-being of an area. However, these measures are less satisfactory for intangible assets or for the assessment of digital domain resources. Contingent value assessments or analysis are seeking to resolve those intangible asset measures. There is also some very interesting assessment work supporting new business investment opportunities as described by the Triple Bottom Line also known as the three pillars: people, planet, profit. An impact investor seeks to enhance social structure or environmental health as well as achieve financial returns and the modes of measurement are proving interesting to this study.
Finally, Social IA looks more closely at individuals, organisations and social macro-systems. The International Principles for Social Impact Assessment (Vanclay, 2003) defines Social IA as including “the processes of analysing, monitoring and managing the intended and unintended social consequences, both positive and negative, of planned interventions and any social change processes invoked by those interventions”. Social IA has a predictive element but successive tools such as Theory of Change have made it more participatory and part of the process of managing the social issues. For our purposes we are happy to include so called social return on investment within the conception of social IA although others within the profession would disagree with this wide church approach. There are many methods and tools for Social IA and we think these may prove especially helpful in considering the life opportunities questions and indicators we need to establish.
What is increasingly clear though is that all these modes of IA have something to offer and that our inclusive approach to this research is a good one. Our challenge is to rationalise now into ways of presenting a cohesive set of methods and guidance that is most useful to our memory organisations and sector.
One way to organise the thinking might be to use the Balanced Scorecard approach to considering the Impact of a digitised resource upon lives. This allows us to balance out the Impacts being assessed and to ensure that multiple perspectives are enabled. By combining economic measures, social and non-financial measures in a single approach, the Balanced Scorecard should provide richer and more relevant information.
In the Balanced Scorecard approach we would suggest the following core headings:
- Social and audience Impacts
- Economic Impacts
- Innovation Impacts
- Internal process Impacts
In this way we can assess the way Impact is occurring both externally and internally to the organisation delivering the digital resource. Allowing a balanced perspective of changes to people’s lives who use the resource and changes to the organisation through the existence of the resource.
There are a number of challenges that many Impact Assessments have sought to address with varying success. These include:-
- timescales in which measurements may take place remain too short for outcomes to become visible and constrained by project timescales;
- a lack of suitable benchmarks or baselines from which to measure change;
- a diverse evidence base;
- the wide range of possible beneficial stakeholders;
- the difficulty in establishing useful indicators for the sector;
- the need to make recommendations to decision-makers based on strong evidence;
- the lack of skills and training in the community of practise;
- and the need to review relevant qualitative as well as quantitative evidence to discover significant measurable outcomes.
These challenges are all part of scope for this study. The need to establish useful indicators for this sector is currently a major concern of this research. Indicators drive the questions and the methods which may be applied to gain significant information to support impact assessment.
What are your thoughts? What are the significant indicators? What are the exemplars out there that you have seen or methods you have tried?