Evaluation
Evaluation is an important part of any planning and management process. It should provide evidence of achievement and where resources have been used effectively, and also provide an awareness of areas for review and improvement. Remember that evaluation is not undertaken in clinical conditions and is therefore not an exact science. However rigorous the methodology, it will, at best, only give an ‘approximation’ of success.
Principles
First of all there are a number of issues to be aware of when planning an evaluation, such as needing to distinguish between the different types of evaluation. For example:
- Impact/Outcome evaluation is the process of assessing at a particular point whether or not you are achieving your objectives.
- Process evaluation is about looking to see if the processes used were working properly and helped you to achieve your objectives.
It is important however also to differentiate between monitoring and evaluation. Although they both involve measuring what has been achieved against the original objectives, the difference is that monitoring involves the continual assessment of progress whereas evaluation involves measurement at a given point in time. Evaluation can be very time consuming and expensive, therefore careful consideration should be given to whether this is undertaken. Should it be for a small scale project then continual monitoring may be sufficient to identify if a project is achieving its objectives.
Evaluation is a tool to assist in the management and control of projects, and can be undertaken at pre-planned points during the life of a project and for different purposes:
- Evaluation prior to implementation is known as ‘pre evaluation or appraisal’, the purpose being to decide whether proposed plans are feasible.
- Evaluation at key points during a project acts as a check to see whether objectives are being met. This is most useful in larger, more costly initiatives or one with an extended timescale. This ‘formative’ type of evaluation provides the opportunity for changes to be made if required.
- Post-project evaluation is designed to see whether the objectives have been achieved, and this commonly called impact or outcome evaluation. What is important, is that evaluation is seen as a dynamic process, carried out at all stages of a project, and should not be undertaken as an ‘afterthought’. http://www.crimereduction.gov.uk/evalpart2.pdf
When and why evaluation is undertaken are important factors, another is the level at which it is done. The three main levels being project, programme and strategy levels, ranging from localised small scale projects to high level strategies such as Crime and Disorder Reduction Strategies.
Question
What types of evaluation does your organisation carry out? Find out whether your organisation has an evaluation policy and try and get hold of copies of completed evaluations to see the kind of work that has been carried out.
Carrying Out Evaluations
As previously outlined, impact or outcome evaluation is the process of assessing, at a particular point in time, whether or not a project is achieving or has achieved its objectives.
It is valuable for:
- Providing evidence of success or failure
- Identifying if costs have been used effectively
- Identifying mistakes thereby avoiding future implementation failure
- Allowing for improvements to be made
- Providing documented process material for replication
- Providing accountability.
Process evaluation should:
- describe and document the activities undertaken
- identify implementation problems
- describe approaches taken to resolving them.
It should also include a description of any lessons learned and how future projects could be improved.
Impact Evaluation should include information on:
|
Background information |
Why the work was undertaken |
|
Implementation details |
Documentation of interventions Who worked on the interventions Descriptions of delivery Problems encountered, options considered for resolving and solutions adopted |
|
Outputs achieved |
Lists of outputs associated with; each intervention |
|
Costs |
Total budgets Costs of each intervention Staff time |
|
Lessons learned |
From each intervention and General lessons to emerge |
When carrying out evaluation or monitoring of a project you should approach it as a project in its own right and ensure that it is planned, well organised and has clear objectives and methods. You then need to:
Set the evaluation objectives
Decide what data to collect to measure objectives
- Collect the data
- Analyse the data
- Plan the logistics
- Document the findings
Evaluation Reports
When documenting the evaluation findings you should consider:
- Setting it out clearly so it can be understood
- The level of complexity and length should vary dependent on audience for which it is intended
- Including an executive summary
- Setting up a way of reviewing subsequent work as a result of the evaluation
The Home Office ‘Passport to Evaluation (http://www.crimereduction.gov.uk/learningzone/passport_to_ evaluation.htm) is a good guide to understanding what is needed in planning, undertaking and delivering an evaluation. It does confirm, as outlined earlier, that evaluation does not provide conclusive proof that an initiative has worked. But there are three steps you should take to ensure that the results are as accurate as possible, and they are:
- At the planning stage, list all the factors that might have an effect on the initiative. Be aware of the area in which it is being undertaken and what other work of a similar nature is going on in the area
- Collect data for your performance indicators from as wide a range of sources as possible,
- Carefully analyse all available sources of data.
Question
What evaluation system does your organisation use? If possible get hold of a copy of the documentation used and familiarise yourself with it. Even if you don’t carry out evaluations yourself it will be useful to know what kind of information you need to record to help the evaluators do their job properly.
Summary
Monitoring and evaluation needs to be planned prior to any intervention or project
implementation.
Monitoring and evaluation should inform future decision making.
Evaluation can be either impact or outcome based or processed based.
Evaluation is not an exact science and can only ever act as an indicator of success.
