5.2 How To Evaluate
How Do You
Gather Evaluation Information?
Just as
there are different types of evaluations to assess the various
components of your project, there are also different ways
of collecting the information you will need. Here is a brief
overview of some common evaluation methods and what they work
best for.
Interviews
with key participants
(used
for process and outcome evaluation)
Conducting
face-to-face interviews is one of the most effective ways
of gathering detailed, information-rich data for your evaluation.
Interviewing people will help your to capture people’s experiences
participating in activities, of their feelings of empowerment,
of their sense of community, of well-being, and so on.
Because
people often express their views and opinions more easily
orally then in writing, you will probably find that interviews
generate a greater volume and depth of information than
questionnaires or evaluation forms.
For
more information on conducting key informant interviews,
please refer to the Community Tool Box, Chapter 3 I, Section
8: Interviews with key participants. http://ctb.Isi.ukans.edu/ctb/c31/c31s8
or to other resources listed at the end of this chapter.
Focus
groups
(used
for process and outcome evaluation)
A focus
group is a small-group discussion guided by a trained leader.
It is used to learn more about opinions on a certain topic,
and then to guide future action.
The group’s
composition and the group discussion are planned to create
a comfortable environment where people will feel free to
talk openly. Members are actively encouraged to express
their opinions, and to respond to other members, as well
as to questions posed by the leader.
Because
focus groups are structured and directed, but also expressive,
they can yield a lot of information in a relatively short
time. For more information on conducting focus groups, please
refer to the Community Tool Box, Chapter 2, Section 4: Conducting
Focus Groups http://ctb.lsi.ukans.edu/ctb/c2/c2s4
or to other resources listed at the end of this chapter.
Participant
observation
(used
more for process evaluation)
Participant
observation allows you to gather information about how your
project as it is actually happening, from the point of view
of a participant. The participant is trained in writing
descriptively, perceiving detail (and separating it from
trivia), and recognizing the subjectivity of their own perceptions.
Observational
evaluation reports should be detailed enough that the reader
develops an understanding of what happened, how it happened,
and the context in which it happened.
|
The
evaluation of the Inclusion project hinged on the role
of the "site historian". In each of the project
sites, a community member, usually a consumer/survivor,
was selected and trained in participant observation,
so that they could monitor and evaluate the process
and outcomes of the project.
The
site historians took part in an intensive training session,
which included an overview of qualitative methods. In
this training, which was provided by the CMHA national1,
site historians learned that their own observations
and insights were an integral part of the overall project
evaluation.
By
using participant observation techniques, site historians
were able to record the development and implementation
of the project in a way that captured the nuances of
the process in each site.
|
Back
to top
Event
logs
(used
more for process evaluation)
Event
logs are written accounts of the activities of the initiative.
They might also be used to record any changes in the community
brought about by the initiative, such as new policies, programs,
or practices related to the initiative’s goals and mission.
An event
log usually contains the following information:
- when
the event took place;
- why
it was important;
- what
happened as a result
- who
was involved;
- what
organizations contributed people and resources.
The people
who fill out event logs will be those members who are taking
action on behalf of the initiative. The log should be completed
soon after the event takes place, so that the details are
still fresh in the mind of the person who is recording them.
Thee event logs can then be given to the people in the group
who are responsible for compiling evaluation information.
Questionnaires
and surveys
(used
mainly for outcome evaluation)
Questionnaires
and surveys can be used when you need to get information
quickly. They don’t necessarily capture the same richness
of detail as interviews, focus groups and participant observation,
but they can be helpful in gathering baseline data and rating
participant satisfaction.
You’ll
find an example of a participant satisfaction survey in
the tools section at the end of his chapter.
Baseline
data
A baseline
provides you with a snapshot of the community at the beginning
of your project. By collecting the same data several times
through the life of your project, you will be, in essence,
doing time-lapse photography. This will help you to see
if changes came about in your community, and you’ll have
a better chance of knowing if those changes were a result
of your project.
Some things
you might measure with baseline data include numbers of
partnerships, numbers of participants, etc.
This is
not a separate strategy, but should be used when you are
conducting questionnaires and surveys, and sometimes with
interviews and focus groups.
Back
to top
How
Do You Evaluate Your Mental Health Promotion Project?
A Review of The
Main Steps
There are
three main steps to developing the evaluation for your mental
health promotion project:
- Clarifying
your project goals and objectives;
- Developing
your evaluation questions and select evaluation methods;
- Developing
a framework for collecting evaluation data;
1. Clarifying
your project goals and objectives
The first
step in any evaluation is to clarify the goals and objectives
of your initiative, for it will help you identify which components
of your project should be evaluated.
You have
probably already completed this step, if you’ve been following
the planning model we’ve set out in this tool kit. Now is
a good time to revisit those objectives, and see if they are
still relevant. If you haven’t yet set your goals and objectives,
now would be a good time to consider the following questions:
- What
are the main things you want to accomplish?
- How do
you propose to accomplish them?
The kinds
of things you’ll want to keep track of in your mental health
promotion initiative include the following:
- what
you did to implement your initiative;
- the events
and process of the initiative as it’s happening;
- what
the results of your project were.
Back
to top
2. Developing
evaluation questions and selecting evaluation methods
The following
questions2
can be used as a template to develop evaluation questions
for your mental health promotion project.
Did we
do what we said we would do?
"WHAT?"
(description of activities)
The responses
to this question describe the work done in your project and
the relevance of this work in meeting your project goals and
objectives. Your objectives provide the criteria against which
you measure success.
What
did we learn about what worked and what didn’t?
"WHY?"
(reasons for success)
The answers
to this question will tell you about people’s perceptions
of what worked well in your
project
and what didn’t work well, so that you can take action to
improve your efforts.
"What
difference did it make that we did this work?
"SO
WHAT?" (outcomes)
The answers
to this question measure your project’s success in changing
knowledge, attitudes, and behaviour. Again, your objectives
provide the criteria against which you measure change both
during and at the end of your project.
"What
could we do differently?
"NOW
WHAT?" (the future of this and other projects)
This question
examines the challenges that your project presented, which
often provide a rich learning opportunity.
How do
we plan to use the evaluation findings for continuous learning?
"THEN
WHAT" (use of evaluation results)
This question
looks at how will you use your evaluation results throughout
the project as well as at the end, and whom you will share
them with.
If you think
about your project in terms of the five questions, it should
help you to design your process and outcome evaluation effectively,
so that you capture the information you need, and it will
help you choose the most appropriate methods for going about
collecting that information.
Back
to top
3.
Developing a framework for collecting evaluation data (need
to change wording here)
The table
that begins on the next page summarizes the different components
of developing an evaluation framework. It combines information
on evaluation questions and methods as well as timelines for
collecting evaluation data.
You can
use the table as a guide to help you through the process of
developing a framework to evaluate your own mental health
promotion project
|
Key
Evaluation Questions For Your Project
|
Type
Of Information Needed To Answer Questions
|
Method
Used To Gather Information
|
Time
Frame For Gathering Information
|
|
WHAT
Did we do what we said we'd do?
- What
activities were undertaken, and how did they link
to meeting project goals and objectives?
- What
were the major achievements of the project, and what
resources did they require?
- Did
the objectives change during the project? How and
why did they change?
|
process
and outcome measurements that describe the project's
activities, including:
- community
services provided - workshops, publications, communications...
- community
actions taken - to encourage change in the community
- products
of planning processes - written objectives, committees,
action plans...
- community
resources and assets identified and used - grants,
donations, in-kind resources...
|
|
|
|
WHY
What did we learn about what worked and what didn't?
- Who
participates?
- Is
the group diverse?
- Do
participants remain involved?
- Why
do people enter and leave your project?
- Are
community members satisfied that your project meets
local needs?
- What
strategies worked well/didn't work well for involving
community members?
- What
strategies worked best/didn't work best for broadening
the base of community support?
- Which
strategies and activities did you change? Why?
- How
realistic and relevant were your project goals and
objectives?
- In
what ways did the planning process work most effectively?
-
What did you learn about working together as a group?
|
process
and outcome measurements that examine reasons for success,
including:
- people
who participate - information on who participates,
how often, why people leave
- media
coverage - publicity given to your initiative
|
- basic
demographic data on participants, e.g. age, gender,
economic status, ethnic background, disability, etc.
- questionnaires
and surveys
|
- beginning,
middle and end of the project
|
|
SO WHAT? What difference did it make that
we did this work?
- What
has changed as a result of the project?
- What
evidence is there to attribute any of attitudes, behaviours,
policies, these changes to the project?
- What
evidence is there to attribute any of these changes
to the project?
- What
other factors outside the project might through have
contributed to these changes?
- Have
participants been involved in the entire process of
the initiative, including defining the issue?
- How
has participants’ mental health been promoted as a
result of participation in the initiative?
- Are
participants satisfied with their experience?
- Were
there any negative results from participation in the
initiative?
- In
what ways did the project contribute to increased
community participation/ strengthening community capacity?
- What
new partnerships were developed in the project?
- What
was the nature of these partnerships, and what was
their contribution?
- Did
the project have any negative results?
|
process and outcome measures that examine outcomes,
including:
- changes
in knowledge, skills, attitudes, behaviours, policies,
practices.. .
|
baseline
data gathered
- questionnaires
and surveys
|
|
|
NOW WHAT?
What could we have done differently?
- What
effective ways for achieving objectives emerged from
this project?
- What
additional support and resources are required to do
the work more effectively in the future?
- What
could be done to expand the network of people involved
in the project?
- Could
the project have been better planned?
|
process and outcome measures that examine outcomes,
including:
- products
of the planning process - action plans, evaluation
plans
|
- questionnaires
and surveys
|
|
|
THEN WHAT?
How do we plan to use evaluation results?
- How
were evaluation results used on an ongoing basis to
contribute to the planning and implementation of the
project?
- How
will final evaluation results be documented and disseminated?
- How
will evaluation results be used for renewed project
planning?
|
process and outcome measures that examine outcomes,
including:
- products
of the planning process - dissemination plans
|
|
|
Back
to top
|
The
framework for evaluating Inclusion in Community was
based on the broad program goals, and was in place before
the project actually got under way. A variety of measures
were used to determine the success of each site’s efforts,
and the as well as the success of the project as a whole.
The
project was evaluated according to key outcome measures
such as:
- number
of community partnerships formed;
- stakeholder
participation in design and implementation of an action
plan in each community;
- extent
of shift in focus among participants from service
model to community process model.
Outcome
Key
indicators were evaluated using a number of measures
such as pre and post-project questionnaires and focus
groups of stakeholders. and end-of-project evaluation
meetings in each site.
Process
A
number of steps were taken throughout the course of
the project in order to monitor the effectiveness of
its ongoing activities. These included participant evaluations
of workshops and meetings, written evaluation of the
process by local participants and a continuing log of
experiences, Iearnings and activities kept by a site
historian.
|
Back
to top
Summary >
1.
For more information on the site historians' training, please
contact the CMHA National office.
|
