Brief 1: Overview of Policy Evaluation


Download Brief 1: Overview of Policy Evaluation


Preview text

Step by Step – Evaluating Violence and Injury Prevention Policies
Brief 1: Overview of Policy Evaluation

This brief provides a definition of policy evaluation, including a

description of the evaluation framework used and a definition

“Policy evaluation uses a

of policy. It also presents information about the general value and potential challenges of conducting policy evaluation.

range of research methods to systematically investigate the

Defining Policy

effectiveness of policy interventions, implementation and processes, and

The CDC definition of “Policy” is “a law, regulation, procedure,

to determine their merit, worth,

administrative action, incentive or voluntary practice of governments and other institutions.”1 Policies generally operate at the systems level and can influence complex systems in ways that can improve the health and safety of a population. A policy

or value in terms of improving the social and economic conditions of different stakeholders.”3

approach can be a cost-effective way to create positive changes

in the health of large portions of the population. There are several types of policy, each of which can operate at different levels (national, state, local, or organizational)2

Legislative policies are laws or ordinances created by elected representatives. Regulatory policies include rules,

guidelines, principles, or methods created by government agencies with regulatory authority for products or

services. Organizational policies include rules or practices established within an agency or organization.

What Is Policy Evaluation?3
Policy evaluation applies evaluation principles and methods to examine the content, implementation or impact of a policy. Evaluation is the activity through which we develop an understanding of the merit, worth, and utility of a policy.
CDC Evaluation Framework
While there are a variety of different approaches to evaluation, this set of briefs utilize the six-step CDC Framework for Evaluation in Public Health, as shown in Figure 1.4
The Framework outlines an ongoing process comprising six steps of program evaluation.2 These six steps are also applicable to policy evaluation and provide a guide for implementing a thorough evaluation. The remaining briefs each discuss the implementation of one or more of these steps. Figure 2 illustrates which steps are discussed in each brief.
Briefs 2, 6, and 7 discuss the steps as they apply to all types of policy evaluation, whereas Briefs 3, 4, and 5 discuss Step 3 (focusing the evaluation design) as it applies to the specific types of policy evaluation.

1 CDC, Office of the Associate Director for Policy. (2011). Definition of policy. PDF available upon request; please contact [email protected] 2 CDC, Office of the Associate Director for Policy. (2011). Definition of policy. PDF available upon request; please contact [email protected] 3 Her Majesty’s Treasury. (2011). The magenta book: Guidance for evaluation. London, UK: Author. Retrieved from http://www.hm-treasury.gov.
uk/data_magentabook_index.htm 4 CDC, Office of the Associate Director for Program. (2012, September). A framework for program evaluation. Retrieved from http://www.cdc.gov/
eval/framework/index.htm
National Center for Injury Prevention and Control

Figure 2. The Briefs in Relation to the Steps in the CDC Evaluation Framework

(1) Engaging Stakeholders

(2) Describing (3) Focusing the Program or the Evaluation
Policy

(4) Gather Credible Evidence

(5) Justify Conclusions

Brief 2







Brief 3







Brief 4







Brief 5







Brief 6





Brief 7

(6) Ensure Use and Lessons
Learned


Standards for Conducting Evaluation
The Framework also includes the following four categories of standards for conducting evaluation to help guide choices along the process:
ƒƒ Utility: Who wants the evaluation results and for what purpose? ƒƒ Feasibility: Are the evaluation procedures practical, given the time, resources, and expertise available? ƒƒ Propriety: Is the evaluation being conducted in a fair and ethical way? ƒƒ Accuracy: Are approaches at each step accurate, given stakeholder needs and evaluation purpose?
Policy Evaluation Versus Program Evaluation
Although policy evaluation and program evaluation have many similarities, there are some important differences as well. Some of these differences include:
ƒƒ The level of analysis required (e.g., system or community level for policy evaluation; program level for program evaluation).
ƒƒ The degree of control and clear “boundaries” may be more challenging with policy evaluation. ƒƒ The ability to identify an equivalent comparison community may be more challenging with policy evaluation. ƒƒ The scale and scope of data collection may be greater with policy evaluation. ƒƒ Policy evaluation may require increased emphasis on the use of surveillance and administrative data. ƒƒ The type and number of stakeholders involved may differ.
Why Is Policy Evaluation Important?
Developing and implementing policy strategies is important in addressing injury and violence prevention at the population level. Although policy has been used effectively in some areas of injury and violence prevention,

2

Step by Step – Evaluating Violence and Injury Prevention Policies

policy strategies in some areas lack a sufficient evidence base. Policy evaluation, like all evaluation, can serve important purposes along the entire chain of the policy process, including1:

ƒƒ Documenting policy development. ƒƒ Documenting and informing implementation. ƒƒ Assessing support and compliance with existing
policies.

ƒƒ Demonstrating impacts and value of a policy. ƒƒ Informing an evidence base. ƒƒ Informing future policies. ƒƒ Providing accountability for resources invested.

Evaluation Within the Policy Process

It is important to understand how policy evaluation fits into the larger policy process. Understanding this context provides an increased understanding of why policy evaluation is critical to advancing the field of policy. Although there are many theories regarding the policy process and mechanisms of policy change, the policy change process is often conceptualized in several key stages as depicted on the top row of Figure 3.2 Evaluation is an integral part of each step in the policy process. Although these steps are laid out in a row, in reality, the steps are circular in nature. The three main types of evaluation, shown in the bottom row of Figure 3, each focus on a different phase of the policy process5: policy content evaluation, policy implementation evaluation, and policy impact evaluation. Figure 3 illustrates the relationship between the main stages of the policy process and the three types of evaluation.

ƒƒ Evaluating Policy Content: Does the content clearly articulate the goals of the policy, its implementation
and the underlying logic for why the policy will produce intended change? Evaluating the development of a policy helps to understand the context, content, and implementation.
ƒƒ Evaluating Policy Implementation: Was the policy implemented as intended? The implementation of a
policy is a critical component in understanding its effectiveness. Evaluation of policy implementation can provide important information about the barriers to and facilitators of implementation and a comparison between different components or intensities of implementation.
ƒƒ Evaluating Policy Impact: Did the policy produce the intended outcomes and impact? Within injury
prevention, the intended impact may be a reduction in injuries or severity of injuries. However, it is important to evaluate short-term and intermediate outcomes as well.
The type of evaluation selected depends on many factors, and often more than one type of evaluation will be needed. Each type of evaluation can provide valuable information for the planning and interpretation of the other types of evaluation (content, implementation, and impact) in addition to uncovering unintentional consequences.6 However, it is critical for each evaluation to be focused so the most appropriate design and methodology is selected.1 The team can develop an overarching set of evaluation questions and then select specific evaluation questions and methods for each particular phase. Appendix A and Appendix B provide examples of the planning, implementation, and dissemination of a policy evaluation.

5 Brownson, R. C., Royer, C., Chriqui, J. F., & Stamatakis, K. A. (2009). Understanding evidence-based public health policy. American Journal of Public Health, 99, 1576–1583.
6 MacDonald, G., Starr, G., Schooley, M., Yee, S. L., Klimowksi, K., & Turner, K. (2001). Introduction to program evaluation for comprehensive tobacco control programs. Atlanta, GA: Centers for Disease Control and Prevention. Retrieved from http://www.cdc.gov/tobacco/tobacco_ control_programs/surveillance_evaluation/evaluation_manual/pdfs/evaluation.pdf

3

Step by Step – Evaluating Violence and Injury Prevention Policies

Policy Evaluation Challenges
While all evaluations encounter challenges, some are particularly relevant to policy evaluation, and some of these are listed below. Many of these challenges can be easily addressed by using an appropriate design, indicators, and methods. Specific solutions to many of these challenges are provided in Briefs 2–7. In addition, Appendix C provides an overview of the challenges and potential solutions when conducting policy evaluation.

Potential Policy Evaluation Challenges

ƒƒ Lack of resources or clear responsibility for

ƒƒ Lack of strong evidence base to support policy

evaluation

ƒƒ Fear of evaluation and lack of familiarity with ƒƒ External and contextual factors such as

policy evaluation methods

economic conditions or public awareness

ƒƒ Lack of “control” over policy implementation ƒƒ Access to appropriate data

ƒƒ Rapid pace of policy

ƒƒ Lack of appropriate measures

ƒƒ Political scrutiny and desire for quick production ƒƒ Difficulty in identifying appropriate comparison

of results

communities

Additional Resources
The Magenta Book: Guidance for Evaluation (Her Majesty’s Treasury). Provides general and technical guidance on policy evaluation. Available at http://www.hm-treasury.gov.uk/data_magentabook_index.htm CDC Evaluation Page: http://www.cdc.gov/eval/ Appendix A: Policy Impact Evaluation Example - Child Restraint Law Expansion Appendix B: Restricting Hours of Alcohol Sales to Prevent Excessive Alcohol Consumption and Related Injuries Appendix C: Challenges and Potential Solutions to Policy Evaluation Appendix D: Glossary

4

Step by Step – Evaluating Violence and Injury Prevention Policies

Preparing to load PDF file. please wait...

0 of 0
100%
Brief 1: Overview of Policy Evaluation