Selected Evaluation Tools from the Federal Evaluators Network1 (July 2016)

Title and web address

Intended audience2

Comments (contents, intended use, etc.)

Demonstrating evaluation’s value – why evaluate?

Evaluation Dialogue Between OMB Staff
and Federal Evaluators,
2006.
http://www.fedeval.net/docs/omb2006briefing.pdf

Both

Briefing slides describe various evaluation purposes, types, and place in the management cycle.

DOE. Case Studies Show the Value of
Program Evaluation,
2009. Introduction to
the "Value of Program Evaluation" Case Study Series
http://energy.gov/eere/analysis/
program-evaluation-eere-resource-documents#value

Manager

Two-page case studies document the benefits of implementing evaluation recommendations, such as streamlined processes, increased productivity, or reduced program costs.

GAO. Designing Evaluations: 2012 Revision.
http://www.gao.gov/products/GAO-12-208G

Both

Guidance paper, see especially Chapter 1, “The Importance of Evaluation Design”

GAO. Performance Measurement and Evaluation, 2011.
http://www.gao.gov/products/GAO-11-646SP

Both

Guidance provides brief description of purposes and types of evaluations.

GAO. Program Evaluation: An Evaluation Culture and
Collaborative Partnerships
Help Build Agency Capacity, 2003.
http://www.gao.gov/products/GAO-03-454

Both

Report provides examples of agencies that use evaluation results to test new initiatives or assess progress toward agency goals.

Millennium Challenge Corporation. Conducting
an Impact Evaluation
.
https://www.mcc.gov/our-impact/independent-evaluations

Manager

Web page outlines issues to consider when selecting projects to evaluate, such as learning potential, strong stakeholder commitment, and feasibility. 

Building a program logic model

DOJ. Developing and Working with Logic Models.
https://www.bja.gov/programs/crppe/logic-models.html

Both

Web pages provide guidance and links to further guidance and examples in criminal justice and other fields. 

DOL. Fully Articulating Your Vision: Using Logic Models
to Support Innovation,
2011.
https://www.workforcegps.org/resources/2016/06/07/07/59/
Fully_Articulating_Your_Vision-_Using_Logic_Models_to
_Support_Innovation_26_Min

Both

A tutorial to help grantees use logic models to articulate goals, identify and organize resources, and focus on results.

EPA. Picturing Your Program: An Introduction to
Logic Modeling,
2013.
 http://www.epa.gov/evaluate/lm-training/index.htm

Both

A web-based training course on developing program logic models and their use in developing performance measures and conducting program evaluations.

Developing performance measures – assessing reliability

DOJ. Understanding Performance Measures.
http://www.ojjdp.gov/grantees/pm/understanding.html

Both

Web pages explain and provide examples for numerous goals of juvenile justice assistance grants.

GAO. Performance Plans: Selected Approaches
for Verification and Validation of Agency Performance
Information,
1999.
 
http://www.gao.gov/products/GGD-99-139

Both

Report provides examples from several agencies of fostering organizational commitment and capacity, assessing quality, and building quality into data collection.

GAO. The Results Act: An Evaluator’s Guide to
Assessing Agency Annual Performance Plans,
1998.
http://www.gao.gov/products/GGD-10.1.20

Evaluator

Guidance for assessing strengths and weaknesses of performance plans required under GPRA, see section 1 on measurement, section 3 on verifying data.

GAO. Assessing the Reliability of Computer-Processed
Data,
2009.
http://www.gao.gov/products/GAO-09-680G

Evaluator

Guidance, including electronic testing of sample data, and reviewing selected system controls.

HHS, National Institutes of Health. PEPH Evaluation
Metrics Manual
, 2012.
http://www.niehs.nih.gov/research/supported/
dert/programs/peph/metrics/

See also web-based training on use of the manual.

Evaluator

Guidance for developing performance measures for Partnerships for Environmental Public Health, including metrics for partnerships, leveraging, products and dissemination, education and training, and capacity building.

USAID. How-to Note: Preparing a Performance
Management Plan
, 2013.
http://usaidlearninglab.org/library/how-note
-preparing-performance-management-plan-pmp

Evaluator

Guidance for developing a plan to monitor, evaluate and analyze progress toward achieving a project’s results as depicted in its logical framework.

Designing an evaluation study
- clarifying the question; consulting stakeholders; selecting
a design, data sources, or evaluative criteria

Evaluation Dialogue Between OMB Staff and
Federal Evaluators,
2006.  http://www.fedeval.net/docs/omb2006briefing.pdf

Both

Briefing slides outline steps in design process, common evaluation questions by program stage, and designs used for different types of federal programs.

Corporation for National and Community Service
(CNCS). Evaluation Resources.

http://www.nationalservice.gov/resources/evaluation

Both Resources for grantees, agency staff and other stakeholders. Webinars on logic models, basic steps in an evaluation, budgeting, writing an evaluation plan, managing an external evaluation, reporting and using results, among other topics.

DOE, EERE. Project Manager’s Guide to Managing
Impact and Process Evaluation Studies,
August 2015.
http://energy.gov/sites/prod/files/2015/09/f26/project_manager
_guide_managing_impact_process_evaluation_studies.pdf

Both Step-by-step approach to help managers of EERE evaluation projects create and manage objective, high quality, independent, and useful impact and process evaluations.

DOE, EERE. Evaluating Realized Impacts of
DOE/EERE R&D Programs,
2014 Final Report.
http://energy.gov/sites/prod/files/2015/05/f22/
evaluating_realized_rd_mpacts_9-22-14.pdf

Evaluator

Guidance for impact evaluations to determine the “realized” economic benefits and costs, energy, environmental benefits, and other impacts of R&D programs.

DOE. Impact Evaluation Framework for Technology
Deployment Programs: An Approach for Quantifying
Retrospective Energy Savings, Clean Energy Advances,
and Market Effects,
2007. http://www1.eere.energy.gov/
ba/pba/pdfs/ impact_framework_tech_deploy_2007_main.pdf

Both

Guidance provides a series of steps and templates to design and implement impact studies of technology diffusion, with a focus on identifying the links between outputs and outcomes for different sectors.

DOJ. Guide to Program Evaluation.
https://www.bja.gov/evaluation/guide/
bja-guide-program-evaluation.pdf

Both

Guidance provides brief discussion of types of evaluation activities, benefits of evaluation, concerns, and constraints. Also covered are performance measures, logic models and links to more detailed guidance.

GAO. Designing Evaluations: 2012 Revision.
http://www.gao.gov/products/GAO-12-208G  

Both

Guidance addresses the design process, including determining the right questions and the feasibility of answering them, evaluation types and designs.

HHS, Administration for Children and Families (ACF).
Common Framework for Research and Evaluation
, 2016. http://www.acf.hhs.gov/sites/default/files/opre/acf_common
_framework_for_research_and_evaluation_v02_a.pdf

Both The framework outlines the roles of various types of research and evaluation studies related to human services. The document describes the purpose of each study type, their different empirical and theoretical justifications, types of study outcomes, and expectations for their design.

HHS, Administration for Children and Families (ACF).
A Framework to Design, Test, Spread, and Sustain
Effective Practice in Child Welfare,
2014.
http://www.acf.hhs.gov/programs/cb/resource/
framework-workgroup?utm_source=Framework
_Workgroup&utm_medium=Crosslink&utm_
campaign=Program_Evaluation


Available also in video format. Children’s Bureau Child Welfare
Evaluation Virtual Summit Series 2013-2014.
http://www.acf.hhs.gov/programs/cb/capacity/program
-evaluation/virtual-summit/framework

Both

The framework charts five phases to guide a user through designing, implementing, and evaluating interventions to build evidence about effectiveness and support integration into routine practice.

HHS, ACF. The Program Manager’s Guide to Evaluation,
Second Edition,
2010.
http://www.acf.hhs.gov/opre/resource/the-program-managers
-guide-to-evaluation-second-edition

Manager

Guidance specifies basic evaluation questions, preparation, and planning. Discusses building logic models, determining objectives, selecting design and data sources.

HHS, Centers for Disease Control and Prevention (CDC).
Other Evaluation Resources.

http://www.cdc.gov/eval/resources/index.htm

Both

Web page collection of public and private sector guides, manuals/assistance with developing logic models, developing effective evaluation plans, data collection methods and resources, and other topics.

HUD. The Healthy Homes Program Guidance Model.
Chapter 6, Evaluating Your Program,
July 2012. http://portal.hud.gov/hudportal/HUD?src=/program
_offices/healthy_homes/HHPGM

Both Guidance and tools to establish or improve healthy homes programs. Evaluation chapter covers planning, logic models, quality data, evaluation measures, and dissemination.

HHS, CDC. Framework for Program Evaluation
in Public Health,
1999.
http://www.cdc.gov/eval/framework/index.htm

Both

Web pages summarize essential elements of program evaluation and outline steps and standards for effective public health program evaluation.

Estimating study costs

GAO.  GAO Cost Estimating and Assessment Guide:
Best Practices for Developing and Managing
Capital Program Costs
, 2009.
http://www.gao.gov/products/GAO-09-3SP

Both

Guidance provides a consistent methodology to be used by federal, state, or local government agencies to develop more reliable cost estimates for government projects of all sizes.

HHS, ACF. The Program Manager’s Guide to Evaluation,
Second Edition, 2010.
http://www.acf.hhs.gov/opre/resource/the-program-managers
-guide-to-evaluation-second-edition

Manager

Guidance describes the kinds of information that can be obtained at different cost levels. See section What Will an Evaluation Cost?

Program Cost Analysis                      

HHS, ACF. Cost Analysis in Program Evaluation:
A Guide for Child Welfare Researchers and
Service Providers,
2013.
http://www.acf.hhs.gov/programs/cb/resource/cost-workgroup?utm_source=Cost_Workgroup&utm
_medium=Crosslink&utm _campaign=Program_Evaluation
.
Available also in video format. Children’s Bureau Child Welfare
Evaluation Virtual Summit Series 2013-2014.

http://www.acf.hhs.gov/programs/cb/capacity/
program-evaluation/ virtual-summit/cost-analysis-videos

Both

Guide provides a framework for integrating cost analysis into program evaluations and guidance for defining scope, purpose, information needed, and core steps involved.

Working with Administrative Data 

FCSM. Data Quality Assessment Tool for
Administrative Data,
2013.
http://fcsm.sites.usa.gov/files/2014/04/
DataQualityAssessmentTool.pdf

Both Tool provides a set of questions to help data users assess the fitness of data for their needs and develop a memorandum of understanding with the data provider.

FCSM. Profiles in Success of Statistical
Uses of Administrative Data,
2009.
http://fcsm.sites.usa.gov/files/2014/04/
StatisticalUsesofARData.pdf

Both Report discusses six challenges encountered in seven successful data-sharing arrangements between various federal, state, and academic entities and four elements of success: vision and support by agency leadership, narrow but flexible goals, infrastructure, and mutual interest.

GAO. Federal Statistical System: Agencies
Can Make Greater
Use of Existing Data but
Progress Needed on Access
and Quality Issues,
2012.
http://www.gao.gov/products/GAO-12-54

Both Report describes how administrative data supplement survey data and recommends that OMB better disseminate information on interagency committees and develop comprehensive guidance for data sharing.

HHS, ACF. Confidentiality Toolkit: A Resource Tool
from the ACF Interoperability Initiative,
2014.
http://www.acf.hhs.gov/about/interoperability#chapter-2

Both Toolkit outlines the individual case planning and program review and evaluation purposes for data sharing across state and local social service agencies, the confidentiality and privacy issues and laws involved in each program, and several example memorandums of understanding between agencies.

HHS, ACF. Getting it Done: Partnering to Overcome
Concerns About Data Sharing
[video]. Children’s
Bureau Child Welfare Evaluation Virtual Summit
Series 2013-2014.

http://www.acf.hhs.gov/programs/cb/capacity/
program-evaluation /virtual-summit/data-sharing-partnering

Manager

Video depicts common barriers to sharing data, such as unclear data needs, and privacy and technology concerns, and discusses strategies for overcoming them.

OMB. Analytical Perspectives, Budget of the United States,
Fiscal Year 2016. Chapter 7.
Building Evidence
with Administrative Data,
2015.
http://www.whitehouse.gov/omb/evidence

Manager

Chapter describes reuse of administrative data for a wide range of purposes; includes examples, barriers, and the privacy, confidentiality, and data security protections that govern their use.

OMB. Guidance for Providing and Using Administrative
Data for Statistical Purposes,
2014. OMB Memorandum M-14-06.
http://www.whitehouse.gov/omb/evidence

Manager

Guidance for use of non-public administrative data for statistical purposes to evaluate the effectiveness of programs and policies; elements include leadership responsibilities,   legal and policy requirements, and best practice tools for monitoring progress.

Working with contractors – writing a task order or RFP, selecting a contractor,  or monitoring contracts

USAID. How-to Note on Evaluation Statements
of Work,
2013.

http://usaidlearninglab.org/library/
how-note-evaluation-statement-work

Both

Guidance for an evaluation statement of work for externally contracted evaluations covers: purpose, audience, and use of the evaluation; background; evaluation questions; design and methodology; deliverables and timeline; expertise needed; logistics and budget.

Reporting findings – tailoring to different audiences, encouraging use

GAO. Program Evaluation: Strategies to Facilitate
Agencies’ Use of Evaluation in Program Management
and Policy Making,
2013. 
http://www.gao.gov/products/GAO-13-570.

Both

Report provides results from a random sample of federal civilian managers on their use of program evaluations, barriers to use, and examples of use and strategies used by agency evaluators to facilitate evaluation influence.

GAO. How to Get Action on Audit Recommendations, 1991.
http://www.gao.gov/products/OP-9.2.1

Both

Guidance for developing high quality, effective recommendations also addresses the importance of monitoring and follow-up.

HHS, ACF. Use of Research Evidence:
Building Two-Way Streets.
[video] Children’s Bureau
Child Welfare Evaluation Virtual Summit Series 2013-2014.
http://www.acf.hhs.gov/programs/cb/capacity/program
-evaluation/virtual-summit/two-way-streets

Both

Video outlines steps to close gaps between the research and evaluation community to improve the relevant use of research in practice.

HHS, CDC. Developing an Effective Evaluation Report, 2013. http://www.cdc.gov/eval/materials/developing-an
-effective-evaluation-report_tag508.pdf

Both Guide for developing an effective public health program or initiative evaluation report. Includes exercises, worksheets, tools, and resources.

The Lewin Group. Getting the Most out of Evaluations:
A Guide to Successful Evaluation Utilization,
2009.
http://www.lewin.com/publications/publication/200906300460/                   

Both

Guide provides action steps in four areas—involving stakeholders, evaluation quality, dissemination, and evaluation culture—to facilitate evaluation use.

USAID. How-to Note on Evaluation Reports, 2012
http://usaidlearninglab.org/library/how-note-preparing-evaluation-reports.

Both

Guidance for developing reports includes requirements and components that also serve as a guide for reviewing the quality of draft reports. 

Evaluating a research portfolio

DOE, Office of Energy Efficiency and Renewable Energy (EERE).
Overview of Evaluation Methods for R&D Programs
, 2007.
http://www1.eere.energy.gov/ba/pba/pdfs/
evaluation_methods_r_and_d.pdf
 

Both

Report provides an overview of a variety of methods for evaluating R&D programs, including examples of use by DOE, NIH, NIST, and NSF.

DOE, EERE. EERE Peer Review Guide, 2004. http://www1.eere.energy.gov/ba/pba/pdfs/
2004peerreviewguide.pdf

Both

Guidance for peer reviews, including, steps to plan, design, and implement external reviews.

DOE. A Framework for Evaluatng R&D Impacts and
Supply Chain Dynamics Early in a Product Life Cycle,
2014.
http://energy.gov/sites/prod/files/2015/05/f22/evaluating
_rd_impacts_supply_chain_dynamics.pdf
Evaluator Guidance for evaluating R&D investments aimed at speeding up the pace of innovation and strengthening domestic manufacturing and supply chains. The framework focuses on the dynamics of innovation and early stages of developing a new product. It identifies core progress and impact metrics and provides an approach to help establish agency attribution.

DOE.EERE. Advanced Manufacturing Office’s Stage-Gate
Innovation Management Guidelines
, 2007.
http://www1.eere.energy.gov/manufacturing/financial/pdfs/
itp_stage_gate_overview.pdf

Both

Guidance for applying the Stage-Gate process for reviewing progress of energy-saving technology development projects.

Integrating evaluation with performance measurement

GAO. Performance Measurement and Evaluation, 2011.
http://www.gao.gov/products/GAO-11-646SP

Both

Guidance provides brief description of their complementary purposes.

GAO. Program Evaluation: Studies Helped Agencies
Measure or Explain Program Performance,
2000. 
http://www.gao.gov/products/GGD-00-204

Both

Report provides examples from several agencies of the complementary use of evaluation and performance measurement.

USAID. Discussion Note: Complexity-Aware Monitoring, 2013.
http://usaidlearninglab.org/library/complexity
-aware-monitoring-discussion-note-brief

Evaluator

Note outlines general principles and promising approaches for monitoring development assistance in complex, changing environments where cause and effect relationships are poorly understood.

Evaluation policy

American Evaluation Association. An Evaluation Roadmap
for a More Effective Government,
revised 2013.
http://www.eval.org/d/do/472

Both

Paper outlines general principles for strengthening federal agency evaluation functions and integrating evaluation into program management.

Council of the Inspectors General on Integrity and Efficiency.
Quality Standards for Inspection and Evaluation,
2012.
https://www.ignet.gov/content/inspections-evaluation-0

Evaluator

Standards for federal Offices of Inspector General to ensure the independence and integrity of units providing information on the efficiency, effectiveness, or sustainability of agency operations, programs, or policies.

GAO. Program Evaluation: Some Agencies
Reported that Networking, Hiring, and Involving
Program Staff help Build Capacity,
2014.
http://www.gao.gov/products/GAO-15-25

Both Report examines GPRAMA implementation and agency capacity to conduct and use evaluations through a survey of agency organizational resources, activities, and policies.

HHS, ACF. Evaluation Policy, November 2012.
http://www.acf.hhs.gov/evaluation-policy

Both

Policy presents key principles that pertain to the planning, conduct, and use of program evaluations

HHS, CDC. Improving the Use of Program
Evaluation for Maximum Health Impact: Guidelines
and Recommendations,
2012.
http://www.cdc.gov/eval/materials/
finalcdcevaluationrecommendations
_formatted_120412.pdf

Both

Guidelines and recommendations to improve evaluation planning and implementation, develop funding opportunity announcements, and build evaluation capacity.

DOL. U.S. Department of Labor Evaluation Policy,
November
2013.
http://www.dol.gov/asp/evaluation/EvaluationPolicy.htm

Both

Policy presents key principles that pertain to the planning, conduct, and use of program evaluations.

Millennium Challenge Corporation. Policy for Monitoring
and Evaluation of Compacts and Threshold Programs,
2012. https://assets.mcc.gov/guidance/policy-050112-
monitoring-and-evaluation.pdf

Both

Guidance for the monitoring and evaluation framework required of each country compact, including logic models, performance indicators, economic analysis and impact evaluation.

PEPFAR. PEPFAR Evaluation Standards of Practice,
September 2015.
http://www.pepfar.gov/reports/guidance/c61317.htm

Both Guidance on implementing standards of practice for evaluation planning, implementation, reporting, and review of evaluations and roles and responsibilities of relevant stakeholders. Tools and templates to support evaluation planning and reporting.

State. Department of State Program Evaluation
Policy,
January 2015.
http://www.state.gov/s/d/rm/rls/evaluation/2015

Both

Policy establishes requirements for program and project evaluation, evaluation plans, utilization of findings, and expected documentation and dissemination.  

USAID. USAID Evaluation Policy, January 2011.
http://www.usaid.gov/evaluation

Both

Policy establishes roles, responsibilities, and key expectations for the design, conduct, dissemination and use of evaluation.


1 For more information, see the Federal Evaluators website, www.fedeval.net.

2 Evaluator, program manager, or both.

Evaluation Tools Training Opportunities