Selected Evaluation Tools from the Federal Evaluators Network1

Title and web address Comments (contents, intended use, etc.)
Demonstrating evaluation’s value – why evaluate?
Evaluation Dialogue Between OMB Staff and Federal Evaluators, 2006.
http://www.fedeval.net/docs/omb2006briefing.pdf		 Briefing slides describe various evaluation purposes, types, and place in the management cycle.
DOE. Case Studies Show the Value of Program Evaluation, 2009. Introduction to the “Value of Program Evaluation” Case Study Series
http://energy.gov/eere/analysis/program-evaluation-eere-resource-documents#value		 Two-page case studies document the benefits of implementing evaluation recommendations, such as streamlined processes, increased productivity, or reduced program costs.
GAO. Designing Evaluations: 2012 Revision.
http://www.gao.gov/products/GAO-12-208G		 Guidance paper, see especially Chapter 1, “The Importance of Evaluation Design”
GAO. Performance Measurement and Evaluation, 2011.
http://www.gao.gov/products/GAO-11-646SP		 Guidance provides brief description of purposes and types of evaluations.
GAO. Program Evaluation: An Evaluation Culture and Collaborative Partnerships Help Build Agency Capacity, 2003.
http://www.gao.gov/products/GAO-03-454		 Report provides examples of agencies that use evaluation results to test new initiatives or assess progress toward agency goals.
Millennium Challenge Corporation. Conducting an Impact Evaluation.
https://www.mcc.gov/our-impact/independent-evaluations		 Web page outlines issues to consider when selecting projects to evaluate, such as learning potential, strong stakeholder commitment, and feasibility.
Building a program logic model
DOJ. Developing and Working with Logic Models.
https://www.bja.gov/programs/crppe/logic-models.html		 Web pages provide guidance and links to further guidance and examples in criminal justice and other fields.
DOL. Fully Articulating Your Vision: Using Logic Models to Support Innovation, 2011.
https://www.workforcegps.org/resources/2016/06/07/07/59/Fully_Articulating_Your_Vision-_Using_Logic_Models_to_Support_Innovation_26_Min		 A tutorial to help grantees use logic models to articulate goals, identify and organize resources, and focus on results.
EPA. Picturing Your Program: An Introduction to Logic Modeling, 2013.
http://www.epa.gov/evaluate/lm-training/index.htm		 A web-based training course on developing program logic models and their use in developing performance measures and conducting program evaluations.
Developing performance measures – assessing reliability
DOJ. Understanding Performance Measures.
http://www.ojjdp.gov/grantees/pm/understanding.html		 Web pages explain and provide examples for numerous goals of juvenile justice assistance grants.
GAO. Performance Plans: Selected Approaches for Verification and Validation of Agency Performance Information, 1999.
http://www.gao.gov/products/GGD-99-139		 Report provides examples from several agencies of fostering organizational commitment and capacity, assessing quality, and building quality into data collection.
GAO. The Results Act: An Evaluator’s Guide to Assessing Agency Annual Performance Plans, 1998.
http://www.gao.gov/products/GGD-10.1.20		 Guidance for assessing strengths and weaknesses of performance plans required under GPRA, see section 1 on measurement, section 3 on verifying data.
GAO. Assessing the Reliability of Computer-Processed Data, 2009.
http://www.gao.gov/products/GAO-09-680G		 Guidance, including electronic testing of sample data, and reviewing selected system controls.
HHS, National Institutes of Health. PEPH Evaluation Metrics Manual, 2012.
https://www.niehs.nih.gov/research/supported/translational/peph/metrics/index.cfm
See also web-based training on use of the manual.		 Guidance for developing performance measures for Partnerships for Environmental Public Health, including metrics for partnerships, leveraging, products and dissemination, education and training, and capacity building.
USAID. How-to Note: Preparing a Performance Management Plan, 2013.
https://usaidlearninglab.org/library/how-note-prepare-and-maintain-performance-management-plan-pmp		 Guidance for developing a plan to monitor, evaluate and analyze progress toward achieving a project’s results as depicted in its logical framework.
Designing an evaluation study – clarifying the question; consulting stakeholders; selecting a design, data sources, or evaluative criteria
Evaluation Dialogue Between OMB Staff and Federal Evaluators, 2006.  http://www.fedeval.net/docs/omb2006briefing.pdf Briefing slides outline steps in design process, common evaluation questions by program stage, and designs used for different types of federal programs.
Corporation for National and Community Service (CNCS). Evaluation Resources.
http://www.nationalservice.gov/resources/evaluation		 Resources for grantees, agency staff and other stakeholders. Webinars on logic models, basic steps in an evaluation, budgeting, writing an evaluation plan, managing an external evaluation, reporting and using results, among other topics.
DOE, EERE. Project Manager’s Guide to Managing Impact and Process Evaluation Studies, August 2015.
http://energy.gov/sites/prod/files/2015/09/f26/project_manager_guide_managing_impact_process_evaluation_studies.pdf		 Step-by-step approach to help managers of EERE evaluation projects create and manage objective, high quality, independent, and useful impact and process evaluations.
DOE, EERE. Evaluating Realized Impacts of  DOE/EERE R&D Programs, 2014 Final Report.
http://energy.gov/sites/prod/files/2015/05/f22/evaluating_realized_rd_mpacts_9-22-14.pdf		 Guidance for impact evaluations to determine the “realized” economic benefits and costs, energy, environmental benefits, and other impacts of R&D programs.
DOE. Impact Evaluation Framework for Technology Deployment Programs: An Approach for Quantifying Retrospective Energy Savings, Clean Energy Advances,and Market Effects, 2007.
http://www1.eere.energy.gov/ba/pba/pdfs/impact_framework_tech_deploy_2007_main.pdf		 Guidance provides a series of steps and templates to design and implement impact studies of technology diffusion, with a focus on identifying the links between outputs and outcomes for different sectors.
DOJ. Guide to Program Evaluation.
https://www.bja.gov/evaluation/guide/bja-guide-program-evaluation.pdf		 Guidance provides brief discussion of types of evaluation activities, benefits of evaluation, concerns, and constraints. Also covered are performance measures, logic models and links to more detailed guidance.
GAO. Designing Evaluations: 2012 Revision.
http://www.gao.gov/products/GAO-12-208G		 Guidance addresses the design process, including determining the right questions and the feasibility of answering them, evaluation types and designs.
HHS, Administration for Children and Families (ACF). Common Framework for Research and Evaluation, 2016.
http://www.acf.hhs.gov/sites/default/files/opre/acf_common_framework_for_research_and_evaluation_v02_a.pdf		 The framework outlines the roles of various types of research and evaluation studies related to human services. The document describes the purpose of each study type, their different empirical and theoretical justifications, types of study outcomes, and expectations for their design.
HHS, Administration for Children and Families (ACF). A Framework to Design, Test, Spread, and Sustain Effective Practice in Child Welfare, 2014.
http://www.acf.hhs.gov/programs/cb/resource/framework-workgroup?utm_source=Framework_Workgroup&utm_medium=Crosslink&utm_campaign=Program_Evaluation
Available also in video format. Children’s Bureau Child Welfare Evaluation Virtual Summit Series 2013-2014.
http://www.acf.hhs.gov/programs/cb/capacity/program-evaluation/virtual-summit/framework		 The framework charts five phases to guide a user through designing, implementing, and evaluating interventions to build evidence about effectiveness and support integration into routine practice.
HHS, ACF. The Program Manager’s Guide to Evaluation, Second Edition, 2010.
http://www.acf.hhs.gov/opre/resource/the-program-managers-guide-to-evaluation-second-edition		 Guidance specifies basic evaluation questions, preparation, and planning. Discusses building logic models, determining objectives, selecting design and data sources.
HHS, Centers for Disease Control and Prevention (CDC). Other Evaluation Resources.
http://www.cdc.gov/eval/resources/index.htm		 Web page collection of public and private sector guides, manuals/assistance with developing logic models, developing effective evaluation plans, data collection methods and resources, and other topics.
HUD. The Healthy Homes Program Guidance Model. Chapter 6, Evaluating Your Program, July 2012.
http://portal.hud.gov/hudportal/HUD?src=/program_offices/healthy_homes/HHPGM		 Guidance and tools to establish or improve healthy homes programs. Evaluation chapter covers planning, logic models, quality data, evaluation measures, and dissemination.
HHS, CDC. Framework for Program Evaluation in Public Health, 1999.
http://www.cdc.gov/eval/framework/index.htm		 Web pages summarize essential elements of program evaluation and outline steps and standards for effective public health program evaluation.
Estimating study costs
GAO.  GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs, 2009.
http://www.gao.gov/products/GAO-09-3SP		 Guidance provides a consistent methodology to be used by federal, state, or local government agencies to develop more reliable cost estimates for government projects of all sizes.
HHS, ACF. The Program Manager’s Guide to Evaluation, Second Edition, 2010.
http://www.acf.hhs.gov/opre/resource/the-program-managers-guide-to-evaluation-second-edition		 Guidance describes the kinds of information that can be obtained at different cost levels. See section What Will an Evaluation Cost?
Program Cost Analysis
HHS, ACF. Cost Analysis in Program Evaluation: A Guide for Child Welfare Researchers and Service Providers, 2013.
http://www.acf.hhs.gov/programs/cb/resource/cost-workgroup?utm_source=Cost_Workgroup&utm_medium=Crosslink&utm
_campaign=Program_Evaluation.Available also in video format. Children’s Bureau Child Welfare Evaluation Virtual Summit Series 2013-2014.
http://www.acf.hhs.gov/programs/cb/capacity/program-evaluation/virtual-summit/cost-analysis-videos		 Guide provides a framework for integrating cost analysis into program evaluations and guidance for defining scope, purpose, information needed, and core steps involved.
Working with Administrative Data 
FCSM. Data Quality Assessment Tool for Administrative Data, 2013.
http://fcsm.sites.usa.gov/files/2014/04/DataQualityAssessmentTool.pdf		 Tool provides a set of questions to help data users assess the fitness of data for their needs and develop a memorandum of understanding with the data provider.
FCSM. Profiles in Success of StatisticalUses of Administrative Data, 2009.
http://fcsm.sites.usa.gov/files/2014/04/StatisticalUsesofARData.pdf		 Report discusses six challenges encountered in seven successful data-sharing arrangements between various federal, state, and academic entities and four elements of success: vision and support by agency leadership, narrow but flexible goals, infrastructure, and mutual interest.
GAO. Federal Statistical System: Agencies Can Make Greater Use of Existing Data but Progress Needed on Access and Quality Issues, 2012.
http://www.gao.gov/products/GAO-12-54		 Report describes how administrative data supplement survey data and recommends that OMB better disseminate information on interagency committees and develop comprehensive guidance for data sharing.
HHS, ACF. Confidentiality Toolkit: A Resource Tool from the ACF Interoperability Initiative, 2014.
http://www.acf.hhs.gov/about/interoperability#chapter-2		 Toolkit outlines the individual case planning and program review and evaluation purposes for data sharing across state and local social service agencies, the confidentiality and privacy issues and laws involved in each program, and several example memorandums of understanding between agencies.
HHS, ACF. Getting it Done: Partnering to Overcome Concerns About Data SharingChildren’s Bureau Child Welfare Evaluation Virtual Summit Series 2013-2014.
http://www.acf.hhs.gov/programs/cb/capacity/program-evaluation/virtual-summit/data-sharing-partnering		 Video depicts common barriers to sharing data, such as unclear data needs, and privacy and technology concerns, and discusses strategies for overcoming them.
OMB. Analytical Perspectives, Budget of the United States, Fiscal Year 2016. Chapter 7. Building Evidence with Administrative Data, 2015.
https://www.gpo.gov/fdsys/browse/collectionGPO.action?collectionCode=BUDGET		 Chapter describes reuse of administrative data for a wide range of purposes; includes examples, barriers, and the privacy, confidentiality, and data security protections that govern their use.
OMB. Guidance for Providing and Using Administrative Data for Statistical Purposes, 2014. OMB Memorandum M-14-06.
https://www.whitehouse.gov/omb/information-for-agencies/memoranda		 Guidance for use of non-public administrative data for statistical purposes to evaluate the effectiveness of programs and policies; elements include leadership responsibilities,   legal and policy requirements, and best practice tools for monitoring progress.
Working with contractors – writing a task order or RFP, selecting a contractor,  or monitoring contracts
USAID. How-to Note on Evaluation Statements of Work, 2013.
http://usaidlearninglab.org/library/how-note-evaluation-statement-work 		Guidance for an evaluation statement of work for externally contracted evaluations covers: purpose, audience, and use of the evaluation; background; evaluation questions; design and methodology; deliverables and timeline; expertise needed; logistics and budget.
Reporting findings – tailoring to different audiences, encouraging use
GAO. Program Evaluation: Strategies to Facilitate Agencies’ Use of Evaluation in Program Management and Policy Making, 2013.
http://www.gao.gov/products/GAO-13-570.		 Report provides results from a random sample of federal civilian managers on their use of program evaluations, barriers to use, and examples of use and strategies used by agency evaluators to facilitate evaluation influence.
GAO. How to Get Action on Audit Recommendations, 1991.
http://www.gao.gov/products/OP-9.2.1		 Guidance for developing high quality, effective recommendations also addresses the importance of monitoring and follow-up.
HHS, ACF. Use of Research Evidence: Building Two-Way Streets.Children’s Bureau
Child Welfare Evaluation Virtual Summit Series 2013-2014.
http://www.acf.hhs.gov/programs/cb/capacity/program-evaluation/virtual-summit/two-way-streets		 Video outlines steps to close gaps between the research and evaluation community to improve the relevant use of research in practice.
HHS, CDC. Developing an Effective Evaluation Report, 2013.
http://www.cdc.gov/eval/materials/developing-an-effective-evaluation-report_tag508.pdf		 Guide for developing an effective public health program or initiative evaluation report. Includes exercises, worksheets, tools, and resources.
The Lewin Group. Getting the Most out of Evaluations: A Guide to Successful Evaluation Utilization, 2009.
http://www.lewin.com/publications/publication/200906300460/		 Guide provides action steps in four areas—involving stakeholders, evaluation quality, dissemination, and evaluation culture—to facilitate evaluation use.
USAID. How-to Note on Evaluation Reports, 2012
http://usaidlearninglab.org/library/how-note-preparing-evaluation-reports.		 Guidance for developing reports includes requirements and components that also serve as a guide for reviewing the quality of draft reports.
Evaluating a research portfolio
DOE, Office of Energy Efficiency and Renewable Energy (EERE).  Overview of Evaluation Methods for R&D Programs, 2007.
http://www1.eere.energy.gov/ba/pba/pdfs/evaluation_methods_r_and_d.pdf		 Report provides an overview of a variety of methods for evaluating R&D programs, including examples of use by DOE, NIH, NIST, and NSF.
DOE, EERE. EERE Peer Review Guide, 2004.
http://www1.eere.energy.gov/ba/pba/pdfs/
2004peerreviewguide.pdf		 Guidance for peer reviews, including, steps to plan, design, and implement external reviews.
DOE. A Framework for Evaluatng R&D Impacts and Supply Chain Dynamics Early in a Product Life Cycle, 2014.
http://energy.gov/sites/prod/files/2015/05/f22/evaluating_rd_impacts_supply_chain_dynamics.pdf		 Guidance for evaluating R&D investments aimed at speeding up the pace of innovation and strengthening domestic manufacturing and supply chains. The framework focuses on the dynamics of innovation and early stages of developing a new product. It identifies core progress and impact metrics and provides an approach to help establish agency attribution.
DOE.EERE. Advanced Manufacturing Office’s Stage-Gate Innovation Management Guidelines, 2007.
http://www1.eere.energy.gov/manufacturing/financial/pdfs/itp_stage_gate_overview.pdf		 Guidance for applying the Stage-Gate process for reviewing progress of energy-saving technology development projects.
Integrating evaluation with performance measurement
GAO. Performance Measurement and Evaluation, 2011.
http://www.gao.gov/products/GAO-11-646SP		 Guidance provides brief description of their complementary purposes.
GAO. Program Evaluation: Studies Helped Agencies Measure or Explain Program Performance, 2000.
http://www.gao.gov/products/GGD-00-204		 Report provides examples from several agencies of the complementary use of evaluation and performance measurement.
USAID. Discussion Note: Complexity-Aware Monitoring, 2013.
http://usaidlearninglab.org/library/complexity-aware-monitoring-discussion-note-brief		 Note outlines general principles and promising approaches for monitoring development assistance in complex, changing environments where cause and effect relationships are poorly understood.
Evaluation policy
American Evaluation Association. An Evaluation Roadmap for a More Effective Government, revised 2016.
http://www.eval.org/d/do/4008		 Paper outlines general principles for strengthening federal agency evaluation functions and integrating evaluation into program management.
Council of the Inspectors General on Integrity and Efficiency. Quality Standards for Inspection and Evaluation, 2012.
https://www.ignet.gov/content/inspections-evaluation-0		 Standards for federal Offices of Inspector General to ensure the independence and integrity of units providing information on the efficiency, effectiveness, or sustainability of agency operations, programs, or policies.
GAO. Program Evaluation: Some Agencies Reported that Networking, Hiring, and Involving Program Staff help Build Capacity, 2014.
http://www.gao.gov/products/GAO-15-25		 Report examines GPRAMA implementation and agency capacity to conduct and use evaluations through a survey of agency organizational resources, activities, and policies.
GAO. Foreign Assistance: Selected Agencies’ Monitoring and Evaluation Policies Generally Address Leading Practices, 2016.
www.gao.gov/products/GAO-16-861R		 Report examines policies for planning, conducting and reporting evaluation of foreign assistance at 6 federal agencies.
HHS, ACF. Evaluation Policy, November 2012.
http://www.acf.hhs.gov/evaluation-policy		 Policy presents key principles that pertain to the planning, conduct, and use of program evaluations
HHS, CDC. Improving the Use of Program Evaluation for Maximum Health Impact: Guidelines and Recommendations, 2012.
http://www.cdc.gov/eval/materials/finalcdcevaluationrecommendations_formatted_120412.pdf		 Guidelines and recommendations to improve evaluation planning and implementation, develop funding opportunity announcements, and build evaluation capacity.
DOL. U.S. Department of Labor Evaluation Policy, November 2013.
http://www.dol.gov/asp/evaluation/EvaluationPolicy.htm		 Policy presents key principles that pertain to the planning, conduct, and use of program evaluations.
Millennium Challenge Corporation. Policy for Monitoring and Evaluation, 2017.
https://www.mcc.gov/resources/doc/policy-for-monitoring-and-evaluation		 Guidance for the monitoring and evaluation framework required of each country compact, including logic models, performance indicators, economic analysis and impact evaluation.
PEPFAR. PEPFAR Evaluation Standards of Practice, September 2015.
http://www.pepfar.gov/reports/guidance/c61317.htm		 Guidance on implementing standards of practice for evaluation planning, implementation, reporting, and review of evaluations and roles and responsibilities of relevant stakeholders. Tools and templates to support evaluation planning and reporting.
State. Department of State Program Evaluation Policy, January 2015.
http://www.state.gov/s/d/rm/rls/evaluation/2015		 Policy establishes requirements for program and project evaluation, evaluation plans, utilization of findings, and expected documentation and dissemination.
USAID. USAID Evaluation Policy, January 2011.
http://www.usaid.gov/evaluation		 Policy establishes roles, responsibilities, and key expectations for the design, conduct, dissemination and use of evaluation.

1 For more information, see the Federal Evaluators website, www.fedeval.net.