What to evaluate and when to evaluate a program depend on several interrelated considerations, such as:
- Stage of the program in its life cycle (i.e., creation of program, program underway, or at or near program closeout phase),
- Decisions to be informed by evaluative information during critical planning, budgeting, implementation, benefits reporting and communication cycles, and
- Any need to meet evaluation requirements.
- An overall program-wide strategic evaluation plan should be developed to map out planned evaluation activity over a multi-year time frame, so the information produced:
- Is timely, without being so frequent that not enough time passes to act on recommendations from the evaluation
- In sync with the program's management and decision processes
- Complements other program data collection activities.
What to Evaluate
There is no hard and fast rule on what programs or program areas should be evaluated. However, here are some circumstances that may suggest your program is in need of an evaluation:
(1) For a new program that is underway or the creation of a new program:
- Desire to understand if and how a new program or pilot works because you want to expand or replicate it later if determined to be successful
- New initiative with lots of uncertainties that need to be better understood to achieve the initiative's full potential
- A new initiative is a major budgetary priority, or addresses an important problem
(2) For an existing program that is underway:
- Program has never been formally evaluated, or has persisted year after year without adequate evidence that it is producing results
- Program activities cover large portions of the program's budget or are highly visible to major stakeholders for other reasons
- A need to re-evaluate an existing program where significant changes were made
- A program (or subprogram activity) appears to be in trouble (e.g., progress or participation lower than expected), or is experiencing better than expected success
- A program that has high potential risks or a complex design
- A program (or subprogram activity) where there is an interest in adapting it elsewhere
- A program that is a major budgetary priority, or addresses an important problem
- A program that has advanced to a stage where consumer and markets benefits should be accruing and the amount of economic return on investment should be determined, such as:
- One that has potential for large benefits, or appears to be cost-effective, and the return on investment is currently not known;
- One that was previously evaluated in an earlier period but had shown no significant realized economic return on investment, in part because accrued value (e.g., in the market) had not been established for its commercialized technologies during the period of the earlier study;
- One previously evaluated but requires an update (e.g., two to five years later);
- One that had performed benefit-cost calculations using an alternative approach not consistent with an updated benefit-cost assessment methodology; or
- Congress directs an economic return on investment study be done
(3) For a program at or near closeout:
- Program that is phasing out or has closed out, where lessons learned and best practices could be identified and potentially applied to the development of a future successor initiative
When to Evaluate
Programs have critical planning, budgeting, implementation, benefits reporting and communication cycles. These elements of a program life cycle can be viewed in three stages –
- Creating a program,
- Program is underway
- Closing out or end of program.
In general, different type of evaluations are carried out over different parts of a program's life cycle.
Accommodating EERE, DOE, and OMB requirements where they exist, the following table illustrates where in the program life cycle offices should plan or consider doing an evaluation and types of evaluations that are sensible for each life cycle phase.
Generally speaking, selected program activities or the entire program should have some evaluation performed over a time period ranging from annually to every two or three years.
|Program Life Cycle Stage
|Type of Program Evaluation
|Creating a program
|Market Needs and Effects Assessment: conducted before the program is initiated.
|Program is underway/ ongoing
Outcome evaluation: conducted once every 2-3 years or annually if results are to be used to support annual Government Performance and Results Act (GPRA) benefits analysis. Outcome evaluations can also be performed at any time to answer ad hoc questions regarding program outcomes.
Impact evaluation: performed once every 3-5 years or annually if results are to be used to support annual GPRA benefits analysis, or as often as cost-benefit analyses will be performed.
In-progress Peer Review: In EERE, in-progress peer reviews of project portfolios or entire programs should be performed on average every two years. EERE In-Progress Peer Review Guide. (An EERE Requirement)
Technical Program Reviews and Advisory Committee Reviews are conducted periodically by senior management, technical teams, or advisory committees.Market Needs and Effects Assessment: repeated as necessary to determine if there is a continuing need for the program's services or for identifying new market segments to target. If major changes are occurring in external factors that may affect program outcomes and impacts (e.g., energy prices and changes in minimum-efficiency standards), it might be necessary to repeat a market assessment every several years.
|Closing out or end of program
Process and/or impact evaluations after the program has ended: conducted at or near closeout or within 1-3 years of the end of the program (apply lessons learned from process evaluation to the design of next generation programs; use impact evaluation to document program effectiveness).Cost-benefit evaluation: conducted within 1-3 years of the end of the program only if needed for accountability, and repeat as necessary (e.g., every 2 to 5 years) to account for ongoing accrual of benefits beyond program closeout.