My name is Kurmit Rockwell, and I'm the FEMP ESPC Program Lead.
The Just in Time training series is designed to give you timely information from program experts to help you award high-quality, on-time ESPC task orders. This is the second of three sessions in this series. In session one, we developed the task order request for proposal, conducted the IGA kickoff meeting, and set up expectations for success through frequent communications including bi-weekly ESCO-Agency meetings.
This session will focus on the technical review of the proposal your ESCO submitted, based on the results of the IGA.
Session three of this series will be focusing on the review of the price proposal, and finally crossing the finish line to task order award.
Your instructors today are very experienced in ESPC. Scott Wolf works for the Oak Ridge National Laboratory and is a FEMP Federal Project Executive. [He] has worked more than 10 years in this field and has been involved in dozens of ESPC projects. He helps agencies overcome barriers and award high-quality ESPC projects. Prior to FEMP, he was an engineer at the Washington State University energy program where he helped develop the award-winning Resource Efficiency Manager Program so heavily used in the Department of Defense and other agencies.
Deb Kephart is a retired federal contracting officer with over 25 years' contracting experience, including over 13 years working with ESPCs. Deb previously worked at the Golden [Field] Office in Colorado where she was Contracting Officer for the DOE IDIQ master contract. Prior to that, she worked for the Navy as a contracting officer with responsibility for awards and administration of multiple Navy and Marine Corps ESPC task orders. She now works for Allegheny Science and Technology to provide support to FEMP, assisting with the ESPC Program.
Bob Slattery works for Oak Ridge National Laboratory as well, and he has many years' experience in the ESPC Program. He's a specialist in measurement and verification and supports FEMP's ESPC Life of Contract Program, where he helps to monitor annual savings achieved by ESPC projects.
Lastly, I want to leave you with this: It's FEMP's goal to provide you with relevant, high-quality training to help you achieve your energy and sustainability goals. So to help serve you better, I'm asking you to please complete an evaluation form at the end of your session today.
I'm now going to turn it over to your first speaker, Scott Wolf, to get us started.
Thank you very much, Kurmit. I really do appreciate that introduction. Today, we're going to talk about the critical path to success and the following discussion topics: First, Evaluating the Proposal: Know Where You Stand and Where You Want to Go. Then, Best Practices for Managing Agency Review. We'll then move on to The Crystal Clear Technical Proposal. This will include ECM Descriptions and Energy and Cost Savings. Then we'll move to the Management Approach. From there we go to Basics, which will include the Measurement & Verification, which we refer to as "M&V." Then we will go discuss Risk, Responsibility, and Performance Matrix. You can see the acronym, RRPM. And then finally, Summary-Best Practices and Lessons Learned. Slide 5, please.
To remind you-you saw this in the previous session-these are the milestones in the ESPC process. We're going to focus on pretty much Phase 3: the Proposal. But just to refresh your memory, we start with Phase 1: Acquisition Planning; then Phase 2: ESCO Selection, Preliminary Assessment, Notice of Intent to Award; then we shift to Phase3: Request for Proposal, Investment Grade Audit. And like I said, we're going to focus right here on Proposal. After that in the process is Task Order Award, Final Design and Construction, Project Acceptance, and then lastly, Post-Acceptance Performance Period. We'll talk about those latter parts in Just in Time Session 3. Next slide, please.
And just to remind you-you've seen this slide as well-it's the Project Development timeline. And key here is the legend. In the bottom left, you can see the Agency Action denoted by a star, and, by a box, the ESCO Action. And you can see the big arrow: We're going to focus here on the evaluation of the proposal. Next slide, please.
And to get you a larger picture of how this process will work during the IGA Presentation through Award, we did borrow a great graphic from the FEMP ESPC Project Development Guide. As you can see in the circled areas, we are going to cover this in the Just in Time 2 series, and the greyed areas-you see three of them-will be covered in the Just in Time 3 version, which you will hopefully hear next. Simply put, the agency responsibilities are in the top bar, beginning with the IGA/Proposal Presentation that the ESCO provides, moving to the IGA/Proposal Review, which is a very formal process, and then the IGA/Proposal Comment Review Workshop, which you do in conjunction with the ESCO. From there on, there tends to be more of an intense Negotiation, back and forth. And of course, below all that, you can see the FEMP services that are provided to support you and your team during this process. Next slide.
And you can see the ESPC project development guidebook here. This is an excellent, 74-page guidebook which you could download for free from the FEMP website. And as Albert Einstein says, you should really be using this Resource Guide on your next project. And I really do like the message here, and when we wrote this book, we adhered to the practice of simplicity: "Everything should be made as simple as possible, but not simpler." Next slide.
Now as you can see, our IDIQ, plus our Task Order RFP, plus our Proposal encompasses the entire Task Order. And this is a graphical representation. The IDIQ includes the general scope, the terms, and the conditions. And the Task Order RFP includes the site/agency-specific requirements, and there are a lot of those for agencies, and they differ all over the board. As Kurmit was suggesting, the investment grade audit does roll over into a proposal. This is a project overview, and includes the technical specifications from the investment grade audit findings; and then you will have the financial schedules, which are really the heart of the proposal, and you should pay special attention to that; and then the subcontracting plan. With that I will turn the next slide over to Deb Kephart.
Well, thank you, Scott. The key ingredient is knowing where you stand and where you want to go with negotiations. You should consider the relationship you've developed with the ESCO throughout the project development cycle. Another critical element is to know when to negotiate.
Because ESPCs require the ESCO to both design and build, agencies will have to live with the ESCO's design and construction for a long period of time. That's why it's very important for agencies to ensure that they do a very comprehensive review before contract award. So, how do we do that? Next slide, please.
First, do a quick review of the proposal. Consider if the proposal contains all the required content as specified within the DOE's IDIQ master contract. So this includes adequately describing the energy conservation measures, or ECMs, that are proposed. It should include a narrative description of what will be installed, the physical changes that will be made, and also specify the major equipment manufacturers and model numbers, if known, for each ECM proposed. And it should also include details on baseline and energy savings calculations as well as assumptions.
Then ESCO's proposed measurement and verification plan should be included and consistent with the IDIQ format. This will be discussed in more detail later in our presentation. And it should include a Risk, Responsibility & Performance Matrix. This is a very important element of the contract and, again, will be discussed in more detail later. And there are financial schedules, called TO schedules. There are five of them and they summarize the financial deal. We talk more about this in our Session 3.
And next, I would assess the completeness and adequacy of the proposal based on the direction provided to the ESCO in the agency's task order request for proposal, or TO-RFP. And then it's time to assess if there is additional information that you need to make a comprehensive analysis of the proposal. A lot of agencies have price analysts that assist COs in their evaluation, and frequently, additional pricing information is needed. The key now is to ensure that you have all the information you need to understand the proposal details and to conduct the evaluation. As necessary, the contracting officer will instruct the ESCO to submit any missing data. Next slide, please.
And next, consider your relationship with the ESCO. How is it? Did you find that the proposal you received is different than what you expected, based on your discussions with the ESCO? These discussions started back at the IGA kick-off meeting, where the agency stated its expectations, and continued throughout the IGA process where the ESCO discussed the capabilities and interim results. Also, were deadlines met? Were there delays due to the ESCO's inability to perform, or because of communication difficulties? Do you feel like this is an ESCO that can effectively manage this project?
Another consideration is the ESCO's design and engineering team. It could be either the prime contractor or a sub. Did they listen to your concerns? Did they communicate effectively? And did they develop a comprehensive proposal that meets your agency's needs?
Conversely, did the ESCO clearly communicate what they could deliver, so that there were no-or maybe only minor-surprises in the proposal? Were the mutually agreed-to deadlines consistently met? And consider the design and engineering team: Did they work well with your group and develop a project that you believe they can successfully accomplish? This is usually good news. Next slide, please.
Additionally, consider if the proposed project progressed as expected. Agencies and ESCOs have expectations about how the project should progress. Are your expectations in sync? Can they be improved through better communications? Did the ESCO surpass your expectations? If so, this is great news, and usually means a high probability of implementing a very successful project. Has your relationship thus far inspired trust and confidence in each other? Has the ESCO listened to your needs, and are they responsive to your requests? It's best to get any difficulties out in the open to ensure that you can have confidence and trust going forward in your collaborative efforts. Sometimes the information provided by the agency's not clear. Sometimes the ESCO's not listening. If there are communication difficulties, agencies should assess how these difficulties can be overcome.
And finally, assess if this is an ESCO who is a good fit for a long-term partnership. All these are just symptoms to consider. No one particular symptom makes for a good or a bad relationship, but these are things to keep in mind to help agencies feel that they've received a proposal that includes a comprehensive scope, that the ESCO can successfully implement the project, and that the agency has confidence that this is an ESCO that they can work with over a long term. An ESPC can last for up to 25 years, so you'll have to get along with each other for a very long time. And if you answered "No" to any of these areas, consider discussing your concerns with the Project Facilitator, or a Federal Project Executive such as Scott, and they may be able to help you on a one-on-one basis. Next slide, please.
So negotiations, they're described as exchanges or discussions between the contractor, or ESCO, in this case, and the government. And they include bargaining and changing assumptions and positions. Remember that it can be a give-and-take situation resulting in an agreement that both parties are comfortable with. The primary objective is to increase the proposal's potential for award by increasing its value based on the requirement. So when do you start negotiating? ESPC negotiations should start after contractor selection and continue throughout the development process. Next slide.
This is a quote from Henry Petroski, a great engineer who studied failure analysis. I'm going to read some key words from what he wrote: "Engineers…are not superhuman. They make mistakes in their assumptions, in their calculations, [and] in their conclusions. That they make mistakes is forgivable; that they catch them is imperative. Thus it is the essence of modern engineering not only to be able to check one's own work but also to have one's work checked and to be able to check the work of others." These are words of wisdom, and a reminder of the need for the government to perform its due diligence in reviewing proposals. Next slide.
The agency works together as a team within their areas of responsibility. And having said that, most people know that within the federal government, the agency contracting officer is the lead negotiator, but negotiates best with the assistance of the technical team and others as necessary. The agency contracting team, consisting of the contracting officer and possibly a contract specialist, the price analyst, and legal staff that are experienced with the applicable procurement laws and regulations; the agency CO has the ability to bind the government and to sign the contract.
And then there's the agency technical team. They know the government's requirements and their technical skills and their expertise is critical to refine the project, including the areas of design, installation, operations, maintenance and repair of the ECMs. Remember that teamwork is critical, and that's what achieves best value to the government. Next slide.
Now, this is best practices and processes for agency management of proposal or IGA review. And staying involved throughout IGA development and effective communication between the ESCO and the agency is critical to achieving a comprehensive and responsive proposal. FEMP is frequently asked to provide lessons learned and best practices for management of the proposal review. We try to provide this information throughout our presentation, and we've summarized some of the key elements for managing the proposal review in the following slides. Next slide, please.
You should know in advance your agency process for proposal evaluation and approvals. And that means that first of all, there has to be a plan, and that plan should include not just the proposal evaluation process, but also a plan to brief those who provide concurrence and approvals prior to award, as well as getting those individuals who have the ability to stop the project involved at the right time. And next, it should be a plan that everyone can agree with. The plan includes knowing who should be involved, when they should be involved, and what they need to review. It also includes identifying the key individuals or functions who need to be briefed so that timely approvals can be obtained. And typically, this includes at least the individuals listed.
ESPCs are FAR Part 15 negotiated contracts, so the CO will be involved throughout the evaluation process and lead negotiations, and they have a responsibility to ensure that all required documentation and approvals are obtained prior to award. Program engineering, including the future COR, will typically lead the evaluation of the proposal, and be involved throughout the evaluation process. And they'll review all technical aspects of the proposal. The others listed will get involved at the appropriate time. Maintenance, or facilities, has a critical role in certain aspects of the proposal review, and the budget office should be brought in to ensure that the payment structure is within acceptable parameters. COs submit their files for legal review to establish legal sufficiency prior to award, and usually, ESPCs require some sort of management, board, or headquarter concurrence or approval prior to award.
So the overall objective is to have a plan in place that ensures the agency performs its due diligence, and of course, communication is so important as you go through this part of the process. Next slide, please.
ESPCs are long-term contracts. Once the contract is signed, it's binding to both parties, the ESCO and the agency. So read every word of your contract and make sure you understand what you're signing up for. Usually, the task order contract consists of at least these three items: the ESCO's proposal, incorporating the results of the investment grade audit, and usually it's incorporated into the contract, or else it's included by reference; there's the agency task order request for proposal, as modified during negotiations, and this includes Section C, the statement of work, and other sections that identify the agency-specific requirements, and this will actually be the basis for the ESCO's proposal; and then there's the Department of Energy's master contract, and this is for overall ESPC terms and conditions.
Proposal revisions will likely be necessary before award, to incorporate the results of the agency evaluations and negotiations. That may also mean that the task order RFP, as incorporated into the contract, should be updated to reflect negotiations. Next slide.
Additional best practices for coordinating proposal evaluation include: Well, convene the evaluation team as early as possible. The team should establish what their review process will be, as well as the review schedule. And discuss this with the ESCO-they're part of the team-because they need to plan their time accordingly, also.
So, get a plan. Usually, the agency evaluation team is established in advance and includes members with different specialties. Sections of the proposal should be assigned so that they're evaluated by the team member within their area of expertise. It's important to set aside sufficient review time to accomplish the task and make the evaluation a priority. Some teams make a commitment to all meet in a conference room for a couple of days or a week in order to have uninterrupted time for the evaluation, as well as completing the necessary file documentation.
It's very important to complete a comprehensive evaluation in a timely manner and deliver on time to meet overall milestones for award. And lastly, think ahead to the next step. For example, if you know who the reviewers will be, such as headquarters or an agency review board, communicate-start briefing them in advance to get them up to speed. Educate them on the ESPC process and what the project will deliver. Determine if you need to get on their review calendar in advance to keep the process moving forward. Thinking ahead helps to prevent delays. Next slide.
It's important to develop a process for handling all the evaluation comments to ensure that they are adequately addressed and that there is documentation on their resolution.
Consider designating one individual, and it's usually the project facilitator, to collect and consolidate all the comments. Most proposals are large, meaning hundreds of pages long, so there can be a long list of comments, concerns, requests for clarification, and questions. There are sometimes 2 or 3 evaluators that will have the same question, so it's important that someone read the comments and consolidate them into a succinct list for the ESCO to address. The CO will coordinate discussion of the questions and concerns with the ESCO. Deliver the evaluation comments to the ESCO on schedule so the ESCO can manage their time effectively.
Additionally, agencies should review the ESCO's responses to their comments in a timely manner. The back-and-forth can be time consuming, so keep the team involved. And build this back-and-forth comment resolution period into your schedule.
Make sure that required changes are agreed to and included in the task order, potentially by asking the ESCO to provide a track-changes version of the proposal or submitting new and revised proposal pages to ensure they are made and incorporated.
When I worked in the field, there might be a list of anywhere from 20 to 200 comments, and every comment would be listed on a row, along with the ESCO's response, and then an indication that the agency agrees with the resolution. Make sure that all comments are adequately addressed. For example, it's not good enough to say that something will be resolved after award. You keep talking until every line item is resolved before moving to award. After agreement on all comments, this document should be included either as an appendix to the proposal or as an attachment to the task order. If there is confusion years later, this is the document that can provide clarity on how certain topics were to be handled.
And of course, make use of FEMP resources that are available to you.
Now I'm going to turn this over to Bob to talk about the technical aspects of the proposal evaluation.
Okay, thanks, Deb. I'll be going over the technical aspects of reviewing proposals throughout this section. And we call this section "The Crystal-Clear Proposal," but sometimes the proposal's not necessarily crystal-clear. In order to get that clarity, you're going to have to read every word of the document to make sure that what was discussed during the investment grade audit is reflected in that proposal. Your final proposal is comprised of two primary sections, Volume I, which is the Technical Proposal, and Volume II, the Price Proposal. And you can find the format requirements for the proposal in Section H.6 of the IDIQ contract.
Today, we'll just be covering the aspects of Volume I, the Technical Proposal. And essentially, the Technical Proposal is a very robust version of the elements that were conveyed in the Preliminary Assessment. These assumptions that were made in the Preliminary Assessment are going to be validated through the investment grade audit phase, and it should be reflected in this Technical Proposal. You'll have energy conservation measures, or ECM, descriptions, and the measurement and verification plan with its associated baselines. And you'll also find within the Technical Proposal the management approach. The management approach details the contractor's organization and how they plan to staff the project during the implementation period, as well as the performance period. And then the risk and responsibility matrix will also be included in this section, and Scott will cover this in greater detail in the later sections. So we'll go to slide 21.
So we'll spend this portion of the session discussing some of these various elements of the Final Proposal in greater detail. For topics like the M&V Plan and Risk and Responsibility, we have dedicated sections later in this presentation that'll go into even greater detail.
But your first step as an agency will be to establish a review team to examine all of the elements we'll discuss here today. You'll want to outline a process that you'll go through for the review as well as a schedule associated with that process.
And as Deb had mentioned, team members will draft comments associated with the proposal. You'll consolidate those comments, eliminate redundancies, and possibly separate those that might be more appropriate for a price negotiation discussion. Those comments will go to the ESCO; the ESCO will provide a response for each. You'll review those ESCO responses and if they are acceptable, you'll move forward. If not, further discussions or negotiations will be required. You're likely going to have a sit-down meeting with the ESCO in order to resolve these outstanding issues associated with each of those comments, as Deb had mentioned. So we'll go to slide 22.
So the heart of the proposal pertains to the work scope. Reviewers should examine the list of recommended ECMs-these energy conservation measures-and ensure that they still make sense, that they're aligned with current project goals and expectations, and then determine whether each ECM is adequately described. Generally, one should be able to read the description and have a basic understanding of current practices, recommended actions, and how energy savings will be achieved.
Some of the questions you'll want to explore are:
Is each ECM description complete and understandable? Are you left with a clear understanding of what's going to be done?
Does it specify what physical locations at your facility will be affected?
Does it go into detail of the potential interface with existing government equipment?
Does the description provide sufficient information pertaining to the brand and model and size of equipment? Do they have "cut-sheets" included in this document?
And does it provide sufficient information pertaining to any required utility interruptions or physical changes to the existing equipment or facilities that will be required in order to get that ECM implemented?
So if the answers to these questions are "Yes," then you can be reasonably certain that the ESCO has adequately addressed this requirement of the proposal.
Overall, there should be no surprises here. The ECM content should reflect the summary of discussions that you had during the entire Investment Grade Audit process. Alright, we'll go to slide 23.
So reviewing energy and cost savings is a critical step because they become the basis for the ESCO's savings guarantee.
The format for a proposal requires the ESCO to provide detailed engineering calculations which support the project's guaranteed savings. And the ESCO should compile these supporting calculations in a logical format that starts with the methodology, then proceeds to assumptions, followed by the calculations themselves.
And when we say "energy and energy cost savings," we're also including water savings, so keep that in mind. So the ESCO should be detailing how much energy and water is saved, and then apply the fuel costs to calculate the cost savings. It's important to make sure that the utility rates they're using in their calculations are the rates that you agreed to with the ESCO earlier. The ESCO will also have to detail energy related cost savings, and we'll define that and discuss that in a later slide.
So from the calculated cost savings, the ESCO will make a guarantee. Now, keep in mind, this is a financial guarantee for the overall project. The individual ECMs themselves do not have a guarantee. The ESCO typically guarantees between 90 to 95 percent of the calculated savings, depending on the technology involved, the complexity of the ECMs, the M&V selected, who is handling operation and maintenance or repair and replacement. Those all factor in. So we'll move to slide 24.
So reviewing the engineering calculations and assumptions essentially involves ensuring that the savings proposed are realistic and reasonable. This includes verifying that the baseline configurations and operating parameters are accurately represented. If the contractor is proposing to save you 75 percent of your energy in your facility, that might raise a red flag because that may be an unrealistic expectation or an unrealistic calculation. Typically, that's not going to be the case to have such a high percentage.
The calculations should be based on sound engineering principles and industry standards and practice. You'll want to check the math, spot check those elements in large spreadsheets to make sure they have the math correct because sometimes folks do make mistakes. We've seen it happen.
So you'll want to review the assumptions used for calculations. Two types of assumptions must be reviewed: those related to operations and those related to performance. Operations-related assumptions deal with occupancy, operating schedules, temperature set-points, etc. Generally, building tenants can provide reliable corroboration or correction of these assumptions. In particular, there are assumptions in these proposals that only the site folks can review and validate. Your project facilitator will be helping you review these proposals, but there are some elements of the scope of the work that you need intimate knowledge of the site in order to determine whether the assumptions are reasonable or realistic. Let's go to the next slide, slide 25.
So as far as energy simulations and spreadsheet models are concerned, these are both allowable for use as calculations by the ESCO. The thing you don't want is to have them in a PDF format. You'd like to have the input and output files for energy simulation models so that you can evaluate those and see if they're reasonable. And if the ESCO's using a spreadsheet model for calculations, you'd like to have the live Excel file so you can look at what formulas are being used and evaluate those appropriately. If we're talking about simulation models, we want to make sure that complies with ASHRAE 90.1, Appendix G, and that they're calibrating their models using meter data in accordance with the FEMP M&V Guidelines. We'll move to slide 26, talk a little bit about O&M savings.
So, payments for an ESPC can also come from what we call "one-time" or "recurring energy-related cost savings." And recurring savings generally result from a reduction in operation and maintenance expenses. Savings can be in the form of either materials or labor. And typically, sites will include material cost savings associated with an ECM. For example, in a lighting ECM, if you go from a conventional fluorescent fixture to an LED technology, the LEDs will typically have a longer service life than the conventional fluorescent fixtures. So there'll be a reduction in the frequency that you're replacing lamps. There would be a reduction in the material costs associated with that ECM on an annual basis. And that savings can be applied to your project.
Consider another example, where you might replace a boiler that was a maintenance headache with a newer technology that didn't need all the maintenance that was required of the older boiler but you don't lay off the boiler operator or the repairman, you just reassign them. Now, granted, you'll have some savings associated with that person doing something different, but you're not going to be able to realize those savings financially, because you made that reduction in staff. Or the dollars aren't financially tangible. In some cases where maintenance is handled by a sub-contract, you can terminate or modify that sub-contract for service and include that savings as an economic element of your project.
An additional example of a one-time cost savings: Let's say you were planning on replacing a chiller before your ESPC project, but instead of replacing the chiller with a like-for-like unit, you decide to incorporate that into the scope of an ESPC project. You do a bunch of other energy conservation measures, and then potentially you could reduce the size or capacity of that chiller and its associated costs, and/or you could put in a more efficient unit. In those cases, you can claim that savings as a one-time savings versus the recurring savings examples that we spoke about.
And I'd like to point out, finally, that FEMP has a guidance document associated with accounting for savings within federal ESPC projects. It's called the Practical Guide to Savings and Payments, and it's available on the FEMP website. If you have any questions in connection with energy-related savings, you can always ask your project facilitator, your federal [project executive], or refer to the FEMP guidance. Alright, we'll jump to slide 27, talk about consequences of some of our decisions.
So these are all elements of the ESPC contract and processes that are interrelated. Sometimes you think you can make a decision in absence of its context. But it can affect elements of the overall project economics and the proposal.
So what do I mean by the first bullet, "Silence means agreement?" If you don't like something, you'd better speak up and make sure you have what you want and it's reflected in the language in the final proposal.
And then in the area of understanding and agreements, if they're not documented, they no longer exist after award. So you'll want to make sure that you document it in the award documentation-either in the final proposal or the TO RFP.
In the area of savings calculations, you want to make sure that these are reasonable and realistic. Again, they apply to both the energy and energy-related savings. For a percentage savings guarantee, the greater the percentage savings guarantee, the greater your payments and the sooner you can retire the debt. And that's something to keep in mind. The lower the savings percentage, the lower the risk theoretically to the contractor, but the longer it takes you to retire the debt.
Then, in the area of escalation rates, we would recommend that you use what's called the Energy Escalation Rate Calculator tool, or the EERC tool. It's based on NIST calculations and has been found to help avoid overly conservative or excessively high assumptions for utility escalation rates. If you're not using the Escalation Rate tool and you're assuming a rate that's too low, you're limiting yourself and your purchasing power of what you can initially install, and/or you're extending the term of the contract. If you're choosing too high, you may have a decoupling between what the actual escalation rates are and what you've estimated. You want to be as accurate as you possibly can, based on the best available information for your escalation rates and the EERC tool serves this role.
In the area of M&V protocols, the level of rigor of the M&V protocol will drive implementation and performance period costs within your project. M&V approach should be balanced against the ECM performance risk and level of savings so that you're not adding unnecessary costs to your project.
And finally, the Risk and Responsibility Matrix: Scott's going to go into this in more detail, but essentially here, the whole point is this is a risk management type of a contract and you need to know the potential impacts of your decisions. Alright, so we'll jump to slide 28.
So again, just to summarize a review of the energy and cost savings: Again you want to validate the math. You want to make sure that the assumptions are reasonable and realistic. And you want to make sure that you agree with the energy rates and labor rates in the proposal.
As a final note on ECM review, traceability of the energy and cost savings is critical in the review of the proposal. Reviewers should ensure that the energy and cost savings by fuel type for each ECM is properly identified and that the ESCO has properly transferred these values to the financial task order schedules that we mentioned previously as part of this price proposal. These will appear on Task Order Schedule #4 and will be a full detail of all of the particulars related to each ECM.
And lastly, there are folks at FEMP that can help you with advanced technologies, such as the staff at the national laboratories. So if you have any questions, you can ask your project facilitator or federal [project executive] and they can get you in touch with these individuals. Alright, we'll jump to the last slide, slide 29.
So the management portion of the proposal consists of the ESCO's organizational structure for the project. It also contains, as we mentioned, the Risk and Responsibility Matrix, and a proposed level of O&M services, as well as repair and replacement services, and a project timeline. Scott will cover the RRPM in greater detail, but in the area of operation and maintenance services, you want to make sure that whatever is proposed complements your resources and expertise. You don't want to take on maintaining a new co-generation plant when you don't have the expertise to do so.
And on the topic of training, look at it and, you know, is what's proposed adequate for your staff both now and in the future? You may want to consider periodic re-training or digitally recording the training for future use.
Overall, is the timeline appropriate for an ESPC? Keep in mind that you can't exceed a 25-year contract term for both the implementation period and the performance period. That's the total maximum. We recommend that you generally do not exceed 24 years because things change during construction and you'll need some buffer in there. There may be some delays and there may be some modifications that occur after the project is awarded. And if you have no more time left on your contract, the only way to fix that problem is to apply funds to it. Sometimes that can be painful and sometimes you don't have funds to do that. So we suggest that you don't push up against the limit.
And then, finally, do you have confidence from the proposal that the ESCO has conveyed an understanding of how to implement the ECMs to meet your site's needs?
So this concludes the content I have for this section, and now I'm going to turn it over to Scott for the Risk, Responsibility, and Performance Matrix section.
Thank you, Bob, I really appreciate that in depth view that you gave on that section. I'm going to talk about the Risk, Responsibility, and Performance Matrix, or we refer to it as RRPM. This certainly is a risk mitigation tool, and very useful for this particular contract. Next slide.
So, what is risk? Well, to the ESCO, there's a chance the guarantee will not be met and a shortfall will be identified. Bob talked about this and talked about how the ESCO will guarantee a percentage of the savings. Basically, the ESCO will have to pay the agency the difference if there is a shortfall. Well, what's the risk to the agency? Well, the chance that the savings you are paying for are not fully realized. For example, the savings might not exceed the payments, which is crucial to this particular contract. Next slide.
So the RRPM is a summary, and it's embedded in the contract, and it's in a very condensed, easy to read format. The purpose is to educate you about the risks: how the contract elements affect the costs and savings, how to tailor the task order to match agency needs. And, again, it's a very defined structure for decision making. You talk about this throughout the agreement. Much like Deb Kephart was talking about, you negotiate throughout the entire process. This is documentation of the agreement that you do make with the ESCO. The required element of the contract is Attachment J-7 of the IDIQ contract. Slide 33, please.
Here I'll talk a little bit about the logic and cost-effectiveness that drive the responsibility allocation. The responsible party plans to cover the costs: The ESCO in the ESPC price, the agency from savings and budgets. Unforeseen costs are paid by the party who caused the costs, or by the party who is responsible for that risk area.
Now, there's three main categories in the risk-responsibility matrix. They are financial, operational, and performance. And connected to each of those main categories are subcategories which I'll tell you a little bit about. But from a large perspective, from the overall perspective, the financial risks are most commonly allocated-it's usually mixed between the agency and the ESCO. Part of financial subcategories of risk include interest rates; energy prices; construction costs; measurement and verification costs, which Bob will talk about in the next segment; non-energy cost savings is another subcategory; as well as delays, whether it's the contractor's fault or the agency's fault; and really what happens if there's major changes in the facility. Those are all financial risk subcategories.
Now as far as operational risks, what do we have? Well, we've got operating hours. Will they change or alter by the cause of the agency? Will they be totally wrong because the ESCO miscalculated or didn't do the data logging properly? Other subcategories of operational risk include loads-equipment loads-they change over time; weather; user participation, because of course, many ECMs require the user to participate in operating them. That's a particular risk.
As far as performance risks and subcategories, you certainly have equipment performance, you have operations-the responsibility for the operations is absolutely negotiable in this contract-you also have preventive maintenance issues and equipment repair and replacement. As we know, equipment doesn't last forever. These are long-term contracts, up to 25 years, and equipment has a finite lifetime, and certainly in here you will see who and how equipment repair and replacement will be done. Next slide, please.
So it's negotiated operation and maintenance and repair and replacement of installed equipment. It's the ESCO's responsibility for all of this by default in this contract. However, the O&M and the R&R tasks can be assigned to ESCO or the agency, depending on what's appropriate and how the whole contract is configured. The general approach on these allocations of responsibility are simply outlined in the Risk, Responsibility, and Performance Matrix. Basically, for each ECM, the task order should specify: who will carry out the operation and maintenance and risk-repair tasks, how performance will be verified, and what to do if the tasks are not performed. It basically will show the cause and the effect throughout the various pieces of the Risk, Responsibility, and Performance Matrix. Next slide, please.
So, if the government performs the operation and maintenance, the task order will contain: the performance requirements-the whats and the whens-okay? So, if you've got performance requirements, will the ESCO be responsible for it, or will the agency? It depends on what makes sense. If the energy management control system is being installed, will the agency operate it or will the ESCO? In ESPC you could go either direction and allocate the performance requirements of that control system, as well as equipment, whether it be boilers or chillers.
Oftentimes, there's requirements for the agency to perform record keeping, particularly if the agency is responsible for the operation and maintenance. Certainly, the ESCO is on the hook for the entire savings, and so they-it is right and correct for them to ask the agency to keep records, to show them that they have done the prescribed maintenance.
Also there are provisions for the ESCO to monitor the agency. This may be various data dumps of information from the energy management control system, log inspections, perhaps there'll be a requirement to take a photograph and various measurements of equipment. Overall, the ESCO is still responsible for the operation and maintenance and repair and replacement.
It's very standard in these contracts that the ESCO provides the operation and maintenance manuals defining the procedures. They absolutely provide training to the agency, and in many cases, this is recorded so it could be shown to new employees as there's a rotation of employees who move up or retire. And it also notifies the site of any deficiencies in O&M and R&R.
Now, we all realize, if the agency does not meet its obligations for the operation and maintenance or repair and replacement, the guarantee may be compromised. And in this case, your contracting officer and a technical team will need to come in and will have to have extended discussions with the ESCO to talk about why the guarantee was compromised, who is responsible, and who should pay. Next slide, please.
Lessons learned are threaded throughout our whole ESPC training segments. RRPM ensures that the important risks are addressed and responsibilities are assigned. The dialog fosters mutual understanding of the entire deal, the entire ESPC deal you have struck. The matrix is certainly a key item in review of the Preliminary Assessment, but it's also very key in the Final Proposal. Use the RRPM to guide your proposal review. The details in the M&V Plan and in other parts should not conflict with the Risk, Responsibility, and Performance Matrix. This does happen, and you should expect it to happen, and you should be aware of these conflicts. And certainly before the contract gets signed, you should correct all the conflicts in the body of the proposal that conflict with the Risk, Responsibility, and Performance Matrix. Don't take a responsibility that your organization can't handle well. This has been a lesson that has been learned over and over by various agencies who have entered into ESPC contracts. But, certainly do take responsibilities that fit well within your organization. This concludes the RRPM section, and I will turn the next slide over to Bob, who will talk about measurement and verification.
Great, thanks, Scott. So, some people think that M&V is just for technical staff, but it is central to the enforcement of savings guarantees within ESPC projects, so a basic level of understanding is important for anyone engaged with a project. M&V is tricky sometimes, since what you're trying to do is measure the absence of something-in other words, energy usage that's gone away-even while a lot of other changes at the facility have gone on at the same time.
For the purposes of today's webinar we'll be keeping things relatively light and just covering the basics of M&V. So we'll go to slide 38.
So what is M&V? Why do you need it? M&V are the activities aimed at determining whether or not your savings guarantee, as proposed by the ESCO, is being met. The M&V is required both under the DOE contract as well as required by statute.
There needs to be at minimum an annual measurement and verification audit and report that's conducted in order to be responsive to the statutory requirements.
When M&V is done right, it can bring benefits beyond meeting just the legal requirements. A good M&V plan, when carried out well, can accurately assess both the cost and the energy savings, and also make sure that you discover performance problems with the equipment so that they can be fixed.
The method of M&V chosen for an ECM also allocates the risk between the agency and the ESCO, and we'll try to have a few examples about that as we go through M&V options a little bit later. So we'll move to slide number 39. Talk about some basic M&V concepts.
So measurement and verification could be considered a bit of an art as well as a science because it's a balance between savings assurance and cost. You don't want to spend more money validating or getting some assurance that you're getting the savings than those savings are actually worth themselves. It needs to be proportional. The rigor that you use for measurement and verification needs to be proportional to the magnitude of the cost savings associated with the energy conservation measure, as well as the risk that the performance will not occur and that the savings will not manifest themselves. It should be proportional to those two elements. A good M&V plan requires that one or more key performance parameters be measured both prior to and after retrofit. If an M&V plan is weak it's just a bunch of words on paper. You really need to review the plan thoroughly and ask yourself, "What is the risk to the contractor?"
As an additional note, your project facilitator can help you with the review of measurement and verification plans if you do need additional assistance in this area. We'll move on to slide 40.
So the M&V that's proposed in the contract should be in accordance with the latest FEMP M&V Guidelines, which is Version 4.0. This came out back in November of 2015. So you'll want to make sure they're referring to the current version of the guidelines. The FEMP Measurement & Verification Guidelines is an application of the IPMVP or the International Performance Measurement and Verification Protocol, and it contains guidance on how this M&V is to be approached within federal projects.
The guidance is available on the FEMP website, in addition to some other related resources, such as the Government Witnessing Guide for Measurement and Verification, and Guidance on Reviewing Measurement and Verification Plans and Reports. Given that Version 4.0 is new, I'll highlight a few changes that were incorporated into the document. In general, it's no longer specific to federal projects. A lot of the references have been removed to federal-specific components, and all of that relegated to one subsection. So there still is guidance on how federal entities are to apply the guidance. But overall, it reads more broadly.
It made some efforts to strengthen Option A. This now suggests that there is some annual measurement activity associated with Option A, even if it's just of a single key parameter. We'll talk about a few examples a little bit later.
And then it goes and clarifies Options C and D approaches as being more short-term approaches, and reverting to other options, either A or B, after an initial usage of C or D.
It provides additional examples of M&V approaches. The previous guidance had a handful of ECM examples; we now have 19 different application examples for you to look at. One of the larger changes, and this impacts the ESCOs, is that the modification of the annual M&V report template for added clarity. The guidelines have always asked the ESCOs to detail costs, savings impacts from any deficiencies within the projects. However, a lot of this activity wasn't occurring on a regular basis. There's now a new table within the templates that will hopefully provide greater clarity as to what's happening in your project when it comes to contractual requirements versus the actual performance of the equipment.
And then finally, there's a requirement in there that ESCOs begin adding Measurement and Verification Report findings into the new eProject Builder system, which, for those of you who aren't familiar with it, this is a new central repository created by FEMP for housing all ESPC data, both the financial task order schedules as well as the annual performance data. Again, this current guideline can be found on the FEMP ESPC Resources website. Alright, so we'll go to slide number 41, talk about M&V throughout the ESPC process.
The DOE contract prescribes the M&V process in general, and specifics are defined in each task order.
ESCOs are required to use templates for the M&V plan, the post-Installation reports, the annual M&V reports that we'll talk about here. And the templates are attached to the DOE IDIQ contract as well, on FEMP's webpage.
But shown here are some of the key elements associated with the measurement and verification process. The first one's the baseline, which is typically defined in the investment grade audit phase and reflected in the final proposal. This is an important element of the overall M&V process because depending on the energy conservation measure and the methodology that's used to validate the savings, you may not be able to re-establish or look at that baseline again after the retrofit.
Then there's the Measurement & Verification Plan itself. I think this is the most important element of the process because it defines the activities that are associated with the M&V process for the entire term of the contract. Some of the things that it contains are:
Documentation of all assumptions and sources of data
What will be verified
Who will conduct M&V activities
What's the schedule of those activities
It contains the details of engineering analysis that's to be performed
How energy savings will be calculated
As well as the utility rates and how they'll be used in the cost savings calculations.
Then we have the post-installation report. This is submitted by the contractor after the installation is complete. And it should verify and demonstrate that the ECM has the ability to perform and generate savings per the Measurement and Verification Plan.
And then we have the annual Measurement & Verification Reports. The activities are defined in the Measurement and Verification Plan. And then the findings or the results of those activities are documented in the annual M&V Reports. Those are required at minimum an annual element. There may be some reasons you may want to have them more frequently, depending on your energy conservation measures or at least some sub-element of a complete report.
Also included in these M&V activities is reporting on maintenance and operation and repair and replacement activities associated with the ECMs and whether that's performed either by the contractor or by the government. So that should be documented in those annual reports as well. One of the things I would recommend is having a meeting with the ESCO when they submit their annual report in order to go over those findings and have an understanding of what's taking place with the equipment and its operation. Alright, we'll move on to slide 42, talk about baselines.
So typically the ESCO defines the baseline as part of the investment grade audit. Baseline physical conditions-such as equipment inventory and conditions, occupancy, control strategies, and so on-are typically determined during the IGA through surveys or inspections, perhaps spot measurements or short-term metering activities. And baselines are established for the purpose of calculating savings by comparing the baseline energy use to the post-installation energy use. It is the agency's responsibility to ensure that the baseline has been properly defined.
And in most cases, after the ECM has been installed, it's impossible to re-create the baseline. Therefore, it's important to properly define and document the baseline conditions. Deciding what needs to be monitored, and for how long, depends on factors such as the complexity of the measure and the stability of the baseline, including the variability of equipment loads and operating hours, as well as other variables that affect the load.
And baseline data are also used to account for any changes that may occur during the performance period, which may require baseline adjustments. Sometimes baselines are affected by weather. This is typically normalized by using what they call TMY-or Typical Meteorological Year-data. It's weather data, a 30-year average that's typically local to a facility. It can be used as part of a regression analysis. Alright, so we'll go to the next slide, slide 43, and talk about some of the various measurement and verification options.
So the FEMP M&V guidelines and the IPMVP, which I mentioned earlier, they group M&V methodologies into four categories: Options A, B, C, and D. The options are generic M&V approaches for energy and water savings projects. Having four options provides a range of M&V approaches to use, depending on the characteristics of the ECMs.
And these options differ in several significant ways:
One is the level at which the savings are measured: Are we looking at a retrofit isolation, just that measure, where the individual system is measured? Or are we looking at things from a whole building perspective, where several systems have been retrofitted and they all interact with each other?
And then there's the duration of the measurement: Are we taking a one-time spot measurement before and after installation? We could come in short-term, say, for several weeks or even years at the beginning of the performance period. Or do we come in just periodically, on a quarter or annual basis, but yet recurring? And then, or course, there's continual. We can have things monitored such as a photovoltaic system. We can monitor the generation output continuously.
In some cases it may be reasonable to assume that key values will be held constant without performance-period measurements. For example, savings from a lighting retrofit are usually determined by measuring operating hours through the investment grade audit, and then fixing those operating hours at a level for the purpose of calculating savings throughout the performance period. And this would be perfectly acceptable.
And there at the bottom of the slide, just some general rules of thumb that you might consider when examining the expenses related to M&V:
Up-front: We typically see that the costs average about 3 percent of the total project investment.
And then when we shift into an annual performance period mode, those annual M&V expenses represent about 3 percent of the annual savings.
But of course, more complex and interactive ECMs may justify more M&V effort, and thus carry a higher cost related to those ratios there.
Alright, we'll move on to the next slide, slide 44, and talk a little bit more about Options A and B versus Options C and D.
So this chart shows the four main M&V options. So A and B are what we call retrofit-isolation methods, whereas C and D are considered whole-building or whole-facility measurements. So again, as the name might imply, "retrofit-isolation," we are looking purely at that particular measure on its own.
So a good example of when we might use an Option A is in the case of lighting. We're looking at just the lighting system. We may come in and take one-time measurements of wattage, before and after; we may measure usage, and then consider those fixed for the period of the contract, and then just do annual inspections.
An example of an Option B: It's also retrofit-isolation, but it typically involves continuous monitoring, or regular recalculation of the savings on an annual basis. Solar photovoltaic systems are a good example of that, where we may have a production meter on the system, where we can continually monitor the output, and therefore have ongoing, year-over-year assessment of the savings for that measure.
When we move into the areas of whole-building methods, we're talking about Options C and D. Option C is a whole-building approach. Let's say you have a case where you're doing both a new boiler, new controls. You might have many interactive effects, it's too difficult, sort of complex to isolate those measures. You would then be able to utilize metering of an entire building, or perhaps even a facility, and then, through a series of regression analysis comparing that metered data to your utility bills and taking into account things like weather and occupancy, we can try to correlate savings through analysis of those utility bills.
Option D: Also whole-building, but this is a computer simulation, a model of your building, and this would be commonly used if you had, again, interactive effects from many ECMs. A good example is just envelope measures, very difficult to isolate by themselves, but if you were to model that, you can estimate the savings from there. Alright, we'll move to slide 45.
So we talked about the savings guarantee. And as Scott mentioned, the savings must exceed the payments in every year of the contract. At least that's how DOE has interpreted the federal guidance.
There's two types of savings that we can use to pay the ESCO. We talked about these earlier. One is energy and water cost savings. The other is energy- and water-related cost savings. And again, if you're going to include it in the economic analysis of the project, the savings-the energy-related-must be realized, because you're going to make payments based on that.
The bottom line is, does the agency have confidence that the savings will be achieved? Remember that you want to be asking yourself the question, "Under what circumstances will the contractor not achieve the estimated savings or the guaranteed savings?" If it's a complex measure with a lot of perceived risk and the ESCO is proposing Option A, you have to ask yourself if you're comfortable with that approach for the life of the contract. Alright, we'll move on to the last slide, slide 46, talk a bit about M&V best practices.
So, in the area of baselines and fixed parameters, we've discussed the importance of these elements as cornerstones to the savings calculations. Make sure you're comfortable with them, as they will be with you for the life of your project.
Stay involved through the life of the project. Agencies' responsibilities don't end after construction and project acceptance. Be sure to witness the annual M&V activities when the ESCO makes their annual M&V audit. And be sure to document this witnessing activity in your contract file. Auditors are looking for this level of contract management.
Review annual reports thoroughly. Make sure things make sense, question the ESCO on things that are unclear, and have them correct report errors or add detail where needed. There are FEMP resources to assist you with the performance contract period requirements, such as guidance documents and FEMP's Life of Contract service, which Kurmit mentioned earlier. The DOE Life of Contract staff located at Oak Ridge National Lab reaches out to each project before and after the ESCO's annual M&V audit to ensure staff is up to speed on their requirements for their project, and we can offer assistance and training where needed.
So that's all I have today on the topic of M&V, and now I'll turn it back to Scott for the next section.
Thank you very much, Bob, you certainly took a lot of the mystery out of M&V and answered a lot of key, key questions, I believe, for the audience. I'm going to talk about best practices and lessons learned. And again, I want to reiterate that best practices that we're going to present here and that we have been presenting throughout the training have been based on dozens and dozens of projects that we have done through the FEMP program. I also want to say that the best practices that we feel are very, very important are recorded and identified on our website, so you as an agency representative could go to our website and could print out the entire copy of our best practices and review that with your project team. Next slide.
So, critical outcomes for review, that are certainly best practices. And again, much of what I'm going to talk about, Bob and Deb already might have gone over, but we'll just reiterate some of these items.
The proposed energy consumption baselines and fixed parameters for calculating savings should be sound, and Bob talked about this. Very critical that your organization agrees with what the proposed baseline was, and how it was developed by the ESCO.
Second bullet: Guaranteed savings from the ECMs are reasonable, and the given analysis of assumptions and savings received on similar projects are sound. Do you agree with it as an organization? Did you have your technical people look at it? Does it make sense from a holistic point of view?
Third: Is the measurement and verification appropriate for the proposed ECMs? Bob talked about all four methodologies-A, B, C, and D. And is what the ESCO's proposing-as what you accepted-is it sound? Many agencies do have prescribed M&V protocol for various ECMs, and you should know if your particular agency does have pretty much canned and boxed protocols for your various measurement and verification with various ECMs.
Is the Risk, Responsibility, and Performance Matrix consistent with the project scope of work and pricing in the IGA and proposal? Again, remember, I spoke about sometimes there are conflicts between what's in the matrix and what's in the proposal. You have to reconcile that and make sure everything is straight and sound.
Lastly, do all stakeholders understand the Risk, Responsibility, and Performance Matrix and the details of any agency performance-period responsibilities, and the needed resources are committed? This is crucial because you, as an agency, may be agreeing for the next, for example, 20 years, to do something, and are your resources committed, and do you have management approvals to do whatever you committed to do, as stated in the proposal? Next slide, please.
Communication is critical. Deb Kephart talked about that in the beginning. Communicate often and frequently. We certainly would like to see bi-weekly meetings happen, where there are notes being taken by the ESCO, and the notes are being published. You'll certainly refer back to this time and time again, sometimes one or two or three years after the event.
The longer it takes to develop a project, the lower the probability for success. I, myself, as a federal project executive, have seen this time and time again: Projects that languish on the vine, that go through extended periods of development, oftentimes fail. Government people change, leadership changes, sometimes the ESCO folks' team changes as well, the longer these things go on. We highly recommend that you have a good, solid schedule to implement a project, anywhere from 12 to 16 months, and just get it done.
Thirdly, avoid scope creep. Oftentimes, there's a lot of afterthought in the development of a project, and scope creeps up by the direction of the agency. Be careful about this. And early on, you should really finalize your scope and what you want to do.
Your decisions certainly have consequences. Maintain complete project documentation during all stages of project development. Now, I know I sound like a contracting officer here, and that's okay, because documentation is so, so important in these. These are long term contracts, and you should document everything and have very solid files on what you do and what you decide.
Document your technical evaluations. This is very important, to document how you made decisions, what kind of technical input you got from various folks from your own agency. Again, take a graded approach for time management. Prioritize those activities based on the importance and value. You're going to be doing a hundred different things in the development of your ESPC project, so make sure you put the important time and effort, and number of hours into the important things, into the high-level things. Not necessarily miniscule details that won't yield you an exceptional project that you're looking for. Next slide, please.
This slide talks about your resources. Be sure to use your FEMP Project Executive as your first, best point of contact. In the next slide, I'll show you a map on the Project Executives in your region.
I also want to talk about your DOE Project Facilitator. This is a required person in all IDIQ contracts, and these people are certainly experienced, and they're expert consultants to guide you and to help you with your ESPC project. Certainly, these experts have seen dozens of projects before. They will your advocate and be on your team to help steer you clear of the hazards on ESPC projects.
Also, FEMP has an ESPC team of national lab experts to help. If you have some interesting conservation measures that go beyond your expertise in your agency, such as cogeneration, biomass, landfill gas, wind turbine technology, geothermal heat pumps, for example, FEMP and the national lab partners could help you understand these technologies and steer you in a good direction to get great agreements with your ESCO on how to implement these technologies.
There's ESPC resources on the FEMP website. You can see the ESPC homepage on the URL on the bottom here, of slide 50, as well as ESPC resources, guidance, and contract documents. This is very important. We have an enormous catalog of guidance and resources. And if you want to see a sample IDIQ contract with various modifications, you could find that at the last URL listed on slide 50 as well. Next slide, please.
As I promised, you, I would show you a map of the Federal Project Executives in your region. And my name is Scott Wolf, and you can see my region in the West, which includes the Marianas, Palau, Guam, American Samoa, and of course, Asia. There's my phone number and email. Please contact me any time you want.
I have two other co-workers. Tom Hattery, he covers the Northeast Region, plus he has all of State Department. And there's his phone number and email. And then in the Southeast Region is Doug Culbreth. He also covers federal agencies in Europe and the Western Hemisphere.
Again, if you want to get a copy of this map on our website, you could see the URL at the bottom. And with that, I will turn over the closing remarks to Kurmit Rockwell.
Thank you, Scott, and thank you, team, for providing Session 2: The Critical Path to Success. And thank you, listeners, for joining us today. I also invite you, if you haven't seen Session 1: Managing the Investment Grade Audit, that's available at our FEMP Training website. And also, as a follow-up to this session, Session 3: Crossing the Finish Line, is available on our website as well.
You can also obtain a certificate of completion by going to this web link. And you'll be asked to fill out a very short survey on how the course was. And I'd really appreciate you taking the time to fill out that evaluation so that we can make your training sessions better and more productive. Thank you very much, and have a great day.