Below is the text version of the webinar, "2020 Peer PReview: Virtually as Good as the Real Thing – Remote Inspections." See the video.
David Nemtzow:
With that said, I am going to introduce my colleague, Jack Mayernik. Jack is at the National Renewable Energy Lab, but he also works full-time and then some for BTO. He does a variety of activities for us, from strategic planning to being our senior analyst to keeping us honest on the facts and many things in between, and including, it turns out, he has the background in the audio-visual business, as you will find out. Jack Mayernik, turning it over to you.
Jack Mayernik:
Great; thanks, David. I'm not sure if, Justin, you can get my video to work again, but I'm really excited that we're the first session of this event, and perhaps that's appropriate because we're talking about virtual inspections and we're meeting in the virtual environment. We have three fantastic speakers who will cover how they've adapted to conducting virtual inspections, as well as some of the emerging technologies that could be paired with inspections in the future to improve their efficacy. We want this to be an interactive panel and hope to have some time at the end after their presentations for discussion and we'd like to answer your questions, so please feel free to use the Q and A or the chat functions, which are available at the bottom of your screen, and throughout the conversation I'll moderate -- throughout the session, I'll be monitoring that, and we'll do our best to get some of your questions answered by the panel. But before we hear from the speakers, we have a short video dramatization of remote energy-efficiency assessments that we'd like to share. Justin, could you roll the film.
Audio of video:
Hello? Good morning, is this Mr. Nemtzow?
Yes, yes, it is.
Excellent. My name is Madeline. I'll be reviewing your comfort systems today to address your complaint of poor performance.
Madeline, can I call you Maddie?
Yes, that's great and isn't it amazing we can use technology like this to complete our work virtually given the need for social distancing?
Nah, it's great because I really want to get this done and we don't want to have somebody traipsing through the house, and even if they have a mask on it's just too risky so we were putting it off. So I'm really glad you can do it virtually. Very exciting.
Awesome. Yeah. I've been impressed with how easy it has been to adapt. So just to do my job properly, I'll need your help. It shouldn't be too difficult and take no more than a few minutes. Does that sound OK to you?
That's great. I always appreciate your help, Maddie. Always.
Right, so awesome. If you could just turn the camera around and show me your heating and cooling system.
Sure, I'm already down in my utility and storage room.
Fantastic.
For this can you see it?
I can, yeah, so if you could just back up a little bit so I can see the whole system, the ducts up at the top -- great. And are we in unconditioned space right now in your home?
Yeah, we are. It's a utility room and storage room that's unconditioned.
Got it. So if your ducts are in unconditioned space, they should probably be sealed and insulated, which might be contributing to some comfort problems ... um ... the water heater?
Right, yes, it's Bradford White.
Awesome, and can you just zoom in, if you can find where the name plate is. ... Great. OK, we'll take a picture of that. All right, awesome. So it looks like you have a mid-range efficiency heater installed in 2013. It still has life left but ...
Yeah, look at what I found, which is a little bit blocked. Oops, oh yeah, you're right, it's mid-range, a little on the heavy side.
Yeah, so installed in 2013, it still has life left, but you might want to consider more efficient options when it needs to be replaced. You mentioned yourself ...
I turned down the temperature.
Oh, good, it's good.
Yeah, you know, the thing is, Maddie, you know, when we bought this water heater, in fact, the air conditioner too, we weren't living in the house then. And we had 10 minutes to make that tough decision. And which we pay for the -- you know we paid the thousands of dollars for the air conditioner and water heater, but somebody else was paying the bills. So we got an efficient one but not as efficient as if we were paying the bills ourselves, I'm afraid. But we did just get a new washer and dryer, which I know isn't part of your regime today, but we got, of course, Energy Star ones, a horizontal dr on the washer. We've been very pleased with that.
That's great, that's awesome. I'm happy to hear that making the trip for more efficient equipment.
There's my carbon-free transportation mode. Do I get any credit for that on the home energy business?
Unfortunately we don't include bicycles in the Home Energy Store, but you do get some bonus points for me.
OK.
You said you're working from home. What do you do for a living?
I work for the federal government at the U.S. Department of Energy.
Really? What do you do there?
Actually, I'm director of the Building Technologies Office.
And we're live from Washington, D.C.!
It's the 2020 BTO Peer PReview!
Jack Mayernik:
I hope that was enjoyable and gave you a sense of what virtual inspections are like. Obviously, a real one is more detailed, and that's what we'll hear about from our speakers shortly. To help moderate their discussion, I'd like to introduce Steve Byers, who is the CEO and co-founder of Energy Logic. He has deep and broad experience in energy efficiency, both on the practical side working hands-on, analyzing energy-efficiency opportunities in buildings, and on the strategic side, growing a successful company that was recently recognized by the EPA with their Sustained Excellence Award. An entrepreneur with boundless energy and a passion for energy efficiency. I can't think of anyone better to moderate this panel. Steve.
Steve Byers:
Thanks so much, Jack. I'm really excited to be here. I'm excited about this session. As David noted, you know, remote virtual inspections have, you know, been happening for a while, and now we're just accelerating that, which is really exciting. I'm assured by everyone at your team that no assessors were harmed in the making of the film. So it's a little joke. Anyway, let me introduce -- let's get on to the panelists. We've got a great set of panelists. This is going to be a really exciting session, and I'm really looking forward to the Q and A afterward, as well. Let's start off with Jason Elton. Jason is a senior consultant at Earth Advantage. He helped create and implement remote quality assurance and mentoring services at Earth Advantage. Jason is qualified as a U.S. Department of Energy Home Energy Score mentor and a QA provider. His experience includes residential energy retrofitting, training, mentoring, quality assurance, and technical support. He holds a BPI building analyst and envelope professional certifications. Jason and his wife are proud parents of three daughters and a yellow lab. So, Jason ...
Jason Elton:
OK, thank-you for that introduction. I'm gonna go ahead and share my slides. That video was awesome, by the way; thank-you for doing that. Put me in a good space.
David Nemtzow:
OK. I was willing, Jason, to sacrifice, to show everybody I have uninsulated ducts. So I'm glad I got some good out of this, now that I'm going to take grief from everybody at BTO for a long time to come.
Jason Elton:
Well, I always appreciate it when people can see humor and even at the top level of the department, if you can. It was awesome. Thank-you. I've spent my career since 2006 working with homeowners, particularly, you know, my field is in the residential energy-efficiency field, which I'm very proud to be part of. And you know, now that I've got everybody here on the Department of Energy or a good portion of you, I have many thanks to you guys. I basically took my career to the next level because of the Recovery Act funding for home performance work here in the Northwest, and it launched my career, as well a lot of my colleagues and trade partners that I've worked with over the years. So much appreciation to you guys. And now I have the great honor of working with Earth Advantage for the last couple years, focusing on Home Energy Score work that we do in the Northwest as well as on the East Coast. And my primary role -- my title sounds bigger than what I really do, which is I work mainly with our, you know, the residential side, helping our assessors be better at their jobs. I feel like that's really my role, is lifting our assessor community up. And I came to Earth Advantage and one of our goals was to help create a remote quality assurance process, which I brought some experience from my last job over. And I could tell you when we talk about this topic I came out of the world where we did almost 100 percent in the field, quality assurance on home performance projects. And when I had a colleague of mine came up to me and showed me on his phone -- and this is probably eight years ago -- he's like, look, I'm doing quality assurance on my phone. And he was flipping through some photos, and I was like, you can't do quality assurance on the phone; you have to be at the house. And he was like, why can't you just put a professional brain in the house? Why do you have to actually go there every time? And that really opened up some thinking on my part, where I was like, yeah, we spend a lot of resources going to all these houses throughout the state just to do quality assurance. Is there a better way? And that really sent me on that pathway and pushed me.
I'm going to talk today about what we do right now and the research that we're doing with PNNL and the Home Energy Score division at your organization. And so currently Earth Advantage supports Home Energy Score program, so we support our assessor community, not only providing them with training and mentoring and doing quality assurance in the field, but we also do remote quality assurance in areas in the Northwest where we don't have staff that we can go out with to a home and actually do in person, you know, scoring and do and compare it to the accessory score. We use a remote quality assurance system, and we use an application that you can download to any mobile device or use on a desktop. And if you go and look at the number of applications that you can now use through mobile devices, there are so many that you can now customize yourself. Once you get a subscription to. The one that we use is called Go Forms, and that's that blue checkmark icon over on the left side. And what our assessors do is they fill all the data that they collect that's required by the Department of Energy's Home Energy Score program, they enter into this form and they upload all the required photos that we need in order to do remote QA. And so what it does is allows them to do it on a mobile device, and managing the photos is a big piece of that. So they enter that in. They get all the data, the photos, that we need. They score the home and the DOE system generates the score, and then we use a public registry system called Green Building Registry to make that validated score available to real estate agents. And our QA team pulls 5 percent of those to do the remote QA process.
So that kind of gives you the landscape of what we currently do. We've been doing this since 2018. And here's a couple examples of that form that you can fill out on the phone. These are screenshots from my iPhone. And you can fill it out in this format, or there's also a list format where it looks a little bit more like an app, where the fields are arranged by category. But it allows you to input data. It does do some calculations, as well as upload building diagram images or images that we need in order to do our job. And it can pull those images right off the library in your photo album. So now that everybody's carrying these smart devices around, there's really not a need for external extra cameras anymore, even though some home inspectors still like to use these. You can just take video or pictures with your mobile device and then suck them right into these applications very quickly. And then you can get rid of them on your phone to not have to deal with all the storage.
So that kind of tells you what we are doing right now and which we continue to do remote QA. And that's been growing slowly over the last few years. And we we get lots of inquiries now because of the pandemic, and even things like the wildfires that we had here in Oregon, which create basically a month where people can't get out as much. And it's even uncomfortable to travel in your car with the particulate levels. And we did some remote QA. We did a little bit more than we would normally do during those times, as well, but we're also involved in some research with Pacific Northwest National Laboratory, which is funding this research, particularly around Home Energy Score remote quality assurance methods as well as remote assessment. So the questions are, can we -- is there another way that we could do remote QA through live stream? And you guys basically demonstrated that approach through video conferencing, which is what we're going to be testing not only with our assessor community to see if we can do live stream sessions with them while they're in the field as an alternative to uploading photos, but can we also do it with homeowners? Could we create a way for assessors to do these sessions with a homeowner and collect all the data that they need in order to score the home? And so we really want to -- we're going to test how accurate that method is and see what are some of the challenges and benefits that come out of that. So the main thing we're trying to solve is issues with accessing a home. And I feel like we're on a trajectory where this is going to become even more in demand, as people have moved to basically remote offices. And people aren't just going out into the field as much, so we need to keep getting those professionals into homes. And so can we do that with technology? So we'll be testing some remote applications and then looking at, are these things actually creating cost savings and time-reducing strategies for partners that are doing quality assurance and mentoring, as well as for assessors who are working with homeowners?
So currently we're developing that methodology on how we're going to do that test. And that's kind of in the stages right now, that we're working on with PNNL. And we're going to be working with our relationships that we already have with our current Home Energy Score partners that we have in Oregon and New York. And we'll be evaluating a few different software applications. We'll be testing those in live sessions, and then we really want to test this on a variety of home configurations, because we know that there could be some homes are very simple. You know, ranch style, but others are very more complex with multiple systems. And we really want to know, how does that remote data that we collect, if we were to work with a homeowner and collect all the data remotely, then we'll be sending somebody out on site and a trained assessor to get the data on in the field. And then we'll do some comparison to that to see how accurate this is. So we hope to document all those results and get feedback from our assessors and the homeowners to see what that experience is like. And kind of that will really help shape guidelines that we develop.
Now here's a couple examples. I mean, in that video you saw Maddie take a photo as she was video streaming there, and that's exactly what you could do. Most applications -- you know, this is an example of myself practicing this with one of our new homes verifiers that was working with a builder to verify air-sealing and mechanical equipment. And I was able to screenshot while doing a video conference through Zoom. So this kind of gives you an idea how that worked, and Maddie demonstrated that excellently.
So you know it's one thing to come up with this strategy, but really, we just need to test through and go through all the considerations that can come up, so that we can help, really, you know, give a good tool set to partners and assessors that are using this approach. So one of the things is you can collect a lot of data about homes online. It's amazing how many times that, you know, when I've done this type of work that I can go online and even real estate listings will have floor plans that were drawn by a company, photos of the inside, and all around plus mechanicals. And I can even go on tax records and pull down a lot of information. So there's a lot of online data. We'll be testing that before we do the live stream to see how much data we can actually get, and then how much data can we actually collect remotely. So we've identified some data points that we know that especially homeowners may struggle with in finding it, depending on their knowledge of their home and understanding the terminology. So there's a lot of, you know, kind of walking a homeowner through what is needed, whereas when we work with an assessor doing remote QA, they really speak the same language as us. You know, we most of the time, we've trained them so they really, communicating is a lot more fluid with a, you know, an energy-efficiency professional, whereas with homeowners we need to really meet them where they're at. So there's a lot of prepping a homeowner for a session and then we're also going to take into account geographical location. You know, where they are located, which you know could affect their bandwidth that they have. They may not have high-speed Internet at their home and maybe using their data plan on their mobile device. And then looking at homeowner demographics, as well. And then what application and device issues can come up. We know there's going to be house configuration complexity, but we also know like when you take that device down into a crawlspace or basement, do you lose connection? I'm anticipating some of that. And so we just -- how do you navigate that? And really getting feedback from homeowners and assessors on those challenges. One thing we know is that not all not on mobile devices -- you can see me holding my tablet -- have a flash built in. Some -- most of them don't. And so if you use a phone, you'll have a flash built in. So we really need to think about low light scenarios, will we be able to see what we need to see.
So this research, we'll be testing two things, which is for conducting a remote assessment with a homeowner, as well as conducting a remote quality assurance with an assessor. So what we're going to the plan is to work with our group of assessors. We're going to identify some ones that can help us do this research. And then we'll conduct 30 remote assessments where we'll work with the homeowner, do the remote assessment, and then the trained assessor will go out and collect the on-site data, so that we can compare and see how accurate it is. We'll also during this time conduct 10 remote quality assurance sessions with our assessors to see if that live stream video conferencing method has benefits or challenges in comparison to the current process I talked about, which is the photo upload. And then we'll be doing some analysis around that.
So the ultimate outcomes of this is, how accurate is a remote assessment approach? Can people feel comfortable putting an official stamp on a remote assessment? And can we find a way to create a tool set for our partners to use so they can use this during these times when there are shutdowns or their construction trades are limited and their ability to go into the field? Can we create a method for them to keep that pipeline moving and be really flexible? If anything, that's what we've learned in the last six to nine months is we need adaptability. And so this is a way of testing this can we adapt and provide and get the tools and systems that we need set up for our community. And do these live stream applications work? And they work in tandem with photo uploads or replacement? And really identify the barriers and limitations of this. So we know what we're up against and maybe there's other applications that are a better fit that we can discover after doing this testing. So we hope to establish a baseline criteria for offering a remote assessment for Home Energy Score and then make recommendations around that.
All right, I thank you for your time, and of course, I'm available for questions. I don't know if we'll be waiting towards the end of this session but I'll be here. Thank-you for your time.
Steve Byers:
Thanks so much, Jason. We're going to go to our next panelist, Valarie. Valarie Evans is the City of North Las Vegas building official. Valarie is a master electrician, has been in the construction industry since 1985. And there's a fun fact about Valarie, is that she likes the crumbs at the bottom of the cereal box more than cereal. And yes, she is all about the crust. So Valarie, take it away.
Valarie Evans:
Thanks, Steve, and yes, I am. Thanks, Jason; that was amazing, and it is really. Let me see if I can share my screen right here quick. Let me show this screen. Can you guys see my screen? Not yet? Hold on one second ... let's share this screen ...
Now you should be able to see my screen. And let me get my presentation up. So this was very informative. So to just be a part of this, and at the rate that everything is moving when it comes to virtual inspection and doing things remotely, it's just such -- it's so fast in how it's moving. And so just a little bit for City of North Las Vegas to participate. Thank-you so much, and thank-you, Jason, we do have a remote inspection program at North Las Vegas. And it has continued to change like in real time at the speed of business. And so up to and including today, it continues to change. Our governor just asked us to stay at home 2.0, so here we go again, trying to do more and more stuff remotely. And I think you're right, like everybody's whole perspective on remote has shifted. So I'm going to talk about how the City of North Las Vegas started a remote program and how it shifted a little bit during COVID. And really what's more important is how -- what are we doing today, and how does technology now moving forward work well? Because like you heard Jason talk about, this isn't something new; it's just something available now. And so even though people have used a remote approach, now it's just even more critical that we do what we can to be able to facilitate not only our positions and help our businesses move forward. But to help our customers and to help and to do it safely, right? To do it safely is the most important thing.
So back in 2019 the building officials in southern Nevada were approached, and it was pretty much like, how can we get solar on rooftops quicker? You know, the solar industry came and how can we make things faster? And so several of the jurisdictions were doing online and online permitting and so it was really critical that we started to make changes so that people weren't waiting to get their systems on their roof. And every time that a system was a week behind, there was a percentage that was lost because of the timeline to get the permitting process. And so I got the luck of the draw to run with remote inspection. And we started making changes. So the word "remote" became like pretty synonymous in North Las Vegas. And at the time they would come into the counter and spend some time in the city hall and they'd bring at some jurisdictions three to five solar permits. And then if once you were done with those you had to get back in line, and so it just wasn't effective at all. And so we all started really working towards online permitting and working towards being more efficient with that. And so at the same time, the plumbing and heating industry came to us, and they said they didn't want us on their ladders for inspection purposes. They didn't want the liability for that, and they didn't want the ladder sitting in the middle of the hallway at the little old family's house, where they were trying to navigate around the ladders that were waiting for our inspection to take place. And the real thing was it's the duty of the person who pulls the permit to make the project accessible for inspection. And sometimes that would be through the contractor doing the work and then giving the homeowner the permit and saying here you go; you have 180 days to complete your inspection and this, and if you don't, this permit is going to expire. And so we really wanted to put the responsibility onto the industry and the contractors who are pulling the permits, and make it safe for everybody.
So we started the remote inspection program. It was the residential remote inspection program, and it was a great benefit. Lots of times if somebody's replacing a water heater or insulating some ducts in somebody's storage room, that doesn't have any insulation in their unconditioned space, for example, they would do the work and then the homeowner would have to take the day off, and then they would give the homeowner the permit or schedule the inspection. And then the homeowner would have to take the day off. And if they were rolled to the next day for inspections, then the homeowner would have to take the day off. And it really made it to where we wanted to benefit everybody.
And so the idea was that if you do an inspection and have it scheduled for remote video and do it all the same day, then the industry only has to be there one time for the installation. The homeowner only has to take a little bit of time off of work and all the inspections can take place. So we really ran with it. We started in July of 2019, and we had this small list. Actually, it wasn't that small; there was several other jurisdictions that were doing residential programs. And they were struggling in getting the industry on board. They were struggling with making it more and more mandatory. And so my intention was to throw a bunch out there and see how it worked. I invited my staff to help me to determine what would you guys be comfortable with doing remote, like when would you, what would you not be comfortable with. And when my -- and I have a master electrician at work, and he says, hey, we'll do electrical service changes, then I know they're comfortable. And I know that this program is going to work. And so we started it with the intention of trying to get 80 to 90 percent of them to go video inspection 100 percent of the time within a year. And that seemed to be a challenge because there was some pushback on the not knowing that you could schedule an inspection, do the work, perform the inspection, 100 percent all in one day. And then like that video at the beginning of this presentation all together, I mean, like that was, when we first started inspection, that was kind of what it was like. It was like somebody running around showing us their bicycles and showing us their washers and their dryers and scrolling in and moving fast through the video. And it really did need to slow down and work together so that inspection could take place, and that we could get the information that we needed. So that it didn't take longer to do a remote inspection than it would take to do a regular inspection if you're there in person.
And so the whole process of a video inspection, I think everybody's kind of up to date on now how to perform video inspections. And I think Jason mentioned how there's software that you can customize to your own needs. And with reporting and customization, there's several stuff out there that would work. And so North Las Vegas picked appointments, plus at the time and it was a really great calendar for me to like get reports out of, and to say, this person can do these inspections and this person can do these inspections. But the general idea was that we worked together with the person on the other end of the inspection process, and that they respond to our call, and that they start from the street to validate the address. And they would go up to the house or go up to the property -- at the time, it was the house; today we do commercial inspections, but at that time we were kind of just doing residential. And we would be able to validate that the house was the right place. We still had paper permits at that time and paper plans, which we don't have anymore, either. So they would be able to validate the permit and the scope of the work and do the inspection. And it was more like those one-and-done type inspections where the inspector normally wouldn't be there for very long. That water heater inspection that Jack was doing, or not that Jack was doing, that they were doing, would have been like a 10-minute inspection, if that. And that would be getting the homeowner to the door and then validating all of the components of that installation and then leaving. And so it really is something that once you get there and you're moving along, that it really does save time. And they're not long inspections to begin with, so they would have the customer follow my inspector's or the inspector's direction. And suddenly what happened was the inspector and the industry in the community started building these relationships. And these relationships were built on now we know how to work together, not like you know what I want to see and I'm telling you what to do, but like how do we work together to fulfill this inspection process and get to the other side of it so that we know that we're code-compliant, life-safety-compliant, and that you are comfortable with the inspection process. Because so many people are used to having inspections in person. And so it's a change for everybody along the path.
And so during COVID-19, we did a stay-at-home order, and we really needed just to keep it moving. There were several families waiting to move into homes, and there was commercial buildings that the tenants were waiting to move into that were facing liquidated damages if they didn't have their businesses up and running. And then I think everybody knows that as remote inspection and as COVID took place, that everything kind of went on Internet purchasing. And like it really has changed the whole entire landscape of how we know our lives were to what they are today. And how can we take that information that we got through this experience and like Jason said, use it to advance technologies and to advance the way we approach things today. So we started do -- I sent everybody home. I had some concern from staff, and so I sent everybody home. And we went through that process of getting people comfortable, getting them set up to where they could perform video inspection from their VPN access at home, where they could access the permitting software. And some of my inspectors would allow some photos like that particular inspection that took place on that video. That would have been a really good one to do just some pictures of, because you could with today's cameras, you could actually get like a lot of detail out of just the cameras that are on the smartphones. And even now since that time you know there's all of these programs that have come up to where you can use a software specific to remote inspection. So that you can take that picture and circle something in the live stream video to send to the customer later on and archive the video or do GIS location. So you know that you're in the right spot and that they're in the right spot, but from what we found there wasn't tons of people or there's not tons of people that are just trying to hide stuff. And so really, just everybody working together is what's really been worked well for us, so they got their access.
And I had 12 inspectors sitting at home. And so everybody like had their own way of doing it and I supported that. I supported that this inspector would call everybody in the morning and schedule a time and this inspector would tell everybody to call them when they were ready and this inspector would work with them to change the time if it was necessary. And so everybody was comfortable in their own process. So there wasn't some hard "this is how you have to do it" kind of a process to the remote inspection process. So as long as they validated the permit information, as long as they were able to access the documentation and communicate with each other, then it really did work, once they started learning how to work with each other. I have a inspector that sits at home now. He's been at home since May, and he has gotten to the point now where he's been doing upwards of 300 inspections a week, all remote video, and he is comfortable with it. And that's a four-day workweek. So he's very efficient, and he's working with the same contractors and developers that he would work with in person if he was doing these inspections.
And so really what we had to do to set up something virtual was to determine what inspections were allowed to be done remotely. And in the beginning we only used Skype with our program, but today we use whatever video works for the customer, whatever platform they're comfortable with. This is a Zoom setup right here, with this little background behind me. But there's Webex or there's you know GoToMeeting, or there's just video Skype, Facetime, whatever works was really what was important. And so and then how was the inspection going to be performed? Was it like video? Was it photos? I have some slides in here on drones, and a drone program within a jurisdiction would be amazing for those places where you just can't get to. You may not be able to get to something because it's in the snow and you have to validate something that you can't get to until the snow freezes off. And so really figuring out how that program was going to work and archiving the video. Are you going to archive the video? So far I haven't engaged in a program to archive video, but in my perception, I'm a commercial combination inspector and I've done out of the ground big box buildings. And I did this Super Walmart when I first started inspections almost 20 years ago. And so what if they have me in court tomorrow and say, hey, do you remember that conduit that they run across this roof over there at that Walmart? It's almost like it could be in our benefit, but it could also be to that detriment of trying to get the customer, everybody's going to want their own video and archive it and figure that out. And so for right now we still haven't engaged with a software that's specific to inspections and we haven't engaged in archiving to this point. But what the customer needs is they need that 4G or better; nowadays you have 5G wireless setup. Whether it's a smartphone or a tablet works well, and the media account that's going to be used is necessary so that customer is comfortable. That connectivity is important that they have. And so just making sure that before the inspection is scheduled that the person has everything they need to perform the inspection. We just had a new stay-at-home 2.0, where I just sent my staff back home. We were back in the office and we were kind of doing shift starts and stuff, but now I just sent people back home so that we're doing remote again. So that the state of Nevada doesn't end up back in a stay-at-home order and everything closes down again to whatever degree that might look like for our governor. And so just making sure everything is available for what they need.
And so what we learned in that, is that communication between the customer and the contractor was imperative. Having their contact information was super important. Making sure those connections were the connectivity of the platform that they were using or their phone was using was important. Those utility reconnections we used to do, tagged, gas tag and electric tag approvals. And so working out with NV Energy and Southwest Gas in southern Nevada was really important so that they could know when an emergency down and out needed to be reconnected, where normally they would just go out there and see a tag on a panel, or see a tag on a gas meter, and they would be fine, they would know what was going on. And so after everything took place, it became that change, it became, it has become that change. And so the big thing that I want to share or talk about today is what happens when we turn stuff back to normal, like how can we take what we have learned, not only through before COVID-19 or before you know there was the obvious platforms all over the place, to a point now where it's commonplace, right. It's commonplace to be on Zoom and to do virtual annual business meetings and virtual ceremonies and stuff. And so how can we take those efficiencies that we learned during COVID and put them to use, and how can we increase the program to allow for more video inspections or to allow for a bigger scope? How can we identify the projects that where we were needing to be in person and not necessarily like everything doesn't work for video. If I'm gonna go do some tilt concrete panels in a 800,000-square-foot building and do the rebar steel for those panels and then doing remote might take a whole lot longer than me just walking through some panels with my iPad showing me my panel designation so that I can validate that the steel is in there and is correct. And so and then being able to follow up on those projects that really need a closer look. Everything is not perfect yet, but I really believe that anything can be used for remote validation, whether you have to work through some difficulties or some challenges to get to the other side of it.
North Las Vegas has just engaged with a remote inspector, and that inspector is going to live in Hawaii, and he's going to do remote inspections for me from Hawaii. So that I can get my person that sits at home now and does inspections, I can get him home and get him back into the office so that he can help us here on some jobs. Because we're just blowing up crazy with construction. During the COVID-19, we did like 11,000 over 11,500 inspections we did a hundred percent everything video. We came back into the office and we really determined what can be shifted so that we can add more to that scope of work. What we did find out was if we used photos, we found out that people could send you way too many photos or they could not have great lighting. But the benefits of it was that you could zoom in on this water heater and you could really look and see things that you wouldn't normally maybe see if somebody was just carrying their phone and running around and showing you the bicycle kind of thing. It really became clear that it worked really well for us. And so there was a lot of benefits for photos. And we were able to continue to working with customers, as after they were sending the pictures you could still communicate with them. We had some situations where the customer would go into the house and they'd lose connectivity and then they'd walk back outside and they'd get connectivity, and so suddenly the remote video isn't an option anymore. But how do I complete that inspection without going out to the job site? We would just have them send us some pictures. We'd validate the pictures and then we would continue on. We would allow that to take place.
There's drones. I know that you know drone inspections is something that could really be a useful tool for like a holistic view from an entire corporation or business or jurisdiction where maybe everybody can use that drone to facilitate their own little process, and then together they could have this massive program or this holistic program that can be really quick to mitigate emergencies or GIS. City-owned facilities or or whatever is necessary to be able to conduct those inspections. It's just the possibilities are endless. We don't have a drone program at North Las Vegas, but there's definitely some pros and cons to drone inspection and how you don't need everything you would need to do an inspection relatively if you were on site. But the cons of it is that you have to make sure somebody is a listed pilot and know of the restrictions and really be able to mitigate that program to the best of its ability. But I think a drone program holistically being used across a corporation could be fantastic. And so there's several instances where drones could be used thermally and doing building thermal envelopes in confined spaces for dangerous applications where normally you wouldn't necessarily have to get in there in person if there was something that was dangerous or something that was remote, you couldn't get to something. That makes it really efficient in having a program of that degree. Like I was saying, you could have a jurisdiction use the fleet of drones to create that program that would be that would cover all of the departments within the jurisdiction -- the public works, police and fire, and cover building inspections and GIS. So there's always that type of the great thing for us, is that what's next.
And you know what is it next? What do we find out that is next? We know that there are some things that are just much more thorough still in person. As we work through those kind of things to determine how to get to the other side of that, then we'll be able to mitigate how to do something that's more complex via video and realize that those inspections that would take longer via a video feed than it would in person, that how can we pare that time down so that we're more comfortable in what we're seeing and that the contractor is more comfortable in what they're showing us. And I think it's really important to know that we need to trust in the technology and be able to embrace that change. There's a learning curve that takes place between everybody in a remote world, and getting through those comfort levels and figuring out how to bridge that has really been what's worked for us in North Las Vegas. And what we do still is we do more of our more complex projects in person. We don't do them via video, however, we're looking at increasing our program even now more with this person that is remote and validating how many inspections do we continue to do.
So I'm getting back to work and seeing that there could be changes to the program. When we started it, it was all about residential, and now it's all about remote. So really just changing up what we can do. Checking and seeing like, what are the actual inspections that are listed in the code that are required? And how I hate to say this out loud, but how many jurisdictions do you know above and beyond that, and for what reasons, right? And so being able to determine like what is required and utilizing that code as that tool is important. And like we talked about, there's new software popping up everywhere. I've gotten several phone calls from several different organizations to try this new software that is perfect for that intimate one-on-one remote inspection to archive and to mark up. And I got to tell you the truth. The truth is that I'm moving that so fast at the speed of business here in North Las Vegas, keeping everything moving forward, that we haven't had an opportunity to slow down long enough to look at something new in technology to move forward to that next level because we're a small house here. So we don't have a lot of a lot of time to spend like looking at new things for us to work with. So we added a bunch of residential scopes after the COVID-19 stay-at-home order. And it included pools and room additions and then masonry walls that aren't retaining in soils. And we threw in the southern Nevada home builders association, as well. They were really grateful to be a part of it. And anything is up for revision. That's the one great thing about change is that there's no pain in it, right? There's only pain in resistance to it. So being able to determine what can be changed. If somebody says, hey, well, I think this is a good idea for your program, and we vet that out. Then it can be changed. And that's what's good about being in a small jurisdiction. It doesn't take a lot of navigating to get through something different. We threw in some commercial work, and we threw in a bunch of fire inspections, as well. I also managed the fire inspections and they did remote video during the whole COVID stay-at-home order, as well. We were home for 12 weeks, so I liked the fact that it was mentioned that the homeowner doesn't want people in their home, like "I feel uncomfortable with you coming in here." So it's not about that I need this done right now; it's about how can we do it within that timeline and still move forward.
Paul does those tenant improvements of B and M occupancies that don't exceed 3,000 square feet, But if a customer called me and said, hey, Val, we need to do this one here and it's 4,000 square feet. Can we do it? Of course, I would allow it because changes to a program are good. So in conclusion, I think change takes change, and you gotta start with what's comfortable and move from there. You have to include your whole entire team in the planning process of something, so that when you hand it off to them and they implement it, that they embrace it and that they move forward with it. And they're willing to look at all the good things that can come out of it. And they don't get uncomfortable about that change that takes place. And then figure out what's next, what gonna pop up that's gonna help us to continue to move forward in remote and archiving and video and continue to grow it forward. So I'm really grateful to be a part of this presentation as well as the Department of Energy, and I thank you so very much for your attendance.
Steve Byers:
Thank-you, wow, thank-you, Valarie. That was fantastic, and I'd like to -- in case folks haven't figured it out, there's a lively chat going on. And there is a Q and A tab, as well. And if you put questions in the Q and A, then they can get answered right inline there. So you'll see the answer along with your question. There's three tabs in the Q and A: open, answered, and dismissed, so you can look there and one that's come up a couple times is this all this going to be available later? And the answer is yes; it's going to take a little time to clean it up for web presentation, etc., but everything is going to eventually be posted on the BTO website, so you'll know that. OK, so now next, drag my notes over ... so next is Jessica Granderson. And Jessica is a staff scientist and deputy director for research building technology and urban systems division at Lawrence Berkeley National Labs. She leads LBNL's -- easy for me to say -- R and D in commercial building analytics, diagnostics, and advanced controls, and before getting involved with building energy and buildings, she worked on engineering and analysis of astronomical instrumentation. So thanks, Jessica, and there you go.
Jessica Granderson:
Thank-you. ... OK, thanks for the introduction. Jason and Valarie gave us some really great examples, I think, of how remote inspections are being used in the field today. I'm going to share with you some of the state of the art in commercial building remote analytics, how the technologies are being used today, and how BTO is investing to further enhance those capabilities of the off-the-shelf technology. So a little bit of background on the tool space that we're talking about. There's always edge cases and hybrids, but most of today's remote analytics use either simulation or data-driven techniques under the hood. Generally speaking, the simulation-based tools are going to take in a set of either more or less detailed user inputs on characteristics like the number of floors, the footprint, orientation details on the envelope, windows, lighting, and mechanical systems, as well as utilities information, and feed those into a simulation model and output recommended improvements. On this slide, I'm showing a really simplified representation of the commercial building energy asset score that folks may be familiar with. Now the market is also offering data-driven tools that operate in a continuous manner. So they ingest data from many different sources to give more frequent feedback on building performance and efficiency opportunities. We call these energy management and information systems, and they analyze and visualize data from meters, control systems, IoT devices, even in some cases distributed energy resources, to give insight on measures and efficiency opportunities. On the right-hand side, I'm showing a couple of screenshots illustrating at the top, HVAC system level fault diagnostics, where each of those colored lines represents a specific problem in a specific piece of equipment. In the middle, an energy use heat map, and at the bottom, time series load profiling. And this is all used to indicate how much energy is being used at what time of day, either at the whole building or at the submetered level.
The technologies are capable of giving us capital retrofit or replacement measures, as well as the low-cost / no-cost more operational insights. And these include recommendations for things like improved scheduling, space temperature set points or economizer use, reset strategies, and over- or under-ventilation. And it's worth emphasizing that the same algorithms that are used to detect and provide feedback on these kind of measures are also of relevance under our new normal conditions. We're really interested in monitoring for proper ventilation, outside air intake, and control strategies that are associated with pre-post occupancy flush, making sure that we've disabled things like demand controlled ventilation or ensuring that we have an appropriate level of turndown or setbacks in our low-occupancy conditions.
These more continuous analytics through energy management and information systems are bringing really deep and cost-effective savings today across the U.S. BTO just concluded the largest ever study to date documenting the technology costs and benefits. This was done in partnership with over 100 organizations using 40 different analytics offerings in over half a billion square feet of install base. The results found that the users of smart meter analytics saw average whole-building savings of 3 percent. Users of the HVAC system specific fault diagnostics were saving 9 percent on average, and these deep whole-building savings were translating into median annual portfolio savings of three million dollars a year and paybacks -- median paybacks -- of two years. So very powerful and effective use of the technology today. With that introduction to the key types of remote analytics for commercial buildings and how they're being used, I want to turn to some highlights on how BTO is working to advance today's state of the art. I want to first acknowledge some of our partners. NYSERDA is providing great feedback in an advisory capacity. Signetron is working with us on methods of data capture and analysis. And TRC is supporting some of our field testing and validation. I want to also acknowledge that Harry Bergman is our technology manager at BTO.
So the opportunity driving this work is that recent advances in things like public data availability, you think disclosures, you think permit data, advances in sensor technology, and falling costs have all converged to bring about an increasing number of data collectors for our buildings. And it's these novel data sources and feature extraction techniques that are bringing promise to do two things: identify building characteristics and assets in new ways and building-specific energy-efficiency measures.
So when I talk about novel data sources and feature extraction, I want to walk you through that landscape with a little bit more detail. And I'll start kind of at the 3 o'clock position on this slide with the red squares. Today's continuous remote analytics technologies are largely leveraging inputs like control systems, sensor data, meter data, and weather data. If we start to move kind of counterclockwise through this image, in this work we're looking to incorporate public data sources. And then today I'm going to focus most on the things in the gray boxes. That's RGB and thermal images, street, aerial and satellite images, as well as LIDAR. And we'll talk about feature extraction techniques. That'll span machine learning, image processing and energy analytics where the vision is that we can conduct fusion across all these sources to improve that ability to identify characteristics, assets and site-specific efficiency opportunities.
Specifically, and Valarie gave me a great setup there, I'm going to give some examples from work with drone-based acquisition of thermal and RGB images. So here's a place where we're really helped by dramatically falling camera and drone costs as well as advances in photogrammetry software, machine learning, and computer vision. And I'll show you how we're adapting these techniques to auto-generate 3D geometry to extract exterior features from the building. I'll show you specific work around windows. There's opportunities additionally for things like rooftop PV or packaged units, as well as identifying thermal anomalies.
So I can walk you through how this is done, and beginning with a generation of 3D geometry from RGB images, the first thing we do is focus on a building of interest and plan our drone flight. That usually comprises overlapping or concentric circles flown around the facility. And we use that to create a whole collection of 2D imagery. From there we post-process the imagery with just off-the-shelf photogrammetry software. And this gives us a 3D reconstruction of the building. You can think something along the lines of like the 2D pointillism you may have seen like in art. But this is adding the third dimension, much more complex with additional metadata that opens the door to additional analysis. So we take the post-processed 3D reconstruction. From there we're able to apply huff transform and polygonization algorithms from computer vision. And that allows us to extract the building footprint as well as the building heights, from which point we can then generate a 3D model of the building in a geo-Json format that allows for easy ingestion by existing analytics applications. So we can stop here and ask ourselves kind of a sanity check. Sure, we can do this, but is it practical? What we found is that we can cover a medium-sized building in about two hours to acquire the images, then kind of depending on the processing -- computer processing power available, about a day to do the post-processing with that photogrammetry and generate the 3D geo-Json model. So you know there's not yet any real deal-breakers here to practical viability. And that was very encouraging to move forward and explore how else we could use this drone-based data capture.
So the next thing I'll show is how we're looking at information around windows. Here we start again with our collection of 2D RGB images, and we take from the computer science domain some promising deep neural network algorithms. These are semantic segmentation models, and we use them to pull out the portions of the image that correspond to windows. And this is shown in this binary mask here, kind of in the middle of the slide. We do this by scraping lots and lots of images out of the public domain, labeling them, applying labeling as to where the windows actually lie, then we augment that set of images. We flip them, we rotate them, we change the contrast to make it even bigger. That allows us to customize and tune the parameters in the learning model for our specific intent to figure out where the windows lie in these 2D images of buildings. Once we have that, we then have to identify where the facades actually lie in those 2D images. And we do that by returning to our 3D model. We project the corners of the facades from that 3D model back onto the RGB images. And we do that by pulling out some of the metadata from those 3D point clouds that lets us match each 3D pixel to a 2D pixel. And now we can bring in more of the Z dimension of these images. So then next, and I'll start on the bottom right-hand portion, we're able to then combine the binary mask of where the windows lie with the knowledge of where the facades are. And we can then do a triangulation to figure out where the windows are relative to each facade corner. Then we're really off to the races. We can project the window back onto the 3D representation. And from there we have all the geometric and volumetric data that we need to compute something like a window-to-wall ratio. And that's an important input for say some of the simulation-based tools where this is often kind of an unknown value that may be defaulted or the user is kind of making their level best guess. We expect that the same methods can be used for things like identifying rooftop PV or the presence and number of packaged rooftop units.
So the last thing I'll show is thermal imaging and anomaly detection. How we do that is the same initial step of planning our drone flight, collecting now 2D imagery, both RGB as well as thermal. We apply the same 3D reconstruction with that off-the-shelf commercial software. And we do that for the RGB and for the thermal image. And we extract that same 3D building model. From there, moving down to the bottom right again, we're able to project the 3D model onto the 2D thermal image. And that lets us pull out the portion of the 2D thermal image that represents the building, as opposed to the surrounding parking lot or the trees or whatever else may have been captured in the image. We can then apply some clustering and morphological operations to extract what are the contours of the detected anomalies from the thermal image that can then be overlaid back onto the thermal image to give the energy analyst a visual of the anomalous regions of the facade. The next step here is then to characterize the magnitude of these anomalies so that the user can understand if they do, in fact, represent meaningful improvement opportunities.
So today's just a preview. I gave some highlights that we're undertaking, a lot of additional work. So looking at aerial images combined with LIDAR for building footprint extraction and creation of 3D geometries, when we don't have access to a drone. Valarie mentioned some of the challenges with drones, and I can assure you that we've encountered almost all of them. We're field-testing this work with efficiency program implementers to understand the value of the new information we're creating and whether it actually helps their processes in the field. Then importantly along the way releasing open-source modules. And this will include the code base, the tuned learning algorithms, as well as the training data sets. And that's really a contribution back to the community to support adoption into existing tools, as well as further extension and adaptation in the development community.
I want to conclude with three key takeaways, the first being that remote analytics technologies are available. They're enabling cost-effective savings today. And this is in terms of both capital and operational benefits, benefits under normal as well as new normal conditions. Secondly, on the more development side of things, these new data sources we have available, extraction and fusion techniques from other technical fields, are supporting, further advancing today's technology capabilities. And third, while I focused a lot on a use case context of, specifically of identifying site-specific efficiency measures, you could imagine a lot of additional applications of these techniques. That might be things around outdoor asset identification more generally, citing and tracking distributed energy resources, even more urban design kind of planning the hardscape, thinking about vegetation water cool surfaces, or in the programmatic sense. We're talking a lot more these days about inventorying, conducting more localized building typologies, so that we can better direct our programs in terms of planning as well as targeting.
I'll conclude there, thank you for your attention, and pass it back to Steve.
Steve Byers:
Thank-you so much. As for Valarie and Jason, that was fantastic. Fascinating stuff. There are a ton of questions, and if you'll bear with me for one second. I'm glad Jack has been diligently grabbing questions out of the chat stream and other places to feed to me, and I've got those up. So let's start with Jessica. Question: Is the report for the analytics study available to the public?
Jessica Granderson:
Absolutely. It's available on BTO's Better Buildings Solution Center as well as at smartenergyanalytics.org. Contact me if you can't find it.
Steve Byers:
Awesome.
David Nemtzow:
Hey, Steve, hey, Stephen ...
Steve Byers:
It's a -- you're muted now, David.
David Nemtzow:
... all right, it's double or nothing. Sorry about all that. Can you hear me now? ... Sorry. Technology in the way. All I was trying to say was we will figure out a way at BTO to get all the information out to folks, the URLs and the reports, so to save you guys some time Googling. So we'll do that after the Peer PReview on Wednesday. We'll collect everything, so Jessica and all other panelists, you can send it to Jeremiah. We'll collect it, then we'll distribute it to all registrants. Sorry about all that. Thanks, Steve.
Steve Byers:
No worries. Thank-you, David. So next question I'll just put it out. I think is going to be primarily for Jason and Valarie, broadly speaking. Just several questions around like, what about and maybe it's going to be a little bit more Jason-related you know, diagnostic testing, issues like that, you know, air leakage, duct leakage, you know, combustion appliance zones, etc. What are your thoughts around dealing with those needs?
Jason Elton:
Yeah, sure, well, with Home Energy Score, it isn't a comprehensive audit so those fields aren't required. You can enter a blower door test and in those cases we require a photo of manometer. So you know we can have an auditor take a photo of how they set it up in a manometer. That doesn't cover every scenario of which doors they had open while that test was running, but obviously you're going to be doing that with folks that have a certain level of training and do them regularly. So if you're doing a home performance type work and you're doing -- you're looking at doing remote QA on that in particular, you're going to want to know what fields and photos you're going to require that people provide you. And so if you're doing CAVS testing, you'd want to see them fill out the data they need to collect to meet that requirement, as well as photos of a manometer. And that gives you an idea. But when I say that, you really need to know who you're working with, what their training level is, and so you have to have a minimum criteria. So if you're working with a trade partner that's doing performance work, you're saying OK, who is the person uploading the data and running the test? They should have a minimum requirement, certification to do that or an experience level. So you need to determine that at the program level.
Steve Byers:
That's a great point. We in the HERS world, we primarily use this for QA and doing QA on the road, or so we'll go live stream, watch them do their test if that's what the function is. Another piece of the puzzle there. That strikes me too as an opportunity in with what you're doing, Jason, where you're not expecting to do diagnostic testing that you can identify situations in homes where you would recommend that a person does come to do that testing. You don't always know, but sometimes you can clearly see situations where that would be the recommendation to get somebody in to do a real test. OK, yeah, so another category of question that cuts across everybody on the panel, I'd say, is there's several questions about liability, privacy, data, you know, gathering privacy liability, etc., as a broad category. Anybody want to jump in with any experiences or thoughts on that topic? I'm sure you're all addressing in certain ways, so if you're doing drone scans of a building, where's your authority stop or start, things like that? Jason, you came off mute; did you want to go?
Jason Elton:
Well, just looking at the -- like when we look up online data and we get photos from either a homeowner or an assessor, we're telling the homeowner that we're collecting photos. So for the time, you know, and generally the way we say is, we're collecting photos of the assets of your home. And we try not to get images of like personal items. If a homeowner had a problem with that, they'll let you know. I have rarely had that. I mean, I've done thousands of inspections and I take photos every time because I want to know like if somebody comes back to me, I'm not going to remember that house. I've never had a homeowner say I couldn't take an image. And then when you -- looking at stuff online, I haven't gone down the legal rabbit hole of what data you're allowed to look at, but my assumption is if somebody is posting a real estate with photos, floor plans, and tax records are out there that show you maybe like, you know, build -- you can look up building permits on most city jurisdictional websites. That's publicly made available data, and we're using it for the purposes of making, you know, home more efficient. We're not -- so maybe a disclosure about how you're using that data. You may need to come up with at program level. But we haven't run into any issues with that to date.
Steve Byers:
That's great; thank-you. There's a couple of questions, I think, are really interesting around -- and Jessica you touched on this a little bit, for sure -- around using computer vision in remote inspections, though. So and the question is like, is anybody doing it yet? So I'll open it up there. Like what's -- maybe Jessica, if you might jump in with what you might see is the potential for using computer vision in inspections that we might be doing remotely, in more of the mode of like what Valarie and Jason are doing.
Jessica Granderson:
Yeah I think it's a terrific opportunity. Some of the, you know, on the technical side of things, so the thing to be aware of is the extent to which you have a sufficient repository of training data that you can use to tune your learning models so that you can then process every new image that comes back in from the field. So I think it's that would pretty much be the crux of it. I think that the techniques that we have available, then there's always a certain level of effort in figuring out which promising algorithm you're going to pull out of the computer vision domain, and then tune for your building application. And that would be some of the initial steps to think about. Can you get the training data, and where do you start before you do all of the tuning? Once you have that, and have validated it, then I think the workflows and processes to turn the crank on that are very accessible.
Steve Byers:
Yeah, I would agree. At Energy Logic, we did a little bit of playing around with computer vision in this exact realm, decided that it wasn't necessarily fruitful for us at the time. I think it's -- for people who don't know, that training data set, that's the grunt work. It takes so much time and effort to build the training sets to get to where you can have confidence in the results. That's kind of the grunt work behind the magic. So Valarie or Jason, anything you want to chime in on computer vision issues? Cool? OK, let's see. I'm gonna go to some broader questions. Bear with me one second while I switch over. So I think this one's relevant very well: So post-COVID, and you answered this to a large degree, Valarie, I think, we probably all kind of know the answer, but post-COVID, that are we going back to the old ways?
Valarie Evans:
I don't know how anybody could go back, whenever, then whenever everything is being shown to be more efficient. Yeah, there's going to be challenges, but it's all in forward motion. So we went from paper plans to being at the counter to writing permits and to physically stamping to 100 percent electronic submittals, inspections. I can't see us going back for that, but to continue to be more efficient as we go forward is definitely going to be something to look at.
Steve Byers:
Yeah, I think that probably is the answer. And I've been given the "we're running out of time" warning. I want to ask one last question. I hope we have time. If there was one thing that I highlighted here, if there's one thing that's missing that could be invented tomorrow, what would that one thing be that would get each of you excited? I'll give each of you like a real quick answer, anything you want.
Jason Elton:
I would just say like, if we ever get to this place where you could have an indoor drone that could move around the house taking thermal imagery, you know, taking dimensions, it seems like way out there, but there's a lot of that technology that exists. I mean, I've got trade partners now that put a device in the rooms and take 3D imagery and video for their clients. So it seems like it's a possibility down the road.
Steve Byers:
Yeah, that's a good one.
Valarie Evans:
I would just have to say financial stability to be able to move any new programs forward, like the ability just to make decisions and run with it and be more efficient in that you open the door.
Steve Byers:
Awesome. Jessica, any fondest desire?
Jessica Granderson:
Fondest desires, well, you know, the researcher's constant desire and delight is more data, all the time, in all ways.
Steve Byers:
Awesome. Well, thank-you all. I'm gonna -- you guys have been awesome and I hand it back over to Jack to wind us down.
Jack Mayernik:
Great; thank-you so much, Steve and Jessica, Jason, Valarie. Those were fantastic presentations. Great way to start us off. I think I'm handing it back over to David or Jeremiah to introduce our next speaker.
David Nemtzow:
I think fair. It's David. I think you're handing over to me, Jack, unless Jeremiah needs everybody's attention. ... Thank-you, yeah, for me, thank-you, Steve and panelists for that conversation. I'd almost, you know -- I'd almost be interested if on the last topic if anybody disagrees with what I said and Valarie said and Steve said, does anybody think we're going back to the old normal post-COVID, but you'll have to save that for the chat function.