You are here

Hi, guys! Welcome. Nice to have you on today. It's always a challenge for me to speak into my computer and not be able to see your faces, but I'll do the best I can.

And we'll just work through these. We're excited to start sharing with you what we've learned in the last few months. I always mention NGLS is a partnership with IALD and IES, has been from the very beginning.

Here's what we'll try to cover today. We'll go through a little bit of background. I'm going to assume most of you know something about NGL and NGLS, but I'll go over that just a little bit.

I'm going to talk about the evaluations themselves a little bit, so we have a framework for what we did. We're going to talk about what we saw, what we've learned, at least initially. We're still working through a lot of that. Competition 2 just took place about a month ago, so we're still working through some of the details, but we'll share what we know so far. And some next steps.

Here's our steering committee. I'll go over this quickly, but just gives you a feel for the folks that go into putting the competitions together. A lot of really good folks. They're very involved. We talk almost every week. It's a really good group.

When we switched over to the connected lighting part of the competitions of NGLS, we brought on board an advisory panel. And they have been instrumental, really, from the beginning in helping us shape and coordinate with what's going on with other groups, utility groups, and their programs, as well as DLC. So they've been a really big part of especially the formation of the new competition.

Those of you that are familiar with the old NGL, we looked at tons of products over the years. We were very luminaire-focused. We published catalogs, gave advice on good products that were out there. We started, like Terry said, in 2008, so it was the wild, wild West. And really played an influential part in documentation and subjective reviews where, you can see in the photos here, it was very hands-on.

But we've moved on. We decided in 2017 to shift our focus, rather than seeing so many products in a short period of time, targeted evaluations, and all that. We shifted over to connected lighting systems. We had a pilot contest in 2016. And we quickly learned it was a really complicated process to look at these connected systems, and so we switched to onsite installation in real-life classrooms. And that's where we are this year, and that's what you're going to see today, is how that has gone.

From the very beginning, we began to see that we really needed to slice and dice things in a way so that when we came away from an evaluation, we had specific tangible information to share, rather than observations that were anecdotal in nature. So the more we could focus ourselves-- we were comparing apples to apples-- helped us to do a better job of that. And so we divided into a number of different types of evaluations and number of competitions. So far, we've done competition 1 and competition 2. I'll talk about that a little more. And then we have the layers of evaluations that have gone on with those. Installation and configuration, performance evaluation, and then ongoing user evaluations.

Here's where we are in the process. In the summer, we did competition 1, the installation and configuration. You'll see a lot of information on that today.

And then the performance evaluation we did in September. We installed competition 2 in January, planned to have that performance evaluation in the spring. And the user evaluations. The group of systems-- there's 12 of them now that are installed-- will then go as a group into the future, in the user evaluations that will continue over the next couple of years.

So the focus of competition 1 and competition 2. It's important to understand our mindset. So remember we needed to focus in, so we're comparing apples to apples. And so we decided to take the easiest ones first. We were so naive, how easy we thought they were going to be.

But our main criteria for easy was really how the systems were marketed. So if you were out looking for a system for your upcoming project, how would it be explained to you, as far as how it would install and how it'd configure in your setting. So, sometimes, we would look at entries and we would say, hmm. I wonder how easy that's going to be.

But if it was marketed as easy, if that's the way it was portrayed, we put it in this bucket. Intended for contractor setup and configuration without prior training, without manufacturer assistance. That we were not assuming that the manufacturer was going to be onsite and there was going to be a special orientation meeting in which you stepped through the installation, and everyone was on board and knew how to do it. We were trying to see how would it be if that did not happen. We know that's the worst-case scenario, but that's how we set this up, and it's important to understand that.

So, defining easy. We really struggle with this, in trying to figure out how to approach it. What really makes it easy? What makes it easy to install and configure? And not only just configure, but to reconfigure if something changes in the space. Who's going to do that? And who decides?

So the role of subjective evaluations. This is the niche that NGL has always played. We have real people in the field looking at products, feeling them, understanding them, touching them, taking them apart. All those kinds of things would have made NGL really different from other kinds of competitions and other kinds of evaluations.

So we wanted to see how we would approach really looking at these connected systems and answering some of these questions. What's too complex? What makes it easy, like I said? And what maybe needs to be modified in these systems, in a lot of different areas? And we're going to talk a lot more about that today.

So they can be used by a broader audience. If these systems are really going to be widespread and used where a manufacturer can not be onsite for every single installation, what really needs to happen to help with that? And could NGLS play a role in that, highlighting those things, and helping the industry move on to deal with some of these root causes of confusion and helping different types of users?

So think about all the challenges that go into controls. And I'm sure these aren't new to you guys. But there's a whole different vocabulary, different language. There's different parts and pieces. Instructions are different. All those kinds of things we thought about when we were formulating the competition.

And really at the core of what we're trying to get to is this issue of configuration complexity. It's one of the major areas of our connected lighting systems programs that Michael Poplawski leads. So how do we deal with this? And we really began to see that NGLS doing things in the field could really try to bring to light some of these things that are very difficult and subtle, when it comes to configuration complexity.

The other thing we tried to do is model the entire process. We did do specs, and had drawings, and had the back and forth with the manufacturers, and had a bill of materials from them, and what would actually ship. So we could see where there would be other issues in the chain with these new systems that had more parts and pieces at different kinds of ways to understand then what's going on. So we did try to model that and we will be reporting on some of that. I'm not talking a lot about that today, looking mostly at the installation and performance evaluations. But we are trying to do this portion of it, as well.

So those evaluations have taken place at Parsons. It's been a really important partnership with them. They've been very supportive. They have a sustainability plan themselves for their campus. So they've been a really good partner and great to work with. And all the systems have been installed on their campus.

This is what we required in our specs from a control performance. I'm not going to spend too much time on it, but it's pretty basic. We had them do two zones so that when we did reconfiguration, we could see if they could change the zones. We did occupancy, daylight harvesting, high-end trim, and those kind of things.

It's just a very basic spec that every-- the manufacturers had this spec beforehand. They knew that's what we were going to look at. The installers had it in hand to look at when they did the configuration. So everyone was on the same page.

So the installation and evaluation process. We did it in a set of stages. They installed the luminaires, and we stopped and had them evaluate that. They installed the controls, and we had them stop and evaluate that. And then we concluded each of the installations with-- well, we videotaped everything, as you'll see.

That's why we have so much video to show you. But we also did a videotaped interview with the installers afterward. We knew that there would be so many subtleties to what's going on with this, because it’s a standalone process, that we wanted just to record everything we possibly could.

We also did a performance evaluation, in which we had a different set of judges come back at a different time, after all the systems were working properly. So they looked at the lighting performance in the room, as well as the ease of use of the system itself. Is the system really performing to the spec that we gave them, those kinds of things. You'll see some video from some of those as well today.

OK. Speaking of videos, I'm going to start with just an overview, just give you a feel for…

[VIDEO 1 -- MUSIC PLAYING]

Well, I think the video worked. It works for me over here, so hope you guys could see it. All of that footage that you just saw was from the first competition-- installations from the first competition.

Here's a few photos of the second, with the retrofit kits. So you can see a few shots here. We've got a few videos from that evaluation as well. To give you a feel for what happened there.

Here are the participating manufacturers. A big difference with this version of NGLS with the connected lighting systems is it's been a very open, cooperative process with the manufacturers. And we're being very open about who entered, and what their systems were, exactly. And we had lots of conference calls with each of the manufacturers, and reporting what we learned.

And we really were wanting it to be a collaborative learning process for everyone. As we were reporting things, we told everyone we're going to talk in generalities. So even though we still used competition language because that's how we want to sort of know the… we're not really going to come out with one winner.

We're going to talk about pros and cons and issues we saw across all the systems but not point and particularly with anything negative. Most of the things that we're talking about-- all the systems had pros and cons on each of the areas. So that's how we are approaching this.

And one of the big takeaways, I think, is the different approaches that each of the manufacturers took. It's obvious they would do that. It's OK that they've done it. It makes sense, and all that. But understanding that really helps you understand the differences in the systems and how to specify them, and all those kinds of things.

So we're going to talk about several of these partnerships meaning some of the systems were totally proprietary, and the luminaires were submitted by the manufacturers. Some of them were partnerships, and the luminaire manufacturer partnered with the control manufacturer, and that was an interesting communication issue, in terms of who was in control of what, and who you asked, and how much they knew about the other. Those kinds of things are things we'll talk about.

And the level of preconfiguration and reconfiguration and the types of tools that were used, and the level of assistance that was assumed-- was it assumed that you had to get on the phone-- those kinds of things were very different across the board. And just how their systems went together. What their wall controls were like, where their sensors were, how much wiring. Those kinds of things were all differences between what we saw.

We did allow manufacturers to be onsite for the installations-- not for the performance evaluation, but for the installation for both of the competitions. It was very useful, and they were very cooperative. It just was a really good-- well, I can’t really say it was a great experience for everyone. I think it could be kind of frustrating for some of the manufacturers that were there, just because we did not allow them to speak or help, so that presented some frustration. But I think they really did learn a lot. And it's an important part of what NGLS is all about.

Communication. Probably at the top of the list of things we discovered all along in so many different aspects was communication that went on, or the lack thereof. It was an important thing to highlight.

We all speak a different language. And the crashing of these worlds together has posed a number of problems. Designers are describing what they want in a certain way that's absorbed by the product engineers and the language they're accustomed to using. And then they're putting out instructions and documentation that the contractors are trying to understand, that are different than how they're accustomed to doing business. So all along the way, there's all these ways that there can be confusion.

Well, I'm going to start another video here to demonstrate that.

VIDEO 2

Task tuning.

Task tuning, required capability, no specific setting specified. This here, I don't understand what--

And this one—high end trim with task tuning. What's that exactly?

It has to be based on a set point.

Based on some set in.

Yes.

This set in is coming from these people, IES.

IES.

IES has some recommended light levels that they--

Yeah.

Based on--

On daylight.

Yes. How much light levels should be in a space for somebody to function properly.

And what… ?

Well, IES, they basically set the light level. So based on the amount of light, the light is going to see the amount of light coming in…

And react to it.

Yes, react to it.

I love that these people-- IES people-- this is what they say to do. It's very enlightening.

Documentation. How much is too much? What's the language that's used there? We had a lot of information that we gathered around this subject. It's extremely important what's in the box, what they can find, how they were supposed to find it, and could they understand what they were reading. Let me show you another video that illustrates this point.

VIDEO 3

I see this now. This is what I can see in the app. It's getting to this point-- documentation, and not the right pieces of documentation, because I have been given every piece of documentation. Do you understand?

I do.

I'm given loads of different pieces of information when this is the one piece of documentation that I actually needed. And that's what I'm seeing.

What else did you get?

I have all of this.

This is an interesting point. We have been talking about the need for documentation, for more clarity. It's a big point trying to figure that out.

And we have found that, in many cases, they had so much that they couldn't find what they really needed. So there's lots of different facets and layers to how to deliver what the installers really need. And you'll be able to see, in videos and in pictures that you'll see all across, what a big mess that had the tendency to happen in these rooms, with so much going on. And trying to find everything. And then you've got all the parts and pieces, and you've got the phone or the computer app that makes passwords. There's just a lot of different issues going on when you get to the mess of installation.

Configuration tools. So this was another big part of what we observed and differences between the approaches, as well as the challenges that went on in each of the rooms that they were trying to get the systems installed. You can see here three different main approaches. Hand-held tools, phone apps, or computer front-end approaches. And it was interesting to observe all the subtleties to that.

And you can see over in this right picture where that was the IT guy in the initial installation. Each of the rooms, as they were working through things, they would have to be, hey, where is the IT guy? We need him over here to help. They were unfamiliar. They were a little intimidated, perhaps. And they wanted the young guy to come over and figure it out for them, because it was unfamiliar to them. Which is an interesting dynamic, I think.

So another video. I told you there were going to be lots of videos today.

VIDEO 4

--the light fixture, but there's a little antenna in this box, and they need to sync up with one another. I've been trying for a couple of minutes to get it to happen. I haven't been having much success.

It's a specific location on it.

[PHONE INDICATOR SOUND]

Unbelievable.

[BLEEP]

That is particular. If I've ever seen particular, that's particular.

That is a very polite word for it!

I don't know.

I thought we were allowed to use four-letter expletives.

Don't get me started. It'll be a long day if you get me started.

It's like three, four letter words all wrapped into one!

Yeah.

You are caught on it? Which is something of a bar code. So to read them regularly, you would hold a camera up to it where it would read it.

But this, apparently, has some sort of an antennae inside it. So I was holding the camera up to the antenna, or holding-- apparently there's something on the back of these Galaxy phones which is right here. The second I held that up to it, it configured with it straight away. But I've been playing with that for 15 minutes.

It's asking me to do it again.

 [BEEPING SOUND]

Wow. That sounds interesting.

Yeah. Well, I've created a profile for it-- room 1108 at 60 16th Street. I'm guessing at some point I'll probably be able to add more light fixtures to that.

That is a correct assumption.

Yes. Yes. So I just created a profile for the room. I'm not quite sure how to control the room from the phone or even if that's possible yet. I haven't reached that far. But for right now, I'm quite satisfied that I have one fixture done, and I'm hoping that the rest of them do not take me three hours each to do.

So that gives you a feel for a particular configuration in each of the rooms. And I show videos to just give you a feel this is typical with different kinds of systems, even though they might have different approaches and they have different challenges. It was interesting to observe the configuration and how they had to get adapted to the different systems.

The number and types of components. You can see here in these photos they ended up with quite a collection of things that they were sorting through and trying to figure out. And it was a new world for them, and much more complicated than what they were accustomed to.

The wall controls. This was interesting. The different approaches-- we tried to categorize, and I've given you a feel for the actual entries that have taken this approach, and what they look like. These are actual pictures from the rooms after the installations. So some of them are a little rough.

But some of them were preconfigured paddle switches. They were typical of what you're accustomed to seeing. And they had certain functions that they had preset to work within that approach.

Then, there were a number of companies that did a preset multi-button switch. So they were still preset, but they had separate buttons that you could give a feel for what was going on, up and down, and a little more instruction.

The configurable paddle switches, they sent the system. They were normal paddle switches, but each paddle did a different thing, a different function. So that was their approach. And then you had configurable multi-button switches. So these are the ones that you could change in the field.

So that's just a way to get that. It's a very important aspect of the different approaches that different manufacturers are taking. And I think it's an important one we'll have to understand and work through. Are there places where things could be standardized more?

Those kinds of things we're thinking about as a committee. Approaches to help with this, to define the way it is, those kinds of things. But it's an interesting aspect of what we observed in the evaluation. And let me show you the last video.

VIDEO 4

Should I push the little one?

Yes.

Oh, is there a little one?

There's a little one.

That's what you mean. There's a skinny one. Fat one and a skinny one.

Why didn't you say so?

Skinny one dims.

So we have four controllers over there?

There's two switches. Each switch has a fat side and a skinny side.

You might have to get up and look at it.

Just go back to the high again. The bifurcated fat, and the bifurcated skinny.

Bifurcate.

That's tough, right?

And if you press and hold the fat side, does it do anything?

Wow.

Oh, yeah. Well that was off, I guess. Sorry. Let's push it again. That's off.

Fades to off. OK, so one side is on and off. The other side dimming.

The big side is on, off. The other side is dimming.

And so they're zoned front of the room, and back of the room.

So the on, off function is zone one. The dim function is zone two, on one of the buttons.

Right. But now I'm trying to dim back up.

What?

And it's not doing anything.

So dim down, but not dim up?

Right. OK.

So, as you can see, even in a room full of lighting professionals-- that was part of the performance evaluation. In each of the rooms, we had to figure out what everybody was trying to do, what their interpretation of our spec was. And I think we were a little surprised by this, and it was an interesting process. So we're still working through characterizing all that, and thinking through how to talk about it all, but mostly they're going to give you a feel for what we saw.

So as I said at the beginning, we're still sorting through everything we saw, everything we've learned, all the video footage, all of that, to come away with lessons learned and recommendations. And, in a lot of cases, it's exposing a problem. But then, trying to figure out what can happen next. What needs to happen next. How can NGLS help with that.

As far as how we plan to release what we found, it's such a short presentation today. And we're not really ready to give you very specific things, because we're still combining what we learned in the second competition. But we do plan to release things, not in a big, full report that's going to go into excruciating detail. But we're going to try to release things as soon as we can, little bits at a time. That's the reason we have this webinar now, so you get a feel for what we did and how we did it, so that you'd be interested in maybe getting involved.

And next steps with this. We do plan a number of conference presentations over the next few months. We're looking at doing articles in some target publications. We will put information on our website, of course. And we're also looking at doing some one-pagers by topic and audience focus, because a lot of these issues are very specific to the different audience approach to what's going on, and what kind of information they might need. So that's one of the ways we're trying to put information together that can be helpful.

And on each of those topics the idea is that we would have-- for many of the topics-- that there would be groups forming to be able to talk about some of these issues. So in some cases, the thought is, we're coming out of this and we're saying, this is how you should do it. So with how manufacturers see us doing it, and that's the best way. That's not the answer for this. We're hoping that we're going to expose issues that we can work on together as an industry.

So how can we help? We have talked about helping industry collaboration on things, like vocabulary is a very big one, so we're all talking the same language. Specification language, models-- how do specifiers, lighting designers really describe what they want, what they need in a space, what they intend, the performance they intend, in a language that the IT guy can understand, that the product engineers can understand? We're in a different, a new world here. And some consensus with that language, I think, could be very important.

We want to continue to help that communication between the specifiers and manufacturers. All the manufacturers, I'll say, have been very open to that, very engaged, really seeing us as valuable in their product development. All these products are very new to the market. And this is really a key time to help them with that.

And so we want to continue to have that communication happen, and communicate everything we learn as we go. So we're going to be receiving more user feedback as the living lab continues. We want to continue to communicate that, as well as these lessons learned that we're talking about today.

And we're framing how we may start looking at things in three ways-- if there's a real consensus in the industry about how things should be done, that begs for some sort of standard, some sort of model. Let's all agree on this common ground. We're all going to do many things different, but maybe there's a common ground that helps us all. So maybe there is a standard that needs to happen in some cases.

If there's multiple ways of doing things that are all valid and important, maybe there are some templates, or some model sets that can be developed, that can help. That people can start with when they're starting onto a project, that help communication and help speed things along. And if there's no consensus, we just don't know what to do, maybe there's more studies that need to happen. The photo on this slide is our connected lighting test bed. And we do have facilities here at PNNL to do some of that testing. So maybe there are places that we really need to look at it more closely, and figure out what to do next.

So how can you get involved? We do plan for future competitions, as you saw on that one slide early on. We still are looking at competitions 3 and 4. We're still looking at an outdoor competition, a connected lighting competition focusing on parking lot systems.

As you're learning things in the field, if you can share those with us, it all adds to the collective story of us learning together. And we are planning on forming some working groups, particularly on specific subjects. We're hoping maybe to gather some folks to start talking about that at LightFair this year.

But we'll hope you'll get involved. It's one of the reasons we wanted to do this webinar today, early, even though we're still working through all the findings and what we're going to share. We know there's a lot of things that we need industry collaboration on, and communication can be really important. So we want you involved with that.

As another way to be involved, we are having an open house at LEDucation. That happens in March. We'll open the systems up. We did this after the performance evaluation, where we let people come through the rooms and play with the systems.

It's a really good learning activity. So if you're in the New York area, and want to be involved with that, that's a little more specific. You can email us, and I can tell you more.

That's all I have. I think we've been gathering questions, if anyone's had any.

Thank you, Ruth. I have a couple of questions here. Here's an easy one. Of all the problems that you mentioned, which has caused the greatest difficulties for the installers?

Well, I think just getting started. As evidenced by what we saw and heard in the videos, everything is new and different. So they were trying to get their minds around what they needed to do, all the different parts and pieces, finding the right kind of app, and all that kind of thing. Once they got their mind around that, they did much better at moving with that first luminaire. The first, with the retrofit kits that was especially true. Doing that very first one was a lot of the challenge. Then they did much better.

There really wasn’t a lot of time to go down the learning curve.

Right.

A second question. Are all of the systems working at this point?

They're close. We do have a few tweaks on a couple of them that were installed in January. When the first installation happened in July, we did have some ongoing work and tweaks. But we did see a big difference in the second competition that most of the systems were ready and functioning. We're in the process now of going back through and making sure they're all functioning to our spec.

So Craig Bernecker at Parsons is-- I should have mentioned this earlier-- so he's heading that team there. And he has students on campus that are going around and confirming performance, verifying things as we're working into the user feedback part of the-- I'll mention we also had a process of handing off the systems to the facilities staff.

So that's part of our next step, for them to understand the systems, and for us to communicate with them that we want them to keep talking with us about what happens as users are in the rooms. What kind of issues they might have. I guess complaints would be one, or if there's just tweaks that need to happen. What goes back and forth there. So that'll be a part of that user evaluation into the future.

All right. I have a couple more questions. Here's one. Early in the presentation, you showed that tunable luminaires were in the more complex installations. Why are they more complex? And I can't tell whether this is a rhetorical question or not.

Well, I don't think we were saying that they were more complex. I think mostly we were saying, part of the reason we're engaged in the competitions in these stages would be for us to be ready for and systems to be developed to a place that would make sense to do this kind of evaluation, and typically the performance evaluation, and all that. So we pushed it along the schedule so that the technology was there, and that we could evaluate them properly in the context of these more complex building integrated kind of systems.

I will say, after doing the first two competitions, we're kind of rethinking what competitions 3 and 4 might look like. And we're definitely open to your thoughts on that and how we might segment those a little more. Slice and dice is the word I use a lot. In our first conception of what competition 3 might be, we probably were a little overzealous, in terms of what we could realistically compare and contrast, and all of that. So we're kind of rethinking that now, and maybe thinking about where color tuning would be another aspect of that.

And it's not that you can't have an easy system that color tunes. We did have one entry that had a color change capability. So maybe that's really that question. That it can be done with an easier system, but we wanted that aspect to develop a little more. And I feel a little better prepared for evaluating those aspects of performance.

Was there a preference for wired versus wireless systems? I guess this is a question of which type did manufacturers tend to provide? I think is what that question really means.

Yes. We definitely saw more wireless systems. There were just different systems that were a little more of a combination, as to did they have certain aspects of the system that were still wired. But for the most part, we saw mostly wireless. Do you want to add to that, Dan?

Well, I'd like to say that since we generally didn't have the same contractor handle more than one system, we really never had a serious way to evaluate one system versus another, from the point of view of a given installer. So we really don't know what the installers would have preferred. We can still do that with the manufacturers provided.

I had misinterpreted the question. Sorry. I misinterpreted the question. So the question was, did the installers like one--

No. And you didn't misinterpret it. I changed it, if that's what the intention was, because I didn't think we could answer.

[LAUGHING]

OK. But that is a point that I neglected to talk about, was that we did have different installers in each of the rooms, because we wanted them each to have a level playing field. But we didn't want to have one installer working through all the systems, and by the time he got to the last system, he had this thing down. And it wouldn't have been a fair comparison.

But I think it is fair to say that anything the installers were not familiar with was harder, right? That makes perfect sense. And we had a lot of instances where they looked at something, and they had a preconception of how to do it. It looked just like something they'd already done.

And so they just threw all the instructions away, and went to doing it how they thought they would do it, just like we do when we pick up our IKEA instructions, right? We think we know how to do it. Without guidance, sometimes, we have a mess.

But that happened on several occasions. I think it's an important thing to note, is that if something is going to look familiar, and the installer should do it in a different way, you've got to really highlight it. Make it very obvious that this is not what you're accustomed to. That's an important thing.

If I were to add, the systems that had serious problems were both wireless and wired. That is to say, both of those types of systems could run into serious problems.

The question asks, how long did it take installers to get the systems up and running? Or, more accurately, what was the range of time?

The initial installations, competition 1, they were luminaires. And so we gave each of the rooms two days. Now, you've got to remember we had all kinds of stopping and doing surveys. And we had videos, interviews at the end, and all that. But we gave them the two days. Some of the systems still weren't ready after two days, right?

We had some issues in that initial competition. I don't know if I can give you numbers. But some of them were way faster than others. In the second competition, which was the retrofit kits, we assumed that one day would be enough for each of the systems.

And we had a couple of them that we had to overlap into the second day. And usually, that had to do with adjustments with the fixture housing, that they just literally couldn't get the kit in, so they had to physically manipulate that, and had to go get equipment. So things like that really held up the process.

But for the most part, both competitions-- like we talked about from the beginning, once they got the first one, particularly with the kits-- but once they did the first one and figured that out, it went much, much faster. But these are some of the numbers that we're going to try to give some information around, our little ranges. We may not report on exact minutes by manufacturer kind of thing, because our process was not real, in that we were stopping and starting, and video people, and surveys, and all of that.

So we probably won't report actual 20 minutes and 2 seconds. But we will try to give ranges to be a little more specific with it. I know that's what everybody wants, but it was difficult in this context for us to be able to provide that in a fair way or an accurate way.

Two more questions. To what degree were the installers equally qualified?

We had two different sets of electrical contractors for the first two competitions. And they were teams that worked together. There typically was an electrician and then a helper.

I would say on competition 1, they all seemed to be pretty equal in their years of experience. And that's one of the things we talked to them about, how many years of experience. But neither sets of teams that were experienced with connected lighting systems, in the second step were not experienced with these kinds of retrofit kits.

So we're trying to get inexperienced installers. For the second installation, in the video you saw, with the guy with the preconfiguration with the documentation, Anthony with the strong Irish accent. He's a team leader. And he definitely had the most experience of his team, it seemed like to me.

And then I heard on the first group of installers, they had their IT guy get rotated around from room to room. That definitely helped them to have-- and we talked to them about that. Is that what you would normally do?

Oh, yeah. Oh, yeah. That happens on real jobs. If we encounter this kind of thing, we're going to get the kid to come and help us. I don't know if that's helpful, but we did try to get a similar level of experience.

Another question. Are you considering evaluating PoE lighting and control systems in the future?

Yes, we are. And we've had that suggested. We've had that suggested. Particularly, we've had an emphasis on it. Here at the lab and the CLTB and the recent reports that have come out. We've been waiting to see if it gets to a place where there's enough systems for us to look at.

So appreciate your views on that. If you feel really strongly about that, send us a note. But yes, we're strongly considering that.

Can I be permitted to editorialize one last question?

Sure.

So, based on your experience so far, would you say this product category is ready for prime time? And I hope we're still friends after this.

[LAUGHING]

We've been surprised by difficulties in the field. So, the term prime time. These systems are out there and being installed. And so I feel like our job is to really help that process.

So I hope they're all ready for prime time soon, and I hope we're helping with that. That's our intention here, that all these manufacturers can be ready for prime time and learn from this, and installers can become more and more comfortable in using them. You want to answer that question, Dan?

No, that's why I asked it. Ruth, thank you again for the presentation. And thank all of you, all who participated in today's webinar, brought to you by the US Department of Energy Solid-State Lighting Program.