How IP Controls are Changing Building Automation Controls

Transcrpit of BAS has Gone IP 

Scott Cochrane 0:00
Boom, here we go. And welcome to our second session. These are the automated buildings free industry sessions. And we’re here for a second session. And this session is we’re going to be talking about how the BAS industry has moved over to primarily an IP industry Bas has gone IP. I’m Scott Cochrane, I’ll be your host for this session. A lot of you were in our first session, I did my intro do a brief on my work for building controls distributor, we distribute products from multiple BAS vendors. We work primarily with contractors BAS contractors that install them. And we’re very fortunate that through our customers, we’ve seen countless projects, countless implementations. And we try to learn from that at conference, apply and share those learnings with the industry and our customers to help us all evolve in new and interesting ways. And so that’s why I’m here hosting this today. And I’m glad to be here. So. So with that, I’d like to introduce my panelists that are with me today. We’ll start with the interim panelists. And I’ll start with Greg Fitzpatrick, Greg, Greg is with Cochrane Supply as well. And Greg helps us by working with consulting engineers and helping design ot networks for bas projects. Greg a little introduction.

Greg Fitspatrick 1:25
Yeah, thank you, Scott. First of all, I’d like to thank you all for coming to this session. I think it’s going to be pretty informative. Like Scott said, I work in business development for Cochrane in engineering. In the area of IoT and integration, my primary responsibility is calling on consulting engineers because they are more behind the times than most of our contractors. Real quick, by show of hands, how many Consulting Engineers that we have here? One, contractors, or systems integrators? How about how about end users? And distributors? Okay, so we don’t have many consulting engineers here. But I’ve been in the industry for over 25 years, I’ve worked as a consulting engineer, ran my own firm for a number of years, I’ve worked for both training carrier, selling equipment controls, turnkey projects, and the likes, also worked as a director of facilities for a large school district. So I’ve been around the block, I’ve seen a lot of things. And I also own my own commissioning firm for years. So I got a chance to see controls being implemented. I’ve gotten the chance to see a lot of technicians out in the field when controls were supposed to be completed, but they weren’t. Right. So when we go functional test controls were completed. So now that we’re entering a new stage, or ecosystem with IP controls, one of our main focuses has been to get the consulting engineering community up to speed with designing systems and allowing us to move forward with that technology.

Scott Cochrane 3:10
Cool. Thanks, Greg. All right. And also we have with us also another Detroiter, I bring my crowd with me, I guess, but this is John Guardiola. He is director of facilities with Bedrock, they started in Detroit. And Bedrock is one of the first property owners that Cochrane supply has worked with that evolved their all of their networks over to IP and over to like their own dedicated OT networks and, and they did this stuff like boom, like eight years ago, is when they started doing this, and now they see the benefits of it at scale. And we’re really fortunate to have John here from bedrock. John.

John Guardiola 3:54
Thanks, Scott. Yeah, John Guardiola, VP of engineering and sustainability. And I think probably the clearest thing for me to say is I’m the old school engineer who before Bas was around Bas was by myself and other engineers running around sitting in mechanical rooms and kind of evolved from there. But when I joined Bedrock, nine years ago, we had a vision of buying lots of properties and expanding in Detroit and Cleveland, and we knew that right away, we would have to, you know, Dan Gilbert is the owner of the company. He used to say we’re more of an IT company that happens to sell mortgages. What that tells you is that everything we did from day one was based on technology and the smart building platforms that we have now was different. We started years ago creating smart specification for smart building and installs etc, that we created put in place and then as we bought buildings, and and the portfolio grew We basically had these specifications that we sat down with our architects, our VP engineers as they were designing either IT or ground ops, etc. And we can roll that. And the last thing, I’ll say to that, and then Scott knows this, because we’ve been partners, and we’ve created in house controls team that oversees this force experts in the field to help solve that. So lots of great experience on these folks. And it’s a it’s definitely a lot of benefits. You know, once again, I’m not the technical guy, but I’m the guy that asks, Can I, should I, I want to and get it done using our IT.

Scott Cochrane 5:39
Yeah, John’s been a fearless business leader for technology for as long as I’ve known him at Bedrock. It’s been a lot. It’s been a very fun journey, and very exciting to talk about. Okay, and also another from end user perspective, we also have with us, Sabine Lam from Google. She’s up here remote, she’s coming to us from California. She’s Google, interestingly enough, they’re interested in buildings, they’re very interested in buildings. And they’ve built quite a team around that. And Sabine is a part of a a very interesting group of people at Google that are, are probably vetting out some of the best practices of tomorrow for for especially for IT networks and buildings to support him. So Sabine, welcome to the program. Thank you for being here.

Sabine Lam 6:25
Thanks. Thanks for having me out. Everybody. Everything’s going well in Vegas. So yes, again, from Google, I run governance and product development. So I’m not only the you know, I’m listening to the business, which is the core business, what do they need? And and we’ll focus on how and how can we do it? I’ve been more scale, how do we automate the process? How do we make it work with the lens of an IT company? So the background? Is it background is big data? And how do we apply that to this industry? Operation technology, so the solution we’re looking at, really are looking to focus on cyber security looking to make that change. We’re not accepting status quo. And we’re large enough, we think that we may have new evidence, and we’ll share our knowledge in the IT world I met, and I’m helping improvement industry and really solve for those complex, complex use cases that everybody has in mind. So

Scott Cochrane 7:33
awesome. Thank you for being here, we are super excited to have you. And then our final panelists, of course, we’d like to get the contractor point of view. So we have Scott Papay from Long Building Technologies in Denver. He’s also coming in remote on the screen here. And Scott has an long building technologies have been early adopters of IP based control systems. They’ve got a vast knowledge now of installing them the pitfalls, the good, the bad. So I’m super excited to have Scott on our panel to talk about some of that. Scott, welcome to the panel.

Scott Papay 8:09
Thank you very much. I wish I could be there with everyone unexpectedly ended up with a cough. Grant.

Scott Cochrane 8:20
Wow, you’re probably doing the right thing, Scott. So good thing, we got our virtual or our, I’d say our garage hybrid environment here that we we built on our own here this morning. But I’m glad to have you. So Scott, a little introduction of yourself in long and some of your philosophy on the IP industry.

Scott Papay 8:38
Sure, yeah. Long building technology’s been around for 50 years. We are in front of the Northwest, the continental United States as well as Alaska. And we are a house that is has installed Schneider Electric, Distech Controls, KMC, and we work on old Legacy product lines from back in the day. And as well as a were by current partner as well. So we’ve kind of seen the evolution of MSTP to IP networks, you know, from ARPANET, back in the day to now Wi Fi technologies and everything else. So yes, we’ve, we’ve been on the front end of this for a while.

Scott Cochrane 9:25
Awesome, and you’ve learned a lot of lessons, right?

Scott Papay 9:29
We’ve learned a ton of lessons. I mean, I’d be happy to go into

Scott Cochrane 9:32
No, save them. We’re gonna get to that. So. Alright, so thank you all my panelists for being here. So. So in order to kind of frame up this conversation again, we’re talking about how the BAS industry is moved over to IP. So let me let me kind of start with what does that mean? What are you actually saying there? Well, what we’re saying it’s kind of simple. It’s that the application controllers that used to be on 485 or lawn have now been moved over to Ethernet, IP, right? And they’re no longer 485. Well, these new controllers, and this is only been I mean, while some manufacturers started this five, six years ago, the mass majority of them really just started this about two, three years ago, where they created complete lines of bas controls that were all IP. So it’s brand new to the BAS industry. Now, when we first realize that is when I partnered, I brought Greg Fitzpatrick, he was working as an independent commissioning agent. And Greg, I took Greg to lunch and said, I’m gonna hire you right now, Greg. And he said, really, because why? I said, Well, we got a lot of work to do with the engineering community, because, because right now we’re going through this epic change where, therefore 85 drawings are wrong. They’re not CyberSecure. And they are not going to meet the owners needs of tomorrow. And so we, we made it our sincere effort to try to enable our local engineering community by sending Greg out and offering to do an education and free IP designs and things of that nature. And so with that, we were fortunate that we, Greg got quite a few projects specified around us, and we got to see those projects get installed last summer. So this past summer, and and these were the first real design projects that were all IP based that we saw get installed. Now, the first thing that we noticed about an all IP based project was that there were no gateways needed. So a lot of you have heard of the Jace are, are a serial to Ethernet gateway of some sort in a system? Well, we found out we don’t need those, they serve no purpose. The IP devices are smart enough, where the gateway is sort of a, you know, a lucky it, really, it’s only there could be added as redundancy and a good design, right? The the controllers themselves, were in prior times when we would specify things like sampling rates, or how much data it was to carry, or things of that nature. Those those specifications were obsoleted. And we replaced them with unlimited data capabilities into the specifications in terms of data sampling, data rates, things of that nature. Huge change. I mean, we saw we saw different people looking at that differently. We’ve got large industrial customers whom had been using bas systems from us for years. They got excited, they were like, wait a minute, I don’t have to worry about the sampling rate anymore. The number of samples, the data I’m bringing back were like, no, because IP, we have the bandwidth now, did you know that like an IP network versus a serial network has got like, like 10 million times more bandwidth to move data? It’s a ridiculous number folks. It’s it’s literally think about this. It’s like 250 6k versus 10 Gig haul, my gosh, you guys know what that’s like? That’s like, that’s like having a string of hair versus a 42 inch pipe. Okay. That’s the net difference in the connection to our new controllers, right? And so an incredible amount of new data can be moved at an incredible rate. And when you talk about people that want to do analytics on buildings, you know, we used to play with what we would call big data in our industry, oh, no, oh, no. We haven’t seen big data yet. These new IP infrastructures with the new new capabilities to move data like never before, we’re going to be able to see analytics like we never did, why? Because we’re going to be able to get every bit of data back into these models for analysis, okay, which was a total limitation before. And as we’ve studied the IP more, we learn more, we learn more, we learn more about these devices not only can communicate at a much more rapid rate, but they also have the ability to do many more things within the application, whether it be to have its own programming tool locally, whether it be to have the ability to be simply, you know, plug and play type capability into, into, into

connected networks. So we’re seeing because of the ease of being able to connect these devices just through things like physical patch cables and stuff like that. We noticed the time went down on installation for the contractors. And Scott’s going to talk a little more but we saw and, frankly, you know, from a contractor point of view, I think Scott would agree there’s a lot of projects now if, if they can do a redesign, they can actually give them a D doc to move it from 45 to all IP. And so we’re starting to see that as a byproduct of it. And as the owner start to take these systems on And then they start to see the benefits. What are they seeing? Well, it’s exponential because as BAS has gone to IP, it’s simply as got caught up with the other things that these people are dealing with, like their elevators and escalators and their fire and security systems. And so now they can holistically start to look at networks and potentially use them for total building control. If the use case makes sense, which again, you know, was never really a cost effective option. And so, really, while we sit here, on, I would say, our third year of this evolution, the capabilities of the industry is exponentially like, they’re 100 times more than they used to be. And we’re just now starting to see people truly take advantage of it. So So again, to frame up the conversation, I think let’s let’s start with, let’s start with the design cycle. And I’m going to I’m going to put Greg on the hot seat here. And I’m going to ask Greg a basic question, Greg. How do consulting engineers take this change, I mean, and I’m not talking like them, bringing in IT people, I’m talking about an MEP with a with a typical 485 bas design, tell us about some of the reaction you get when you propose and start talking about IP,

Greg Fitspatrick 16:13
they have been literally kicking and screaming. And if you know anything about the consulting engineering world, is that they don’t like change, they don’t respond well to change. Because your specification has to be updated, you have to think about coordination within your office. So if you’ve got a mechanical and electrical engineering firm, then you’ve got coordination between the mechanical engineers who are designing the systems and the electrical engineers, right. So if we’re talking about IP networks, or OT networks, now we’re talking about getting multiple disciplines involved. So what we’ve done is we’ve gone out and tried to make it easy. We’ve also introduced what I call a systems integration drive, right, which is a topology drawing to gives the systems integrator, enough information such that he wouldn’t even have to dig into the drawings to put together a price or to know if whether or not that project is something he’s interested in. So when you look at a systems integration drawing, it shows everything from the physical cabinet. Right? What we like to call our OT network, MDF cabinet that shows your server, if there’s a on premise server being deployed, it shows your server shows where your web supervisor is going to be deployed, it shows your network management switches, your front end network management switches, your VPN, your firewall, it shows every floor, the switches, the IP switches that accommodate the technology, and it shows the technology that’s got to be integrated. So what we’ve been doing is pushing that drawing out to the consulting engineering community, but we’ve been designing systems for them. And not only for the engineers, but also for MSI to get them to embrace the technology to know that they’ll have a partner in holding their hands to get them through the first project or two, right? Because I’ve had consulting engineers tell me that they now treat controls or BAS, like fire protection, they put on a performance spec and a one line diagram. And if you’re a systems integrator in here, by show of hands, how many times have you seen complete documents for a true integration project? By show of hands? Right, exactly. So what we’re trying to do is tell the consulting engineers that, hey, look, our ecosystem has taken a sharp left turn right now. And this is technology that a lot of young people are interested in. So get some of the young people at the firm interested in the technology, and start learning with technology, so you can get it in a set of plans and specifications. So that’s been our approach.

Scott Cochrane 19:03
Yeah, and so, but the really hard thing about the engineering community was that the MEPs, in the old designs would rely on the BAS vendor to supply essentially the operating system, the network requirements, the, you know, they kind of did it all for you in the, in the old days, right? Well, with these new IP based systems, the MEP is now challenge with standing up an IP network. Now, in some cases, they work with the owners and they these networks are things that the owners provide, but when the owner doesn’t provide it, now, the MEP actually has to provide a working it network, right, which is something that typically was left to the IT people to do, right. And so it’s a change. It’s a change for our industry now. Now we’ve seen Greg and I have seen a lot of in the design side where they bring it people into Help. And of course, that’s great, right? Because they bring a lot of expertise and things of that nature. But it does not really it does not relieve the mmap, the the actual mechanical engineer from understanding what that design is and why it’s designed that way, the MEP is still responsible for working HVAC system. And that includes that network that supports the controllers that are on it. And so, as Greg mentioned, kicking and screaming, no, I’m sorry, you have to learn what an IP addresses Mr. C, it is what it is. So so, you know, we saw that quite a bit. And but, but now on now that Greg’s had some successes. And now the we’re finding MEP s feel empowered, right with this new knowledge. And with the new flexibility in their designs, would you say so? Greg?

Greg Fitspatrick 20:49
Yeah, absolutely. And one of my approaches also has been to put a little beer on the MVP, right? Because when most cars, or most engineers are designing systems, they’re always thinking risk, right? They want to put enough on the drawings, to show intent, but not be responsible for the details. Right. So now we’ve got all this IP technology, which exposes their design to the internet, and cybersecurity risk now, right. So now you start talking cybersecurity risks to MEPS. Now, they’re interested in hearing our story, and taking on our assistance, and designing systems and helping them put together specifications that eliminate that risk, right, and help them with things like cybersecurity policies and things like that, that are typically not in a BAS specification. So we’ve been pushing out division 25, because most projects now are integration projects, right? So if you take a look at the CSI format, of division, 25, integrated automation, there’s section subsection, subsection, subsection, what we’ve done is we’ve eliminated a lot of that unnecessary information that we broke it down into all the hardware and software and policies that you will see in one of our systems integration drives. So if you’re at MSI, and you’re looking at a project that one of our engineers put together, you will be able to look at the systems integration, dry, look at the schedule, and look at the specifications. And they would all make sense. And they would all match because I take a modular approach to it, because that’s how consulting engineers think they think of systems as modular, right, you’re designing a chilled water system, you’ve got your chiller, you’ve got your pumps, you’ve got your valves, you’ve got your piping. So our success has come from taking that modular approach, and truly trying to educate them and not push projects or products.

Scott Cochrane 22:52
It’s awesome. It’s so so John, moving over to the inducers perspective in terms of so so bedrock, bedrock has essentially, you know, the property management group has its own IT group within, they’ve stood up a very immense ot network that connects to all the building controls managed by IT professionals. So you know, when you talk about owner, lad, you know, we always say everybody asked me, Scott, what are the best out, you know, best implementations, best integrations, I would say, when they’re owner lead. John, can you speak to what it means owner lead when it comes to like pushing this new technology and specifying it? Could you speak a little bit about that?

John Guardiola 23:33
Sure. What I mentioned earlier about getting involved from day one, being the being the being a building owner wanting to drive the technical specifications and what we needed or wanted as we grew, we created a thought process and it turned into a team talking about Bas and integration of content experts, where we wrote our own specs, we wrote our own specifications. We had people in the house who were content experts. I brought a gentleman into the team, and he, his education was network. It networks, there was all bas didn’t know what a VAV box was all these things, right. And then we also hired content experts who were either you know, install or the previous programs, etc. And we created a team that we then led the charge on and in our buildings and, you know, you’d mentioned you know, we have we supply our own network. You’ve mentioned the putting that on an MVP or the owners plan. We supply our own network, we have our own standards there. Our team ITT, we’re side by side, they build it and give the specs for security, etc, that we don’t take it. We’re on Google Cloud now because of the vast number of points we have 110 Billy

Scott Cochrane 24:50
got another customers to be in to hear that.

John Guardiola 24:55
And so, at the end of the day, what that’s created is a And I’ll say this out loud, you will talk about the struggle with the contractors and understand the VA as a portion. They have their requirements for building, whatever it is we need MEP, right. But there was always that struggle to disconnect to our nine to automate that Oh, is that right and also how we build our dummy supply the network’s we give not only mechanical specs on the BAS specs, and we provide these to any architect or MEP engineer that comes in the door on the buildings. We sit down, we’re upfront and every design meeting with our construction team, making sure that these are implemented into the drawing, we have the drawings made, like you had mentioned before, and also everything’s pointed out. And we’ve evolved to the point where it’s literally separate now on any construction or project that we do, where it’s MEP integration, it’s a network, they’re all separate sets of drawings, separate sets of spec that we can we basically oversee all of that. And it’s been great for us because it’s been able to keep us in the know of what we can do. Right, and our systems for the building owners. And then as we hired a Chief Sustainability Officer in the middle of last year, we’re getting a lot more company goals around sustainability, whether it’s energy reduction, you know, decarbonisation, etc. And it’s very nice to be able to say, Yeah, you know, we have that we have these points, we can create a process for you for tracking our energy by the submenus for tenants, all the way to base building, you know, smart spec, meters that we have. And it’s really helped us give a tangible ROI on all these projects so that we can continue to get the capital to upgrade our systems and be more sustainable.

Scott Cochrane 26:44
Yeah, and it’s so impressive owner light in their case on as John mentioned. So, you know, you talk about owner lead, and you talk about business integration within the owner. And as John mentioned, he’s got usually three different working parts on any project, any construction project, and they’re all coordinated with at his level to ensure that everything is looked at, right. So he has his traditional construction folks. He’s got a BAS team, and he’s got an IT team. And he corn, and they coordinate all three of those on every project to make sure that the technology is successfully implemented, meets the facility requirements, meets the ESG requirements, and meets all the cybersecurity requirements that it needs and the routing and all that good stuff. And, and so that it’s a to me, it’s a it’s a great example of success. Now, when we introduced IP controls to that group, it didn’t take long to get in their stack. Needless to say, why it’s because they understand their network, they understand their needs, they understand how their network works, and at the owner level, if you understand that bringing in these new IP based products is not that difficult. Not that difficult. Now, of course, as we’ve brought them in, we’ve created some potential cybersecurity concerns. But think, thank goodness, we got Sabine and people like Google looking at that. And Sabine, Could you could you talk to this a little bit about first, could you speak to like how John talked about how bedrock does it could you speak how Google does it to ensure that, that you guys are getting the best technology that you guys are looking for?

Sabine Lam 28:23
So you know, we start with the business case, just like you mentioned, there’s so much we want to do without knowing all this data into a better place and are clearly we can’t do it with an account. We’re seeing more and more places, this will be IP enabled, you got to apply it, principles to it. So we’re looking at open source, we’re looking for flexibility and adaptability of the building and infrastructure we are adopting is what we call the horizontal infrastructure infrastructure. We want you to know what is in our building, we want to manage the IP devices in our building. And we want to control them by connecting connecting them to our software defined network. And so why while we’re applying all those IP principle, and so there’s the concept of those very verifying the the security aspects of the device. So where we can we will install IP devices again, there’s a lot of opportunities with that already mentioned the you know the throughput, but if you have the device on the network, we can quickly send the data to cloud and they get very optimal platform for for small building applications. We can manage that fleet we can we can see it we can manage it. And when I say you know fleet management is around firmware and configuration updates, configuration changes. You know that that like reacting normally or pulling data into cloud and analyzing it and Understanding how to better optimize the body, but we want to turn around and control the system right back. And then we can do that very good Mr. Fraser IP device, the troubleshooting the asset management, you know, if you just think of more reasons, not for J, vulnerability will not be fantastic if you just download Firmware Upgrade to all your IP device that you know, could be vulnerable at one at one point, I don’t think any one of us is actually capable to do that for our portfolio. But that is the angle. So full visibility and manageability of your network. I can’t remember what your other question was like how we go about it. But like I said, we look at it principle we’re moving away from OT or operation technology specific protocols, we’re moving towards it protocols like and TGT. Like ipv6, there’s a lot of things we’re looking at that are that are very common in the IT world and verified and proven. And that’s what we’re asking the industry to move towards. So what will be installed in our building our IP devices that are capable of meeting their specific requirements. And I think along the way, we’re kind of sharing our findings with the manufacturer, if we find a device that is not on par with with our security goals, we share that information and they fix their firmware, and they fix different things. So they can meet the requirement. And you know, in Return Value to like I said, each our industry, it’s a costly process. But they say that as far as cost goes, when you start installing those devices, I do not believe that increase the cost, you know, the this, we want to address those costs, it is novel. Like to say, people push back, once they get over a hurdle and understand that the capability and opportunity, and they start embracing all the value you can get out of IP devices, you will realize that the insulation costs decreases, you know, it’s all on an existing network, you don’t have to install another network necessarily, or on a managed network or random network, it is on an existing network. And so it decreases the solution. In the end, it decreases like the the maintenance cost over time. And like I said, the installer and the technician, you know, and the MSI get used to the requirements. And that leads to just kind of flat or decrease costs. Once you start automate configuration. That’s, you know, again, they can also lower the cost of installation and commissioning. So

Scott Cochrane 32:47
no, that was great to be in. Yeah. So follow up question for you. So So when when, when you’re looking at like an IP based controller versus a serial 485 controller, you mentioned and it was an important thing is that the IP controls on a network are visible, manageable, updatable, versus the 485 devices are kind of dumb, they sit under a gateway, they require a special application not accessible on IP. So my question to use have been is which one is the bigger cybersecurity risk?

Sabine Lam 33:23
Well, it depends. So she was anything that’s managed, is a very high risk, right? You will know anything about it, you try to put it under the radar, you think it’s not a problem? If he gives you a check system hacks, maybe won’t touch your ID now. But the tagline is Google HVAC system was hacked to us, you know, it will be a big PR downside. It’s a big problem for employees. There’s a risk, you know, health and productivity risks. So unmanaged is a very scary world. Yeah, we, we just, you just can’t ignore it anymore. You know, the attacks are there, things are happening. It used to be we know that technology is not secure enough. Let’s put it on the side and this you want to talk our IoT network and we’ll protect it. That is not a true it is not at least an idea that we support everything we need for IT network. Now we have multiple layers of security. So it’s sure if it’s on your IT network, there’s risk to your your IT network, obviously, but you know, it’s converged, but it’s segmented. But we’ve never really brought in touch to it. But we have a really strong IT team with a lot of experience I can help us now a try their biggest and best practices earlier and know what might happen and if we know a vulnerability, we can go and fix it right away. If it’s if you know the vulnerability, and we know he’s right. It’s all about work. It has to go and walk into the building and connect to a certain device and have great mobility. We have 10s of 1000s of IP of devices in our building. We cannot do that it’s not scalable. So the only way to scale the only way to manage it is to sit. So you can do it haphazardly. Obviously you do work with people that are strong insights in the world and understand it network and can can protect your ability of building and no data in general. But I think putting your head in the sand and ignoring what was on the unmanaged network is just not, it’s not the way to go anymore. And besides that, if you try to solve some use cases, how can you integrate different systems that aren’t different network? You can’t do that? How can you react and take that data and make decision and then push it that you can push that radius? If you find out that Billy’s too hot or too cold to this, you know, nobody’s in the building or wherever and you want to get your setpoint? How do you do that on an unmanaged network that you can’t connect to? And so, you know, there’s, there’s risk in every area, it’s managed risk versus unmanaged rent, it’s a choice.

Scott Cochrane 35:57
Awesome. Thank you for being that was awesome, awesome. To want to, we’re, we’re gonna do questions at the end, hold on, hold on, save it, we’re gonna do questions at the end. Going open it up yet, we’re almost to there. So. So we’re moving over so so um, you know, looking not just so we, you know, we talked about the design, we’re looking at, like the requirements of the owner and the needs, and really the goals. Let’s talk about it from a contractor point of view for a minute. Okay. We also have Scott coupe with us, Scott, as I mentioned, has been putting in IP controls for the better part, geez, I would say, six, seven years now, this company has been putting in IP based systems. And Scott, could you just share with us some of your key learnings in general to start?

Scott Papay 36:48
Absolutely. And I’d like to piggyback on really what, you know, some of the things that others have brought up already and that is, insecurity is, is first and foremost in our mind as as the systems that we’re installing, and buildings and systems were integrating to are becoming more and more integrated. And we were seeing commonly where we have point of sale systems on the same network as elevators as the same network is parking as Bas, credit cards and everything else. So security becomes extremely important. And so as a contractor, certainly one of the things that we’re really focusing on is security with our customers. However, one of the fun things that we found out about IP based controllers over the years is where as there used to be a significant kind of step increase in the cost of the controller, relative to traditional MSTP controller, that cost is narrowed significantly for most product lines that I’m aware of. But even if that differential in the hardware remains, the savings that can be had on the labor side, the commissioning side of the projects, the installation side, far exceeds that. So we are typically finding situations were just on the software download configuration, and commissioning of those IP based controllers. In some instances, we’re getting 70% labor savings 70% Wow. So you know, controller that used to take us, you know, an hour to program or to get configured and checkout now it’s 15 minutes. Partly that has to do with you alluded to the larger pipe of communication bandwidth. So we can really download configurations across the network much more rapidly to controllers rather than serial devices. But the other thing is that there the onboard diagnostics and the power of those controllers is significantly greater. And so from a technician standpoint, where we are realizing significant labor savings, not just on the commissioning as far as the local controller and loading the controllers, configuring them, but also a network troubleshooting. And how many of us have had video to an old Rs 485 network? And there’s a ground loop on the on the secondary power, and it just pulls your network down and you have no idea where it is right we those traditional troubleshooting tools that are that are really well understood and well adopted in the IT community aren’t possible on a traditional Rs 45 network. So with standard cabling practices, and with standard it, managed IT network tools, control shooting up communication becomes much simpler. And Sabina also also talks about a horizontal network architecture. And we adopted that same philosophy as well. Many of you, I’m sure in the room are put in network controllers, such as the case, which is a piece of hardware, we have more or less almost stocks, buying cases for our buildings, we’re putting in the supervisory software on a flat architecture that is compliant with the best OT or it practices today. And so now we’re all that we need to do is install the software on an industrial kind of hardened PC with no moving parts, and now we have effectively replaced the chasis in the building. So that’s the hardware savings that can be had. The other thing that we’ve noticed is that installing subcontractors for controls, just like across our entire industry and construction, there’s a shortage of skilled labor in all the trades and controls, electrician of controls, installing electrician is, as you know, a specialty that that goes beyond even a traditional division, you know, 16 or 16, it’s been a while since I’ve even used cell phone numbers, that’s a traditional jobsite electrician

as far as the skills required to install the systems. And so what we found is in many instances, for less sophisticated projects that don’t have a lot of pipe and wire for like a chilled water plant or something that we need to stick build on site that we can actually utilize a structured cabling, subcontractor to come and install that system where they’re using pre made cables, you know, cat six cables, and with the ends already on them. So they’ve been tested. And so it’s literally plug and play. And so we found that not only is the skill level required lower, but also the installation time in many instances is faster. And then again, the troubleshooting is simpler, because we’re using the same cables for communications as we are to talk to maybe a wall mounted sensor, a lot of the sensors that are coming out now are utilizing the standard, RJ 45 connector connectors. And so there’s a lot of savings to be had, in reality, even if the upfront cost of the hardware is a little bit more.

Scott Cochrane 42:47
Thanks, Scott. And yeah, and with that, there’s absolutely no question that IP based projects versus 485 projects are certainly more cost effective, they certainly bring more benefits to the owner. And folks, the horse has left the barn NACCHO. And so And with these titles, the other thing I was gonna say these benefits out there, I mean, it’s over the decision making is becoming, you know, exponential towards IP. And Scott, I’m sorry, you’re gonna say,

Scott Papay 43:17
but I was just gonna say it. We are no longer installing traditional Rs 45 serial based communications networks, in in buildings, unless it’s, you know, a retrofit, or in a small addition rather to an existing facility. So anything Greenfield or even brownfield where we’re putting an IP based controllers across across the board, and really doing to do otherwise at this point is really doing your customers a disservice. i It’s the future. It’s frankly, the past at this point. Right, Sabine, I mean, IP based controllers.

Sabine Lam 44:00
I’ll go I’ll go even further. Right. Yeah, you mentioned people push back. That’s true. We have that within Google too. Like the operation team is familiar with one implementation and you try to change that on them that they just question and taken screaming, and they don’t want change. I also push it on the manufacturer, and you know, I’m telling the manufacturer, good cooperation, good relationship, the manufacturer that I trust is also the one that is showing those opportunity to the customer, you know, so not only the building owner needs to push it, but if we don’t know it, then the manufacturer has the responsibility to show where the technology is going and really, you know, encourage the change as well like it comes from different angle, including like us, including the manufacturer and I feel like it’s too easy to just continue the same design like but design is copy pasted from building to building from you and manufacturer happily, you know getting the business because it’s the same design so they Get the same business and that and that. And that, to me is just really a disadvantage for the industry. And if we so if we want to get somewhere, everybody is in it. Everybody has really to share the same message and move forward and help who, you know, help resisting crowd for good reasons. Right. Everybody’s resistant for a very, very good reason, but help help everyone see the light. So yeah, I think there’s a lot I

Scott Papay 45:26
like, I like being brought up. I like this have been brought up manufacturers. And that’s been one of the things that we’ve had to contend with, we’re all dealing with right now is supply chain issues. And again, a flat architecture. I mean, how many of us in the room that install Niagra based products had to deal with? Unexpected long lead times? Or surcharges? Right? We did a lot. And so if you install a flat architecture, you bypass all that you’re buying software, right. And then also just chipsets in general IP based chipsets are much more readily available than some of the more specialty serial based ones because they’re legacy. So I totally agree with everything Sabine just said,

Scott Cochrane 46:14
Well, I do too. I agree with Sabina, especially as we work with many of the large bas manufacturers. And their incompetence inside of the people develop these products in terms of IP is astronomical to us that the people that are building these products often do not have expertise in it. And you need to talk to your manufacturers like Sabine did and be like, No, this is you folks got to get your you know what together if you’re going to be building these controls on IP networks, because you know what they’re doing, they’re leaving us vulnerable, vulnerable as an industry to be hacked to be messed with. Because there’s, it is not that simple. And there is a pure, there’s true competence and and our vendors are manufacturers of these controls, they need to continue to force themselves to get more confident in that area. When they had 485, they could just do it, they could do it however they want it. Nobody could say anything. Well, that’s not the case anymore. That’s not the case. And they they need to understand that our manufacturers do. They’re building us some great products. So folks, I don’t want to I don’t want to give them too much grief. But I just like to see a little bit more in terms of like their thought in terms of cybersecurity and other areas of it that we feel like they’re still weak, so, so be careful.

Greg Fitspatrick 47:35
It’s got to add something from a contractor’s perspective. I know we’ve got a lot of integrators out here. How many of you guys have actually installed an IP project by show of hands? Okay, so we do have a few. I know that just like the design community, or some reluctancy to get into it. Now like to share real brief story about one of our customers, we’ve got a customer who’s runs a small controls group, at a large contractor, large mechanical contract as a small controls group. Three years ago, I did a presentation on an OT network solution that we sell, right? purely based on the technology itself. Three years afterwards, he gives me a call, he goes, Hey, Greg, you remember that solution? That ot network stuff? You talked to me two years ago about? Well, I think I have a project, a large municipal project that had four buildings, huge project had four buildings, and an engineer out of Texas laid the job. I took a look at the drawings, the drawings were about someone guess what percentage you think the drawings were complete 10% 10%. They’re a little more complete than 10%. But coming from a consulting engineering background, I understood the intent, right. But there wasn’t enough information in there for them to really bid the project. And his group told him that he had to split the project up in scope between a large controls entity that they typically use, which was a branch office, and his smaller controls grew. My advice to him was he who controls the data, or the data architect, and controls the network is going to have that customer forever. So I split the job up for him. Did all the drawings, we suggested that we would do the drawings, we will lay out the entire network for free. So you asked why would we do that? Well, a because it was a fiber based solution. We were able to lay out the fiber so they can get a number from a structure cable guy because they couldn’t do it with the existing drawings. Be it helped us to build the hardware in the software because everything was included on the drawings that we provided. See, those drawings would eventually turn into submittal drawings right And D, they took the drawings to a pulse bet interview. Now, if you’re consulting contract, a consulting engineer, and you lay it on a job that you do in your heart was about 50%, complete. And someone showed up with the entire network laid out from top to bottom, all the way down to the sub networks. I’m talking the VPN, the firewall, there was a couple of servers on premise servers, cell modem, the whole nine yards, all this stuff is laid out. Would you choose that contractor that laid it all out? No, that’s because you’re more savvy, right. But the engineer who didn’t put together a complete set of documents would choose that contractor. That job is currently under construction. It’s almost done. And like, Scott pay said, I’ve talked to the I’ve talked to the systems integrator, and they said that, now that they’re about 50, or 60%, into the job, that the turnaround time on installing these IP devices, has probably went and saved them 50% in labor, up to this point. So for us, we can assist you in designing these IP jobs, if you’re afraid to take that leap as a contractor, right? Because there’s a lot of things involved.

Scott Cochrane 51:23
Yep. And we’ll also

Scott Papay 51:24
want to go ahead, Scott, I was also gonna, yeah, just one more thing, I guess his point he wanted to make earlier, and we keep talking about the resistance that we’re running into kind of as an industry for primarily, I think, at the consulting engineering community. And what I would encourage everyone here who is a system integrator, or manufacturer’s rep, is to understand that we, us who are at the ground level, with the systems are the experts, it’s, it’s incumbent upon us to educate the consulting engineering community. Right. And so, we’re in the early stages, if some of you are looking at being early adopters, when it comes to, or rather, looking to do your first IP based project, you know, there’s definitely some some, some pitfalls, right, some things to watch out for. And what I would really encourage you to do is to take, take a traditional job, if the system is not laid out to be IP based, take a traditional MSTP job and work with your manufacturer work with Cochran you know, supply work with experts, you know, that are in the industry and lay out an IP based network and bring that back to the to the engineer and say, This is what we want to do. And there’s no cost impact of the job. We want to bring this customer, we want to make sure that they’re utilizing the latest technology that’s available in the industry. And here’s how we’re going to do it. And I guarantee you, they’re, they’re gonna say thank you. And that’ll be the result.

Greg Fitspatrick 53:04
And they’re gonna ask for your drawings.

Scott Cochrane 53:05
take that plunge, folks. Go ahead. Don’t be scared, right? Just run out and go switch that 485 job over IP right away. So all right, we have time for some audience questions.

Scott Cochrane 53:23
Any questions? Okay, we’ll say, Well, you know, we only got seven minutes.

Question from Audience  53:29
I can wrap it up into Thank you. Thank you. So, yeah, we get it. IP is the way to go. And we’re sitting in a room with controls contractors, and not many people have gone there. I would ask the panel, how you respond to Gartner or NIST. And I, and I want everyone to hear this not as an additional complication for what you’re about to jump into. But actually good news where Gartner analysts say, really using it traditional IT for the OT ain’t gonna work. It’s too complicated. It’s too expensive. It’s too risky. It’s too cyber secure. So I would say to everyone here, are you guys pushing the MEP community to take a look at new technology and not try to bring the legacy technology into our fledgling IP space? Good question.

Scott Cochrane 54:37
And I’ll start so so briefly on. You’re exactly right, Bill. I think Greg or somebody said it, you know, where we’re trying to serve things in in I would say spoonfuls that people can eat and know the MEPs are not ready for the entire entire IT industry to learn, right? So yes, we we see benefit in helping them specify products that are geared in including it based products that are geared towards bas. And there were as a distributor, we have multiple products, including bills, the Toasty box. is a fantastic product, by the way for everybody. Thank you, Bill for that. No, we do enjoy it. Never Never send a tech to a job site without one just saying, if you ever if you want a remote community member send them without a tosibox. But but we we certainly find that pushing products that are geared towards our industry, from an IT perspective, have made it a lot easier for our industry to accept this new change. I would like to see anybody else want to answer that question?

Sabine Lam 55:56
I think you need to start with a standard, right? What does that mean to be secure from this network? What other requirements? So that’s how you help the manufacturer? That’s how you help the MVP? Like, what even? What does it mean to be secure how you go about it. And you have alliances are out there that are getting together? There’s the rec symposium we do, you know, there’s multiple organizations that are spending time around that, let’s make sure we’re aligned on what we’re asking the manufacturer, because if each of us asked for something different, it’s not gonna be very successful. So kind of getting behind the proper standard of what is the minimum set of requirements that the manufacturer has to meet? And then sharing the pain of how do you go and test it, because, you know, it’s great to do lip service of something, but until it’s verified, I wouldn’t trust any IP device that goes on our network. So we’ll have a qualification lab qualification labs out there, that goes through the list of tests, we get 100 tests per device of things that need to be met before they can go on our network. And that’s one first layer of security. There’s many more layers after that, right. But I think first defining what it is and all lining behind it behind it is the first step.

Scott Cochrane 57:10
Okay, spell. Great question. Next question.

Question from Audience 57:18
But it pertains to this discussion. I’ve just got two words, zero trust, zero trust for whom Paul network devices, often network networked about zero trust. Because if you go by that policy,

Scott Cochrane 57:32
policy and owners rap, and he’s speaking to the cyber risk of all these new IP devices, which is a mess,

Question from Audience 57:41
right. And then I would like to also add, the industry is changing in the IT community as well. As the being mentioned, software defined networks. Now you’re able to create a device profile, specific to a device that allows you to get that integrated into that software. And now your network can stand, you know, goes and scans the device knows it’s a trusted device and allows you to plug into it. Yeah,

Scott Cochrane 58:09
I’m for a cybersecurity and scaling out projects, that will be super handy. So that’s exciting.

Question from Audience 58:14
And it’s starting to become more available. Not at the high dollar guys. But now is I mean, I know you guys use it. So

Scott Cochrane 58:22
cool. Good stuff. One more question.

Question from Audience 58:26
So this whole talk was about encouraging us to go from Rs 485 to IV, which is great. It’s something like No, no one in this room has really done that. Based off the hands on assault earlier. As all you guys have more experience doing it, what are some of the pitfalls we should look out for going from participating firefight?

Scott Cochrane 58:51
Yeah, I mean, I’ll just say that, it, it requires that you educate everybody in the process to make it go right. All the way back from the owner to the consultants step everybody has, you know, because because there’s it is more complex, and there are more things that need to be considered. And if you just go and slam an IP system into a building without understanding what the cyber risk might be of doing that, you could be making a big mistake. Right. So I think it’s really understanding and education is really the biggest pitfall because that’s what it takes to do it. Right. more from my panelists.

Scott Papay 59:30
Scott mean, I could, I could chime in here as well. I mean, you’re asking a really pragmatic question, right, which is how do I go from being a system integrator that’s only done serial comms up till now, to doing IP based controls? It seems like a really big lift, and there is a lot of education and sure that is the answer. And I would just say the best way to do it is OJT. As in fight find it, find Have a candidate project and work with your manufacturer, they’ve already crossed this bridge for you. They, I’m sure if you call up your manufacturers today, instead of that old order where I would have gone and bought the same old stuff you’ve been making for last 1520 years, I want to buy what some of your newer stuff that’s been out for the last four or five years, can you help me lay out a network? Right? You can also work with experts like Bill who’s in the audience, right, and some of his products around tosibox, he’d be happy to help you lay out in that work. Sure. Greg Fitzpatrick, there’s a lot of knowledge in the room right now. And also, income are inherent in your manufacturers that can help you make that leap. It’s, it’s not as scary as it seems. And I would just encourage you all to really take the leap and do it. You might struggle you skin, your knees a little bit on that first project, but you’ll learn a lot along the way, and you just gotta go. Gotta go.

Sabine Lam 1:01:00
I’ll, I’ll add to that is, it also depends on how large of a real estate owner you are. So if it’s, if you have a large real estate owner like Google, it’s important to work with you it division, obviously, and, and just kind of combined effort and really create that team that the understand both it OT and build up that relationship and understand where their concerns are and withstand what the OT world concerns are and work it together. That is not obviously a solution for everybody. So if you know, much smaller company, I imagine can work with solution, provider global supply chain solution, there are some out there, there are experts out there that can install it for you and manage the network for you and provide some guidance at that level, their consultants and others. Yeah, but I totally agree that I wouldn’t go go at it on your own and could be a complete disaster, and you really get to the opposite to what you’re trying to accomplish. So

Scott Cochrane 1:01:58
Trotter, Greg, anything in closing?

Greg Fitspatrick 1:02:01
Just real quick, the project that I mentioned to you guys, before that the integrator did his first IP project, the same speech that Scott pages gave about starting with a smaller project, just taking the project and cutting the teeth. Because remember, the technology is the technology. Right? So once you deploy the technology, whether it’s a big project or a small project, it’s still scalable, right? So I said the same thing to them back. I’m think three years ago, they didn’t do it. And they cut their teeth on project that included four buildings, multi storey the whole nine yards. But, you know, for them, we’re supporting them. Right. Lucky for them were supported enough. But like I said, we can support you on anything from a small to a large project.

John Guardiola 1:02:53
No, and I would echo grayguns beans, comments on the high tea. I mean, we’re so close with it, we collaborate to the point where the built controls team started with engineering is now direct report through it, it is on the network software all outside of tying our other systems.

Scott Cochrane 1:03:15
facilities folks, that bedrock report to a Chief Technology Officer. How cool is that?

John Guardiola 1:03:20
And they still work with myself on the mechanical install and use part of it. So it’s, it’s changed so much in the last couple of years. And we did the same thing with IP We started small, you know, it’s a small one and tested it out with our, you know, our contractor partners, etc. Yeah,

Scott Cochrane 1:03:39
I think that was some good consistent advice. Start small and take the plunge. Well, everybody, thank you, panelists. Thank you. That was awesome.

If you have time, you can watch a video of the session by clicking here