Rick Kaun is VP Solutions at Verve Industrial Protection and has worked in the industrial sector for more than 20 years on a wide variety of IT security projects. Plant Services Editor-in-Chief Thomas Wilk spoke with Rick recently about what plants are doing to safeguard their OT assets in a world where IT and OT infrastructures are increasingly converging.
PS: Before we dive into the questions about the topic, can you tell us a little bit about yourself and your interest in the convergence of these two worlds?
RK: Sure, in 2001, about 20 years ago, I started in the IT department at a company called Matrikon, which many of your listeners and readers may have heard of. Back in the day, it was one of the biggest developers of OPC drivers, and for me, that was the emergence of IT trying to get into OT, bringing visibility to monitoring, process improvements, loop tuning, etc.
I was asked to set up a department that was tasked with safeguarding the infrastructure that we're going into, so when we went to improve our visibility, we weren't decreasing the security or the reliability of the environment. We built a services-based organization that we started to grow. It was shortly after that we started to see some NERP-CIP type of function and traction. Fast forward to today, Matrikon was bought by a company called Honeywell, which, of course, more people have probably heard of than Matrikon. And I liked the opportunity to work within large, global, complex clients, which was very interesting for my own resume and learning. But I found that Honeywell was a little bit difficult given its size, so I joined a vendor-neutral, smaller company again.
The reason I like that is because when I work with a facility, there's more than one label or OEM vendor in that space. If you could bring a holistic view to this environment, you're providing much better services, in my opinion, to the end-user, helping them understand the different nuances of security within an environment regardless of the OEM label.
I now work for Verve Industrial. We were actually originally a systems integrator company. We do a lot of DCS programming, PLC upgrades, a lot of historian work for loop tuning, and some process improvements and monitoring capabilities. Along the way, we built the Verve Security Center platform as well. I like to think that in the last 20 years, after working on a few security projects, I can share some hang-ups and obstacles that some of your fellowship might encounter, and ways to maybe avoid them.
PS: From what we hear from our readers, they're encountering a lot of these issues. You put your finger on one of the problems they're experiencing, which is just the sheer variety of asset classes they're asked to manage and the sheer variety of vendors who manufacture those assets. There's not always time to make sure you can get the same equipment from the same vendor in order to build out a production line. It's a challenge to get all the assets to work together, much less work together securely.
RK: Back to my days at an OEM vendor trying to bid projects, unless security is written in the spec, your vendor is not going to bake it into the product because it costs more. And when they want minimum compliance and maximum margin, you're not putting anything optional in there, so often these things are built without security or with minimal security.
We're getting better. I remember once upon a time I was dispatched to an OSB mill because the vendor decided that a BGP packet, for example, was a great starting point for their new plug-and-play toolset. As soon as it hit a smart 3COM switch, which is what BGP was, nothing got through. So, I had to make both sides as dumb as possible to make that work. We're using new technology, but not necessarily the way it's intended sometimes.
PS: Let me ask you about the responsibilities among plant teams for these kinds of functions. You know, our readers, some are learning to have these conversations, some have more dexterity with these conversations, but I think there's an open question in a lot of ways on how security responsibilities should be divided up between the OT and the IT teams. Our readers often report up through the operations function, maintenance and reliability especially, plus the operators themselves. So they're curious to hear your thoughts, what are these teams on the hook for, for security? What do you see people on the hook for, and what would you say they should let IT take care of, or ask them to take care of?
RK: It's different for every single organization, depending on how mature IT is, and how much collaboration there is between the two departments. But if I stick to a purely philosophical perspective, maybe that might help. You and I talked about the notion that cybersecurity in OT isn't necessarily around honeypots and catching the bad guy and doing the stuff we see in the movies, like tracing back to the evil guys. The reality is, we want to keep safe, reliable, expected operations running as much as possible, and so the ultimate responsibility lies with operations, maintenance, and the entire OT team.
Now, the challenge there is that we need to have this knowledge and awareness of the toolsets on the IT side. Let's face it, you could pick an entire career just on networking, or just on email, or just on databases. There's too much for any one person to know, so I can't find someone with all operational and security knowledge. I will need a team, right?
What Verve advocates is something we call “think global, act local.” What that means is, if you do it right, a small team of a few IT and a few OT professionals can work together to first assess risk. If Blue Keep comes out, for example, we can drop that into our database and see exactly where that lights us up. But then you need the OT context. I can't simply say it's a critical risk on 400 assets. I need the OT site. I need the OT guy or an OT toolset to say, "Okay, here's everywhere that it is. But within these assets, some of these are critical to safe operations."
For example, maybe it's a safety system, maybe it's a lab information management system, maybe it's something that's working on a product spec or safety system. Those are going to have a very different priority, but also a different handling than redundant corporate domain controllers in the DMZ. So right there, I can tell you that same Blue Keep risk has different context for me in an operations role on one versus the other. Where am I going to test first? What am I going to start with? And the only way to come up with smart decisions like that is to have the context, and I challenge my OT readers and followers that are listening to this: inventory is not just IP and Mac addresses; it's everything about that device, plus its operational context.
Carrying on the example: You take that context, and then you work with IT and say, "Okay, what's this Blue Keep all about?" Well, I can't necessarily apply the patch because either the vendor hasn't approved it yet, or they looked at it and they don't approve it. What we need to do in OT, which is the hallmark of anything, is what I call “compensating controls.” If I can apply the patch, what is my plan B, C, or D? If the IT guy that's reading the vulnerability can tell me, "Well, just disable the remote desktop or the guest account in as many places as possible," we as an organization have now gone and said, "Okay, rather than a raw risk indicator, we've taken a notice and we've understood it through this combination of IT and OT, and we're now going to provide compensating controls to most of our systems and full patch where possible."
I've done what I always claim for IT people: I will get the OT team there as close as I can and as fast as we can, but we may take a different path to get there. That's really the only way to have meaningful risk reduction and a sustainable, collaborative environment.
PS: I'm curious to know, Rick, you mentioned that these teams often share the process of assessing the risk in the first place and building out that plan. Are there other areas of overlap that you've noticed where the teams would share responsibilities, such as proving out the ROI on these kinds of investments?
RK: Where I'd like to see more, and I don't see as much: forward-looking teams are when we get into proactive maintenance. Verve had a client where IT came up with a list of system hardening things they'd like to see in all of their systems, and we threw that in group policy and just let it turn on. And, of course, that scares every operator everywhere – small things they're unaware of on the IT side, like a banner login. Well, if you're at a pipeline or a transmission line, an auto-reboot with an auto-login in case it goes offline is a fundamental need, or somebody has to be physically dispatched to this distant thing in a Skidoo or a quad out in the middle of nowhere.
Now, let's go and be proactive. What can we actually do to minimize our attack surface? What can we do to start to write security into our spec? When I upgrade systems, if I'm still getting equipment that requires me to go backwards and be compatible to Windows 7, I'm not doing my company any favors. Now, a big rip-and-replace and heavy lift to completely change your vendor label is not necessarily reasonable either. But with those proactive things, where either IT says, "Oh, we're going to do vulnerability scanning here,” or OT says, "Well, we're going to stick with our current vendor. I need Windows 7 boxes," we're missing the opportunity to cooperate or do multiple forward-looking things, as well as be reactive as risks emerge.
Listen to the entire interview
PS: You know, we scheduled this conversation to talk about IT/OT specifically as a network strategy. In the seven years that I've been with Plant Services, it has gone from being an emerging strategy to something which is being deployed day-to-day. What are some of the things that you can share with our readers on what the reality of IT/OT convergence looks like on a day-to-day level?
RK: Great question. Everybody does it to differing levels of success. The ones that do it right are using this “think global, act local” perspective. For example, one of Verve’s manufacturing clients leveraged our Verve Security Center technology, and the result was they were able to see all of their risks. They found five systems that were dual-homed around the firewall. So, we saw violations of their network architecture. On those five systems, four of them were running TeamViewer. It turns out, they were bypassing the remote access policy and expectation. They did find a lot of things that were not patched for WannaCry, but they also found exploitable firmware on PLC levels.
That particular organization took this information and said, "This is our new standard. We're going to put this into Plant No. 1." And they had 50 or 60 of them in different levels of criticality, and then their results produced a plan. They said, “Let's work together to build what we're doing day-one versus month-one versus year-one and build a standard." That same standard gets rolled out to the next plant and the next plant, and eventually they're building this corporate collaborative environment. It's wonderful. I can't say enough about it.
But the real challenges come from the environments that don't do it well. Verve has another client where the CISO said, "I will do what I need to do, and I have enough work to do on the IT side that we'll get to OT later." There are two things wrong with that. Number one, OT is being left behind. And number two, if I make a decision on the IT side and 18 months later I try to implement it into OT, I'm pretty baked-in with that solution. If it doesn't work on the OT side, I'm either going back to the drawing board, or I'm only doing half the job.
What I would implore your team to do, or your listeners to do is to fight for that seat at the table, not because you're going to take over running whitelisting tools or a SOC, but because you have very specific requirements that can or can't happen when IT makes those decisions. And IT, for the most part, they're not doing it out of malice. They're doing it arguably out of ignorance. I can't tell you the number of times I've sat with a big-four OT specialist and said, "You can't do scan-based vulnerabilities. In OT, you just don't. You knock stuff off. And so, you're going to miss systems, or you're going to miss openings or opportunities.” So, let's create a vulnerability plan that matches corporate's requirement, but let's not put the tool first and hope we can come back later and retrofit it to OT.
To answer your question, we need to have a louder voice. We need to say, "I'm on board. I want to get where you need to, but you must believe me that we have to take a different path or maybe use different toolsets."
PS: We've heard that's a challenge too because a lot of people on the OT side may not have the similar vocabulary that IT would expect out of them. In some ways, its different worlds, different goals, even though the goal of each department is to help make money for the company. They've got different missions to go about doing that. On the OT side, it's asset care, asset operation. On the IT side, make sure the systems are maintained.
RK: The other real challenge is that in IT, the general trend is that for any size or shape of organization, there’s more than one or two people in IT. There's a lot of specialization within IT, so the person who does the vulnerability analysis sends it to somebody else who does vulnerability management or risk reduction in a different department. If you can get your budget to an advocate that goes to the network team, the endpoint team, and the CISO and advocates on your behalf for all operations, they're able to navigate those otherwise silos of information.
I find that's a huge challenge in trying to get things done in OT, again because of that compensating control. I may not be able to patch, but I need to know that I can do other compensating controls, so we know what we have and what actions we’ve taken. In the IT side, this would span multiple divisions. On the OT side, we don't have that luxury. If we could at least dedicate somebody who is responsible for advocating on behalf of operations to navigate amongst the different departments, we'd have a much better success rate.
We're seeing it actually with a client that has a very formal corporate HQ and a full DMZ. They’re bringing every business unit into it, and we have an advocate as the consultant. We are advocating on behalf of the different business units because I've already talked to the network and infrastructure personnel – we did it with business unit one, so I've now learned how to navigate those. If we can leave that knowledge behind, that operating entity will be much more successful in the maintenance portion once the project's over.
PS: That's really interesting. I want to go back to something you said about the way that IT teams have a lot of specialization in them, because in some ways that's not so dissimilar from maintenance and reliability teams. You may have a vibration specialist on hand to monitor the motors, you might have a thermal technician who is in charge of electrical panel maintenance, and yet each department can look monolithic to the others. I'm just bringing it up as a possible point of sympathy between the two teams that, you know what, you may appear to be a monolith, but each side has specializations, and the more one learns about the other, the more they can appreciate, okay, what are the data demands on the reliability side and how might this specialist in IT help out with that problem?
RK: You're absolutely right, and that goes both ways. I'd advocate on the OT side, and I do so because I'm biased. I have seen organizations where IT has designated an OT outreach person. In both cases, it helps to navigate multiple divisions on both sides.
If I come in and say I'm the OT advocate, and we're going to do something around vulnerability scanning or something else, I'm going to need to know my options from IT. Then I’ll explain it to the vibration guys, the operators, etc. because they're all going to have different needs for different systems.
PS: Let me close the conversation then, Rick, with a question on partnerships, and I bring this up, especially because you have been involved with a bunch of different companies and you've seen these situations from various perspectives. Right now you're with Verve, which is an IT/OT partner. When you meet plant teams, how many, at least initially, think they can do it on their own, and don't realize that a partner can help? Versus how many are already down the path and are mature enough in what they understand about the job to say, "Okay, we know we need partners for A, B, C areas?" And who owns those conversations?
RK: To be honest, most of the operators I talk to realize they don't have the staff and maybe don't have the knowledge. But the challenge for them is there's so much noise in this space. There are literally hundreds of millions of dollars in venture capital being put into different marketing strategies and new technologies. I've been in this for 20 years. I've seen the silver bullet come not once, but twice, and in both cases, it's never lived up to its hype.
Many industrial organizations recognize they need to do something, but they're not necessarily sure what. It’s important to get experience on what to do, how fast to do it, and what makes the most sense from the vendor community, other peers, and industry trade shows like ICSJWG and Public Safety Canada for North America, which are two federally-run, information-sharing trade shows. Since they're federally run, you can't buy your way onto the podium, and you must have a good topic. And if you're a vendor, you must pay to stand at a booth. These are great places to educate yourself on getting started with cyber security.
What IT and OT teams everywhere fail to realize is the maintenance portion. I can't just do something and hope it's good. In a former role at a previous company, we conducted a walk-down on inventory. Years later, here I am at Verve, and this past client still needs help with inventory because nobody touched it for four years, and it's off by 592% because of the explosion in IoT.
Nobody understands the maintenance. In my 20 years of experience, I've been everywhere from Saudi Aramco headquarters, to mom-and-pop, pulp and paper manufacturers in the middle of nowhere. I've seen what does and doesn't work. You don't have to buy my solution, but you should at least ask me questions if you're doing it for the first time. I've seen it 300 times. I'm not trying to sell you; I'm trying to help you. And if you do use my software, great, and if not, at least we've made the world a better place.
If I've spent 25 years drinking beer and eating wings, I'm not going to get fit by next weekend just by jumping on a treadmill three times. There's a little bit of work to be done, but we don't need to be afraid of it. We can make it manageable and sustainable. There are lots of good opportunities and ideas out there. We just need to be unafraid of the divide between IT and OT. I think there's a very productive future, but to bring it back to the theme, IT and OT both have to be at the table, and OT has to say, "Here's what operationally or functionally needs to happen," and IT needs to say, "Well, here are your options within those parameters." That's the only way for us to succeed.