Seth Carpenter is a cybersecurity technologist at Honeywell Process Solutions and an expert in the area of secure networks and systems. In January, Plant Services chief editor Thomas Wilk had the chance to talk with Carpenter about the state of plant cybersecurity in 2018. The conversation took place days after the Meltdown and Spectre chip vulnerabilities became widely known and publicized, so the conversation began by addressing the two elephants in the digital waiting room.
PS: Given the Meltdown and Spectre news, the conversation about industrial cybersecurity has taken on a new urgency. What are some of the things that our readers in maintenance, reliability, and operations ought to know about how these threats might affect what they are doing?
SC: The big story here, just like any major breaking malware, is that as much as possible, make sure that you’re up to date, that you’re running supported operating systems, and look to announcements from different vendors talking about compatibility or patches or anything like that coming out. The people who are already on top of things and have their patching process in place, they are going to be in fairly good shape here. Vendors are working everywhere to make sure they’ve got patches available.
Likewise, having good security defenses in place is always going to help. “Defense in Depth” is something that’s really ingrained in industry, as we’re used to the days when these systems were completely disconnected from the rest of the world. Most of the time the threat is from another connected system, or maybe a malicious ad, or something like that, that may not be very applicable in an industrial control system. But still, making sure those firewalls are in place, preventing direct internet connectivity from your processing control environment – good security hygiene, if you will – all those good practices are going to help out for this, just like they will with any major malware outbreak. And if there is a group that hasn’t been as diligent in their security practices, this is another good wake-up call, and we just keep getting one after the other.
PS: What are some of the questions or conversations you’d encourage maintenance, reliability, and operations teams to have with IT? Are there any specific questions they might want to ask about the Spectre and Meltdown vulnerabilities?
SC: Some of those questions to ask include, “What’s applicable to our systems?” and “Can we apply the patches?” It’s a little more complicated in this case than just a simple Windows patch. Right now there are patches coming out specifically targeted at Meltdown, so obviously that’s going to be very timely targeted. In some cases, depending on what antivirus they are running, that may need to be patched first to then allow the Windows update to go through. So there’s a very strategic conversation about what needs to happen for our systems, what systems are vulnerable, and understanding what needs to happen there.
There’s a broader conversation as well. How are we reacting to things, and what are we doing systemically? We tend of think of people, processes, and technology when we are talking about cybersecurity. The people aspect is critical. It’s encouraging that you’re talking about the maintenance-reliability group working closely with IT. Traditionally, there’s been a bit of a divide between the IT teams and the OT teams.
A really positive trend that we’ve been seeing is those teams working more closely together and understanding how they can each bring their expertise to these kinds of problems and help solve them, and having the people working together and then having the processes in place to know: “Here’s where we get our qualified patches. Here’s how we distribute them. Here’s how we verify that they’ve actually been put out to the machines.” You know, just closing the loop on all those things.
There are obviously technology solutions that can help out, and it’s going to be dependent on each team. Maybe they are working with their IT teams to use a centralized patching solution, or rely on managed services teams to get patches installed on the system.
Honeywell does have some additional tools, such as the Industrial Cybersecurity Risk Manager, that will assess the systems, the machines within the system, make sure they are up-to-date. So there are a number of solutions there we can look at.
PS: Do you see the Spectre and Meltdown vulnerabilities as more of an anomaly when it comes to the flaws in the processors themselves? Or do you see this as probably the way the future is going to go, where this is a lot more common than say a Stuxnet type of attack? Or is it going to be a balance?
SC: It’s hard to predict. You never know where the next vulnerabilities will come from. I suspect there will be a lot of people taking a really close look at processor microarchitecture, seeing if there are other vulnerabilities. There have been attacks in the past based on similar exploits, but they generally were very hard to exploit and more theoretical. This has been one where it had a very clear exploitation path. And, yeah, this was a bit of a wakeup call in that area. But will this be the same vector for the next big vulnerability? I can’t necessarily say.
Another kind of encouraging thing from a cybersecurity perspective is that this is something that does have visibility much higher up the food chain, even at the C-Suite. It’s more common, still not as common as we’d like to see, but there are often dedicated cybersecurity officers within the company.
They are calling down saying, “Hey, are we vulnerable? Are we patched? What are we doing?” And putting funding and support behind these efforts to make sure that they are maintaining cybersecurity in those systems. Having that executive support is really critical, and then having an understanding of state of the whole system.
PS: When you see research on what’s happening in industry regarding cybersecurity, do you think that plants are where they should be along this path?
SC: We see a really wide range. There are some customers that have very sophisticated practices. They’ve got dedicated departments; they’ve got incident response teams. They are the ones coming to us saying, “Hey, are you considering XYZ technology? We’d really like to see if that has an application in our systems.”
Then you get to the other extreme where we’ve got customers that may still be running completely unsupported operating systems and not running firewalls or don’t have access controls in place. So it’s hard to say where the industry is as a whole, because it’s such a wide spectrum.
Security is always going to be a bit of a trade-off. You could spend all day, every day just making sure you lock down these systems as tight as possible. But at the end of the day, they are a business. They are trying to produce a product. So what’s the right balance of tightening things down and having security in place versus safety and availability and meeting your production quotas? Each customer needs to assess where they are and understand where they want to be.
PS: Let me throw something into the mix with those considerations. One of the issues that comes up is the relative value of the data that operational teams are collecting and analyzing in comparison to, say, the business data, the CRM data, things that might be more on the IT side. What’s your take on the relative value of this data? Because the value of vibration signatures from a motor is probably going to be a lot different than sales data from the past three years, for example.
SC: Well, we are going into a bit of a brave new world right now. The industrial internet of things and connected plants and all this are new and exciting, and we’re finding that data has more and more value. We can build analytical models around it and maybe do predictions for failures that might happen with equipment or a lot of things of this nature. At the same time, just recognizing that this is more data, it often needs to leave the site, so this has also been a really natural place for customers to take a look at cybersecurity in that context. You know, even sometimes data that you think, “Well, what’s somebody going to do with that?” It’s kind of amazing what you can infer from a really rich set of data.
And likewise, where we are looking at cybersecurity, there’s some really sensitive data that might be collected by some of these systems. Understanding the patch levels, and what antivirus software is running and so forth. Even though it may not be sensitive financial data, like a sales spreadsheet, or something like that, there is really critical information about the plants that can relate to their safety and security.
That’s the other kind of big trend that we are seeing. As customers are asking, “What do we do with this data?” and “What do we want to get out of it?” and “Do we start looking at IIoT?” they are also having this conversation, “How do we make sure security is baked into this process?” and “How do we make sure this data is only going places where it’s well controlled and we know what’s moving where?”
PS: In our research on predictive maintenance technologies, we always ask, “What’s your comfort level with sharing this data, from third-party service providers to OEMs?” Our readers have reported back to us that they are still pretty uncomfortable with the notion of sharing data outside the plant. What’s your take on how people can make a case to share data securely to get benefits from external analytics teams that might be brought on for a temporary period, or sharing with OEMs who can also help analyze that data?
SC: With the wide range in maturity in cybersecurity programs out there, we also see a really wide range of comfort in sharing that data outside of the confines of the plant. Some customers think, “Yeah, let’s get it in there; let’s get it somewhere so we can analyze it.” We see a lot of interest in customers getting information out of the OT environment and into something like a corporate SIEM (Security Information and Event Management system) for doing analysis along with all of the other corporate IT systems, so we are seeing a lot more interest in feeding things in there so they can do that analysis at a corporate level.
And then we also see customers who are saying, “Yeah, I want visibility into what’s happening in these devices at my plant, but I don’t want that to ever leave the plant.” So there is also a desire for technologies that can run on-premise. Then there is probably some between those two ends. So, there’s a different degree of comfort within customers, and our job at is to make sure that we’ve got some solutions that meet those particular needs and interests.
As a general trend, I think we are seeing people be a bit more comfortable with that data moving onto systems doing analysis, as long as they see some value created. Nobody wants to just export their process data, their system data without ever seeing a return on that. But in the case of preventative maintenance, you start looking at, well, what would be the cost of downtime if that broke unexpectedly. If that helps you repair and maintain a critical piece of equipment and avoid a failure, we are talking some real savings by avoiding those.
Sometimes it’s a matter, too, of getting that data in front of experts that you may not have available. Everyone is trying to do more with less, right? We might have less budget, or fewer people to do the work. Maybe you have some guy that just knew this system in and out, and he’s getting close to retirement. And so, what are you going to do to make sure that we’re keeping those assets up and running?
Some of these IIoT initiatives are also allowing plants to bring in an external expert that can get a look at that asset data or build a model around it. But again, teams have to understand what the expected return on investment is, make sure that that communication pathway is as secure as it can be, and make sure that they are aware of everything that is happening within the system.
PS: What thoughts do you want to close on?
SC: I think that, in general, we see new conversations happening; we see the awareness there. These are very different conversations than from 10 years ago, where we’d talk about security and they’d say, “Well, yeah, that would be nice if we had it.” But no one was willing to really invest the time and money and training to make that happen. It’s been really encouraging to see customers now coming to us and saying, “Hey, what can we do in this area? We are looking at that.
”At the same time, we know there are some systems out there that are not even hitting the real security basics. Every time we see a new malware scare come in, we know there will be more in the future. Hopefully that’s a chance for teams to stop and assess and look at their readiness. And, you know, sometimes the hardest part is just taking that first step, and Honeywell has a really great team of consulting cybersecurity experts that can come help put that plan together.
I think the most important thing is, when vulnerabilities like Meltdown and Spectre happen, maybe this can be the driving point to say, “Hey, I know we’ve got some places that we can improve, so let’s take this as an opportunity to do just that.”
I’m overall, in general, a really optimistic person. I see things moving in a great direction. At the same time, we know there are problems, we know there are new attacks. There’s scary things out there as the world becomes increasingly connected. So, to me it’s really exciting to be in a place where we are trying to make sure that we’re getting those business and reliability and uptime benefits, and at the same time making sure that we can protect the security of the systems as best we can. It’s a really exciting time to be working in this field.