As much as industry emphasizes informational technology (IT) and operational technology (OT) convergence, a more connected world presents more cybersecurity challenges. Cybercriminals are aware of these potential vulnerabilities as the sweeping digitization of plants and factories continues, and we’ve seen an increase in cyberattacks on industrial facilities.
The 2024 ARC Industry Forum hosted several cybersecurity case studies and panels, with many end users sharing their experiences, practices and lessons learned. Two sessions on Tues., Feb. 6 focused largely on the people side of cybersecurity. IT and operations do not always see eye-to-eye or speak the same language. As one speaker at the forum put it: they drink at different bars.
However, enabling cybersecurity and protecting industrial assets requires collaboration and a cross-functional team between IT and OT. Both need to understand the strengths and skillsets of each other in order to meet IT challenges without interrupting operations.
Bridging the IT/OT gap
Richard Eckhart, industrial cyber strategy advisor, and Blake Gilson, industrial cyber expertise manager at ExxonMobil, joined the forum to discuss their cybersecurity investigation journey across the company’s many physical sites. Industrial cybersecurity defense increasingly requires technology to provide protection and a continual focus on practices and processes. The key to linking those together is well-trained people and good communication between IT and OT teams.
As cybersecurity challenges grow, so do the skill sets needed to meet those challenges on-site. Part of developing a cybersecurity program for some organizations is growing and nurturing those needed skillsets and talents. “We want to be able to develop skill sets that are relevant across both [IT and OT], while appreciating that there's going to be some differentiation of skills and experience between both organizations,” Eckhart says. “So we want to harness all that talent and experience and really leverage that from an OT cyber perspective. From a process perspective, we want to harmonize our processes where it makes sense.”
Oftentimes, both sides speak a different language professionally and their playbooks for investigations and attack response can be very different. It helps the more IT teams understand OT and vice versa. “We built the talent of the OT team to have those skill sets, so they can appreciate and make those right discussions as they execute investigation from an OT perspective,” Eckhart said.
Gilson described how the team improved its cybersecurity risk management strategies by first focusing on their detection capabilities. Only a cross-disciplinary team could give them the right visibility into their operations.
“Starting in this journey, we discovered a lot of initial feedback wasn't adversarial action related but system anomalies. Items that produced more operational liability questions or configuration mistakes or missed hardening opportunities with a system itself,” Gilson said.
With many sites and unique environments, each site needed to develop the OT cybersecurity knowledge needed in that particular environment, but the knowledge can quickly become cumulative. At a large company like ExxonMobil with many sites and thousands of assets, the knowledge gained at one site can be applied to others with the right team organization and communication in place.
“Now you have people that understand your environment well enough that they may not know the expert internal details of that particular substation, at that particular refinery, but we've seen that enough in another area that we're better positioned to troubleshoot and perform investigation functions in those environments we have yet to discover. So you're growing and developing this workforce by combining certain capabilities as you progress,” Gilson said.
Building a culture of cyber-safety
Sarah Myers, operations enablement manager at ExxonMobil, joined a cybersecurity case study session later in the day, and her focus was largely on building a cybersecurity culture. Much the way industry has made safety a part of everyone’s job, the same needs to be done for cybersecurity, she argued.
“Cybersecurity needs to be treated at the frontline of the last person, the last row, just like safety is. We need to train our people in drills and emergency response, and they're really the most important part of what you can do to ensure that you have good cybersecurity processes at your site,” Myers said.
Nearly all cybersecurity breaches involve some kind of human error, Myers said. However, people can also be part of a strong defense with the right training. “What we want to do with our people, is move them from compliance, to be told what to do, to a committed compliance,” Myers said. “And they truly believe that cybersecurity is just as important as the rest of their jobs. They understand, in their role, why cybersecurity is important. They talk to their colleagues about it.”
It can be difficult to measure cybersecurity culture and progress being made. “How do you count how many cyberattacks didn't happen because people follow the right procedure?” Myers asked. Regular training ensures that culture is taught and nurtured on a regular basis.
“People are the most important cyber barrier. That can be your biggest weakness, but also your biggest strength,” Myers said. ExxonMobil provides many different levels and aspects of cybersecurity training based on position and experience. It includes computer-based and classroom training for all employees annually, including management.
Michael Elhers, global senior security manager at Olin, who joined the cybersecurity session earlier in the day, highlighted cybersecurity best practices at his company, but the conversation continued to circle back to people and communication. In addition to building a cybersecurity culture and training, the IT security mission needs support, especially from OT upper management, Ehlers said.
“You need that change advocate. I cannot stress how important it is to have a change advocate,” Ehlers said. “You generally need someone up in management, sometimes the vice president on the operation side, and you certainly need someone that understands the operations.”
The change advocate can be critical to building those bridges between IT and OT. “If you're on the IT side, you have that responsibility for cybersecurity in your organization, and you have an OT footprint and you're responsible for security. You have got to have a change advocate that you can partner with,” Ehlers said.
Source of truth: the importance of an asset inventory
Outside of training employees on cybersecurity practices and establishing a strong cybersecurity culture, developing an accurate asset inventory is an essential first step for an operation’s cybersecurity plan.
“If you don't know what is on your OT network, how can we even start to do security?” Ehlers asked. “What is your source of truth for servers on your network?”
The same is true for all devices. Completing an asset inventory and managing vulnerabilities across the entire organization can be an overwhelming exercise at the start, but prioritization is key for accessing vulnerabilities.
“We have to be able to figure out how to prioritize them in a systematic fashion to allocate your scarce resources appropriately,” Gilson said. He also suggests that instead of focusing on vulnerabilities alone, reorient the question around attack vectors instead. “That is going to be that point of additional compromise that could be a cascading consequence to other islands, just like we can war game scenarios or threat vectors getting into our environment, understanding those pathways and defending those are in fact the key perimeters to doing it,” Gilson added.
While the importance of inventory in the OT and the process control space can't be understated, Gilson said, it shouldn’t be something that slows other cybersecurity objectives. It’s something to work toward at the start, but other opportunities or risk reduction measures can be done with a partial understanding of asset inventory.
Device source of truth is something that’s a continual process at his facility, Ehlers said, and all those connections between network and devices are investigated on a monthly basis.
Michael Elliott, senior manager of global OT security at Kenvue, a division of Johnson & Johnson, also made the argument for including network connections along with the asset inventory. “We need to understand the assets of course, no argument there, but the other thing that is really critical to know about assets is what are they talking to, how are they networked?” Elliott asked. “What are you organizationally doing about getting together a good accurate network diagram?”
IT challenges: collaboration, not convergence
The more IT can understand about the OT environment through training and cross-functional team interactions, the better it can protect. But the necessary firewalls and protections can make it hard for IT to get what it needs from the OT environment. OT might not want others in their environment, and rightfully so, but it requires OT to push any data that IT needs from OT networks out to the business network. “I think the biggest challenge that we ran into is getting access to the data in the OT environment,” Eckhart said.
The business network can initiate into the OT network, but not without additional verifications and protections in place. There are processes to identify cyber-relevant data from the OT environments, scrape it out and push it to a common platform, where both sides have access to it, but still requires careful coordination between IT and OT.
As much as industry has embraced IT/OT convergence, the expression doesn’t work quite as well for cybersecurity. “The word convergence can have a negative impact in the conversation, so it's really about collaboration,” Eckhart added. “From an OT perspective, their mission is reliability and safety, continuity of operations to generate profitability. They do not want to compromise that mission, and we need to respect that. And so when we bring in folks to the table, they have to come in with the perspective that everything that they may want to do to enhance cybersecurity for an OT environment, it may not be feasible,” Eckhart said.
These conversations can be difficult, he said, but with the right people in the room with the right cross-functional experts and open minds, the discussions can lead to progress, as it did with ExxonMobil.
“Our journey from an ExxonMobil perspective has been many, many years, and we still have a long way to go. That journey will never end. But we've found our success in bringing the right disciplines to the table,” Eckhart said.
Case study: malware attack
During the second cybersecurity session on Feb. 6, Stuart McCoy, OT data and connectivity engineer at WestRock, a packaging manufacturer, discussed his experience with a malware attack in January 2021. The organization experienced widespread production outages across their fleet over a long period of time, which McCoy’s wife refers to as the “black winter.”
“Until you're sitting in that seat, it is very hard to manage a situation. So, while I don't pretend to speak for everyone, I can give you a little insight into what happens during one of these events, and it is quite challenging to say the least,” McCoy said. “You go through your initial assessment. What's there? Where's the dam at? How long has it been there? How fast is this thing moving around? How's it getting around? How do we stop this without stopping manufacturing.”
Having a disconnect playbook can be critical in a malware incident. McCoy defined this playbook as a “set of processes to follow that will allow you to quickly and safely disconnect your process control local networks from your business network that you're using in your IT networks, but allow the PCs to function normally.” Sites that have a disconnect playbook know how to quickly and properly disconnect, but sites without them are left to scramble when time is most precious. “A standard disconnect playbook was created for every site. This is something that is reviewed and updated annually,” McCoy said.
McCoy also returned to a common theme throughout the two cybersecurity sessions. “Communication is key during this period. It's paramount,” he said.
“The way we look at our manufacturing process has changed. The way we look at our data has changed, and we are obligated to provide that data securely to folks making data-driven decisions,” McCoy said.
The incident also helped the organization make the leap to the cloud. “With the proper architecture and design, you can sleep at night, knowing that you've got manufacturing data populating your cloud,” McCoy said.