With the move to digitally transform and modernize manufacturing operations comes increased interconnectivity between business and industrial control systems. Many of today’s process control systems run on continuously evolving Microsoft Windows operating systems. These open systems bring many benefits, such as direct connectivity to process control network nodes and advanced data sharing; they also open the door to the potential for cyberattacks and safety risks.
Protecting against cyberattacks is critical to reducing safety risks, guarding intellectual property, and protecting profits. Data breaches resulting from cyberattacks can be devastating. Critical updates, reboots, and patches are required to ensure the highest level of security and reliability. For systems that haven’t been updated in several years, manufacturers will most likely need to migrate their aging infrastructure as system software support, security updates and software fixes for older releases become obsolete.
To keep pace with all, a planned and well-disciplined patch management approach helps to maintain and ensure a safe, secure, and highly productive and efficient manufacturing environment. Without proper implementation and thorough execution, however, manufacturing facilities are susceptible to cybersecurity threats and vulnerability attacks, either intentional or unintentional, and could potentially incur a large amount of unplanned downtime.
With this scary threat lurking in cyberspace, why aren’t patch management best practices higher on a manufacturer’s priority list?
Depending on facility requirements, patch management is a necessary but somewhat daunting and unpredictable process that if not handled properly can quickly impact the entire enterprise. For many facility personnel, frustration sets in when control system software updates or Windows operating system patches and reboots cause untimely system delays or shutdowns. In many cases, these delays are unnecessary and can be prevented with proper patch management planning and the right team of experts.
Patching across IT/OT lines
One of the greatest challenges in patch management is the separate yet often blurred boundary lines between information technology (IT) and operational technology (OT). As these IT/OT lines converge, greater synergy and collaboration are needed, especially as cyber threats now target the whole enterprise.
As most production facilities typically run 24/7/365, a planned – or sometimes unplanned – interruption in system availability brings up some valid concerns regarding timing on updates, patches, and reboots. To alleviate these concerns, key personnel from both IT and OT must work together and consider several factors in their patch management procedures, and that includes being prepared for the following situations:
How often and when are patching updates applied? When to patch varies and depends greatly on a facility’s specific requirements. A general rule of thumb is to apply updates on a regular basis, preferably monthly, although that is not always feasible depending on the scope of a facility’s patching needs and whether the Windows updates have been OEM-tested, approved, and deemed safe to use.
What day are process servers and stations updated with respect to new patches for enhanced security and protection? Typically, these come out weekly from Microsoft. Many manufacturers use a customized Windows Server Update Service (WSUS) deployment for periodic patching.
When will the somewhat uncomfortable Windows operating system reboot take place? Ideally, these reboots should be scheduled at a time when they cause the least disruption to production or, if required, during a partial or full shutdown with proper system backup measures in place.
How long will the Windows reboot take? Some Microsoft patches require more than ‘just a reboot’ period. Typically, a normal reboot of a Windows server takes 2-3 minutes (in a virtual environment) and is ready for service thereafter. Other Microsoft patches may require about 15-30 minutes of applied time before fully returning to service. If systems have never been patched, then an initial storm of updates may need to be applied to bring the server into target compliance. Also, if the Windows server is grossly behind in patches, then returning to service can take an indeterminate amount of time but typically requires a phased approach over several days to apply patches and reboot.
Will the server even come back from a reboot? It is good practice to create a snapshot of the server prior to applying updates as patches that are qualified for application can sometimes “break” operating systems and can cause a server to experience the dreaded blue screen of death. In the event of a catastrophic server failure, this approach helps roll back to the machine’s prior state before the patch was applied. While this failure issue still looms and requires resolution, at a very minimum, the server/workstation can return to service and a plan can be put together to address the root cause of why the operating system crashed after the update was applied and the server/workstation rebooted.
OEM-applied patches
Unlike a business or personal home computer, which in most cases are patched as needed, process control system patches must be OEM-tested and approved. Typically, an OEM allows a limited number of patches and posts them on their site for use on their systems.
With these patch limitations, facility personnel must identify which patches the OEM posts are recommended for their systems and then properly install them. When installing the necessary Windows updates and patches, a control system may handle them flawlessly or sometimes they may not, which can cause a shut down for potentially long periods of time and costing millions of dollars in lost production. By only applying patches that are OEM pre-approved, tested, and vetted, the potential for undesirable outcomes is greatly reduced.
Patching and beyond
Patch management has many facets and depending on a company’s requirements can run the gamut from simple updates to major installation updates and modifications. Whether manufacturers need to implement Windows updates and patches, install virus protection, or tackle a large-scale system software migration, a team of dedicated expert resources is required, especially where resource bandwidth is an issue.
In these instances, choose a third-party partner who has a clear understanding of patch management policies and procedures and works on multiple vendor platforms. The ideal partner is someone who can help provide expert guidance on implementation and execution so that systems continue to operate seamlessly and are protected from security vulnerabilities. A team of experts with the right mix of process control IT skills with both IT and OT knowledge can prove invaluable to help:
- identify specific challenges related to patching services, software migration and virus protection
- apply security patches to production systems remotely and onsite
- troubleshoot issues surrounding patch deployment, application, and verification
- alleviate the burden of testing
- leverage a simulated test environment to ensure systems will open and run properly after updates are applied
- maintain system stability and performance
- help identify and install recommended OEM patches
- develop a support plan to stay current and maintain continuity
- perform cybersecurity assessments regularly
- perform patch management across the enterprise for control system servers and stations, as well as the whole process computer network
- help migrate control system software and Windows platforms to current releases with no interruptions to production processes
- help with any process controller firmware updates
- provide additional security measures and virus protection.
A quality patch management program is too valuable to ignore. The benefits in safeguarding intellectual property and reducing risk far outweighs the alternative – the potential for millions or even billions of dollars in lost production. With a regimented patch management approach and the right IT/OT team in place, manufacturers can keep cyberattacks at bay and remain secure in the knowledge that their operations will stay up and running with little to no interruption to production.
This article is part of our monthly Tactics and Practices column. Read more Tactics and Practices.