Don't "set it and forget it" when it comes to your automation systems

Do: Get off the ground on the right foot

Just because you have a specification for how a control system should be built or a software platform should be customized to address your teams’ priorities doesn’t necessarily mean that’s how it will get built. And in the critical commissioning and startup stage, it’s vital to identify and resolve any discrepancies as quickly as possible.

Running standardized site acceptance testing is a crucial first step to ensure that time and money aren’t wasted addressing connectivity or functionality issues once a new control system or software platform is up and running, says Nate Kay, P.E., project manager with Martin CSI, a member of the Control System Integrators Association. “Sometimes (a system) doesn’t get installed exactly the way it was designed maybe on the process side,” he says. “If there were any changes made on the fly or issues that occurred during the installation … we work with the contractors and the customer to determine where the problem is.”

In addition, it’s highly valuable to incorporate automation systems into your overall maintenance strategy, notes Richard Gaines, director of the Integrated Smart Operations Center (iSOC) at the New York Power Authority. iSOC, a central monitoring and diagnostics center in White Plains, NY, relies on predictive analytics software and smart technologies deployed across NYPA’s 16 power plants, more than 1,400 circuit miles of transmission lines, and substations statewide to detect operational anomalies and help prevent asset failures and power interruptions. A fully digitized, integrated approach to physical asset management means ensuring that the maintenance of data-collection tools isn’t an afterthought.

“When sensors are installed, we have to get them into the standard PM program that we use for all instrumentation,” says Gaines, a 36-year veteran of the NYPA. “Whether we’re doing gas analysis on a transformer or we’re looking at pressures, temperatures, and flows on a pump or turbine, our philosophy is that monitoring and diagnostic instrumentation shouldn’t be any different from the other instrumentation we use in process control. Even if it’s just (for) monitoring and diagnostics, it’s still just as important as (other) instrumentation for process control. So it gets rolled into our maintenance management system.” 

Do: Verify your data

Once the data starts streaming in from your new, automated condition-monitoring technologies, how can you know – with certainty, and over time – that the data really is valid? After all, for users of these connected technologies, condition and asset performance data underpin decisions about whether, how, and when to service equipment and adjust operating parameters to make processes run more efficiently. “Data quality is key,” says John Rosenberger, director of iWAREHOUSE GATEWAY and global telematics for The Raymond Corp., “because the collection systems could be sending data (to the cloud), but if they’re sending junk” – inaccurate or incomplete data – “now I’m throwing off a lot of those predictors that I have thresholds and controls set for.”

Periodic data validation, which can be performed by an independent third party (such as an outside laboratory) or by an OEM as part of a service agreement, is essential for ensuring data integrity. At iSOC, as part of a transformer monitoring system, oil gets sampled from transformers every six months and sent to a lab to ensure that values from the samples align with values taken from the transformers’ online sensors.

Once the third-party lab completes its analysis of an oil sample, the NYPA downloads it and runs it through an application that does a comparison between the lab sample and the instrument sample, Gaines says. “Our monitoring and diagnostics engineers will then determine whether there’s concurrence or a discrepancy between the two samples,” he says.

A focused effort on process instrumentation monitoring and maintenance, as well as having some redundancy in instrumentation, has proved useful in alerting the iSOC team when there might be a problem with one of its automation technologies, Gaines notes. “Often what we see is it’s not necessarily the machine itself that has a health issue; it might be the instrument on the machine,” he says. “When we’re profiling data for instance coming off a pressure transmitter, and we see that the trend line is flat, it’s not changing, we know that the instrument failed in place.”

The effort to validate data and quickly identify deviations from expected values is supported by improved communication and collaboration across traditionally siloed departments within an organization. One of the benefits of today’s network-connected automation technologies is that they’re narrowing the divides between IT and OT teams and between the office and the plant floor. “A lot of systems in the past have been stand-alone; they existed isolated on their own network so you relied on maintenance personnel to follow a regular preventive maintenance schedule for (them), but what we’re seeing more of is networking that ties into the whole industrial internet of things side, where instead of having this isolated system inside a box where only a few maintenance people interface with it, we’re starting to tie that in to the rest of the plant and also bridge the IT-OT (gap),” Martin CSI’s Kay says.

He offers an example: At one plant with which Martin CSI has worked, “the business side maintains the inventory, what they’ve ordered, where it goes; the maintenance side, they see the levels in real time on the screens,” Kay describes. “And we’ve started to work to merge those two together so that you have that data from the plant floor and the business side can compare it with their orders, their information, and you can detect problems that way. So, for example, if the inventory is way off and doesn’t coincide with what the plant floor sees…we might generate a preventive maintenance task so the maintenance department will look at it and see, is a level probe off, are the operators bypassing the safety procedure to get more out when they shouldn’t, that type of thing.”

{pb}

From a cybersecurity perspective, a true collaboration between IT and OT on maintenance of automation technologies – rather than a begrudging acceptance of the other’s mandates – is the smartest move, industry pros agree. “Typically the IT mentality is on security, keeping everything secure, and the control system mentality is pretty much keeping everything up and running, so getting those two sides to understand each other’s priorities more and come to some middle ground is needed,” Kay says. Dan O’Brien, marketing director for Honeywell’s Connected Plant offering, says he sees growing willingness from operations departments to work with IT departments, because as automation technologies and platforms are used to monitor the health of more assets, the two departments have a common enemy: compromised or faulty data, which can have disastrous safety and business consequences.

“I think operations knows they can’t become a secure facility without IT,” O’Brien says. “I see a more cooperative landscape.” An easing of frictions may stem in part from the fact that an organization’s IT and OT departments aren’t in it alone but together, so to speak, when it comes to automation maintenance today: The trend toward subscription-based management of automation platforms by either the platform vendors themselves or system integrators means teams can offload management of technology in which neither may have expertise. “Technology is getting more complicated, and the workforce is not really there to support it,” says Will Worth, director of engineering at CSIA member Quantum Solutions.

Don't: Relegate updates to the back burner

It’s so easy to do, whether it’s your personal smartphone or your desktop at work: You get a notice that an update or a patch is available, and you think: “I’m busy now. Later.” And then you find that you’re never not busy, and that an update is never not an inconvenience, and then...

“It’s very easy, if you’re not patching your system or you’re not getting your upgrades done on a regular basis, that the system becomes obsolete,” says Rob Patterson, product manager for the digital services offering in GE Industrial Solutions. Failure to perform regular updates or urgent patching for one or more operating systems or software platforms can stymie interoperability between systems, render it difficult to receive timely support should a problem arise, and expose a network to immediate and significant cybersecurity risks.

“I see sections of enterprises or even facilities where (the attitude is), ‘Yeah I’ll get to it,’ ” when it comes to updates, The Raymond Corp.’s Rosenberger says. “You don’t want to wait until you get stung.” Part of the fix gets back to improved IT-OT collaboration. “They have to be more cooperative as far as, ‘OK, IT needs to do these updates; I’m going to schedule a gap between my first and second shifts for that,’ ” he says.

Maintenance of automation systems is an investment of resources as much as maintenance of heavy machinery is, and maximizing your return on investment in these automation tools demands a similarly strategic approach, Patterson and Worth suggest. Plants need to “incorporate (service) into their total cost of ownership for a system, because the biggest issue with systems is that it’s real easy for a factory or location to let them collect dust on the keyboards,” Patterson says. “It gets harder for them to continue to get that payback for what they purchased the system for. Day 1 they see a good payback, (but) over the years it starts to dwindle, and it’s typically because the systems aren’t maintained.”

Operating practices of an automation system may change, or users may find that they new ways of clearing data clutter to get to the data that will be most useful for them – it takes a willingness to banish a “set it and forget it” mentality to make sure that automation tools are operating as well as they could. “It’s kind of sad when (plants) have a powerful tool that they don’t make full use of,” Worth says.

Don't: Neglect security for any of your connected assets

One of the biggest problems plants run into when it comes to automation maintenance – from both a security and an enterprise asset management perspective – is that they don’t have an accurate or adequate inventory of all of their network-connected assets in use. “They don’t even know what equipment they have half the time,” says Quantum Solutions’ Worth.

That’s a major issue, because hackers looking for security vulnerabilities will find easy targets in overlooked network connections, and it’s easy for plant supervisors, plant managers, and IT departments to overlook such connected assets as conveyors and powered industrial equipment like forklifts when they’re strategizing about cybersecurity, says The Raymond Corp.’s Rosenberger. “Smart” conveying and material handling equipment, as with bolt-on industrial sensors, “are just like other high-tech electronic assets in your enterprise, and they need to be treated as such,” Rosenberger says.

“If I have the IT director sitting there, I’ll ask…‘How often do you update the firmware or software for your routers?’ ” he continues. “This piece of equipment is now connected to your network that was connected to your router that was compromised. And all of a sudden you start seeing those lights go off – ‘Oh, yeah, you’re right…’ And they start going down that road themselves, (of) ‘Oh, wait a minute, I have all my scan guns, vehicle-mounted computers, etc., that are now communicating via WiFi or cellular networks that are vulnerable to these types of data security issues.” And whatever piece of connected equipment you’re dealing with, as GE Industrial’s Patterson notes, “If you never look to see if you have a virus, you’re not going to know that you have a virus until it’s too late.”

Addressing vulnerabilities starts with getting an up-to-date automation inventory. A system integrator like Quantum can “go out and sniff out everything that you have connected to the network,” Worth says, and build a mostly autonomous inventory from there. “A centralized place to have an inventory of all of your automation assets is very appealing,” he comments. Because regardless of who (in-house or external) is using what to monitor the automation inventory, “without something…keeping an eye on it, keeping it up to date, it just gets carried away and it’s hard to recover,” says Worth.

The overall landscape for maintenance of automation systems is improving. The common thread among those that are “doing it right” when it comes to automation maintenance? As always, communication. “They seem to have a good relationship between management, engineering, and maintenance, where they can communicate with each other, understand each other,” Worth says. “There are other plants where communication seems to be broken down; they all kind of operate in a vacuum. Those are the ones where we see more problems.”

To learn more, read "Build it better next time."