Identify the symptoms of equipment failure

Production automation and control networks provide diagnostic data to get to the root of the cause.

In brief:

  • Not only system diagnostics, but also diagnostics from the production process itself, already exists on the network. This can help improve productivity or OEE.
  • Although the data exists, getting it to the right person at the right time is critical for good decision-making in the machine-control environment.
  • When the control network and the enterprise network are connected, the same malicious attacks that can be made on the enterprise network now have the opportunity to strike the controls network. Data needs to be accessible, but proper virtual private networks and firewalls need to be used to protect the process.

For years, medical professionals have warned us to treat the cause, not the symptom. That’s sound advice, whether it applies to a person or manufacturing equipment. Industrial plants, however, must first identify the symptoms so they can intervene proactively to minimize downtime, or at least conduct a root-cause analysis when failure occurs.

But where does one find a machine doctor who can work 24/7? The answer probably already exists in your system control network. Smarter equipment means smarter diagnostics, so the data’s available, but how you use it is up to you.

New equipment, new information

Figure 1: Data management software can track available raw material, production parameters, or even troubleshooting, fault codes, and downtime. (Source: Siemens Industry)
Figure 1: Data management software can track available raw material, production parameters, or even troubleshooting, fault codes, and downtime. (Source: Siemens Industry)

The Siemens Industry manufacturing plant in West Chicago, Illinois, implemented new equipment from Prima Power as it ramps up new production capabilities, especially of its motor control centers, and the new integrated production system also provides highly useful information. The plant is using data from the Prima Power software for three main areas, explains John Hayden, manufacturing engineering manager, control components and systems engineering, at the plant. “Our materials organization is using it in real time to know exactly how much raw material is available for each type and thickness of material,” he says (Figure 1). “Manufacturing engineering is using the data for information such as sheet utilization, number of hits, and punch life. And maintenance uses it for diagnosing troubleshooting, fault codes, and downtime.”

Siemens’ West Chicago plant has transitioned from a manually loaded coil steel shear process, which required significant setup for each different raw material, where parts were cut to the right part length, and then transferred manually to a shearing station, where they were sheared to right width, says Hayden. “These parts would then be transferred manually to a turret press, where they were manually loaded for punching, and then finally unloaded manually and transferred to a brake press, where they were formed, which required significant setup for each part,” he explains.

The new automated process has raw material stored in a cassette that’s called down to the turret press at the time it’s required and loaded automatically, explains Hayden. “Once it’s punched, it’s retained in a WIP cassette until the automated brake is ready. The WIP cassette is called down and loaded automatically, squared, and conveyed into the brake for forming. Setup time is done automatically, as well as eliminating the time-consuming manual brake setup.”

The plant also has a preventive maintenance system tool, Mpulse, that maintains the schedule for each piece of equipment. “As part of our Siemens Starpower factory program, we also review our maintenance activities and the associated downtime, on a quarterly basis to identify needed additions or improvements to the system to eliminate unexpected downtime,” says Hayden.

Find a sustainable maintenance plan not conflicting with production planning, suggests Lutz Ehrlich, punching/automation product manager at Prima Power (www.primapower.com). “Become proactive in the maintenance and cleanliness of the complete system,” he says. “Treat the system like an airplane. Don’t attempt to check the engine gearbox oil at 30,000 ft.”

From paper making …

Based in Alpharetta, Georgia, Cellu Tissue Holdings, a division of Clearwater Paper (www.clearwaterpaper.com) has a strategic focus on consumer-oriented, private-label tissue products. The company manufactures a combination of internally converted tissue products, machine-glazed paper and tissue hard rolls.

Cellu Tissue’s tissue segment produces both converted and hard-roll tissue, including facial and bath tissue, kitchen towels, napkins and folded paper towels, and its machine-glazed paper segment produces and sells machine-glazed paper hard-rolls to third-party converters who manufacture fast food and commercial food wrap, gum wrappers, coffee filters, foil lamination, wax papers, and butter wraps.

The company’s Interlake facility in St. Catharines, Ontario, has two production lines: the PM2 wet crepe machine, which primarily manufactures coffee filter base paper, and the TM3 through-air-dried tissue machine, which primarily manufactures premium kitchen towels.

A multitude of issues and situations within the papermaking process can cause system performance to decline and possibly lead to downtime. Technicians are busy simply keeping up with preventive maintenance and corrective services.

At the St. Catharines mill, the HP 2002 quality control system (QCS) on the No. 2 paper machine was contributing to rising maintenance costs. It was increasingly difficult to obtain technical support and spare parts for the legacy system, which was originally installed in 1986. Because of a lack of Ethernet network connectivity, technicians had to record data manually. Justification for replacing the outdated QCS was based on the implementation of a modern monitoring and data collection system that would reduce annual maintenance expenses.

New upstream sensor arrays are installed in the former, in the press section, and inside the dryers themselves. Though the visibility provides a means to optimize the process and improve quality, these sensors and upstream sensor arrays also are coupled with new developments in multivariable predictive control applications for both machine-direction (MD) and cross-direction (CD) control.

The St. Catharines mill enlisted Honeywell to provide a new Da Vinci QCS for its No. 2 paper machine, similar to the system installed on the sister No. 3 tissue machine in 2006. Da Vinci uses an open system architecture, multivariable predictive control, and precision basis weight measurement.

The Da Vinci QCS’s application server runs the performance MD and CD controls, and it supports precision platform and measurement sensors. Together, these provide supervisory MD controls, supervisory CD controls, supervisory color control, process information, historical trending, statistical analysis, and printed reports.

When the rare stoppage occurs, the new automation system announces exactly where the fault is at the main control panel.

Cellu Tissue’s QCS upgrade allowed for implementing Honeywell’s remote monitoring service, too. Ethernet network connectivity with Da Vinci QCS allows technicians located in Honeywell’s Houston office to monitor the mill’s scanner equipment and diagnose problems. Remote monitoring capabilities range from continuously monitoring process variable data to pulling trend information and providing recommendations. Technicians can even take control of the system remotely, if needed.

A virtual private network (VPN) connection between the mill and Honeywell allows only authorized personnel who are configured to access the network. A Honeywell centralized server loaded with monitoring software is responsible for polling and alerting of potential system problems.

Honeywell personnel log into the equipment health monitoring tools daily to log alerts, including low-priority ones, and collect trending data. Monthly reports, including CD power spectrum (machine signature), reel report overview, shift production report, and standardized summary report, assist the St. Catharines facility in addressing improvement opportunities. Periodic meetings also are held to review alerts, trending logs, and recommendations.

“The Ethernet connectivity of the Experion MX QCS, from the system servers all the way down to individual scanning sensors, enables data transfer of process and system health information to local mill-wide systems, local or corporate data historians, and Honeywell remote service infrastructure,” explains Ross MacHattie, director of product marketing for pulp and paper at Honeywell Process Solutions (www.honeywell.com). “Quality data is collected from high-speed scanning to provide a statistical record of the paper as it is being made. This can be kept in an historian for process troubleshooting or customer complaint analysis. Remote analysis of the data can be done by process experts in near-real time to monitor the process for instability or other problems.”

Production data is sent to manufacturing execution systems for trim optimization and as-shipped product records, explains MacHattie. “System health information can be used by remote service personnel to keep tabs on the QCS performance without having to be in the mill every day,” he says. “They can even talk to the individual sensors remotely through Ethernet connections, which results in better predictive maintenance.”

During 2009, QCS remote monitoring reduced the Honeywell maintenance contract at the St. Catharines mill by more than 50%.

… To paper printing

The RockTenn (www.rocktenn.com) printing plant in Jacksonville, Florida, was experiencing gradually increasing unreliability of its automation systems. The systems produced too many false alarms, and there was no feedback as to where the problem was and no diagnostic tools.

Figure 2: The press is 14 ft wide and nearly 150 ft long and was built by Cobden Chadwick of England, a firm now out of business. (Source: AutomationDirect)
Figure 2: The press is 14 ft wide and nearly 150 ft long and was built by Cobden Chadwick of England, a firm now out of business. (Source: AutomationDirect)

“The plant had to test each interlock physically until it located the fault,” says Andy Banaczyk, head of maintenance at RockTenn. The programmable logic controllers (PLCs) controlling the printing plant were obsolete and too expensive for upgrade with the original manufacturer, so Banaczyk decided to replace all of them.

The RockTenn plant prints more than 900 huge rolls of paper each month. The plant prints the outsides of pasteboard cartons that will contain beer, yogurt, drink pouches, and other products. The press is 14 ft. wide and nearly 150 ft. long (Figure 2). It was built by Cobden Chadwick of England, a firm now out of business.

The paper is received in rolls that are mounted at the center of the press. The paper is threaded through the machine and around a large temperature-controlled cylinder. After printing, the paper passes through a dryer and an inspection station, before being re-rolled.

The process is continuous: as one feed roll is emptied, a mechanism cuts the paper and splices on the waiting end of another full feed roll. The same thing happens at the other end: as the printed roll reaches capacity, a cutter cuts the paper and splices the end onto the empty core waiting above the now-filled printed roll. The filled printed roll is then moved out and the empty core roll is lowered in its place.

Support systems for the printing operation include off-line re-winders, plate-making equipment, automated storage systems, overhead cranes, waste treatment, and compressors.

The obsolete PLCs monitored and controlled the entire press from one central location. With so many heavy moving parts, the press was ringed with safety gates and interlocks.

Banaczyk began the retrofit project by replacing some of the original PLCs with AutomationDirect PLCs, but he soon realized he needed help. He called on W.L. Smith Electronics (www.wlsmithelectronics.com), which designed, built, and supplied many of the control enclosures. Smith also programmed many of the PLCs that were installed into various recesses and enclosures around the press.

After some of the controls had been changed, Banaczyk wanted to add inter-PLC communications and data transfers to allow the PLCs to share status information. He called on Expert Automation Design (EAD, www.eanda-technical.com), a systems integrator whose specialty is upgrading older automation systems, and first tested EAD’s programming ability on two small off-line projects — an elevator that stores the printing sleeves and the sprinkler controller for the grounds.

Banaczyk didn’t have the original PLC code and had no way to convert the old code, so EAD had to rewrite the control and safety code for each new PLC, with the help of AutomationDirect’s DirectSoft programming software. Banaczyk implemented a voice-broadcast system using the audio capabilities of the C-more touchscreen. He recorded many messages tied to specific actions or operations. When the press starts, three voice messages are heard in succession: “Caution — stand back. Hands up. Starting the press.” If a gate is opened, the voice announces which gate to alert operators. If the press won’t start, the voice announces why and where the problem lies. Banaczyk can change the messages or the triggers as needed.

The new PLCs include DirectLogic D2-260 and D0-05 models connected to C-More touchscreens, all from AutomationDirect. The plant also replaced many of the pushbuttons, selector switches, and indicator lights. “We set up a communications system, so that all the PLCs in the building, with a few minor exceptions, are linked by an Ethernet network,” explains By Glenn Erickson, president of EAD. “We can call up any linked PLC and monitor its status or make changes to the PLC logic, all from any of several PCs around the plant.”

The entire system also is tied to a PC running DirectSoft software, which allows Internet access from Web browsers. Any authorized person can log in from anywhere in the world and monitor the system or make changes, which has resulted in reduced downtime and service calls, says Banaczyk.

Figure 3: The main control panel tells operators and maintenance personnel exactly where the problem is located. (Source: AutomationDirect)
Figure 3: The main control panel tells operators and maintenance personnel exactly where the problem is located. (Source: AutomationDirect)

Some PLCs control temperature, safety systems, and press operations. Some simply monitor operation. The PLCs pass and exchange data among themselves as needed over the network in a peer-to-peer network configuration with no PLC as a master, eliminating the single point of failure. To date, there are 30 new PLCs with nearly 2,500 corresponding I/O points.

The first project, the updated safety system, reduced unnecessary stoppages dramatically. “I couldn’t count the number of stoppages we used to have,” says Banaczyk. “I just know it was a nightmare before, and no problems now.” And when the rare stoppage occurs, the new automation system announces exactly where the fault is at the main control panel (Figure 3). With fewer safety system faults and quick resolution of fault causes, this item has dropped off the charts as an area of concern, says Banaczyk.

Is there a doctor in the network?

One advantage of using the controls network for diagnostic information is that the network already exists, and many times the diagnostic information is readily available on the network, explains Matthias Hofmann, general manager, factory automation business, industrial automation systems, at Siemens Industry. “This includes not only system diagnostics, but also diagnostics from the production process itself, to improve productivity or OEE,” he says.

Automation systems with diagnostic information that can be presented in an actionable way can reduce the time it takes when unplanned downtime stops a machine, continues Hofmann. “With safety integrated in the automation system, productivity and machine uptime can be improved, also,” he says. “Regarding energy efficiency, an automation system can measure and collect data on machine or process energy usage and determine which components can be shed if peak demands are approaching. This information also can be used to schedule production to minimize energy usage and avoid exceeding peak demands.”

An enormous amount of data is available on the control network, confirms Mike Hannah, manager, networks and infrastructure, at Rockwell Automation (www.rockwellautomation.com). Getting the data to the right person at the right time is critical for good decision-making in the machine-control environment. The information from the network is simplified into actionable formats with tags, faceplates, and Web pages that use standard Web browsers, says Hannah. “Data can be sent directly to the HMI, which is typically where the operator or maintenance personnel would go to view the information and point them in the correct direction for action,” he says. “It also would be possible to send SMS or email messages to the appropriate personnel. Ideally, you want the information to be sent as quickly as possible to the person responsible for getting the machines back up and running.”

“One issue arising about control network data is the connection between the control network and the enterprise network,” explains Bobby Thornton, HMI and communications product engineer at AutomationDirect (www.automationdirect.com). “When these are connected, the same malicious attacks that can be made on the enterprise network now have the opportunity to strike the controls network. Data needs to be accessible, but proper virtual private networks and firewalls need to be used to protect the process.”