MKVIe IO Pack Failures

C

Thread Starter

Chutney

Hi everyone,

I'm just trying to find out if other MKVIe TMR users are experiencing a high rate of IO Pack failures? Last week we changed out 9 maybe 10 IO Packs. This failure rate seems exceptionally high. The MKVIe was installed 2006-2007 to give an idea of age of IO Packs. It has mainly been Rev C that have failed.

Also I'm hoping someone can give a explanation of the "Module Required" check box on the modify module pop up box within Toolbox ST. There isn't much of an explanation in GEH-6700.

If anymore info is required let me know.

Cheers,
Chutney
 
Pray tell--which I/O Packs are failing? Is it random packs, or more generally DI/DO packs, or analog I/O packs?

What are the Diagnostic Alarms which are present on the packs before they require replacement?

Has something changed recently which has caused the temperature inside the panel/compartment/room where the I/O Packs are located to increase, possibly substantially?

And, lastly, grounds--is the Mark VIe control system powered by a 125 VDC battery, and if so, is there a 125 VDC Battery Ground alarm continually active or intermittently active? Battery charger output filter capacitors have been known to cause voltage spikes which can damage control system components.

Does the Mark VIe use DC-AC converters (DACAs, specifically) to provide power in addition to the 125 VDC--or some other type of DC-AC converter? Is it possible there is a problem with the DC-AC converter, or if it's a UPS or inverter providing the power to the converter, could it be a problem with the UPS or inverter?

GE DACA modules are NOT filters, and the filters used on the power supply to the panel are really more to prevent electronic noise from getting OUT of the panel, as opposed to smoothing any voltage/current spikes from getting into the panel. Worse, GE DACA modules can actually magnify any voltage spikes on their input making the likelihood of damage to control system components even more likely if the power source is not clean.

Please write back to let us know what you find!

As for the "module required" check-box, I don't presently have access to a ToolboxST, but would suggest looking at GEH-6721 for help with configuration and Diagnostic alarm issues.
 
Thanks for your reply CSA.

Basically you name it they have failed!
PAIC, PDIA, PTUR, PDOA, PTCC....

Typically we aren't getting any diagnostic before they fail, just the alarms below when the IO Pack fails, in this example T:
Inputs unhealthy on IO Module 25, T pack IONet 3 - Message Timeout [Alarm ID 1537, UCCC-0, Controller R]
Inputs unhealthy on IO Module 25, T pack IONet 3 - Message Timeout [Alarm ID 1537, UCCC-0, Controller T]
Inputs unhealthy on IO Module 25, T pack IONet 3 - Message Timeout [Alarm ID 1537, UCCC-0, Controller S]

Errors 20 and 24 can then ben seen on IO Diagnostics.

Its also worth noting that when these IO Packs fail they are completely dying, all the lights go off on the front.

Sometimes we are seeing a 28V PWR Supply low alarm on an unrelated IO Pack (for example when a PDIA pack failed we had the 28v low alarm on a PRTD pack less then a second before the PDIA died). I'm inclined to think that as the IO Pack dies it is affecting the 28V on the system rather then the low voltage on an unrelated IO Pack affecting another.

I've only been on this site since 17th of October, talking to the guys onsite this isn't a new issue. IO Packs have been failing regularly for some time. I don't believe there have been any changes to temperature, location etc.

On the drawings I've seen it states panel is fed by 240Vac, 50hz from a UPS. I don't have any details on the UPS to hand. But I can do some investigating.

Cheers,
Chutney
 
Dear Chutney,

at recent conference I heard from other users of the MKVIE like yourself experiencing issues with premature I/O pack failures. The suspicion from most users was it was due to heat. I know that aero derivative users have experienced major issues with I/O packs and switch failures that have been traced to heat with switches and I/O packs mounted at the unit itself. But even users running frame machines with all I/O in a climate controller PEECC are experiencing large numbers of failures.

I would hit up GE in a PAC case or Controls Connect and see what they have to say.

The GE staff that attending the conference did not seem aware of these I/O pack issues. But I know you are not alone. And the MKVIE hardware is still going through the growing stage in my opinion. The other users in the conference had contacted GE about the failures and expressed concern about heating. This heating was evident when the plastic of the I/O pack was removed. They had inquired about adding fans to the cabinets to help with cooling. GE advised against it, which I personally feel is wrong. GE's point was that forced cooling stirs up dust etc, but my feeling is if you suspect heat as a failure mode, then decrease temperature of the device, at least to test the theory. In my opinion GE overstated the temperature limits of the MKVIE hardware, and they are still working on upgrades, and learning at the expense of users. Again just my opinion. I hope that other users will chime in with their experience.
 
Chutney,

You say the Mark VIe was installed in 2006-2007; that's not old but it's not young, either.

You can see the effects of heat yourself, especially when the I/O packs are installed in vertical rows. And TMR packs on terminal boards are all very close vertically. The heat generated by the packs rises, and when there are packs vertically directly above other packs, the heat just "multiplies" as it rises from the bottom towards the top of the cabinets. And heats the upper packs more than the lowest packs.

I concur with MIKEVI--fans are definitely warranted in Mark VIe panels, <b>BUT</b> housekeeping is also VERY important. Dust and humidity are rampant in many places and housekeeping is always "someone else's" job.... If control compartments are not maintained properly, and dust and humidity are allowed to enter and build up in the compartment, and in the control panel, then there are going to be problems.

Adding filters to the control panels can be very problematic if the filters are too restrictive. Even with fans drawing air in through the filters. In my opinion, fans should be located in the upper section of the doors, drawing air in through louvers in the lower section of the door. As air temperature increases as it's drawn in from the bottom, it naturally tends to rise--and a fan or fans drawing air to the top of the panel and then exhausting the warm air out of the panel is the best option. And I'm not talking about small "muffin" fans, either. Fans should be moving substantial amounts of air to provide sufficient cooling.

But, again, if housekeeping isn't good or is non-existent the fans are just going to draw more dirt and humidity in to the panel with the air and it's going to collect on electronics--and that's a bad combination (electronics, dust, and humidity).

It certainly does seem like the temperature ratings of the packs are a little high (meaning they won't withstand as much temperature as stated). But I think those ratings were made with single packs--NOT with packs arranged vertically, such on TMR terminal boards, and NOT when the packs were stacked in tall vertical columns.

You haven't told us if the packs which are failing are centrally located in a single control panel, or if they are in distributed junction boxes, or some combination of the two.

I also believe power supply problems can have a lot to do with pack failures, but that's just a personal opinion with no actionable data to back it up.

A lot of OEMs have a very difficult time with understanding actual long-term operating problems--they just don't have long-term operational experience; they just design and contract the building of control systems, not operating them long-term. They should be looking at the number of spares they're selling (which they make more money on than when sold in complete control systems!) and identifying problems. But, the profit motive doesn't always allow for that (capitalism being what it is).

There are a lot of common problems among various sites, but there are also a lot of site-specific variables, too. But, do check for yourself and see how the heat increases in the vertical columns of packs. I think you (and anyone with Mark VIe control systems reading this thread) will be surprised. Install some temporary T/Cs at various heights in a stack and record the temperatures, when the turbine isn't running, and then when it is--with the doors closed. Then install some fans and again monitor the temperatures when running and not running. You should see a difference, probably marked. If not, try different fans or fans in different locations, or ducting air from air conditioners to the lower door louvers. You might even try putting some small fans inside the lower area of the panel pushing air up to the top while monitoring the temporary T\Cs.

Get some data, and provide it to GE. They might (or might not) do something with it.

And write back to let us know what you learn as you work on trying to understand this problem.
 
Hi Chutney,

Yes, I/O pack failures are common and we are seeing a lot of failures these days. WORLD OF CONTROLS can repair your defective packs and back it up with a 12 month warranty. There are a few components that need an upgrade which will avoid any failure. Would you please send me an e-mail at [email protected].
 
Hi Chutney,

Since the failure is not limited to any specific type of IO packs, we can consider 2 prime suspects, as mentioned by CSA, Panel Temperature and Power supply.

For the IO pack temperature monitoring, each MarkVie system IO pack are equipped with inbuilt temperature monitors. You may find the temperature of each pack by going into ToolboxST application.

Example: G1(Device name)-> Hardware(tab)-> PDIO-1A2A-> Variables(tab)
Temperature measured within the pack will be shown on a variable (IOPackTmpr_R for <R> pack). This will be in DegF units. Please monitor the temperature for all the IO packs, and this should give us an idea about the cause.

I don't see any response from you about the query from CSA regarding ground alarms. If you have a PPDA module for monitoring power supply status, please trend the voltage levels of 28VDC supply to confirm that power supply is healthy.

Module Required selection on the IO packs are to inform the controllers, whether it is a critical module. This is unchecked in case if the IO pack is just a mounted spare. In case a Required module is not healthy related to <R> controller, then <R> controller cannot go to "Controlling" mode when powered up and will remain in a status "Inputs disabled" until the pack becomes healthy.
 
A lot of "failed" Mark VIe I/O packs get returned to GE with absolutely nothing wrong with them--the real culprit is all to often improper grounding, a bad network switch, or a power supply issue. My guess from my experience is grounding, as far too many sites have inadequate or improperly built grounding pits. The problems can come apparently out of nowhere, as soil conditions or other site changes can change grounding performance.

Another issue I've seen is simply not tightening down the bolts on the I/O pack enough for a good grounding. Worth a check.
 
Module required just means that the Mark VIe won't go into a Controlling state without that I/O pack. During startup the controller goes through a series of states (see the description of ControlState_x intrinsic in GEH-6700 for a list). Outputs are not enabled until the controller is in the Controlling state.

If you uncheck Module Required, you can get to Controlling while that pack is offline. This is useful during commissioning or when something needs to be taken out of service.
 
Top