Expensive Fuses, Lightning Bolts and… a Ghost? Call for Disaster Recovery
Disaster Recovery! What’s that got to do with a ghost?
Most of us have experienced a long dark night with the unexpected noise. Footsteps on the stairs. A creaking door… STOP! Sorry got a bit ahead of myself then. Restart… Most of us have experienced near or full scale disasters as a result of mains related issues. Whether these are short dips in mains voltage, total failure of the mains power, excessive floating voltages or random conditions such as high voltage surges and spikes. No matter how much you spend on an uninterruptible power supply (UPS) for protection it does not guarantee continual up time, nor does it provide an absolute guarantee that your equipment is protected. On some of the sites I have been involved with their mains supply is so bad the UPS is often nothing better than an expensive but reasonably effective fuse. In short, the UPS is sacrificed to protect the more expensive communications equipment.
Anyway, we all know ghosts don’t really exist… Don’t we?
On one occasion I remember an entire data centre going down when a mains lead was inserted into the back of a piece of comms equipment to power it up. The UPS engineers told me the failure was most likely due to arcing as the plug was inserted into the kit. The subsequent harmonics blew the output stage of the UPS. It was confirmed several days later when the UPS was back in operation that there was nothing actually wrong with the comms equipment.
“…the UPS was delivering 400V over the 240V feeds to our cabinets – OUCH!”
On another occasion we had a report on a Sunday morning from one of our customers that their entire network was down. We hadn’t received any automated messages and when we tried to gain remote access everything was dead. When we arrived on site all of our equipment was dead. We investigated further and found the room UPS had become defective and was delivering 400V over the 240V feeds to our cabinets.
Some of the equipment was fried and needed replacing but there were some old favourites from a Canadian manufacturer called Memotec that just went back to normal as soon as their fuses were replaced. Over the years we have found some vendors equipment to be more robust than others to the point where we could predict fairly accurately what spares we would need to take following any major disasters. We supported a lot of Memotec equipment and it had a well-earned reputation for reliability. Their power supplies weighed a ton but they never let you down.
The moral of the story here is that we buy and install a UPS to protect us from mains irregularities but it is easy to get lulled into a false sense of security and overlook the the potential problems that can be introduced by a UPS. However rare these incidents may be, they need to be considered and included in Disaster Recovery and Business Continuity plans.
“…I have long since totally dismissed the advice that lightning never strikes in the same place twice”
The majority of my career has been spent supporting customers in remote locations that were neither mains power or telecommunications friendly. Lightning was a constant unpredictable menace and in some locations a recurring menace. I have long since totally dismissed the advice that lightning never strikes in the same place twice and if you have ever supported a customer on the edge of a natural lightning bowl you will understand exactly where I am coming from.
Lightning can be a problem almost anywhere. On one occasion, we were called in as a third-party rescue team when entire hospital network was taken out by one strike. We soon discovered that the data cables ran between floors through external steel conduit. I am not criticising this method of installation. This was a fairly common cable pathway method in old buildings that were never designed for the masses of data cabling that appeared in the last quarter of the last century (that makes it sound ‘really old’ doesn’t it?). In some instances, external pathways were used to avoid disturbing asbestos in the building fabric. On this occasion, the lightning struck the steel conduit rather than the deteriorating lightning arrestor system. The data cabling comprised a foil shielded twisted pair wired in a bus topology (the system pre-dated Ethernet) and the cable was attached to every piece of networking equipment from one end of the hospital to the other.
“…crammed into one tool bag that Mary Poppins (as I live and breathe) would have been proud of”
This was back in the days when we carried a proper tool kit everywhere we went. All crammed into one tool bag that Mary Poppins (as I live and breathe) would have been proud of. Thinking about it now, that could that explain why I had to visit the chiropractor every couple of weeks for several years. Our tool kits were certainly put to good use at the hospital that day. The entire hospital network was down and our hopes of finding a single common problem were dashed when we realised the bus cabling had delivered the lightning to every terminal server on the network. Given the urgency of the situation we managed to make some good units out of the parts of the faulty units to get some areas up and running. We repaired some units on site but the remainder were beyond any repair, especially those where the circuit board track had vacated the board. Some of the units were so bad their internal components resembled coal.
Lightning strikes are certainly not a thing of the past. I have encountered them most years of my career. There are some pictures at the bottom of this article of a victim of a lightning strike hitting a telephone line and literally blasting holes in the equipment on the end. As you can see the lightning punched a hole through the steel case. Incidentally, many of our remote customers are served by telephone poles and when I did my training (from memory because it was over 40 years ago) lightning suppressors were only fitted if there were more than four overhead spans.
“…my colleagues thought it was great fun to tell me about the ghostly sounds and sightings ‘witnessed’ in the building”
Lightning wasn’t the only electrostatic issue we experienced. Electronic equipment has always been susceptible to electrostatic discharge but from around the early eighties to the mid to late nineties static problems were very common but widely misunderstood and often mis-diagnosed. When I worked for Cable & Wireless one of my customers were losing contact with their message switch at between 4am and 5am most week day mornings.
I was asked to investigate the problem and having determined the equipment was operating as it should I decided to camp on site to see if I could witness the problem. The fact that the system was stable over weekends led me to believe the system was failing due to an environment issue. The site was a very old building in London and my colleagues thought it was great fun to tell me about the ghostly sounds and sightings the local staff had ‘witnessed’ in the building.
“I heard a distant sound of a creaking door followed by a screeching… then the door swung open with a crash and…”
Other than the equipment fans, air conditioning and disk drives there were no other sounds, ghostly or otherwise. I thought I was in for a really quiet night when I heard a distant sound of a creaking door followed by a screeching… well actually it was more of a ‘squeaking’. Then the comms room door swung open with a crash and… in came a squeaky old trolley with elderly gentleman in tow – the cleaner.
After exchanging pleasantries and allowing some colour to come back into my face we got talking. He said he had worked for Cable & Wireless for 40 years until his retirement a few years ago. He told me he had really enjoyed his career and he was now working as a cleaner as a part time retirement job. He then came over to my closely guarded equipment cabinet, took out a dry synthetic duster and vigourously dry polished the perspex cabinet door to make that Cable & Wireless logo on the door shine above all others and… the message switch crashed… for the last time. Was my friendly cleaner visiting from the other side? I will never know, but no one mentioned the cleaner to me before my overnight stay.
“Was my friendly cleaner visiting from the other side? I will never know, but no one mentioned the cleaner to me before my overnight stay.”
We still have to take care to protect our systems from electrostatic charge but fortunately modern equipment is much less susceptible than the older systems. In addition to this, the experience gained through being around when most of the current technologies were first introduced allows us to design around all of the problems we are aware of.
A Recent Victim of a Lightning Strike…
