Check out our White Paper Series!
A complete library of helpful advice and survival guides for every aspect of system monitoring and control.
1-800-693-0351
Have a specific question? Ask our team of expert engineers and get a specific answer!
Sign up for the next DPS Factory Training!
Whether you're new to our equipment or you've used it for years, DPS factory training is the best way to get more from your monitoring.
Reserve Your Seat TodayThis story demonstrates what the term "You don't know, what you don't know" is really all about. It also further illustrates how just having any off-the-shelf monitoring can you leave you exposed.
An energy service provider in the western United States recently upgraded their Master Alarm System. They were in need of a central location for monitoring of their remote sites. Some of the power producing sites capable of producing over 5 gigawatts include: Solar farms, natural gas stations and clean burning coal sites. With a customer base of over 1.3 million people, encompassing 50,000 square miles - gives you an idea of how remote some of their locations are.
This particular project was slotted for upgrading of 32 of their remote sites. Remote huts can vary by industry. But generally, they consist of multiple pieces of equipment, all which are critical to operations. Although monitoring systems certainly encompass all aspects of a remote hut, the story is centered on our client's emergency power systems.
The emergency power system at each remote site used a generator (diesel fuel) as the power source. If you are familiar with generators, you probably are already aware of the essential need of performing routine testing. Most emergency generator testing requirements indicate a minimum of a monthly test. A generator test, consists of energizing the generator and running it under load for a set duration of time.
The National Fire Protection Association (NFPA) is widely known as a codes and standards organization. The codes and standards set forth by the NFPA include a section on Emergency and Standby Power Systems. NFPA-110 8.4.2 indicates:
Emergency Power Supplies in service shall be exercised at least once monthly for a minimum of 30 minutes, using one of the methods:
- Loading that maintains the minimum exhaust gas temperatures as recommended by the manufacturer
- Under operating temperature conditions and not less than 30 percent of the EPS standby nameplate kW rating
Generator testing occurs regardless of your monitoring status. However, introducing generator monitoring provides many benefits for your organization. Any capable monitoring solution performs several monitoring points, examples being: fuel levels, ground faults and battery conditions.
Our client was running automated generator testing on a weekly basis, at each of their 32 remote sites. Each site was scheduled at a different time and day of the week.
Properly tracking if tests were initiated as planned proved to be a challenging task in itself. The purpose of monitoring your emergency power system, is to provide you a glimpse into the overall health of your emergency power infrastructure. A necessary step to ensure your emergency power serves you during a critical outage. But monitoring your emergency power system also serves a secondary function, to alert your operators of a potential loss of power at your remote site.
When emergency power is being utilized, knowing how long the emergency power can be sustained as well as notifying crews to address the outage is key in maintaining critical infrastructure.
A basic example of how alarm masters function when monitoring emergency power systems is: an alarm will activate when emergency power is in effect. Operators generally have to acknowledge the alarm in order to "clear" it.
For our client, every week 32 alarms would routinely be generated - indicating a generator was activated. They also did not have a solid method of tracking generator exercise. As an alarm would trigger for the weekly exercise, the operator would clear it.
When an alarm triggers frequently for a routine event, you ultimately have operators who consider them non-alarms. These are what we refer to as "nuisance alarms". Alarm events that are expected to occur and trigger frequently.
This is what our client was experiencing with their existing system: A clunky alarm master, cluttered with nuisance alarms. They also had frequent alarms occurring for their generator exercises which required routine acknowledgment.
Operators eventually became conditioned and started discounting these alarms, writing them off as routine generator testing events. They also had 32 alarms to juggle in trying to track the weekly exercises.
Operators have different shifts and work schedules. All of this created an unfavorable environment for our client. Two major storms were brewing in this environment. A cumbersome system of tracking emergency power exercises and the possibility of mistaking a true power loss situation as an exercise.
It would be understandable if an Operator mistook a true event as exercises when their environment has conditioned them to do so. After all, loss of conventional power is not something that occurs very often. If your alarm master for remote site monitoring produces this type of environment - it has defeated its intended purpose.
As was mentioned at the beginning, our client had recently upgraded their alarm-master system. They had decided to implement the T/Mon LNX, the flagship line of master alarm-master systems at DPS Telecom. The T/Mon was chosen for various reasons such as the ability to integrate with a slew of devices (even legacy ones), customization offerings and its overall proven reliability. They also opted to utilize the Turn-Up Service offered by DPS Telecom.
The Turn-Up Service allowed them to complete their upgrade project in a quick and efficient manner. It made sense for them to hire the people who engineered the device to also configure and deploy it as well. The Turn-Up Service is not just an installation service, it can also provide other value such as on-site training as well as customization opportunities.
It wasn't long into the configuration process, that the engineers recognized the nuisance-alarm storm that had been building up.
The experience the engineers brought with them quickly identified this as a problem. They were able to provide an equally quick solution - Derived Alarms. Derived Alarms may already be a term in your vocabulary. Perhaps you have heard the phrases "soft alarm", "virtual alarm", or "intelligent alarming". All of the terms refer to the same general concept - using software logic to produce an alarm. In this case, the correct alarm was: putting an end to the nuisance-alarm fiasco the weekly generator exercises were causing.
Derived Alarm Automation is not a new concept. Andrew from DPS Telecom demonstrates in this YouTube video how to set up Derived Alarms on one of DPS' products.
The DPS engineers sprung into action. By inputting the expected generator exercise schedules of all 32 remote sites into the T/Mon LNX, the engineers were able transform the nuisance alarms into meaningful alarms.
The T/Mon now took the schedule into account, only producing an alarm if the generator exercise did not run as expected. Of course the T/Mon LNX still provided alarms when generators were initiated outside the schedule, preserving the critical indicator of emergency power being utilized.
Now, our client's operators were being provided with meaningful data, a less cluttered screen and no longer allowed conditioning for them to discount alarms. They also now had an intuitive method of alerting when an generator exercise did not occur. This was all simply done by eliminating 32 weekly expected alarms and replacing them with "did not run" alarms!
In the scheme of things, this alone provided genuine added benefit. But for our client, they were soon to discover just how much they did not know. The very next day following the implementation of the derived alarming logic, oddly enough an alarm was produced: One of the sites did not (apparently) initialize its weekly generator exercise. We were confident in our T/Mon's alarm accuracy, but every new installation warrants verification. This unusual alarm absolutely sparked a need for further investigation.
Upon a visit and inspection of the remote site, the newly developed derived alarm was proven accurate. The generator, in fact, did NOT initialize its weekly exercise.
Unfortunately for our client, during this very same site visit, they discovered that the site's generator had been vandalized by copper thieves.
It turns out the theft had gone undetected for weeks. This really drove home the cliche saying of, "You don't know what you don't know." The nuisance alarms and cluttered screen certainly played a role in the non-detection.
Ensuring 32 alarms are occurring on a weekly basis is a very counterintuitive means of ensuring generator exercises are happening. Fortunately, now our client is in a better position of determining the status of their: generators, generator exercises and overall infrastructure.
This certainly was a lesson learned for our client. The take away is if you keep relying on cumbersome systems, eventually you will be left holding the bag. Your operators will be clearing nuisance alarms, mistaking them for exercises and potentially missing a critical event. You also could end up overlooking a missed exercise. Your critical infrastructure is on the line, along with your reputation. The entire reason for your alarm master's existence, is to provide you information in a meaningful format. This way, your able to react accordingly.
One of the ways T/Mon reduces your risk is with GFX maps. Cleverly named T/GFX (convergence of T/Mon & GFX), it's a graphic display of your entire network. By using maps, icons and photos you are presented with geographical view of your remote sites. Combined with multi-layered graphics, you can zoom in to a device level view of your real equipment.
Merging this map view with your alarm statuses, your are given a clear view of your network status. Alarms are color coded by severity, giving easy interpretation of your alarm status to operators. Not only is there a topical viewpoint of your network, you also have the option of utilizing the Change of State screen. A unique way of viewing your alarms you no longer have to hunt for what has changed. The COS screen will list it for you, showing your alarms that have changed from "normal" to "alarm" (or vice versa).
If that is not enough choices of views, you also have yet another option - Standing Alarms screen. As the name suggests, it displays the currently failed alarms. As an system operator, you have all of the current threats to your network in one screen. Both the COS and Standing Alarms screens feature a text message window. This window can provide an explanation of the alarm, or you could even input specific procedures for system operators to follow. They will instantly know how to handle the alarm, or whom to call if needed.
DPS Telecom has a litany of RTU offerings for your needs, each providing the same smart logic to reduce your risk nuisance alarms. The NetGuardian 832A is an example of a smart RTU which DPS manufactures, giving you similar functionality of derived alarms. The NetGuardian 832A has a feature referred to as a "qualification timer". It provides yet another way to handle a nuisance alarm, by delaying the way an alarm is received.
A simple example of how a qualification timer would be implemented is with sensitive types of sensors. Door sensors, at times, can provide a false alarm. Temporary vibrations sometimes set these alarms (picture a train passing or low flying plane). The door sensor may trigger a momentary alarm indicating the door is open.
However, in the scenario illustrated, it would be a false alarm. Setting a qualification timer on this sensor, would transform this into a meaningful alarm. By having the NetGuardian disregard this alarm unless being active for (lets say) 5 seconds, you know have eliminated the false alarm.
The NetGuardian is capable of providing alarm notifications to several devices at once. When you send alarms to multiple devices, you create a real robust notification system. When an alarm is activated a notification is sent to any combination of devices: Multiple SNMP Managers, SMS-to-email or emails. You can configure any mix of devices totaling up to 8.
The NetGuardian works symbiotically with T/Mon to create a robust, redundant system. As the NetGuardian is capable of derived alarms, as well as controls, you are provided a true redundant system. When unexpected situations arise, like when mother nature throws a hurricane at your central location, your T/Mon may be unavailable for durations longer than anticipated. In these types of situations your NetGuardian will still continue to work, providing you and your network with continued coverage and meaningful information.
This is what DPS Telecom does best. We strive to provide all of our clients with meaningful information that assist in keeping them operating and overall success. We are a manufacturer of quality, and as you just read reliable equipment. With alarm masters such as our T/Mon LNX, you can be provided with the same reliability as all of our clients. Perhaps you don't have 32 remote huts. Our equipment has several scalable options to fit your needs. Regardless of the size of your organization you can be assured that there are options for you.
But we just don't manufacture equipment. We also provide our own tech support for our products. The same engineers who perform on-site installations, also provide the tech support. This provides all of our clients with high quality support by engineers who have intimate knowledge of how the systems function. Call us today at 559-454-1600 to learn how we can help you and your company can integrate reliability into your operations.
Andrew Erickson
Andrew Erickson is an Application Engineer at DPS Telecom, a manufacturer of semi-custom remote alarm monitoring systems based in Fresno, California. Andrew brings more than 17 years of experience building site monitoring solutions, developing intuitive user interfaces and documentation, and opt...
