Check out our White Paper Series!
A complete library of helpful advice and survival guides for every aspect of system monitoring and control.
1-800-693-0351
Have a specific question? Ask our team of expert engineers and get a specific answer!
Sign up for the next DPS Factory Training!
Whether you're new to our equipment or you've used it for years, DPS factory training is the best way to get more from your monitoring.
Reserve Your Seat TodayI recently visited a DPS client (a fiber ISP) to perform some generator troubleshooting at a fiber hut in a rural area.
This site contained:
In my 16 years at DPS, I've been involved in hundreds of projects where a generator needed to be monitored. In many respects, not much has changed since I started in 2006.
We still monitor propane/diesel tank levels today. We're still interested in the voltage output. We still worry about oil pressure and engine temperatures. The laws of physics have changed very little in the last few decades.
What has changed is the way that an RTU like the TempDefender must talk to generators. In the old days, nearly everything was contact closures. We could also strap temperature and vibration sensors onto the generator as "acid tests" that would continue to work even if a generator was offline.
Today, generators have moved toward being "smarter", much like many other modern devices. That means communication using machine-to-machine (M2M) protocols instead of simple contact closures. For generators, the most common protocol used to day is Modbus.
Modbus empowers generator owners to collect literally thousands of different data points covering all aspects of generator status. This is great, but configuring protocol communications is far more complex than wiring in simple contact closures.
That's where we were running into trouble at this fiber hut. That's why DPS sent me for a visit to resolve the issue.
When I arrived at the site by mid-morning, the CAT generator technician and my client were examining some unrelated generator issues.
As I arrived, we shifted to troubleshooting the problems they were seeing with Modbus and our TempDefender RTU.
In short, about 20% of all Modbus polls were not completing successfully. This was then causing SNMP notifications to be sent to Solarwinds. In the monitoring industry, we call these "nuisance alarms". To create an effective monitoring system for the on-site generator, we needed to stop this nuisance.
I've seen quite a few machine-to-machine communication problems in my time at DPS, and "intermittent" is a very special case. Usually things 100% work by virtue of being properly configured or 100% fail because something is set incorrectly.
When you see transient/intermittent failures, you're usually on the fringe of some kind of randomness.
Suspecting that noise could be the issue as the unshielded CAT 5e cable passed by the ATS, we tried an alternate cable temporarily strung across the floor. Unsurprisingly for a cable run that was only about 50 feet, noise wasn't the issue.
We also tried adding termination resistors at the generator, at the TempDefender (internal jumpers are fairly convenient for this), and then both simultaneously. This may have helped a bit, but we still had an issue.
I then hopped on a phone call with a DPS engineer back in California, and he had the right idea.
Modbus is a half-duplex serial communications protocol. Half duplex means that data can only flow in one direction at a time. This is opposed to full duplex, where data can flow in both directions simultaneously.
The RS-485 specification for Modbus defines timings for when the RTS (Request to Send) and CTS (Clear to Send) Modbus control signals can be asserted. Specifically, the RS-485 specification states that:
"The time between assertion of RTS and deassertion of RTS (the head time) shall be greater than 3 character times but no greater than 15 character times. The time between deassertion of RTS and assertion of CTS (the tail time) shall be greater than 15 character times but no greater than 60 character times."
In Modbus, the RTS signal is used to control when data is transmitted. The Modbus master (our TempDefender G2 RTU) will assert RTS, and then the Modbus slave (then CAT GCCP 1.2 Control Panel for the D60 Generator) will assert CTS to indicate that it is ready to receive data.
If the RTS Head time is too short, the Modbus slave may not be able to assert CTS in time. This results in a Modbus timeout error. If the RTS Tail time is too long, the Modbus master may timeout waiting for CTS. Either of these timeouts will result in a Modbus communication error.
We looked at our built-in TempDefender debug and found what looked like our culprit...
After some further troubleshooting, we were able to determine that the RTS Head and RTS Tail times were not properly configured on the TempDefender and generator's Modbus (RS-485) ports. We adjusted the timings to the following:
These settings resolved our Modbus/RS-485 troubles:
TempDefender RTS Head: 30 ms
TempDefender RTS Tail: 5 ms
CAT GCCP 1.2 Control Panel Inter-Frame Delay: 10 ms
Baud rate (both devices): 9600
If you're seeing Modbus communication errors, be sure to check your RTS Head and RTS Tail timings. Improperly configured timings can result in Modbus communication timeouts.
Site visits are underutilized in today's world. As powerful as videoconferencing technology can be, there's just no substitute for actually being somewhere.
That's why we routinely schedule site visits to conference rooms and remote sites like yours. We want to say "hi", look at your actual equipment sites, talk to you about what you're trying to accomplish, and make recommendations.
We've also found something good for both of us: Your boss will sometimes walk into the room (if you're not already the boss yourself). That helps us all get on the same page and really get a project moving forward.
We live and breathe remote monitoring here at DPS. Just give me a call and tell me what you're trying to accomplish. We'll plan out a project together.
You can reach me at 1-800-693-0351 or email me at sales@dpstele.com
Andrew Erickson
Andrew Erickson is an Application Engineer at DPS Telecom, a manufacturer of semi-custom remote alarm monitoring systems based in Fresno, California. Andrew brings more than 17 years of experience building site monitoring solutions, developing intuitive user interfaces and documentation, and opt...
