Check out our White Paper Series!
A complete library of helpful advice and survival guides for every aspect of system monitoring and control.
1-800-693-0351
Have a specific question? Ask our team of expert engineers and get a specific answer!
Sign up for the next DPS Factory Training!
Whether you're new to our equipment or you've used it for years, DPS factory training is the best way to get more from your monitoring.
Reserve Your Seat TodayMy day began early when I arrived at the client's HQ at 9:30 am. Our goal was a security-driven network cutover, shifting their T/Mon remote monitoring servers from the current IT network to a more secure OT network.
The benefits of such a move were clear: reducing potential attack vectors, isolating sensitive operational environments, and creating a robust barrier against cyber threats.
At 9:41 am, we set off for the cutover site, the destination of our critical operation. Upon our arrival at 10:00 am, I could feel a buzz of anticipation in the air. I always prepare for cutovers like this one, but things can always go wrong.
At 10:20 am, we signaled the start of the cutover operation by notifying affected teams within the organization. I found myself on a conference call with various client stakeholders.
I immediately began verifying the new on-site workstation's access to the Primary and Secondary T/Mons, noting both were showing recent Digitize alarm activity. (Digitize is a DPS Telecom company as of 2022, so now I represent both T/Mon and the Digitize Prism LX system).
By 10:30 am, IT reinforcements had arrived to facilitate network cabling changes. Everything was falling into place.
As the NetGuardian IP address was changed to the new OT address, instructions went out to HQ users to monitor alarms using the Primary T/Mon. I noticed an issue: a firewall effect that prevented web interface access from the new workstation. The staff on the conference call reassured me this was expected due to firewall rules.
I continued working through the morning, focusing on the Secondary T/Mon. Its job was to poll the local NetGuardian at its new OT IP address. However, it remained in its usual passive role, not taking over polling activity from the Primary T/Mon. With a bit of advice from Rich Howell from DPS Telecom's T/Mon Engineering team, I worked to force the polling takeover by the Secondary T/Mon.
Our work paid off by noon - the Secondary T/Mon successfully separated from the Primary T/Mon. This resulted in both T/Mons polling all RTUs, causing alarm from the fleet of NetGuardians to be split between them.
We needed a bit more time. My quick consultation with a client rep led to approval for a brief downtime for the Primary T/Mon to adjust its IP settings.
But then, there was another wrinkle: A preliminary configuration change led to an IP conflict. However, we quickly confirmed and resolved the issue by 1:10 pm.
This was progress. The Primary T/Mon was now at its new OT IP address and back online.
As the clock ticked towards the 2 pm deadline, the client's leadership decided it was time to start the reversion process.
Under the pressure of the deadline, we managed to revert the Primary T/Mon, Secondary T/Mon, and the local NetGuardian to their original IP addresses and settings.
As the 2 pm deadline struck, we completed our short-term mission to avoid getting stuck in an in-between state during the daily busy period. The long-term goal of completing the cutover from IT to OT network would wait for another day.
Despite the day's challenges, it illustrates the intricacies of IT and OT network separation. When you're working on a network transition like this one, you can't be too careful.
Our next step was to plan for a smoother next attempt at this cutover.
To ensure there would be no problems with the next cutover attempt, we wanted to go "back to basics" and verify that nothing about this client's database would hold up the failover from Primary T/Mon to Secondary T/Mon.
Prior to the "Preliminary T/Mon Failover & Synchronization Verification" in the client's network, DPS Engineering would test the following at DPS HQ in California:
This initial testing cannot be a complete simulation of the client's IT/OT networks, but it demonstrates the basic functions of T/Mon and the validity of the cutover procedure.
In the IP Cutover steps (1-26), you need to change step 13 to manually change IP address in the Primary Configuration since you won't be able to "Push these new database changes from Secondary" now that the IP of the primary is different.
This "IT network to OT network" cutover is only partially completed. More remains to be done.
I'll continue writing anonymized details like the above in future articles. This way, you can learn more about cutovers and T/Mon in a real-world context.
If you're considering or actively planning a network cutover from IT to OT, call us at DPS Telecom. With our expertise in network monitoring (NetGuardian RTUs and the T/Mon alarm management platform), we can help you achieve successful cutovers without the usual headaches and delays.
We not a consultant for general cutovers, but we're happy to speak with you for a few minutes to share some "tips and tricks" to help you succeed.
Give us a call at 1-800-693-0351 or email info@digitize-inc.com
Andrew Erickson
Andrew Erickson is an Application Engineer at DPS Telecom, a manufacturer of semi-custom remote alarm monitoring systems based in Fresno, California. Andrew brings more than 17 years of experience building site monitoring solutions, developing intuitive user interfaces and documentation, and opt...
