A Taste of Our Own Medicine

Submitted by Brian Noone, Network Systems Manager for NetScout Systems

As a network manager, don’t you just hate when someone complains about network performance, but the problem disappears as soon as you start looking for it? Here at NetScout even we run into that type of issue.

One afternoon this summer, we began receiving complaints from a group of engineers that their Internet connectivity was getting slow at sporadic intervals. In order to isolate the problem, the network team logged onto nGenius Performance Manager and accessed the specific probe that was monitoring the affected network segment. The data from the probe was able to show us that there were indeed drops in network utilization appearing at regular intervals – approximately 30 seconds worth every 30 minutes – but it did not really provide us with sufficient information to isolate the root cause.

We attempted to schedule a packet capture using the existing nGenius probe, but missed on the timing and had to wait for the next recurrence. After missing the problem timeframe twice, we decided to pull out the “big guns”. In other words, put an nGenius InfiniStream on the network segment in question. 

The nice thing about the nGenius InfiniStream is that it continuously captures and records up to 15TB of packet data while also providing robust data mining to help winnow the timeframe to the precise moment required – in this case, 60-seconds worth of packets. By analyzing the packet decode, we were able to determine that a desktop was spoofing the IP of the router at 30-minute intervals and causing ARP poisoning. Although the invalid ARP entry was cleared out within 30 seconds, the users on the network segment still noticed the loss of connectivity.

Once the network team isolated the MAC address of the system, we took the system off the network and had our desktop support team investigate. Over the next hour or so, using the data provided by the InfiniStream, we identified and isolated three additional systems that were exhibiting the same malicious behavior.

Meanwhile, the desktop support team determined that all four systems were infected with malicious code that manipulated or replaced the arp.exe file but had not been detected by the company’s antivirus software. The malicious code was also generating unsolicited connections to Internet sites that were capable of causing even more trouble.

As a last step, the network team took two preventative measures: We set alerts within nGenius Performance Manager that would trigger notifications of systems attempting to connect to the identified websites and we also blocked any related traffic with our existing security system.

Fortunately, NetScout’s network has been running smoothly ever since; no Performance Manager alerts have been generate and network performance and user experience remains satisfactory.

Have any of your own stories to share with us?