Even in the Quietest Moments ...

Published: 2013-12-03. Last Updated: 2013-12-03 20:16:05 UTC
by Rob VandenBrink (Version: 1)

I recently had a migration from one internet uplink to another to do for a client. As with many organizations, they have about 40% of their workforce at head office, and 60% (and sometimes more) of their workforce operating remotely, so taking the Firewall and especially the VPN services offline is a very big deal. There is no good time to take things down given that their sales force has people in just about every time zone, there are just times that are "less bad" than others.

Anyway, we settled on a weeknight, starting at around 6pm EST after their shipping was completed. As in most projects of this type, we finally got most of the "important" users off the network by about 7:30 and where ready to start.

On logging into their firewall, I (as I always do), did a quick check of the real time logs, just because you never know what you'll see. Imagine my surprise when I saw that it was still pretty busy, and the traffic being logged was mostly this:

So what is that? Their internal workstation network is 192.168.122.0/24, a nice respectable RFC1918 address range. However, we're seeing lots of outbound requests for 192.168.1.0/24, which I would expect to see more on home networks. Now I could see a simple, persistent series of requests to a single 192.168.1.x address, perhaps a home printer that is a user's default, or a home DLNA server, but this looks a whole lot like reconnaissance doesn't it? It's a sweep of the entire 192.168.1.0 network, using ICMP. Looking deeper at the packet, as you'd expect these are echo requests (Type 0, code 0), otherwise known as plain old "ping".

So, what's so interesting about this you ask?
My answer would be - recon of a network that's the default subnet for many home network routers is always suspect in an enterprise. In fact, recon of any type in an enterprise network should be considered suspect. People connect to what they need for to do their jobs, network sweeps are almost always an indicator of compromise, it's almost always malware looking for something else to infect, or an attacker looking for their next foothold. Happily, if the network being scanned isn't in the internal routing table, if it's not black-holed internally it usually shows up in the firewall logs.

In this case, it was malware looking for PNP devices (printers, cameras, TVs, but especially home firewalls), which Johannes wrote up in January ( https://isc.sans.edu/diary/Exposed+UPNP+Devices/15040 ). Even though it's what you might consider "old", it sees sustained use ( https://isc.sans.edu/port.html?port=1900 ) in malware and sees continued success, mostly becuase almost nobody patches or fixes their home routers.
It could also just as easily have been malware looking for default home router credentials or one of the home router backdoor vulnerabilities (which Manuel wrote up here https://isc.sans.edu/diary/Old+D-Link+routers+with+coded+backdoor/16802 )

It was just lucky that I caught this activity on film, the network was so quiet that the malware activity just popped up without looking for it. It's funny that when you have a quiet network, sometimes all that's left is the malware and attack traffic.

Of course, when we did a reverse lookup on the workstation IP, it was their HR manager. You know, the HR manager who insisted that we remove the internet filters from their account so they could be active on social media? No risk at all having malware on that station !!

But hopefully, this should illustrate why it's so important that part of your day-to-day tasks to secure your organization should be to look at your logs. And not just glancing at the logs scrolling by on your firewall - review the text logs that you store in disk, that's where you'll find those "gold" log entries. Make friends with the grep or findstr commands and do some mining for malware, it'll be the most productive time you spend most days! Heck, just looking at a directory of your logs by day is often enough - if yesterdays logs was 10 times the size you normally see, often that's an IOC (Indicator of Compromise) all on it's own. Relying simply on tools to alert you of problems and not looking at your logs is passing up the easiest and earliest detection of problems you'll ever get. As we say over and over - "there's gold in them thar logs!"

Even reviewing the logs of your home network can give you valuable information ( https://isc.sans.edu/diary/Collecting+Logs+from+Security+Devices+at+Home/14614 ). You'll find the same indicators of compromise or attack in home logs as at work, and if nothing else, it's good practice for reviewing logs for enterprise gear - home gear and enterprise gear all log the same things, if they're configured correctly that is...

By the way, did you catch the other problem that our captures showed? Yes, the NTP server that the firewall was synced to had been retired, so the date and time on the firewall was completely out of whack. If you've ever tried to correlate logs from multiple systems, perhaps to get a complete picture of how a compromise happened or what it did after it got it's toe-hold, you'll appreciate just how big a deal syncing your clocks is.