Using NMAP to Assess Hosts in Load Balanced Clusters

Using NMAP to Assess Hosts in Load Balanced Clusters
Recently I've been seeing more clients using DNS load balancing (GSLB) to load-balance traditional web, API and other services - it's an easy way to load balance between datacenters for instance. What I've noticed lately, especially with servers that use cloud "as a service" offerings, is that: Not all cluster members neccessarily have the same ports available Not all cluster members might even have the same web components installed If there isn't a TLS front-end (and we're seeing less of this over time), that means that certificates are now maintained per-server, which means that one or more can easily expire "under the radar" So, how do we work through this problem of "my DNS target is now multiple different hosts, each with their own IP", and add to that, now dozens or hundreds of other hosts (from other organizations) now reside on those same IP addresses? By default, nmap will only assess the first IP returned for the DNS query against your hostname. In fact, it comes right out and tells you that when this situation comes up: nmap -Pn -sT -v -p80,443,8443,9443 somehost.somedomain.com Starting Nmap 7.92 ( https://nmap.org ) at 2022-05-18 14:08 Eastern Daylight Time Initiating Parallel DNS resolution of 1 host. at 14:08 Completed Parallel DNS resolution of 1 host. at 14:08, 0.02s elapsed Initiating Connect Scan at 14:08 Scanning somehost.somedomain.com (1.2.3.4) [4 ports] Discovered open port 443/tcp on 1.2.3.4 Discovered open port 80/tcp on 1.2.3.4 Discovered open port 8443/tcp on 1.2.3.4 Discovered open port 9443/tcp on 1.2.3.4 Completed Connect Scan at 14:08, 0.08s elapsed (4 total ports) Nmap scan report for somehost.somedomain.com (1.2.3.4) Host is up (0.065s latency). Other addresses for somehost.somedomain.com (not scanned): 5.6.7.8 13.12.11.10 rDNS record for 1.2.3.4: some-rev-dns.some-other-domain.com PORT STATE SERVICE 80/tcp open http 443/tcp open https 8443/tcp open https-alt 9443/tcp open tungsten-https Read data files from: C:\Program Files (x86)\Nmap Nmap done: 1 IP address (1 host up) scanned in 0.23 seconds This is of course less than optimal, the workaround is to use the "--resolve-all" switch, so that each IP is scanned (as the dns name provided). Because of how web servers work, if you just scan the IP address you are quite often not scanning the service you think you are - in many cases for instance you might be "looking" at the apache or IIS default server rather than your customer's web server. nmap -Pn -sT -v -p80,443,8443,9443 --resolve-all somehost.somedomain.com Starting Nmap 7.92 ( https://nmap.org ) at 2022-05-18 15:20 Eastern Daylight Time Initiating Parallel DNS resolution of 3 hosts. at 15:20 Completed Parallel DNS resolution of 3 hosts. at 15:20, 0.04s elapsed Initiating Connect Scan at 15:20 Scanning 3 hosts [4 ports/host] Discovered open port 80/tcp on 1.2.3.4 Discovered open port 80/tcp on 5.6.7.8 Discovered open port 80/tcp on 13.12.11.10 Discovered open port 443/tcp on 13.12.11.10 Discovered open port 443/tcp on 1.2.3.4 Discovered open port 8443/tcp on 1.2.3.4 Discovered open port 443/tcp on 5.6.7.8 Discovered open port 8443/tcp on 5.6.7.8 Discovered open port 9443/tcp on 5.6.7.8 Discovered open port 9443/tcp on 13.12.11.10 Completed Connect Scan at 15:20, 0.14s elapsed (12 total ports) Nmap scan report for somehost.somedomain.com (5.6.7.8) Host is up (0.043s latency). rDNS record for 5.6.7.8: some-rev-dns-5.some-other-domain.com PORT STATE SERVICE 80/tcp open http 443/tcp open https 8443/tcp open https-alt 9443/tcp open tungsten-https Nmap scan report for somehost.somedomain.com (13.12.11.10) Host is up (0.044s latency). rDNS record for 13.12.11.10: some-rev-dns-13.some-other-domain.com PORT STATE SERVICE 80/tcp open http 443/tcp open https 9443/tcp open tungsten-https Nmap scan report for somehost.somedomain.com (1.2.3.4) Host is up (0.037s latency). rDNS record for 1.2.3.4: some-rev-dns.some-other-domain.com PORT STATE SERVICE 80/tcp open http 443/tcp open https 8443/tcp open https-alt Read data files from: C:\Program Files (x86)\Nmap Nmap done: 3 IP addresses (3 hosts up) scanned in 1.05 seconds Key things to note: All 3 hosts got scanned All 3 hosts got scanned as the target service somehost.somedomain.com Not all 3 hosts had the same services - when I looked at this closer, the missing services were API interfaces, which were only fully implemented on the "primary" host. You might think that this is a pretty big "oops", but the load balancer might know about that, and will only load balance 8443 between the correct 2 hosts and 9443 between the correct (other) two hosts for that service. It's tough to say without a failover to test it, and that's all you can suggest to the client - "you should test the failover for those two services" Using this same switch with nmap scripts can also be useful, especially the "SSL-" family of nmap scripts. ssl-cert and ssl-enum-ciphers in particular should always give you the same results for each cluster member. "Should" being the operative word! Have you found some missed configs or "housekeeping opportunities" when scanning load balanced clusters? If your NDA permits, please share any war stories using our comment form! =============== Rob VandenBrink rob@coherentsecurity.com	Rob VandenBrink 579 Posts ISC Handler May 25th 2022
Thread locked Subscribe	May 25th 2022 1 month ago
Thank you for a great writeup.	Anonymous
Reply Quote	Jun 4th 2022 4 weeks ago