Threat Level: green Handler on Duty: Tom Webb

SANS ISC: Are You Protecting your Backdoor ? - Internet Security | DShield SANS ISC InfoSec Forums


Sign Up for Free!   Forgot Password?
Log In or Sign Up for Free!
Are You Protecting your Backdoor ?

Hardly anybody has physical access to critical public facing servers. Usually, they are located in a data center, hours away from the system administrators charged with managing them. Doing a system reboot, or trouble shooting a system that has dropped offline, is accomplished by either calling some third party support person at the data center, or via remote consoles. Neither solution is perfect.

Using remote "hands" often requires that these individuals will need access to root/admin passwords, or other credentials used to access these systems. Managing these credentials isn't easy, in particular "under stress" if you systems are down. To get things done, bad decisions are made in a rush if no plan was defined ahead of time. 

The alternative (and my preferred option) is a solid "out-of band" management solution for the systems. This typically includes some form of remote power controller and a "KVM over IP" solution. Not sure if anybody still uses dialup lines to access these consoles, but of course, there should be some secondary IP connection to reach them. There are a number of different vendors that offer solutions for this, but in my experience, many of them suffer from some basic flaws:

Logging: No half way security conscious system administrator would turn off logging for failed and successful login attempts. However, many of these remote access solutions do not even have a logging option, and if they do it is often not configured. Many loging features only allow for limited loging on the device itself, and do not allow to simply send the logs to a syslog server. SMTP traps exist in some cases, but again have to be enabled. I can't remember seeing anything that I would consider "robust" logging, like logging over SSL, from any KVM or RPC device.

Access Control: Sometimes, you find features like RADIUS, which at least allows for some central management of credentials. But remember, you need to be able to use these systems if "things don't work right". So often it is necessary to have local credentials. Better systems at least allow for multiple users and offer different privilege levels. But in some cases, the user can not change their password after the administrator set it for them. This requires the device's administrator to communicate the password to the user which is less then ideal.

Out of data Java applets: Many modern KVMs use a Java applet to provide remote access. These applets are often signed with expired, or self signed, certificates. Newer versions of Java will not allow you to run these applets, so you need to keep an older version around just in case. Aside from having to accept the invalid signature, this puts your system at risk if this older Java version is vulnerable. Some vendors provide updates for the firmware fixing the Java signature issues, but this will only help you as long as your device is still supported.

SSL Configuration: It is usually possible to load your own SSL certificate, which is a minimum requirement to obtain a reasonable SSL connection (you don't want to use the default configuration!!!). But I have seen devices that do only allow MD5 signed certificates. No real certificate authority will provide them anymore due to the documentd weakness of MD5. It is possible to setup your own CA (and sometimes a good idea in cases like this), but you still probably should not use MD5 signed certificates. Aside from that, many of these devices still max out at SSLv3, which is less and less supported. 

So how do you control these "backdoors"? Any great ideas on staying up to date with patches, and limiting/logging access to them? I didn't even mention simple firmware vulnerabilities like authentication bypass that plague these devices.

---
Johannes B. Ullrich, Ph.D.
STI|Twitter|LinkedIn

I will be teaching next: Defending Web Applications Security Essentials - SANS Munich July 2019

Johannes

3532 Posts
ISC Handler
why dont you forget about that KVM with the out of date, not-signed java applets?

Why not putting the admin interfaces behind a firewall and only allow 1 (Jump)server to administer these. That server requires 2FA login, with client side certificate authentication. The server itself only allows the login port (RDP)
Peter B

1 Posts
The problem with firewalling these devices is what if the firewall needs remote administration? In other words, if that firewall goes down, then you can't get to your power switches or KVM over IP devices. Its better than nothing, but still leaves a problem spot.
Moriah

133 Posts
I think a better solution might be a Linux box with a frame grabber so it could capture the video out of the server(s), and appropriate means to transmit keyboard and mouse signals as outputs, so they could be inputs to the managed device. At least that way, you would have a box that was routinely updatable.
Moriah

133 Posts
Most servers already have a built-in "out-of-band" solution. Almost all models support SOL (serial over LAN) via IPMI; which provides a text console across network. This is good enough for Linux, and Windows support this via SAC (Special Administrative Console). Some OS configuration is required.

Newer models usually have built-in "Light-out-management" (LOM), where you can view the graphical console and remotely cold reset, power on and power off. Different vendors have different names, iLO for HP, DRAC for Dell, and RSA for IBM. Some people are unaware of the feature and purchase a KVM over IP solution.

For Dell DRAC, this is a web interface with default login credentials. So do take note and change the defaults.
There is even a python tool to automate the hacking via this method as per https://www.trustedsec.com/september-2012/owning-dell-drac-awesome-hack/
Mike7

42 Posts
Our rule of thumb is to have a non-routable subnet in each datacenter specifically for things like DRACs BMCs IPMI interfaces, etc. Then we'll have a gateway system that has one NIC in this non-routable subnet and another NIC in some "normal" inside network segment. Access to a DRAC or BMC means opening an ssh tunnel or running a vnc client to connect to this gateway system. Monitoring systems (like nagios/cacti) can have a NIC in these subnets for the purpose of monitoring chassis health (via DRAC/IPMI/whatever).

Provided one keeps the gateway systems secure, it raises the bar to others accessing these often not patched enough devices.
Brent

118 Posts

Sign Up for Free or Log In to start participating in the conversation!