As Your Admin Walks Out the Door ..
One of our readers (thanks Gebhard) mailed us a link to an article on what the press is apparently now calling a "Revenge Wipe" - a system administrator who has left the organization, and as a "last hurrah", deletes or locks out various system or infrastructure components.
In this case, the organization was a hosting company in the Netherlands (Verelox). In the case of cloud providers, a disgruntled admin may have access to delete entire networks, hosts, and associated infrastructure. In the case where it's a smaller CSP, the administrator may also have access to delete customer servers and infrastructure as well. In Verelox's situation, that seems to have been the case (from their press release at least)
The classic example of this is the City of San Francisco in 2008), where their main administrator (Terry Childs) refused to give up the credentials to their "FiberWAN" Network Infrastructure, even after being detained by law enforcement (he eventually did give the credentials directly to the Mayor). I've listed several other examples in the references below - note that this was not a new thing even in 2008 - this has been a serious consideration for as long as we've had computers.
So, how should an organization protect themselves from a situation like this?
Back up Job Responsibilities:
Know who has access to what. Have multiple people with access to each system. Having any system with only a single administrator can turn into a real problem in the future. DOCUMENT things. BACKUP your configurations in addition to your data.
Use Authorization:
It can be difficult, but wherever possible use Admin accounts with only the rights required. It’s very easy to build an “every Admin has all rights” infrastructure. It’s likely more difficult to build a “why does the VMware admin need the rights to delete an entire LUN on the San” config – but it’s important to think along those lines wherever you can.
Use a back-end directory for authentication to network infrastructure:
What this often means is that folks implement NPS (RADIUS) services in Active Directory. This allows you to audit access and changes during regular production, and also allows you to deactivate network administrator accounts in one place
Where you can, use Two Factor Authentication
Use 2FA whereever possible, this makes password attacks much less of a threat. 2FA is a definite "easy implement" for VPN and other remote access, also for administration of almost all Cloud Services for your organization.
Just as a side note - I am still seeing that many smaller CSPs have not gone forward with 2FA - if you are looking at any new Cloud services, adding Two Factor Authentication as a "must-have" is a good way to go.
Deal with "Stale" Accounts:
Keep track of accounts that are not in use. I posted a powershell script for this (targeting AD) in a previous story ==> https://isc.sans.edu/diary/The+Powershell+Diaries+-+Finding+Problem+User+Accounts+in+AD/19833
Deal with "Service Accounts":
Service accounts are used in Windows and other operating system to run things like Windows Services, or to allow scripts to login to various systems as they run. The common situation is that these service accounts have Domain Administrator or local Root access (depending on the OS).
Know in your heart that the person you are protecting the organization from is the same person who likely created one or all of these accounts.
Be sure that these service accounts are documented as they are created, so that if a mass change is required it can be done quickly.
Know that these use a central directory (such as AD or LDAP), so that if you need to change them or disable them, there is one place to go.
I posted a PowerShell script in a previous story to inventory service accounts in AD ==> https://isc.sans.edu/forums/diary/Windows+Service+Accounts+Why+Theyre+Evil+and+Why+Pentesters+Love+them/20029/
Restrict Remote Access:
Be sure that your administrative accounts don't have remote access (VPN, RDP Gateway, Citrix CAG etc). This falls into the same category as "don't allow Administrators to check mail or browse the internet while logged in as a Domain Admin or root privileges.
On the day:
On the day of termination, be sure that all user accounts available to our administrator are deactivated during the HR interview. If you've used a central authentication store this should be easy (or at least easier)
Also force a global password change for all users (your departing admin has probably done password resets for many of your users), and if you have any stale accounts simply deactivate those.
For Service accounts, update the passwords for all of these. This is a good time to be sure that you aren't following a pattern for these passwrods - use long random strings for these (L33t speak versions of your company or product name are not good choices here).
I'm sure that I've missed some important things - please, use our comment for to fill out the picture. This is a difficult topic, since many of us are admins for one thing or another this really hits close to home. But for the same reason, it's important that we deal with it correctly, or as correctly as the situation allows.
References:
https://www.schneier.com/blog/archives/2008/07/disgruntled_emp.html
https://www.wired.com/2016/06/admin-faces-felony-deleting-files-flawed-hacking-law/
===============
Rob VandenBrink
Compugen
Comments
Use Separation of Duties:
Not exactly the definition of Separation of Duties. Better, don’t have all your cookies in one cookie jar, especially if the cookie jar is about to depart the organization. Divide permissions so that one admin for installation and the DBA and Storage staff are not OS Admins nor are they backup admins. For the truly paranoid/superior provider, split super-critical functions to require “dual signing” – can’t wipe drives, backups or purge DB without two staff (i.e. an admin and an executive).
Use Authorization:
As noted above, split Admin/root authorization between different roles at different levels of organization. Also, Admins must have separate general account for use when email, etc. and a privileged account for use when administering. Have audit permissions separate from admin (i.e. Audit/syslog by ISSO, not Admins). This is also Separation of Duties.
Deal with "Stale" Accounts:
That is, deal with Ghost accounts – accounts that those with admin level privileges can create and HIDE. When an admin is about to leave is WHEN TO run scripts in AD to check for unattributed accounts AND PRIVILEGE ESCALATIONS (especially since admins have general and privileged accounts). Did the admin’s general login suddenly get promoted to Admin? GHOST ACCOUNTS search includes every local (console), network (e.g. AD) and remote access (VPN, etc.).
On the day:
Well, this depends if the departure is (1) planned and known, or if it is (2) sudden, and unexpected. If #1, then on giving notification, the admin account should be locked – the soon to depart can use their general account only. This can be a hit for organizations with few admins. But the cost could be worse in the example of being wiped. A slightly less obtrusive (but less effective) means is to turn on all auditing for everything the soon to depart is involved in (desktop included) without his or her knowledge. Of course, you do already have all use of administrative privilege and changes (escalations) to privileges already being audited on every system, right? (No? NOW is the time to engage that). Audit is the soft option – you may get hit, but you are going to have all the evidence you need ready for after event action (e.g. legal, criminal, etc.-and you are coordinating with Legal department to get their requirements too). And you do have all your audit logs dumped to a central syslog (that the departing has no access), right? Oh, and when it all goes to hell, your backups are tested and procedures are up to date, right? And you have all your configurations documented so anything that you must do a bare metal rebuild is at hand? And you should consider available login restriction – time, location, etc. for the period prior to departing. The admins might normally need to login at 1 a.m. for problems or 4 a.m. to do backups, but the soon to depart must only be allowed to login 9-5 at office (not from home or VPN).
/InfoSecurity.Master/
Anonymous
Jun 19th 2017
7 years ago
Anonymous
Jun 20th 2017
7 years ago
1. The larger issue was was the possibility of violence, one IT manager actually was a killer. Look for early help with threat assessment, e.g., from local chapter members of the Assn of Threat Assessment Professionals www.atapworldwide.org.
2. Even before the facts were confirmed it was necessary to action to determine if the subject was using privileges to snoop on the investigation. Log and review mail admin access to exec emails. Use keyloggers and review proxy logs to learn more about the subject's situation and intentions.
3. For high stakes interviews design the environment to minimize risk, allow the subject to leave easily, station a cop across the hall to manage blowups.
4. After the exit interview consider setting up a honeypot that the subject's old VPN account connects to. This can document malicious, and likely illegal, activity.
5. Be generous. It doesn't help anyone to rub the departing manger's face in the dirt. Offer a severance package and counseling assistance (which can report threats of violence).
Anonymous
Jun 20th 2017
7 years ago
As to #1, absolutely agree! Background checks (periodic also), and assistance from a variety of Threat perspectives (personal, personnel, etc. as you noted) should be taken in to account.
#2. This is a separate scenario, and lets call it the prequel. Here is where having ISSO role and permissions separate from IT and the Admins is critical. This brings up questions
-Do you have an Office of Inspector General (IG) or comparably qualified and competent staff? These are NOT your IT/Infosec forensics staff. In federal agencies, IG can be a “mini-FBI”. If not on staff (and possibly a separate department), get outside support. It may be a local or federal law enforcement depending on your organizations and the type of activities you are suspecting. And of course, your Legal department or other counsel is also usually required. But to your original assertion, (“if the subject was using privileges to snoop on the investigation”), SEPARATE THE DIFFERENT PERMISSIONS of admin level to several IT Admins and reserve highly critical permissions to the ISSO and CIO.
- Have procedures and specify roles for investigations (again, not talking to SANS style forensics only). This reiterates the need for an Information Risk Committee (e.g. “Risk Executive” per FISMA) basically comprised of the Executive Board/CEO, CISO, Internal Audit, IG/Legal (whichever you have). Define who is Accountable (authority), Responsible (executes), Consulted (provide opinion and guidance) in particular. Run your investigation here, not in the subject’s department. There are several contexts to "Risk Management" to address.
#3. Yes. #4 OK, maybe. Did I mention CHANGE ALL ADMIN and SYSTEM ACCT PASSWORDS?
#5. This is the prequel’s prequel. If you have enough suspicion, or if the subject gives the “two-week notice”, escort out immediately. Not after two weeks, not end of the day-Now. And “Do No harm”-pay for the two weeks or more, provide other support as you suggested. Dealing with personnel issues personally beats any IT activities by far. It could defuse any emotions for the subject as well as supports the organization’s reputation.
Anonymous
Jun 20th 2017
7 years ago