Threat Level: green Handler on Duty: Remco Verhoef

SANS ISC: Cyber Security Awareness Month - Day 19 - Remote User VPN Access – Are things getting too easy, or too hard? - Internet Security | DShield SANS ISC InfoSec Forums


Sign Up for Free!   Forgot Password?
Log In or Sign Up for Free!
Cyber Security Awareness Month - Day 19 - Remote User VPN Access – Are things getting too easy, or too hard?

It seems lately to me that in IT  we no longer seem to have downtime, even in traditional "9 to 5" companies.  Laptops, smartphones, iPads and every other gadget out there all are internet connected, and more and more people seem to be online every waking moment.  And if they’re online, chances are they’re VPN’d in to keep tabs on things at work while they’re surfing social sites, playing flash games or whatever.  This is especially true now that VPN access is so easy, in fact it's now included in a number of smart phones and tablets.

Which brings us to the poor folks in IT.  Since everyone is online 24-7, and we’re seeing business sales offices or business partners from 12 timezones over with VPN connections in, this brings up a whole raft of problems:

When exactly can we do system maintenance?   I’m tired of waking up at oh-dark-early, only to find 6 users logged that you need to track down before you can start an upgrade.  You can’t seem to pick any time as a maintenance window without causing someone a problem. Who gets access to what.  All too often people have skipped over the data classification and server zoning steps.  Without those done, just exactly what is that business partner allowed to have when they’re VPN’d in?


The prevalence of cheap laptops, tablets, phones and electronic doo-dads, all with internet access and VPN access (especially now that we have SSL VPNs) seriously starts to blur the line as to what the corporate desktop is.  Worse yet, it blurs the line over who has bought and paid for that corporate desktop.  No matter what our policies say, we have way too many personally owned devices out there that have VPN access to corporate resources, but don’t have corporate security tools, logging or, well, anything else.  But you can bet they’ve got malware on them from the kids in the family ! (or the grown-up kids).  And just exactly how do you enforce a VPN policy and deny access to someone who wants to work after hours for free?  It’s a real challenge to make that point to a senior manager.


We’d really like to hear about any challenges you have faced on the topic of VPN access, and how you have solved them.  Even if in your view you lost the battle on one issue or another, please share – someone else may have a different approach that might help you out.   As always – our comment form stands ready to field any and all comments, questions and answers !

 

=============== Rob VandenBrink Metafore

Rob VandenBrink

489 Posts
ISC Handler
Your maint. windows do not actually need to be affected by VPN - any more than your carriers allow you to affect theirs. When AT&T decides that a POP router is getting an upgrade... it gets one. The provide notice etc, but... it gets one.

Likewise, having a pile of users on at 2am is not an issue, provided that they were given notice that "2am is MY time." Personally, we generally don't do 2am maint - we use daytime maint. outages to exercise various disaster recovery facets. Aside from file-shares (where dumping a server would cost someone a bunch of edits), half the time we don't announce a darned thing - just take the server away, and see how people (and systems, and human processes) react. If all goes well, no customer will notice. And with mission crit systems... most employees won't notice, either.
Steven

42 Posts
S,

I can understand your thinking, but at many organizations that kind of IT BS just doesn't fly, period. sounds like you work in a very IT friendly environment.

you wrote:
>>"Your maint. windows do not actually need to be affected by VPN - any more than your carriers allow you to affect theirs."<<

here, we are not even allowed to update DNS entries without a change request form being filled out and a time window for said change to be submitted for approval. It must be nice to be able to just tell your users "tough luck". we are not allowed to do that here.

in our situation I have found the best solution is to work within the confines of our system and in doing so I have gotten upper management to agree that a time window with 48 hours notice is generally sufficient to prepare end users for any downtime they might experience due to the IT change request.

we submit the request which states how long the planned operation will take, which business units will be affected, and to what degree. details a backout plan in case the operation fails and a few other minor informational items. this gives our management and users exactly the information they need to understand what will be down and when it will be down. it is subject to management approval. if some departments are too busy we are asked to schedule a different time.

sometimes frustrating, but overall, working WITH our users and getting them to work WITH us has really been a huge benefit to everyone involved in any IT changes or upgrades.
Blagarswinth

23 Posts
IT "BS"?

You're being a tad myopic. This is a life safety environment. We spend a pile of effort to make sure things work when other things fail. This has nothing to do with "IT Friendly". It is all about "Process Integrity".

Last I checked, reality doesn't care about "convenience". Servers fail, PRIs get cut, and old ladies have strokes while driving cars, and drive those cars through the wall of your building, into your switch room. They do not schedule these events, either.

And somewhere in that reality, if you're going to label something "mission critical", and call it "High Avail", then you really need to live up to that expectation. And that means users need to live up to that expectation, too. I think the error in your perception is that you expect some major service or process interruption during these maint. windows. You're mistaken; as I said, for the easy stuff, the customer doesn't notice; for the important stuff, the user doesn't notice, either.

Repeat after me: "Process Integrity". And then repeat the 2nd mantra: "A DR process must be exercised."

You cannot get more "working WITH the users" than that. The IT systems exist for the sake of the Process, not the users. And guess what - those users exist for the sake of the Process, too. If the user cannot maintain Process Integrity during an outage of some sort? Then they are as defective as anything else.

And as I said, you cannot get more "working WITH users" than that.
Steven

42 Posts
um yeah. ok.

the issue here is not what happens when there is an UNEXPECTED emergent situation with your IT infrastructure. the issue being discussed is PLANNED maintenance and the availability of systems 24/7 because thats how people want to work...not because you promised high availability.

you've completely confused the issues and crossed into an entirely different realm when you start talking about disasters like a car driving into the switch room. where did you get that?.

there is no error in perception here. sometimes there ARE major interruptions when doing IT upgrades...you have to know that. as I said...IT friendly...your IT people appear to have much more say in what gets worked and when (provided its not an emergency) than we do here.

i dont mean to make this an argument but you seem to think you can just cookie cut some IT solution and plug it into any company.

all I am getting at here is that we can not just take down a server for maintenance whenever we want simply because we informed the users we were going to do so. it must be nice to be able to do that since its apparently what you do where you work. bring that crap over here and you wont last 2 weeks ok? your system WONT work here in this organization. kk thanks for playing.
Blagarswinth

23 Posts
I have to say I agree with S here. In a "mission critical, high availability" environment, taking one server down should NOT cause any inconvenience to the users, since there should be monitoring, automatic failover, and automatic recovery when the system comes online. You should literally be able to pull the plug on a critical machine and have another system able to automatically take over the role of that machine, including taking its IP if necessary, almost instantly. This is why there are clustering systems like Linux-HA. I think that's what S is getting at here.
Blagarswinth
4 Posts
please tell me where i can apply for a job in the IT utopia you guys live in. must be nice.
Blagarswinth

23 Posts
please tell me where i can apply for a job in the IT utopia you guys live in. must be nice.
Blagarswinth

23 Posts
You're showing a lack subtle of experience.

I got that story from a call I was on. After we finished cutting her out of the car, I noticed that there wasn't a single TBU mixed in with the pile of fine Dell logos that we were standing on. I then pawed through many layers of crap to find anything to salvage before we foamed over the gasoline that was soaking into the everything. Go figure, the guy had a dozen Raid5 arrays, so he "didn't need a backup." He had no plan for those arrays being scattered across the floor by a car moving at 30mph, or the building catching fire, or the pipes feeding the toilet on the next floor up being sheared (and I mean *all* the pipes). Too bad - it was the end of his business, and eventually had to sell his house as well. But hey! No time for downtime, users needed it 24/7! Website Too Critical! Users! kk thx for playing! Look at me! Too Critical! Users more important than the GOAL we're trying to achieve!!

I should point out that our 600 call per year district handles about 3 car-buildings per year. Bigger districts will do more, they are considered a routine call. The fact that you have no concept of this? Is significant, and you should consider what that fact means.

As far as confusing issues - no. Outage is outage. Doesn't matter if by car or by keyboard - the only difference is the scope of impact, level of pre-plan, and response.

Losing a single server should not create a crisis. If it does, then that setup isn't adequate for the risks involved. Period. q.v. "single point of failure". If you choose to hedge on a failure not taking place... you're gaming the odds, which means cost (or more often, lack of engineering skill) is more important than uptime. Period. When you eventually lose the bet? it cannot be a crisis - you accepted the risk, earlier. Period.

Because really - if uptime actually matters, your first priority is to make it possible. If you can't, then you hire someone who can. And if you can't afford them, THEN you consider taking risks. Right? But at that point, "uptime" is no longer the top dog - "Risk" is.

So as I said - in an environment where process integrity actually matters, maint. windows are trivial. It's almost as if we'd planned for them, or something, instead of reacting to them AFTER we turned everything on.

Another error - there should be no such thing as an UNEXPECTED emergent situation. We use words like "eventualty" in this field. Learn that word. It forms the backbone of proper computer and network and software and process engineering. And once you learn that word, you discover that "Reboot Wednesday" is Yet Another Eventuality, just like Vacations, Earthquakes, Death, and Taxes. You can even accomplish such goals as rebooting a server without people noticing, it's amazing!

With the "eventuality" culture comes the attitude that Excuses are not Results. If your stuff is so important that it must not go away... yet it CAN go away? You're doing it wrong. And seriously - nobody cares why. If it's that important? Hmm... Excuses... or Results. Pick one.

Since your users will kill you if you dump a single box... well, we can guess that you're not after Results as far as Uptime is concerned. Kind of makes you wonder how Google manages all of their machines, cheaply, without asking your permission before they maintain them. Magic!

Utopia starts with you, figuring out what your priorities are, and not confusing them (uptime vs hedge as top priority, for example.) And as I said - playing the hedge is a totally legitimate tactic - but it is not about uptime, it is about odds. And once your goals are clear, you can actually get Results.
Steven

42 Posts
wonderfully patronizing email. i wonder. is there anyone smarter or better than you on the planet?

seriously. i KNOW all the stuff youre saying.

now...bring all that over to my CEO and see how fare you get with any of it.

the issue you are not considering here...***the decisions are not MINE to make***

if they were then your arguments might be valid. so...i say once again. enjoy your utopia because many of us dont get to live in that wonderful little world of yours.

eventually...unexpected....semantics. give me a break. because you use the work eveutally rather than the word unexpected its somehow a different situation? wow.
Blagarswinth

23 Posts
S wrote:
The fact that you have no concept of this? Is significant, and you should consider what that fact means.

really? you know me? you know something...ANYTHING about what i have an understanding of? you know where i work and how large my organization is? what we do? what we deal with on an annual basis as far as calls go? hrm...you must be good.

and what does your example of the car have to do with anything? again...youre all up in arms about redundancy and backups and what??? thats not even what we are talking about.

"the only difference is the scope of impact, level of pre-plan, and response."

gosh....the level of pre-plan, scope of impact, and response....IS what makes those situations different...wow.

you're just making way too many assumptions based on knowledge about our situation at this company and my level of expertise that you know nothing about.
Blagarswinth

23 Posts
final thought.

you sound very much like a millionare telling a homeless guy on the street...

"you should really check into a motel and have a shower...clean up and eat at Ruth's Cris every night. I am sure you will enjoy life more and be more healthy."

to which the homeless guy would no doubt reply...

"thanks for the advice, a-hole...you think i LIKE eating rotten banana peels for dinner?"

I really hope you get the analogy so i dont have to actually call you an a-hole for you to understand.
Blagarswinth

23 Posts

Sign Up for Free or Log In to start participating in the conversation!