Where is my data? When hosting providers go away

Where is my data? When hosting providers go away
Most of us host part or maybe even all of our infrastructure at hosting providers. They provide you with floor space, rack space, or in cloud environments with platforms and software for you to use. As with all of these solutions there are pros and cons to having your hardware hosted. In cloud environments the hardware and often software typically belongs to the provider and only the data belongs to you. What could go wrong? As security professionals we get to discuss the risks of these kinds of arrangements and most of us will raise the risk of the provider going south or the data being unavailable for other reasons. The answer we often get is along the lines of “oh that never happens and we have backups”. Unfortunately that doesn’t always help and losing data isn’t the only issue as has been aptly demonstrated this week when a number of datacentres Belgium and the Netherlands closed up shop. http://tweakers.net/nieuws/90104/belgische-tak-datahouse-is-failliet-verklaard.html http://datanews.knack.be/ict/nieuws/datahouse-belgium-failliet-verklaard/article-4000351627180.htm http://www.ispam.nl/archives/33702/datahouse-belgium-failliet-verklaard-op-verzoek-van-scarlet-business/ http://www.intall.nl/onderwerp/2818-Datahouse_Belgie_failliet_Is_Datahouse_NL_de_volgende In a nutshell the provider was declared bankrupt, the doors closed and connections were cut. As the articles state customers were denied access to their servers whilst the bankruptcy processes were established. In a number of cases connectivity to servers was cut, denying access to the data. So what risks are there when a hosting provider goes bust? Denied access to physical servers – In many hosting situations the line between who owns what is difficult and often physical access will be denied until ownership can be demonstrated. In the mean time you may have expensive equipment sitting in a datacentre that you can no longer access. Denied access to data (internet) – This can happen a number of ways. The Internet connection may be removed which obviously cuts your and anyone else’s access. Sometimes machines are shut down. Whilst many administrators may decide to keep things running, after all earning some money is better than none, to cut cost supporting services may be reduced and if something breaks it is unlikely to get fixed. Denied access to data (Local) – You may decide to go pick up your data, but getting access may not be that easy. So unless you can retrieve it remotely you may have to kiss that good bye. Backups – Any backups taken by the hosting provider are unlikely to be accessible. Depending on the systems used to manage backups it may be quite a task to get them. Even if you get the physical tapes (if used) you are unlikely to get the backup catalogue, so retrieving data will be difficult. Disclosure of data – Physical access usually trumps most of the controls many of us place on our hosted environment and in the cloud we do not have any control. So it is quite likely that you will not be able to deny access of third parties to your data. Least of all you will be left with the cost of moving operations to an alternate location and as most of use who have been involved with datacentre moves know that is not a trivial task. It would be mean to just leave it there, so what can be done about this to mitigate the risks? Denied Access – If denied access to your servers or data a DR environment is probably your best bet. Being able to run up services elsewhere provides processing capabilities whilst lawyers sort out getting physical assets back. However tempting it may be it is probably not a great idea to have the production environment and your DR environment hosted by the same organisation. Backups – Make your own. Do not rely on the hosting provider to do all the backups. Alternatively make sure that backups are stored elsewhere, including the catalogue so you can readily identify the data on the tapes, if needed. Disclosure of data – This is probably the most difficult one and makes you wish that the mission impossible slogan (this message will self-destruct in …) is an actuality. Not many of us are in the habit of full disk encryption on servers, but that may be the only way and won’t help in a PAAS or SAAS situation. Some of you may have been in this situation and others can no doubt learn from your experience so if you are able to I’d love to see your experiences or additional risks and controls I may have missed. Mark	Mark 391 Posts ISC Handler
Subscribe	Jul 20th 2013 6 years ago
"Do your own computing on your own computer with your copy of a freedom-respecting program. If you use a proprietary program or somebody else's web server, you're defenceless. You're putty in the hands of whoever developed that software." -- Richard Stallman	Moriah 133 Posts
Quote	Jul 20th 2013 6 years ago
Don't forget about the experiences of the tens of millions of Megaupload's legitimate data storage customers: 2012: http://www.wired.com/threatlevel/2012/06/feds-megaupload-data/ "Federal authorities say they may shut down cloud-storage services without having to assist innocent customers in retrieving data lost in the process." 2013: http://www.slashgear.com/megaupload-loses-petabytes-of-data-as-euro-host-pulls-plug-19287093/ It does pose an interesting consideration for e-discovery. If you are compelled to turn over data during a lawsuit but the government already confiscated it for unrelated reasons and then facilitated its destruction, did the government cause the spoliation and taint the case? And if the government is a party to the case, did their confiscation of the data for another reason also violate the discovery order for the case in question? Because the confiscation may be considered overly broad or even give them access to attorney-client confidential documents of the other party.	Anonymous
Quote	Jul 20th 2013 6 years ago
If you happen to have a functional backup and can move your operation to another hosting service, it may be obvious to change all passwords, but frequently that is overlooked in the tense moments of moving your servers.	benbrandbjm.com 5 Posts
Quote	Jul 20th 2013 6 years ago
As stated above, hosted (cloud) data/services solution introduce a new set of technical and security challenges. However, it is more of a trade vs. simply a new set of added challenges. In a non-hosted environment, you have layers upon layers of hardware and infrastructure challenges to deal with. Server migrations, physical security, environmental controls, etc. Miss one and you could very well have downtime. For example, a huge in-house datacenter/server room was brought to its knees (at my previous employer) because of a faulty air conditioning system. Using a hosted/cloud service transfers much of the technical, physical, and environmental risks to the hosting provider. However, the new set of challenges includes cloud security and the availability of your data should the provider experience a problem. With hosting, it's extremely important to focus on the CIA triad with respect to your data now that you don't have a massive server room or datacenter to maintain. I don't see the "cloud" as an improvement or enhancement of any kind. I simply see it as an alternative to building and maintaining your own server room or datacenter. Whether you out-source it (cloud) or do-it-yourself, the same quantity of challenges and risks exist... they are just different.	da1212 69 Posts
Quote	Jul 23rd 2013 6 years ago

Most of us host part or maybe even all of our infrastructure at hosting providers. They provide you with floor space, rack space, or in cloud environments with platforms and software for you to use. As with all of these solutions there are pros and cons to having your hardware hosted. In cloud environments the hardware and often software typically belongs to the provider and only the data belongs to you. What could go wrong?

As security professionals we get to discuss the risks of these kinds of arrangements and most of us will raise the risk of the provider going south or the data being unavailable for other reasons. The answer we often get is along the lines of “oh that never happens and we have backups”. Unfortunately that doesn’t always help and losing data isn’t the only issue as has been aptly demonstrated this week when a number of datacentres Belgium and the Netherlands closed up shop.

http://tweakers.net/nieuws/90104/belgische-tak-datahouse-is-failliet-verklaard.html

http://datanews.knack.be/ict/nieuws/datahouse-belgium-failliet-verklaard/article-4000351627180.htm

http://www.ispam.nl/archives/33702/datahouse-belgium-failliet-verklaard-op-verzoek-van-scarlet-business/

http://www.intall.nl/onderwerp/2818-Datahouse_Belgie_failliet_Is_Datahouse_NL_de_volgende

In a nutshell the provider was declared bankrupt, the doors closed and connections were cut. As the articles state customers were denied access to their servers whilst the bankruptcy processes were established. In a number of cases connectivity to servers was cut, denying access to the data. So what risks are there when a hosting provider goes bust?

Denied access to physical servers – In many hosting situations the line between who owns what is difficult and often physical access will be denied until ownership can be demonstrated. In the mean time you may have expensive equipment sitting in a datacentre that you can no longer access.
Denied access to data (internet) – This can happen a number of ways. The Internet connection may be removed which obviously cuts your and anyone else’s access. Sometimes machines are shut down. Whilst many administrators may decide to keep things running, after all earning some money is better than none, to cut cost supporting services may be reduced and if something breaks it is unlikely to get fixed.
Denied access to data (Local) – You may decide to go pick up your data, but getting access may not be that easy. So unless you can retrieve it remotely you may have to kiss that good bye.
Backups – Any backups taken by the hosting provider are unlikely to be accessible. Depending on the systems used to manage backups it may be quite a task to get them. Even if you get the physical tapes (if used) you are unlikely to get the backup catalogue, so retrieving data will be difficult.
Disclosure of data – Physical access usually trumps most of the controls many of us place on our hosted environment and in the cloud we do not have any control. So it is quite likely that you will not be able to deny access of third parties to your data.

Least of all you will be left with the cost of moving operations to an alternate location and as most of use who have been involved with datacentre moves know that is not a trivial task.

It would be mean to just leave it there, so what can be done about this to mitigate the risks?

Denied Access – If denied access to your servers or data a DR environment is probably your best bet. Being able to run up services elsewhere provides processing capabilities whilst lawyers sort out getting physical assets back. However tempting it may be it is probably not a great idea to have the production environment and your DR environment hosted by the same organisation.
Backups – Make your own. Do not rely on the hosting provider to do all the backups. Alternatively make sure that backups are stored elsewhere, including the catalogue so you can readily identify the data on the tapes, if needed.
Disclosure of data – This is probably the most difficult one and makes you wish that the mission impossible slogan (this message will self-destruct in …) is an actuality. Not many of us are in the habit of full disk encryption on servers, but that may be the only way and won’t help in a PAAS or SAAS situation.

Some of you may have been in this situation and others can no doubt learn from your experience so if you are able to I’d love to see your experiences or additional risks and controls I may have missed.

Mark

391 Posts
ISC Handler

"Do your own computing on your own computer with your copy of a freedom-respecting program. If you use a proprietary program or somebody else's web server, you're defenceless. You're putty in the hands of whoever developed that software." -- Richard Stallman

Moriah

133 Posts

Don't forget about the experiences of the tens of millions of Megaupload's legitimate data storage customers:

2012: http://www.wired.com/threatlevel/2012/06/feds-megaupload-data/

"Federal authorities say they may shut down cloud-storage services without having to assist innocent customers in retrieving data lost in the process."

2013: http://www.slashgear.com/megaupload-loses-petabytes-of-data-as-euro-host-pulls-plug-19287093/

It does pose an interesting consideration for e-discovery. If you are compelled to turn over data during a lawsuit but the government already confiscated it for unrelated reasons and then facilitated its destruction, did the government cause the spoliation and taint the case?

And if the government is a party to the case, did their confiscation of the data for another reason also violate the discovery order for the case in question? Because the confiscation may be considered overly broad or even give them access to attorney-client confidential documents of the other party.

Anonymous

If you happen to have a functional backup and can move your operation to another hosting service, it may be obvious to change all passwords, but frequently that is overlooked in the tense moments of moving your servers.

benbrandbjm.com

5 Posts

As stated above, hosted (cloud) data/services solution introduce a new set of technical and security challenges. However, it is more of a trade vs. simply a new set of added challenges. In a non-hosted environment, you have layers upon layers of hardware and infrastructure challenges to deal with. Server migrations, physical security, environmental controls, etc. Miss one and you could very well have downtime. For example, a huge in-house datacenter/server room was brought to its knees (at my previous employer) because of a faulty air conditioning system.

Using a hosted/cloud service transfers much of the technical, physical, and environmental risks to the hosting provider. However, the new set of challenges includes cloud security and the availability of your data should the provider experience a problem. With hosting, it's extremely important to focus on the CIA triad with respect to your data now that you don't have a massive server room or datacenter to maintain.

I don't see the "cloud" as an improvement or enhancement of any kind. I simply see it as an alternative to building and maintaining your own server room or datacenter. Whether you out-source it (cloud) or do-it-yourself, the same quantity of challenges and risks exist... they are just different.

da1212

69 Posts