Questions about cloud storage

I have another bizarre job that is over my paygrade. 70 high end cameras over 7 locations for a government facility in Tobago. No problems with the basic system except for the final requirement. They are asking for 12 months video storage, off site. My suppliers are suggesting a LARGE server set. For that long storage, requires somewhere over 500TB. 25K. Is this an application that could be securely handled by utilizing a cloud? I have no knowledge about the cloud system. Any advice out there? Security is very important here.

You can't trust the cloud....

Hey, I'm really sorry the cloud ate your data.

As long as they have the bandwidth to get it offsite, the answer is yes.

It's not video, but I'm doing the same thing on a much smaller scale for my wife's veterinary clinic.

It's got two backup schemes running for her practice management system. One to a local network attached drive array, and a second that uses an application called Crashplan that gives you the choice of sending an encrypted backup out to their servers for a fee (that they probably contract from someone else) or, free of charge, I can use the application to backup the data on another NAS at my house.

So yes, it's possible, but I can't give you much advice for the scale you are looking at, other that taking a look at aws.amazon.com and looking over some of the tutorials they offer there.

"I hear" that wally world is moving to the microsoft cloud. 500TB would be very small change for them. MS Azure. Amazon has theirs and there's one more big cloud supplier too, but amazon and MS are the biggest.

It certainly could. If security is really important, I'd recommend that the customer should handle the encryption themselves beforehand rather than leaving it to the host.

However the cost will be astronomical - this place will be cranking out video about half the size of Netflix's entire library every year. It's worth checking whether Tobago has enough bandwidth to handle that. If "off site" just means "not in the building" they might want to consider setting up a local data center. This would also allow them to use WAN connections or shipped storage systems to reduce the bandwidth requirements. That would probably be cheaper.

More and more is going to the cloud.

Netflix? yeah. They use Amazons cloud service for all those shows you watch. The multi-distributed storage helps provide the content very quickly depending where you are and provides redundancy.

BTW, an in-between option they might want to consider is sending an Amazon Snowball box around to the different locations and loading it up with the month's video, and then sending that back to Amazon. No bandwidth problems there and no need to build a data center.

You'd have to figure out your run-rate costs for the cloud storage. Let's take a look at Amazon for a quick example, using their S3 storage service.

I got this info from here: https://aws.amazon.com/s3/pricing/

Three tiers, Standard, Standard with infrequent access, Glacier

First 50 TB/month $0.023/GB, $0.0125/GB, $0.004/GB

Next 450 TB/month $0.022/GB, $0.0125/GB, $0.004/GB

Over 500 TB/month $0.021/GB, $0.0125/GB, $0.004/GB

So, for your 500 TB (500,000 GB) of data,

Standard - (50000 * $0.023) + (450,000 * $0.022) = $1150 + $9900 = $11,050/mnth

Std - Inf. access (500000 * $0.0125) = $6,250/mnth

Glacier ( 500000 * $0.004 ) = $2,000/mnth

So, just in carrying costs, at the cheapest you will be spending $24,000 per year for the archived data. Then, there are the access costs. There are charges per 1000 operations. Costs are going to vary based on what tier you are using, and how you software works for pushing data into S3.

The big pluses for using cloud services are you don't have to worry about power, cooling, hardware costs, support costs, etc. If you are in a situation where capex is hard to get approved but opex is easy, then cloud services become a lot more attractive. I'm sure there are tools out there to help you with better pricing, but this is just the back of the envelope calculation to give you a starting point.

This does not take into account the cost of bandwidth. That can be huge if you are trying to push a ton of data very quickly.

Isn't the "cloud" just an off site server that's not yours?

One that you have to use the global internet to use?

And if the size is so big that servers need to be added, while the up front cost is high, my impression of the long term storage (assuming archiving and deleting is done correctly) would be cheaper do have your own.

Perhaps I have that wrong.

Does it need to be accessible spinning disk. One could think old school and write backups and defined intervals and put them in cold storage ...

In reply to alfadriver:

You are correct on all points.

In reply to cwh:

Many of the US Government entities use Amazons cloud storage service for their backups. If price is no object that's a pretty solid way to go. Rolling your own would be quite a bit cheaper but that comes with a lot of maintenance and possible interconnect problems when you start talking that amount of data since most ISPs get a little funny when you start talking Netflix sized data consumption.

The other possibility would be to do a local fiber loop like many university's install. That's going to be expensive upfront but if it's a long term type deal it might save a lot of hassle and money over time. Again that comes with the need for local maintenance.

Caveat: I do this E36 M3 for a living.

Cloud can be very attractive and there are some solutions that are ITAR and FIPS 140-2 compliant which other government agencies like the DOJ are using today to enable cloud storage so it is possible. Most of the cloud providers are achieving something like 13 nines of data availability when you're utilizing multiple data centers.

Couple of questions out of the gate: What is the access method that the camera system is using to write the data. Is it something REST-ful already, is it a block protocol (iSCSI/SAS/FC), or is it a file protocol (SMB/CIFS/NFS)?

When the cameras are writing, what is the rate at which they store data? (Be careful on this one because it will depend a lot on what and when the cameras are recording. Darker recordings actually consume more capacity than brighter ones, so indoor cameras where things are always brightly lit require less bandwidth/capacity than outdoor cameras that will capture hours of darkness).

What's the growth expectation? 70 cameras today, how many coming in the next 1, 2, and 3 years? Do you anticipate bringing in any cameras with higher resolutions/capture rates that we need to plan for?

What is the Internet bandwidth available?

What data sovereignty concerns exist?

Once the video is written from the camera, what is the playback requirement? Is this kept as a just-in-case and rarely, if ever, recalled? When a recall is necessary, what is the SLA for recall? Will the recall be done to the same site where the cameras are located, or is there a use case to view the data in a remote location?

Video data isn't going to compress, so you're probably looking at about $15-20k/month for cloud storage, assuming no egress fees. Does that break the bank?

If you have another data center (and we'll go back to the bandwidth question), it might make sense to look at a solution for private cloud storage -- put cloud-type storage on-prem where the video is being captured and then replicate that to another site. You can stage that replication so that the last week's worth of data is local and remote and the rest is purged locally and only remote, or you can simply do a one-for-one replica and keep all the data in both sites. Ignoring cost of money and the OPEX versus CAPEX issue, for a three year solution, the cloud storage costs would probably give you nearly $600,000 to spend and that's not outside the realm of possibility to purchase something on-prem or -- if it needs to be an OPEX -- do a utility cents-per-gb-per-month model on private on-prem gear.

It will all end up in a big data center in Utah no matter what and at no cost to you. The trick would be how to retrieve it.

^LOL true

Anyone else read this as: "there is something very secret and interesting hidden in Tobago that the Government doesn't want you to know about. Probably alien technology."

pinchvalve wrote: Anyone else read this as: "there is something very secret and interesting hidden in Tobago that the Government doesn't want you to know about. Probably alien technology."

It crossed my mind. This level of security is probably unprecedented in Caribbean governance.

You may be right, Gameboy. We are getting more and more strange requests from our islanders.

I work in this industry. You need to ask your customer if cloud storage meets their security requirements. Assuming it is US Federal Government, the answer is most likely "no," unless the cloud storage in question is actually owned and operated by said Government.