In the "upload 3tbs to the cloud" thread we were discussing setting up a shared backup system. The original suggestion was to use BitTorrent Sync but I think we should go with something FLOSS so we're not totally at the mercy of anyone's whims (BTSync is closed-source freeware from BitTorrrent Inc). Also because it syncs changes automatically, damage to your local data (or something that looks like it, like a drive not being mounted) could be propagated immediately and automatically while the service is running, which is obviously far from ideal for a backup system.
So the first thing we'd have to settle on is what software we're going to use. Here's what I've looked at so far:
Tahoe-LAFS: Very powerful and flexible, highly secure with default encryption but pretty freakin' complex to set up - the complex setup is just "up-front" work and it would be easy for anyone to join in once those details are hammered out by the most uber-nerdy among us 
, but I think the biggest problem is the unequal distribution of workload - at least one person would have to run a "helper server" involving heavy upload bandwidth usage and CPU usage or uploads would be painfully slow. But in terms of features this looks like the best tool for the job by a long shot. Interface-wise it uses SFTP/SCP protocol.
Syncthing: This one is more similar to BTsync in operation. Pretty simple, equal distribution of workload, but no file-level encryption (so you'd have to encrypt the files locally first using something like EncFS). Also like BTsync it propagates changes immediately while running.
Any other suggestions?
GameboyRMH wrote:
Also because it syncs changes automatically, damage to your local data (or something that looks like it, like a drive not being mounted) could be propagated immediately and automatically while the service is running, which is obviously far from ideal for a backup system.
I'm pretty sure it can be configured so this doesn't happen.
I don't like that LAFS would require a helper server of some sort. Unless its a fairly lightweight server and if it went down it wouldn't compromise the ability for someone to recover files.
Don't like that sync thing has no encryption.
I'd still lean toward BT Sync, even if it is closed source. Do the others have Pi builds?
Libre software means you can always compile from source, but Syncthing has a generic linux-arm build as well. From what I understand Tahoe-LAFS is all python-based so it's platform-independent, you just need python with the right libraries.
Syncthing's encryption situation is the same as BitTorrent Sync - it has transport encryption but no file encryption.
The Tahoe-LAFS helper server is just a performance booster, it can work without it but uploads will be vastly quicker with it. There are no security or fault-tolerance downsides to having one.
I'm not much of a Linux geek(I run Ubuntu on 1 box), but I might be interested in joining this party. Do you need true server-class machines, or can a PC potentially work?
Not much power is needed (except for Tahoe-LAFS' Helper server). A regular ol' PC or even a RasPi can work.
Could you take a small step backwards and describe what the benefits and purpose of a shared backup is for the less technically savvy among us?
Basically, it's a convenient off-site backup system that would only cost a contribution of storage space.
If your house burns down with your backup drives inside or through extremely bad luck you lose your backup and live data at the same time, you won't suffer data loss. It's also a backup you can access from anywhere and with a high download speed - could be handy if you're travelling and you need to get to them, and your backup drives are in a drawer at home.
It's a shame the old, distributed version of Wuala is gone. I've not seen an equivalent.
GameboyRMH wrote:
Not much power is needed (except for Tahoe-LAFS' Helper server). A regular ol' PC or even a RasPi can work.
How much is needed for the helper? Can there be more than one helper? I'd dedicate a Pi2 to this. 95% of the time it would be idle and could be used as a helper. If others used a similar setup, the total helper capacity would scale pretty quickly.
Multiple helpers is a feature that will be added soon, the current version only allows one. The helper needs lots of upload bandwidth most of all, but lots of processing power is important too. The RasPi2 is a big step up from the original in processing power, but I don't know if it would be enough. It's certainly worth a try, even a helper that's short on processing power is far better than not having one.
What kind of bandwidth are we talking? I have a surplus HP server running in my basement with FreeNAS that I have not done anything with yet.
More is better, what's your upload bandwidth?
(Don't need crazy business connection speeds, just above-average 1st-world speeds.)
I have terrible upload bandwidth because TWC. 2mbps. I pay extra for that too. 
That said, I don't have a ton to backup so it doesn't concern me that much. Also crossing my fingers for reasonable bandwidth sometime soon.
I am in the same boat as Darwin.
Ugh, do we all have awful E36 M3ty upload speeds? I have 1mbit/s up on a good day. I think the helper server should have at least 10mbit/s.
I very nearly got a 20 down/ 5 up FTTH connection, but the old telecom monopoly bought out the competition just before I got the upgrade, and that was the end of that
So, BTSync doesn't need the server, correct? The only opposition to it is the fact that it is closed source?
Closed source and closed protocol (which could be reverse-engineered later though), no file-level encryption (backup buddies will be able to read your files unless you encrypt locally first) and I haven't yet found a way to work around the auto-damage-propagation problem.
Edit: Also, even without the helper server, Tahoe-LAFS should be faster than BTSync once there are more than 3-4 users.
Cool.
Does Tahoe-LAFS have an app (or a 3rd party app) that I can use to backup stuff from my phone to the cloud?
Not really...only found a 3rd-party download-only client for Android, and it needs to be compiled:
https://github.com/ctrlaltdel/TahoeLAFS-android
If you can get an SFTP/SCP client on your Android device though, you could connect that way - using an SSH tunnel to your server perhaps.
I have an iPhone. I could probably BT Sync or Owncloud -> server, which is copied into the cloud using TahoeLAFS though.
OK so I guess we're gonna go with Tahoe-LAFS...so here are the things we need to decide on:
-
How much space we're each going to contribute. I can spare somewhere between 100-500GB. It's more efficient for us all to share the same amount (similar to RAID5/6)
-
Who's going to run the introducer server. There's no downside to running this on the same system as a storage server, requirements are vanishingly light, but this server needs to be very reliable and on a very reliable connection. I'd do it but my connection is pretty unreliable nowadays.
-
Who's going to run the helper server, if anyone
-
Garbage collection parameters. To keep the system from filling up with old junk - potentially even from people who uploaded things and stopped using it or lost their capability strings (which are the keys to the kingdom in this system) - we'll have to use garbage collection where you have to "renew" your shares periodically or your files could be deleted. Since our system will be used for fairly long-term storage, I'm thinking we could have the lease last a year and ask for renewal ("asking for renewal" really means "deleting expired files") every 3-6 months. The default setting is much more demanding with a lease lasting a month and renewal required weekly.
Once all this is done I can put together a guide to make joining in as easy as possible.
OK the introducer server's on you.
Anyone else interested? If we don't get at least 10 people we'll have to use a non-default erasure coding level.
moxnix
HalfDork
10/16/15 8:12 a.m.
How much data transfer are we looking at here?
On FIOS but I hear verizon starts complaining when you hit 4+TB a month.
