Creating a cloud-based virtual hard drive
I’ve been storing my backups “in the cloud” for a number of years now. For the past 3 or so years I’ve been using Amazon S3 and various linux-based solutions to store my backups of absolutely can’t afford to lose things.
During that time I’ve used Jungledisk and S3FS, but now I’ve moved onto S3Backer. So far so good.
Here is a quick guide to what I did to set up s3Backer on my Ubuntu 10.04 LTS server. The instructions on the project site are a little limited, and I didn’t find any useful “how to” blog posts.
- Compile and install s3Backer and dependencies
- Create your mount points. You will need one mount point for s3backer (which is used for the files stored on s3) and then one mount point for the file system (which S3backer looks after and saves to your S3 mount)
- Start s3backer. The command I’m using is
s3backer --blockSize=1024k --size=250g --listBlocks --vhost --ssl --timeout=250 --blockCacheThreads=3 --maxUploadSpeed=35k --accessFile=/etc/passwd-s3backer --prefix=s3 buggles /s3
The blocksize option is good to have at 1024 when you are using it for backups; good balance of file system cache and put requests to S3.
The vhost option is needed because this is an EU bucket.
The timeout option I need as the server sits on the end of my ADSL, with a limited upstream. s3backer runs 20 simultaneous threads by default, so I’ve limited the upload speed to 35k and dropped the number of threads to 3, which means it will take at least 90 seconds for 3mb to transfer (blocksize times number of threads divided by bandwidth cap) – hence a safe timeout of 250.
I’ve put my password file in /etc (rather than in Root’s home directory which is the default) as you can see by the accessFile option.
- Make your file system. I went for ext4. There were some warnings on the project site that certain file systems would create too many “empty” files in your s3 account (and therefore cost you money even if they weren’t in use by the virtual disk). On creation, a 250 gig ext4 disk used 17mb of storage in my s3 account.
mke2fs -t ext4 /s3/file
- Finally, time to mount your cloud-based disk. This is the one you actually use and backup to.
mount -t ext4 -o loop /s3/file /backups
- Create some cronjob scripts that rsync your files across to your virtual disk. Here’s an example of the kind of rsync command I use:
rsync -rptgoDv --del --inplace /stuff/ /backups/stuff
I currently store about 160 gig of backups in my Amazon S3 account for a costs of approximately 30 USD a month.