Automatically Backup, Bundle, and Encrypt Important Files

Posted on: by

I am like everyone else and if creating a backup is hard, I won’t do it.  And hard is a pretty low threshold to reach.  Since it is that time of year that all my tax documents start flowing in, and I have been really working on my paperless workflows, I decided I wanted to find an easy way to not only backup these important files locally, but I wanted an effortless way to securely save a copy offsite.

I suppose I could use a service like Crashplan or one of the similar services, but I already have plenty of offsite storage and really don’t need another monthly service fee.  Originally, I had been creating an encrypted DMG in OSX and then moving a copy to Google Drive.  This had a number of flaws.  First, the files actually reside on my home file server, not my local machine.  Second, it was a manual process.  Finally, it is dependent on OS X.  If I need that information I really would prefer it to be platform independent.

I decided that instead of a DMG to bundle all the files up, I would create an ISO.  The makes it easy to mount anywhere if I need to get at the files or restore them.  I know I could use tar to bundle them up, but it simply did not seem as elegant a solution.

I chose to encrypt the files with gpg.  It is an opensource tool that I could get on either Linux or OS X and, though I haven’t tried, I am guessing it would work on Windows as well.

To create a backup I am doing two things.  First, I create an onsite backup onto a second hard drive.  That doesn’t need to be encrypted and can easily enough be done with a cron job and rsync.  Second, I use Dropbox to save an offsite copy of the encrypted backup.  I chose Dropbox over Google Drive for a simple reason, it works well on Linux, which is the OS of my file server.

Now for the code:

Backup to Second Hard Drive

As I wrote earlier, this is done using rsync and cron.  I set cron to run rsync every hour to backup any changes.  You can set this to as often as you feel comfortable.

rsync -ah /files/to/backup/ /backup/destination/

Test Sensitive Files for Changes

That is easy enough. Now I want a particular directory within this to be the one that I bundle and offsite. These files do not change very often and I only want to offsite and encrypt a new bundle if something has changed. I couldn’t find a good built in solution to handle this recursively but after doing some searches, it looks like the best option is some variation on using a sha1 stamp to test if any files or directories have changed. I decided to use the ls command to get a list of all the files and directories modification time stamps recursively and pipe that to generate a sha1 stamp. The ls command has two flags that I use to make this work. The first is the -R flag which will list all the files recursively on a given directory. The second is the –full-time flag, which is like doing an ls -l except you get full-iso time stamps. I run this with cron. I run it once a day, but you can run it as often as you like.



sha=`ls -R --full-time $WATCH_PATH | sha1sum | awk '{print $1}'`

	# first time through, save the sha and then quit.
	echo $sha > $PREVIOUS_SHA_FILE

if [[ ${sha} != ${PREVIOUS_SHA} ]]; then
	# something has changed
	# code to run goes here

	# Update the previous_sha_file
	echo $sha > $PREVIOUS_SHA_FILE

Generate ISO, Encrypt, and Save To Dropbox

If the previous script detects something has changed, then it runs this script which bundles up the files into an ISO and then encrypts them, finally it moves them to a folder in my Dropbox which will then take over uploading to the server.



# Make the iso
/usr/bin/mkisofs -o $OUTPUT_FILE $INPUT_DIR

# encrypt the iso
gpg --yes --batch --passphrase=$PASSPHRASE -c $OUTPUT_FILE

# remove iso

# Move encrypted file to Dropbox for offsite copy

I am sure there are a number of ways to solve this problem, but I hope this gives you some ideas on how to solve a similar problem you might be having.