Saturday, February 11, 2017

Time Machine backups on DreamCompute

I really like the simplicity of Time Machine for backing up my Macs. It makes restoring lost files and changing computers extremely easy. However, my laptop is rarely on my home network so it would go months without completing a full backup. My solution was to set up an openvpn server and bridge it with my home network. This allowed my laptop to appear as if it was on my home network regardless of where I was connected. This also allowed me to print to my network printer and VNC to machines on my home network when I was away. I also set it up where I could, when needed, tunnel all traffic through my home network so traffic would be encrypted even if I was on a public unencrypted wifi network.

I had been using BackBlaze for offsite backups but I wanted to see if I could get offsite TimeMachine working. A few notes before I get rolling:

  1. This uses a lot of bandwidth for initial backup. I was saturating my 100Mbps FIOS connection before I started packet shaping. However, incremental backups are usually small so it isn’t much of an issue after initial backup.
  2. If you follow the directions below, you can’t do a full system restore from it as a local VPN connection is required. So, if something happened to both your computer and your local backup, restoring would be still be possible but more complicated as you would need to copy the sparse image somewhere local.
  3. If you have a large (>100GB) amount of data to back up, this can get expensive as only 100GB is included with a DHC account.
  4. This should not be your primary backup. Because of the latency, browsing and restoring files lags a bit.

Setting up a server

I will be setting up an Ubuntu instance on DreamCompute. On that instance, I will be running an OpenVPN server, an AFP daemon, and an avahi daemon.

The first thing you need to do is create an Ubuntu 16.04 instance using the directions here. Even while backups were running, I never exceeded a load of .5 on a single proc instance or more than 100MB of RAM so a semisonic instance is more than powerful enough.

You need to create a volume for the backups to be stored on and attach it to the instance you just created. Make sure the volume you create is at least as big as the drive in the computer you are backing up. You should only use this volume for backups as Time Machine will grow to use all the space you give it to store incremental snapshots. Follow the directions here to set up a volume.

You also need to open up UDP port 1194 for your OpenVPN connection. Instructions for setting up a security group on DreamCompute are here.

Install dependencies

Now that we’ve got the server up and running, first thing we need to do is install needed dependencies!

sudo su
apt-get install openvpn easy-rsa netatalk avahi-daemon

Create and attach volume

Partition and format your volume. I used xfs but it doesn’t really matter.

parted -s /dev/vdb mklabel gpt mkpart P1 xfs 0% 100%
mkfs.xfs /dev/vdb1

Create the direction where you want the backups stored.

mkdir /mnt/backups

Edit /etc/fstab and add a line for your backup mount

/dev/vdb1 /mnt/backups xfs defaults 0 0

And get it mounted.

mount -a

Set up users

While it is possible to store all backups under the same user, I find it easier to manage backups if each computer’s backup is stored under its own user. For each computer you want to back up set up a user.


adduser luke
mkdir /mnt/backups/luke
chown -R luke /mnt/backups/luke

Set up certificate authority and generate keys

For computers to be able to connect to your OpenVPN server, you need to set up a certificate authority.

mkdir /etc/openvpn/easy-rsa/
cp -r /usr/share/easy-rsa/* /etc/openvpn/easy-rsa/

Edit /etc/openvpn/easy-rsa/vars and update the values to match your information.

Build the certificate authority

cd /etc/openvpn/easy-rsa/
source vars
./clean-all
./build-ca
./build-key-server timemachine
./build-dh
cp timemachine.crt timemachine.key ca.crt dh2048.pem /etc/openvpn/

Now, generate a key for each of the computers you wish to back up.

cd /etc/openvpn/easy-rsa/
source vars
./build-key luke

Copy ca.crt and the generated files to each machine you are going to be backing up using scp or rsync.

/etc/openvpn/ca.crt
/etc/openvpn/easy-rsa/keys/luke.crt
/etc/openvpn/easy-rsa/keys/luke.key

Next we have to set up the OpenVPN server itself. Copy the sample server config file and decompress it into /etc/openvn

cp /usr/share/doc/openvpn/examples/sample-config-files/server.conf.gz /etc/openvpn/
gzip -d /etc/openvpn/server.conf.gz

Edit /etc/openvpn/server.conf For timemachine you want it to be a tap device, not tun so remove the semicolon from behind “dev tap” and add one behind “dev tun”.

dev tap
;dev tun

You also need to update ca, cert, and key to point to the correct files.

ca ca.crt
cert timemachine.crt
key timemachine.key

OpenVPN configuration is done. Time to start it up!

service openvpn start

Set up AFP

TimeMachine requires either a local disk or a disk mounted via AFP or SMB. AFP will be deprecated once apple moves to APFS. Unfortunately, as of the writing of this blog, the SMB implementation in Ubuntu doesn’t support an extension that TimeMachine requires. You can check https://bugzilla.samba.org/show_bug.cgi?id=12380 to see if that has changed. So, we will be using AFP here. Luckily, the default install includes a config file for Apple volumes we just have to tweak it for our setup. Edit /etc/netatalk/AppleVolumes.default and add this line:

/mnt/backups/$u "Time Machine" options:tm

Now you just have to restart the service!

service netatalk restart

Set up avahi

The last thing we have to set up is avahi. This is Apple’s zeroconf networking protocol that will allow our Mac to find the server to back up to it. There’s no configuration changes we need to make sure. Just start the service.

service avahi-daemon start

Set up OpenVPN on mac

While you can use the OpenVPN command line client or TunnelBlick, I have found Viscosity much more powerful and easy to use.

For address, use the ip of your DreamCompute instance.

We will be moving a lot of data over this connection. Make sure LZO compression is on.

Use the ca, cert, and key that you copied over earlier.

Start backups!!!

From System Preferences, go to Time Machine. From there click “Add Backup Disk.” Select the TimeMachine mount you created and the backup will start!

If you ssh back into the server, you should be able to see the sparse image.

ubuntu@timemachine:/mnt/backups/luke$ ls
Luke.sparsebundle Network Trash Folder Temporary Items

posted by admin at 9:37 pm  

Sunday, November 4, 2012

Site update check script (or how I knew the Viriginia Bar results within 5 seconds)

Joy just passed the Virginia bar (congrats baby!!!). The way that bar results work  in Virginia, is at some random time over the span of a couple days the list of everyone who passed gets posted to http://www.vbbe.state.va.us/bar/barresults.html This results in 1500 people staring at their computer endlessly beating the refresh key. To save Joy the stress on her fingers I wrote up a script to do it for her. I thought of two different ways of doing it.

1. Using the document.lastModified property

2. Checking the filesize of that webpage.

2 seemed like a cleaner/quicker option but did have the extremely unlikely chance that the modified page would be exactly the same size as the unmodified page. To fix this, I decided to do a checksum instead of a file size check. I also had it either dump the page to shell or email a copy of the source code just in case it got changed back quickly. I wrote the script in a way I could reuse it in the future. It should work on any *nix system. However, if you want an email instead of a shell output you will need to have mutt installed and outgoing mail properly configured. It will probably not work correctly on most dynamically generated sites. I’m sure there are better ways of going about this but this worked well for me and let Joy and me know results were up within 5 seconds of them being posted. Example usage and output below.

[harlow]$ ./sitecheck.sh
Usage: sitecheck.sh $site [$email] [$sleep (default 30)]

harlow]$ ./sitecheck.sh betterthanbacon.me/test.html luke@lukeodom.com 5
Sun Nov 4 19:57:13 PST 2012 no change
Sun Nov 4 19:57:18 PST 2012 no change
Sun Nov 4 19:57:23 PST 2012 no change
Sun Nov 4 19:57:28 PST 2012 no change
Change detected. Sending email to luke@lukeodom.com
Sun Nov 4 19:57:38 PST 2012 no change
Sun Nov 4 19:57:43 PST 2012 no change

Script can be downloaded at

http://betterthanbacon.me/sitecheck.sh

Or it’s dumped at:

#! /bin/bash

#Check for at least one variable

if [ $# == 0 ]
then
echo “Usage: sitecheck.sh \$site [\$email] [\$sleep (default 30)]”
exit
fi

#Create temp file

tmp=`mktemp`

#Create initial md5 sum

curl -s $1 > $tmp
md5old=`md5sum $tmp | awk ‘{print $1}’`

#Get site again, compare to inital md5 checksum, rinse, repeat

while true; do
curl -s $1 > $tmp
md5=`md5sum $tmp | awk ‘{print $1}’`
if [ $md5 != $md5old ]
then
if [ -z $2 ]
then
echo “change detected”
cat $tmp
else
echo Change detected. Sending email to $2
echo $1 |  mutt -a $tmp -s “$1 updated” $2
fi
md5old=$md5
else
echo `date` no change
fi
if [ -z $3 ]
then
sleep 30
else
sleep $3
fi
done

posted by admin at 8:02 pm  

Wednesday, June 17, 2009

New Gadgets

So, most people would consider it a bad time to be buying new gadgets since I am only weeks away from moving to Los Angeles. Well, I think any time is a good time to buy gadgets and this week I got three.
First Gadget: Canon CanoScan Lide 100
This was a gadget of necessity. My bank does not have a local branch so I have to deposit my check by scanning them. Earlier last week my scanner of 9 1/2 years (a mustek 1200 UB plus) died. I have used that scanner for many many projects over the years but right now scanning checks is all I do with it so I grabbed the cheapest new scanner on Amazon.

BIG thanks to the developers of SANE. Having SANE prolonged the life of the scanner for 4 years. When I got my mac laptop in 2005 mustek has not updated their drivers but SANE had. Thanks guys!!!
Second Gadget: Acer x223w
When I get a new gadget I immediately post it on Amazon for the price of the next best one. Last year, when I got my 19″ Acer I did that and this week it sold allowing me to purchase a new 22″ for roughly the same price!!!!
Third Gadget: iPhone
I have been with Alltel forever and my contract is up. The iPhone is awesome. I really am not too fond of going to AT&T but its worth it. I also looked at the Palm Pre and Blackberry Storm but in the end the iPhone plays nicer with my 12″ Powerbook and it has the AppStore.

posted by admin at 11:00 am  

Friday, January 16, 2009

The nuts and bolts of this blog

Here’s what this blog is made of and why I made it this way.

The webhost – dreamhost. Why dreamhost? I was formerly the webmaster of a newspaper that used dreamhost. I liked it and was used to it so I used it. Why not host on blog site such as wordpress? Pretty much I just wanted my own server to play with 😀

The software – wordpress.  I am a huge fan of open-source software and have worked with it before and WordPress is great. I even set my father-in-law up blogging 😀 

The theme – Dark Knight.  No, I am not a huge batman fan. I stare at a computer all day and personally find it easier on the eyes to have white text on a black background. My web browser and email client also use the PitchDark theme.

posted by admin at 7:30 am  

Powered by WordPress Copyright © 2011 Luke Odom