bluebottle

Tarsnap backups on Windows and Linux

For the past month or so I’ve been running Tarsnap to backup my home desktop and remote servers. Tarsnap is an online (payware) backup system that’s written by Colin Percival, and is probably the most technically impressive backup system I’ve seen. As well as that, it’s damn cheap.

What’s so great about Tarsnap? Here’s three things:

• Snapshot backups. Every backup you create with tarsnap is a standalone ’snapshot’ of your data, totally independent of all other snapshots.
• Backup deduplication. So your backup includes a 300mb log file that gets a few megs of data written to it every day. Tarsnap recognises this, and for each new snapshot only the changed data has to be uploaded and stored.
• Crazy Ass Security. While mild-mannered Colin Percival works on Tarsnap during the day, by night he’s the FreeBSD Security Officer. He’s been there since 2005, so must be doing something right. Plus, check out this page. How can something with that many mentions of AES, SHA and RSA be insecure?! :-)

This post isn’t a HOWTO on setting up Tarsnap, there’s a comprehensive tutorial on the subject already. This post is just to document how I use Tarsnap, on both Linux and Windows.

Linux

My Linux servers perform a simple daily backup of everything in certain folders. Technically, I could simply backup “/” and exclude the directories I don’t want, but smaller archives are faster to restore from, and there’s no monetary penalty for having more snapshots. This is my script:

#!/bin/bash
for dir in $(cat /root/tarsnap-dirs) ; do
        nice tarsnap -c -f $(hostname -s)-$(date -u +%Y%m%d-%H%M%S)-$(echo $dir | tr -d '/') --one-file-system -C / $dir
done
 
# Delete backups more than n days old
# n=10
# tarsnap --list-archives | sort | cut -d- -f1-2 | uniq | tail -n +$n > /tmp/temp.$$
# tarsnap --list-archives | fgrep -f /tmp/temp.$$ | while read archive ; do
#     echo Deleting $archive
#     tarsnap -d -f $archive
# done
# rm /tmp/temp.$$
 
tarsnap --print-stats

(This script runs from crontab, so the output gets mailed to me daily. The email is sent to the address specified as the MAILTO variable in /etc/crontab.)

My .tarsnaprc looks like this:

keyfile /root/tarsnap-key-abraxo.key
cachedir /root/tarsnap-cache/
exclude /root/tarsnap-cache/
humanize-numbers

Each day, a backup of each folder listed in the file ‘tarsnap-dirs’ is created, with names like: ‘bluebottle-20091209-200001-homeaj’. There’s commented out support for deleting old archives too, but my monthly costs are so low I keep everything.

Windows

My Windows setup is basically identical, but since there’s no native Windows (or msys) support for Tarsnap, you have to make do with Cygwin. Explaining how to install Cygwin is far beyond the scope of this document, but it’s pretty simple. Apart from the standard Tarsnap dependancies, you will probably also want to install ’ssmtp’, which will let you email Tarsnap’s output to yourself like Unix cron does.

(You can generate ssmtp’s config file by running ’ssmtp-config’.)

Again, my script:

#!/bin/bash
 
log=/tmp/tarsnap.log.$$
 
cat <$log
From: Alex Jurkiewicz 
Subject: tarsnap run $(hostname)-$(date +%Y%m%d)
To: Alex Jurkiewicz 
 
EOF
 
tarsnap -c -f $(hostname)-$(date +%Y%m%d-%H%M%S)-homeaj -C /home/ aj >>$log 2>&1
tarsnap -c -f $(hostname)-$(date +%Y%m%d-%H%M%S)-CUsersAlex -C /cygdrive/c/Users/ Alex >>$log 2>&1
tarsnap --print-stats >>$log 2>&1
 
cat $log | /usr/sbin/ssmtp.exe alex@bluebottle.net.au
rm $log

Because I want to exclude a lot of directories on Windows, I put these in my ~/.tarsnaprc file:

exclude Desktop/
exclude AppData/Local/Temp/

And so on. So there you have it. Simple, painless backups with my favourite new toy, Tarsnap.

Ubuntu 9.10 nvidia-settings – xorg.conf save fail

Ubuntu 9.10 creates a (relatively) minimal xorg.conf file compared to what older Ubuntus have. This causes the nvidia-settings app to be unable to parse the file (and therefore save a new version), specifically because of this missing line from the “Screen” section:

Device         "Default Device"

If you add this line to your default screen configuration in /etc/X11/xorg.conf, nvidia-settings should be happy again.

Gripe: didn’t anyone pick this up during testing?

Scripted FreeBSD reimaging

Recently at work I’ve been working on a system to automatically revert FreeBSD systems to a known good state – something like a VM snapshot, but for physical machines too. The system is reasonably fast, portable across different hardware configurations and very easy to use. It’s turned out that this system has worked quite well, and although reasonably simple is the result of much research and time, so I’m posting some pointers here.

The Problem

The push for this came from a need to improve the testing environments our developers have been using. The old testing environments were VMs, more or less handmade at some point in the distant past to provide a rough approximation of our live environment. Since that time the developers had mostly looked after them themselves, tweaking, fixing and forking the VMs as they saw fit. New developers would copy an existing developer’s VM, and branch off from there.

This system was bad. There were problems ranging from minor version differences all the way up to entirely missing subsystems, not to mention the total lack of documentation. Every developer changed their VM slightly differently, so troubleshooting problems was as much exploration and discovery as debugging. Worse, the slowly growing number of environments made keeping them in sync (or something approximating that) with the live environment increasingly difficult. Something had to be done.

The Solution

What I’ve ended up going with is a very basic system that builds on as much of our previous infrastructure as possible. The key component is our ‘live environment image’, an OS image that we base all our live servers off. The new development environments are also based off this image, and the solution consists of a set of scripts that automate the conversion of the live environment image into a development environment. So that’s the high-level concept, how do the nitty gritty details work?

The system is made up of two separate installs of FreeBSD on one disk. The first install is a 5gb disk slice, containing a minimal install of FreeBSD. This is the Reimaging OS. The second install is on another slice taking up the rest of the disk and is the actual Development Environment.

The Reimaging OS

This copy of FreeBSD isn’t just a fresh install. There are a few changes, the most important are the installation of bash and these two lines added to the machine’s /etc/rc.local file, which start the magic:

scp -i /root/key reimageuser@fileserver:dev-environment-reimage.sh /tmp/
/usr/local/bin/bash /tmp/dev-environment-reimage.sh

And what does this script do? Well, in short:

1. Format the partitions making up the Development Environment.
2. Restore the base ‘live environment image’ into the Development Environment.
3. Modify the fresh Development Environment to actually be suitable for use as a Development Environment, rather than a live server.
4. Change the default boot slice to the second slice, and reboot the machine.

So now the machine reboots, and when the bootloader starts up, it loads…

The Development Environment

Remember step three above? The hand-wavey “turn the live environment into a dev. environment” step? Well, you can’t fully complete that step from the Reimaging OS. There are some things you just don’t know. What hostname should you set for the machine? What email address should all outgoing mail be redirected to? It turns out that the first time a dev. environment boots, the developer has to answer some questions. After developers reimage their machine, the final step is to run a script on first login: dev-environment-rechristen.sh. This script goes through and makes all the changes to the system that require user input, plus a few workplace specific changes.

Of course, the developer also needs to be able to kick off a reimage of their environment at some point in the future. How do they do that? Another script sets the default boot OS to the Reimaging OS and reboots the machine. Developer goes away for a coffee and comes back to a fresh machine.

The Scripts

So that’s how it works, how about some sample code? Please note that these are edited versions of the scripts I run with anything even vaguely revealing about our configuration/infrastructure stripped out. They’re a starting point for you, but you still need to do a fair amount of work to get these scripts working for you.

• reimage-setup.txt. This is a short script I wrote to help quickly deploy the Reimaging OS to new machines. Put it on a thumbdrive with a FreeBSD dump(8) image, then boot off a FreBSD install CD, and select Fixit -> Live CD Filesystem. Mount the USB key at /mnt (something like mount /dev/da0s1 /mnt) and then run the script like so: /mnt/reimage-setup.sh /dev/ad0 (where /dev/ad0 is the drive you want to use for the dev. environment. The script (or something like it) has worked well for me across a number of heterogeneous systems, but, of course, YMMV. For the record, the rest of these scripts are written in bash, so make sure that’s available in your OSes.
• reimage-do.txt. This script is the basic skeleton of our dev-environment-reimage.sh script. It shares many of the assumptions the previous script makes: what the partitions are, where they are located and so on. I’ve excised the more site-specific customisation, but it’s ready to go otherwise.
• reimage-begin.txt. This script completes the trio. It’s what you run from within the development environment to kick off the reimaging process. Nice and simple, and again shares all the assumptions the previous two scripts made.
• Bonus: a starting point for your reimage-rechristen.sh script. A couple of functions that ours does. Most of the rest of our script does site-specific work, like checking some files out of an SVN repository, changing the hostname in a few config files, and so on.

HOWTO: Wordpress pretty URLs with Nginx 0.6 (the proper way)

A lot of guides around the net for Nginx + Wordpress suggest something like this:

if (!-e $request_filename) { rewrite ^/blog /blog/index.php?q=$1 last; }

This is Bad. Try this:

if (!-e $request_filename) { rewrite ^/blog /blog/index.php last; }

What’s the difference? Well, the latter form doesn’t set a GET variable. Why is that better? wp-super-cache, should you choose to use it, refuses to create supercache files when the GET URL has variables in it. If you turn on the debug log, you see errors like these stream past:

Supercache caching disabled. Non empty GET request.
Supercache disabled: GET or feed detected or disabled by config.

You can find a full example of the required Super Cache config here.
 
 
Update:
For nginx >0.6.36 you can use the try_files directive instead:

location /blog { try_files $uri $uri/blog.index.php; }

There’s a full example with this new method in this mailing list post.

Windows 7 (64-bit) on Dell Vostro 1310

Everything works out of the box! Well, almost everything:

• Install new Nvidia drivers from the Nvidia website. The included ones are too ancient to be much good, and don’t support Aero or other fancy stuff.
• Touchpad driver doesn’t support scroll zones or middle click emulation. Install the touchpad driver for the Latitude E6400 (this one in particular). This driver has somewhat weird default settings. I dropped sensitivity a few notches and reduced the scroll detection threshold to minimum.