nanoBSD for servers – Part 1

Warning:  This is more of a rambling brain-dump than a carefully revised and edited, or even coherent post.


For a long time now, I've been intrigued by two useful features of FreeBSD:  nanoBSD and ZFS. 

I like the nanoBSD concept of managing servers as deployable read-only images on ping-pong partitions.  That would allow me to build and test an OS/application image and then push it over the network to a remote server without having to go through the mergemaster hassle.  It also provides for an instant failsafe in case the upgrade fails.

I've been using ZFS on production servers for about 2 years now.  I like that it can tolerate unexpected power interruptions and that I can snapshot file systems before attempting risky modifications.  The ability to back-out of bad upgrades can spare me long weekends and trips to a remote site.


Over the years and in the back of my mind, I've been searching for a way to meld the characteristics of nanoBSD and ZFS to create an extremely reliable server able to tolerate far less than ideal environments and also easy to maintain remotely (including OS upgrades).  In the worst case, I'd like to have the option of walking somebody through booting a recovery image over the phone so I can finish the job remotely.

The problem with ZFS by itself for purely remote administration manifests when it comes time for an OS upgrade.  Snapshots make it possible to roll back, but you still need a reboot to single-user mode and other risky business in order to complete the upgrade itself.  It also takes a considerable amount of time (I like to build from source to apply custom build and configuration options).

The problem with nanoBSD for a regular server is that /var is a memory disk and nothing stored there will persist across a reboot.  Gone will be the logs, databases, mail, and other important things.  There are some simple ways to work around this, but I want complete separation of the OS media and the data/configuration media.  I'd really like to have a read-only OS image on one device with ping-pong binary upgrades and all the configuration and data on a raidz pool that's backed-up to another machine using dirvish or rsnapshot.


I'm not the first to attempt some sort of "bigger nanoBSD" or "nanoBSD + ZFS" evolution:

I think it's cleaner to separate the configuration and data from the OS and installed software.  Data and configuration can be on a raidz pool while the operating system and applications are static and could reside on a separate disk, compact flash, USB drive, CD-ROM, etc.  Should the OS media fail, I want the system to automagically revive itself after replacement of a new OS drive without reconfiguration or restoration from backup.

I've considered the idea of updating the applications separately from the OS (something like GoboLinux), but I think that is more trouble than it's worth.  Libraries, applications, and OS are all tied together and it makes more sense to rev a single image where libraries applications and OS are compatible rather than try to maintain separate applications and OS and track a compatibility matrix.


My plan is to start with nanoBSD and add a zfs pool with file systems and mount points at /cfg and /var.  I'll symlink a directory like /etc/local to /usr/local/etc so I can store /usr/local/etc config data in /cfg along with the /etc data.

If the zfs pool is unavailable, there will be a fallback to a writable memory disk /var and a default recovery configuration on an OS image /cfg partition.

The upgrade procedure will be modified to include creation of zfs snapshots for /cfg and /var so that they can be updated along with the OS, but also rolled-back in case of failure.


I think FreeNAS 8 comes very close to doing what I intend to do, but but its focus is heavily on being a NAS appliance and not as much on being a remotely managed general purpose server.  I'll probably pare down the FreeNAS nanoBSD config file as a starting point and add in my particular customizations.


Some particular issues I will need to consider:

  • GEOM labeling of the OS drive so it can always be found (tunefs -L)  (added already in (not-so)recent versions of nanobsd?)
  • Zpool naming scheme and zfs mount point scheme
  • handling of zpool.cache and/or importing zpools after swapping OS drives / upgrading

It turns out we don't need zpool.cache at all. It is just as safe to import the zpools we know about on boot. Rather than play games to keep this file happy, we should just do the import always. Email with pjd@ confirms that unless we're booting from ZFS, this file is completely optional and won't really buy us anything since we have another database of zpool data.

We should modify the zpool commands to operate on /etc/zpool.cache and not worry about it from there. This property can be set early in boot so we don't need to modify our backend, I believe. I'll investigate and add the appropriate early boot commands (as well as adding the zpool imports to ix-fstab). This will be more robust anyway and a lot easier to implement than the mount -uw / dancing.


And that's about it for my research and brainstorming.  I'm going to take a couple stabs at actually implementing these ideas and hopefully post some successful results soon.


I just received my BSDA exam results today and I'm relieved to have passed :)

I would encourage others to give it a shot as well.  Even though it's supposedly targeted towards junior admins, even developers and senior admins will probably find aspects of it challenging.  In the course of study, I discovered some new tools and better ways of doing the same job I've been doing for the last decade. 

The exam lived up to most of its promises–the questions were straight-forward and clearly worded and it definitely separates those with real-world experience from those without.  Beware Linux admins — the BSDA exam has some Linux-y wrong answers so you'd better be sure of the differences in BSD before attempting it.  For FreeBSD admins, a little bit of exposure to the other BSDs is a must, especially the installers, pkgsrc, and upgrade procedures.  I don't think I could have passed without having installed each operating system a couple times and learned some of the basic differences between them and FreeBSD.  That's not to say that I had to become an expert in those operating systems, however–I think it's sufficient to just be aware of what's different and what what those different things are called.

I almost skipped BSDA entirely because I felt BSDP would be more appropriate and I was holding out for that exam.  In point of fact, there are relatively few BSDA certificate holders out there and so it is still a relatively exclusive club.  There are no BSDA boot-camps or leaked questions from the exams and the exam itself is psychometrically valid, so I think it's a good goal for BSD administrators looking to validate their abilities and experience.  It's a good run-up to the upcoming BSDP exam, which I think will in some ways be easier than BSDA since it will be less broad and more focused on specific real-world tasks.

One thing that keeps surprising me is that out of the huge number of people who registered IDs to be able to sign up for exams, only a very small fraction have actually followed-through and attempted the exam.  What's holding people back?  The exam is cheap enough that it's practically free, it's challenging and exclusive enough that it's a worthwhile goal, and the inconvenience of taking it at a convention is offset by the positive experience of meeting and interfacing with other working professionals. 

Certification doesn't matter in my workplace, but I'm still proud to be certified.  I'm proud to have an outside group validate my own personal achievement, I'm proud to be doing my part to advocate BSD and promote BSD awareness, and I'm proud to be doing my small part to support certification–especially for my fellow admins in other countries (and now even here, too), where jobs and education are scarce and any kind of professional training or certification can give them a leg-up.

WordPress Permalinks and 404

The default permalink naming scheme used by WordPress is bad.  Most people wind up changing it and most documentation on the subject of changing permalinks centers around redirecting the old links to the new ones so you don't erase all your hard-earned credibility.

This post is about a totally different issue when changing permalinks.  A very few people will change their permalink structure and then find that every link clicked on the WordPress site results in a 404.  This has nothing to do with migration or redirection–even links to posts made after the permalink change will 404.

The problem is in the apache configuration.  If you're getting 404's, here's how to fix it:  First of all, WordPress relies on mod_rewrite for non-default permalinks, so make sure that module is loaded.  Second, there's a .htaccess file with the rewrite rules that needs to be able to take effect, so check the httpd.conf file section for your WordPress directory and try adding

AllowOverride All

And that should fix it.

Using Fixit and chroot to repair a non-booting system

The trick:

You want to boot off a live CD (or USB stick) and use your system as if you had booted into single user mode.


The problem:

Booting off a CD means you have to mount your filesystems to a subdirectory.  You keep editing the livecd's files by mistake instead of your installation's.


The solution:

  • Boot to a fixit/livefs
  • Locate the root partition and mount to /mnt
  • mount the devfs to /mnt/dev
  • chroot to /mnt
  • mount -a if you want to automatically bring up the other partitions

This is basic stuff, but it took me a while to learn this trick–Mainly because booting off a CD to fix something isn't a frequent occurence.  The tips here are chroot and devfs

Chroot  changes the apparent root directory to someplace else.  In other words it maps "/" to a subdirectory of your choosing.  You can mount your broken rootfs to /mnt and then use chroot to make the current shell behave as if you had mounted the filesystem to "/".  Now you can access /boot and /etc instead of /mnt/boot and /mnt/etc

But /dev will be conspicuously missing after a chroot.  That's why it's important/useful to mount devfs to /mnt/dev before doing the chroot.  Then you'll be able to access your devices for mounting, partitioning, or MBR/bootloader repairs while in the chroot shell.


Here's an example from top to bottom:

Booting the CD and selecting Fixit

Select media



Here's what / looked like before and after the chroot to /mnt:

Before and After chroot


Now the userland paths are set up just as if you had booted into the broken system.  This is a much better environment from which to do tests and repairs in most cases. 


And just to be complete, here is how you can explicitly unhide devfs nodes (though I haven't needed to yet):

mount -t devfs devfs /mnt/dev
devfs -m /mnt/dev rule apply path null unhide