Reflecting on Day 1

Yesterday I obtained and installed the 4 major BSD's in a virtual machine.  Each had its own compatibility issues with VirtualBox, but I found ways around that.  Were I to do it all over again, I might try running aqemu on a PC-BSD host instead of VirtualBox on a Windows host.

I noticed that every installer gave me an opportunity to drop to a shell to troubleshoot a broken system.  All but DragonFlyBSD had an upgrade option.  It turns out that NetBSD and OpenBSD prefer binary upgrades, while DragonFlyBSD prefers source upgrades and FreeBSD is fairly comfortable with both.  I'll revisit upgrading on another day.

OpenBSD, NetBSD, and FreeBSD let you choose what base parts of the operating system you wish to install.  OpenBSD and NetBSD call these sets, while FreeBSD refers to them as distributions or distribution sets.  DragonFlyBSD doesn't give you much of a choice. 

FreeBSD's sysinstall lets you install to UFS1 and UFS2 filesystems, with or without softupdates.  NetBSD's sysinst offers up FFSv1, FFSv2, and LFS.  OpenBSD seems to only support installing to UFS1. 

Today's Objectives

• Configure basic network settings (hostname, resolver, IP address, DHCP, etc.)
• Determine open ports on each system
• Verify availability of network services
• Query DNS
• Configure NTP
• Enable sshd and disable root logins
• Set and remove interface aliases
• Review IPv4 and IPv6 theory

In a word: Networking.  Probably the first thing you want to do after installing a system is get it connected to the network so you can download additional updates and software.  All of the tasks listed for today are actually quite easy and any UNIX admin should know how to do these on at least one operating system.  The main point of today's lab is to explore the subtle differences between each operating system.

Network Theory Review

If you've ever gone for a basic network certification like CCNA, there should be little to worry about.  My biggest concern is with IPv6 theory, since I've never had a need to deal with it in 11 years of administering servers.  I'm just going to leave some links here that cover all the theory listed in the BSDA requirements publication:

IPv4

Classful Networks
CIDR
Subnetworks
Subnetting Reference
Network Notation Conversions

IPv6

IPv6 overview
IPv6 Autoconfiguration
Types of IPv6 Addresses
More Information

DHCP for all

I want to start with the simple case of getting all the config information via DHCP.  My VM host is set to be a DHCP server to all the virtual interfaces. 

Right away I notice that only NetBSD's dhclient supports the -r option to release a lease.  Otherwise there's not much special here — just issue dhclient interface to obtain or renew a lease from the command-line.

Making it stick

Enabling automatic DHCP at boot for an interface is a little different depending on the operating system.  For FreeBSD and DragonFlyBSD, Edit your /etc/rc.conf to include the following:

Where fxp0 is your network interface. 

Under NetBSD, edit /etc/rc.conf and set the following:
dhclient=YES
This will turn dhclient on for all interfaces.  To list the interfaces explicitly, add:
dhclient_flags="wm0"
where wm0 is the list of interfaces to run dhclient against.

In OpenBSD, make a file /etc/hostname.em0 were em0 is the interface name, and set its contents to dhcp.

Manual Network Configuration

Now we want to set an address, subnet mask, default route, hostname, and DNS servers manually.  Let's look to the handbooks for instructions:

FreeBSD and DragonFlyBSD:
http://www.freebsd.org/doc/handbook/config-network-setup.html
Scroll down to Configuring the Network Card.  Some lines are added to /etc/rc.conf and /etc/resolv.conf

NetBSD:
http://www.netbsd.org/docs/network/index.html#configuration_files
NetBSD seems to have a couple ways to configure the network settings.  It supports /etc/rc.conf settings, as well as /etc/myname, /etc/mygate, /etc/ifconfig.IF, and /etc/resolve.conf.  So much for KISS…

OpenBSD:
http://www.openbsd.org/faq/faq6.html#Setup
Looks like /etc/myname, /etc/mygate, /etc/hostname.IF, and /etc/resolv.conf.  That's not so bad.

Don't forget to be aware of how the different systems change the order of name resolution:
http://www.bsdnewsletter.com/bsda-book/Change_the_order_of_name_resolution.html

Verifying Network Settings

In order to verify the current network settings on any system, we should all know:

ifconfig -a
netstat -rn | grep default
cat /etc/resolv.conf
hostname

Determine Open Ports

Looking at the list of commands given for this section of the BSDA requirements, I think they want us to know how to list open sockets and files as well.  Things to note:  OpenBSD doesn't have sockstat, lsof and nmap are not in the base distributions.

Here are the commands I like to use to check for open ports and established connections:

sockstat -cl
sockstat -l
netstat -a | grep LISTEN
netstat -a | grep ESTABLISHED

Of course nmap is used to probe local and remote systems for open and filtered ports.  I've never used lsof but apparently it stands for "list open files" and it is useful for determining what process has a death grip on a file.  Or you could just use fstat.

Enable NTP on All Systems

Let's get all the servers to have the correct time via NTP.

DragonFlyBSD:

dragonfly# echo dntpd_enable="YES" >> /etc/rc.conf
dragonfly# /etc/rc.d/dntpd start
Starting dntpd.

FreeBSD:

freebsd# echo ntpd_enable="YES" >> /etc/rc.conf
freebsd# /etc/rc.d/ntpd start
Starting ntpd.

NetBSD:

netbsd# echo ntpd=YES >> /etc/rc.conf
netbsd# /etc/rc.d/ntpd start
Starting ntpd.

OpenBSD:
OpenBSD is a little bit different.  There is no /etc/rc.d/ directory and the /etc/rc.conf file should not be edited.  Instead, put startup overrides in /etc/rc.conf.local.  See http://www.openbsd.org/faq/faq10.html#rc for more startup information.

openbsd# echo ntpd_flags="" >> /etc/rc.conf.local
openbsd# ntpd
openbsd#

Some BSDs also have a command ntpdate which synchronizes time once and exits.  Ntpd's servers can be set in /etc/ntpd.conf.  Ntpd will not normally make a large adjustment to the system clock.  To force ntpd to set the time now instead of synchronizing slowly, use ntpd -s or ntpd -q depending on the operating system.

Enable SSH Logins For !Root

Disabling root logins via ssh is too easy.  Just edit /etc/ssh/sshd_config and uncomment or edit:
PermitRootLogin no

Now to enable sshd at startup and manually start it on each system:

DragonFlyBSD and FreeBSD:

# echo sshd_enable="YES" >> /etc/rc.conf
# /etc/rc.d/sshd start
Starting sshd.

NetBSD:

netbsd# echo sshd=YES >> /etc/rc.conf
netbsd# /etc/rc.d/sshd start
Starting sshd.

OpenBSD:

openbsd# echo sshd_flags="" >> /etc/rc.conf.local
openbsd# /usr/sbin/sshd
openbsd#

Nothing to it.

Verify Availability of Network Services

So we're supposed to know how to use ping, traceroute, telnet, and nc (netcat) to verify and troubleshoot network services.

OK, there's not that much to know about ping and traceroute that you won't turn up in the first 2 minutes of searching.  Remember that ping isn't the end-all to verifying a hosts's availability since ICMP echo messages are often blocked by routers and firewalls.  Failed replies and timeouts when running traceroute are not necessarily problems for similar reasons.  Times are round-trip.  Anybody taking this exam ought to have experience with both of these.

Telnet is great for testing TCP connections.  Remember it won't work for UDP.  Let's use it to verify SSH on all our lab systems.  Remember that /etc/services maps names to ports, so we can check it and see that ssh is on port 22.  We can use either of these commands to see if sshd is responding:

# telnet localhost 22
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
SSH-2.0-OpenSSH_5.5
^]
telnet> quit
Connection Closed.
#

# telnet localhost ssh
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
SSH-2.0-OpenSSH_5.5
^]
telnet> quit
Connection Closed.
#

So you specify the port as a parameter on the command-line.  The port can be either numeric or a word from /etc/services.  Control-] is the way to escape to the telnet prompt and forcibly close the connection.  This is still child's play.

How about netcat.  I've used nc for creatively piping things across the network, but not so much for verifying services.  Have a tutorial about netcat if you've never used it before.  OpenBSD and FreeBSD have netcat in the base, so let's portscan localhost and verify http on google:

# nc -z localhost 1-100
Connection to localhost 22 port [tcp/ssh] succeeded!
Connection to localhost 25 port [tcp/smtp] succeeded!
#

#nc -z google.com 80
Connection to google.com 80 port [tcp/http] succeeded!
#

OK wonderful.  It's not as fast as nmap (no parallelization), but nc scanned tcp ports 1 to 100 and found ssh and smtp open.  Then we used nc to verify something is listening to port 80 on google.com.  Use the -u option to test udp ports.

Querying a DNS Server Like a Pro

All platforms support nslookup, dig, and host. All three of these utilities do pretty much the same thing, but dig is the preferred tool these days.

nslookup is old and quirky.  It can be a pain in the butt to learn to use properly and all the time I put into it is probably why I have such a hard time stopping using it 🙂

host is the simplest to use, but can still tell you what you need to know.

dig is the most configurable, giving you the most control over what and who you query. 

I think it's important to remember that these tools are their own recursive resolvers.  They don't use the system's facilities for name resolution and therefore they don't care much about nsswitch.conf or your hosts file.  If you don't specify a server, however, then they will look for one in resolv.conf.

I think the skills they are looking for go beyond simply checking if a hostname resolves and doing a reverse lookup on an IP address.  You should know how to use nslookup, dig, and host to test and troubleshoot a DNS server.  Know how to query a server and verify that it is authoritative, be able to query for specific record types (SOA, A, MX, PTR, TXT, NS) and know what they are and the information they are supposed to contain.  Also know how to use these tools to attempt a zone transfer, which is important both for verifying that the transfer works and for verifying security (it's bad form to transfer your zones to just anybody who asks).  It goes without saying that you're expected to understand how DNS works and you should have some experience troubleshooting and maintaining a DNS server.

Here are some great links if you need brushing up in this area:
http://www.bsdnewsletter.com/bsda-book/Query_a_DNS_server.html
http://uw713doc.sco.com/en/NET_tcpip/dnsC.nslook.html
http://www.tuaw.com/2007/01/29/monday-man-page-dig-host-and-nslookup/

Set and Unset Interface Aliases

This is very basic and yet also important.  A NIC is usually associated with a single, primary IP address, but it's possible to configure additional, simultaneously active addresses called aliases.  There are many reasons why you might need to or want to do this.  You may want to run multiple services on a single machine but with different configurations, maybe you want to use IP based virtual hosting (necessary for https virtual hosting), perhaps you want to NAT individual internal subnets to different external IP addresses for different traffic shaping policies or just to be able to differentiate the source networks for future troubleshooting.  Maybe you need something entirely different.  In any case, knowing how to configure an interface for multiple IP addresses (and how to remove those addresses without taking the interface down) is just something we expect of a UNIX sysadmin.

At this time, the latest distributions of all the BSDs support the ifconfig options alias and -alias.  You can also use delete instead of -alias, but it may not work for IPv6 addresses on all systems.  FreeBSD and DragonFlyBSD also support add as a synonym for alias.  I really think this is a bit of trivia and I hope they don't ask questions about which operating systems support which arguments.  That information is in the man pages.  What's more important is that you know how to add and remove aliases without clobbering the primary IP address.

So let's just add some aliases:

# ifconfig em1
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 08:00:27:d1:b8:ee
        priority: 0
        media: Ethernet autoselect (1000baseT full-duplex)
        status: active
        inet6 fe80::a00:27ff:fed1:b8ee%em1 prefixlen 64 scopeid 0x2
        inet 192.168.56.101 netmask 0xffffff00 broadcast 192.168.56.255

# ifconfig em1 inet 10.1.1.1 netmask 255.255.255.0 alias
# ifconfig em1 inet 10.1.2.1 netmask 255.255.255.0 alias
# ifconfig em1 inet 10.1.3.1 netmask 255.255.255.0 alias

# ifconfig em1
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 08:00:27:d1:b8:ee
        priority: 0
        media: Ethernet autoselect (1000baseT full-duplex)
        status: active
        inet6 fe80::a00:27ff:fed1:b8ee%em1 prefixlen 64 scopeid 0x2
        inet 192.168.56.101 netmask 0xffffff00 broadcast 192.168.56.255
        inet 10.1.1.1 netmask 0xffffff00 broadcast 10.1.1.255
        inet 10.1.2.1 netmask 0xffffff00 broadcast 10.1.2.255
        inet 10.1.3.1 netmask 0xffffff00 broadcast 10.1.3.255
# 

And now remove a couple aliases:

# ifconfig em1 10.1.2.1 -alias

# ifconfig em1
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 08:00:27:d1:b8:ee
        priority: 0
        media: Ethernet autoselect (1000baseT full-duplex)
        status: active
        inet6 fe80::a00:27ff:fed1:b8ee%em1 prefixlen 64 scopeid 0x2
        inet 192.168.56.101 netmask 0xffffff00 broadcast 192.168.56.255
        inet 10.1.1.1 netmask 0xffffff00 broadcast 10.1.1.255
        inet 10.1.3.1 netmask 0xffffff00 broadcast 10.1.3.255

# ifconfig em1 10.1.3.1 delete

# ifconfig em1
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 08:00:27:d1:b8:ee
        priority: 0
        media: Ethernet autoselect (1000baseT full-duplex)
        status: active
        inet6 fe80::a00:27ff:fed1:b8ee%em1 prefixlen 64 scopeid 0x2
        inet 192.168.56.101 netmask 0xffffff00 broadcast 192.168.56.255
        inet 10.1.1.1 netmask 0xffffff00 broadcast 10.1.1.255
# 

Making the aliases stick

Now to preserve aliases across reboots in the various BSDs:

DragonFlyBSD and FreeBSD:
Add a line to /etc/rc.conf for each alias like:

ifconfig_em0_alias0="inet 192.168.1.5 netmask 255.255.255.255"

Of course change em0 to your actual interface name and increment the number at the end of alias# for each alias.

OpenBSD:
Add aliases to the /etc/hostname.if files:

# cat /etc/hostname.dc0
inet 192.168.0.2 255.255.255.0 NONE media 100baseTX
inet alias 192.168.0.3 255.255.255.255
inet alias 192.168.0.4 255.255.255.255

NetBSD
Add aliases to /etc/ifaliases in the format address interface netmask:

# cat /etc/ifaliases
10.1.2.1 wm0 255.255.255.255

Wrapping Up

Now we've explored the differences in network configuration between OpenBSD, DragonFlyBSD, FreeBSD, and NetBSD.  We should be able to get essential networking up immediately after installation and configure sshd for remote administration.  We can troubleshoot routing and DNS problems and verify that network services are listening and responding.

I didn't cover the route command but you should know how to add and delete routes, including the default route to the gateway.  All but FreeBSD support the command route show, but it's more compatible to use netstat -r.