Optimising FreeBSD and it's kernel

This is a very simple step by step guide on optimising your FreeBSD server or workstation. It doesn't go into a great amount of detail, but after spending several months searching for one source of simple optimisation information and failing, I wrote this paper. All the suggestions listed here are known optimisations available to you if you know where to find them and have the time to do so. There is nothing secret, or special or amazing in this paper, just information on how you can optimise your system.
It can mostly be applied to the other BSDs too, but not Linux. There are plenty of Linux documents out there, so go find one of those. I'm sure there are several HOWTO's. This document is true as of the release of FreeBSD 4.8. Some parts of it, such as optimising your kernel, can also be applied to some previous releases.

---

Before we proceed any further you should know that as with any system tuning, we have to accept the possibility that something will break. If you're going to recompile the kernel, please read the following URL first. If possible print a copy of the page and keep it with you. It'll help you if your new kernel doesn't boot:
http://www.freebsd.org/handbook/kernelconfig-trouble.html

First we need to make sure we have the right source files downloaded. The best place to start, is the original source files that were intended for the release of FreeBSD you are using. We assume you have a fairly recent release - not more than 2 revisions (approx 6 to 8 months) old. Execute the following command and follow the steps to get the sources needed to recompile a kernel:

• # /stand/sysinstall
• Choose 'Configure'
• Choose 'Distributions'
• Choose 'src'
• Choose 'sys' and then choose 'OK', and 'OK' again.
• Choose an FTP site to download the sources from. If you're prompted to 'skip over the network configuration now', choose 'Yes'. After the install is complete, choose the 'Exit' options until you exit sysinstall.

Congratulations, you now have a know working kernel source tree installed in /usr/src/sys!
Now, cd /usr/src/sys/i386/conf and follow the next steps.

You should see two files in this directory, GENERIC and LINT. These are both kernel configuration files and contain all the data the kernel needs from the user, to compile. They are very easy to read and edit.
GENERIC is the Generic kernel configuration which should work with every system. LINT contains a list of all possible kernel configuration file directives. We won't worry about LINT just yet. First lets trim the GENERIC kernel down and get comfortable with that. Follow these steps to success:

• Copy the GENERIC file to a new file. Traditionally, this new file has the same name as the hostname of your machine.
• Edit this new file in your favourite text editor. I prefer vi and have written a cheat sheet for new vi users at:
  http://www.silverwraith.com/papers/vi-quickref.txt
• There are only a few pitfalls and special cases to watch out for in this file. As you can see it just a plain text file and any line starting with a # is considered a comment and ignored by the compiler.
  • The word 'GENERIC' in the line 'ident GENERIC' is the name of your kernel. This can be any single alphanumeric word with no spaces or punctuation. I recommend you keep it short. Again, it is traditional to name this to be the same as your machine hostname and kernel configuration file name but it is not required. It is informational only.
  • The maxusers line does not actually limit the maximum number of users. It is used internally with an algorithm in the param.c file to determine the sizes and numbers of certain tables. This can remain at the default. As of FreeBSD 4.7 this is set to '0' by default, which causes the actual value to be auto-sized depending on the amount of memory that you have.
  • Most, if not all items in your kernel config will have a comment toward the end of the line. Anything labelled with '[KEEP THIS!]' should not be removed. FreeBSD needs these things! Anything labelled '(required!)' should also be kept if you're going to use that type of device. For example, if you're going to use SCSI devices, don't comment out 'scbus'. If you're not using any SCSI devices, then you should comment this out. Some PCI network cards require the inclusion of 'device miibus'. These are noted in your kernel configuration file. If you don't use these NICs you can also comment this out.
  • Now go through the entire file and comment out anything you don't think you'll need. Effectively every line contains a driver. Commenting it out will mean that particular piece of hardware will not work your system, even though it may be recognised as present in the system. Thus it is bad to comment out your own network card, but good to comment out any cards you don't have. Don't worry if you comment out something you will need later. You can always recompile fairly quickly and simply.
• After you've finished editing the configuration file, save it and exit from the editor.
• Issue the command: config , where is your configuration file. This only takes a moment and it will tell you which directory is your compile directory. That is where we will compile the kernel. The config command creates the directory structure and files which the next steps use in compiling the kernel.
• cd to the directory that config gave and issue these commands in turn, each after the previous has finished:
  • # make depend
  • # make
  • # make install
  • Hopefully the above will all work without producing an error. If you get an error, see your local FreeBSD Guru. If you got no error, consider the installation of the new kernel a success!

That is it. Really! If something breaks and your new kernel will not boot, DON'T PANIC! Read the 'Kernel Does Not Boot' section of the URL at the top of the page.

---

Optimising compiling of code, the kernel
and debugging kernels

This section is for the slightly more advanced compilation of code and kernels. We will also briefly mention how to compile a kernel with the debugging symbols built in. The debugging kernel can be useful when your server or workstation is kernel panicing frequently and you want to find out why. There is a great article from OnLamp.com on how to debug a kernel, available at:
http://www.onlamp.com/pub/a/bsd/2002/04/04/Big_Scary_Daemons.html

---

Optimising code and kernel compilation

First lets look at optimising the compilation of C code itself. One of the key tasks when doing this is to start with as simple a configuration as possible. Hopefully if you've read everything above you'll be at this stage already :-)

You should set a few specific options in /etc/make.conf that will optimise the compilation of new code on your machine. This means all code will be compiled with these options. If you set the right ones, they can significantly improve the speed and efficiency with which code is compiled and executed. They will also help to reduce memory consumption. If you have installed the portupgrade package from /usr/ports/sysutils/portupgrade, you should execute this command after setting these options and updating your ports collection. It will cause every port you have installed, to have it's latest version downloaded and recompiled with the optimisations. I think it is worth it:

portupgrade -ra

Updating your ports collection en masse can have other consequences which are worth realising. If the current installed version of your port has undergone a major update (eg exim3 to exim4), a straight upgrade in this way could break your configuration. Use with care! The compilation process can take a while, depending on the speed of your CPU, the amount of memory you have, your internet connection and so on, so if you're on a slow link and wish to download the distfiles for each package first and do the updating offline, you can issue this command to download the distfiles for each port first:

portupgrade -Fra

So without further ado, the list of optimisations follows. Initially you won't have an /etc/make.conf, you can just start editing a file with that name (if you want to see every possible option to put in the file, reference /etc/defaults/make.conf):

CPUTYPE=cpu_type where cpu_type is one of i386, i486, i586, i586/mmx i686, p2, p3, p4,
     k6, k6-2, or k7.  gcc v2, which comes with FreeBSD doesn't yet support the latest Athlon
     optimisations, so if you have a Thunderbird or other such CPU, just set k7 for now. gcc v3 has
     better support and this will be available in FreeBSD 5 when it is released toward the end of
     2002.  NOTE: An important point to remember: Most CPU's of the same family are backward
     compatible.  That is, the K7 is backward compatible with K6 code. Also, Intel-CPU code is
     mostly universally compatible across CPU families. However, if you compile code for a
     non-Intel family CPU type and later upgrade to a newer Intel CPU, there is a good chance you
     may encounter problems unless you recompile your code.

CFLAGS= -O3 -pipe -funroll-loops -ffast-math -O3 sets optimisation level 3 where the
     largest number of practical optimisations can be made to your code (everything except the
     kernel). It also does sacrifice binary size for speed.  -pipe causes code to be passed between
     processes using pipes during compilation rather than using temporary files, which has obvious
     I/O advantages.  -funroll-loops causes iterating loops with a known number of iterations to be
     unrolled into faster executions.  -ffast-math breaks IEEE/ANSI strict math rules. One way it
     does this is by assuming that the square root of numbers are non-negative. This shouldn't be
     used if you're compiling code that relies on the exact implementation of IEEE or ANSI rules
     for math functions. Unless you're writing your own code that does just this you shouldn't have
     a problem with setting this. It should help reduce your compile times.

COPTFLAGS= -O2 -pipe -funroll-loops -ffast-math This is the same as the "CFLAGS"
     statement, except it's used on the kernel when you compile it.  We choose -O2 instead of -O3,
     because -O3 is known to produce broken kernels. Officially, only -O is supported, but I have
     never had a problem with -O2. It should be noted at this point that one difference between -O2
     and -O3 is that -O3 produces slightly larger code during its optimising. In normal situations
     this is OK, but we want to compact the kernel down it is another reason to stick with -O2.
     This also has the same effect as adding the 'makeoptions COPTFLAGS' lines to the
     kernel config as discussed below.

---

kernel optimisation

In you kernel configuration, add the following line after the 'machine' (i386, i486, etc) types near the top:

makeoptions    COPTFLAGS="-O2 -pipe -funroll-loops -ffast-math"

This does two things. First the -O2 switch tells the compiler to optimise the compilation. This takes advantage of internal compiler tricks to do this. You could use -O3 to implement even more optimisation tricks, but these aren't supported and -O3 is known to cause many stability issues. -O2 may or may not work for you depending on how many things you have compiled into the kernel and how non-standard your hardware is.

TOP_TABLE_SIZE=number where number is a prime number, at least twice the number of lines
     in /etc/passwd.  This statement sets the size of the hash that the
     top(1) uses for usernames when it runs.

options         CPU_WT_ALLOC

should be set if you have an AMD K5/K6/K6-2 or Cyrix 6x86 chip. It provides for the kernel to enable cache Write Allocation for the L1 cache, which was disabled by default on these chips.

options        NFS_NOSERVER

If you are running a workstation or server when you know that you will not be acting as an NFS server, you can add the above line to your kernel configuration to disable NFS server code. NFS servers allow other servers and workstations to mount parts of their filesystems using the Network FileSystem protocol.

options         NSWAPDEV=number

Another way of saving kernel memory is to define the maximum number of swap devices. Your kernel needs to allocate a fixed amount of bit-mapped memory so that it can interleave swap devices. I set the preceding parameter to 1 on my workstation and 2 on my servers. I rarely run out of swap space on a workstation but if I need to add more to a server, I can easily create another partition

---

Building a debugging kernel

Another option you have when compiling your kernel is to build with the debugging symbols. This can be fundamental in determining the reason your kernel panics, if it does. Be warned though - compilation time can increase slightly when doing this. To make a debug kernel, add the following line to your kernel configuration:

makeoptions     DEBUG=-g

This doesn't actually install a kernel with full debugging symbols as /kernel. The /kernel that gets installed is the stripped down regular kernel, but a separate kernel.debug file is in /usr/src/sys/compile/Your_Kernel_Name/kernel.debug. If your kernel panics and leaves behind a core file, the kernel.debug file is used to get the debugging symbols when you actually do the debug. See the OnLamp article for more on this.
One final thing you need to do if you're going to be building a debug kernel, is to have your system actually dump the memory to the swap partition. You can do this by adding the following like to /etc/rc.conf and rebooting:

dumpdev="/dev/ad0s1b"

Where "/dev/ad0s1b" is your swap partition as defined in /etc/fstab.
NOTE: Your swap partition needs to be at least 1Mb (1024Kb) bigger than your total amount of memory. When your system crashes, the memory will get dumped to your swap partition. When your system returns, your swap partition will be enabled and your disks will be fsck'd before they are mounted. During this process, fsck uses a small amount of swap space. It would be preferable if you had twice as much swap space as memory in this situation.