From 5.1 RC2 to current (part I: the kernel)

The first post showed how NetBSD 5.1 RC2 could be installed, this week an announcement was made that NetBSD 5.1 RC3 is available which turns my installation into something outdated. Nevertheless this also gives me the ideal excuse (or forces me to, depends on which sentence you like best) to upgrade my installation to current. In this context is nothing more than a fancy for CVS head, or tracking the most current developments, this way (in conjunction to subscribing to several of the netbsd  mailing lists ) I hope to get a bit familiar with the source code and the NetBSD development process, and this obviously opens the door for some contributions by myself (the last one being a long term and idealistic goal, and I don't have a clue if I will ever reach that point).

An advantage of playing around with Virtualbox, as I'm doing now, is that one can easily export the appliance (read: make a full backup) and import an appliance (read: make a full restore), or even run multiple versions in parallel. So before you start doing this, just make a full backup, this will allow you to play around and to break things, and falling back to a stable situation is just a couple of clicks (and some patience) away.

This blog post is just something I'm writing up for my own reference, I'm not reinventing the wheel myself, so I'm also just following the documentation a first suggested way to make use of the binary images which the NetBSD project provides (accessible at http://nyftp.netbsd.org/pub/NetBSD-daily/ ), this is a solution, but I call this the easy way out, thus I will not follow this path, if I want to modify code, I need to be able to have these images build myself.  For this reason I also choose to host the CVS sources on my Linux host OS, this way I can mount the CVS sources in several NetBSD guests (over NFS). Thus on my local system I obtain a CVS copy:

edb@lapedb:~$ mkdir netbsd 
edb@lapedb:~$ cd netbsd/
edb@lapedb:~/netbsd$ export CVS_RSH="ssh
edb@lapedb:~/netbsd$ export CVSROOT="anoncvs@anoncvs.NetBSD.org:/cvsroot"
edb@lapedb:~/netbsd$ cvs co -P src
...
edb@lapedb:~/netbsd$ du -sh src
1.4G src

Next I set up NFS locally (note: lapedb is my Linux host os, flying_spaghetti_monster is my NetBSD guest):

edb@lapedb:/etc$ apt-get install nfs-common nfs-kernel-server
edb@lapedb:/etc$ cat /etc/exports  | tail -n 1
/home/edb/netbsd/src 192.168.10.0/24(rw,sync,no_subtree_check,no_root_squash)

And on the guest I can run

flying_spaghetti_monster# mount -t nfs 192.168.10.100:/home/edb/netbsd/src /mnt/nfs/
flying_spaghetti_monster# cd /mnt/nfs/
flying_spaghetti_monster# cvs update 
Warning: Permanently added the RSA host key for IP address '204.152.190.16' to the list of known hosts.

This also gives me the opportunity to do a little benchmarking, I compared the delay in time between a cvs update running native on the Linux host, and a cvs update from the NetBSD guest mounted over NFS (with the options set to sync and async). This while the repository is up to date:

native on Linux host: 
real 1m19.054s
user 0m1.400s
sys 0m1.396s

nfs sync on NetBSD guest
      117.33 real         0.80 user        57.79 sys
nfs async on NetBSD guest
      111.22 real         0.66 user        55.75 sys

So this means that by adding the NFS/Virtualization I loose about 38 seconds on a CVS update. Okay, a bit higher than my expectations, but still very workable. Okay, the story so far, we have the CVS sources available in our sandbox so we can start playing.

We start by following the crosscompilation chapter in the NetBSD guide

flying_spaghetti_monster# cd /mnt/nfs/
flying_spaghetti_monster# ./build.sh -m i386 tools
===> build.sh command: ./build.sh -m i386 tools
===> build.sh started: Sat Jun 19 15:35:20 CEST 2010
===> NetBSD version:   5.99.31
===> MACHINE:          i386
===> MACHINE_ARCH:     i386
===> Build platform:   NetBSD 5.1_RC2 i386
===> HOST_SH:          /bin/sh
===> No $TOOLDIR/bin/nbmake, needs building.
===> Bootstrapping nbmake
...
===> Tools built to /mnt/nfs/obj/tooldir.NetBSD-5.1_RC2-i386
===> build.sh ended:   Sat Jun 19 15:55:51 CEST 2010
===> Summary of results:
         build.sh command: ./build.sh -m i386 tools
         build.sh started: Sat Jun 19 15:35:20 CEST 2010
         NetBSD version:   5.99.31
         MACHINE:          i386
         MACHINE_ARCH:     i386
         Build platform:   NetBSD 5.1_RC2 i386
         HOST_SH:          /bin/sh
         No $TOOLDIR/bin/nbmake, needs building.
         Bootstrapping nbmake
         TOOLDIR path:     /mnt/nfs/obj/tooldir.NetBSD-5.1_RC2-i386
         DESTDIR path:     /mnt/nfs/obj/destdir.i386
         RELEASEDIR path:     /mnt/nfs/obj/releasedir
         Created /mnt/nfs/obj/tooldir.NetBSD-5.1_RC2-i386/bin/nbmake
         makewrapper:      /mnt/nfs/obj/tooldir.NetBSD-5.1_RC2-i386/bin/nbmake-i386
         Updated /mnt/nfs/obj/tooldir.NetBSD-5.1_RC2-i386/bin/nbmake-i386
         Tools built to /mnt/nfs/obj/tooldir.NetBSD-5.1_RC2-i386
         build.sh ended:   Sat Jun 19 15:55:51 CEST 2010

This build.sh is actually a really nice system, it has just build a crosscompiler which runs on NetBSD 5.1_RC2, and will compile for i386. But I could actually run the same script on my native Linux box and use build.sh to construct a toolchain to build NetBSD for any architecture.

Now that we have the tools in place, let's start with compiling our kernel. But first we collect some proof, tihs is our current kernel:

flying_spaghetti_monster# uname -a 
NetBSD flying_spaghetti_monster 5.1_RC2 NetBSD 5.1_RC2 (GENERIC) #0: Fri May 21 04:32:33 UTC 2010  builds@b8.netbsd.org:/home/builds/ab/netbsd-5-1-RC2/i386/201005210534Z-obj/home/builds/ab/netbsd-5-1-RC2/src/sys/arch/i386/compile/GENERIC i386

The kernel configurations are located in sys/arch/i386/conf/ and we will now build the GENERIC kernel.

flying_spaghetti_monster# ls -hal ./sys/arch/i386/conf/GENERIC
-rw-r--r--  1 1000  1000   56K Jun  1 04:40 ./sys/arch/i386/conf/GENERIC
flying_spaghetti_monster# ./build.sh -u -m i386 kernel=GENERIC
===> build.sh command: ./build.sh -u -m i386 kernel=GENERIC
===> build.sh started: Mon Jun 21 22:23:48 CEST 2010
===> NetBSD version:   5.99.31
===> MACHINE:          i386
===> MACHINE_ARCH:     i386
===> Build platform:   NetBSD 5.1_RC2 i386
===> HOST_SH:          /bin/sh
===> TOOLDIR path:     /mnt/nfs/obj/tooldir.NetBSD-5.1_RC2-i386
===> DESTDIR path:     /mnt/nfs/obj/destdir.i386
===> RELEASEDIR path:     /mnt/nfs/obj/releasedir
===> makewrapper:      /mnt/nfs/obj/tooldir.NetBSD-5.1_RC2-i386/bin/nbmake-i386
===> Updated /mnt/nfs/obj/tooldir.NetBSD-5.1_RC2-i386/bin/nbmake-i386
===> Building kernel without building new tools
#    objdir  /mnt/nfs/sys/arch/i386/compile/obj
===> Building kernel:  GENERIC
===> Build directory:  /mnt/nfs/sys/arch/i386/compile/obj/GENERIC
Build directory is /mnt/nfs/sys/arch/i386/compile/obj/GENERIC
Don't forget to run "make depend"
depending the kern library objects
#    create  kern/__cmsg_alignbytes.d

....

#    link  GENERIC/netbsd
/mnt/nfs/obj/tooldir.NetBSD-5.1_RC2-i386/bin/i486--netbsdelf-ld -Map netbsd.map --cref -T /mnt/nfs/sys/arch/i386/conf/kern.ldscript -Ttext c0100000 -e start -X -o netbsd ${SYSTEM_OBJ} ${EXTRA_OBJ} vers.o
NetBSD 5.99.31 (GENERIC) #0: Mon Jun 21 23:06:06 CEST 2010
   text    data     bss     dec     hex filename
9684860  541396  554716 10780972         a4812c netbsd

===> Kernels built from GENERIC:
  /mnt/nfs/sys/arch/i386/compile/obj/GENERIC/netbsd
===> build.sh ended:   Mon Jun 21 23:06:27 CEST 2010
===> Summary of results:
         build.sh command: ./build.sh -u -m i386 kernel=GENERIC
         build.sh started: Mon Jun 21 22:23:48 CEST 2010
         NetBSD version:   5.99.31
         MACHINE:          i386
         MACHINE_ARCH:     i386
         Build platform:   NetBSD 5.1_RC2 i386
         HOST_SH:          /bin/sh
         TOOLDIR path:     /mnt/nfs/obj/tooldir.NetBSD-5.1_RC2-i386
         DESTDIR path:     /mnt/nfs/obj/destdir.i386
         RELEASEDIR path:     /mnt/nfs/obj/releasedir
         makewrapper:      /mnt/nfs/obj/tooldir.NetBSD-5.1_RC2-i386/bin/nbmake-i386
         Updated /mnt/nfs/obj/tooldir.NetBSD-5.1_RC2-i386/bin/nbmake-i386
         Building kernel without building new tools
         Building kernel:  GENERIC
         Build directory:  /mnt/nfs/sys/arch/i386/compile/obj/GENERIC
         Kernels built from GENERIC:
          /mnt/nfs/sys/arch/i386/compile/obj/GENERIC/netbsd
         build.sh ended:   Mon Jun 21 23:06:27 CEST 2010
===> .
flying_spaghetti_monster# file /mnt/nfs/sys/arch/i386/compile/obj/GENERIC/netbsd
/mnt/nfs/sys/arch/i386/compile/obj/GENERIC/netbsd: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, for NetBSD 5.99.31, not stripped

Jay, this means we have a kernel image available. Now we will need to boot from it, so we copy it over the original kernel (after creating a backup):

flying_spaghetti_monster# mv /netbsd /netbsd.old
flying_spaghetti_monster# cp /mnt/nfs/sys/arch/i386/compile/obj/GENERIC/netbsd /netbsd
flying_spaghetti_monster# ls -hal /netbsd*
-rwxr-xr-x  1 root  wheel   11M Jun 21 23:14 /netbsd
-rwxr-xr-x  1 root  wheel   11M May 21 06:32 /netbsd.old 
flying_spaghetti_monster# shutdown -r now 

And we cross our fingers. If things fail, we can fall back to the original kernel by following the following steps:

However after a reboot this fallback mechanism was already necessary (boot the old kernel, and copy netbsd.old to netbsd because the GENERIC kernel has loadable module support by default and the filesystems are compiled in (out) as modules, hence just booting the GENERIC results in a kernel which can't mount its root filesystem. The workaround is to either put the proper modules there or to build the kernel with the MONOLITHIC config, this config should not rely on modules. If we take a look at the config file for the MONOLITHIC kernel:

# $NetBSD: MONOLITHIC,v 1.11 2010/04/09 14:02:05 ahoka Exp $
#
# Non MODULAR, used mostly as a reference as to what we modularized.
#
include "arch/i386/conf/GENERIC"

no options MODULAR

options  EXEC_AOUT # exec a.out binaries
options  EXEC_ELF32 # exec ELF binaries
options  EXEC_SCRIPT # exec #! scripts
options  COREDUMP

options  P1003_1B_SEMAPHORE # p1003.1b semaphore support
options  AIO  # POSIX asynchronous I/O, built as a module
options  MQUEUE  # POSIX messsage queues, built as a module

options  COMPAT_OSSAUDIO # OSS (Voxware) audio driver compatibility
options  COMPAT_SVR4 # binary compatibility with SVR4
options  COMPAT_IBCS2 # binary compatibility with SCO and ISC
options  COMPAT_LINUX # binary compatibility with Linux
options  COMPAT_FREEBSD # binary compatibility with FreeBSD

# File systems
file-system  FFS  # UFS
file-system  EXT2FS  # second extended file system (linux)
file-system  LFS  # log-structured file system
file-system  MFS  # memory file system
file-system  NFS  # Network File System client
file-system  NTFS  # Windows/NT file system (experimental)
file-system  CD9660  # ISO 9660 + Rock Ridge file system
file-system  MSDOSFS  # MS-DOS file system
file-system  FDESC  # /dev/fd
file-system  KERNFS  # /kern
file-system  NULLFS  # loopback file system
file-system  OVERLAY  # overlay file system
file-system  PROCFS  # /proc
file-system  UMAPFS  # NULLFS + uid and gid remapping
file-system  UNION  # union file system
file-system  CODA  # Coda File System; also needs vcoda (below)
file-system  SMBFS  # experimental - CIFS; also needs nsmb (below)
file-system  PTYFS  # /dev/ptm support
file-system  TMPFS  # Efficient memory file-system
file-system PUFFS  # Userspace file systems (e.g. ntfs-3g & sshfs) 

We see that it just extends the GENERIC configuration and adds the file system support. However after building the monolithic kernel

flying_spaghetti_monster# ./build.sh -u -m i386 kernel=GENERIC

And following the same steps (copying it to /netbsd) and rebooting we are now running our own-rolled kernel:

flying_spaghetti_monster# uname -a
NetBSD flying_spaghetti_monster 5.99.31 NetBSD 5.99.31 (MONOLITHIC) #0: Wed Jun 23 23:49:20 CEST 2010  root@flying_spaghetti_monster:/mnt/nfs/sys/arch/i386/compile/obj/MONOLITHIC i386

Next step, building userland and using a generic kernel with module support.

Post scriptum:
Later experimentations showed that the NFS mount turned out to be rather painful and it did not fulfill my needs (mainly talking about performance here). So in a later phase I started using build.sh to build on my Linux host (when I set up the NFS I was not aware that build.sh was powerful enough to do this.