|
|
The Network People Solutions for Hosting Providers |
|
|
|
||
 |
Booting FreeBSD via PXE (Preboot Execution Environment) Version: v1.2 (also available: v1.1 (for FreeBSD 4) and v1.0) Audience: Unix System Administrators Objective: Document the steps necessary to boot and/or install a FreeBSD 5.x system using a DHCP, NFS, & TFTP server. Background: FreeBSD can boot and install off a variety of mediums. The common and most useful are floppies, cd-roms, and the network (ppp, NFS, & ftp). All the methods work essentially the same, you bootstrap the helpless machine off floppies (yes, even the CD boot uses floppy emulation) and the mini-FreeBSD system then has enough smarts to initialize CD-ROM drives, Ethernet, and PPP interfaces. This all works fairly good for installing FreeBSD on one machine but what happens when you want to install it on 20, or 50, or 1,000 machines? Right now the process is pretty much a manual one on every machine. This is expensive, error prone, and slow. Excuse: I had quite a few reasons to embark upon this project.
At BSDCon 2000 I sat in on a panel taught by Doug White on automated system installations. It gave me the impetus to charge forth and conquer the beast known as PXE. I'm too lazy to look up some of the facts (again) so anywhere I use <>, please feel free to send me the relevant information and URL's where I can verify and I'll update my documentation. Some time ago <date please> Intel developed a technology known as PXE and began blessing their wonderful EtherExpress Pro Server Adapters with this spiffy new feature. <Some 3Com NIC cards> also include PXE technology. PXE is designed to allow a NIC card to fetch a configuration from a DHCP server and boot up a computer via it's network interface. John Baldwin and Paul Saab at FreeBSD saw the usefulness of this feature and wrote a little boot loader appropriately named pxeboot. Pxeboot is included in FreeBSD 4.1 and higher. So, how does it work you ask? Very well, I must say. Once you satisfy it's many dependencies, things work great. Getting everything satisfied wasn't as simple as I'd hoped. Anyway, follow the steps below to reach enlightenment. Requirement 1: FreeBSD distribution. It would be nice for FreeBSD to just magically appear on your hard drive but alas, we've got to get it from somewhere. I suppose you could fetch it from the FTP server but I simply copied the CD contents to a NFS exported file system on my server.
Requirement 2: Ethernet adapter with PXE boot roms. The Intel Management adapters all include PXE but even our newest batches required flash updating to get FreeBSD booted properly. This is what my oldest Intel's looked like at first:
This version of PXE bios does not work. It will load the pxeboot loader but fails soon thereafter. A visit to Intel's web site had me downloading a file named 8255x.zip. Within that archive was three very useful things. The first was a directory name 8255x containing the boot ROM's for the Intel 8255x adapters. Logical huh? :-) The other useful tool was the fboot.exe program. I created a DOS boot floppy and copied these programs onto the boot floppy. I then installed five Intel adapters into the PCI slots of my server and proceeded to update them all. After installing the new firmware the PXE bios looks like this:
Requirement 3: DHCP Server. I already had ISC-DHCP 3.0b installed so I merely had to add a couple lines to my DHCP configuration. Here's what a working configuration looks like:
Requirement 4: DNS server. I'm not sure it's its necessary but I'm sure it's at least a good idea. Create an entry in your DNS records for dhcpserver.yourdomain.com and records for all the addresses in your DHCP pool. It's saves us a lot of time waiting for DNS timeouts when we've got DNS set up correctly. Requirement 5: TFTP server. The DHCP configuration tells the booting client that it's supposed to grab the filename "pxeboot" from the TFTP server (next-server) at 192.168.254.3. Here's how mine is set up:
This is pretty easy to configure. On most systems, simply comment out the tftp line in your /etc/inetd.conf and restart inetd (killall -HUP inetd). NOTE: TFTP has virtually no built in security. You should only enable a TFTP server on an internal (trusted) network or use a firewall to restrict access to it. At a minimum, use TCP wrappers. Requirement 6: PXEBOOT. Copy the pxeboot file from your /usr/src/sys tree to the /tftpboot directory and you're all set:
Once your machine has loaded the NIC cards PXE bios, it will (assuming it's the chosen boot device) make the tftp request for the file "pxeboot" from the tftp server. The tftp server, being properly configured will hand it the file "pxeboot" which is comparable to the FreeBSD loader program. NOTE: The pxeboot program can be compiled to fetch the loader via TFTP or NFS. NFS is the default but you can add this "LOADER_TFTP_SUPPORT=YES" to your /etc/make.conf and recompile pxeboot (#cd /usr/src/sys/boot; make clean; make depend; make; cp i386/pxeldr/pxeboot /tftpboot). Requirement 7: Boot loader. Once PXE boot is loaded it will fetch the files it needs from the /boot directory that's defined within the root-path directive your DHCP server handed it. Since we've defined a root path of /usr/local/export/pxe, it'll be looking within the /boot directory there for the second and third stage boot loaders. Here's what we've got set up there:
-r-xr-xr-x 1 root wheel 512 boot1 -r-xr-xr-x 1 root wheel 7680 boot2 -r-xr-xr-x 1 root wheel 163840 loader -rw-r--r-- 1 root wheel 504 loader.rc -rw-r--r-- 1 root wheel 105 loader.rc-freebsd-ide -rw-r--r-- 1 root wheel 105 loader.rc-freebsd-mailserver -rw-r--r-- 1 root wheel 93 loader.rc-freebsd-mylex -rw-r--r-- 1 root wheel 101 loader.rc-freebsd-scsi You can get these files in a couple ways. You can either snag them off the mfsroot.flp (follow the instructions on Alfred Perlstein's page) or just copy them from your source tree (like I did):
You'll need to create the contents of the loader.rc file to look something like this:
I also wanted to have the ability to select which mfsroot I wanted to boot off so I figured out how to script the loader process a little bit. You'll notice that the loader.rc calls another file (ex. /boot/loader.rc-freebsd-ide). Here's all that file looks like:
Requirement 8: NFS Server. Since we're going to use pxeboot's default retrieval method of NFS, we'd better export the /usr/local/export/pxe directory. Once most systems it's as easy as putting a line in /etc/exports that looks like this: "/usr -alldirs -maproot=root -ro". That exports the entire /usr file system with read only permissions. Once you've added that line to the exports file, you need to restart (or start) mountd. To set everything up on a FreeBSD system, just do this:
If you want this machine to be a NFS server every time you reboot, add this to your /etc/rc.conf: "nfs_server_enable="YES". NOTE: NFS security is adequate at best. Use IPFIREWALL or IPFILTER (I use IPFIREWALL) to restrict access to it. Requirement 9: Bootable Kernel. At this point your new machine has just run the loader program which, as instructed by the loader.rc file, is going to try loading the file named "kernel" from our root path. Lucky for us, we've got a perfectly good kernel floating around in our / directory so we just copy it from there over to our exported directory.
There are some requirements for this kernel, most specifically the MFS support but if you just use a GENERIC kernel, things will work out just fine for you. The reason you don't want to use the kernel off the boot.flp or kern.flp images is that they pause at the kernel configuration screen. Using a custom kernel also gives you the flexibility to load klm's (kernel loadable modules) for your custom install packages if you need to (I did). Once the kernel loads we'll proceed to the next line of the loader.rc file and that's fetching the mfsroot file. The mfsroot is just that, a memory based file system that's stored in a file. Requirement 10: MFSROOT. You have a couple options here. The first is to simply use the mfsroot disk image that comes with the version of FreeBSD you are installing. There are instructions for doing so on Alfred Perlstein's page. If your needs are modest and can be met with the tools provided on the mfsroot floppy then have at it. However, that defeats half the purpose of doing this exercise in the first place. I create a directory with the entire FreeBSD CD on my drive. I do it like this:
The real power behind this is that you can create your own mfsroot with no size restrictions. Weehee, let's make a really big 25 megabyte mfsroot:
There, now we've got 25 megs of space mounted on /mnt to play with. The basic rule of thumb here to keep in mind is that you are building a root file system for FreeBSD. Everything the kernel, sysinstall, and your install programs needs from the root file system is what you need to have in your /mnt directory. To get started we'll copy the contents of the distribution mfsroot.flp to our new mfsroot:
Voila, now we've got a bootable mfsroot partition and tons of space to play with. One thing to keep in mind, when you place any of your spiffy utilities on there, make sure you compile them statically. This is usually as easy as changing into the appropriate source directory, editing the Makefile and adding "-static" to the CFLAGS. You've got plenty of room to hack up your own sysinstall or anything else that you want/need on your root partition. At this point, you can simply umount the mfsroot (umount /mnt/mfs; mdconfig -d -u 0) or build it up the way you want/need. We have some pretty ugly kludges we've made which including hacking sysinstall and a few other goodies that use about 20 megs. I'm sure you'll find some creative ways to use all that space. Just drop in your statically compiled binary and run with it. My system looks like this at boot time:
Requirement 11: Install.cfg. If you're installing FreeBSD via the Ethernet adapter, (as opposed to just using PXE to netboot) then you'll want to script sysinstall. Why go though all this bother if you have to sit in front of the box and answer questions? Anyway, now our loader.rc has requested the mfsroot file and pulled our 25 meg image across it's 100Megabit interface. The loader.rc also tells the kernel to get it's root file system from memory disk 0 (md0c) that we loaded. The kernel then had it's root partition set up based on the contents of our mfsroot partition. Once it checks out our hardware it looks for /sbin/init which doesn't exist and then falls back to running sysinstall. Sysinstall checks to see if the file /install.cfg exists and if so, uses it to control it's behavior. The script syntax is documented in "man sysinstall" which you might need to install (cd /usr/src/release/sysinstall; make install). There's a sample install.cfg installed at: /usr/src/usr.sbin/sysinstall/install.cfg and here's what my install.cfg looks like:
Last modified on 1/29/07. |
|
