Building GRUB for iMac Netboot EFI

GRUB2 Built on Ubuntu 14.04 to use in iMac Netboot via EFI

Steps:

1. download grub
2. install packages to build
3. create configure script
4. configure for efi 64bit
5. patch grub to automatically configure network on iMac efi
6. make grub
7. create bootstrap grub.cfg to load the real configuration from the network
8. build efi netboot grub image
9. create full grub.cfg to be pulled from server by grub
10. install grub image, grub.cfg on TFTP server

See the attached files for more details.

Script to document the steps
git-diff output for patches to GRUB

Netbooting Apple Clients with ipxe or grub

Still trying to image Apple clients (iMacs) with FOG.  In a previous post I discussed setting up the DHCP server to send boot files to the iMac via the Netboot protocol.

FOG sends iPXE to the PCs to control the boot process during imaging.  We can't just use the same iPXE file for the iMacs, because the PCs are booting using legacy BIOS and the iMacs are using EFI.  iPXE supports EFI, so we built an iPXE for EFI file to send to the iMAC.  It is downloaded, but when the iMac tries to run it, the apple loader seems to choke on it, then boots to the default (harddrive).  The same iPXE EFI file runs fine on the iMac if you install it on the hard drive or a USB drive and run it using rEFIt or rEFInd.  I was unable to figure out why there is a difference, and was unable to attract any attention from the iPXE developers.  Without any kind of output from the netboot loaded image, I didn't see how to start debugging the process.

After some googling, I found these links that indicated that GRUB2 might be able to do what we wanted.

iPXE forums
GRUB mailing list

So, I downloaded the GRUB source and built a 64-bit GRUB EFI binary to send to the iMac.  If you do this, be sure the follow the instructions in the GRUB mailing list post.  So, when the mac loaded the grub image, it dropped to the grub shell.  Awesome, if you ask me.

After messing around in the shell a bit, I was able to get GRUB to load the grub.cfg from the network server, and present the menu.  I thought I was almost home.  Wrong. :(

From the instructions, I was expecting the network config file to be loaded automatically.  I need it to load automatically, if this is going to be useful for imaging 50+ macs.

After 2+ days of messing in the GRUB shell and source code, I've found that, at least on our iMacs, there is a problem with the EFI network interface.  BTW, the modularity of the GRUB code is very nice.

Here's the problem as I see it now.  When GRUB loads, it does all of it's initialization code, including detecting network cards and configuring GRUB's view of those cards, by pulling information from EFI using efi API calls.  Well, the cards (yes, both Ethernet and WiFi) are found.  However, the WiFi card wasn't connected to an AP, so it fails the configuration detection, as expected.  And the Ethernet card configuration detection fails when trying to open the PXE protocol.  This failure isn't surprising since the iMac doesn't support PXE.  Thus, GRUB has no network connection.

Mysteries still exist.  Remember that I was able to download the GRUB configuration file from the network after messing around in the GRUB shell for a while.  So, where did the network connection come from?  That's the mystery.  If I let the iMac boot into the GRUB shell, and just leave it sitting there for a while (30 seconds to 30 minutes), the Ethernet interface in GRUB magically becomes configured with the correct settings!   I was just lucky when I first was able to download the configuration file.  I had waited long enough to have the magic happen first.

The GRUB mailing list post mentioned using the net_bootp command to force this magical configuration.  However, that doesn't work for me.  It fails in one of two ways.  1)  If run without arguments, then it tries to send BOOTP packets out on every network card.  It tries the WiFi card first, and when this fails to send, the net_bootp command aborts without trying the Ethernet card, leaving bad temporary addresses for both cards.  2)  If run with the Ethernet card as an argument, it sends several BOOTP packets out to the network over the card successfully, as seen on the DHCP server.  However, this doesn't make the card magically configured, so the rest of GRUB still can't communicate on the network.

I've had trouble trying to contact live developers for GRUB to see if anyone has insight.  I filed a bug report, but haven't heard anything after a couple of days.  So, I assume there aren't any active developers who are EFI network experts.  If there were, this probably would have been solved by now.

So, it looks like I'm going to be learning DeployStudio for now. :(

Edit:

OK, writing up this message helped to clarify my thinking.  When the "net_bootp efinet0" command is run in the GRUB shell, it actually sends a BOOTREQUEST message to the DHCP server.  The DHCP server logged this message: "BOOTREQUEST from c4:2c:03:1c:68:2b via 144.38.197.1: BOOTP from dynamic client and no dynamic leases".  I have "allow bootp" configured, but they DHCP server doesn't use that to control responses to BOOTREQUEST messages.  Re-reading the man page for dhcpd.conf shows that the "dynamic-bootp" option to the "range" directive enables responses.  Now, the server responds with a BOOTREPLY, which is enough to trigger the DHCP response handling code in the network portion of GRUB.  So, with this configuration, GRUB can be used to Netboot and bootstrap an iMAC!  Yea!

Netbooting Apple clients using linux server

We're trying to unify our client system imaging to one tool.  We've been using FOG for PCs and DeployStudio for MACs.  Now that FOG supports EFI and GPT, it may be possible to use FOG for both.

One of the pre-requisites is to have the clients boot from the network.  PCs use the standard PXE to do this, and is supported by configuring DHCP to tell the PC which server and file to download to start the boot process.  Apple uses a proprietary modified BOOTP protocol named "Netboot".  It turns out that ISC DHCP server is flexible enough to send the correct information to Apple Netboot clients, so they know which server to pull the boot file from and which file to download.

Once the client knows the server and file, it uses TFTP to download the initial boot file.  If that boot file is smart enough to use other networking protocols, then they can be used to download additional support files to complete the boot.

There are several useful websites for helping to configure ISC DHCP to communicate correctly with Apple Netboot clients:

Excellent example configuration file with detailed comments
Older instructions with additional information and links

Using these links, I was able to get several different iMacs to download boot files.  This blog doesn't discuss the creation of valid boot files to send to iMacs.

So, this was a success.

Where is GRUB2 on my disk anyways?

Preparing our client PC system for imaging before Fall semester, we ran into another problem.  The imaging system we use, FOG, recently made some updates to support GRUB2.  This was welcome news, in years past we have had to downgrade to grub-legacy to get the dual boot system working after imaging.  Initial tests produced booting systems.  However, we started having boot problems very soon after that.  It looked like GRUB2 was using the disk differently in our production case than in our test case.

In order to understand and fix the problem, I spent a while with "the google", trying to understand how GRUB2 puts itself onto the disk.  After a lot of time reading several different websites, including GNU's GRUB page, the Wikipedia GRUB article, and ArchLinux's GRUB page, I was learning, but not enough.  I downloaded the source for GRUB2, got psyched up to read some detailed assembly, then found a great post by someone who already had done it.

Here is a summary of what I learned, regarding the placement of GRUB2 on MBR partitioned disks.  The assumption here is that GRUB is installed in the MBR of the disk.

Grub puts the "boot.img" into the MBR.  This is ~440 bytes of boot code that only know enough to load and jump to the first sector (512 bytes) of the "core.img".  So, in this code, there is stored the disk location of 1 sector.  This code is generic, so it can load from anywhere on the disk.  If you really want to know that location, you can reverse engineer but dumping the bytes of your MBR and finding it.  However, this is usually stored in the "embedding area".  This is the set of unused sectors after the MBR and before the first partition.

The first sector of the "core.img" is the "diskboot.img" file.  This code has the responsibility of loading the rest of "core.img" into memory and jumping to it.  Part of the "diskboot.img" sector is a list of sector collections to load, and the order to load them.  Each collection has a beginning sector and a number of sectors to load.  Again, if you want to reverse engineer, you can dump these data from the end of the block and see the location and size of each collection.  These collections are usually also in the "embedding area".

After "core.img" is loaded, it has enough information and code to use the linux file system, instead of using block access.  It loads the GRUB configuration file, and any other GRUB modules necessary to do its job.  These files are usually stored in "/boot/grub".  The beauty of this system, is that you can have rather large amounts of data for GRUB, and you don't have to re-write the bootloader if the files are moved within the file system's data blocks.  Again, the actual location is configurable and can be found from dumping the on-disk copy of "core.img", if you really need to.

OK, for our setup, the problem turned out to be the size of the "embedding area".  FOG assumed it to be 62 sectors in size, and always captured and restored those sectors.  BTW, the Wikipedia article, and most of the online documentation supports that idea.

After using dd and od to examine the contents of "diskboot.img" on my system, I saw that the total size of "core.img" was closer to 100 sectors, but it was stored in the "embedding area".  After wondering how that could work without wiping out the first partition's data, I used sfdisk -d to examine the actual location of the first partition.  It started at sector 2048, not sector 64.  There was plenty of room.

After making minor changes to FOG's scripts, it saves the entire "embedding area", and restores it again.  Presto!  GRUB2 is imaged just fine.

It turns out that Windows 7 puts the first partition 1MB=2048sectors into the disk, making for a larger "embedding area".  However, an Ubuntu only system will put the first partition at sector 64.  GRUB2 is smart enough to know how much space is available.  It puts enough into "core.img" to mount and load the rest.  But it might put more in "core.img" if there is more space.

The initial tests with FOG were Ubuntu only, so the 62 sector "core.img" assumption worked great.  Our production, dual-boot systems we install Windows 7, then Ubuntu, so Windows can be happily in the first 2 partitions.  These weren't working, because we were only copying the first 62 of ~100 sectors.

Thanks again to the FOG developers and support staff, the GNU project, especially the GRUB developers, and the producers of the website that finally gave the information I needed to understand my problem.

MBR Partition Tables

While trying to prepare our client system image to be replicated across the departmental client PCs, we ran into a few snags.  One of them was the imaging system we use for PCs, FOG, didn't support the use of extended and logical partitions.  Unfortunately, this was something I knew a couple of years ago, but had forgotten.  So, we whipped up a patch for FOG so that extended and logical partitions are supported, and the good folks at the project integrated them into the project.

Now, in order to make sure we got it right, I spent a bit of time re-learning the details of MBR partition tables.  Turns out there are a number of good resources describing the details.  The Wikipedia articles listed here were very helpful.  MBR article, extended partition article, partition types article

I'll write a quick summary here.

Note:  I'm discussing disks that use the historical, old, MSDOS partitioning system.  There are many other partitioning schemes, but this is the one I care about right now.

The Master Boot Record (MBR) is the first sector (512 bytes) on the disk.  In addition to boot-time start-up code, there are four 16 byte partition records (64 bytes total) stored in this data.  That's why there can only be 4 primary partitions on a disk.  There is no more room.  Each of the records stores information the kind of partition, the starting location and the ending location of the partition.

To overcome the 4 partition limit, extended partitions were added.  There are 2 common partition types that mean extended.  They differ in the way addresses are represented, but are mostly the same.  The codes are 0x05 and 0x0F.  Only one of the primary partitions can be classified as extended.  The extended partition's space is then available to be divided into logical partitions.

The first sector (512 bytes) of the extended partition, has another MBR-like structure, with slots for defining 4 partitions.  The first entry defines the first logical partition's starting position and size.  The second entry defines the next logical partition's starting position and size.  The third and fourth entries are not used.

Each logical partition also has an MBR-like structure that describes the logical partition itself, and the next logical partition.  The last logical partition shows its next logical partition as all zeros, indicating the end of the list.  Many logical partitions can be represented using this linked-list style approach.

If you care about the actual format and details of the records, read the articles referenced above.

For our purposes, we need to know which sectors contain the partition table structure, how to copy them, and how to restore them.  FOG worked only with primary partitions, because it was only copying the MBR at the front of the disk, not the MBR-like blocks in the extended and logical partitions necessary to re-create the extended and logical partitions.  We need to re-construct all of those elements to get an identical partition table with extended and logical partitions.

We could write a tool to do that, and make sure we get all of the bit-twiddling correct, and send it through a testing cycle to get all the bugs worked out.  Or, we could use software that has already been vetting by millions of users, even if they don't know it.

The sfdisk command can read the partition table and display a straightforward and precise textual representation:

$ sudo sfdisk -d
# partition table of /dev/sda
unit: sectors
/dev/sda1 : start=     2048, size=   204800, Id= 7, bootable
/dev/sda2 : start=   206848, size=390623232, Id= 7
/dev/sda3 : start=390830080, size= 15624192, Id=82
/dev/sda4 : start=406454272, size= 72060928, Id= f
/dev/sda5 : start=406456320, size= 72058880, Id=83

This format can then be used to create a new partition table with the same structure.  sfdisk does it for us.

Thanks to the FOG developers and support team, the Wikipedia contributors, and software developers that made this possible.