iMac (late-2013) Triple boot Mac OSX, Windows 7, Ubuntu Linux

In our lab and classroom setup, we have about 50% iMac hardware, the other 50% being from the Dell Optiplex series.  Our iMacs are as new as late-2013 models, and as old as mid-2011 models.  Our OS X is Mavericks (OS X 10.9.4); Windows is Windows 7, SP1; Linux is Ubuntu 14.04 (Trusty).

The PC hardware is used in Windows 7 and Ubuntu Linux dual-boot.  And users will choose the OS based on the coursework at hand.  The iMac hardware is used mostly in OS X.  However, there are times when the Windows 7 and Ubuntu Linux functionality is needed on the iMac hardware.

In creating the master iMac install image on a new late-2013 model iMac, several problems were encountered.  There are posts on blogs and forums, and all over the 'net that describe how to create a triple boot Mac machine.  Some are better than others, but none was completely what I was looking for.  Here are some of the best I drew from to create our solution.

discussions.apple.com: How to triple boot

discussions.apple.com: Partitioning disk for triple boot

GPT fdisk tool

support.apple.com: Late-2013 iMac specific help

rEFInd Boot Manager

technet.microsoft.com: Using Dism to add drivers to Windows .wim (Windows 8 version)

technet.microsoft.com: Using Dism to add drivers to Windows .wim (Windows 7 version)

arstechnica.com: Make bootable USB install/recovery drive for OS X 10.9 (Mavericks)

Sorry, but I got carried away.  This is a long post.  I just didn't want to lose the information between now and next year when I'll need to do this again.

The Issues

1. Apple uses EFI 1.10.
2. This implies that the partitioning scheme is GPT.
3. Windows 7 can install to GPT, but only in UEFI mode (EFI 2.x).  Note the mismatch with Apple EFI version.  So, on Apple hardware, Windows must install in MBR/BIOS/CSM mode.
4. The disk must be GPT partitioned, with a "hybrid MBR" partition table to fool Windows 7 installer and installed OS to use the correct portion of the disk.
5. Apple hardware requires drivers that aren't present in the default installer for Windows 7.  In particular, late-2013 model iMac has USB3, and those drivers are not present.  This makes the installation and setup process unable to use the keyboard or mouse.
6. Apple provides Boot Camp Assistant to help in the setup for installing Windows and providing the Apple specific drivers.  BCA has several limitations, that caused this system not to work out of the box.  
7. The lack of USB3 drivers prevent the Windows installer from finding the USB3 drivers from a USBstick or USB DVD drive, during the installation.  Catch 22.  
8. BCA also only allows partitioning a disk that has a single HFS+ partition into 2 partitions, one for OSX and one for Windows.  Apple's Disk Util won't resize them later and add partitions, without breaking things.

The Solutions

1. We have to deal with EFI 1.10 limitations
2. GPT partitioning is great.  We will have to deal with more flexible hybrid MBR tables though.
3. BCA helps build a USB stick that can boot from the Apple Boot Manager.  After Windows is installed, it will boot find from the Apple Boot Manager, or using rEFInd, or other boot manager.
4. GPT fdisk, will allow us to create the more flexible hybrid MBR table that is consistent with the GPT table, so all OSes use the disk consistently.
5. BCA will provide the Apple specific Windows drivers.  The Dism tool in Windows will add them to the setup and installation sources on the USB stick.
6. BCA will only be used in limited capacity, to work around its limitations.  But its useful features will be used.
7. The Windows tool, Dism, will be used to add Apple specific drivers to the setup and installation sources.  No more chicken and egg problem.
8. GPT fdisk and Apple Disk Util will be used to create the desired set and sizes of partitions.

The Process

Here is the process that I used.  If you don't like to use the command line, I'm sorry.  This process will require you to use the terminal in OS X, the cmd prompt in Windows, and the shell in the Linux installer.  If there was a GUI that did all of this, we wouldn't need to document these issues.

This is the outline of the process.  Those parts of the process that require more details are documented below this section.

Note that I inserted the USB sticks in the back of the iMac, not into the keyboard USB extension.  It makes a difference, particularly when booting the Windows and Linux installers.

CAUTION.  I used a system that didn't have any valuable data on it.  This process will destroy the data on the disk, and on the 3 USB sticks.  This is not for "upgrading" an existing system. CAUTION.

1. Create a USB stick installer for OS X.
2. Boot the installer.  Use Disk Util to create the number, order and size of partitions desired for the final system.  Our setup has a Windows partition, a "Sharing" partition, a OS X partition, a Linux partition, and a Linux swap partition.  In that order.  Set all partitions to be ExFAT type, except for the OS X partition, it should be Mac OS X Extended (Journaled).
3. Install a fresh copy of OS X on the system, using the installer app on the USB stick. Reboot into the installed OS, and complete all software updates.
4. Install GPT fdisk on OS X system.  Use GPT fdisk to create the hybrid MBR that protects the non-windows accessible disk space, and exposes the Windows and Sharing partitions.  The Windows partitions should have the boot flag set.  Reboot OS X.
5. Put a USB stick in the system.  Use the Disk Util to erase its contents and replace with an empty FAT file system.
6. Obtain an iso of Windows 7.  (I used Windows 7 Enterprise, SP1), copy the iso to the OS X system.
7. Launch BCA.  Ignore the message that says BCA can't work because there are too many partitions.  Check first 2 options to install Windows 7 installer on the USB stick, and download the drivers and other software to the USB stick.  Safely eject the USB stick.
8. On an existing Windows system, use Dism to add the USB3 and graphics drivers to all of the .wim images on the USB stick (sources/boot.wim and sources/install.wim).  Also, add the other Apple hardware drivers to the installation .wim image (sources/install.wim). Safely remove the USB stick from Windows, and insert back in iMac.
9. Reboot iMac, holding the option/alt key at boot time to interact with the Apple Boot Manager. Select the windows option that represents the USB stick installer.
10. During the Windows installation, when selecting an installation destination, be sure to choose the correct MBR partition number.  Format it so that NTFS will be correctly used for installation.
11. Every time the Windows installer reboots, use the option/alt key to enter the boot manager.  This time select the option that represents the internal drive partition of Windows.  This should be named whatever you chose in the Disk Util.  Leave the Windows installer USB stick in throughout the installation process.
12. At the end of the Windows installation process, Boot Camp installer launches and installs Apple software updaters, etc.
13. Now, remove the USB stick with the Windows installer, and you have a dual boot system.  Use the option/alt key at boot time to activate the boot manager and select the operating system of choice.
14. Create a bootable USB stick with the Ubuntu installer.  I used the mkusb script, and the Ubuntu mini.iso available at your favorite Ubuntu mirror in (/ubuntu/dists/trusty/main/installer-amd64/current/images/netboot/mini.iso).  I image you could use another iso file.  Use the command line: sudo ./mksub mini.iso on a linux system.  Be sure you know which device is your USB stick, or you might wipe out your system.
15. Insert the Ubuntu installer USB stick into the iMac, reboot, and use the boot manager to select the installer.  The Apple boot manager will say "WINDOWS" as the title for the USB stick.  Don't worry, it's not Windows.  Select it, and do your installation.
16. When choosing the disk partition(s) to use, be careful, the GPT partition table is presented.  Don't delete or create partitions.  Just use and format the ones you reserved.  Note that this will still re-write the partition table, because the partition types will change for the ones used.
17. Don't use the GRUB installation option from the menu.  Instead, drop to shell, and manually install GRUB for EFI.  See details below.
18. Fix the hybrid MBR.  In chroot'd shell, apt-get install gdisk;  Follow same partitioning options as before to recreate the hybrid MBR.  (Maybe using gdisk on OS X to set the types of the Linux and swap partitions in the first place would have avoided this need.)
19. Select "Continue without boot loader," and "Finish the installation."  Ubuntu still isn't available from the Apple boot manager, but don't worry.
20. Test that both OS X and Windows still boot fine.
21. Boot into OS X, install and configure rEFInd, so that all 3 OSes are available at boot time.

Details

Making USB Stick OS X Installer

1. Boot an OS X system.  Download and install the OS X installer from the App Store.
2. Insert a USB stick that will be erased and used to create the installer.  8 GByte minimum.
3. Format the USB stick using Disk Util as "Mac OS Extended (Journeled)
4. Create the installer from the command line.  This command line assumes the location of the OS X installer app, and the name of the USB stick volume.  Please verify those 2 values before running this from the terminal. 
5. sudo /Applications/Install\ OS\ X\ Mavericks.app/Contents/Resources/createinstallmedia --volume /Volumes/OSXInstaller --applicationpath /Applications/Install\ OS\ X\ Mavericks.app --nointeraction
6. Safely eject the USB stick.

Using GPT fdisk to Create Hybrid MBR Table

1. Be sure you know you disk device name.  It should be /dev/disk0, but check using the df command.
2. Start GPT fdisk:  sudo gdisk /dev/disk0
3. Examine current GPT: p
4. You should see more partitions than expected.  The EFI System Partition is hidden from you in Disk Util.  Also, a new "Recovery HD" partition was added when installing OS X.  Examine the positions, names, and types of the partitions to be sure they make sense.  Note the numbers of the Windows and Sharing partitions.  If you don't have a Sharing partition, that's fine.  However, these partitions should be adjacent on the disk.
5. Enter the "recovery and transformation" command mode: r
6. Examine the current hybrid MBR, just for reference: o
7. Create a new hybrid MBR: h
8. Enter the GPT partition numbers to add (Windows and Sharing): 2 3
9. Do put an 0xEE partition to protect the front of the disk: y
10. For each partition, accept the type of 0x07 (NTFS): 07
11. For the Windows partition, make bootable: y
12. For the other partitions, don't make bootable: n
13. Now, protect the rest of the disk with another partition: y
14. I used type 0x80 (minix), because 0xEE did confuse OS X: 80
15. Use the "o" and "p" options to make sure things make sense.
16. Finally, save the changes to disk: w
17. Reboot system immediately following.

Installing Apple Drivers on Windows Installer

1. There are two links above for how to use the Dism tool.  The one to use depends on the OS you are using to patch the Windows 7 installer.  One is for Windows 7, the other is for Windows 8.  I used Windows 7, so my examples will show that.
2. Boot your existing Windows system.  Insert the Windows 7 Installer USB stick created by BCA.
3. From the USB stick, copy sources/boot.wim and sources/install.wim from the USB stick to the disk.  For example, I put them on my Desktop.  My USB stick was mounted as drive K.
4. Open an privileged cmd prompt (Run as administrator).
5. List the images in the .wim file with: Dism /Get-ImageInfo /ImageFile:C:\Users\myuser\Desktop\boot.wim
6. Note the name of the images.
7. My boot.wim file had 2 images, and install.wim had 1 image.  I installed the USB3 and Graphics drivers from BootCamp into all images.  I installed ALL of the BootCamp drivers in the install.wim image.
8. Make a directory to mount the images on: mkdir C:\offline
9. One at a time, I processed each of the images with these steps:
10. Mount a particular image in a .wim file: Dism /Mount-Image /ImageFile:C:\Users\myuser\Desktop\boot.wim /Name:"Microsoft Windows Setup (x64)" /MountDir:C:\offline
11. Be sure you put the name in quotes, and get it right.  That's the selection mechanism within the .wim file.
12. Add all of the USB3 and graphics drivers: Dism /Image:C:\offline /Add-Driver /Driver:"K:\$WinPEDrivers$" /Recurse
13. If its an install image, add ALL drivers: Dism /Image:C:\offline /Add-Driver /Driver:"K:\BootCamp\Drivers" /Recurse
14. Unmount the image, saving changes: Dism /Unmount-Image /MountDir:C:\offline /Commit
15. Copy both boot.wim and install.wim back to the sources directory on the USB stick.
16. Safely eject the USB stick.

GRUB EFI Installation

The grub-pc package is for BIOS/MBR.  We don't want it, instead we need the grub-efi package.  The Ubuntu installer doesn't have an option from the menu for this yet.  So, we'll do this during the installation, instead of "Install the GRUB boot loader on a hard disk."

1. Use alt-F2 or "Execute a shell", to obtain a shell prompt
2. chroot /target
3. apt-get update; apt-get install grub-efi-amd64
4. mkdir /EFI; mount /dev/sdb1 /EFI
5. grub-install --target x86_64-efi --efi-directory /EFI
6. update-grub
7. umount /EFI
8. Exit the shell

Building GRUB for iMac Netboot EFI

GRUB2 Built on Ubuntu 14.04 to use in iMac Netboot via EFI

Steps:

1. download grub
2. install packages to build
3. create configure script
4. configure for efi 64bit
5. patch grub to automatically configure network on iMac efi
6. make grub
7. create bootstrap grub.cfg to load the real configuration from the network
8. build efi netboot grub image
9. create full grub.cfg to be pulled from server by grub
10. install grub image, grub.cfg on TFTP server

See the attached files for more details.

Script to document the steps
git-diff output for patches to GRUB

Netbooting Apple Clients with ipxe or grub

Still trying to image Apple clients (iMacs) with FOG.  In a previous post I discussed setting up the DHCP server to send boot files to the iMac via the Netboot protocol.

FOG sends iPXE to the PCs to control the boot process during imaging.  We can't just use the same iPXE file for the iMacs, because the PCs are booting using legacy BIOS and the iMacs are using EFI.  iPXE supports EFI, so we built an iPXE for EFI file to send to the iMAC.  It is downloaded, but when the iMac tries to run it, the apple loader seems to choke on it, then boots to the default (harddrive).  The same iPXE EFI file runs fine on the iMac if you install it on the hard drive or a USB drive and run it using rEFIt or rEFInd.  I was unable to figure out why there is a difference, and was unable to attract any attention from the iPXE developers.  Without any kind of output from the netboot loaded image, I didn't see how to start debugging the process.

After some googling, I found these links that indicated that GRUB2 might be able to do what we wanted.

iPXE forums
GRUB mailing list

So, I downloaded the GRUB source and built a 64-bit GRUB EFI binary to send to the iMac.  If you do this, be sure the follow the instructions in the GRUB mailing list post.  So, when the mac loaded the grub image, it dropped to the grub shell.  Awesome, if you ask me.

After messing around in the shell a bit, I was able to get GRUB to load the grub.cfg from the network server, and present the menu.  I thought I was almost home.  Wrong. :(

From the instructions, I was expecting the network config file to be loaded automatically.  I need it to load automatically, if this is going to be useful for imaging 50+ macs.

After 2+ days of messing in the GRUB shell and source code, I've found that, at least on our iMacs, there is a problem with the EFI network interface.  BTW, the modularity of the GRUB code is very nice.

Here's the problem as I see it now.  When GRUB loads, it does all of it's initialization code, including detecting network cards and configuring GRUB's view of those cards, by pulling information from EFI using efi API calls.  Well, the cards (yes, both Ethernet and WiFi) are found.  However, the WiFi card wasn't connected to an AP, so it fails the configuration detection, as expected.  And the Ethernet card configuration detection fails when trying to open the PXE protocol.  This failure isn't surprising since the iMac doesn't support PXE.  Thus, GRUB has no network connection.

Mysteries still exist.  Remember that I was able to download the GRUB configuration file from the network after messing around in the GRUB shell for a while.  So, where did the network connection come from?  That's the mystery.  If I let the iMac boot into the GRUB shell, and just leave it sitting there for a while (30 seconds to 30 minutes), the Ethernet interface in GRUB magically becomes configured with the correct settings!   I was just lucky when I first was able to download the configuration file.  I had waited long enough to have the magic happen first.

The GRUB mailing list post mentioned using the net_bootp command to force this magical configuration.  However, that doesn't work for me.  It fails in one of two ways.  1)  If run without arguments, then it tries to send BOOTP packets out on every network card.  It tries the WiFi card first, and when this fails to send, the net_bootp command aborts without trying the Ethernet card, leaving bad temporary addresses for both cards.  2)  If run with the Ethernet card as an argument, it sends several BOOTP packets out to the network over the card successfully, as seen on the DHCP server.  However, this doesn't make the card magically configured, so the rest of GRUB still can't communicate on the network.

I've had trouble trying to contact live developers for GRUB to see if anyone has insight.  I filed a bug report, but haven't heard anything after a couple of days.  So, I assume there aren't any active developers who are EFI network experts.  If there were, this probably would have been solved by now.

So, it looks like I'm going to be learning DeployStudio for now. :(

Edit:

OK, writing up this message helped to clarify my thinking.  When the "net_bootp efinet0" command is run in the GRUB shell, it actually sends a BOOTREQUEST message to the DHCP server.  The DHCP server logged this message: "BOOTREQUEST from c4:2c:03:1c:68:2b via 144.38.197.1: BOOTP from dynamic client and no dynamic leases".  I have "allow bootp" configured, but they DHCP server doesn't use that to control responses to BOOTREQUEST messages.  Re-reading the man page for dhcpd.conf shows that the "dynamic-bootp" option to the "range" directive enables responses.  Now, the server responds with a BOOTREPLY, which is enough to trigger the DHCP response handling code in the network portion of GRUB.  So, with this configuration, GRUB can be used to Netboot and bootstrap an iMAC!  Yea!

Netbooting Apple clients using linux server

We're trying to unify our client system imaging to one tool.  We've been using FOG for PCs and DeployStudio for MACs.  Now that FOG supports EFI and GPT, it may be possible to use FOG for both.

One of the pre-requisites is to have the clients boot from the network.  PCs use the standard PXE to do this, and is supported by configuring DHCP to tell the PC which server and file to download to start the boot process.  Apple uses a proprietary modified BOOTP protocol named "Netboot".  It turns out that ISC DHCP server is flexible enough to send the correct information to Apple Netboot clients, so they know which server to pull the boot file from and which file to download.

Once the client knows the server and file, it uses TFTP to download the initial boot file.  If that boot file is smart enough to use other networking protocols, then they can be used to download additional support files to complete the boot.

There are several useful websites for helping to configure ISC DHCP to communicate correctly with Apple Netboot clients:

Excellent example configuration file with detailed comments
Older instructions with additional information and links

Using these links, I was able to get several different iMacs to download boot files.  This blog doesn't discuss the creation of valid boot files to send to iMacs.

So, this was a success.

Where is GRUB2 on my disk anyways?

Preparing our client PC system for imaging before Fall semester, we ran into another problem.  The imaging system we use, FOG, recently made some updates to support GRUB2.  This was welcome news, in years past we have had to downgrade to grub-legacy to get the dual boot system working after imaging.  Initial tests produced booting systems.  However, we started having boot problems very soon after that.  It looked like GRUB2 was using the disk differently in our production case than in our test case.

In order to understand and fix the problem, I spent a while with "the google", trying to understand how GRUB2 puts itself onto the disk.  After a lot of time reading several different websites, including GNU's GRUB page, the Wikipedia GRUB article, and ArchLinux's GRUB page, I was learning, but not enough.  I downloaded the source for GRUB2, got psyched up to read some detailed assembly, then found a great post by someone who already had done it.

Here is a summary of what I learned, regarding the placement of GRUB2 on MBR partitioned disks.  The assumption here is that GRUB is installed in the MBR of the disk.

Grub puts the "boot.img" into the MBR.  This is ~440 bytes of boot code that only know enough to load and jump to the first sector (512 bytes) of the "core.img".  So, in this code, there is stored the disk location of 1 sector.  This code is generic, so it can load from anywhere on the disk.  If you really want to know that location, you can reverse engineer but dumping the bytes of your MBR and finding it.  However, this is usually stored in the "embedding area".  This is the set of unused sectors after the MBR and before the first partition.

The first sector of the "core.img" is the "diskboot.img" file.  This code has the responsibility of loading the rest of "core.img" into memory and jumping to it.  Part of the "diskboot.img" sector is a list of sector collections to load, and the order to load them.  Each collection has a beginning sector and a number of sectors to load.  Again, if you want to reverse engineer, you can dump these data from the end of the block and see the location and size of each collection.  These collections are usually also in the "embedding area".

After "core.img" is loaded, it has enough information and code to use the linux file system, instead of using block access.  It loads the GRUB configuration file, and any other GRUB modules necessary to do its job.  These files are usually stored in "/boot/grub".  The beauty of this system, is that you can have rather large amounts of data for GRUB, and you don't have to re-write the bootloader if the files are moved within the file system's data blocks.  Again, the actual location is configurable and can be found from dumping the on-disk copy of "core.img", if you really need to.

OK, for our setup, the problem turned out to be the size of the "embedding area".  FOG assumed it to be 62 sectors in size, and always captured and restored those sectors.  BTW, the Wikipedia article, and most of the online documentation supports that idea.

After using dd and od to examine the contents of "diskboot.img" on my system, I saw that the total size of "core.img" was closer to 100 sectors, but it was stored in the "embedding area".  After wondering how that could work without wiping out the first partition's data, I used sfdisk -d to examine the actual location of the first partition.  It started at sector 2048, not sector 64.  There was plenty of room.

After making minor changes to FOG's scripts, it saves the entire "embedding area", and restores it again.  Presto!  GRUB2 is imaged just fine.

It turns out that Windows 7 puts the first partition 1MB=2048sectors into the disk, making for a larger "embedding area".  However, an Ubuntu only system will put the first partition at sector 64.  GRUB2 is smart enough to know how much space is available.  It puts enough into "core.img" to mount and load the rest.  But it might put more in "core.img" if there is more space.

The initial tests with FOG were Ubuntu only, so the 62 sector "core.img" assumption worked great.  Our production, dual-boot systems we install Windows 7, then Ubuntu, so Windows can be happily in the first 2 partitions.  These weren't working, because we were only copying the first 62 of ~100 sectors.

Thanks again to the FOG developers and support staff, the GNU project, especially the GRUB developers, and the producers of the website that finally gave the information I needed to understand my problem.

MBR Partition Tables

While trying to prepare our client system image to be replicated across the departmental client PCs, we ran into a few snags.  One of them was the imaging system we use for PCs, FOG, didn't support the use of extended and logical partitions.  Unfortunately, this was something I knew a couple of years ago, but had forgotten.  So, we whipped up a patch for FOG so that extended and logical partitions are supported, and the good folks at the project integrated them into the project.

Now, in order to make sure we got it right, I spent a bit of time re-learning the details of MBR partition tables.  Turns out there are a number of good resources describing the details.  The Wikipedia articles listed here were very helpful.  MBR article, extended partition article, partition types article

I'll write a quick summary here.

Note:  I'm discussing disks that use the historical, old, MSDOS partitioning system.  There are many other partitioning schemes, but this is the one I care about right now.

The Master Boot Record (MBR) is the first sector (512 bytes) on the disk.  In addition to boot-time start-up code, there are four 16 byte partition records (64 bytes total) stored in this data.  That's why there can only be 4 primary partitions on a disk.  There is no more room.  Each of the records stores information the kind of partition, the starting location and the ending location of the partition.

To overcome the 4 partition limit, extended partitions were added.  There are 2 common partition types that mean extended.  They differ in the way addresses are represented, but are mostly the same.  The codes are 0x05 and 0x0F.  Only one of the primary partitions can be classified as extended.  The extended partition's space is then available to be divided into logical partitions.

The first sector (512 bytes) of the extended partition, has another MBR-like structure, with slots for defining 4 partitions.  The first entry defines the first logical partition's starting position and size.  The second entry defines the next logical partition's starting position and size.  The third and fourth entries are not used.

Each logical partition also has an MBR-like structure that describes the logical partition itself, and the next logical partition.  The last logical partition shows its next logical partition as all zeros, indicating the end of the list.  Many logical partitions can be represented using this linked-list style approach.

If you care about the actual format and details of the records, read the articles referenced above.

For our purposes, we need to know which sectors contain the partition table structure, how to copy them, and how to restore them.  FOG worked only with primary partitions, because it was only copying the MBR at the front of the disk, not the MBR-like blocks in the extended and logical partitions necessary to re-create the extended and logical partitions.  We need to re-construct all of those elements to get an identical partition table with extended and logical partitions.

We could write a tool to do that, and make sure we get all of the bit-twiddling correct, and send it through a testing cycle to get all the bugs worked out.  Or, we could use software that has already been vetting by millions of users, even if they don't know it.

The sfdisk command can read the partition table and display a straightforward and precise textual representation:

$ sudo sfdisk -d
# partition table of /dev/sda
unit: sectors
/dev/sda1 : start=     2048, size=   204800, Id= 7, bootable
/dev/sda2 : start=   206848, size=390623232, Id= 7
/dev/sda3 : start=390830080, size= 15624192, Id=82
/dev/sda4 : start=406454272, size= 72060928, Id= f
/dev/sda5 : start=406456320, size= 72058880, Id=83

This format can then be used to create a new partition table with the same structure.  sfdisk does it for us.

Thanks to the FOG developers and support team, the Wikipedia contributors, and software developers that made this possible.