Bye Bye DVD

The DVD is a comparatively long-lasting (50+ years) optical storage media. Yet in today's notebooks, you will rarely find a built-in DVD drive. This may be related to the unpleasant form factor, considering the DVD's capacity of about 4GB single-sided or 8GB double-sided, which offers much less storage capacity than a modern tiny microSD card. For booting from removable media, the easily portable USB flash pen with a practical keyring connector has long been a favorite for most users instead of the CD or DVD. So, wouldn't it be appropriate for Knoppix to just give up on the old-fashioned DVD and start distributing only flash disk images instead?

As much as this seems to be a reasonable step, there are still pros and cons to both formats (Table 1).

Probably the most important issue is the easy handling of distributed images. A DVD ISO, containing the ISO 9660/Rock Ridge/Joliet filesystem, is always exactly as big as the included (unpartitioned) data; there is no adjustment needed when burning.

A hard disk or flash disk image, however, contains a partition table with fixed values for sector starts and ends of partitions. An image with a partition table for a 7.8GB drive will cause data corruption sooner or later when installed on a 7.6GB drive, since the filesystem of the last partition extends beyond the physically available space. At least, the last partition would have to be expanded or shrunk to match the physical end of data. Installing a 8GB image on a 1TB hard disk would leave the remaining unpartitioned space unused. It's quite inflexible.

The flash-knoppix tool circumvents this limitation when creating a bootable flash disk by first partitioning the target disk for using all available space and then creating filesystems that match the partitioning before copying over all files.

Volume Records vs. Disk Partition Tables

By its standard, the ISO 9660 filesystem used for DVDs does not use the first 32KB for anything; these bytes are usually set to zero (Figure 1), after which a volume descriptor is located at offset 0x8800 that starts with CD001.

Figure 1: First 32KB in ISO 9660 filesystem are set to zero.

Figure 2 shows the El Torito specification [1] located after the first 32KB set to zero and indicates the boot system ID.

Figure 2: Offset volume descriptor.

Several of these volume descriptors can be found in the ISO 9660 image, but this one is also a boot record, which contains the boot catalog's address encoded in little-endian format at byte 71 to 74 from the records offset. This referenced boot catalog then points to the isolinux boot loader, which loads the kernel and initial ramdisk.

As you can see, the procedure of the computer's BIOS booting the ISO 9660/El Torito way is a little more complex than booting from a Master Boot Record (MBR) disk.

Now just doing a 1:1 copy of the ISO image to flash disk will not create a bootable medium, since the BIOS does not even attempt to search for ISO volume descriptors or boot records for media other than CD or DVD. Additionally, the standard sector size of a CD/DVD is specified as 2048 bytes, whereas for disks the sector size is 512 bytes or 4096 bytes, so all offsets given in blocks or sectors would be wrong.

When booting with the MBR method from flash or hard disk (Figure 3), the BIOS will read the first 446 bytes of the disk (starting at sector  ). These first bytes contain a very tiny boot code that usually just reads the partition table at byte 446 and redirects to the next boot record at the start of a system partition marked as bootable, or to an offset that contains a secondary stage boot loader.

Figure 3: Booting with MBR from a flash disk.

Right after the boot code, from byte 446 to 511, the MS-DOS style partition table follows, which defines the four primary partition offsets, types and sizes, ended by a 0x55AA signature (which may be used by the BIOS to check if this really looks like a standard partition table).

The partition table at offset 1BE results in the following partitioning shown with fdisk -l (Figure 4).

Figure 4: Partition table at offset 1BE.

Merging ISO 9660 El Torito and MBR Booting

As mentioned before, a USB flash disk will usually not become bootable by just copying the DVD's ISO image over it. The most apparent solution is adding an MBR with a boot code and a partition table in the first 512 bytes of the image, which are not used by the ISO 9660 filesystem anyway. The result is a hybrid (El Torito + MBR) booting system that can be burned to DVD or flashed to a disk with no changes.

For Knoppix, the hybrid image partition table (Figure 5), which is injected into the first sector of the image, contains three partitions. The first "empty" one is the original ISO 9660 filesystem; the other two will be explained in the next sections.

Figure 5: The hybrid image partition table for Knoppix contains three partitions.

The boot code is created from the isolinux hybrid boot loader, which comes as isohdpfx.bin with isolinux. For actually creating the image, I use xorriso.

After many experiments, the final command line for creating the final Knoppix image can be seen in Listing 1.

In theory, the steps of inserting additional information into the image could have been done with dd and slices of binaries, but using xorriso is just more convenient.

Most Linux distros already use this method in their installers and live images in one way or the other. However, there are some disadvantages in this approach of an "all-in-one" image, which made me hesitate to switch to the hybrid image format until now:

• When flashing to disk instead of burning a DVD, the first partition containing the ISO 9660 filesystem will be read-only, simply because ISO 9660 does not support any write operations regardless of writeability of the installed medium.
• Some BIOSes get confused when they see a partition table (or anything but zero bytes) in the first sector of a DVD and refuse to even display it in the boot menu.
• The additional partition table in the first sector may be interpreted with the sector size of 2048 bytes instead of the standard 512 bytes of the DVD when trying to just mount it from a drive.
• The ISO 9660 filesystem starts at sector   (which is the same sector where the MBR and partition table are located). When inserting a partition entry for mounting the ISO, this also may confuse some software that assumes that a partition can only start at sectors higher than  . In fact, the ISO filesystem really overlaps the MBR and partition table, which are embedded in the unused first blocks of the filesystem.
• Since the partition offsets and sizes specified in the partition table have to be set to a fixed number, they will most likely never exactly match the physical size of the target disk, and they are also not aligned to "cylinders" (which causes warnings in some partitioning programs).

Consequently, some computers will no longer boot the DVD after switching to hybrid boot format.

Compatibility?

The big question for me was (and still is): How many computers are affected by not being able to boot the new format. Sure, refusing to boot a DVD that still follows the valid standard for El Torito booting can be considered a bug in the BIOS (aka, "It's not my fault…").

However, I'd rather renounce new features than lose the capability to boot on as many different computers as possible. Also, it is not transparent to users why a Knoppix 7 DVD booted without problems, while version 8 won't even show up in the boot menu anymore, in those (hopefully) rare cases.

On the other hand, the hybrid format finally allows distributing an image that works (on most computers) for DVDs, as well as USB flash disks, by just writing the same data. It seems to me that it's unavoidable in the long run because of the disappearance of DVD drives from modern computers.