Klaus Knopper answers your Linux questions

Ask Klaus

Article from Issue 180/2015
Author(s):

Klaus Knopper answers your Linux questions.

Cloning Live System USB Flash Drives

Dear Klaus,

First, I want to thank you once again – this time for solving my problem with 3G connectivity in issue 178 [1]. Knoppix 7.5.0 now runs like a charm on my old Eee PX 901Go, from which I could finally drop the way-too-old Xandros I still used because of the 3G issue.

I have another question: As I have made already a few USB sticks for friends with Knoppix 7.5.0, I found myself always doing the same postinstall jobs (e.g., adding a few programs, changing the keyboard default layout, adding plugins to the web browser, …). It would be nice if I could just make a new clone of such a modified Knoppix 7.5.0 that includes all those changes.

I tried to copy one stick to another on a Linux machine using

dd if=/dev/<modified_stick> of=/dev/<stick_for_friend>

but this almost always failed to work, even when both sticks were the same type and make. I also tried to recreate the ReiserFS you use on the target stick, but that also got me in trouble.

Could you describe how to accomplish this the right way? Maybe it is possible to add this procedure as a new option in the next version of Knoppix – something like Install personalized KNOPPIX to flash disk?

Kind regards, Hugo Coolens

 

Hello again, Hugo,

Your question touches on the general problem of properly cloning an installed operating system to another medium of different size, which gives me an occasion to discuss the topic here. :-)

Some Knoppix downloaders might wonder why I still distribute Knoppix as a DVD or CD ISO image, even though those drives are getting rare nowadays. The reason is simply because these images are still easier to handle on the side of the computer creating the installed medium than are those for hard disks or flash drives. CD- and DVD-ROMs, at least by standard, do not have a partition table that could cause trouble at bootup, and CD/DVD burning programs can always handle images that are smaller than the theoretical maximum size of the medium.

The script flash-knoppix included in Knoppix can create USB flash drives from a Knoppix ISO image, without having to burn a DVD first, just by appending the name of the ISO image to the command.

Flash drive images are more difficult. Some BIOSes refuse to boot from pen drives with "incorrect" partition tables. It is mostly OK to have a partition table that does not cover all the available space, but partition tables that claim that the device capacity is larger than it really is will cause trouble – ranging from write errors, by attempting to access blocks outside of the physical limit, to non-bootability. Even with the same brand and given size of a flash disk, the real capacity might be less by just a few kilobytes because of a varying reserve for defect block management; therefore, if you buy an "8GB" drive, it could be anything between 6.5 and 7.5GB of usable capacity. The reserved bytes are not visible, nor accessible, without reprogramming the drive's internal controller.

Listing 1 shows two so-called 8GB flash drives of the same brand from the same manufacturer. Note the difference in the usable capacity and, thus, the different perfect partitioning. You actually have a 50/50 chance that the target drive matches or exceeds the capacity of the source drive. Only if both drives are exactly equal do you get a perfect copy.

Listing 1

Output of "fdisk -l" for Two Flash Pen Drives

knopper@knoppix:~$ fdisk -l /dev/sdd
Disk /dev/sdd: 7,6 GiB, 8103395328 bytes, 15826944 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xc41ed7aa
Device     Boot   Start      End Sectors  Size Id Type
/dev/sdd1  *       2048  9170943 9168896  4.4G  c W95 FAT32 (LBA)
/dev/sdd2       9170944 15826943 6656000  3.2G 83 Linux
knopper@knoppix:~$ fdisk -l /dev/sdc
Disk /dev/sdc: 7,5 GiB, 8022654976 bytes, 15669248 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x647bbfb5
Device     Boot    Start      End  Sectors  Size Id Type
/dev/sdc1  *        2048 10049535 10047488  4.8G  c W95 FAT32 (LBA)
/dev/sdc2       10049536 15669247  5619712  2.7G 83 Linux

If you do a 1:1 copy with dd from an 8GB to a 16GB drive, the 16GB drive will usually work fine, and you could even extend the last partition to cover the rest of the drive. For Knoppix, this would be an option to provide a "minimum size" flash drive image of around 4.5GB that can, after being copied to an 8GB drive, expand the almost-zero-sized last partition to reach the full capacity of the drive. I'll have to think about this variant again when programs for flashing pen drives with an image are included and become a standard in other operating systems someday.

Unfortunately, it's difficult to resize an installed OS "on the fly" by copying just the data stream, because the stored filesystem information needs to be smaller or equal to the partition size. Otherwise, you will get an invalid or damaged filesystem on the target when its last partition is smaller than that of the source drive – even if you have enough "free space" available at the end of the filesystem – and a truncation is possible.

The flash drive installation script flash-knoppix solves that problem by first partitioning the target drive perfectly (utilizing the full drive capacity), creating a new FAT32 filesystem on the first partition and a ReiserFS filesystem on the second partition, then copying over the source drive content file by file. Then, a master boot record is put on the first sector of the target device; a Syslinux partition boot record will transfer control to the Syslinux bootloader when booting from the device.

Transferring system and personal configurations to the target device is not an option, yet. Only the bootloader configuration in boot/syslinux/*.cfg is copied to the target by default. If you also want to transfer the content of the overlay partition after flashing the drive, you could mount the empty filesystem (e.g., target drive /dev/sdc here),

mount /media/sdc2

and then copy all the changed files from the running system to the target with:

sudo cp -av /KNOPPIX-DATA/* /media/sdc2/

Be aware that this includes all your personal settings and WLAN and website passwords stored in your home directory in the current session! For this reason, I have not yet included the option in the flash-knoppix script; it would be just to easy to give away personal secrets and passwords when your only intention is to copy a working system for friends.

You could exclude /KNOPPIX-DATA/home from your copy command with

rsync -Hav --exclude=/home/ /KNOPPIX-DATA/* /media/sdc2/

instead. Still, you might have private information in /etc, like system-wide password and network settings, that you've forgotten.

Infos

  1. "Ask Klaus!" by Klaus Knopper, Linux Pro Magazine, issue 178, September 2015, pg. 48, http://www.linuxpromagazine.com/Issues/2015/178/Ask-Klaus

The Author

Klaus Knopper is an engineer, creator of Knoppix, and co-founder of LinuxTag expo. He works as a regular professor at the University of Applied Sciences, Kaiserslautern, Germany. If you have a configuration problem, or if you just want to learn more about how Linux works, send your questions to: mailto:klaus@linux-magazine.com.

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • Ask Klaus!

    New fdisk behavior, booting Knoppix from USB, and using PHP mail() in Apache.

  • Professor Knopper's Lab – Hybrid Image Mode

    As DVD drives disappear from modern notebook computers, Knoppix offers a hybrid image mode that works for both DVDs and flash drives.

  • Ask Klaus!

    Klaus Knopper is the creator of Knoppix and co-founder of LinuxTag expo. He currently works as a programmer, teacher, and consultant. If you have a configuration problem, or if you just want to learn more about how Linux works, send your questions to: klaus@linux-magazine.com

  • Ask Klaus!

    Klaus Knopper is the creator of Knoppix and co-founder of LinuxTag expo. He currently works as a teacher, programmer, and consultant. If you have a configuration problem, or if you just want to learn more about how Linux works, send your questions to: klaus@linux-magazine.com

  • Ask Klaus!

    Klaus Knopper is the creator of Knoppix and co-founder of LinuxTag expo. He currently works as a teacher, programmer, and consultant. If you have a configuration problem, or if you just want to learn more about how Linux works, send your questions to: klaus@linux-magazine.com

comments powered by Disqus

Direct Download

Read full article as PDF:

Price $2.95

News