Subject: Re: writing iso file to usb stick with dd - garbled files - usb eject error?

On 2025-08-04, David Christensen <dpchrist@holgerdanske.com> wrote:

Thank you, Thomas, for finding that. So, I AIUI the d-i does not modify
the USB flash drive, but the motherboard firmware can (BIOS/EFI/UEFI)
and some operating systems can. The result is that the ISO checksum and
the USB flash drive checksum will differ. So, verification of the USB
flash drive must be done immediately after burning and later checksums
are not meaningful.

I don't understand this. The iso image isn't modified, and you can verify the integrity of that image independent of the drive itself.

David

--- PyGate Linux v1.5.12
* Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)

Subject: Re: writing iso file to usb stick with dd - garbled files - usb eject error?

On Thu, Mar 05, 2026 at 08:23:53PM +0100, Thomas Schmitt wrote:
[...]

ISO images which are bootable by EFI from USB stick contain a partition
table which points to some data range inside the image file.
In case of Debian netinst ISO this range is the content of data file /boot/grub/efi.img . So the EFI partition is even inside the ISO 9660 filesystem:

[...]

In modern Ubuntu ISOs the EFI partition is outside the ISO 9660
filesystem, but nevertheless inside the image file.

[...]

The offending mainboard firmwares and operating systems feel entitled
to add files to the FAT filesystem in the EFI partition. This changes
the content of /boot/grub/efi.img in the Debian ISO and thus the
checksum of the data range of the original image on the USB stick.
In the Ubuntu image the change does not hit the ISO 9660 filesystem
but still the byte range for which its download checksum was computed.

Ouch.
Is there any more systematic account for that? A way to find out (post
mortem, perhaps?) Or a list of bad firmwares?
I'm asking, because we had a couple of Linux install fests, and the install sticks were some times (not often, but more than once) not bootable after
the install. We always assumed fat-finger (aka pilot error), for example, partitioning the "wrong" target media (the install stick itself). But perhaps there are other viable hypotheses...
Cheers
--
t


--- PyGate Linux v1.5.12
* Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)

Subject: Re: writing iso file to usb stick with dd - garbled files - usb eject error?

On Thu, Mar 05, 2026 at 09:31:16PM +0100, Thomas Schmitt wrote:

Hi,

tomas@tuxteam.de wrote:

Ouch.

Habits of proprietary software. They decide what's on your USB stick.
Not because of malice but because of sound economic interest (at our
expense, usually).

That's why we love them, right? Now, as they realised they can't stamp
us out, they decided to colonize us (resp. our brains). I must admit,
they seem to be succeeding :-(

Is there any more systematic account for that?
Or a list of bad firmwares?

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1056998
reveiled that Lenovo's EFI and Microsoft's Windows operating system
placed files into the EFI partition's filesystem.

Oh, thanks a million.

There is no indication that these files hamper booting or using the
Debian ISO on the USB stick.

OK, so in "my" cases, the pilot error hypothesis is the most likely
(as often).

A way to find out (post mortem, perhaps?)

In case that the overall checksum of the image on USB stick does not
match, you need to compare image file and USB stick content.

In order to verify that it's not a manipulation outside the EFI
partition, consider to do something like this:
Ask a partition editor or xorriso for the partition table of the
image. Identify the block range of the EFI partition.
Compute the checksum for the range from block 0 up to the start of the
EFI partition. Compute the checksum for the range from first block
after the EFI partition up to the image file end.
Do the same two checksum computations with the USB stick base device
(e.g. /dev/sdc) and compare the results.
If these checksums match, then the alteration is only in the EFI
partition.

[...]
Thanks for the very detailed (as always) recipe. I'll keep it close
to my heart and perhaps dry-run it before the next install party
(these days, luckily, this kind of party seems to be taking off
again!)
Cheers
--
t


--- PyGate Linux v1.5.12
* Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)

Subject: Re: writing iso file to usb stick with dd - garbled files - usb eject error?

On Thu, Mar 05, 2026 at 01:37:11PM -0800, Mike Castle wrote:

I believe this whole class of problems is why tools like Rufus (https://rufus.ie/) were developed.

Wait a moment, and to clear any misunderstanding: I'm talking
about the content of the installation media being shredded
while being used. I can't imagine what a difference it could
ever make whether you wrote that media with dd, bit-bang or
whatever other method?

The following might be of interest to some: https://github.com/pbatard/rufus/wiki/FAQ#why-doesnt-rufus-recommend-dd-mode-over-iso-mode-for-isohybrid-images-surely-dd-is-better

I think I'll not click on a github link this time, thanks.
Cheers
--
t


--- PyGate Linux v1.5.12
* Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)

Subject: Re: writing iso file to usb stick with dd - garbled files - usb eject error?

On Fri, Mar 06, 2026 at 09:36:46AM +0100, Thomas Schmitt wrote:

Hi,

Mike Castle wrote:

I believe this whole class of problems is why tools like Rufus (https://rufus.ie/) were developed.

tomas@tuxteam.de wrote:

Wait a moment, and to clear any misunderstanding: I'm talking
about the content of the installation media being shredded
while being used. I can't imagine what a difference it could
ever make whether you wrote that media with dd, bit-bang or
whatever other method?

I think "shredding" is too hard. I'd call it "splashing about".

About the difference:
Pete Batard, the author of Rufus, writes in
https://github.com/pbatard/rufus/wiki/FAQ#user-content-Why_doesnt_Rufus_recommend_DD_mode_over_ISO_mode_for_ISOHybrid_images_Surely_DD_is_better

"Another nail in the coffin of "DD is better", is that you cannot
add runtime content validation, which is something that Rufus can
add to any UEFI based media (through UEFI MD5SUm) to enable boot-time
check that your media has not been corrupted, which, given the
unreliable nature of flash based USB drives, is arguably much better
than a once-off write-time check. Writing an ISOHybrid in DD mode can
never give you that. "

Ah, thanks for doing the reading for me. As I assume "dd" here is just
a strawman: rufus has some magic extra bootloader thingy which checks fingerprints of chosen files at boot. Good for them, but this is part
of image construction -- dumping that image bitwise to media is, at
its root, just another dd.
So the rufus people should rather criticise the construction of the boot
image, that would be adequate, not the "dd" itsef, right?
Or perhaps that's Microsoft thinking, where the browser (in former times
[1]) or "AI" (nowadays) is part of the "operating system"?
Confusing things doesn't make them clearer, methinks.
Cheers & thanks for the patient clarification
[1] https://en.wikipedia.org/wiki/Removal_of_Internet_Explorer
--
tom s


--- PyGate Linux v1.5.12
* Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)

Subject: Re: writing iso file to usb stick with dd - garbled files - usb eject error?

On Fri, Mar 06, 2026 at 11:37:49AM +0100, Thomas Schmitt wrote:

Hi,

i wrote:

Pete Batard, the author of Rufus, writes in
https://github.com/pbatard/rufus/wiki/FAQ#user-content-Why_doesnt_Rufus_recommend_DD_mode_over_ISO_mode_for_ISOHybrid_images_Surely_DD_is_better

tomas@tuxteam.de wrote:

Ah, thanks for doing the reading for me.

I assure you that the Rufus FAQ is not poisonous. :))

If you ever are interested to get an impression what a large user base
on MS-Windows does to the hobby life of a good willing developer of
free software, then skim over it.
I as developer of xorriso can afford to be mercilessly friendly to the
few users who contact me. He needs a pitch fork to keep them off his
neck.

I don't envy them. I only took issue with the idea that any tool can
have "magical powers" beyond dd, since dd is just the last, dumb,
bit-for-bit copy to the media, which has to be done anyway. Surely
the only magic has to be then in the content copied?
Unless, that is, that magical tool has a way to change the stick's
embedded controller's firmware (I know, that has been done, but...).

As I assume "dd" here is just
a strawman: rufus has some magic extra bootloader thingy which checks fingerprints of chosen files at boot.

The quoted section of the FAQ is about Rufus "ISO" mode versus its
"DD" mode.

Ah, OK, something to watch out for: if anyone spells "dd" as "DD",
all in capitals, they are talking about "Rufu's DD mode", which
might -- or might not -- have anything to do with dd.

[...]

Yep, the Rufus man dislikes the concept of isohybrid.

We isohybrid producers like it because it delivers a single blob which
has the sole purpose to boot on nearly every x86 machine. Beauty and usability of the USB stick as storage device are not the main goal. (Technical background of isohybrid can be found in
https://sources.debian.org/src/libisofs/1.5.6.pl01-1.1/doc/boot_sectors.txt Debian's ISO isohybrid layout much like the one in
SYSLINUX isohybrid for MBR, UEFI and x86-Mac
which of course refers to previous sections about x86 boot lures.)

Both concepts cannot cover every use case.
Rufus needs to know internals about the boot related files in the ISO. isohybrid might not work on some ideosyncratic x86 firmware for which
Rufus could apply specific workarounds.

Thanks. So in a nutshell, the magic is in the content, and
Rufus produces a somewhat different magic than isohybrid.
Good ol' dd is not to blame, just up to that confusing name "DD mode"
chosen for that Rufus option...
Cheers
--
t


--- PyGate Linux v1.5.12
* Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)

Subject: Re: writing iso file to usb stick with dd - garbled files - usb eject error?

On 2026-03-06, <tomas@tuxteam.de> <tomas@tuxteam.de> wrote:

On Thu, Mar 05, 2026 at 01:37:11PM -0800, Mike Castle wrote:

I believe this whole class of problems is why tools like Rufus
(https://rufus.ie/) were developed.

Wait a moment, and to clear any misunderstanding: I'm talking
about the content of the installation media being shredded
while being used. I can't imagine what a difference it could
ever make whether you wrote that media with dd, bit-bang or
whatever other method?

I guess you can override "read-only" if you're a raving capitalist.

The following might be of interest to some:
https://github.com/pbatard/rufus/wiki/FAQ#why-doesnt-rufus-recommend-dd-m=

ode-over-iso-mode-for-isohybrid-images-surely-dd-is-better

I think I'll not click on a github link this time, thanks.

Cheers
--=20
t

--D4DhzOTBSum2vqFu
Content-Type: application/pgp-signature; name="signature.asc"

--D4DhzOTBSum2vqFu--

--- PyGate Linux v1.5.12
* Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)

Subject: Re: writing iso file to usb stick with dd - garbled files - usb eject error?

On 2026-03-06, Thomas Schmitt <scdbackup@gmx.net> wrote:

I already knew it and always found it too excited for a topic which
leaves a lot of room for discussion and but-what-if arguments.
Pete Batard is as proud of his work as we Linux isohybriders are of
ours. But his text lets him appear deeply annoyed by the expectations
of GNU/Linux developers and users. (I cannot judge what a user of
MS-Windows expects as natural. To my meager experience they mostly
expect sudden disaster.)

On that derniŠre note, I bought a Lenovo Slimpad 5 for my better half.
She wanted Windows; the machine has Windows 11 on it, and it's a real
pain in the ass. Very, very annoying. I would've thrown the machine out
the window, if it hadn't cost me 800?.

We're on the third floor.

YMMV.

Some of the technical arguments against "dd" are valid. That's why i
made a shell script which shall keep the non-Windows users from
overwriting their hard disk by mistake and also cares for the perils
of previously GPT partitioned USB sticks.
https://packages.debian.org/unstable/xorriso-dd-target
https://wiki.debian.org/XorrisoDdTarget

Have a nice day :)

Thomas

--- PyGate Linux v1.5.12
* Origin: Dragon's Lair, PyGate NNTP<>Fido Gate (3:633/10)