grub-mkrescue: Problem with MBR partition table at start of EFI partition

grub-mkrescue: Problem with MBR partition table at start of EFI partition

[Thomas Schmitt]

Hi,

this is about how exactly to solve a diagnosed and worked-around problem.

Guix is one of the few distros which make their installation ISOs by
help of grub-mkrescue. The EFI boot manager of an old Macbook got stuck
when such an ISO was presented on USB stick. I.e. it not only did not
boot the ISO but showed no other EFI partitions either.

A Debian Live 9 ISO does not impose such a problem.
The same Guix ISO on DVD rather than USB stick works well, too.

The owner of the machine is Florian Pelz (Cc'ed). He characterizes it as:
  MacBook Pro (13-inch, Mid 2010)
MacOS Yosemite reports:
  Model Name:            MacBook Pro
  Model Identifier:      MacBookPro7,1
  Boot ROM Version:      MBP71.003F.B00
  SMC Version (system):  1.62f7

During a longish bug hunt it turned out that the decisive difference
is the first block of the FAT filesystem image which Debian creates by
mkfs.fat whereas grub-mkrescue creates it by mformat.

The mformat-made image has an MBR partition table entry which claims the
whole image as partition 1:

  Device            Boot Start   End Sectors  Size Id Type
  /mnt/iso/efi.img1 *        0  2879    2880  1.4M  1 FAT12

If this partition entry is zeroed, then the EFI boot manager works even
when the USB stick with this modified ISO is plugged in.

Actually it suffices to change the start LBA from 0 to 1, so that the
partition does not start by the partition table. (I count this as proof
of an endless loop in EFI while it is exploring something like
extended partitions. Strange is that the main MBR partition 1 of the
Debian ISO points to that main MBR, too. So it seems to harm only in
sub partition tables. We repacked the Guix ISO to MBR rather than GPT.
Still stuck.)

The EFI FAT image in Debian Live 9 has no partition table entry in block 0.

Production of the partition entry can be suppressed by mformat option -k.
But then block 0 of the result is not marked as MBR and also causes
program "file" to give out a very sparse characterization:

  DOS floppy 2880k

Here is the "file" report about an image made without -k:

  DOS/MBR boot sector, code offset 0x3c+2, OEM-ID "MTOO4018",
  sectors/cluster 2, root entries 240, sectors 5760 (volumes <=32 MB) ,
  sectors/FAT 9, sectors/track 36,
  serial number 0x6c528b1f, unlabeled, FAT (12 bit), followed by FAT

An image made by a modified grub-mkrescue with mformat option -k
worked well for Florian Pelz.

---------------------------------------------------------------------------

Questions:

1: Is there any use for the partition entry in the EFI partition of a
   grub-mkrescue ISO ?

2: Is there any use for the information which mformat does not insert
   if option -k is set ?

If the partition entry is not needed but the other MBR-like bytes are
needed:

3: Is there a good example in GRUB util sources about how to open a file,
   seek to its byte 446, and write 16 zero bytes starting there ?
   (I'd like to propose a patch for grub-mkrescue.c which zeros the
    partition entry directly after the mformat run.)

---------------------------------------------------------------------------


Have a nice day :)

Thomas

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[Vladimir 'phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 3667 bytes --]

On Mon, 22 Apr 2019, 01:42 Thomas Schmitt, <scdbackup@gmx.net> wrote:

> Hi,
>
> this is about how exactly to solve a diagnosed and worked-around problem.
>
> Guix is one of the few distros which make their installation ISOs by
> help of grub-mkrescue. The EFI boot manager of an old Macbook got stuck
> when such an ISO was presented on USB stick. I.e. it not only did not
> boot the ISO but showed no other EFI partitions either.
>
> A Debian Live 9 ISO does not impose such a problem.
> The same Guix ISO on DVD rather than USB stick works well, too.
>
> The owner of the machine is Florian Pelz (Cc'ed). He characterizes it as:
>   MacBook Pro (13-inch, Mid 2010)
> MacOS Yosemite reports:
>   Model Name:            MacBook Pro
>   Model Identifier:      MacBookPro7,1
>   Boot ROM Version:      MBP71.003F.B00
>   SMC Version (system):  1.62f7
>
> During a longish bug hunt it turned out that the decisive difference
> is the first block of the FAT filesystem image which Debian creates by
> mkfs.fat whereas grub-mkrescue creates it by mformat.
>
> The mformat-made image has an MBR partition table entry which claims the
> whole image as partition 1:
>
>   Device            Boot Start   End Sectors  Size Id Type
>   /mnt/iso/efi.img1 *        0  2879    2880  1.4M  1 FAT12
>
> If this partition entry is zeroed, then the EFI boot manager works even
> when the USB stick with this modified ISO is plugged in.
>
> Actually it suffices to change the start LBA from 0 to 1, so that the
> partition does not start by the partition table. (I count this as proof
> of an endless loop in EFI while it is exploring something like
> extended partitions. Strange is that the main MBR partition 1 of the
> Debian ISO points to that main MBR, too. So it seems to harm only in
> sub partition tables. We repacked the Guix ISO to MBR rather than GPT.
> Still stuck.)
>
> The EFI FAT image in Debian Live 9 has no partition table entry in block 0.
>
> Production of the partition entry can be suppressed by mformat option -k.
> But then block 0 of the result is not marked as MBR and also causes
> program "file" to give out a very sparse characterization:
>
>   DOS floppy 2880k
>
> Here is the "file" report about an image made without -k:
>
>   DOS/MBR boot sector, code offset 0x3c+2, OEM-ID "MTOO4018",
>   sectors/cluster 2, root entries 240, sectors 5760 (volumes <=32 MB) ,
>   sectors/FAT 9, sectors/track 36,
>   serial number 0x6c528b1f, unlabeled, FAT (12 bit), followed by FAT
>
> An image made by a modified grub-mkrescue with mformat option -k
> worked well for Florian Pelz.
>
> ---------------------------------------------------------------------------
>
> Questions:
>
> 1: Is there any use for the partition entry in the EFI partition of a
>    grub-mkrescue ISO ?
>
Nope. An artefact of mformat that we don't really want. I just didn't know
about -k (or maybe it didn't exist back then)

>
> 2: Is there any use for the information which mformat does not insert
>    if option -k is set ?
>
Probably no

>
> If the partition entry is not needed but the other MBR-like bytes are
> needed:
>
> 3: Is there a good example in GRUB util sources about how to open a file,
>    seek to its byte 446, and write 16 zero bytes starting there ?
>    (I'd like to propose a patch for grub-mkrescue.c which zeros the
>     partition entry directly after the mformat run.)
>
I don't think so.

>
> ---------------------------------------------------------------------------
>
>
> Have a nice day :)
>
> Thomas
>
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
>

[-- Attachment #2: Type: text/html, Size: 5084 bytes --]

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[Thomas Schmitt]

Hi,

i wrote:
> > 1: Is there any use for the partition entry in the EFI partition of a
> >  grub-mkrescue ISO ?

Vladimir Serbinenko wrote:
> Nope. An artefact of mformat that we don't really want. I just didn't know
> about -k (or maybe it didn't exist back then)

> > 2: Is there any use for the information which mformat does not insert
> >    if option -k is set ?

> Probably no

So how about adding "-k" to

  http://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkrescue.c#n812

  rv = grub_util_exec ((const char * []) { "mformat", "-C", "-f", "2880", "-L", "16", "-i",
        efiimgfat, "::", NULL });

and waiting for some EFI that will take offense ?
(Then at least we know why we need more than what -k gives.)

For now we have one old Macbook firmware which likes it.


Have a nice day :)

Thomas

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[Daniel Kiper]

On Sun, Apr 21, 2019 at 03:43:52PM +0200, Thomas Schmitt wrote:
> Hi,
>
> this is about how exactly to solve a diagnosed and worked-around problem.
>
> Guix is one of the few distros which make their installation ISOs by
> help of grub-mkrescue. The EFI boot manager of an old Macbook got stuck
> when such an ISO was presented on USB stick. I.e. it not only did not
> boot the ISO but showed no other EFI partitions either.
>
> A Debian Live 9 ISO does not impose such a problem.
> The same Guix ISO on DVD rather than USB stick works well, too.

How do you create USB stick from ISO image? Have you tried latest Guix image?

> The owner of the machine is Florian Pelz (Cc'ed). He characterizes it as:
>   MacBook Pro (13-inch, Mid 2010)
> MacOS Yosemite reports:
>   Model Name:            MacBook Pro
>   Model Identifier:      MacBookPro7,1
>   Boot ROM Version:      MBP71.003F.B00
>   SMC Version (system):  1.62f7
>
> During a longish bug hunt it turned out that the decisive difference
> is the first block of the FAT filesystem image which Debian creates by
> mkfs.fat whereas grub-mkrescue creates it by mformat.
>
> The mformat-made image has an MBR partition table entry which claims the
> whole image as partition 1:
>
>   Device            Boot Start   End Sectors  Size Id Type
>   /mnt/iso/efi.img1 *        0  2879    2880  1.4M  1 FAT12
>
> If this partition entry is zeroed, then the EFI boot manager works even
> when the USB stick with this modified ISO is plugged in.

Numbers here and in another email suggest that this is a floppy image.
And floppy images do not have MBR. They have something called boot
sector which is a bit different thing. I have a feeling that simply Macs
tested by you does not like floppy images. If you create regular HDD
images then probably everything will work.

Daniel

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[pelzflorian (Florian Pelz)]

On Wed, Apr 24, 2019 at 10:32:08PM +0200, Daniel Kiper wrote:
> On Sun, Apr 21, 2019 at 03:43:52PM +0200, Thomas Schmitt wrote:
> > Hi,
> >
> > this is about how exactly to solve a diagnosed and worked-around problem.
> >
> > Guix is one of the few distros which make their installation ISOs by
> > help of grub-mkrescue. The EFI boot manager of an old Macbook got stuck
> > when such an ISO was presented on USB stick. I.e. it not only did not
> > boot the ISO but showed no other EFI partitions either.
> >
> > A Debian Live 9 ISO does not impose such a problem.
> > The same Guix ISO on DVD rather than USB stick works well, too.
> 
> How do you create USB stick from ISO image? Have you tried latest Guix image?
> 

This applies to all Guix images, including
guixsd-install-0.16.0.x86_64-linux.iso,
guixsd-install-0.15.0.i686-linux.iso, and current Guix git master.  It
is created by a grub-mkrescue invocation.  See:

https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/build/vm.scm

Some patches are applied to grub-mkrescue.  See those prefixed with
“grub” at:

https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/patches

For completeness, the GRUB package is defined here to use these patches:

https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/bootloaders.scm



> > The owner of the machine is Florian Pelz (Cc'ed). He characterizes it as:
> >   MacBook Pro (13-inch, Mid 2010)
> > MacOS Yosemite reports:
> >   Model Name:            MacBook Pro
> >   Model Identifier:      MacBookPro7,1
> >   Boot ROM Version:      MBP71.003F.B00
> >   SMC Version (system):  1.62f7
> >

Note that more recent Macbooks appear not to be affected (at least an
“early 2014” MacBookAir6,2 is not affected).


> > The mformat-made image has an MBR partition table entry which claims the
> > whole image as partition 1:
> >
> >   Device            Boot Start   End Sectors  Size Id Type
> >   /mnt/iso/efi.img1 *        0  2879    2880  1.4M  1 FAT12
> >
> > If this partition entry is zeroed, then the EFI boot manager works even
> > when the USB stick with this modified ISO is plugged in.
> 
> Numbers here and in another email suggest that this is a floppy image.
> And floppy images do not have MBR. They have something called boot
> sector which is a bit different thing. I have a feeling that simply Macs
> tested by you does not like floppy images. If you create regular HDD
> images then probably everything will work.
> 
> Daniel


I am not very familiar with partitioning.  Is any image created by
grub-mkrescue a floppy image?  Creating a disk image without
grub-mkrescue works and boots fine on the USB drive, so do the above
ISO images when burned to a DVD, but the goal is that users can dd the
ISO to a USB drive and it just works.

Regards,
Florian

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[Thomas Schmitt]

Hi,

Daniel Kiper wrote:
> How do you create USB stick from ISO image?

By plain copy to the base device, as usual for a grub-mkrescue ISO.
Like

  dd if=image of=usb_stick


> Have you tried latest Guix image?

Florian builds Guix ISOs locally. So i assume that it was newest Guix.
Nevertheless, the problem is clearly not in Guix content but in the
Start LBA 0 of the partition table entry inside the EFI partition image.


> Numbers here and in another email suggest that this is a floppy image.
> And floppy images do not have MBR. They have something called boot
> sector which is a bit different thing.

Yes. It already has been pointed out to me privately that it should
rather be called a "VBR".
But EFI specs talk of "MBR" in the System Partition. So i am in good
company. As mathematician i always agree to "If it walks like a duck ...".

UEFI 2.4, 12.3.1 "System Partition"

  "A System Partition supports backward compatibility with legacy systems
   by reserving the first block (sector) of the partition for compatibility
   code. On legacy systems, the first block (sector) of a partition is
   loaded into memory and execution is transferred to this code.
   EFI firmware does not execute the code in the MBR. The EFI firmware
   contains knowledge about the partition structure of various devices,
   and can understand legacy MBR, GPT, and “El Torito.”"

(The description of BIOS loading partition boot code is wrong. It is a
 class of device MBRs which contain code to chainload partition boot code.
 I guess, GRUB has such an MBR variant in its collection, too.)

The last sentence might mean that EFI shall support extended MBR partitions.
I doubt that it is an invitation to support a tree of partition tables with
arbitrary tree width.
I also doubt that the author had a mental relapse to the MBR of the device,
which is mentioned in the paragraph before the one above.

  "A System Partition is a partition in the conventional sense of a
   partition on a legacy system. For a hard disk, a partition is a
   contiguous grouping of sectors on the disk where the starting sector
   and size are defined by the Master Boot Record (MBR), which resides
   on LBA 0 (i.e., the first sector of the hard disk) (see Section 5.2),
   or the GUID Partition Table (GPT), which resides on logical block 1
   (the second sector of the hard disk) (see Section 5.3.1). For a
   diskette (floppy) drive, a partition is defined to be the entire media.
   A System Partition can reside on any media that is supported by EFI
   Boot Services."


> I have a feeling that simply Macs
> tested by you does not like floppy images. If you create regular HDD
> images then probably everything will work.

It was grub-mkrescue which created that floppy image by mformat
and submits it to xorriso as EFI partition.

My questions were whether the partition entry is necessary (Vladimir
expects that it is not), and whether an MBR signature and the other
fields in block 0 are needed, which mformat creates without option -k
(Vladimir's answer was less firm).

I am now proposing to Guix to use my script-in-the middle

  https://dev.lovelyhq.com/libburnia/libisoburn/raw/master/frontend/grub-mkrescue-sed.sh

which is submitted to grub-mkrescue by option --xorriso=.

Its main purpose is to achieve other partition layouts with mountable
ISO partition. Meanwhile it learned how to zeroize the partition table
in the EFI partition and, if desired additionally, to zeroize the MBR
signature of the partition.

Be invited to try with a BIOS and EFI enabled GRUB configuration:

  wget https://dev.lovelyhq.com/libburnia/libisoburn/raw/master/frontend/grub-mkrescue-sed.sh
  chmod u+x grub-mkrescue-sed.sh

  export MKRESCUE_SED_MODE=mbr_hfs
  export MKRESCUE_SED_IN_EFI_NO_PT=yes

  grub-mkrescue --xorriso=./grub-mkrescue-sed.sh \
                -iso_mbr_part_type 0x83 -partition_offset 16 \
                ...your usual options to grub-mkrescue...

(  export MKRESCUE_SED_DEBUG=yes
 would let the script print input xorriso arguments and the arguments
 which it actually hands over to xorriso.)

The result has this partition layout:

  $ /sbin/fdisk -l output.iso
  ...
  Disklabel type: dos
  ...
  Device      Boot Start   End Sectors  Size Id Type
  output.iso1 *       64 28695   28632   14M 83 Linux
  output.iso2      28696 34455    5760  2.8M ef EFI (FAT-12/16/32)

Advantages in my eyes are:

- No GPT backup which will be misplaced on USB stick after dd.
  (Linux and partition editors complain.)

- Mountable ISO partition with non-zero partition start for pleasing
  partition editors and  being able to claim partition 2 as part of the
  filesystem range of the base device ISO:
     $ expr $(/sbin/isosize /dev/sdc) / 512
     34456
     $ expr $(/sbin/isosize /dev/sdc1) / 512
     28632
  This maintains the property of the original grub-mkrescue GPT layout,
  that an ISO image on a USB stick or DVD+RW still can tell its original
  size for checksumming.

- It is supposed to boot on Florian's Macbook because of no partition table
  in the EFI partition. (I'm not sure whether Florian aready tested it.
  Our test results are from byte hacking experiments on a vanilla Guix ISO.)


Have a nice day :)

Thomas

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[pelzflorian (Florian Pelz)]

On Thu, Apr 25, 2019 at 10:18:08AM +0200, Thomas Schmitt wrote:
>   wget https://dev.lovelyhq.com/libburnia/libisoburn/raw/master/frontend/grub-mkrescue-sed.sh
>   chmod u+x grub-mkrescue-sed.sh
> 
>   export MKRESCUE_SED_MODE=mbr_hfs
>   export MKRESCUE_SED_IN_EFI_NO_PT=yes
> 
>   grub-mkrescue --xorriso=./grub-mkrescue-sed.sh  \
>                 -iso_mbr_part_type 0x83 -partition_offset 16 \
>                 ...your usual options to grub-mkrescue...
> 

This works (with -o mbrhfs.iso as the usual options).  I can select
between “EFI Boot” and “GRUB 2.02” which both boot fine into a GRUB
console.  Without --xorriso=./grub-mkrescue-sed.sh the boot process
gets stuck before showing a boot drive selection screen.

Regards,
Florian

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[Vladimir 'phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 5996 bytes --]

There is an easy way to check if -k can be used always: have a look at
windows install CD/DVD. If it doesn't have partion table then we can drop
it as well.

On Thu, 25 Apr 2019, 18:16 Thomas Schmitt, <scdbackup@gmx.net> wrote:

> Hi,
>
> Daniel Kiper wrote:
> > How do you create USB stick from ISO image?
>
> By plain copy to the base device, as usual for a grub-mkrescue ISO.
> Like
>
>   dd if=image of=usb_stick
>
>
> > Have you tried latest Guix image?
>
> Florian builds Guix ISOs locally. So i assume that it was newest Guix.
> Nevertheless, the problem is clearly not in Guix content but in the
> Start LBA 0 of the partition table entry inside the EFI partition image.
>
>
> > Numbers here and in another email suggest that this is a floppy image.
> > And floppy images do not have MBR. They have something called boot
> > sector which is a bit different thing.
>
> Yes. It already has been pointed out to me privately that it should
> rather be called a "VBR".
> But EFI specs talk of "MBR" in the System Partition. So i am in good
> company. As mathematician i always agree to "If it walks like a duck ...".
>
> UEFI 2.4, 12.3.1 "System Partition"
>
>   "A System Partition supports backward compatibility with legacy systems
>    by reserving the first block (sector) of the partition for compatibility
>    code. On legacy systems, the first block (sector) of a partition is
>    loaded into memory and execution is transferred to this code.
>    EFI firmware does not execute the code in the MBR. The EFI firmware
>    contains knowledge about the partition structure of various devices,
>    and can understand legacy MBR, GPT, and “El Torito.”"
>
> (The description of BIOS loading partition boot code is wrong. It is a
>  class of device MBRs which contain code to chainload partition boot code.
>  I guess, GRUB has such an MBR variant in its collection, too.)
>
> The last sentence might mean that EFI shall support extended MBR
> partitions.
> I doubt that it is an invitation to support a tree of partition tables with
> arbitrary tree width.
> I also doubt that the author had a mental relapse to the MBR of the device,
> which is mentioned in the paragraph before the one above.
>
>   "A System Partition is a partition in the conventional sense of a
>    partition on a legacy system. For a hard disk, a partition is a
>    contiguous grouping of sectors on the disk where the starting sector
>    and size are defined by the Master Boot Record (MBR), which resides
>    on LBA 0 (i.e., the first sector of the hard disk) (see Section 5.2),
>    or the GUID Partition Table (GPT), which resides on logical block 1
>    (the second sector of the hard disk) (see Section 5.3.1). For a
>    diskette (floppy) drive, a partition is defined to be the entire media.
>    A System Partition can reside on any media that is supported by EFI
>    Boot Services."
>
>
> > I have a feeling that simply Macs
> > tested by you does not like floppy images. If you create regular HDD
> > images then probably everything will work.
>
> It was grub-mkrescue which created that floppy image by mformat
> and submits it to xorriso as EFI partition.
>
> My questions were whether the partition entry is necessary (Vladimir
> expects that it is not), and whether an MBR signature and the other
> fields in block 0 are needed, which mformat creates without option -k
> (Vladimir's answer was less firm).
>
> I am now proposing to Guix to use my script-in-the middle
>
>
> https://dev.lovelyhq.com/libburnia/libisoburn/raw/master/frontend/grub-mkrescue-sed.sh
>
> which is submitted to grub-mkrescue by option --xorriso=.
>
> Its main purpose is to achieve other partition layouts with mountable
> ISO partition. Meanwhile it learned how to zeroize the partition table
> in the EFI partition and, if desired additionally, to zeroize the MBR
> signature of the partition.
>
> Be invited to try with a BIOS and EFI enabled GRUB configuration:
>
>   wget
> https://dev.lovelyhq.com/libburnia/libisoburn/raw/master/frontend/grub-mkrescue-sed.sh
>   chmod u+x grub-mkrescue-sed.sh
>
>   export MKRESCUE_SED_MODE=mbr_hfs
>   export MKRESCUE_SED_IN_EFI_NO_PT=yes
>
>   grub-mkrescue --xorriso=./grub-mkrescue-sed.sh \
>                 -iso_mbr_part_type 0x83 -partition_offset 16 \
>                 ...your usual options to grub-mkrescue...
>
> (  export MKRESCUE_SED_DEBUG=yes
>  would let the script print input xorriso arguments and the arguments
>  which it actually hands over to xorriso.)
>
> The result has this partition layout:
>
>   $ /sbin/fdisk -l output.iso
>   ...
>   Disklabel type: dos
>   ...
>   Device      Boot Start   End Sectors  Size Id Type
>   output.iso1 *       64 28695   28632   14M 83 Linux
>   output.iso2      28696 34455    5760  2.8M ef EFI (FAT-12/16/32)
>
> Advantages in my eyes are:
>
> - No GPT backup which will be misplaced on USB stick after dd.
>   (Linux and partition editors complain.)
>
> - Mountable ISO partition with non-zero partition start for pleasing
>   partition editors and  being able to claim partition 2 as part of the
>   filesystem range of the base device ISO:
>      $ expr $(/sbin/isosize /dev/sdc) / 512
>      34456
>      $ expr $(/sbin/isosize /dev/sdc1) / 512
>      28632
>   This maintains the property of the original grub-mkrescue GPT layout,
>   that an ISO image on a USB stick or DVD+RW still can tell its original
>   size for checksumming.
>
> - It is supposed to boot on Florian's Macbook because of no partition table
>   in the EFI partition. (I'm not sure whether Florian aready tested it.
>   Our test results are from byte hacking experiments on a vanilla Guix
> ISO.)
>
>
> Have a nice day :)
>
> Thomas
>
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
>

[-- Attachment #2: Type: text/html, Size: 7229 bytes --]

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[Thomas Schmitt]

Hi,

Vladimir Serbinenko wrote:
> There is an easy way to check if -k can be used always: have a look at
> windows install CD/DVD. If it doesn't have partion table then we can drop it
> as well.

Eek. After that i would have to scrub my eyes with a wire brush.

But this question is already decided by the EFI capable ISOs of Debian,
Ubuntu, Archlinux, Fedora, and openSUSE. None of them has a partition
table entry. I am not aware of failure reports which point to this as
culprit.

Still open is the question whether mformat -k omits more preparations
which may be needed in some situations.

-----------------------------------------------------------------------

Meanwhile i found an example in grub-mkrescue.c about how to patch the
mformat result immediately after it was produced:

  if (source_dirs[GRUB_INSTALL_PLATFORM_SPARC64_IEEE1275]
      ...
      out = grub_util_fopen (sysarea_img, "wb");
      if (!out)
        grub_util_error (_("cannot open `%s': %s"), sysarea_img,
                         strerror (errno));
      memset (buf, 0, 512);
      fwrite (buf, 1, 512, out);
      ...
      fclose (out);

With due adaptions this gesture could well zeroize the 64 byte of
partition table in mformat's (pseudo-)MBR.

-----------------------------------------------------------------------

Have a nice day :)

Thomas

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[Chris Murphy]

On Wed, May 1, 2019 at 1:31 AM Thomas Schmitt <scdbackup@gmx.net> wrote:
>
> But this question is already decided by the EFI capable ISOs of Debian,
> Ubuntu, Archlinux, Fedora, and openSUSE. None of them has a partition
> table entry. I am not aware of failure reports which point to this as
> culprit.

Fedora ISO's use isohybrid and have multiple partitions on their ISOs:
MBR, GPT, and APM.
https://mjg59.dreamwidth.org/11285.html


-- 
Chris Murphy

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[Thomas Schmitt]

Hi,

Chris Murphy wrote:
> Fedora ISO's use isohybrid and have multiple partitions on their ISOs:
> MBR, GPT, and APM.

Well, the GPT is not valid because the MBR partition table is not
"protective". It has a partition of type 0xEF embedded in a partition
of type 0x00. Barely compliant to UEFI specs.
Besides Fedora used by Debian, Ubuntu, Arch, Gentoo, and others.
More or less the current mainstream.

Nevertheless, this thread is about the MBR partition table entry at
the start of the EFI FAT image, not at the start of the ISO filesystem.
The ISO should have a partition table which marks the EFI partition.
(Only Microsoft seems not to bother ...)


Have a nice day :)

Thomas

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[Thomas Schmitt]

Hi,

Chris Murphy wrote:
> [Microsoft image CCCOMA_X64FRE_JA-JP_DV9]
> 0000b040  91 ef 01 00 00 00 00 00  00 00 00 00 00 00 00 00  |........
> 0000b050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |........
> 0000b060  88 00 00 00 00 00 01 00  07 02 00 00 00 00 00 00  |........

It comes to me that i forgot to ask for a look into the EFI partition
at 2048-bye block address 0x0207 = 519.
What do you get as hex dump of byte 1,062,912 to 1,063,423 ?


Have a nice day :)

Thomas

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[Thomas Schmitt]

Hi,

iwrote:
> What do you get as hex dump of byte 1,062,912 to 1,063,423 ?

I found the answer in the hex dump:

00103800  eb 3c 90 4d 53 44 4f 53  35 2e 30 00 02 01 01 00  |.<.MSDOS5.0.....|
00103810  02 e0 00 40 0b f0 09 00  12 00 02 00 00 00 00 00  |...@............|
00103820  00 00 00 00 00 00 29 c0  56 ef 00 45 46 49 53 45  |......).V..EFISE|
00103830  43 54 4f 52 20 20 46 41  54 31 32 20 20 20 fa 33  |CTOR  FAT12   .3|
00103840  c0 8e d0 bc 00 7c 16 07  bb 78 00 36 c5 37 1e 56  |.....|...x.6.7.V|
00103850  16 53 bf 3e 7c b9 0b 00  fc f3 a4 06 1f c6 45 fe  |.S.>|.........E.|
00103860  0f 8b 0e 18 7c 88 4d f9  89 47 02 c7 07 3e 7c fb  |....|.M..G...>|.|
00103870  cd 13 72 79 33 c0 39 06  13 7c 74 08 8b 0e 13 7c  |..ry3.9..|t....||
00103880  89 0e 20 7c a0 10 7c f7  26 16 7c 03 06 1c 7c 13  |.. |..|.&.|...|.|
00103890  16 1e 7c 03 06 0e 7c 83  d2 00 a3 50 7c 89 16 52  |..|...|....P|..R|
001038a0  7c a3 49 7c 89 16 4b 7c  b8 20 00 f7 26 11 7c 8b  ||.I|..K|. ..&.|.|
001038b0  1e 0b 7c 03 c3 48 f7 f3  01 06 49 7c 83 16 4b 7c  |..|..H....I|..K||
001038c0  00 bb 00 05 8b 16 52 7c  a1 50 7c e8 92 00 72 1d  |......R|.P|...r.|
001038d0  b0 01 e8 ac 00 72 16 8b  fb b9 0b 00 be e6 7d f3  |.....r........}.|
001038e0  a6 75 0a 8d 7f 20 b9 0b  00 f3 a6 74 18 be 9e 7d  |.u... .....t...}|
001038f0  e8 5f 00 33 c0 cd 16 5e  1f 8f 04 8f 44 02 cd 19  |._.3...^....D...|
00103900  58 58 58 eb e8 8b 47 1a  48 48 8a 1e 0d 7c 32 ff  |XXX...G.HH...|2.|
00103910  f7 e3 03 06 49 7c 13 16  4b 7c bb 00 07 b9 03 00  |....I|..K|......|
00103920  50 52 51 e8 3a 00 72 d8  b0 01 e8 54 00 59 5a 58  |PRQ.:.r....T.YZX|
00103930  72 bb 05 01 00 83 d2 00  03 1e 0b 7c e2 e2 8a 2e  |r..........|....|
00103940  15 7c 8a 16 24 7c 8b 1e  49 7c a1 4b 7c ea 00 00  |.|..$|..I|.K|...|
00103950  70 00 ac 0a c0 74 29 b4  0e bb 07 00 cd 10 eb f2  |p....t).........|
00103960  3b 16 18 7c 73 19 f7 36  18 7c fe c2 88 16 4f 7c  |;..|s..6.|....O||
00103970  33 d2 f7 36 1a 7c 88 16  25 7c a3 4d 7c f8 c3 f9  |3..6.|..%|.M|...|
00103980  c3 b4 02 8b 16 4d 7c b1  06 d2 e6 0a 36 4f 7c 8b  |.....M|.....6O|.|
00103990  ca 86 e9 8a 16 24 7c 8a  36 25 7c cd 13 c3 0d 0a  |.....$|.6%|.....|
001039a0  4e 6f 6e 2d 53 79 73 74  65 6d 20 64 69 73 6b 20  |Non-System disk |
001039b0  6f 72 20 64 69 73 6b 20  65 72 72 6f 72 0d 0a 52  |or disk error..R|
001039c0  65 70 6c 61 63 65 20 61  6e 64 20 70 72 65 73 73  |eplace and press|
001039d0  20 61 6e 79 20 6b 65 79  20 77 68 65 6e 20 72 65  | any key when re|
001039e0  61 64 79 0d 0a 00 49 4f  20 20 20 20 20 20 53 59  |ady...IO      SY|
001039f0  53 4d 53 44 4f 53 20 20  20 53 59 53 00 00 55 aa  |SMSDOS   SYS..U.|

An MBR signature 55 aa. But cleartext where the partition table should be.

If Microsoft ISOs shall really be considered standard, then any kind of
garbage is ok in a potential partition table at the start of the EFI
partition.

The firmware of the affected Macbook probably tolerates everything
except four zeros at byte offset 454 (here: 0x1039c6), where Microsoft
has 20 61  6e 64.
(It feels like two cars speeding on a salt lake. Probability for collision
 is low. Just close your eyes and trust in your luck.)


Have a nice day :)

Thomas

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[pelzflorian (Florian Pelz)]

On Fri, May 10, 2019 at 09:09:25AM +0200, Thomas Schmitt wrote:
> The firmware of the affected Macbook probably tolerates everything
> except four zeros at byte offset 454 (here: 0x1039c6), where Microsoft
> has 20 61  6e 64.


I would like to test but on this bootable German Windows 10
32-bit+64-bit USB install medium, the content is different.  How would
I find the offset in the USB image (you call it offset 454) which I
should zero out to break Windows?

Regards,
Florian

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> I would like to test but on this bootable German Windows 10
> 32-bit+64-bit USB install medium, the content is different.  How would
> I find the offset in the USB image (you call it offset 454) which I
> should zero out to break Windows?

That would be offset 454 in the EFI boot image: Start LBA field of MBR
partition 1.

You may learn the block address of the image from xorriso:

  xorriso -indev "$ISO" -report_el_torito plain 2>/dev/null \
  | grep 'El Torito boot img : .* UEFI'

should say something like

  El Torito boot img :   2  UEFI  y   none  0x0000  0x00   1     515

The last number 515 is the 2048-byte block address of the EFI image.
(That's from the japanese ISO mentioned at debian-user.)

If you write four zero bytes at byte 515 * 2048 + 454 = 1055174, then
there is start LBA 0 in the slot of partition 1. This should trigger
the stalled boot process of the Macbook.

But maybe the other bytes from 446 to 461 hamper recognition as partition
table entry. So more similar to the mformat image would be to zeroize
bytes 446 to 509 to clear the partition table, and then to write to bytes
446 to 461 what we see as partition slot 1 in the grub-mkrescue EFI
partition:

  80  00  01  00  01  01  12  4f  00  00  00  00  40  0b  00  00

-------------------------------------------------------------------------

If you are at it, i'd like to see a confirmation that the ISO indeed
boots via El Torito from USB stick. (Interesting for GRUB as background
info.)

Please in the end spoil El Torito by zeroizing the address of the
Boot Catalog in the Boot Record at 2048-byte block 17 (decimal). Byte
offset 71 to 74, little endian.

The ISO hexdumped by Chris Murphy has 0x00000016 = 22 :

  00008840  00 00 00 00 00 00 00 16  00 00 00 00 00 00 00 00

So

  dd if=/dev/zero bs=1 count=4 seek=34887 conv=notrunc of="$ISO"


You will probably see error messages when inspecting by software.
Like

  $ xorriso -indev "$ISO" -report_el_torito plain
  ...
  libisofs: WARNING : Wrong or damaged El-Torito Catalog. El-Torito info will be ignored.

Or a clueless assessment without checking the catalog signature:

  $ isoinfo -d -i "$ISO"
  ...
  El Torito VD version 1 found, boot catalog is in sector 0
  ...
  Eltorito validation header:
    Hid 0
    Arch 0 (x86)
    ID ''
    Key 0 0
    Eltorito defaultboot header:
        Bootid 0 (not bootable)
        Boot media 0 (No Emulation Boot)
        Load segment 0
        Sys type 0
        Nsect 0
        Bootoff 0 0

This is still less intrusive than trying to remove the Boot Record
or to camouflage it as something else.

If my theory is right, then EFI should not recognize the USB stick
as a device with EFI partition any more.


Have a nice day :)

Thomas

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[pelzflorian (Florian Pelz)]

On Fri, May 10, 2019 at 03:46:56PM +0200, Thomas Schmitt wrote:
> Hi,
> 
> Florian Pelz wrote:
> > I would like to test but on this bootable German Windows 10
> > 32-bit+64-bit USB install medium, the content is different.  How would
> > I find the offset in the USB image (you call it offset 454) which I
> > should zero out to break Windows?
> 
> That would be offset 454 in the EFI boot image: Start LBA field of MBR
> partition 1.
> 
> You may learn the block address of the image from xorriso:
> 
>   xorriso -indev "$ISO" -report_el_torito plain 2>/dev/null \
>   | grep 'El Torito boot img : .* UEFI'
> 
> should say something like
> 
>   El Torito boot img :   2  UEFI  y   none  0x0000  0x00   1     515
>

This is not an install ISO but a USB install medium.  The USB drive
was written by the Windows Media Creation Tool.  Copying instead a
Windows ISO created by the Windows Media Creation Tool to a USB drive
does *not* yield a bootable USB drive; it does not even boot on other
computers.

So mine is a USB drive already, not an ISO image.  I do:

$ sudo xorriso -indev "stdio:/dev/sdb" -report_el_torito plain
GNU xorriso 1.5.0 : RockRidge filesystem manipulator, libburnia project.

xorriso : NOTE : Loading ISO image tree from LBA 0
libisoburn: WARNING : No ISO 9660 image at LBA 0. Creating blank image.
Drive current: -indev 'stdio:/dev/sdb'
Media current: stdio file, overwriteable
Media status : is written , is closed
Media summary: 1 session, 3932160 data blocks, 7680m data,     0 free
Volume id    : 'ISOIMAGE'
xorriso : NOTE : No El Torito information was loaded

Apparently it is wrong to look at such Windows 10 USB media and I
should rather look at older Windows installation media.  I will try to
procure a Windows 7 install ISO.  I will report back tomorrow.

Regards,
Florian

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> This is not an install ISO but a USB install medium.  The USB drive
> was written by the Windows Media Creation Tool.  Copying instead a
> Windows ISO created by the Windows Media Creation Tool to a USB drive
> does *not* yield a bootable USB drive; it does not even boot on other
> computers.

This would bring the MS-World back in sync with my understanding of UEFI
specs and other info sources.

But there are substantial rumors that the ISO boots from USB stick.
Microsoft is said to advertise it for DVD+R DL and USB stick.
(I could not get my hands on ISO or advertising yet ...)


Whatever, if your USB stick has an EFI partition, then you could try to
spoil it for your Macbook by writing the partition slot or the whole
partition table. The byte offsets like 446 or 454 are counted from the
start of the EFI partition.


Have a nice day :)

Thomas

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[pelzflorian (Florian Pelz)]

On Fri, May 10, 2019 at 06:27:17PM +0200, Thomas Schmitt wrote:
> Hi,
> 
> Florian Pelz wrote:
> > This is not an install ISO but a USB install medium.  The USB drive
> > was written by the Windows Media Creation Tool.  Copying instead a
> > Windows ISO created by the Windows Media Creation Tool to a USB drive
> > does *not* yield a bootable USB drive; it does not even boot on other
> > computers.
> 
> This would bring the MS-World back in sync with my understanding of UEFI
> specs and other info sources.
> 
> But there are substantial rumors that the ISO boots from USB stick.
> Microsoft is said to advertise it for DVD+R DL and USB stick.
> (I could not get my hands on ISO or advertising yet ...)
> 
> 

The Windows 8.1 ISO image that I downloaded from Microsoft yesterday
also cannot be booted from a USB drive on neither this Macbook or
another computer.  I will continue testing the Windows 10 USB install
medium:

On Fri, May 10, 2019 at 03:46:56PM +0200, Thomas Schmitt wrote:
> Hi,
> 
> Florian Pelz wrote:
> > I would like to test but on this bootable German Windows 10
> > 32-bit+64-bit USB install medium, the content is different.  How would
> > I find the offset in the USB image (you call it offset 454) which I
> > should zero out to break Windows?
> 
> That would be offset 454 in the EFI boot image: Start LBA field of MBR
> partition 1.
> 
> You may learn the block address of the image from xorriso:
> 
>   xorriso -indev "$ISO" -report_el_torito plain 2>/dev/null \
>   | grep 'El Torito boot img : .* UEFI'
> 
> should say something like
> 
>   El Torito boot img :   2  UEFI  y   none  0x0000  0x00   1     515
> 
> The last number 515 is the 2048-byte block address of the EFI image.
> (That's from the japanese ISO mentioned at debian-user.)
> 
> If you write four zero bytes at byte 515 * 2048 + 454 = 1055174, then
> there is start LBA 0 in the slot of partition 1. This should trigger
> the stalled boot process of the Macbook.
> 

florian@florianmacbook ~$ sudo fdisk -l /dev/sdb
Disk /dev/sdb: 7.5 GiB, 8053063680 bytes, 15728640 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x3093e6cb

Device     Boot Start      End  Sectors  Size Id Type
/dev/sdb1  *     2048 15728639 15726592  7.5G  c W95 FAT32 (LBA)
florian@florianmacbook ~$ sudo dd if=/dev/sdb of=windoof
Password: 
15728640+0 records in
15728640+0 records out
8053063680 bytes (8.1 GB, 7.5 GiB) copied, 324.929 s, 24.8 MB/s

So I calculate 512*2048+454=1049030.

I do:

florian@florianmacbook ~$ sudo dd if=/dev/zero bs=1 count=16 seek=1049030 of=/dev/sdb

It still boots.  I believe what I calculated was wrong.  I do not continue.



> But maybe the other bytes from 446 to 461 hamper recognition as partition
> table entry. So more similar to the mformat image would be to zeroize
> bytes 446 to 509 to clear the partition table, and then to write to bytes
> 446 to 461 what we see as partition slot 1 in the grub-mkrescue EFI
> partition:
> 
>   80  00  01  00  01  01  12  4f  00  00  00  00  40  0b  00  00
> 

Regards,
Florian

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[Thomas Schmitt]

Hi,

> The Windows 8.1 ISO image that I downloaded from Microsoft yesterday
> also cannot be booted from a USB drive on neither this Macbook or
> another computer.

I wonder how other people can get to the impression that their attempts
succeeded. (Will have to wait for somebody who is willing to go on with
experiments after perceived success.)


> I will continue testing the Windows 10 USB install medium:
> /dev/sdb1  *     2048 15728639 15726592  7.5G  c W95 FAT32 (LBA)

That's not an EFI System Partition.
So i assume the USB stick boots by BIOS and x86 machine code in the MBR.
Unless we believe in EFI firmware that hops on arbitrary partitions ...


> So I calculate 512*2048+454=1049030.
> florian@florianmacbook ~$ sudo dd if=/dev/zero bs=1 count=16 seek=1049030
> of=/dev/sdb
> It still boots.  I believe what I calculated was wrong.

You zeroized the LBA field but also the block count field.

It should have been count=4 for the most minimal surgery (LBA=0x00000000).
With count=16 you would have to seek to 512*2048+446 and afterwards would
have to write a new partition table entry with LBA=0, non-zero block count,
and probably a non-zero partition type.


>  I do not continue.

Then let's disinfect our brains and go on with some useful work. :))


Have a nice day :)

Thomas

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[pelzflorian (Florian Pelz)]

On Sat, May 11, 2019 at 02:05:25PM +0200, Thomas Schmitt wrote:
> > I will continue testing the Windows 10 USB install medium:
> > /dev/sdb1  *     2048 15728639 15726592  7.5G  c W95 FAT32 (LBA)
> 
> That's not an EFI System Partition.
> So i assume the USB stick boots by BIOS and x86 machine code in the MBR.
> Unless we believe in EFI firmware that hops on arbitrary partitions ...
> 
> 

This is strange indeed.  The Macbook displays “EFI Boot” in the boot
selection menu.  efi/boot/bootia32.efi and efi/boot/bootx64.efi exist.
It could be a decoy though.  I do not know how to verify on Windows if
the boot used UEFI.  Shall I continue testing?

Regards,
Florian

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> Shall I continue testing?

I'm into ISO images. But it might be interesting to see what Microsoft
deems to be bootable and is willing to promote by its weight.


> I do not know how to verify on Windows if the boot used UEFI.

You could zeroize the x86 machine code in the MBR:

  dd if=/dev/zero bs=1 count=446 of=/dev/sdb

Then BIOS/CSM legacy mode should not have anything to start with.
(It would be interesting to know whether there is useful machine code
 in the MBR at all.)

Similarly you could rename the efi/boot directory in order to hide the
EFI start-up programs and thus to prevent EFI booting.


Have a nice day :)

Thomas

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[pelzflorian (Florian Pelz)]

I guess this is off-topic and we cannot learn anything about the
permissibility of mformat -k from Windows?

Michael Schierl already explained everything, but still:

I reinstate the Windows 10 USB installation medium’s content.

On Sat, May 11, 2019 at 07:31:23PM +0200, Thomas Schmitt wrote:
> > I do not know how to verify on Windows if the boot used UEFI.
> 
> You could zeroize the x86 machine code in the MBR:
> 
>   dd if=/dev/zero bs=1 count=446 of=/dev/sdb
>

After this dd it still boots.

> Then BIOS/CSM legacy mode should not have anything to start with.
> (It would be interesting to know whether there is useful machine code
>  in the MBR at all.)
> 
> Similarly you could rename the efi/boot directory in order to hide the
> EFI start-up programs and thus to prevent EFI booting.
> 
>

Renaming bootx64.efi to bootx64.efi.bkup makes the Windows 10 install
medium no longer appear in the boot device selection menu.

Regards,
Florian

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[Thomas Schmitt]

Hi,

Florian Pelz wrote:
> I guess this is off-topic and we cannot learn anything about the
> permissibility of mformat -k from Windows?

Yes.

The on-topic conclusion so far is that i cowardly propose to zeroize
the MBR partition table in the mformat result before populating it
by mcopy.
(Code snippets from SYS_AREA_SPARC around line 855 could be transplanted
 and adapted to zeroize bytes 446 to 509 of efiimgfat.)

Exploring the other differences between mformat with and without -k is
probably useless, because the big truck makes its own lane anyway.


Have a nice day :)

Thomas

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[Daniel Kiper]

On Sat, May 11, 2019 at 10:39:25PM +0200, Thomas Schmitt wrote:
> Hi,
>
> Florian Pelz wrote:
> > I guess this is off-topic and we cannot learn anything about the
> > permissibility of mformat -k from Windows?
>
> Yes.
>
> The on-topic conclusion so far is that i cowardly propose to zeroize
> the MBR partition table in the mformat result before populating it
> by mcopy.
> (Code snippets from SYS_AREA_SPARC around line 855 could be transplanted
>  and adapted to zeroize bytes 446 to 509 of efiimgfat.)
>
> Exploring the other differences between mformat with and without -k is
> probably useless, because the big truck makes its own lane anyway.

The thread is long and I am a bit lost. So, may I ask you to write
a summary of your findings, what works and what does not and how the issue
should be fixed and why this fix should land in the upcoming release.

Daniel

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[Thomas Schmitt]

Hi,

Daniel Kiper wrote:
> may I ask you to write a summary of your findings,

A Macbook took offense from the MBR partition table entry in the EFI FAT
image which grub-mkrescue produces by help of mformat(1).
Vladimir stated that this partition table entry is not intentional and
that the information in block 0, which would be missing after using mformat
option -k, is probably not needed.

Vladimir's proposal to look into the EFI partitions of MS-Windows ISOs
led to the insight that their MBR partition table space is filled with
cleartext. I.e. not-zero is usual. Credibly looking partition table entry
starting at LBA 0 is not usual.

(Further we learned that Microsoft has its own protocol implemented in EFIs
 by which they boot USB sticks with MBR partitioning and no partition of
 type 0xEF.)


> what works and what does not

The only thing that was found to not work is a MBR partition entry 1 which
starts at LBA 0. Probably it needs to have non-zero block count and non-zero
type.


> how the issue should be fixed

Safest seems to be not to change the mformat run in grub-mkrescue but
rather to postprocess its result, e.g. before the run of mcopy which
populates the image.
Postprocessing would mean to zeroize the whole range of MBR partition
table entries. That's bytes 446 to 509 of the FAT image.


> why this fix should land in the upcoming release.

It may well wait until the start of the next release cycle, although i deem
it easy to test and of low risk. Most Linux distros have zeros where i
propose to put them. They make their EFI images by mkfs.fat(8).


Have a nice day :)

Thomas

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[Thomas Schmitt]

Hi,

Daniel Kiper wrote:
> may I ask you to write a summary of your findings,

A Macbook took offense from the MBR partition table entry in the EFI FAT
image which grub-mkrescue produces by help of mformat(1).
Vladimir stated that this partition table entry is not intentional and
that the information in block 0, which would be missing after using mformat
option -k, is probably not needed.

Vladimir's proposal to look into the EFI partitions of MS-Windows ISOs
led to the insight that their MBR partition table space is filled with
cleartext. I.e. not-zero is usual. Credibly looking partition table entry
starting at LBA 0 is not usual.

(Further we learned that Microsoft has its own protocol implemented in EFIs
 by which they boot USB sticks with MBR partitioning and no partition of
 type 0xEF.)


> what works and what does not

The only thing that was found to not work is a MBR partition entry 1 which
starts at LBA 0. Probably it needs to have non-zero block count and non-zero
type.


> how the issue should be fixed

Safest seems to be not to change the mformat run in grub-mkrescue but
rather to postprocess its result, e.g. before the run of mcopy which
populates the image.
Postprocessing would mean to zeroize the whole range of MBR partition
table entries. That's bytes 446 to 509 of the FAT image.


> why this fix should land in the upcoming release.

It may well wait until the start of the next release cycle, although i deem
it easy to test and of low risk. Most Linux distros have zeros where i
propose to put them. They make their EFI images by mkfs.fat(8).


Have a nice day :)

Thomas

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[Daniel Kiper]

On Tue, May 14, 2019 at 08:04:01AM +0200, Thomas Schmitt wrote:
> Hi,
>
> Daniel Kiper wrote:
> > may I ask you to write a summary of your findings,
>
> A Macbook took offense from the MBR partition table entry in the EFI FAT
> image which grub-mkrescue produces by help of mformat(1).
> Vladimir stated that this partition table entry is not intentional and
> that the information in block 0, which would be missing after using mformat
> option -k, is probably not needed.

Again, this is an FDD 2880K image and there is no MBR there. Your
interpretation is wrong. I think that simply some Macbooks have issues
with such FDD images.

> Vladimir's proposal to look into the EFI partitions of MS-Windows ISOs
> led to the insight that their MBR partition table space is filled with
> cleartext. I.e. not-zero is usual. Credibly looking partition table entry
> starting at LBA 0 is not usual.
>
> (Further we learned that Microsoft has its own protocol implemented in EFIs
>  by which they boot USB sticks with MBR partitioning and no partition of
>  type 0xEF.)

What kind of image is used on MS ISO Win image? FDD or HDD one? If FDD
then you cannot interpret boot sector as MBR. This is wrong.

> > what works and what does not
>
> The only thing that was found to not work is a MBR partition entry 1 which
> starts at LBA 0. Probably it needs to have non-zero block count and non-zero
> type.

Again, on what type of image? FDD or HDD?

> > how the issue should be fixed
>
> Safest seems to be not to change the mformat run in grub-mkrescue but
> rather to postprocess its result, e.g. before the run of mcopy which
> populates the image.
> Postprocessing would mean to zeroize the whole range of MBR partition
> table entries. That's bytes 446 to 509 of the FAT image.

I would not do that blindly. First of all look at mformat code and check
why above mentioned data is not zeroed. If this is not obvious ask
authors about that. Please CC grub-devel then too.

> > why this fix should land in the upcoming release.
>
> It may well wait until the start of the next release cycle, although i deem
> it easy to test and of low risk. Most Linux distros have zeros where i
> propose to put them. They make their EFI images by mkfs.fat(8).

If mkfs.fat images work on Macbooks then I would leave mformat as
default and add an option to switch between mformat and mkfs.fat.
Or we can try to create HDD image using mformat. Or both.

Daniel

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[Thomas Schmitt]

Hi,

> Again, this is an FDD 2880K image and there is no MBR there. Your
> interpretation is wrong.

In respect to MBR partition table, mformat source has at
  https://sources.debian.org/src/mtools/4.0.23-1/mformat.c/#L1375
code with this comment:

  /* install fake partition table pointing to itself */


> What kind of image is used on MS ISO Win image? FDD or HDD one?

Probably FDD (however this might be exactly defined).
There is a 0xaa 0x55 signature, piossibly x86 boot code, and clearly
no partition table.

Chris Murphy exposed on
  https://drive.google.com/open?id=1JgPv8EBHDn5A7hTEwewGK1MuFHvHbJnr
as block 0 of the Microsoft El Torito boot image for EFI:

00103800  eb 3c 90 4d 53 44 4f 53  35 2e 30 00 02 01 01 00  |.<.MSDOS5.0.....|
00103810  02 e0 00 40 0b f0 09 00  12 00 02 00 00 00 00 00  |...@............|
00103820  00 00 00 00 00 00 29 c0  56 ef 00 45 46 49 53 45  |......).V..EFISE|
00103830  43 54 4f 52 20 20 46 41  54 31 32 20 20 20 fa 33  |CTOR  FAT12   .3|
00103840  c0 8e d0 bc 00 7c 16 07  bb 78 00 36 c5 37 1e 56  |.....|...x.6.7.V|
00103850  16 53 bf 3e 7c b9 0b 00  fc f3 a4 06 1f c6 45 fe  |.S.>|.........E.|
00103860  0f 8b 0e 18 7c 88 4d f9  89 47 02 c7 07 3e 7c fb  |....|.M..G...>|.|
00103870  cd 13 72 79 33 c0 39 06  13 7c 74 08 8b 0e 13 7c  |..ry3.9..|t....||
00103880  89 0e 20 7c a0 10 7c f7  26 16 7c 03 06 1c 7c 13  |.. |..|.&.|...|.|
00103890  16 1e 7c 03 06 0e 7c 83  d2 00 a3 50 7c 89 16 52  |..|...|....P|..R|
001038a0  7c a3 49 7c 89 16 4b 7c  b8 20 00 f7 26 11 7c 8b  ||.I|..K|. ..&.|.|
001038b0  1e 0b 7c 03 c3 48 f7 f3  01 06 49 7c 83 16 4b 7c  |..|..H....I|..K||
001038c0  00 bb 00 05 8b 16 52 7c  a1 50 7c e8 92 00 72 1d  |......R|.P|...r.|
001038d0  b0 01 e8 ac 00 72 16 8b  fb b9 0b 00 be e6 7d f3  |.....r........}.|
001038e0  a6 75 0a 8d 7f 20 b9 0b  00 f3 a6 74 18 be 9e 7d  |.u... .....t...}|
001038f0  e8 5f 00 33 c0 cd 16 5e  1f 8f 04 8f 44 02 cd 19  |._.3...^....D...|
00103900  58 58 58 eb e8 8b 47 1a  48 48 8a 1e 0d 7c 32 ff  |XXX...G.HH...|2.|
00103910  f7 e3 03 06 49 7c 13 16  4b 7c bb 00 07 b9 03 00  |....I|..K|......|
00103920  50 52 51 e8 3a 00 72 d8  b0 01 e8 54 00 59 5a 58  |PRQ.:.r....T.YZX|
00103930  72 bb 05 01 00 83 d2 00  03 1e 0b 7c e2 e2 8a 2e  |r..........|....|
00103940  15 7c 8a 16 24 7c 8b 1e  49 7c a1 4b 7c ea 00 00  |.|..$|..I|.K|...|
00103950  70 00 ac 0a c0 74 29 b4  0e bb 07 00 cd 10 eb f2  |p....t).........|
00103960  3b 16 18 7c 73 19 f7 36  18 7c fe c2 88 16 4f 7c  |;..|s..6.|....O||
00103970  33 d2 f7 36 1a 7c 88 16  25 7c a3 4d 7c f8 c3 f9  |3..6.|..%|.M|...|
00103980  c3 b4 02 8b 16 4d 7c b1  06 d2 e6 0a 36 4f 7c 8b  |.....M|.....6O|.|
00103990  ca 86 e9 8a 16 24 7c 8a  36 25 7c cd 13 c3 0d 0a  |.....$|.6%|.....|
001039a0  4e 6f 6e 2d 53 79 73 74  65 6d 20 64 69 73 6b 20  |Non-System disk |
001039b0  6f 72 20 64 69 73 6b 20  65 72 72 6f 72 0d 0a 52  |or disk error..R|
001039c0  65 70 6c 61 63 65 20 61  6e 64 20 70 72 65 73 73  |eplace and press|
001039d0  20 61 6e 79 20 6b 65 79  20 77 68 65 6e 20 72 65  | any key when re|
001039e0  61 64 79 0d 0a 00 49 4f  20 20 20 20 20 20 53 59  |ady...IO      SY|
001039f0  53 4d 53 44 4f 53 20 20  20 53 59 53 00 00 55 aa  |SMSDOS   SYS..U.|


> If FDD then you cannot interpret boot sector as MBR. This is wrong.

It's others who interpret it and at least one takes the "fake" partition
table entry by mformat for a reason to early stall its boot process.

I just have to explain users of grub-mkrescue ISOs why they don't work.
The plight of the programmer whose software was the last to touch it.


> > The only thing that was found to not work is a MBR partition entry 1 which
> > starts at LBA 0. Probably it needs to have non-zero block count and non-zero
> > type.

> Again, on what type of image? FDD or HDD?

Made by the mformat run of grub-mkrescue.
Vladimir can probably tell you more about what kind of image he expects.


> First of all look at mformat code

I did before starting this thread.
The bytes in question are intended as "fake" partition table.


> If mkfs.fat images work on Macbooks then I would leave mformat as
> default and add an option to switch between mformat and mkfs.fat.

Why that ? Either mformat's result is ok or it is not.
Offering an alternative could start a new branch of cargo cult. Explaining
the differences will quickly raise the question why staying with the
unusual block0 bytes from mformat at all.

I understand that GRUB wants to use a GNU program, and i am careful not
to spoil anything that works. Thus my proposal as it is.


I will not be overly sad if the proposal is rejected. After all it is
quite easy to override the ideas of grub-mkrescue before the ISO gets
packed up. In practice, it's preference for GPT makes more problems than
its preference for mformat.


Have a nice day :)

Thomas

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[Daniel Kiper]

On Wed, May 15, 2019 at 12:57:58PM +0200, Thomas Schmitt wrote:
> Hi,
>
> > Again, this is an FDD 2880K image and there is no MBR there. Your
> > interpretation is wrong.
>
> In respect to MBR partition table, mformat source has at
>   https://sources.debian.org/src/mtools/4.0.23-1/mformat.c/#L1375
> code with this comment:
>
>   /* install fake partition table pointing to itself */

OK, but why it is also done for FDD images?

> > What kind of image is used on MS ISO Win image? FDD or HDD one?
>
> Probably FDD (however this might be exactly defined).
> There is a 0xaa 0x55 signature, piossibly x86 boot code, and clearly
> no partition table.
>
> Chris Murphy exposed on
>   https://drive.google.com/open?id=1JgPv8EBHDn5A7hTEwewGK1MuFHvHbJnr
> as block 0 of the Microsoft El Torito boot image for EFI:
>
> 00103800  eb 3c 90 4d 53 44 4f 53  35 2e 30 00 02 01 01 00  |.<.MSDOS5.0.....|
> 00103810  02 e0 00 40 0b f0 09 00  12 00 02 00 00 00 00 00  |...@............|
> 00103820  00 00 00 00 00 00 29 c0  56 ef 00 45 46 49 53 45  |......).V..EFISE|
> 00103830  43 54 4f 52 20 20 46 41  54 31 32 20 20 20 fa 33  |CTOR  FAT12   .3|
> 00103840  c0 8e d0 bc 00 7c 16 07  bb 78 00 36 c5 37 1e 56  |.....|...x.6.7.V|
> 00103850  16 53 bf 3e 7c b9 0b 00  fc f3 a4 06 1f c6 45 fe  |.S.>|.........E.|
> 00103860  0f 8b 0e 18 7c 88 4d f9  89 47 02 c7 07 3e 7c fb  |....|.M..G...>|.|
> 00103870  cd 13 72 79 33 c0 39 06  13 7c 74 08 8b 0e 13 7c  |..ry3.9..|t....||
> 00103880  89 0e 20 7c a0 10 7c f7  26 16 7c 03 06 1c 7c 13  |.. |..|.&.|...|.|
> 00103890  16 1e 7c 03 06 0e 7c 83  d2 00 a3 50 7c 89 16 52  |..|...|....P|..R|
> 001038a0  7c a3 49 7c 89 16 4b 7c  b8 20 00 f7 26 11 7c 8b  ||.I|..K|. ..&.|.|
> 001038b0  1e 0b 7c 03 c3 48 f7 f3  01 06 49 7c 83 16 4b 7c  |..|..H....I|..K||
> 001038c0  00 bb 00 05 8b 16 52 7c  a1 50 7c e8 92 00 72 1d  |......R|.P|...r.|
> 001038d0  b0 01 e8 ac 00 72 16 8b  fb b9 0b 00 be e6 7d f3  |.....r........}.|
> 001038e0  a6 75 0a 8d 7f 20 b9 0b  00 f3 a6 74 18 be 9e 7d  |.u... .....t...}|
> 001038f0  e8 5f 00 33 c0 cd 16 5e  1f 8f 04 8f 44 02 cd 19  |._.3...^....D...|
> 00103900  58 58 58 eb e8 8b 47 1a  48 48 8a 1e 0d 7c 32 ff  |XXX...G.HH...|2.|
> 00103910  f7 e3 03 06 49 7c 13 16  4b 7c bb 00 07 b9 03 00  |....I|..K|......|
> 00103920  50 52 51 e8 3a 00 72 d8  b0 01 e8 54 00 59 5a 58  |PRQ.:.r....T.YZX|
> 00103930  72 bb 05 01 00 83 d2 00  03 1e 0b 7c e2 e2 8a 2e  |r..........|....|
> 00103940  15 7c 8a 16 24 7c 8b 1e  49 7c a1 4b 7c ea 00 00  |.|..$|..I|.K|...|
> 00103950  70 00 ac 0a c0 74 29 b4  0e bb 07 00 cd 10 eb f2  |p....t).........|
> 00103960  3b 16 18 7c 73 19 f7 36  18 7c fe c2 88 16 4f 7c  |;..|s..6.|....O||
> 00103970  33 d2 f7 36 1a 7c 88 16  25 7c a3 4d 7c f8 c3 f9  |3..6.|..%|.M|...|
> 00103980  c3 b4 02 8b 16 4d 7c b1  06 d2 e6 0a 36 4f 7c 8b  |.....M|.....6O|.|
> 00103990  ca 86 e9 8a 16 24 7c 8a  36 25 7c cd 13 c3 0d 0a  |.....$|.6%|.....|
> 001039a0  4e 6f 6e 2d 53 79 73 74  65 6d 20 64 69 73 6b 20  |Non-System disk |
> 001039b0  6f 72 20 64 69 73 6b 20  65 72 72 6f 72 0d 0a 52  |or disk error..R|
> 001039c0  65 70 6c 61 63 65 20 61  6e 64 20 70 72 65 73 73  |eplace and press|
> 001039d0  20 61 6e 79 20 6b 65 79  20 77 68 65 6e 20 72 65  | any key when re|
> 001039e0  61 64 79 0d 0a 00 49 4f  20 20 20 20 20 20 53 59  |ady...IO      SY|
> 001039f0  53 4d 53 44 4f 53 20 20  20 53 59 53 00 00 55 aa  |SMSDOS   SYS..U.|
>
>
> > If FDD then you cannot interpret boot sector as MBR. This is wrong.
>
> It's others who interpret it and at least one takes the "fake" partition
> table entry by mformat for a reason to early stall its boot process.
>
> I just have to explain users of grub-mkrescue ISOs why they don't work.
> The plight of the programmer whose software was the last to touch it.
>
>
> > > The only thing that was found to not work is a MBR partition entry 1 which
> > > starts at LBA 0. Probably it needs to have non-zero block count and non-zero
> > > type.
>
> > Again, on what type of image? FDD or HDD?
>
> Made by the mformat run of grub-mkrescue.
> Vladimir can probably tell you more about what kind of image he expects.
>
>
> > First of all look at mformat code
>
> I did before starting this thread.
> The bytes in question are intended as "fake" partition table.
>
>
> > If mkfs.fat images work on Macbooks then I would leave mformat as
> > default and add an option to switch between mformat and mkfs.fat.
>
> Why that ? Either mformat's result is ok or it is not.

I would say that in most cases it works but is some it does not. This
is difficult to say right now where is the bug. Is it in Macbook
firmware or in mformat? Anyway, for some reason somebody decided to
create fake partition table even for FDDs. It looks that it confuses
Macbooks but not others. So, first of all I would like to understand why
somebody decided to do that before blindly dropping fake partition
table. I think that you should ask about that mformat folks on relevant
mailing list.

> Offering an alternative could start a new branch of cargo cult. Explaining
> the differences will quickly raise the question why staying with the
> unusual block0 bytes from mformat at all.

First of all I would like to understand the problem and then we will
make a decision what we do.

> I understand that GRUB wants to use a GNU program, and i am careful not
> to spoil anything that works. Thus my proposal as it is.
>
>
> I will not be overly sad if the proposal is rejected. After all it is

I have not rejected your proposal. At this point we are looking for the
best solution for the issue.

> quite easy to override the ideas of grub-mkrescue before the ISO gets
> packed up. In practice, it's preference for GPT makes more problems than
> its preference for mformat.

This is not good idea. I would like to see a fix or at least a kind of
workaround in GRUB upstream.

Daniel

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[Thomas Schmitt]

Hi,

i quoted mformat.c:
> >   /* install fake partition table pointing to itself */

Daniel Kiper wrote:
> OK, but why it is also done for FDD images?

According to its man page, mformat formats floppies. So the question
would be why it creates a partition table at all.



> > Either mformat's result is ok or it is not.

> I would say that in most cases it works but is some it does not. This
> is difficult to say right now where is the bug

Whoever is the culprit when stepping into a pitfall, grub-mkrescue should
try to avoid known bug triggers.

I understand from Vladimir's reply that he did not think of partition
tables or floppies when employing mformat in grub-mkrescue.c. He wanted
a FAT filesystem with usual entrails.
Usual is to have no data in it which look like a partition table entry
which describes a partition starting at block 0.
(Microsoft hasn't. The larger Linux distros haven't.)

So i propose the change as fine tuning of the original developer's
intention. (Vladimir will correct me if i'm wrong.)


> So, first of all I would like to understand why somebody decided to do that

As for the Macbook, my theory is that it tries to explore what it believes
to be an extended partition. Newer Macbooks are said not to make this
mistake any more.

As for mtools, binary search on
  http://svn.savannah.gnu.org/viewvc/mtools/trunk/mformat.c?view=log
yields that the comment came in with revision 4 in line 958
  http://svn.savannah.gnu.org/viewvc/mtools/trunk/mformat.c?revision=4&view=markup
Revision 2, line 800 shows no trace of a partition table until in the
end 0xaa55 is written.
Revision 4 is from 2002 and has as commit message "3.9.8, modified files".

Committer is "aknaf", probably the same as "alainknaff" who committed
to mtools last year.
  https://savannah.gnu.org/users/alainknaff
Maybe he still can remember why he added the partition entry.

--------------------------------------------------------------------------

> > After all it is quite easy to override the ideas of grub-mkrescue
> > before the ISO gets packed up. In practice, it's preference for GPT
> > makes more problems than its preference for mformat.

> This is not good idea. I would like to see a fix or at least a kind of
> workaround in GRUB upstream.

I was not happy with Vladimir's decision to use GPT instead of MBR partition
table. (Because of non-mountable partitions and the GPT backup, which needs
to be relocated after copying the ISO onto USB stick.)
Nevertheless, if GRUB wants then GRUB gets, provided that i understand what
is desired and find a way to fiddle it into the existing libisofs code.

In the course of a few bug hunts i developed the pseudo-xorriso script which
can be put between grub-mkrescue and real xorriso by option --xorriso=.
  https://dev.lovelyhq.com/libburnia/libisoburn/raw/master/frontend/grub-mkrescue-sed.sh
It gets to see the xorriso arguments and the temporary directory with the
prepared files. So it can modify them.

It spares grub-mkrescue the work to implement options like
  --mbr-rather-than-gpt
  --without-hfsplus-and-apm
  --in-efi-no-partition-table,
  --do-it-like-isohybrid-u
They all are more in the realm of xorriso than of GRUB. But the user can
hardly influence those aspects by adding some xorriso options to the
grub-mkrescue arguments.

The script is part of xorriso's project and mainly applies to x86 EFI
bootable ISOs. Of course it speculates that not much will change in
grub-mkrescue's relation to xorriso. I am subscribed here and hope to notice
any patch that would change the xorriso arguments or the fundamental layout
of the prepared file tree.


Have a nice day :)

Thomas

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[Daniel Kiper]

On Thu, May 16, 2019 at 02:18:51PM +0200, Thomas Schmitt wrote:
> Hi,
>
> i quoted mformat.c:
> > >   /* install fake partition table pointing to itself */
>
> Daniel Kiper wrote:
> > OK, but why it is also done for FDD images?
>
> According to its man page, mformat formats floppies. So the question
> would be why it creates a partition table at all.

Yeah, could you ask mformat folks about that?

> > > Either mformat's result is ok or it is not.
>
> > I would say that in most cases it works but is some it does not. This
> > is difficult to say right now where is the bug
>
> Whoever is the culprit when stepping into a pitfall, grub-mkrescue should
> try to avoid known bug triggers.

Yep but on the other hand we have to not break anything which works right now.

> I understand from Vladimir's reply that he did not think of partition
> tables or floppies when employing mformat in grub-mkrescue.c. He wanted
> a FAT filesystem with usual entrails.
> Usual is to have no data in it which look like a partition table entry
> which describes a partition starting at block 0.
> (Microsoft hasn't. The larger Linux distros haven't.)
>
> So i propose the change as fine tuning of the original developer's
> intention. (Vladimir will correct me if i'm wrong.)

After some thinking I have an itching to add the "-k" option to the
mformat. However, I am not convinced that this should go into the
release. There is a chance that we will break other platforms and fix
the only one. So, I would suggest to make this as a known issue and fix
it after the release. Does it make sense?

> > So, first of all I would like to understand why somebody decided to do that
>
> As for the Macbook, my theory is that it tries to explore what it believes
> to be an extended partition. Newer Macbooks are said not to make this
> mistake any more.
>
> As for mtools, binary search on
>   http://svn.savannah.gnu.org/viewvc/mtools/trunk/mformat.c?view=log
> yields that the comment came in with revision 4 in line 958
>   http://svn.savannah.gnu.org/viewvc/mtools/trunk/mformat.c?revision=4&view=markup
> Revision 2, line 800 shows no trace of a partition table until in the
> end 0xaa55 is written.
> Revision 4 is from 2002 and has as commit message "3.9.8, modified files".
>
> Committer is "aknaf", probably the same as "alainknaff" who committed
> to mtools last year.
>   https://savannah.gnu.org/users/alainknaff
> Maybe he still can remember why he added the partition entry.

Could you ask him about that? Please keep grub-devel in the loop.

> --------------------------------------------------------------------------
>
> > > After all it is quite easy to override the ideas of grub-mkrescue
> > > before the ISO gets packed up. In practice, it's preference for GPT
> > > makes more problems than its preference for mformat.
>
> > This is not good idea. I would like to see a fix or at least a kind of
> > workaround in GRUB upstream.
>
> I was not happy with Vladimir's decision to use GPT instead of MBR partition
> table. (Because of non-mountable partitions and the GPT backup, which needs
> to be relocated after copying the ISO onto USB stick.)

I am not sure I understand. Which GPT do you refer to?

Daniel

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[Thomas Schmitt]

Hi,

i wrote:
> > So i propose the change as fine tuning of the original developer's
> > intention. (Vladimir will correct me if i'm wrong.)

Daniel Kiper wrote:
> After some thinking I have an itching to add the "-k" option to the
> mformat.

I am not courageous enough to go that far.


> So, I would suggest to make this as a known issue and fix
> it after the release. Does it make sense?

Sure. There is a workaround, after all.


> >   https://savannah.gnu.org/users/alainknaff
> > Maybe he still can remember why he added the partition entry.

> Could you ask him about that? Please keep grub-devel in the loop.

Will do.


> > I was not happy with Vladimir's decision to use GPT instead of MBR partition
> > table. (Because of non-mountable partitions and the GPT backup, which needs
> > to be relocated after copying the ISO onto USB stick.)

> I am not sure I understand. Which GPT do you refer to?

The GPT in the grub-mkrescue ISO, which xorriso creates according to
the options sent from grub-mkrescue.

An ISO made by grub-mkrescue on Debian with "i386-efi", "i386-pc", and
"x86_64-efi" in /usr/lib/grub gets this partition table

  $ /sbin/fdisk -l output.iso
  ...
  Disklabel type: gpt
  Disk identifier: 8E0B9F07-AB1F-4B0B-BA12-764F83FF76EC

  Device      Start   End Sectors  Size Type
  output.iso1    64   339     276  138K Microsoft basic data
  output.iso2   340  6099    5760  2.8M EFI System
  output.iso3  6100 34131   28032 13.7M Apple HFS/HFS+
  output.iso4 34132 34731     600  300K Microsoft basic data

Partitions 1 and 4 are fillers. Not mountable.
Whether partition 3 is mountable depends on the kernel's capability to
work with HFS+ block size 2048. It is supposed show a HFS+ version of the
ISO's file tree. But i am not sure whether this was ever challanged.
The ISO is only mountable via the base device (start 0).

The fillers are necessary because the EFI partition image is a file
inside the ISO filesystem. Number 1 protects vital parts of the ISO from
being claimed by user-made partitions. Number 4 protects cargo cult
padding to prevent the TAO CD Linux read-ahead bug caused by ambiguous
specs for SCSI command READ CAPACITY ... whew, without breathing.

I prefer this layout

  Disklabel type: dos
  Disk identifier: 0x00000000

  Device      Boot Start   End Sectors  Size Id Type
  output.iso1 *       64 28695   28632   14M 83 Linux
  output.iso2      28696 34455    5760  2.8M ef EFI (FAT-12/16/32)

It gets achieved by

  export MKRESCUE_SED_MODE=mbr_hfs
  grub-mkrescue --xorriso=./grub-mkrescue-sed.sh -o output.iso minimal \
                -partition_offset 16 -iso_mbr_part_type 0x83

where "minimal" is a dummy directory with a dummy file.

The pseudo-xorriso moves the EFI image out of the GRUB-prepared tree
(e.g. /tmp/grub.Q98DOe) and replaces the xorrisofs options for EFI-in-ISO
and GPT by options for EFI-appended and MBR partition table.
The options in the second line get forwarded by grub-mkrescue as xorrisofs
options.

Partition 1 is mountable as ISO 9660 filesystem.

This run produces HFS+ and Apple Partition Map. The HFS+ range is not
represented in the MBR partition table, because it is inside the ISO range.

----------------------------------------------------------------------
History and motivation:

Back in 2012, xorriso offered no way to put an El Torito image outside
the ISO filesystem's range. So the choice was to have two EFI images or
to have the EFI partition inside the ISO's data file range.
I am not aware how much Vladimir considered the duplication. His wishes
were for the EFI partition inside the ISO.

Since a while, xorriso can point by an El Torito catalog entry to an
appended partition. So the EFI image can be stored outside the ISO's
claimed range and a mountable ISO partition does not have to overlap
with the EFI partition.

There remains the problem that tools inquire the ISO's claimed size
to determine the size of the overall ISO+EFI image, but the partition's
filesystem must not claim more than the partition's size.
Other tools hate the start LBA 0 of the mountable ISO partition.

Both can be solved at the cost of a second ISO superblock and directory
tree which match a partition start at 64 * 512 bytes. That's the job of
  -partition_offset 16
It creates an ISO superblock for the whole device, which claims the whole
ISO+EFI image as its range. The second superblock claims only the ISO
partition as its range.

We had an example of an EFI implementation, which hated any MBR partition
table type except 0x00 and 0xef. So grub-mkrescue-sed.sh uses type 0x00
by default.
Nevertheless, for the purpose of grub-mkrescue plus data partition on
a USB stick, it is better to have a non-zero MBR partition type. So my
above example uses
  -iso_mbr_part_type 0x83

----------------------------------------------------------------------

Have a nice day :)

Thomas

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[Chris Murphy]

Sorry, I lost track of this thread, but read through it and doesn't
appear I need to provide anymore info.

I would not be surprised if there are either bugs or intentional
oddity with the firmware in Apple hardware. Only recently is it based
on UEFI 2.x, for a long time it was based on Intel EFI 1.10 with a lot
of Apple only stuff added on top of that. It would not at all surprise
me if much of this is non-standard, not documented, and varies from
model to model. In fact it would not surprise me if it varied quite a
lot based on firmware revision for the same model.

For a long time, Apple's strategy for (back then) Mac OS X and Windows
dual boot was to use Intel/Apple EFI boot for OS X; and to use a CSM
mode to present BIOS to Windows. My vague recollection to trigger CSM
enabled:

1. boot disk must have "hybrid MBR" i.e. PMBR replaced with an MBR
with a partition that "protected" the LBA range for Mac OS X
partitions, and a second partition as a mirror of the GPT's Windows
partition. Super screwy and I have a million posts on Apple forums how
to fix that crazy shit when users would try to install a 3rd operating
system, which of course replaced the hybrid MBR with a PMBR and would
blow up the ability to boot Windows.
2. Bootloader code in the first 440 bytes of LBA 0.
Those two things must be true to even get the user visible option to
boot Windows
3. Choose the icon labeled Windows (either in the firmware's graphical
boot manager; or the Mac OS X startup panel which I guess wrote some
hint into NVRAM to boot Windows).

Also, I found that if I formatted the internal drive's EFI system
partition as FAT12, the firmware would faceplant. No error, no booting
of anything, just a black screen. Reformat as either FAT16 or FAT32
and it was OK.

Anyway, the point is, I'm not surprised goofy things happen with Apple
firmware. Perhaps circa 2015 or 2016 there's hardware that will EFI
boot Windows but I don't have such hardware.

So yeah, with Macs you very well might have to poke it with a stick to
find out how it works. And then whatever software change you need to
make in the pre-boot environment, even just a partition map, could be
so weird that you have to poke a lot of other hardware with that same
stick the same way with the same moon phase, to make sure you haven't
accidentally triggered some goofy firmware bug in otherwise working
hardware. I sorta sympathize with people who buy expensive hardware
and it does crazy shit they didn't think they were buying. But, life
is full of unfair, and this is pretty predictable what happens when
companies depart from specs and standards. And it's a big reason why I
haven't bought Apple hardware since 2011.

But in the meantime, I also have ISO images from Apple, the original
ones for this 2011 Macbook Pro, if anyone wants to look at that
madness.

I can tell you the Microsoft Windows ISO (udf) dd imaged to a USB
stick will not boot either my Macbook Pro or my hp spectre (2016). I
have to MBR or GPT partition a USB stick, format it FAT32, and cp -a
the files onto it, that the HP will boot, still the Mac will not (that
2011 Mac will not USB boot with the CSM; CSM booting is only possible
with optical and internal drives).

Anyway, esoteric stuff.

---
Chris Murphy

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[pelzflorian (Florian Pelz)]

On Fri, Jun 14, 2019 at 06:15:00PM -0600, Chris Murphy wrote:
> I would not be surprised if there are either bugs or intentional
> oddity with the firmware in Apple hardware. Only recently is it based
> on UEFI 2.x, for a long time it was based on Intel EFI 1.10 with a lot
> of Apple only stuff added on top of that. It would not at all surprise
> me if much of this is non-standard, not documented, and varies from
> model to model. In fact it would not surprise me if it varied quite a
> lot based on firmware revision for the same model.
>

The changes recommended by Thomas Schmitt not only made the iso image
created by grub-mkrescue work for my Mid 2010 Macbook Pro, they also
made it work on a Late 2009 iMac (irrelevant details see
<https://bugs.gnu.org/35856>).  It appears post-2010 Macs boot all
grub-mkrescue iso images without issue.

Regards,
Florian

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[Chris Murphy]

On Tue, Apr 30, 2019 at 5:43 PM Vladimir 'phcoder' Serbinenko
<phcoder@gmail.com> wrote:
>
> There is an easy way to check if -k can be used always: have a look at windows install CD/DVD. If it doesn't have partion table then we can drop it as well.

I think there's no partition whatsoever, plain UDF. I have a Microsoft
produced (not OEM) Windows 10 ISO.

blkid:
/dev/loop0: UUID="5d9a00004d532055" LABEL="CCCOMA_X64FRE_EN-US_DV9" TYPE="udf"

mount
/dev/loop0 on /mnt/iso type udf (ro,relatime,utf8)

dmesg
[1277852.561098] UDF-fs: warning (device loop0): udf_load_vrs: No anchor found
[1277852.566766] UDF-fs: Scanning with blocksize 512 failed
[1277852.573609] UDF-fs: warning (device loop0): udf_load_vrs: No anchor found
[1277852.579255] UDF-fs: Scanning with blocksize 1024 failed
[1277852.585615] UDF-fs: INFO Mounting volume 'UDF Volume', timestamp
2018/04/13 00:00 (1000)

So it's a 2048 byte block size (expected) optical image.

sudo dd if=/dev/loop0 bs=1M count=4 2>/dev/null | hexdump -C >
win10-4M-hexdump.txt

That file as win10-4M-hexdump.txt.tbz can be located here for a little
while; if more than the first 4 MiB are needed, let me know.
https://drive.google.com/open?id=1JgPv8EBHDn5A7hTEwewGK1MuFHvHbJnr



-- 
Chris Murphy

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[Thomas Schmitt]

Hi,

Vladimir 'phcoder' Serbinenko wrote:
> > have a look at windows install CD/DVD.

Chris Murphy wrote:
> I think there's no partition whatsoever, plain UDF.
>  LABEL="CCCOMA_X64FRE_EN-US_DV9"

debian-user has a thread about a similar image: CCCOMA_X64FRE_JA-JP_DV9
  https://www.mail-archive.com/debian-user@lists.debian.org/msg741966.html

The thread starter, Mark Fletcher, inspected it by
  xorriso -indev .../Win10_1809Oct_v2_Japanese_x64.iso \
          -report_system_area plain -report_el_torito plain
There is an ISO 9660 superblock and El Torito boot images for BIOS and
EFI. No MBR signature, no other partition table known to xorriso.
  https://www.mail-archive.com/debian-user@lists.debian.org/msg741984.html

And then RESOLVED:
  https://www.mail-archive.com/debian-user@lists.debian.org/msg742063.html
  "The issue was Secure Boot. Specifically, in the case of my test machine,
   it's too old to support secure boot, and in the case of the final target
   machine, secure boot was turned off in the BIOS so that I could boot
   Buster which is what this machine was previously running. When I
   switched the CSM and also turned on Secure Boot (both settings needed to
   be changed) in the BIOS, the machine finally booted from the USB stick,
   and all proceeded smoothly from there."

Whatever happened there, it looks like it has booted via El Torito.
That's contrary to all belief and known specs.


> https://drive.google.com/open?id=1JgPv8EBHDn5A7hTEwewGK1MuFHvHbJnr
>
> 00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |......
> *
> 00008000  01 43 44 30 30 31 01 00  20 20 20 20 20 20 20 20  |.CD001.

Absolutely zero System Area. No BIOS boot code for hard-disk-like devices.
The first non-zero bytes are the ISO 9660 Primary Volume Descriptor.


> 00008800  00 43 44 30 30 31 01 45  4c 20 54 4f 52 49 54 4f
>  |.CD001.EL TORITO|

Here begins the El Torito Boot Record.


> 0000b000  01 00 00 00 4d 69 63 72  6f 73 6f 66 74 20 43 6f  |....Micr
> 0000b010  72 70 6f 72 61 74 69 6f  6e 00 00 00 4c 49 55 aa  |rporatio
> 0000b020  88 00 00 00 00 00 08 00  05 02 00 00 00 00 00 00  |........
> 0000b030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |........
> 0000b040  91 ef 01 00 00 00 00 00  00 00 00 00 00 00 00 00  |........
> 0000b050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |........
> 0000b060  88 00 00 00 00 00 01 00  07 02 00 00 00 00 00 00  |........

El Torito Boot Catalog announces images for BIOS and EFI.


> 00009800  00 42 45 41 30 31 01 00  00 00 00 00 00 00 00 00  |.BEA01...

... and there is UDF, of course ...


So how could this boot if not via El Torito ?

It is known that OVMF boots El Torito from virtual hard disk.
Tianocore's EDK-2 indiscriminately registers FAT filesystems from GPT,
MBR partition table, and El Torito. At least at it's Pre-EFI stage:
  https://github.com/tianocore/edk2/blob/master/FatPkg/FatPei/Part.c
  https://github.com/tianocore/edk2/blob/master/FatPkg/FatPei/Eltorito.c

But can Microsoft Inc. really bet on all firmwares to ignore specs and
traditions ?
(I feel lost right now. Vladimir, are you somewhere out there ?)


Have a nice day :)

Thomas

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[Michael Schierl]

[Please cc: me since I am not subscribed to that list]


Hello,


I know this is getting off-topic, but since I am surprised how many
people here (have to) base their knowledge of booting Windows install
media from rumors and "advertising", I decided to bring in my knowledge
and personal experience about the Windows installer boot process.


>> > This is not an install ISO but a USB install medium.  The USB drive
>> > was written by the Windows Media Creation Tool.  Copying instead a
>> > Windows ISO created by the Windows Media Creation Tool to a USB drive
>> > does *not* yield a bootable USB drive; it does not even boot on other
>> > computers.
>>
>> This would bring the MS-World back in sync with my understanding of UEFI
>> specs and other info sources.
>>
>> But there are substantial rumors that the ISO boots from USB stick.
>> Microsoft is said to advertise it for DVD+R DL and USB stick.
>> (I could not get my hands on ISO or advertising yet ...)

Yes, any Windows install ISO starting from Windows Vista supports
installation from a USB stick. However, that does *not* mean that you
can use a tool like Unix's "dd" to write it to the drive and expect it
to boot (of course you can try, but it won't boot; valid for all Windows
up to latest Windows 10 Insider version). Instead, assuming that you
already have some Windows machine, you can either use the Media Creation
Tool to write the ISO to your USB, or you can use the command line
utilities included in the ISO itself.

On the other hand, in the Windows world this rarely causes any confusion
since no Windows version I've encountered so far contains native tools
to write an ISO (or any other image file) to a USB stick as Unix' "dd"
command would. Starting from Vista you can natively write ISO files to
optical media and since Windows 8 you can mount ISOs as a drive without
burning them, but that "just dd it to a USB key" feature of isohybrid is
mainly useful to people who already have some kind of Unix running.

In case you ever run into the situation to create a USB key from an
Windows install ISO file from another OS than Windows (and you want that
media to boot both on UEFI and CSM), you'd have to resort to third-party
tools like ms-sys that can replicate this installation process, or
chainload the Windows boot loader from another bootloader that supports
it (e.g. from Grub - to get this discussion back on-topic). Or if you
already have such a USB stick of any other Windows version (since
Vista), just mount it, delete all the files and copy all the files from
inside the ISO to it.

Another thing to keep in mind is that until (at least) Windows 7, the
Disk Management GUI refused to change partitioning of external
"removable" mass storage devices, so in case you ever "dd" a smaller ISO
to a larger stick, you would have to resort to third-party software or
the command-line "diskpart" command to regain your USB key's capacity
back. That may be another reason that dd-able ISOhybrid files are mostly
used for Linux installs (since Linux does not have that restriction) and
not used for Windows install media.

> I wonder how other people can get to the impression that their attempts
> succeeded. (Will have to wait for somebody who is willing to go on with
> experiments after perceived success.)

They probably did not use "dd" in the first place to get their ISO to
the USB stick.

>> I will continue testing the Windows 10 USB install medium:
>> /dev/sdb1  *     2048 15728639 15726592  7.5G  c W95 FAT32 (LBA)
>
> That's not an EFI System Partition.
> So i assume the USB stick boots by BIOS and x86 machine code in the MBR.
> Unless we believe in EFI firmware that hops on arbitrary partitions ...

I do not believe that EFI firmware hops on arbitrary partitions either.

On the other hand, all 64-bit EFI firmware versions I encountered so far
(to be honest, I believe all of them were "designed to run Windows"), do
EFI boot from the *first* MBR-style partition of an *external* drive
(e.g. USB), if the partition type is either 0x0B or 0x0C, the partition
is marked bootable, and it contains a FAT32 filesystem with an
/EFI/BOOT/BOOTX64.EFI file on it. I have encountered machines who do not
boot it if only one of these conditions is unmet (different partition
type, not marked bootable, not the first partition slot, or drive was
attached internally via SATA).

I do not know if this is specified anywhere, but if it did not work,
people would complain quickly. My own multiboot USB key (which uses GRUB
for chainloading multiple Linux ISOs via loopback.cfg) also uses that
exact disk layout, and I never had any machine where it could not boot.
(As it has a USB-OTG port, I can even load new ISOs to it from my
smartphone). On the other hand, if I partitioned the USB stick in GPT
mode and used an EFI System Partition as primary partition, Windows
would not assign a drive letter to it when I attach it, and older OSes
would not be able to read the stick at all. (I never heard that the
Windows installer "broke an USB stick", but I heard it multiple times
about Linux installers that used isohybrid. Getting rid of all
partitions always "cured" the USB stick again, but many Windows users
are unable to fix this themselves).


And that is exactly what happens when you "write" a Windows Installer
ISO to a USB key:

It will create a MBR-style partition table with one "bootable" 0x0C
entry spanning the entire device, formats it at FAT32 and copies all the
files in the ISO to it. (In case install.wim is larger than 4GB and the
media creation tool is recent enough, it will run SplitWim to create
install.swm, install2.swm etc. from it and put those on the device
instead). Afterwards bootsect.exe is executed, which writes a MBR and a
partition boot record so that booting in legacy mode works (this is not
required for booting in UEFI mode, but does not hurt either).

So, while the ISO used a UDF filesystem, the USB stick will use a FAT32
filesystem, and since bootmgr can read both, installation will succeed
in either case.

> I do not know how to verify on Windows if the boot used UEFI.  Shall I continue testing?

On a booted (installed) Windows, the easiest way is running msinfo32.exe
(which shows "BIOS Mode: UEFI" in case booted from UEFI, and won't show
it in case it is booted in legacy mode).


When you are in the installer (or any other Preinstallation Environment
or Recovery Environment), you'd have to go to the registry editor (in
case of a standard install media, press next to get to the second
install screen, press Shift+F10 to open a command prompt, and run
"regedit" from there) and check the value of

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\PEFirmwareType

It is 1 for Legacy mode and 2 for UEFI mode.


Another option would be to either damage the bootx64.efi file or the
MBR/PBR of the drive.

Also there are two versions of the Boot Configuration Database; the
Legacy one as \BOOT\BCD and the EFI one as \EFI\MICROSOFT\BOOT\BCD (this
is used by dual-arch ISOs so that in BIOS mode you can choose if you
want 32- or 64-bit, but in UEFI mode it will pick 64-bit automatically).
If you delete/rename one of these, the boot manager will stop with an
error instead of booting.


Regards,


Michael

Re: grub-mkrescue: Problem with MBR partition table at start of EFI partition

[Thomas Schmitt]

Hi,

> On the other hand, in the Windows world this rarely causes any confusion
> since no Windows version I've encountered so far contains native tools
> to write an ISO (or any other image file) to a USB stick as Unix' "dd"
> command would.

A Wallace Line for ISO-zooa. This explains a lot.


> that "just dd it to a USB key" feature of isohybrid is
> mainly useful to people who already have some kind of Unix running.

Rufus has a "DD" mode, meanwhile with auto-detection of worthy ISOs:
  https://github.com/pbatard/rufus/issues/1148


> [Disk Management GUI refusal ...]
> That may be another reason that dd-able ISOhybrid files are mostly
> used for Linux installs (since Linux does not have that restriction) and
> not used for Windows install media.

Actually in GNU/Linux and similar systems it is darn dangerous to dd
an ISO onto a USB stick. You normally need privileges which suffice to
overwrite your system disk.

I wonder whether it was Vladimir Serbinenko or H. Peter Anvin who
first had the idea to let MBR x86 boot code hop onto the El Torito
no-emulation boot image. I guess the term "isohybrid" is by hpa.


> On the other hand, all 64-bit EFI firmware versions I encountered so far
> (to be honest, I believe all of them were "designed to run Windows"), do
> EFI boot from the *first* MBR-style partition of an *external* drive
> (e.g. USB), if the partition type is either 0x0B or 0x0C, the partition
> is marked bootable, and it contains a FAT32 filesystem with an
> /EFI/BOOT/BOOTX64.EFI file on it.

This is quite what the rumors say, if i replace their riddling details
by yours.
It also matches what Florian reports about his USB stick's partitioning.

In contrast, UEFI 2.4, 5.2.1 Legacy Master Boot Record (MBR):

  "Table 14. Legacy MBR Partition Record
   [...]
   BootIndicator : Byte Offset 0 :
   0x80 indicates that this is the bootable legacy partition. Other
   values indicate that this is not a bootable legacy partition.
   This field shall not be used by UEFI firmware.
   [...]
   If an MBR partition has an OSType field of 0xEF (i.e., UEFI System
   Partition), then the firmware must add the UEFI System Partition GUID
   to the handle for the MBR partition using InstallProtocolInterface().
   This allows drivers and applications, including OS loaders,
   to easily search for handles that represent UEFI System Partitions."


> if it did not work, people would complain quickly.

Yeah. This is the reason why i care to learn. Big truck slipstream.
(Even if i afterwards need to get hot water and iodine ...)


> On the other hand, if I partitioned the USB stick in GPT
> mode and used an EFI System Partition as primary partition, Windows
> would not assign a drive letter to it when I attach it, and older OSes
> would not be able to read the stick at all.

I am a big fan of MBR partition table with partition of type 0xEF.
Therefore i provide a script-in-the-middle for grub-mkrescue to be
submitted as option --xorriso=
  https://dev.lovelyhq.com/libburnia/libisoburn/raw/master/frontend/grub-mkrescue-sed.sh
It can let grub-mkrescue produce a partition layout like:

  Device                     Boot Start   End Sectors  Size Id Type
  grub-mkrescue-offst16.iso1 *       64 24171   24108 11.8M 83 Linux
  grub-mkrescue-offst16.iso2      24172 29931    5760  2.8M ef EFI (FAT

(Made with xorriso options -partition_offset 16 -iso_mbr_part_type 0x83)
Partition 1 is mountable as ISO 9660. The whole device too. The latter
claims the full image size including EFI partition as its filesystem size.

------------------------------------------------------------------------

Many thanks for sharing this info. It will help me a lot when next time
i see optimistic proposals about how to make ISOs bootable from USB stick.

If anybody knows a public document where the tolerant firmware habit is
adopted by some standards entity (or market power), then i would be glad
to learn.


Have a nice day :)

Thomas