Dell BIOS issue reading Disk Extended data

Dell BIOS issue reading Disk Extended data

[Guilherme G. Piccoli]

Hello Dell folks, I'm Guilherme Piccoli from Canonical - first of all,
apologies for the out-of-nowhere communication. We've been investigating
an issue that seems to date long time ago, and eventually we could
narrow it to what appears to be a Dell BIOS bug. Notice I'm also looping
a kernel x86 ML and grub-devel, just for the purpose of archiving such
discussion in public lists, to help others that may find such an issue
in the future.

Since I don't have contacts of Dell representatives, I've just raised a
list of people from Dell contributing to kernel in the last 2 years -
maybe one of you could point me towards the path of a proper
contact/channel to discuss such an issue. If not, I'm sorry for the noise.
Let me detail the problem we're observing - notice all of this is about
legacy BIOS mode, not UEFI.

After creating a HW RAID on a Dell PowerEdge R730 (RAID5, total of 8T),
GRUB fails to load its modules, dropping to "rescue mode". After a lot
of investigation, we narrowed the issue to a bad return from BIOS to
service 48h, int 13h [0] - this is the way GRUB collects disk size
information. To double-check that, I've booted Linux in 16-bit realmode
and with that, I could observe that EDD module [1] gets the same wrong
value as total sectors - both GRUB and kernel EDD returns 0xFFFFFFFF.
The correct value would be 0x3A3600000 according to SCSI Read Capacity
16 command (tested through the sg_readcap tool). In the P.S. session
below there are details of the outputs collected by GRUB
instrumentation, kernel EDD and sg_readcap tool.

There are some workarounds to that, like having a smaller partition
_before_ the rootfs in the disk topology, to hold grub modules and
linux/initrd images - in that case it seems the BIOS responds the int
13h/48h service with proper values, but this issue dates from a while
ago [3][4], so I'm hereby seeking a proper discussion with Dell firmware
engineers to understand if that could be fixed or at least to understand
the root cause of such limitation.
Thanks in advance,


Guilherme


[0]
https://en.wikipedia.org/wiki/INT_13H#INT_13h_AH=48h:_Extended_Read_Drive_Parameters

[1]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/boot/edd.c

[2] https://askubuntu.com/q/867047

[3] https://askubuntu.com/q/416418


P.S. GRUB debug output [dump of struct grub_biosdisk_drp in
grub_biosdisk_get_diskinfo_real() function]:

size=1e, flags=9
cyl=0, heads=0, sec=0
bytesp_s=200, total=ffffffff,


kernel EDD output:
[    0.741378] edd[0]->total_secs=ffffffff


sg_readcap output:
$ sg_readcap /dev/sdb
READ CAPACITY (10) indicates device capacity too large
  now trying 16 byte cdb variant
Read Capacity results:
   Protection: prot_en=0, p_type=0, p_i_exponent=0
   Logical block provisioning: lbpme=0, lbprz=0
   Last logical block address=15625879551 (0x3a35fffff), Number of
logical blocks=15625879552
[...]

RE: Dell BIOS issue reading Disk Extended data

[Limonciello, Mario]

> 
> Hello Dell folks, I'm Guilherme Piccoli from Canonical - first of all,
> apologies for the out-of-nowhere communication. We've been investigating
> an issue that seems to date long time ago, and eventually we could
> narrow it to what appears to be a Dell BIOS bug. Notice I'm also looping
> a kernel x86 ML and grub-devel, just for the purpose of archiving such
> discussion in public lists, to help others that may find such an issue
> in the future.
> 
> Since I don't have contacts of Dell representatives, I've just raised a
> list of people from Dell contributing to kernel in the last 2 years -
> maybe one of you could point me towards the path of a proper
> contact/channel to discuss such an issue. If not, I'm sorry for the noise.
> Let me detail the problem we're observing - notice all of this is about
> legacy BIOS mode, not UEFI.

Most of the guys you CC'ed from Dell work on client related things, not
servers.  So I'll move some of the client guys off of this thread into BCC.
@K, Narendra can you please raise this with the appropriate team?

> 
> After creating a HW RAID on a Dell PowerEdge R730 (RAID5, total of 8T),
> GRUB fails to load its modules, dropping to "rescue mode". After a lot
> of investigation, we narrowed the issue to a bad return from BIOS to
> service 48h, int 13h [0] - this is the way GRUB collects disk size
> information. To double-check that, I've booted Linux in 16-bit realmode
> and with that, I could observe that EDD module [1] gets the same wrong
> value as total sectors - both GRUB and kernel EDD returns 0xFFFFFFFF.
> The correct value would be 0x3A3600000 according to SCSI Read Capacity
> 16 command (tested through the sg_readcap tool). In the P.S. session
> below there are details of the outputs collected by GRUB
> instrumentation, kernel EDD and sg_readcap tool.
> 
> There are some workarounds to that, like having a smaller partition
> _before_ the rootfs in the disk topology, to hold grub modules and
> linux/initrd images - in that case it seems the BIOS responds the int
> 13h/48h service with proper values, but this issue dates from a while
> ago [3][4], so I'm hereby seeking a proper discussion with Dell firmware
> engineers to understand if that could be fixed or at least to understand
> the root cause of such limitation.
> Thanks in advance,
> 
> 
> Guilherme
> 
> 
> [0]
> https://en.wikipedia.org/wiki/INT_13H#INT_13h_AH=48h:_Extended_Read_Drive_Para
> meters
> 
> [1]
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x
> 86/boot/edd.c
> 
> [2] https://askubuntu.com/q/867047
> 
> [3] https://askubuntu.com/q/416418
> 
> 
> P.S. GRUB debug output [dump of struct grub_biosdisk_drp in
> grub_biosdisk_get_diskinfo_real() function]:
> 
> size=1e, flags=9
> cyl=0, heads=0, sec=0
> bytesp_s=200, total=ffffffff,
> 
> 
> kernel EDD output:
> [    0.741378] edd[0]->total_secs=ffffffff
> 
> 
> sg_readcap output:
> $ sg_readcap /dev/sdb
> READ CAPACITY (10) indicates device capacity too large
>   now trying 16 byte cdb variant
> Read Capacity results:
>    Protection: prot_en=0, p_type=0, p_i_exponent=0
>    Logical block provisioning: lbpme=0, lbprz=0
>    Last logical block address=15625879551 (0x3a35fffff), Number of
> logical blocks=15625879552
> [...]

Re: Dell BIOS issue reading Disk Extended data

[Guilherme Piccoli]

On Fri, Jan 22, 2021 at 3:41 PM Limonciello, Mario
<Mario.Limonciello@dell.com> wrote:
>
> >
> > Hello Dell folks, I'm Guilherme Piccoli from Canonical - first of all,
> > apologies for the out-of-nowhere communication. We've been investigating
> > an issue that seems to date long time ago, and eventually we could
> > narrow it to what appears to be a Dell BIOS bug. Notice I'm also looping
> > a kernel x86 ML and grub-devel, just for the purpose of archiving such
> > discussion in public lists, to help others that may find such an issue
> > in the future.
> >
> > Since I don't have contacts of Dell representatives, I've just raised a
> > list of people from Dell contributing to kernel in the last 2 years -
> > maybe one of you could point me towards the path of a proper
> > contact/channel to discuss such an issue. If not, I'm sorry for the noise.
> > Let me detail the problem we're observing - notice all of this is about
> > legacy BIOS mode, not UEFI.
>
> Most of the guys you CC'ed from Dell work on client related things, not
> servers.  So I'll move some of the client guys off of this thread into BCC.
> @K, Narendra can you please raise this with the appropriate team?
>

Thanks Mario and K, Narendra - do we have any news about that?
Cheers,


Guilherme

RE: Dell BIOS issue reading Disk Extended data

[K, Narendra]

Internal Use - Confidential

> -----Original Message-----
> From: Guilherme Piccoli <gpiccoli@canonical.com>
> Sent: Friday, March 12, 2021 9:14 PM
> To: Limonciello, Mario; K, Narendra
> Cc: Barabash, Alexander; Engel, Amit; Guilherme G. Piccoli;
> halves@canonical.com; Jay Vosburgh; Dan Streetman; Gavin Guo;
> x86@kernel.org; grub-devel@gnu.org
> Subject: Re: Dell BIOS issue reading Disk Extended data
> 
> 
> [EXTERNAL EMAIL]
> 
> On Fri, Jan 22, 2021 at 3:41 PM Limonciello, Mario
> <Mario.Limonciello@dell.com> wrote:
> >
> > >
> > > Hello Dell folks, I'm Guilherme Piccoli from Canonical - first of
> > > all, apologies for the out-of-nowhere communication. We've been
> > > investigating an issue that seems to date long time ago, and
> > > eventually we could narrow it to what appears to be a Dell BIOS bug.
> > > Notice I'm also looping a kernel x86 ML and grub-devel, just for the
> > > purpose of archiving such discussion in public lists, to help others
> > > that may find such an issue in the future.
> > >
> > > Since I don't have contacts of Dell representatives, I've just
> > > raised a list of people from Dell contributing to kernel in the last
> > > 2 years - maybe one of you could point me towards the path of a
> > > proper contact/channel to discuss such an issue. If not, I'm sorry for the
> noise.
> > > Let me detail the problem we're observing - notice all of this is
> > > about legacy BIOS mode, not UEFI.
> >
> > Most of the guys you CC'ed from Dell work on client related things,
> > not servers.  So I'll move some of the client guys off of this thread into BCC.
> > @K, Narendra can you please raise this with the appropriate team?
> >
> 
> Thanks Mario and K, Narendra - do we have any news about that?
> Cheers,

Hi Guilherme,

Sorry for the delay in responding.  

I checked with the team about the findings shared in this thread. Please find the details -

The behavior seems to be working as designed. Legacy BIOS Option ROM implements logical block addressing support only up to 2TB. 
The option ROM will respond with 2TB as the size when queried by INT13h function 48h. For booting from volumes > 2TB, UEFI is needed.

With regards,
Narendra K

Re: Dell BIOS issue reading Disk Extended data

[Guilherme Piccoli]

On Tue, Mar 30, 2021 at 2:43 PM K, Narendra <Narendra.K@dell.com> wrote:
> > On Fri, Jan 22, 2021 at 3:41 PM Limonciello, Mario
> > <Mario.Limonciello@dell.com> wrote:
> > >
> > > >
> > > > Hello Dell folks, I'm Guilherme Piccoli from Canonical - first of
> > > > all, apologies for the out-of-nowhere communication. We've been
> > > > investigating an issue that seems to date long time ago, and
> > > > eventually we could narrow it to what appears to be a Dell BIOS bug.
> > > > Notice I'm also looping a kernel x86 ML and grub-devel, just for the
> > > > purpose of archiving such discussion in public lists, to help others
> > > > that may find such an issue in the future.
> > > >
> > > > Since I don't have contacts of Dell representatives, I've just
> > > > raised a list of people from Dell contributing to kernel in the last
> > > > 2 years - maybe one of you could point me towards the path of a
> > > > proper contact/channel to discuss such an issue. If not, I'm sorry for the
> > noise.
> > > > Let me detail the problem we're observing - notice all of this is
> > > > about legacy BIOS mode, not UEFI.
> > >
> > > Most of the guys you CC'ed from Dell work on client related things,
> > > not servers.  So I'll move some of the client guys off of this thread into BCC.
> > > @K, Narendra can you please raise this with the appropriate team?
> > >
> >
> > Thanks Mario and K, Narendra - do we have any news about that?
> > Cheers,
>
> Hi Guilherme,
>
> Sorry for the delay in responding.
>

No problem Narendra K, thanks for your response!

> I checked with the team about the findings shared in this thread. Please find the details -
>
> The behavior seems to be working as designed. Legacy BIOS Option ROM implements logical block addressing support only up to 2TB.
> The option ROM will respond with 2TB as the size when queried by INT13h function 48h. For booting from volumes > 2TB, UEFI is needed.

So, just confirming, Dell doesn't support > 2TB offsets even using the
extended 64-bit offset from service 48h? I just want to confirm that
is a support limitation from Dell, I've tested another vendor and that
works in their machine, I could address > 2TB devices.

In case this is unsupported by Dell, do you have an official
documentation I can refer, to our customer?
Thanks again,

Guilherme

Re: Dell BIOS issue reading Disk Extended data

[Jordan Uggla]

This should *not* be made default because grub's native drivers don't
get very much testing, and they do cause grub to hang on some
hardware, but you can likely get around this limitation with:

grub-install --target=i386-pc --disk-module native /dev/sdX

If you want to test it without installing grub in this way to your
primary drive(s), you can boot from a grub USB and run "nativedisk" at
the grub shell. This will switch from firmware based disk access to
entirely grub native driver access (while accounting for the device
name changes and updating $prefix accordingly). For safety reasons,
this will disable firmware based access to all drives.

This works by including grub's native drivers in the embedded
core.img. For GPT disks (or arrays), which I assume you're using with
8TB capacity, grub's core.img is embedded in the BIOS Boot Partition.
As long as the BIOS Boot Partition is within 2TB of the beginning of
the disk (usually, it's just the first partition, and is 1MiB large)
you'll be able to load it just fine through INT13 and from that point
on (if your hardware is supported) you'll no longer be depending on
the boot firmware or be affected by its disk access bugs.


On Wed, Mar 31, 2021 at 7:45 AM Guilherme Piccoli
<gpiccoli@canonical.com> wrote:
>
> On Tue, Mar 30, 2021 at 2:43 PM K, Narendra <Narendra.K@dell.com> wrote:
> > > On Fri, Jan 22, 2021 at 3:41 PM Limonciello, Mario
> > > <Mario.Limonciello@dell.com> wrote:
> > > >
> > > > >
> > > > > Hello Dell folks, I'm Guilherme Piccoli from Canonical - first of
> > > > > all, apologies for the out-of-nowhere communication. We've been
> > > > > investigating an issue that seems to date long time ago, and
> > > > > eventually we could narrow it to what appears to be a Dell BIOS bug.
> > > > > Notice I'm also looping a kernel x86 ML and grub-devel, just for the
> > > > > purpose of archiving such discussion in public lists, to help others
> > > > > that may find such an issue in the future.
> > > > >
> > > > > Since I don't have contacts of Dell representatives, I've just
> > > > > raised a list of people from Dell contributing to kernel in the last
> > > > > 2 years - maybe one of you could point me towards the path of a
> > > > > proper contact/channel to discuss such an issue. If not, I'm sorry for the
> > > noise.
> > > > > Let me detail the problem we're observing - notice all of this is
> > > > > about legacy BIOS mode, not UEFI.
> > > >
> > > > Most of the guys you CC'ed from Dell work on client related things,
> > > > not servers.  So I'll move some of the client guys off of this thread into BCC.
> > > > @K, Narendra can you please raise this with the appropriate team?
> > > >
> > >
> > > Thanks Mario and K, Narendra - do we have any news about that?
> > > Cheers,
> >
> > Hi Guilherme,
> >
> > Sorry for the delay in responding.
> >
>
> No problem Narendra K, thanks for your response!
>
> > I checked with the team about the findings shared in this thread. Please find the details -
> >
> > The behavior seems to be working as designed. Legacy BIOS Option ROM implements logical block addressing support only up to 2TB.
> > The option ROM will respond with 2TB as the size when queried by INT13h function 48h. For booting from volumes > 2TB, UEFI is needed.
>
> So, just confirming, Dell doesn't support > 2TB offsets even using the
> extended 64-bit offset from service 48h? I just want to confirm that
> is a support limitation from Dell, I've tested another vendor and that
> works in their machine, I could address > 2TB devices.
>
> In case this is unsupported by Dell, do you have an official
> documentation I can refer, to our customer?
> Thanks again,
>
> Guilherme
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel