PXEgrub development on grub2

PXEgrub development on grub2

[Lars Nooden]

I see that all new work is on Grub2.  What timeline is there for
bringing the pxegrub functionality forward to grub2 from legacy grub?

Regards
-Lars

Re: PXEgrub development on grub2

[Felix Zielcke]

Am Mittwoch, den 09.09.2009, 17:07 +0300 schrieb Lars Nooden:
> I see that all new work is on Grub2.  What timeline is there for
> bringing the pxegrub functionality forward to grub2 from legacy grub?
> 
> Regards
> -Lars
> 

I don't know what pxegrub can do, but GRUB 2 has PXE support:
http://grub.enbug.org/PXEBOOT


-- 
Felix Zielcke
Proud Debian Maintainer

Re: PXEgrub development on grub2

[Lars Nooden]

Felix Zielcke wrote:

> I don't know what pxegrub can do, but GRUB 2 has PXE support:
> http://grub.enbug.org/PXEBOOT

I think that's what I was looking for.  I'll play with it.  Thanks!

Regards,
-Lars

Re: PXEgrub development on grub2

[Michal Suchanek]

HEllo

2009/9/9 Felix Zielcke <fzielcke@z-51.de>:
> Am Mittwoch, den 09.09.2009, 17:07 +0300 schrieb Lars Nooden:
>> I see that all new work is on Grub2.  What timeline is there for
>> bringing the pxegrub functionality forward to grub2 from legacy grub?
>>
>> Regards
>> -Lars
>>
>
> I don't know what pxegrub can do, but GRUB 2 has PXE support:
> http://grub.enbug.org/PXEBOOT
>

This would suffice for reading config and/or default from tftp so the
boot selection can be changed remotely on systems that have PXE.

Is it possible to do something similar on systems that do not have PXE
(ie systems without PXE BIOS or Apple EFI)?

Thanks

Michal

Re: PXEgrub development on grub2

[Seth Goldberg]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 832 bytes --]



Quoting Michal Suchanek, who wrote the following on Wed, 9 Sep 2009:

> HEllo
>
> 2009/9/9 Felix Zielcke <fzielcke@z-51.de>:
>> Am Mittwoch, den 09.09.2009, 17:07 +0300 schrieb Lars Nooden:
>>> I see that all new work is on Grub2.  What timeline is there for
>>> bringing the pxegrub functionality forward to grub2 from legacy grub?
>>>
>>> Regards
>>> -Lars
>>>
>>
>> I don't know what pxegrub can do, but GRUB 2 has PXE support:
>> http://grub.enbug.org/PXEBOOT
>>
>
> This would suffice for reading config and/or default from tftp so the
> boot selection can be changed remotely on systems that have PXE.
>
> Is it possible to do something similar on systems that do not have PXE
> (ie systems without PXE BIOS or Apple EFI)?

   Have you looked at gPXE? http://etherboot.org/wiki/index.php

  --S

Re: PXEgrub development on grub2

[Michal Suchanek]

2009/9/9 Seth Goldberg <Seth.Goldberg@sun.com>:
>
>
> Quoting Michal Suchanek, who wrote the following on Wed, 9 Sep 2009:
>
>> HEllo
>>
>> 2009/9/9 Felix Zielcke <fzielcke@z-51.de>:
>>>
>>> Am Mittwoch, den 09.09.2009, 17:07 +0300 schrieb Lars Nooden:
>>>>
>>>> I see that all new work is on Grub2.  What timeline is there for
>>>> bringing the pxegrub functionality forward to grub2 from legacy grub?
>>>>
>>>> Regards
>>>> -Lars
>>>>
>>>
>>> I don't know what pxegrub can do, but GRUB 2 has PXE support:
>>> http://grub.enbug.org/PXEBOOT
>>>
>>
>> This would suffice for reading config and/or default from tftp so the
>> boot selection can be changed remotely on systems that have PXE.
>>
>> Is it possible to do something similar on systems that do not have PXE
>> (ie systems without PXE BIOS or Apple EFI)?
>
>  Have you looked at gPXE? http://etherboot.org/wiki/index.php
>

It sounds good. Looks like gPXE should work on EFI.

I will try to build an image.

Thanks

Michal

Re: PXEgrub development on grub2

[Felix Zielcke]

Am Donnerstag, den 10.09.2009, 14:39 +0200 schrieb Michal Suchanek:
> 2009/9/9 Seth Goldberg <Seth.Goldberg@sun.com>:
> >
> >
> > Quoting Michal Suchanek, who wrote the following on Wed, 9 Sep 2009:
> >
> >> HEllo
> >>
> >> 2009/9/9 Felix Zielcke <fzielcke@z-51.de>:
> >>>
> >>> Am Mittwoch, den 09.09.2009, 17:07 +0300 schrieb Lars Nooden:
> >>>>
> >>>> I see that all new work is on Grub2.  What timeline is there for
> >>>> bringing the pxegrub functionality forward to grub2 from legacy grub?
> >>>>
> >>>> Regards
> >>>> -Lars
> >>>>
> >>>
> >>> I don't know what pxegrub can do, but GRUB 2 has PXE support:
> >>> http://grub.enbug.org/PXEBOOT
> >>>
> >>
> >> This would suffice for reading config and/or default from tftp so the
> >> boot selection can be changed remotely on systems that have PXE.
> >>
> >> Is it possible to do something similar on systems that do not have PXE
> >> (ie systems without PXE BIOS or Apple EFI)?
> >
> >  Have you looked at gPXE? http://etherboot.org/wiki/index.php
> >
> 
> It sounds good. Looks like gPXE should work on EFI.
> 
> I will try to build an image.
> 

Though that doestn't change that pxe.mod is currently only build for
i386-pc and nothing else

-- 
Felix Zielcke
Proud Debian Maintainer

Re: PXEgrub development on grub2

[Robert Millan]

On Wed, Sep 09, 2009 at 05:04:35PM +0200, Michal Suchanek wrote:
> HEllo
> 
> 2009/9/9 Felix Zielcke <fzielcke@z-51.de>:
> > Am Mittwoch, den 09.09.2009, 17:07 +0300 schrieb Lars Nooden:
> >> I see that all new work is on Grub2.  What timeline is there for
> >> bringing the pxegrub functionality forward to grub2 from legacy grub?
> >>
> >> Regards
> >> -Lars
> >>
> >
> > I don't know what pxegrub can do, but GRUB 2 has PXE support:
> > http://grub.enbug.org/PXEBOOT
> >
> 
> This would suffice for reading config and/or default from tftp so the
> boot selection can be changed remotely on systems that have PXE.
> 
> Is it possible to do something similar on systems that do not have PXE
> (ie systems without PXE BIOS or Apple EFI)?

We should have network card drivers like GRUB Legacy had.  In GRUB Legacy,
they were imported from Etherboot.  In GRUB 2 we can do the same, as long
as they're GPL-compatible.

-- 
Robert Millan

  The DRM opt-in fallacy: "Your data belongs to us. We will decide when (and
  how) you may access your data; but nobody's threatening your freedom: we
  still allow you to remove your data and not access it at all."

Re: PXEgrub development on grub2

[Seth Goldberg]

>> It sounds good. Looks like gPXE should work on EFI.
>>
>> I will try to build an image.
>>
>
> Though that doestn't change that pxe.mod is currently only build for
> i386-pc and nothing else

  Does anyone have any patches that enable PXE on EFI-firmware-based systems?

  --S

Re: PXEgrub development on grub2

[Robert Millan]

On Thu, Sep 10, 2009 at 02:17:56PM -0700, Seth Goldberg wrote:
>
>>> It sounds good. Looks like gPXE should work on EFI.
>>>
>>> I will try to build an image.
>>>
>>
>> Though that doestn't change that pxe.mod is currently only build for
>> i386-pc and nothing else
>
>  Does anyone have any patches that enable PXE on EFI-firmware-based systems?

What we need is firmware-independant support.  I can live with the PXE-on-BIOS
hack if it's useful, but what we're supposed to have is a driver framework like
the one in GRUB Legacy, not a stub for each firmware type.

Reliing on firmware is problematic, because it's usually proprietary and buggy,
we shouldn't do it if it can be avoided.

-- 
Robert Millan

  The DRM opt-in fallacy: "Your data belongs to us. We will decide when (and
  how) you may access your data; but nobody's threatening your freedom: we
  still allow you to remove your data and not access it at all."

Re: PXEgrub development on grub2

[Seth Goldberg]

Hi,

Quoting Robert Millan, who wrote the following on Fri, 11 Sep 2009:

> On Thu, Sep 10, 2009 at 02:17:56PM -0700, Seth Goldberg wrote:
>>
>>>> It sounds good. Looks like gPXE should work on EFI.
>>>>
>>>> I will try to build an image.
>>>>
>>>
>>> Though that doestn't change that pxe.mod is currently only build for
>>> i386-pc and nothing else
>>
>>  Does anyone have any patches that enable PXE on EFI-firmware-based systems?
>
> What we need is firmware-independant support.  I can live with the PXE-on-BIOS
> hack if it's useful, but what we're supposed to have is a driver framework like
> the one in GRUB Legacy, not a stub for each firmware type.
>
> Reliing on firmware is problematic, because it's usually proprietary and buggy,
> we shouldn't do it if it can be avoided.


   I strongly disagree with you in this specific case.  Our experience in 
Solaris has demonstrated that PXE firmware is surprisingly robust (when the 
right combination of API calls (i.e. those tested by Windows) are used).  We 
have been successfully using PXE-based firmware for netbooting for many years 
now, and we would like to continue to do so.  Maintaining a driver collection 
for NICs is futile, IMHO.  Using the firmware that's there, and that's 
reliable should be the goal.  Not all firmware is our enemy :).

  --S

Re: PXEgrub development on grub2

[Michal Suchanek]

2009/9/11 Seth Goldberg <Seth.Goldberg@sun.com>:
> Hi,
>
> Quoting Robert Millan, who wrote the following on Fri, 11 Sep 2009:
>
>> On Thu, Sep 10, 2009 at 02:17:56PM -0700, Seth Goldberg wrote:
>>>
>>>>> It sounds good. Looks like gPXE should work on EFI.
>>>>>
>>>>> I will try to build an image.
>>>>>
>>>>
>>>> Though that doestn't change that pxe.mod is currently only build for
>>>> i386-pc and nothing else
>>>
>>>  Does anyone have any patches that enable PXE on EFI-firmware-based
>>> systems?
>>
>> What we need is firmware-independant support.  I can live with the
>> PXE-on-BIOS
>> hack if it's useful, but what we're supposed to have is a driver framework
>> like
>> the one in GRUB Legacy, not a stub for each firmware type.
>>
>> Reliing on firmware is problematic, because it's usually proprietary and
>> buggy,
>> we shouldn't do it if it can be avoided.
>
>
>  I strongly disagree with you in this specific case.  Our experience in
> Solaris has demonstrated that PXE firmware is surprisingly robust (when the
> right combination of API calls (i.e. those tested by Windows) are used).  We
> have been successfully using PXE-based firmware for netbooting for many
> years now, and we would like to continue to do so.  Maintaining a driver
> collection for NICs is futile, IMHO.  Using the firmware that's there, and
> that's reliable should be the goal.  Not all firmware is our enemy :).

Real hardware drivers should provide much better functionality than
PXE (other firmwares may do better, though).

Still using PXE and other firmware as the networking driver allows
driving huge class of devices in the same way (although perhaps not
optimally). Also there will be likely devices that have firmware
drivers but not gPXE/Linux/... drivers.

Thanks

Michal

Re: PXEgrub development on grub2

[Seth Goldberg]

>
> Real hardware drivers should provide much better functionality than
> PXE (other firmwares may do better, though).

   No argument there.  For Solaris, the PXE firmware has proven reliable enough 
for us not to investigate other alternatives.  Sure, there are ranges of 
PXE packet throughput between NICs of the same speed, but a few more 
seconds of load time are no big deal.  For what we need, PXE is sufficient.

  --S

About firmware facilities

[Robert Millan]

On Fri, Sep 11, 2009 at 02:07:10PM -0700, Seth Goldberg wrote:
>
>   I strongly disagree with you in this specific case.  Our experience in  
> Solaris has demonstrated that PXE firmware is surprisingly robust (when 
> the right combination of API calls (i.e. those tested by Windows) are 
> used).  We have been successfully using PXE-based firmware for netbooting 
> for many years now, and we would like to continue to do so.  Maintaining 
> a driver collection for NICs is futile, IMHO.  Using the firmware that's 
> there, and that's reliable should be the goal.  Not all firmware is our 
> enemy :).

Reliing on proprietary firmware is a compromise.  We don't install the blobs
ourselves, so we're not responsible for them, but it is still problematic
because user has less freedom (firmware bugs is just the most notable
consequence of this).

So our compromise is to use firmware when we have no other choice, or when
the alternative is not reasonable (e.g. not mature or complete enough).

My goal as maintainer is to encourage development of a usable and complete
driver framework.  I'm open to discussion about accepting code for using
hardware support from firmware, but keep in mind it's not our primary goal.

In the specific case of network hardware, I'm more reluctant because it's
a regression compared to what we had in GRUB Legacy.

-- 
Robert Millan

  The DRM opt-in fallacy: "Your data belongs to us. We will decide when (and
  how) you may access your data; but nobody's threatening your freedom: we
  still allow you to remove your data and not access it at all."

Re: About firmware facilities

[Seth Goldberg]

Quoting Robert Millan, who wrote the following on Sat, 12 Sep 2009:

> On Fri, Sep 11, 2009 at 02:07:10PM -0700, Seth Goldberg wrote:
>>
>>   I strongly disagree with you in this specific case.  Our experience in
>> Solaris has demonstrated that PXE firmware is surprisingly robust (when
>> the right combination of API calls (i.e. those tested by Windows) are
>> used).  We have been successfully using PXE-based firmware for netbooting
>> for many years now, and we would like to continue to do so.  Maintaining
>> a driver collection for NICs is futile, IMHO.  Using the firmware that's
>> there, and that's reliable should be the goal.  Not all firmware is our
>> enemy :).
>
> Reliing on proprietary firmware is a compromise.  We don't install the blobs
> ourselves, so we're not responsible for them, but it is still problematic
> because user has less freedom (firmware bugs is just the most notable
> consequence of this).
>
> So our compromise is to use firmware when we have no other choice, or when
> the alternative is not reasonable (e.g. not mature or complete enough).
>
> My goal as maintainer is to encourage development of a usable and complete
> driver framework.  I'm open to discussion about accepting code for using
> hardware support from firmware, but keep in mind it's not our primary goal.
>
> In the specific case of network hardware, I'm more reluctant because it's
> a regression compared to what we had in GRUB Legacy.

   I agree that choice is very important.  In this case, our choice is to rely 
on PXE firmware, since we've had excellent experiences with it.  We added an 
UNDI network driver to legacy GRUB, so from our perspective, not having PXE in 
GRUB2 is the regression :).

  Thanks,
  --S

Re: About firmware facilities

[Robert Millan]

On Sun, Sep 13, 2009 at 01:54:33PM -0700, Seth Goldberg wrote:
>
>
> Quoting Robert Millan, who wrote the following on Sat, 12 Sep 2009:
>
>> On Fri, Sep 11, 2009 at 02:07:10PM -0700, Seth Goldberg wrote:
>>>
>>>   I strongly disagree with you in this specific case.  Our experience in
>>> Solaris has demonstrated that PXE firmware is surprisingly robust (when
>>> the right combination of API calls (i.e. those tested by Windows) are
>>> used).  We have been successfully using PXE-based firmware for netbooting
>>> for many years now, and we would like to continue to do so.  Maintaining
>>> a driver collection for NICs is futile, IMHO.  Using the firmware that's
>>> there, and that's reliable should be the goal.  Not all firmware is our
>>> enemy :).
>>
>> Reliing on proprietary firmware is a compromise.  We don't install the blobs
>> ourselves, so we're not responsible for them, but it is still problematic
>> because user has less freedom (firmware bugs is just the most notable
>> consequence of this).
>>
>> So our compromise is to use firmware when we have no other choice, or when
>> the alternative is not reasonable (e.g. not mature or complete enough).
>>
>> My goal as maintainer is to encourage development of a usable and complete
>> driver framework.  I'm open to discussion about accepting code for using
>> hardware support from firmware, but keep in mind it's not our primary goal.
>>
>> In the specific case of network hardware, I'm more reluctant because it's
>> a regression compared to what we had in GRUB Legacy.
>
>   I agree that choice is very important.  In this case, our choice is to 
> rely on PXE firmware, since we've had excellent experiences with it.  We 
> added an UNDI network driver to legacy GRUB, so from our perspective, not 
> having PXE in GRUB2 is the regression :).

Well, you have the freedom to disagree with anything we do and bring your
customized GRUB to a different direction :-)

Anyhow, my priority for GRUB is strong driver-based support.  We could recruit
someone to develop the framework in next year's GSoC (unless somebody steps
in, of course).

-- 
Robert Millan

  The DRM opt-in fallacy: "Your data belongs to us. We will decide when (and
  how) you may access your data; but nobody's threatening your freedom: we
  still allow you to remove your data and not access it at all."

Re: About firmware facilities

[Brendan Trotter]

Hi,

On Tue, Sep 15, 2009 at 1:02 AM, Robert Millan <rmh@aybabtu.com> wrote:
> Well, you have the freedom to disagree with anything we do and bring your
> customized GRUB to a different direction :-)
>
> Anyhow, my priority for GRUB is strong driver-based support.  We could recruit
> someone to develop the framework in next year's GSoC (unless somebody steps
> in, of course).

Why stop there?

If proprietory ethernet ROMs aren't good enough, then what about
proprietory SCSI ROMs, and proprietory firmware/BIOS?

Surely a boot manager wouldn't be a good manager if it didn't include
it's own replacements for all of these things; and perhaps it should
also include it's own replacement for proprietory OS's too...

Why are you worrying about such silly things when the multi-boot
specification (which actually is relevant) is still severely borked?

>  The DRM opt-in fallacy: "Your data belongs to us. We will decide when (and
>  how) you may access your data; but nobody's threatening your freedom: we
>  still allow you to remove your data and not access it at all."

Sigh. I think I understand now - lack of logical thinking leads to
lack of rational behavior.

"Our data belongs to us. We will decide when (and how) other people
may access our data (potentially including allowing them to lease it
under terms and conditions that include DRM, if they agree to these
terms and conditions)"


Cheers,

Brendan

Re: About firmware facilities

[Pavel Roskin]

On Tue, 2009-09-15 at 04:27 +0930, Brendan Trotter wrote:
> Hi,
> 
> On Tue, Sep 15, 2009 at 1:02 AM, Robert Millan <rmh@aybabtu.com> wrote:
> > Well, you have the freedom to disagree with anything we do and bring your
> > customized GRUB to a different direction :-)
> >
> > Anyhow, my priority for GRUB is strong driver-based support.  We could recruit
> > someone to develop the framework in next year's GSoC (unless somebody steps
> > in, of course).
> 
> Why stop there?
> 
> If proprietory ethernet ROMs aren't good enough, then what about
> proprietory SCSI ROMs, and proprietory firmware/BIOS?

We are already doing it.  There is functional ATA support, USB support
is under development.  GRUB can serve as BIOS together with Coreboot.

> Surely a boot manager wouldn't be a good manager if it didn't include
> it's own replacements for all of these things; and perhaps it should
> also include it's own replacement for proprietory OS's too...
> 
> Why are you worrying about such silly things when the multi-boot
> specification (which actually is relevant) is still severely borked?

Patches are welcome.

> >  The DRM opt-in fallacy: "Your data belongs to us. We will decide when (and
> >  how) you may access your data; but nobody's threatening your freedom: we
> >  still allow you to remove your data and not access it at all."
> 
> Sigh. I think I understand now - lack of logical thinking leads to
> lack of rational behavior.

Ad hominem arguments are not welcome here.

-- 
Regards,
Pavel Roskin

Re: About firmware facilities

[Brendan Trotter]

Hi,

On Tue, Sep 15, 2009 at 4:41 AM, Pavel Roskin <proski@gnu.org> wrote:
> On Tue, 2009-09-15 at 04:27 +0930, Brendan Trotter wrote:
>> Hi,
>>
>> On Tue, Sep 15, 2009 at 1:02 AM, Robert Millan <rmh@aybabtu.com> wrote:
>> > Well, you have the freedom to disagree with anything we do and bring your
>> > customized GRUB to a different direction :-)
>> >
>> > Anyhow, my priority for GRUB is strong driver-based support.  We could recruit
>> > someone to develop the framework in next year's GSoC (unless somebody steps
>> > in, of course).
>>
>> Why stop there?
>>
>> If proprietory ethernet ROMs aren't good enough, then what about
>> proprietory SCSI ROMs, and proprietory firmware/BIOS?
>
> We are already doing it.  There is functional ATA support, USB support
> is under development.

But, are you doing it for valid technical reasons?

> GRUB can serve as BIOS together with Coreboot.

I know. It'll break my code.

The multi-boot specification says "However, other machine state should
be left by the boot loader in normal working order, i.e. as
initialized by the bios (or DOS, if that's what the boot loader runs
from)."; although I seem to remember it saying words to the effect of
"the firmware should be left in a usable state".

Due to limitations in the original multi-boot specification my code
switches back to real mode and uses the BIOS to do memory detection,
do video mode detection, switch video modes and gather other
information. If GRUB is running on coreboot or UEFI I can't do this
(and can't work around the limitations any other way). Unless the
multi-boot specification is changed significantly (such that a
multi-boot compliant code needs no work-arounds, or at least so that
multi-boot compliant code can determine what sort of firmware there
was and use the firmware) then GRUB as a whole becomes useless to me.

The current draft (http://grub.enbug.org/MultibootDraft) is a small
step in the right direction; but it has changed much in 3 years.

>> Why are you worrying about such silly things when the multi-boot
>> specification (which actually is relevant) is still severely borked?
>
> Patches are welcome.

I'm not sure it's possible to write patches for a multi-boot compliant
boot loader without a specification that defines "multi-boot
compliant".


Cheers,

Brendan

Re: About firmware facilities

[Michal Suchanek]

2009/9/14 Brendan Trotter <btrotter@gmail.com>:
> Hi,
>
> On Tue, Sep 15, 2009 at 4:41 AM, Pavel Roskin <proski@gnu.org> wrote:
>> On Tue, 2009-09-15 at 04:27 +0930, Brendan Trotter wrote:
>>> Hi,
>>>
>>> On Tue, Sep 15, 2009 at 1:02 AM, Robert Millan <rmh@aybabtu.com> wrote:
>>> > Well, you have the freedom to disagree with anything we do and bring your
>>> > customized GRUB to a different direction :-)
>>> >
>>> > Anyhow, my priority for GRUB is strong driver-based support.  We could recruit
>>> > someone to develop the framework in next year's GSoC (unless somebody steps
>>> > in, of course).
>>>
>>> Why stop there?
>>>
>>> If proprietory ethernet ROMs aren't good enough, then what about
>>> proprietory SCSI ROMs, and proprietory firmware/BIOS?
>>
>> We are already doing it.  There is functional ATA support, USB support
>> is under development.
>
> But, are you doing it for valid technical reasons?
>

Many BIOSes are undeniably broken without any upgrade path provided by
the original hardware manufacturer.

On the hand, many more BIOSes are usable.

If you think of the BIOS as a part of the hardware then the question
is if there is value in supporting broken hardware. There are also
numerous bugs in the silicon of many (probably most) PCs which are
worked around in drivers or by not using a particular hardware
feature.

It may be more practical to throw away a piece of deficient or
sub-spec hardware if it is easily replaced but many people are in a
situation when they have dozens of systems with a particular defect.

Using native ATA drivers in a boot loader might be good way around
problems with BIOS drive size limitations and the like or it may just
bring different problems. We will know only when grub2 is widely
deployed on a wide variety of systems.

However, it is necessary to keep in mind that the firmware might be
able to initialize and use obscure devices for which grub does not
have drivers and on which more traditional boot loaders would work. Be
it disks, networking devices or anything else.

Thanks

Michal

Re: About firmware facilities

[Vladimir 'phcoder' Serbinenko]

On Mon, Sep 14, 2009 at 10:12 PM, Brendan Trotter <btrotter@gmail.com> wrote:
> Hi,
>
> On Tue, Sep 15, 2009 at 4:41 AM, Pavel Roskin <proski@gnu.org> wrote:
>> On Tue, 2009-09-15 at 04:27 +0930, Brendan Trotter wrote:
>>> Hi,
>>>
>>> On Tue, Sep 15, 2009 at 1:02 AM, Robert Millan <rmh@aybabtu.com> wrote:
>>> > Well, you have the freedom to disagree with anything we do and bring your
>>> > customized GRUB to a different direction :-)
>>> >
>>> > Anyhow, my priority for GRUB is strong driver-based support.  We could recruit
>>> > someone to develop the framework in next year's GSoC (unless somebody steps
>>> > in, of course).
>>>
>>> Why stop there?
>>>
>>> If proprietory ethernet ROMs aren't good enough, then what about
>>> proprietory SCSI ROMs, and proprietory firmware/BIOS?
>>
>> We are already doing it.  There is functional ATA support, USB support
>> is under development.
>
> But, are you doing it for valid technical reasons?
>
Yes. I have a borked firmware right on this laptop.
>> GRUB can serve as BIOS together with Coreboot.
>
> I know. It'll break my code.
>
> The multi-boot specification says "However, other machine state should
> be left by the boot loader in normal working order, i.e. as
> initialized by the bios (or DOS, if that's what the boot loader runs
> from)."; although I seem to remember it saying words to the effect of
> "the firmware should be left in a usable state".
>
Firmware if present is left in usable state. However it may simply not
be present.
> Due to limitations in the original multi-boot specification my code
> switches back to real mode and uses the BIOS to do memory detection,
> do video mode detection, switch video modes and gather other
> information.

Have you actually read the multiboot specification? Booter passes info
about memory and video mode in mbi (video for multiboot isn't
implemented yet). If you need firmware for basic bootup you're clearly
doing something wrong and are firmware-dependent. Of course it's your
freedom to make suboptimal software.
> If GRUB is running on coreboot or UEFI I can't do this
> (and can't work around the limitations any other way). Unless the
> multi-boot specification is changed significantly (such that a
> multi-boot compliant code needs no work-arounds, or at least so that
> multi-boot compliant code can determine what sort of firmware there
> was and use the firmware) then GRUB as a whole becomes useless to me.
Just say in clean text what changes you need so we can discuss it.
Don't expect us to understand the statement "your standard is borked,
change it".
>
> The current draft (http://grub.enbug.org/MultibootDraft) is a small
> step in the right direction; but it has changed much in 3 years.
>
It mainly introduces multi-CPU and another MBI format compared to multiboot1
>>> Why are you worrying about such silly things when the multi-boot
>>> specification (which actually is relevant) is still severely borked?
>>
>> Patches are welcome.
>
> I'm not sure it's possible to write patches for a multi-boot compliant
> boot loader without a specification that defines "multi-boot
> compliant".
>
Read it here: http://www.gnu.org/software/grub/manual/multiboot/multiboot.html
>
> Cheers,
>
> Brendan
>
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> http://lists.gnu.org/mailman/listinfo/grub-devel
>



-- 
Regards
Vladimir 'phcoder' Serbinenko

Personal git repository: http://repo.or.cz/w/grub2/phcoder.git

Re: About firmware facilities

[Brendan Trotter]

Hi,

On Tue, Sep 15, 2009 at 6:19 AM, Vladimir 'phcoder' Serbinenko
<phcoder@gmail.com> wrote:
> On Mon, Sep 14, 2009 at 10:12 PM, Brendan Trotter <btrotter@gmail.com> wrote:
>> On Tue, Sep 15, 2009 at 4:41 AM, Pavel Roskin <proski@gnu.org> wrote:
>>> On Tue, 2009-09-15 at 04:27 +0930, Brendan Trotter wrote:
>>> We are already doing it.  There is functional ATA support, USB support
>>> is under development.
>>
>> But, are you doing it for valid technical reasons?
>>
> Yes. I have a borked firmware right on this laptop.

Sounds like you need better firmware - Coreboot might be a good
alternative. I assume GRUB is meant to be a boot loader though (rather
than a project to rewrite every piece of dodgy software ever written).

>>> GRUB can serve as BIOS together with Coreboot.
>>
>> I know. It'll break my code.
>>
>> The multi-boot specification says "However, other machine state should
>> be left by the boot loader in normal working order, i.e. as
>> initialized by the bios (or DOS, if that's what the boot loader runs
>> from)."; although I seem to remember it saying words to the effect of
>> "the firmware should be left in a usable state".
>>
> Firmware if present is left in usable state. However it may simply not
> be present.

So if the firmware is present, GRUB won't alter the state that the
firmware left the hardware in (or if it does it'll restore all
hardware to the default firmware state before starting the OS);
including all devices that GRUB uses via. it's own device drivers
(except for specific things mentioned in the multi-boot specification,
like the A20 gate)?

>> Due to limitations in the original multi-boot specification my code
>> switches back to real mode and uses the BIOS to do memory detection,
>> do video mode detection, switch video modes and gather other
>> information.
>
> Have you actually read the multiboot specification? Booter passes info
> about memory and video mode in mbi (video for multiboot isn't
> implemented yet). If you need firmware for basic bootup you're clearly
> doing something wrong and are firmware-dependent. Of course it's your
> freedom to make suboptimal software.

I've read the multi-boot specification. I've also read the code in
GRUB-legacy that does memory detection, and I'm unwilling to allow my
code to rely on it for "quality control" reasons. Without going into
details, GRUB-legacy tends to do a "minimal" job and then expects the
user to fix the problem if/when it goes wrong (but even then it only
offers a "uppermem" command without providing a way for the user to
specify a complete system memory map).

For video, I believe there's patches to make GRUB-legacy comply with
the original multi-boot specification (but for dual boot setups I
can't expect people to replace their main OS's existing GRUB with a
patched version). Even if all versions of GRUB did support the video
stuff in multi-boot, the multi-boot specification is still
ridiculously inadequate.

My code gives the user a choice of any video modes that are supported
by the video card and my code (and doesn't allow them to choose video
modes that aren't supported by my code), including double scanned
modes (if supported by the video card) and a variety of refresh rates
(if supported by the video card). It also uses the monitor's EDID
information to try to prevent video modes that are supported by the
video card but not supported by the monitor from being used. This is
mostly automated, so that (for e.g.) the user can replace the monitor
with another monitor that doesn't support the video mode that was
previously used, and the software will find the best video mode that
the new monitor does support without the user changing anything.

>> If GRUB is running on coreboot or UEFI I can't do this
>> (and can't work around the limitations any other way). Unless the
>> multi-boot specification is changed significantly (such that a
>> multi-boot compliant code needs no work-arounds, or at least so that
>> multi-boot compliant code can determine what sort of firmware there
>> was and use the firmware) then GRUB as a whole becomes useless to me.

> Just say in clean text what changes you need so we can discuss it.
> Don't expect us to understand the statement "your standard is borked,
> change it".

Fair enough...

The video support is already mentioned above.

It'd be nice if other output methods were exposed to the OS too. For
example, if GRUB is configured to use a serial terminal it'd be good
if it told the OS details for the serial port, so that the OS can
continue using the same serial terminal without resetting/restarting
an established connection or requiring the user to configure it in 2
separate places (in GRUB and in the OS).

The same would apply to internationalization - GRUB should tell the OS
which language/locale it used, so that the OS can automatically use
the same language as a default (especially for OS installation CDs).

For memory detection, ACPI 3.0 allows the BIOS (" INT 15H, E820H") to
return extended attributes - mostly only a volatile/non-volatile flag.
This isn't in GRUB's information. ACPI 3.0 also allows the BIOS to
return areas of the type "AddressRangeUnusable" (e.g. faulty RAM). If
the BIOS reports that the area from 0x00100000 to 0x001FFFFF is
faulty, then how is GRUB planning to handle that? My own code (if
using it's own boot loaders rather than GRUB) can tolerate RAM
failures anywhere in RAM except for about 64 KiB in low memory.

While I'm on the subject, I also want a list of RAM areas that the
boot loader has relied on. That way if a 24/7 server detects that any
RAM that the boot loader relies on has become faulty it can warn the
user that the computer may not be able to boot. Currently I assume
that all multi-boot compliant boot loaders (including GRUB) rely on
all RAM below 0x00100000, but it's a dodgy hack I'd rather avoid.

In the memory map, GRUB should mark areas that it has used to pass
information to the OS (e.g. the multi-boot information structure) as
"multi-boot reclaimable" (in a similar way that ACPI uses the "ACPI
reclaimable" type). This would make it easier for the OS to avoid
overwriting this data before it attempts to read it. The area types
need to be "architecture independent" types too (e.g. GRUB converts
ACPI area types, UEFI area types, etc into "standard multi-boot area
types").

The "Boot device" field in the multi-boot information structure should
be improved to handle a variety of conditions; including if the disk
was an emulated disk (e.g. "El Torito" emulating a hard drive). The
BIOS drive number isn't much use (especially if the firmware is
coreboot, UEFI, OpenFirmware, etc), and should be replaced with
something that identifies the corresponding drive structure (this
includes USB). If the boot device was ethernet then the "Boot device"
field should contain a reference to a "network card information
structure" that includes the ethernet card's MAC address (and any IP
addresses, etc from DHCP, if applicable). If the boot device was
firmware (e.g. GRUB and OS embedded in the firmware's flash) then the
"Boot device" field should indicate this. Under no circumstances
should "Boot device" field be invalid (bit 1 in the `flags' word
should always be set).

The drive structures. Here, the "BIOS device number" should be a
"firmware specific identifier". The OS must be able to uniquely
identify each device without using the "firmware specific identifier".
For an example, something like the information returned by "INT 0x13,
ah = 0x48" (see http://www.ctyme.com/intr/rb-0715.htm) would be a good
place to start (e.g. "host bus, interface type, interface path, device
path"). The boot loader (GRUB) should list as many storage devices as
possible (but the boot loader shouldn't be required to guarantee that
all storage devices are listed, or that complete information is
present for all devices that are listed). Note: "storage devices"
means hard drives, floppy drives, CD-ROMs (with and without CDs
inserted), Zip drives, USB flash, USB floppy/CD, tape backup units,
etc. Note: I don't care how obsolete floppy, zip and/or tapes are, the
specification should cover it (even if boot loaders don't).

The "boot loader name" field is nice, but it needs a "boot loader
version" field to go with it. A "firmware type" field is also needed.

The OS image also needs a different magic number to indicate that the
OS image is designed for future versions of the multi-boot
specification (rather than the old/current version). If the OS image
uses the new magic number, then the OS image must also include an
"version of the multi-boot specification that this image complies
with" field. If the OS image indicates that it's intended for a newer
version of the multi-boot specification than the boot loader complies
with, then the boot loader refuses to boot and displays a "this boot
loader needs to be upgraded" error. If the OS image has the old magic
number, and if the firmware is "PC BIOS" then the boot loader should
boot the old OS image. If the OS image has the old magic number, and
if the firmware is not "PC BIOS" then the boot loader refuses to boot
and displays a "this OS requires a PC BIOS" error message.

In a similar way, if the OS image has the old magic number and the
boot loader is not running on 80x86 then the boot loader refuses to
boot and displays an error message. If the OS image has the new magic
number then it must also include a field that indicates which
architecture the OS is intended for, and the boot loader must check
this field and display a "this OS is intended for a different
architecture" error message (and refuse to boot) if it's wrong.

>> The current draft (http://grub.enbug.org/MultibootDraft) is a small
>> step in the right direction; but it has changed much in 3 years.
>>
> It mainly introduces multi-CPU and another MBI format compared to multiboot1

By "multi-CPU" you mean different architectures, rather than
SMT/SMP/NUMA support?

>>>> Why are you worrying about such silly things when the multi-boot
>>>> specification (which actually is relevant) is still severely borked?
>>>
>>> Patches are welcome.
>>
>> I'm not sure it's possible to write patches for a multi-boot compliant
>> boot loader without a specification that defines "multi-boot
>> compliant".
>>
> Read it here: http://www.gnu.org/software/grub/manual/multiboot/multiboot.html

That's the old version of the multi-boot specification (for
GRUB_legacy). I'm looking forward to a new version of the multi-boot
specification (for GRUB 2) that is designed to support different
architectures, different types of firmware on the same architecture,
etc; and hopefully other improvements.


Cheers,

Brendan

Re: About firmware facilities

[Colin Watson]

I don't have relevant experience regarding multiboot in general, but ...

On Tue, Sep 15, 2009 at 08:53:12AM +0930, Brendan Trotter wrote:
> The same would apply to internationalization - GRUB should tell the OS
> which language/locale it used, so that the OS can automatically use
> the same language as a default (especially for OS installation CDs).

Speaking as an OS installer developer, I'd actively prefer GRUB not to
try to do this via specialised methods like multiboot. I don't want to
have to maintain language lists and what-have-you in yet another place,
cope with all the usual slight variations (e.g. things like pt_BR, or
God-help-us something like the hopelessly inadequate language list in
the USB keyboard specification), and cope with the fact that there are
still more boot loaders out there than just GRUB and so some other
mechanism needs to be maintained anyway.

Any OS installer that can take advantage of this kind of thing almost
certainly already has facilities to simply pass the appropriate locale
code as a boot parameter. GRUB should just use that. That means it's
accessible to the scripting interface, and so any necessary fixups will
be much easier to understand and maintain.

-- 
Colin Watson                                       [cjwatson@ubuntu.com]

Re: About firmware facilities

[Brendan Trotter]

Hi,

On Tue, Sep 15, 2009 at 9:13 AM, Colin Watson <cjwatson@ubuntu.com> wrote:
> I don't have relevant experience regarding multiboot in general, but ...
>
> On Tue, Sep 15, 2009 at 08:53:12AM +0930, Brendan Trotter wrote:
>> The same would apply to internationalization - GRUB should tell the OS
>> which language/locale it used, so that the OS can automatically use
>> the same language as a default (especially for OS installation CDs).
>
> Speaking as an OS installer developer, I'd actively prefer GRUB not to
> try to do this via specialised methods like multiboot. I don't want to
> have to maintain language lists and what-have-you in yet another place,
> cope with all the usual slight variations (e.g. things like pt_BR, or
> God-help-us something like the hopelessly inadequate language list in
> the USB keyboard specification), and cope with the fact that there are
> still more boot loaders out there than just GRUB and so some other
> mechanism needs to be maintained anyway.

An OS could choose to use the information provided by GRUB, or choose
to ignore the information provided by GRUB.

Assuming GRUB implements something that isn't hopelessly inadequate,
I'd prefer to attempt to use the information provided by GRUB (and
then have a fall-back). However, I may be a little unusual, in that
I'm willing to do a lot of extra work to avoid a little unnecessary
end-user configuration.

With this in mind, as an OS installer developer, what form of language
identifier would you consider the 'least hopelessly inadequate"? What
if GRUB used the exact same language identifiers that (for e.g.)
Ubuntu already uses?



Cheers,

Brendan

Re: About firmware facilities

[Colin Watson]

On Tue, Sep 15, 2009 at 09:26:55AM +0930, Brendan Trotter wrote:
> An OS could choose to use the information provided by GRUB, or choose
> to ignore the information provided by GRUB.

Sure, but the less code that needs to go in the boot loader the better.

> Assuming GRUB implements something that isn't hopelessly inadequate,
> I'd prefer to attempt to use the information provided by GRUB (and
> then have a fall-back). However, I may be a little unusual, in that
> I'm willing to do a lot of extra work to avoid a little unnecessary
> end-user configuration.

Don't get me wrong, I wasn't suggesting end-user configuration - I'd
rather have it in grub-mkconfig than in the core, that's all.

> With this in mind, as an OS installer developer, what form of language
> identifier would you consider the 'least hopelessly inadequate"? What
> if GRUB used the exact same language identifiers that (for e.g.)
> Ubuntu already uses?

You mean glibc locale identifiers with any territory and character set
information stripped off, and the variant stripped off unless it's
Portuguese or Chinese since those are the only currently significant
cases where language variants have significant enough differences to be
worth considering as essentially separate languages?

Oh, and locale configuration tends to go together with keyboard
configuration to some extent (at least for the purpose of setting
defaults); since text entry is possible in GRUB it makes some sense to
be able to configure the keymap, and you want to be able to pass that on
to the operating system as well. Language-to-keymap mapping is a
non-trivial problem often involving dispute resolution among local users
to figure out the most appropriate default, and, for us, the keymaps
themselves are maintained as part of the xkeyboard-config project and
change quite frequently.

Beyond the provision of some generic core facilities such as setting a
keymap, this honestly doesn't seem suitable for the boot loader core,
and nor does it seem likely to be particularly portable among anything
other than quite closely related operating systems. Of course it does
pretty much exactly what we need, but it's far too ad-hoc to live
outside a scripting language, IMO.

If GRUB made up its own set of identifiers, then somebody would have to
maintain the list, and my expectation would be that churn would be
substantial and frequent. If it used some kind of generic superset, I'd
expect that we'd end up ignoring it and doing our own thing anyway.

I've just seen this done wrongly far too many times.

Cheers,

-- 
Colin Watson                                       [cjwatson@ubuntu.com]

Re: About firmware facilities

[Brendan Trotter]

Hi,

On Tue, Sep 15, 2009 at 9:58 AM, Colin Watson <cjwatson@ubuntu.com> wrote:
> On Tue, Sep 15, 2009 at 09:26:55AM +0930, Brendan Trotter wrote:
>> An OS could choose to use the information provided by GRUB, or choose
>> to ignore the information provided by GRUB.
>
> Sure, but the less code that needs to go in the boot loader the better.
>
>> Assuming GRUB implements something that isn't hopelessly inadequate,
>> I'd prefer to attempt to use the information provided by GRUB (and
>> then have a fall-back). However, I may be a little unusual, in that
>> I'm willing to do a lot of extra work to avoid a little unnecessary
>> end-user configuration.
>
> Don't get me wrong, I wasn't suggesting end-user configuration - I'd
> rather have it in grub-mkconfig than in the core, that's all.
>
>> With this in mind, as an OS installer developer, what form of language
>> identifier would you consider the 'least hopelessly inadequate"? What
>> if GRUB used the exact same language identifiers that (for e.g.)
>> Ubuntu already uses?
>
> You mean glibc locale identifiers with any territory and character set
> information stripped off, and the variant stripped off unless it's
> Portuguese or Chinese since those are the only currently significant
> cases where language variants have significant enough differences to be
> worth considering as essentially separate languages?
>
> Oh, and locale configuration tends to go together with keyboard
> configuration to some extent (at least for the purpose of setting
> defaults); since text entry is possible in GRUB it makes some sense to
> be able to configure the keymap, and you want to be able to pass that on
> to the operating system as well. Language-to-keymap mapping is a
> non-trivial problem often involving dispute resolution among local users
> to figure out the most appropriate default, and, for us, the keymaps
> themselves are maintained as part of the xkeyboard-config project and
> change quite frequently.
>
> Beyond the provision of some generic core facilities such as setting a
> keymap, this honestly doesn't seem suitable for the boot loader core,
> and nor does it seem likely to be particularly portable among anything
> other than quite closely related operating systems. Of course it does
> pretty much exactly what we need, but it's far too ad-hoc to live
> outside a scripting language, IMO.
>
> If GRUB made up its own set of identifiers, then somebody would have to
> maintain the list, and my expectation would be that churn would be
> substantial and frequent. If it used some kind of generic superset, I'd
> expect that we'd end up ignoring it and doing our own thing anyway.

If GRUB 2 supports internationalization for it's own menu, etc, then
they'll need to do something to track which language and which
keyboard mapping anyway. All I'm suggesting is that (in addition to
all the work needed anyway) the multi-boot information structure
(passed to the OS/kernel) include identifiers that indicate which
language (and keyboard mapping) that GRUB used. It's probably about 6
additional lines in the code that creates the multi-boot information
structure.

I honestly don't care where GRUB gets this information from (whether
it can be modified during boot using GRUB's menu, or if it's set when
GRUB is installed or compiled, or if it's specified by some sort of
script, or whatever). I only care about the header in a multi-boot
compliant OS image and the data passed by multi-boot compliant boot
loaders to the OS (where "multi-boot compliant" includes GRUB, but
doesn't exclude any other implementations of the multi-boot
specification).


Cheers,

Brendan

Re: About firmware facilities

[Vladimir 'phcoder' Serbinenko]

On Tue, Sep 15, 2009 at 1:23 AM, Brendan Trotter <btrotter@gmail.com> wrote:
> Hi,
>
> On Tue, Sep 15, 2009 at 6:19 AM, Vladimir 'phcoder' Serbinenko
> <phcoder@gmail.com> wrote:
>> On Mon, Sep 14, 2009 at 10:12 PM, Brendan Trotter <btrotter@gmail.com> wrote:
>>> On Tue, Sep 15, 2009 at 4:41 AM, Pavel Roskin <proski@gnu.org> wrote:
>>>> On Tue, 2009-09-15 at 04:27 +0930, Brendan Trotter wrote:
>>>> We are already doing it.  There is functional ATA support, USB support
>>>> is under development.
>>>
>>> But, are you doing it for valid technical reasons?
>>>
>> Yes. I have a borked firmware right on this laptop.
>
> Sounds like you need better firmware - Coreboot might be a good
> alternative. I assume GRUB is meant to be a boot loader though (rather
> than a project to rewrite every piece of dodgy software ever written).
>
Coreboot is a minimal firmware. It has no drivers
>>>> GRUB can serve as BIOS together with Coreboot.
>>>
>>> I know. It'll break my code.
>>>
>>> The multi-boot specification says "However, other machine state should
>>> be left by the boot loader in normal working order, i.e. as
>>> initialized by the bios (or DOS, if that's what the boot loader runs
>>> from)."; although I seem to remember it saying words to the effect of
>>> "the firmware should be left in a usable state".
>>>
>> Firmware if present is left in usable state. However it may simply not
>> be present.
>
> So if the firmware is present, GRUB won't alter the state that the
> firmware left the hardware in (or if it does it'll restore all
> hardware to the default firmware state before starting the OS);
> including all devices that GRUB uses via. it's own device drivers
> (except for specific things mentioned in the multi-boot specification,
> like the A20 gate)?
>
No. Usuable means only that firmware isn't destroyed. Any device may
be in a different state
>>> Due to limitations in the original multi-boot specification my code
>>> switches back to real mode and uses the BIOS to do memory detection,
>>> do video mode detection, switch video modes and gather other
>>> information.
>>
>> Have you actually read the multiboot specification? Booter passes info
>> about memory and video mode in mbi (video for multiboot isn't
>> implemented yet). If you need firmware for basic bootup you're clearly
>> doing something wrong and are firmware-dependent. Of course it's your
>> freedom to make suboptimal software.
>
> I've read the multi-boot specification. I've also read the code in
> GRUB-legacy that does memory detection, and I'm unwilling to allow my
> code to rely on it for "quality control" reasons. Without going into
> details, GRUB-legacy tends to do a "minimal" job and then expects the
> user to fix the problem if/when it goes wrong (but even then it only
> offers a "uppermem" command without providing a way for the user to
> specify a complete system memory map).
>
What is "minimal job" and "quality control"? We use standard
E820+(optionally)badram command. I've seen no OS do any more than
this.
>
> My code gives the user a choice of any video modes that are supported
> by the video card and my code (and doesn't allow them to choose video
> modes that aren't supported by my code), including double scanned
> modes (if supported by the video card) and a variety of refresh rates
> (if supported by the video card). It also uses the monitor's EDID
> information to try to prevent video modes that are supported by the
> video card but not supported by the monitor from being used. This is
> mostly automated, so that (for e.g.) the user can replace the monitor
> with another monitor that doesn't support the video mode that was
> previously used, and the software will find the best video mode that
> the new monitor does support without the user changing anything.
The mode listed in multiboot header is only suggested one. User or
bootloader may choose any other mode. But I agree that multiboot2
should have a way for better video mode suggesting.

> It'd be nice if other output methods were exposed to the OS too. For
> example, if GRUB is configured to use a serial terminal it'd be good
> if it told the OS details for the serial port, so that the OS can
> continue using the same serial terminal without resetting/restarting
> an established connection or requiring the user to configure it in 2
> separate places (in GRUB and in the OS).
I'll think about it
>
> The same would apply to internationalization - GRUB should tell the OS
> which language/locale it used, so that the OS can automatically use
> the same language as a default (especially for OS installation CDs).
Just add lang=$lang to multiboot line
>
> For memory detection, ACPI 3.0 allows the BIOS (" INT 15H, E820H") to
> return extended attributes - mostly only a volatile/non-volatile flag.
> This isn't in GRUB's information. ACPI 3.0 also allows the BIOS to
> return areas of the type "AddressRangeUnusable" (e.g. faulty RAM).
This is mostly unnecessary. Basically you need only to know if you can
use a memory range or not. The only useful additional code would be
ReclaimMemory
> If
> the BIOS reports that the area from 0x00100000 to 0x001FFFFF is
> faulty, then how is GRUB planning to handle that? My own code (if
> using it's own boot loaders rather than GRUB) can tolerate RAM
> failures anywhere in RAM except for about 64 KiB in low memory.
GRUB can't do this right now because it doesn't recieve badram info
soon enough. And even if it does most kernels expect first MiB to be
usable.
>
> While I'm on the subject, I also want a list of RAM areas that the
> boot loader has relied on. That way if a 24/7 server detects that any
> RAM that the boot loader relies on has become faulty it can warn the
> user that the computer may not be able to boot. Currently I assume
> that all multi-boot compliant boot loaders (including GRUB) rely on
> all RAM below 0x00100000, but it's a dodgy hack I'd rather avoid.
Such list is a blatant encapsulation breach. If you want such test,
add it to bootloader, not OS.
>
> In the memory map, GRUB should mark areas that it has used to pass
> information to the OS (e.g. the multi-boot information structure) as
> "multi-boot reclaimable" (in a similar way that ACPI uses the "ACPI
> reclaimable" type). This would make it easier for the OS to avoid
> overwriting this data before it attempts to read it.
This information is available with a simple loop over mbi. I would
rathjer avoid overcomplicating the standard because it increases a
chance of having "half-compliant" OSes and "half-compliant" booters.
> The area types
> need to be "architecture independent" types too (e.g. GRUB converts
> ACPI area types, UEFI area types, etc into "standard multi-boot area
> types").
It already is.
>
> The "Boot device" field in the multi-boot information structure should
> be improved to handle a variety of conditions; including if the disk
> was an emulated disk (e.g. "El Torito" emulating a hard drive). The
> BIOS drive number isn't much use (especially if the firmware is
> coreboot, UEFI, OpenFirmware, etc), and should be replaced with
> something that identifies the corresponding drive structure (this
> includes USB).
Boot device shouldn't be used at all. It was a mistake. Booter has no
good way to know how OS will see the device. You should pass this
parameter via commandline either as device name or UUID. You have
scripting to automate this
> If the boot device was ethernet then the "Boot device"
> field should contain a reference to a "network card information
> structure" that includes the ethernet card's MAC address (and any IP
> addresses, etc from DHCP, if applicable).
I already discussed this with Seth Goldberg but no time to write
multiboot ammendment yet.
>

> The "boot loader name" field is nice, but it needs a "boot loader
> version" field to go with it.
it's a part of name. This field is more for displaying anyway and OS
shouldn't do any checks based on it
> A "firmware type" field is also needed.
Can you respond in appropriate thread?
>
> The OS image also needs a different magic number to indicate that the
> OS image is designed for future versions of the multi-boot
> specification (rather than the old/current version). If the OS image
> uses the new magic number, then the OS image must also include an
> "version of the multi-boot specification that this image complies
> with" field. If the OS image indicates that it's intended for a newer
> version of the multi-boot specification than the boot loader complies
> with, then the boot loader refuses to boot and displays a "this boot
> loader needs to be upgraded" error. If the OS image has the old magic
> number, and if the firmware is "PC BIOS" then the boot loader should
> boot the old OS image. If the OS image has the old magic number, and
> if the firmware is not "PC BIOS" then the boot loader refuses to boot
> and displays a "this OS requires a PC BIOS" error message.
Already implemented through feature fields
>
> In a similar way, if the OS image has the old magic number and the
> boot loader is not running on 80x86 then the boot loader refuses to
> boot and displays an error message. If the OS image has the new magic
> number then it must also include a field that indicates which
> architecture the OS is intended for, and the boot loader must check
> this field and display a "this OS is intended for a different
> architecture" error message (and refuse to boot) if it's wrong.
>
multiboot1 and multiboot2 have different magics
>>> The current draft (http://grub.enbug.org/MultibootDraft) is a small
>>> step in the right direction; but it has changed much in 3 years.
>>>
>> It mainly introduces multi-CPU and another MBI format compared to multiboot1
>
> By "multi-CPU" you mean different architectures, rather than
> SMT/SMP/NUMA support?
I mean ARM, PPC, MIPS and so on.
>
>>>>> Why are you worrying about such silly things when the multi-boot
>>>>> specification (which actually is relevant) is still severely borked?
>>>>
>>>> Patches are welcome.
>>>
>>> I'm not sure it's possible to write patches for a multi-boot compliant
>>> boot loader without a specification that defines "multi-boot
>>> compliant".
>>>
>> Read it here: http://www.gnu.org/software/grub/manual/multiboot/multiboot.html
>
> That's the old version of the multi-boot specification (for
> GRUB_legacy). I'm looking forward to a new version of the multi-boot
> specification (for GRUB 2) that is designed to support different
> architectures, different types of firmware on the same architecture,
> etc; and hopefully other improvements.
>
multiboot1 ISN'T depreceated and grub2 supports multiboot1
>
> Cheers,
>
> Brendan
>
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> http://lists.gnu.org/mailman/listinfo/grub-devel
>



-- 
Regards
Vladimir 'phcoder' Serbinenko

Personal git repository: http://repo.or.cz/w/grub2/phcoder.git

Re: About firmware facilities

[Brendan Trotter]

Hi,

On Tue, Sep 15, 2009 at 6:29 PM, Vladimir 'phcoder' Serbinenko
<phcoder@gmail.com> wrote:
> On Tue, Sep 15, 2009 at 1:23 AM, Brendan Trotter <btrotter@gmail.com> wrote:
>> On Tue, Sep 15, 2009 at 6:19 AM, Vladimir 'phcoder' Serbinenko
>> <phcoder@gmail.com> wrote:
>>> On Mon, Sep 14, 2009 at 10:12 PM, Brendan Trotter <btrotter@gmail.com> wrote:
>>>> On Tue, Sep 15, 2009 at 4:41 AM, Pavel Roskin <proski@gnu.org> wrote:
>>>>> On Tue, 2009-09-15 at 04:27 +0930, Brendan Trotter wrote:

>>>>> GRUB can serve as BIOS together with Coreboot.
>>>>
>>>> I know. It'll break my code.
>>>>
>>>> The multi-boot specification says "However, other machine state should
>>>> be left by the boot loader in normal working order, i.e. as
>>>> initialized by the bios (or DOS, if that's what the boot loader runs
>>>> from)."; although I seem to remember it saying words to the effect of
>>>> "the firmware should be left in a usable state".
>>>>
>>> Firmware if present is left in usable state. However it may simply not
>>> be present.
>>
>> So if the firmware is present, GRUB won't alter the state that the
>> firmware left the hardware in (or if it does it'll restore all
>> hardware to the default firmware state before starting the OS);
>> including all devices that GRUB uses via. it's own device drivers
>> (except for specific things mentioned in the multi-boot specification,
>> like the A20 gate)?
>>
> No. Usuable means only that firmware isn't destroyed. Any device may
> be in a different state

Any device (that the firmware assumes is in a certain state) may be
left in a different state (that the firmware no longer knows about)?

For a very simple example, imagine if the BIOS leaves the floppy motor
on, and GRUB's own floppy driver uses the floppy and then turns the
motor off. Then the OS uses the firmware to read from floppy, but the
firmware thinks the floppy motor is still on and attempts to read from
the floppy without turning the floppy motor on.

If GRUB has it's own device drivers, and GRUB doesn't restore devices
to the state that the firmware expects the devices to be in, then the
firmware is unusable.

If an OS can't use the firmware, then the OS must rely on GRUB for
everything instead, including strange "OS specific" things that nobody
has seen any other OS do before.

>>>> Due to limitations in the original multi-boot specification my code
>>>> switches back to real mode and uses the BIOS to do memory detection,
>>>> do video mode detection, switch video modes and gather other
>>>> information.
>>>
>>> Have you actually read the multiboot specification? Booter passes info
>>> about memory and video mode in mbi (video for multiboot isn't
>>> implemented yet). If you need firmware for basic bootup you're clearly
>>> doing something wrong and are firmware-dependent. Of course it's your
>>> freedom to make suboptimal software.
>>
>> I've read the multi-boot specification. I've also read the code in
>> GRUB-legacy that does memory detection, and I'm unwilling to allow my
>> code to rely on it for "quality control" reasons. Without going into
>> details, GRUB-legacy tends to do a "minimal" job and then expects the
>> user to fix the problem if/when it goes wrong (but even then it only
>> offers a "uppermem" command without providing a way for the user to
>> specify a complete system memory map).
>>
> What is "minimal job" and "quality control"? We use standard
> E820+(optionally)badram command. I've seen no OS do any more than
> this.

My code tries "int 0x15, eax=0xE820" expecting 24 bytes per area (ACPI
3.0); then it tries "int 0x15, eax=0xE820" expecting 20 bytes per
area. If "int 0x15, eax=0xE820" isn't supported by the BIOS then you
can assume it's an old computer (and old computers are painful).

It tries "int 0x15, ax=0xE801", then "int 0x15, ah=0xC7", then "int
0x15, ah=0x8A", then "int 0x15, ah=0xDA88", then "int 0x15, ah=0x88",
then CMOS locations 0x70 and 0x71. If all of this fails (which does
happen on some computers) then it does manual probing.

Some of the old BIOS functions have limited range - for example
they'll return "number of KiB blocks at 0x00100000" as a 16-bit
integer, and can't return more than 64 MiB. In this case my code won't
know if the value returned by the BIOS has been limited to 0xFFFF, so
it'll do manual probing to detect any more RAM above the reported
area. Some computers have an "ISA hole" from 0x00F00000 to 0x00FFFFFF.
Because of this all older BIOS functions that report "amount of RAM at
0x00100000" may return 14 MiB (from 0x00100000 to 0x00EFFFFF) when
there's actually another area of RAM at 0x01000000. In this case my
code won't know if the value returned by the BIOS is right or not, and
it'll do manual probing to detect any more RAM at 0x00100000. If my
code has to do manual probing, then it assumes there's an "ISA hole"
from 0x00F00000 to 0x00FFFFFF (regardless of whether there is or not)
as this hole was used for memory mapped video cards (which would seem
like RAM).

For all BIOS functions used my code avoids all known BIOS bugs (and
there's plenty of them). This includes "sanitizing" the data returned
from "int 0x15, eax=0xE820" - sorting the list and handling any
overlapping areas.

I've never needed to provide a way for the end-user to override my
memory detection.

From what I remember, Linux uses about 3 of these BIOS functions
(probably the same 3 BIOS function GRUB uses) and does work around
most of the BIOS bugs. Linux does (need to) provide a way for the
end-user to override its memory detection (and Linux does provide
adequate facilities for this).

I checked the code for GRUB 1.96 and it uses 3 BIOS functions ("int
0x15, eax=0xE820", "int 0x15, ax=0xE801" and "int 0x15, ah=0x88") and
doesn't work around any BIOS bugs at all. The memory detection code in
Linux is a better (but the memory detection code in Linux is still not
good enough for my purposes, because in rare cases the end-user may
need to override it).

>> For memory detection, ACPI 3.0 allows the BIOS (" INT 15H, E820H") to
>> return extended attributes - mostly only a volatile/non-volatile flag.
>> This isn't in GRUB's information. ACPI 3.0 also allows the BIOS to
>> return areas of the type "AddressRangeUnusable" (e.g. faulty RAM).
> This is mostly unnecessary. Basically you need only to know if you can
> use a memory range or not. The only useful additional code would be
> ReclaimMemory

To handle standby states correctly the OS may need to know which areas
are volatile and which areas aren't (which can include knowing the
difference between volatile system areas and non-volatile system
areas). Some OSs also want to know if there's any faulty RAM present
in the system or not (and additional information about any area
reserved for "hot-plug" RAM, and NUMA ranges, but that information
comes from ACPI tables not BIOS functions so the OS can get this
information without GRUB).

>> If
>> the BIOS reports that the area from 0x00100000 to 0x001FFFFF is
>> faulty, then how is GRUB planning to handle that? My own code (if
>> using it's own boot loaders rather than GRUB) can tolerate RAM
>> failures anywhere in RAM except for about 64 KiB in low memory.
> GRUB can't do this right now because it doesn't recieve badram info
> soon enough. And even if it does most kernels expect first MiB to be
> usable.

You're right - all kernels that are designed to use "multi-boot
specification version 1" expect to be loaded at 0x00100000 and that
RAM below the EBDA is usable. I'm not sure what kernels designed for
"multi-boot specification version 2" expect...

>> While I'm on the subject, I also want a list of RAM areas that the
>> boot loader has relied on. That way if a 24/7 server detects that any
>> RAM that the boot loader relies on has become faulty it can warn the
>> user that the computer may not be able to boot. Currently I assume
>> that all multi-boot compliant boot loaders (including GRUB) rely on
>> all RAM below 0x00100000, but it's a dodgy hack I'd rather avoid.
> Such list is a blatant encapsulation breach. If you want such test,
> add it to bootloader, not OS.

When the OS is running and detects a RAM fault, you want the OS to run
a copy of GRUB (maybe inside an emulator or something) so the OS can
tell GRUB about the RAM fault, and GRUB can tell the OS if the RAM
fault might cause problems if the computer is rebooted (and so the OS
can send an email to the network administrators or something *before*
the computer is rebooted)?

You can't assume that the OS that is running is the same OS that
installed GRUB; or that the OS that is running has access to wherever
GRUB is installed; or that GRUB will be able to detect any faulty RAM
during boot.

>> In the memory map, GRUB should mark areas that it has used to pass
>> information to the OS (e.g. the multi-boot information structure) as
>> "multi-boot reclaimable" (in a similar way that ACPI uses the "ACPI
>> reclaimable" type). This would make it easier for the OS to avoid
>> overwriting this data before it attempts to read it.
> This information is available with a simple loop over mbi. I would
> rathjer avoid overcomplicating the standard because it increases a
> chance of having "half-compliant" OSes and "half-compliant" booters.

I'd rather have "fully compliant" OSes that are easier to write than
"fully compliant" OSes that are a pain in the neck to write because
you have to parse everything in the multi-boot information structure
before you can write to any RAM (except for your own ".bss").

If I can't rely on the firmware (like I currently do) then I have to
rely on GRUB, and have to copy everything from the multi-boot
information structure into my ".bss". So, how much extra space do I
need to allow in my ".bss"? What is the maximum number of drive
structures? What's the maximum number of memory map entries? What's
the maximum length of the command line? The multi-boot specification
doesn't say.

The alternative is for the OS to create it's own memory management
structures in its ".bss" that excludes RAM used by the multi-boot
information structure, then use these memory management structures to
allocate "actually free" RAM. In this case, the OS ends up building
the equivalent of a system memory map that includes "multi-boot
reclaimable" areas because the boot loader didn't.

>> The area types
>> need to be "architecture independent" types too (e.g. GRUB converts
>> ACPI area types, UEFI area types, etc into "standard multi-boot area
>> types").
> It already is.

You mean (from the multi-boot specification) "`type' is the variety of
address range represented, where a value of 1 indicates available ram,
and all other values currently indicated a reserved area."?

No sane OS complies with the specification (because the specification
isn't adequate, and doesn't allow ACPI reclaimable areas to be
reclaimed or anything else). All sane (non-compliant) OSs assume that
GRUB copies data "as is" from "int 0x15, eax=0xE820" directly into the
multi-boot information structure (because this is what "GRUB-legacy"
actually does do), and that 'type' means the same as it does for "int
0x15, eax=0xE820".

>> The "Boot device" field in the multi-boot information structure should
>> be improved to handle a variety of conditions; including if the disk
>> was an emulated disk (e.g. "El Torito" emulating a hard drive). The
>> BIOS drive number isn't much use (especially if the firmware is
>> coreboot, UEFI, OpenFirmware, etc), and should be replaced with
>> something that identifies the corresponding drive structure (this
>> includes USB).
> Boot device shouldn't be used at all. It was a mistake. Booter has no
> good way to know how OS will see the device. You should pass this
> parameter via commandline either as device name or UUID. You have
> scripting to automate this

A user who's using GRUB to boot Ubuntu decides to install my code in
another partition, then modify GRUB's configuration (in Ubuntu) so
that GRUB will also boot my code. Now my code needs to rely on the
user to not stuff up GRUB's configuration for my code?

>> The "boot loader name" field is nice, but it needs a "boot loader
>> version" field to go with it.
> it's a part of name. This field is more for displaying anyway and OS
> shouldn't do any checks based on it
>> A "firmware type" field is also needed.
> Can you respond in appropriate thread?

Ok - where is the appropriate thread?

>> The OS image also needs a different magic number to indicate that the
>> OS image is designed for future versions of the multi-boot
>> specification (rather than the old/current version). If the OS image
>> uses the new magic number, then the OS image must also include an
>> "version of the multi-boot specification that this image complies
>> with" field. If the OS image indicates that it's intended for a newer
>> version of the multi-boot specification than the boot loader complies
>> with, then the boot loader refuses to boot and displays a "this boot
>> loader needs to be upgraded" error. If the OS image has the old magic
>> number, and if the firmware is "PC BIOS" then the boot loader should
>> boot the old OS image. If the OS image has the old magic number, and
>> if the firmware is not "PC BIOS" then the boot loader refuses to boot
>> and displays a "this OS requires a PC BIOS" error message.
> Already implemented through feature fields

Is there a private version of the multi-boot specification that I'm
not aware of yet; or does GRUB fail to comply with the current
multi-boot specification?

>> In a similar way, if the OS image has the old magic number and the
>> boot loader is not running on 80x86 then the boot loader refuses to
>> boot and displays an error message. If the OS image has the new magic
>> number then it must also include a field that indicates which
>> architecture the OS is intended for, and the boot loader must check
>> this field and display a "this OS is intended for a different
>> architecture" error message (and refuse to boot) if it's wrong.
>>
> multiboot1 and multiboot2 have different magics

See above.

>>> Read it here: http://www.gnu.org/software/grub/manual/multiboot/multiboot.html
>>
>> That's the old version of the multi-boot specification (for
>> GRUB_legacy). I'm looking forward to a new version of the multi-boot
>> specification (for GRUB 2) that is designed to support different
>> architectures, different types of firmware on the same architecture,
>> etc; and hopefully other improvements.
>>
> multiboot1 ISN'T depreceated and grub2 supports multiboot1

See above.


Cheers,

Brendan

Re: PXEgrub development on grub2

[Joey Korkames]

>> > I don't know what pxegrub can do, but GRUB 2 has PXE support:
>> > http://grub.enbug.org/PXEBOOT
>> >
>> 
>> This would suffice for reading config and/or default from tftp so the
>> boot selection can be changed remotely on systems that have PXE.

I use PXE:UNDI all the time with Grub2, per that wiki. Works fine.

>> 
>> Is it possible to do something similar on systems that do not have PXE
>> (ie systems without PXE BIOS or Apple EFI)?
> 
> We should have network card drivers like GRUB Legacy had.  In GRUB Legacy,
> they were imported from Etherboot.  In GRUB 2 we can do the same, as long
> as they're GPL-compatible.
> 

s/Etherboot/gPXE?
I use gPXE's UNDI layer by way of pxelinux and that works fine. That 
project has stayed active for quite a long time and I'd think it best to 
integrate/shim their network stack onto GRUB2 than to reinvent the wheel 
(such as what pxelinux->gpxelinux has done). I believe that code is all 
GPL(.v?)

-joey

 

Re: PXEgrub development on grub2

[Vladimir 'phcoder' Serbinenko]

Joey Korkames wrote:
>>> > I don't know what pxegrub can do, but GRUB 2 has PXE support:
>>> > http://grub.enbug.org/PXEBOOT
>>> >
>>>
>>> This would suffice for reading config and/or default from tftp so the
>>> boot selection can be changed remotely on systems that have PXE.
>
> I use PXE:UNDI all the time with Grub2, per that wiki. Works fine.
>
>>>
>>> Is it possible to do something similar on systems that do not have PXE
>>> (ie systems without PXE BIOS or Apple EFI)?
>>
>> We should have network card drivers like GRUB Legacy had.  In GRUB
>> Legacy,
>> they were imported from Etherboot.  In GRUB 2 we can do the same, as
>> long
>> as they're GPL-compatible.
>>
>
> s/Etherboot/gPXE?
> I use gPXE's UNDI layer by way of pxelinux and that works fine. That
> project has stayed active for quite a long time and I'd think it best
> to integrate/shim their network stack onto GRUB2 than to reinvent the
> wheel (such as what pxelinux->gpxelinux has done). I believe that code
> is all GPL(.v?)
>
Quick check says that core is under GPLv2+. Some drivers are GPLV2-only
but we can live without them. Another possibility is lwIP but it has no
drivers, only TCP/IP stack. I started experiments with lwIP but haven't
gone far. It shouldn't hold anyone back. If you port gpxe to grub2 then
be sure all files you use are under GPLv2+ and change it to GPLv3+. Your
code should go to grub-extras because it's not original work. This has
also an advamtage of forcing not to add any code in kernel and in
particular avoid increasing core size for non-network setup
> -joey
>
>
>
>
>
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> http://lists.gnu.org/mailman/listinfo/grub-devel
>

Re: About firmware facilities

[Vladimir 'phcoder' Serbinenko]

Brendan Trotter wrote:
>> No. Usuable means only that firmware isn't destroyed. Any device may
>> be in a different state
>>     
>
> Any device (that the firmware assumes is in a certain state) may be
> left in a different state (that the firmware no longer knows about)?
>
> For a very simple example, imagine if the BIOS leaves the floppy motor
> on, and GRUB's own floppy driver uses the floppy and then turns the
> motor off. Then the OS uses the firmware to read from floppy, but the
> firmware thinks the floppy motor is still on and attempts to read from
> the floppy without turning the floppy motor on.
>
> If GRUB has it's own device drivers, and GRUB doesn't restore devices
> to the state that the firmware expects the devices to be in, then the
> firmware is unusable.
>
>   
Most OSes should use their own drivers to access devices.
> If an OS can't use the firmware, then the OS must rely on GRUB for
> everything instead, including strange "OS specific" things that nobody
> has seen any other OS do before.
>
>   
If nobody uses a particular feature in firmware then you shouldn't use
it either. Unused firmware features are often buggy. Moreover firmware
on x86 is useful only for bootstrap and once bootstrap is completed you
should forget it exists except some firmware-specific tasks as setting
boot device.
>>>>> Due to limitations in the original multi-boot specification my code
>>>>> switches back to real mode and uses the BIOS to do memory detection,
>>>>> do video mode detection, switch video modes and gather other
>>>>> information.
>>>>>           
>>>> Have you actually read the multiboot specification? Booter passes info
>>>> about memory and video mode in mbi (video for multiboot isn't
>>>> implemented yet). If you need firmware for basic bootup you're clearly
>>>> doing something wrong and are firmware-dependent. Of course it's your
>>>> freedom to make suboptimal software.
>>>>         
>>> I've read the multi-boot specification. I've also read the code in
>>> GRUB-legacy that does memory detection, and I'm unwilling to allow my
>>> code to rely on it for "quality control" reasons. Without going into
>>> details, GRUB-legacy tends to do a "minimal" job and then expects the
>>> user to fix the problem if/when it goes wrong (but even then it only
>>> offers a "uppermem" command without providing a way for the user to
>>> specify a complete system memory map).
>>>
>>>       
>> What is "minimal job" and "quality control"? We use standard
>> E820+(optionally)badram command. I've seen no OS do any more than
>> this.
>>     
>
> My code tries "int 0x15, eax=0xE820" expecting 24 bytes per area (ACPI
> 3.0); then it tries "int 0x15, eax=0xE820" expecting 20 bytes per
> area. If "int 0x15, eax=0xE820" isn't supported by the BIOS then you
> can assume it's an old computer (and old computers are painful).
>
> It tries "int 0x15, ax=0xE801", then "int 0x15, ah=0xC7", then "int
> 0x15, ah=0x8A", then "int 0x15, ah=0xDA88", then "int 0x15, ah=0x88",
> then CMOS locations 0x70 and 0x71.
Read code. GRUB fallback to old methods if newer aren't available.
>  If all of this fails (which does
> happen on some computers) then it does manual probing.
>   

> Some of the old BIOS functions have limited range - for example
> they'll return "number of KiB blocks at 0x00100000" as a 16-bit
> integer, and can't return more than 64 MiB. In this case my code won't
> know if the value returned by the BIOS has been limited to 0xFFFF, so
> it'll do manual probing to detect any more RAM above the reported
> area. Some computers have an "ISA hole" from 0x00F00000 to 0x00FFFFFF.
> Because of this all older BIOS functions that report "amount of RAM at
> 0x00100000" may return 14 MiB (from 0x00100000 to 0x00EFFFFF) when
> there's actually another area of RAM at 0x01000000. In this case my
> code won't know if the value returned by the BIOS is right or not, and
> it'll do manual probing to detect any more RAM at 0x00100000. If my
> code has to do manual probing, then it assumes there's an "ISA hole"
> from 0x00F00000 to 0x00FFFFFF (regardless of whether there is or not)
> as this hole was used for memory mapped video cards (which would seem
> like RAM).
>   
Have you thought that manual probe may detect MMIO as additional "RAM"?
Video RAM isn't the only case of MMIO.
I prefer to function correctly on bug-free firmware at cost of quirks on
buggy ones rather than other way round
> For all BIOS functions used my code avoids all known BIOS bugs (and
> there's plenty of them). This includes "sanitizing" the data returned
> from "int 0x15, eax=0xE820" - sorting the list and handling any
> overlapping areas.
>   
have you ever looked at mmap folder?
> I've never needed to provide a way for the end-user to override my
> memory detection.
>
>   
Neither did we. But test your manual probing at 4GiB system - it's
likely to detect all MMIO addresses as RAM.


>>> For memory detection, ACPI 3.0 allows the BIOS (" INT 15H, E820H") to
>>> return extended attributes - mostly only a volatile/non-volatile flag.
>>> This isn't in GRUB's information. ACPI 3.0 also allows the BIOS to
>>> return areas of the type "AddressRangeUnusable" (e.g. faulty RAM).
>>>       
>> This is mostly unnecessary. Basically you need only to know if you can
>> use a memory range or not. The only useful additional code would be
>> ReclaimMemory
>>     
>
> To handle standby states correctly the OS may need to know which areas
> are volatile and which areas aren't (which can include knowing the
> difference between volatile system areas and non-volatile system
> areas). Some OSs also want to know if there's any faulty RAM present
> in the system or not (and additional information about any area
> reserved for "hot-plug" RAM, and NUMA ranges, but that information
> comes from ACPI tables not BIOS functions so the OS can get this
> information without GRUB).
>
>   
I'm ok with defining additional types in multiboot1. But OS considering
multiboot type to be BIOS type is buggy
>>
>> GRUB can't do this right now because it doesn't recieve badram info
>> soon enough. And even if it does most kernels expect first MiB to be
>> usable.
>>     
>
> You're right - all kernels that are designed to use "multi-boot
> specification version 1" expect to be loaded at 0x00100000 and that
> RAM below the EBDA is usable. I'm not sure what kernels designed for
> "multi-boot specification version 2" expect...
>
>   
Read what I said
>> Such list is a blatant encapsulation breach. If you want such test,
>> add it to bootloader, not OS.
>>     
>
> When the OS is running and detects a RAM fault, you want the OS to run
> a copy of GRUB (maybe inside an emulator or something) so the OS can
> tell GRUB about the RAM fault, and GRUB can tell the OS if the RAM
> fault might cause problems if the computer is rebooted (and so the OS
> can send an email to the network administrators or something *before*
> the computer is rebooted)?
>
> You can't assume that the OS that is running is the same OS that
> installed GRUB; or that the OS that is running has access to wherever
> GRUB is installed; or that GRUB will be able to detect any faulty RAM
> during boot.
>
>   
You're in circular logic. You assume that booter is using faulty RAM but
supplying RAM it used correctly.
>>
>> This information is available with a simple loop over mbi. I would
>> rathjer avoid overcomplicating the standard because it increases a
>> chance of having "half-compliant" OSes and "half-compliant" booters.
>>     
>
> I'd rather have "fully compliant" OSes that are easier to write than
> "fully compliant" OSes that are a pain in the neck to write because
> you have to parse everything in the multi-boot information structure
> before you can write to any RAM (except for your own ".bss").
>
> If I can't rely on the firmware (like I currently do) then I have to
> rely on GRUB, and have to copy everything from the multi-boot
> information structure into my ".bss". So, how much extra space do I
> need to allow in my ".bss"? What is the maximum number of drive
> structures? What's the maximum number of memory map entries? What's
> the maximum length of the command line? The multi-boot specification
> doesn't say.
>
>   
First do a small parse and count how many memory the structures you need
to take.

>>> The "Boot device" field in the multi-boot information structure should
>>> be improved to handle a variety of conditions; including if the disk
>>> was an emulated disk (e.g. "El Torito" emulating a hard drive). The
>>> BIOS drive number isn't much use (especially if the firmware is
>>> coreboot, UEFI, OpenFirmware, etc), and should be replaced with
>>> something that identifies the corresponding drive structure (this
>>> includes USB).
>>>       
>> Boot device shouldn't be used at all. It was a mistake. Booter has no
>> good way to know how OS will see the device. You should pass this
>> parameter via commandline either as device name or UUID. You have
>> scripting to automate this
>>     
>
> A user who's using GRUB to boot Ubuntu decides to install my code in
> another partition, then modify GRUB's configuration (in Ubuntu) so
> that GRUB will also boot my code. Now my code needs to rely on the
> user to not stuff up GRUB's configuration for my code?
>
>   
You can simply tell him to add "source" line
>>> The "boot loader name" field is nice, but it needs a "boot loader
>>> version" field to go with it.
>>>       
>> it's a part of name. This field is more for displaying anyway and OS
>> shouldn't do any checks based on it
>>     
>>> A "firmware type" field is also needed.
>>>       
>> Can you respond in appropriate thread?
>>     
>
> Ok - where is the appropriate thread?
>
>   
ML search is your friend
>>> The OS image also needs a different magic number to indicate that the
>>> OS image is designed for future versions of the multi-boot
>>> specification (rather than the old/current version). If the OS image
>>> uses the new magic number, then the OS image must also include an
>>> "version of the multi-boot specification that this image complies
>>> with" field. If the OS image indicates that it's intended for a newer
>>> version of the multi-boot specification than the boot loader complies
>>> with, then the boot loader refuses to boot and displays a "this boot
>>> loader needs to be upgraded" error. If the OS image has the old magic
>>> number, and if the firmware is "PC BIOS" then the boot loader should
>>> boot the old OS image. If the OS image has the old magic number, and
>>> if the firmware is not "PC BIOS" then the boot loader refuses to boot
>>> and displays a "this OS requires a PC BIOS" error message.
>>>       
>> Already implemented through feature fields
>>     
>
> Is there a private version of the multi-boot specification that I'm
> not aware of yet; or does GRUB fail to comply with the current
> multi-boot specification?
>
>   
No. Read specification again

Re: About firmware facilities

[Robert Millan]

On Tue, 2009-09-15 at 04:27 +0930, Brendan Trotter wrote:
> On Tue, Sep 15, 2009 at 1:02 AM, Robert Millan <rmh@aybabtu.com> wrote:
> > Anyhow, my priority for GRUB is strong driver-based support.  We could recruit
> > someone to develop the framework in next year's GSoC (unless somebody steps
> > in, of course).
> 
> Why stop there?
> 
> If proprietory ethernet ROMs aren't good enough, then what about
> proprietory SCSI ROMs, and proprietory firmware/BIOS?

As I said, we have to compromise sometimes.  But if you aren't satisfied
with the freedom we provide, you're more than welcome to help us go
further.  For example, you could help us improve the coreboot port.

On Mon, Sep 14, 2009 at 03:11:37PM -0400, Pavel Roskin wrote:
> > 
> > Sigh. I think I understand now - lack of logical thinking leads to
> > lack of rational behavior.
> 
> Ad hominem arguments are not welcome here.

Indeed.  Brendan, we accept reasonable criticism, but this kind of rethoric
is not welcome here, and is particularly useless if your aim is trying to
persuade us.

-- 
Robert Millan

  The DRM opt-in fallacy: "Your data belongs to us. We will decide when (and
  how) you may access your data; but nobody's threatening your freedom: we
  still allow you to remove your data and not access it at all."

Re: About firmware facilities

[Brendan Trotter]

Hi,

On Sat, Sep 19, 2009 at 11:36 PM, Vladimir 'phcoder' Serbinenko
<phcoder@gmail.com> wrote:
> Brendan Trotter wrote:
>>> No. Usuable means only that firmware isn't destroyed. Any device may
>>> be in a different state
>>
>> Any device (that the firmware assumes is in a certain state) may be
>> left in a different state (that the firmware no longer knows about)?
>>
>> For a very simple example, imagine if the BIOS leaves the floppy motor
>> on, and GRUB's own floppy driver uses the floppy and then turns the
>> motor off. Then the OS uses the firmware to read from floppy, but the
>> firmware thinks the floppy motor is still on and attempts to read from
>> the floppy without turning the floppy motor on.
>>
>> If GRUB has it's own device drivers, and GRUB doesn't restore devices
>> to the state that the firmware expects the devices to be in, then the
>> firmware is unusable.
>>
> Most OSes should use their own drivers to access devices.

Most device manufacturers should provide full documentation so that
programmers can write drivers to access the devices; and manufacturers
should provide hardware samples (and documentation) to these
programmers so that the device driver is ready before the device is
made available to the general public. Unfortunately the real world
just doesn't work the same as "should".

The multi-boot specification says the firmware is left in a usable
state. If GRUB doesn't leave the firmware in a usable state, then
either GRUB is wrong or the multi-boot specification is wrong. You
can't have it both ways.

Of course I'm forgetting that GRUB also supports chainloading (e.g.
the chainloaded OS tries to use the firmware to load more of it's
data, and the firmware fails because GRUB left a device in an
unexpected state) - non-compliance with the multi-boot specification
isn't the only issue.

>> If an OS can't use the firmware, then the OS must rely on GRUB for
>> everything instead, including strange "OS specific" things that nobody
>> has seen any other OS do before.
>>
> If nobody uses a particular feature in firmware then you shouldn't use
> it either. Unused firmware features are often buggy. Moreover firmware
> on x86 is useful only for bootstrap and once bootstrap is completed you
> should forget it exists except some firmware-specific tasks as setting
> boot device.

So, can I rely on GRUB to (for e.g.) setup video in a way that is
suitable for my code, or do I need to use the firmware myself (and
hope that GRUB hasn't left a device in an unexpected state)?

>>>>>> Due to limitations in the original multi-boot specification my code
>>>>>> switches back to real mode and uses the BIOS to do memory detection,
>>>>>> do video mode detection, switch video modes and gather other
>>>>>> information.
>>>>>>
>>>>> Have you actually read the multiboot specification? Booter passes info
>>>>> about memory and video mode in mbi (video for multiboot isn't
>>>>> implemented yet). If you need firmware for basic bootup you're clearly
>>>>> doing something wrong and are firmware-dependent. Of course it's your
>>>>> freedom to make suboptimal software.
>>>>>
>>>> I've read the multi-boot specification. I've also read the code in
>>>> GRUB-legacy that does memory detection, and I'm unwilling to allow my
>>>> code to rely on it for "quality control" reasons. Without going into
>>>> details, GRUB-legacy tends to do a "minimal" job and then expects the
>>>> user to fix the problem if/when it goes wrong (but even then it only
>>>> offers a "uppermem" command without providing a way for the user to
>>>> specify a complete system memory map).
>>>>
>>>>
>>> What is "minimal job" and "quality control"? We use standard
>>> E820+(optionally)badram command. I've seen no OS do any more than
>>> this.
>>>
>>
>> My code tries "int 0x15, eax=0xE820" expecting 24 bytes per area (ACPI
>> 3.0); then it tries "int 0x15, eax=0xE820" expecting 20 bytes per
>> area. If "int 0x15, eax=0xE820" isn't supported by the BIOS then you
>> can assume it's an old computer (and old computers are painful).
>>
>> It tries "int 0x15, ax=0xE801", then "int 0x15, ah=0xC7", then "int
>> 0x15, ah=0x8A", then "int 0x15, ah=0xDA88", then "int 0x15, ah=0x88",
>> then CMOS locations 0x70 and 0x71.
> Read code. GRUB fallback to old methods if newer aren't available.

I read the code (for both GRUB 1.96 and GRUB 0.97) and wrote down
exactly which BIOS functions GRUB does use in my last post. You didn't
read the code (and didn't read what I wrote either), and now you're
telling me to read the code?

>>  If all of this fails (which does
>> happen on some computers) then it does manual probing.
>
>> Some of the old BIOS functions have limited range - for example
>> they'll return "number of KiB blocks at 0x00100000" as a 16-bit
>> integer, and can't return more than 64 MiB. In this case my code won't
>> know if the value returned by the BIOS has been limited to 0xFFFF, so
>> it'll do manual probing to detect any more RAM above the reported
>> area. Some computers have an "ISA hole" from 0x00F00000 to 0x00FFFFFF.
>> Because of this all older BIOS functions that report "amount of RAM at
>> 0x00100000" may return 14 MiB (from 0x00100000 to 0x00EFFFFF) when
>> there's actually another area of RAM at 0x01000000. In this case my
>> code won't know if the value returned by the BIOS is right or not, and
>> it'll do manual probing to detect any more RAM at 0x00100000. If my
>> code has to do manual probing, then it assumes there's an "ISA hole"
>> from 0x00F00000 to 0x00FFFFFF (regardless of whether there is or not)
>> as this hole was used for memory mapped video cards (which would seem
>> like RAM).
>>
> Have you thought that manual probe may detect MMIO as additional "RAM"?
> Video RAM isn't the only case of MMIO.

Yes. For old computers there's plenty of unused area in the physical
address space and PCI devices are almost always assigned areas
starting from higher addresses and working down (which leaves a
massive "unused" area between the end of RAM and the first memory
mapped PCI device. For older systems (with ISA) the only usable space
is the "ISA hole" just below 0x01000000 (I already explained that my
code does probe this area).

For newer computers (e.g. anything made in the last 15 years) where
this might be a problem, the BIOS functions work and nothing needs to
be probed anyway.

> I prefer to function correctly on bug-free firmware at cost of quirks on
> buggy ones rather than other way round

I prefer to function correctly on modern BIOSs (where GRUB currently
works), and function correctly on almost all old BIOSs (where GRUB
currently fails) and function correctly on almost all buggy BIOSs
(where GRUB currently fails).

>> For all BIOS functions used my code avoids all known BIOS bugs (and
>> there's plenty of them). This includes "sanitizing" the data returned
>> from "int 0x15, eax=0xE820" - sorting the list and handling any
>> overlapping areas.
>>
> have you ever looked at mmap folder?

There is no mmap folder in the source code for GRUB 1.96 or GRUB 0.97.

>> I've never needed to provide a way for the end-user to override my
>> memory detection.
>>
> Neither did we. But test your manual probing at 4GiB system - it's
> likely to detect all MMIO addresses as RAM.

It's extremely unlikely that a computer with 4 GiB of RAM will fail on
all of the previous BIOS functions. If all of the previous BIOS
functions do fail, then you're probably running on an 80486 or older
computer which is unlikely to have more than 128 MiB of RAM.

If you think GRUB's memory detection never needs to be overridden then
you're obviously not testing it on anything that predates "int 0x15,
eax = 0xE820".

>>>> For memory detection, ACPI 3.0 allows the BIOS (" INT 15H, E820H") to
>>>> return extended attributes - mostly only a volatile/non-volatile flag.
>>>> This isn't in GRUB's information. ACPI 3.0 also allows the BIOS to
>>>> return areas of the type "AddressRangeUnusable" (e.g. faulty RAM).
>>>>
>>> This is mostly unnecessary. Basically you need only to know if you can
>>> use a memory range or not. The only useful additional code would be
>>> ReclaimMemory
>>
>> To handle standby states correctly the OS may need to know which areas
>> are volatile and which areas aren't (which can include knowing the
>> difference between volatile system areas and non-volatile system
>> areas). Some OSs also want to know if there's any faulty RAM present
>> in the system or not (and additional information about any area
>> reserved for "hot-plug" RAM, and NUMA ranges, but that information
>> comes from ACPI tables not BIOS functions so the OS can get this
>> information without GRUB).
>>
> I'm ok with defining additional types in multiboot1. But OS considering
> multiboot type to be BIOS type is buggy

I agree - most OSs that use multi-boot are buggy because they don't
comply with the specification (except mine, because I ignore GRUB's
memory map and get the information directly from the BIOS). The
question is which new types would be needed to ensure that
non-compliance isn't "deemed necessary" by OS developers in the
future, and how GRUB will know if the kernel image will understand the
new types correctly or if the kernel is an older (buggy) kernel that
(incorrectly) assumes ACPI types.

>>> GRUB can't do this right now because it doesn't recieve badram info
>>> soon enough. And even if it does most kernels expect first MiB to be
>>> usable.
>>>
>>
>> You're right - all kernels that are designed to use "multi-boot
>> specification version 1" expect to be loaded at 0x00100000 and that
>> RAM below the EBDA is usable. I'm not sure what kernels designed for
>> "multi-boot specification version 2" expect...
>>
>>
> Read what I said

In which way does existing kernels (that were designed for
GRUB-legacy) include future kernels (that might be designed to support
features that have been/could be introduced with GRUB2)?

>>> Such list is a blatant encapsulation breach. If you want such test,
>>> add it to bootloader, not OS.
>>
>> When the OS is running and detects a RAM fault, you want the OS to run
>> a copy of GRUB (maybe inside an emulator or something) so the OS can
>> tell GRUB about the RAM fault, and GRUB can tell the OS if the RAM
>> fault might cause problems if the computer is rebooted (and so the OS
>> can send an email to the network administrators or something *before*
>> the computer is rebooted)?
>>
>> You can't assume that the OS that is running is the same OS that
>> installed GRUB; or that the OS that is running has access to wherever
>> GRUB is installed; or that GRUB will be able to detect any faulty RAM
>> during boot.
>>
>>
> You're in circular logic. You assume that booter is using faulty RAM but
> supplying RAM it used correctly.

No. All RAM is OK when the boot loader boots the OS, but then
(possibly several months of running "24 hours per day" later) a RAM
fault occurs and the OS detects it, and the OS tells the user (or
administrator) that rebooting might cause problems due to the RAM
fault (because the OS knows that the faulty RAM will be used by the
boot loader).

>>> This information is available with a simple loop over mbi. I would
>>> rathjer avoid overcomplicating the standard because it increases a
>>> chance of having "half-compliant" OSes and "half-compliant" booters.
>>>
>>
>> I'd rather have "fully compliant" OSes that are easier to write than
>> "fully compliant" OSes that are a pain in the neck to write because
>> you have to parse everything in the multi-boot information structure
>> before you can write to any RAM (except for your own ".bss").
>>
>> If I can't rely on the firmware (like I currently do) then I have to
>> rely on GRUB, and have to copy everything from the multi-boot
>> information structure into my ".bss". So, how much extra space do I
>> need to allow in my ".bss"? What is the maximum number of drive
>> structures? What's the maximum number of memory map entries? What's
>> the maximum length of the command line? The multi-boot specification
>> doesn't say.
>>
>>
> First do a small parse and count how many memory the structures you need
> to take.

For something like a "live" CD; during boot you want the OS to do a
small parse and determine how much memory these structures will take,
then write to a read-only boot CD to change the kernel's ".bss" size?
And you want the OS to do this before GRUB has allocated memory for
the kernel or executed any of the OS's code?

>>>> The "Boot device" field in the multi-boot information structure should
>>>> be improved to handle a variety of conditions; including if the disk
>>>> was an emulated disk (e.g. "El Torito" emulating a hard drive). The
>>>> BIOS drive number isn't much use (especially if the firmware is
>>>> coreboot, UEFI, OpenFirmware, etc), and should be replaced with
>>>> something that identifies the corresponding drive structure (this
>>>> includes USB).
>>>>
>>> Boot device shouldn't be used at all. It was a mistake. Booter has no
>>> good way to know how OS will see the device. You should pass this
>>> parameter via commandline either as device name or UUID. You have
>>> scripting to automate this
>>>
>>
>> A user who's using GRUB to boot Ubuntu decides to install my code in
>> another partition, then modify GRUB's configuration (in Ubuntu) so
>> that GRUB will also boot my code. Now my code needs to rely on the
>> user to not stuff up GRUB's configuration for my code?
>>
> You can simply tell him to add "source" line

How does my code know if the user has set this "source" line correctly?

If someone is making a bootable CD that's meant to be used on 100
different computers, how should they set the "source" line?

>>>> The OS image also needs a different magic number to indicate that the
>>>> OS image is designed for future versions of the multi-boot
>>>> specification (rather than the old/current version). If the OS image
>>>> uses the new magic number, then the OS image must also include an
>>>> "version of the multi-boot specification that this image complies
>>>> with" field. If the OS image indicates that it's intended for a newer
>>>> version of the multi-boot specification than the boot loader complies
>>>> with, then the boot loader refuses to boot and displays a "this boot
>>>> loader needs to be upgraded" error. If the OS image has the old magic
>>>> number, and if the firmware is "PC BIOS" then the boot loader should
>>>> boot the old OS image. If the OS image has the old magic number, and
>>>> if the firmware is not "PC BIOS" then the boot loader refuses to boot
>>>> and displays a "this OS requires a PC BIOS" error message.
>>>>
>>> Already implemented through feature fields
>>
>> Is there a private version of the multi-boot specification that I'm
>> not aware of yet; or does GRUB fail to comply with the current
>> multi-boot specification?
>>
> No. Read specification again

The current multi-boot specification (version 0.6.95)? The one at:
http://www.gnu.org/software/grub/manual/multiboot/multiboot.html?

For the "flags" field in the kernel's Multiboot header, this version
of the specification says "Naturally, all as-yet-undefined bits in the
`flags' word must be set to zero in OS images." and there are no flags
defined that allows a kernel to indicate that it supports other types
of firmware (or any other feature/s introduced by GRUB2).

Can I assume that one or more of these "as-yet-undefined" bits have
been defined in some private version of the multi-boot specification
that I'm not aware of?


Cheers,

Brendan

Re: About firmware facilities

[Vladimir 'phcoder' Serbinenko]

Brendan Trotter wrote:
> Hi,
>
> On Sat, Sep 19, 2009 at 11:36 PM, Vladimir 'phcoder' Serbinenko
> <phcoder@gmail.com> wrote:
>   
>> Brendan Trotter wrote:
>>     
>>>> No. Usuable means only that firmware isn't destroyed. Any device may
>>>> be in a different state
>>>>         
>>> Any device (that the firmware assumes is in a certain state) may be
>>> left in a different state (that the firmware no longer knows about)?
>>>
>>> For a very simple example, imagine if the BIOS leaves the floppy motor
>>> on, and GRUB's own floppy driver uses the floppy and then turns the
>>> motor off. Then the OS uses the firmware to read from floppy, but the
>>> firmware thinks the floppy motor is still on and attempts to read from
>>> the floppy without turning the floppy motor on.
>>>
>>> If GRUB has it's own device drivers, and GRUB doesn't restore devices
>>> to the state that the firmware expects the devices to be in, then the
>>> firmware is unusable.
>>>
>>>       
>> Most OSes should use their own drivers to access devices.
>>     
>
> Most device manufacturers should provide full documentation so that
> programmers can write drivers to access the devices; and manufacturers
> should provide hardware samples (and documentation) to these
> programmers so that the device driver is ready before the device is
> made available to the general public. Unfortunately the real world
> just doesn't work the same as "should".
>
> The multi-boot specification says the firmware is left in a usable
> state. If GRUB doesn't leave the firmware in a usable state, then
> either GRUB is wrong or the multi-boot specification is wrong. You
> can't have it both ways.
>
> Of course I'm forgetting that GRUB also supports chainloading (e.g.
> the chainloaded OS tries to use the firmware to load more of it's
> data, and the firmware fails because GRUB left a device in an
> unexpected state) - non-compliance with the multi-boot specification
> isn't the only issue.
>
>   
Well we do have some sanitasation code as grub_stop_floppy. If you see
somewhere it's not enough, submit patch
>>> If an OS can't use the firmware, then the OS must rely on GRUB for
>>> everything instead, including strange "OS specific" things that nobody
>>> has seen any other OS do before.
>>>
>>>       
>> If nobody uses a particular feature in firmware then you shouldn't use
>> it either. Unused firmware features are often buggy. Moreover firmware
>> on x86 is useful only for bootstrap and once bootstrap is completed you
>> should forget it exists except some firmware-specific tasks as setting
>> boot device.
>>     
>
> So, can I rely on GRUB to (for e.g.) setup video in a way that is
> suitable for my code, or do I need to use the firmware myself (and
> hope that GRUB hasn't left a device in an unexpected state)?
>
>   
You can rely on GRUB once I implement it.
>>>>>>> Due to limitations in the original multi-boot specification my code
>>>>>>> switches back to real mode and uses the BIOS to do memory detection,
>>>>>>> do video mode detection, switch video modes and gather other
>>>>>>> information.
>>>>>>>
>>>>>>>               
>>>>>> Have you actually read the multiboot specification? Booter passes info
>>>>>> about memory and video mode in mbi (video for multiboot isn't
>>>>>> implemented yet). If you need firmware for basic bootup you're clearly
>>>>>> doing something wrong and are firmware-dependent. Of course it's your
>>>>>> freedom to make suboptimal software.
>>>>>>
>>>>>>             
>>>>> I've read the multi-boot specification. I've also read the code in
>>>>> GRUB-legacy that does memory detection, and I'm unwilling to allow my
>>>>> code to rely on it for "quality control" reasons. Without going into
>>>>> details, GRUB-legacy tends to do a "minimal" job and then expects the
>>>>> user to fix the problem if/when it goes wrong (but even then it only
>>>>> offers a "uppermem" command without providing a way for the user to
>>>>> specify a complete system memory map).
>>>>>
>>>>>
>>>>>           
>>>> What is "minimal job" and "quality control"? We use standard
>>>> E820+(optionally)badram command. I've seen no OS do any more than
>>>> this.
>>>>
>>>>         
>>> My code tries "int 0x15, eax=0xE820" expecting 24 bytes per area (ACPI
>>> 3.0); then it tries "int 0x15, eax=0xE820" expecting 20 bytes per
>>> area. If "int 0x15, eax=0xE820" isn't supported by the BIOS then you
>>> can assume it's an old computer (and old computers are painful).
>>>
>>> It tries "int 0x15, ax=0xE801", then "int 0x15, ah=0xC7", then "int
>>> 0x15, ah=0x8A", then "int 0x15, ah=0xDA88", then "int 0x15, ah=0x88",
>>> then CMOS locations 0x70 and 0x71.
>>>       
>> Read code. GRUB fallback to old methods if newer aren't available.
>>     
>
> I read the code (for both GRUB 1.96 and GRUB 0.97) and wrote down
> exactly which BIOS functions GRUB does use in my last post. You didn't
> read the code (and didn't read what I wrote either), and now you're
> telling me to read the code?
>
>   
GRUB2 already uses fallback chain. I'm ok with having more fallbacks but
only as long as they are sane.
> Yes. For old computers there's plenty of unused area in the physical
> address space and PCI devices are almost always assigned areas
> starting from higher addresses and working down (which leaves a
> massive "unused" area between the end of RAM and the first memory
> mapped PCI device. For older systems (with ISA) the only usable space
> is the "ISA hole" just below 0x01000000 (I already explained that my
> code does probe this area).
>
>   
You can't rely on BIOS mmap'ing the same way on all old computers.


> and function correctly on almost all old BIOSs (where GRUB
> currently fails) and function correctly
Maintaining support for very old computers is pain. I'm ok to add the
code as fallback (except probably manual probing) but not to claim we
support old computers
>  on almost all buggy BIOSs
> (where GRUB currently fails).
>
>   
If you speak about sorting and overlapping region then grub resolves
them in mmap.mod but multiboot spec doesn't require mmap to be sorted.
>>> For all BIOS functions used my code avoids all known BIOS bugs (and
>>> there's plenty of them). This includes "sanitizing" the data returned
>>> from "int 0x15, eax=0xE820" - sorting the list and handling any
>>> overlapping areas.
>>>
>>>       
>> have you ever looked at mmap folder?
>>     
>
> There is no mmap folder in the source code for GRUB 1.96 or GRUB 0.97.
>
>   
SVN?
>>> I've never needed to provide a way for the end-user to override my
>>> memory detection.
>>>
>>>       
>> Neither did we. But test your manual probing at 4GiB system - it's
>> likely to detect all MMIO addresses as RAM.
>>     
>
> It's extremely unlikely that a computer with 4 GiB of RAM will fail on
> all of the previous BIOS functions. If all of the previous BIOS
> functions do fail, then you're probably running on an 80486 or older
> computer which is unlikely to have more than 128 MiB of RAM.
>
>   
It's unlikely to have more than 64 MiB either. The problem is that mobos
may also have a bug similar to GateA20 and causes the same memory to be
detected twice at 2 or more different addresses. That's another reason
against manual probing. I'm ok to accept more BIOS functions fallback
but manual probing is bad. GRUB2 should be able to work even without all
memory detected and if you want to add manual probing you can supply an
external module which uses mmap.mod
> If you think GRUB's memory detection never needs to be overridden then
> you're obviously not testing it on anything that predates "int 0x15,
> eax = 0xE820".
>
>   
>>>>> For memory detection, ACPI 3.0 allows the BIOS (" INT 15H, E820H") to
>>>>> return extended attributes - mostly only a volatile/non-volatile flag.
>>>>> This isn't in GRUB's information. ACPI 3.0 also allows the BIOS to
>>>>> return areas of the type "AddressRangeUnusable" (e.g. faulty RAM).
>>>>>
>>>>>           
>>>> This is mostly unnecessary. Basically you need only to know if you can
>>>> use a memory range or not. The only useful additional code would be
>>>> ReclaimMemory
>>>>         
>>> To handle standby states correctly the OS may need to know which areas
>>> are volatile and which areas aren't (which can include knowing the
>>> difference between volatile system areas and non-volatile system
>>> areas). Some OSs also want to know if there's any faulty RAM present
>>> in the system or not (and additional information about any area
>>> reserved for "hot-plug" RAM, and NUMA ranges, but that information
>>> comes from ACPI tables not BIOS functions so the OS can get this
>>> information without GRUB).
>>>
>>>       
>> I'm ok with defining additional types in multiboot1. But OS considering
>> multiboot type to be BIOS type is buggy
>>     
>
> I agree - most OSs that use multi-boot are buggy because they don't
> comply with the specification (except mine, because I ignore GRUB's
> memory map and get the information directly from the BIOS). The
> question is which new types would be needed to ensure that
> non-compliance isn't "deemed necessary" by OS developers in the
> future, and how GRUB will know if the kernel image will understand the
> new types correctly or if the kernel is an older (buggy) kernel that
> (incorrectly) assumes ACPI types.
>
>   
We don't support buggy kernels. But I'm ok to either not use values 2-5
at all or use them only same way as BIOS does. Please go ahead and write
a patch for multiboot texinfo and post it into separate thread
>>>> GRUB can't do this right now because it doesn't recieve badram info
>>>> soon enough. And even if it does most kernels expect first MiB to be
>>>> usable.
>>>>
>>>>         
>>> You're right - all kernels that are designed to use "multi-boot
>>> specification version 1" expect to be loaded at 0x00100000 and that
>>> RAM below the EBDA is usable. I'm not sure what kernels designed for
>>> "multi-boot specification version 2" expect...
>>>
>>>
>>>       
>> Read what I said
>>     
>
> In which way does existing kernels (that were designed for
> GRUB-legacy) include future kernels (that might be designed to support
> features that have been/could be introduced with GRUB2)?
>
>   
multiboot1 is still supported.
>>>> Such list is a blatant encapsulation breach. If you want such test,
>>>> add it to bootloader, not OS.
>>>>         
>>> When the OS is running and detects a RAM fault, you want the OS to run
>>> a copy of GRUB (maybe inside an emulator or something) so the OS can
>>> tell GRUB about the RAM fault, and GRUB can tell the OS if the RAM
>>> fault might cause problems if the computer is rebooted (and so the OS
>>> can send an email to the network administrators or something *before*
>>> the computer is rebooted)?
>>>
>>> You can't assume that the OS that is running is the same OS that
>>> installed GRUB; or that the OS that is running has access to wherever
>>> GRUB is installed; or that GRUB will be able to detect any faulty RAM
>>> during boot.
>>>
>>>
>>>       
>> You're in circular logic. You assume that booter is using faulty RAM but
>> supplying RAM it used correctly.
>>     
>
> No. All RAM is OK when the boot loader boots the OS, but then
> (possibly several months of running "24 hours per day" later) a RAM
> fault occurs and the OS detects it, and the OS tells the user (or
> administrator) that rebooting might cause problems due to the RAM
> fault (because the OS knows that the faulty RAM will be used by the
> boot loader).
>
>   
It doesn't. Which RAM is used depends on things like version or even
command order. I prefer to have full badram support in booter.
>>>> This information is available with a simple loop over mbi. I would
>>>> rathjer avoid overcomplicating the standard because it increases a
>>>> chance of having "half-compliant" OSes and "half-compliant" booters.
>>>>
>>>>         
>>> I'd rather have "fully compliant" OSes that are easier to write than
>>> "fully compliant" OSes that are a pain in the neck to write because
>>> you have to parse everything in the multi-boot information structure
>>> before you can write to any RAM (except for your own ".bss").
>>>
>>> If I can't rely on the firmware (like I currently do) then I have to
>>> rely on GRUB, and have to copy everything from the multi-boot
>>> information structure into my ".bss". So, how much extra space do I
>>> need to allow in my ".bss"? What is the maximum number of drive
>>> structures? What's the maximum number of memory map entries? What's
>>> the maximum length of the command line? The multi-boot specification
>>> doesn't say.
>>>
>>>
>>>       
>> First do a small parse and count how many memory the structures you need
>> to take.
>>     
>
> For something like a "live" CD; during boot you want the OS to do a
> small parse and determine how much memory these structures will take,
> then write to a read-only boot CD to change the kernel's ".bss" size?
> And you want the OS to do this before GRUB has allocated memory for
> the kernel or executed any of the OS's code?
>
>   
I meant parsing MBI near entry point and taking account of it further
for memory allocations.
>>>>> The "Boot device" field in the multi-boot information structure should
>>>>> be improved to handle a variety of conditions; including if the disk
>>>>> was an emulated disk (e.g. "El Torito" emulating a hard drive). The
>>>>> BIOS drive number isn't much use (especially if the firmware is
>>>>> coreboot, UEFI, OpenFirmware, etc), and should be replaced with
>>>>> something that identifies the corresponding drive structure (this
>>>>> includes USB).
>>>>>
>>>>>           
>>>> Boot device shouldn't be used at all. It was a mistake. Booter has no
>>>> good way to know how OS will see the device. You should pass this
>>>> parameter via commandline either as device name or UUID. You have
>>>> scripting to automate this
>>>>
>>>>         
>>> A user who's using GRUB to boot Ubuntu decides to install my code in
>>> another partition, then modify GRUB's configuration (in Ubuntu) so
>>> that GRUB will also boot my code. Now my code needs to rely on the
>>> user to not stuff up GRUB's configuration for my code?
>>>
>>>       
>> You can simply tell him to add "source" line
>>     
>
> How does my code know if the user has set this "source" line correctly?
>
> If someone is making a bootable CD that's meant to be used on 100
> different computers, how should they set the "source" line?
>
>   
source includes your script. You can ensure your script to be correct.
>>>>> The OS image also needs a different magic number to indicate that the
>>>>> OS image is designed for future versions of the multi-boot
>>>>> specification (rather than the old/current version). If the OS image
>>>>> uses the new magic number, then the OS image must also include an
>>>>> "version of the multi-boot specification that this image complies
>>>>> with" field. If the OS image indicates that it's intended for a newer
>>>>> version of the multi-boot specification than the boot loader complies
>>>>> with, then the boot loader refuses to boot and displays a "this boot
>>>>> loader needs to be upgraded" error. If the OS image has the old magic
>>>>> number, and if the firmware is "PC BIOS" then the boot loader should
>>>>> boot the old OS image. If the OS image has the old magic number, and
>>>>> if the firmware is not "PC BIOS" then the boot loader refuses to boot
>>>>> and displays a "this OS requires a PC BIOS" error message.
>>>>>
>>>>>           
>>>> Already implemented through feature fields
>>>>         
>>> Is there a private version of the multi-boot specification that I'm
>>> not aware of yet; or does GRUB fail to comply with the current
>>> multi-boot specification?
>>>
>>>       
>> No. Read specification again
>>     
>
> The current multi-boot specification (version 0.6.95)? The one at:
> http://www.gnu.org/software/grub/manual/multiboot/multiboot.html?
>
> For the "flags" field in the kernel's Multiboot header, this version
> of the specification says "Naturally, all as-yet-undefined bits in the
> `flags' word must be set to zero in OS images." and there are no flags
> defined that allows a kernel to indicate that it supports other types
> of firmware (or any other feature/s introduced by GRUB2).
>   
Every payload is supposed to work on any firmware.
> Can I assume that one or more of these "as-yet-undefined" bits have
> been defined in some private version of the multi-boot specification
> that I'm not aware of?
>   
No

You speak about many things in a single thread. In this cases many of
things tend to be forgotten. Can you make precise patches for things I'm
ok with (I'm not maintainer so not the one who decides) and put other
ideas in separate threads (one idea per thread)