Re: [ANNOUNCE] Gujin graphical bootloader 0.4

Re: [ANNOUNCE] Gujin graphical bootloader 0.4

[Etienne Lorrain]

> This is indeed a good structure, but this wide interface is a pain to
> keep stable, and having bootloaders call it directly is a genuinely
> bad idea.  It will lock us into an interface, or cause major breakage,
> when we have to do necessary revving of this interface.

 Note that this interface is so stable that the structure did not change
 for a long time. If someone wants to change it completely, he will
 have to rewrite tools which are accessing this structure (rdev) and
 also the bootloaders which are setting up fields into it already.
 This will involve re-coding real-mode i8086 assembly, and there is less
 and less people knowing how to do it.

> Instead, the proper time to deal with this is at kernel link time.
> The PC-BIOS stuff should go in, say arch/i386/pcbios, and you then can
> have other platforms (say, for example, arch/i386/linuxbios) which has
> its own setup code.  You then link a kernel image which has the
> appropriate code for the platform you're running on, and you're set.

 I can see your point about keeping functions which set up the structure
 and function which uses it in a single kernel release.
 In fact, all the functions setting this structure are in the same
 file in Gujin, vmlinuz.[ch]. This file could one day go into the Linux
 kernel, but we are no more speaking of compatibility.

 My main problem is the order the things are done: First load compressed
 files at defined addresses, then call a kernel function which callback the
 BIOS, then uncompress files.

 Once Gujin has started, I have a complete C environment so I can load
 files, treat errors, display messages. I can do this either from cold
 boot or from DOS (think of the first install of Linux on a system).

 A good solution would be to have the kernel being two (or three) GZIP
 files concatenated, the first would be the real-mode code to setup
 the structure only, the second would be the protected-mode code of the
 kernel (and the third the initrd). The first part would be a position
 independant function getting some parameters (address/max size of the
 structure to fill in) and returning information like microprocessor
 minimum requirement, video mode supported (number of BPP, or text only),
 address the kernel has been linked (to load a kernel at 16 Mb), ...

 Then I would call this setup function before doing invalid things like
 writing in the "DOS=HIGH" area. Note also that Gujin do not keep the
 compressed kernel/initrd files, it reads a block and uncompress it
 immediately because of the 64Kb limit on the data section.

 This interface would still not handle a distribution where there is
 few kernel files:
 /boot/Linux-2.4.8-SMP
 /boot/Linux-2.4.8-UP
 /boot/Linux-2.4.8-386
 /boot/Linux-2.4.8-Pentium
 And the bootloader should just select automagically the right kernel.

 I have to say that I simply do not have time to do such a thing,
 because I have a lot of other problems in Gujin: it is already
 a Linux-3.0 bootloader, not Linux-2.5 .
 Moreover, going from a simple solution (loading the binary image of
 an ELF file) to a complex one (as described) to solve problem which
 may appear in the future is not my way of thinking: it is already
 complex enought to do simple software.

  Etienne.

___________________________________________________________
Do You Yahoo!? -- Vos albums photos en ligne, 
Yahoo! Photos : http://fr.photos.yahoo.com

Re: [ANNOUNCE] Gujin graphical bootloader 0.4

[Keith Owens]

On Mon, 13 Aug 2001 14:05:05 +0200 (CEST), 
Etienne Lorrain <etienne_lorrain@yahoo.fr> wrote:
> A good solution would be to have the kernel being two (or three) GZIP
> files concatenated, the first would be the real-mode code to setup
> the structure only, the second would be the protected-mode code of the
> kernel (and the third the initrd). The first part would be a position
> independant function getting some parameters (address/max size of the
> structure to fill in) and returning information like microprocessor
> minimum requirement, video mode supported (number of BPP, or text only),
> address the kernel has been linked (to load a kernel at 16 Mb), ...

Before you go too far, there is already an standard for boot loading,
EFI (Extensible Firmware Interface).  Originally from Intel but it is
open.  http://developer.intel.com/technology/efi.  IA64 uses this and
nothing but this, it already loads kernels in ELF format.  There is no
point in inventing yet another boot interface, unless you cannot do
what you want in EFI.

Re: [ANNOUNCE] Gujin graphical bootloader 0.4

[Eric W. Biederman]

Keith Owens <kaos@ocs.com.au> writes:

> On Mon, 13 Aug 2001 14:05:05 +0200 (CEST), 
> Etienne Lorrain <etienne_lorrain@yahoo.fr> wrote:
> > A good solution would be to have the kernel being two (or three) GZIP
> > files concatenated, the first would be the real-mode code to setup
> > the structure only, the second would be the protected-mode code of the
> > kernel (and the third the initrd). The first part would be a position
> > independant function getting some parameters (address/max size of the
> > structure to fill in) and returning information like microprocessor
> > minimum requirement, video mode supported (number of BPP, or text only),
> > address the kernel has been linked (to load a kernel at 16 Mb), ...
> 
> Before you go too far, there is already an standard for boot loading,
> EFI (Extensible Firmware Interface).  Originally from Intel but it is
> open.  http://developer.intel.com/technology/efi.  IA64 uses this and
> nothing but this, it already loads kernels in ELF format.  There is no
> point in inventing yet another boot interface, unless you cannot do
> what you want in EFI.

Well unless someone removes the architecture specific assumptions of EFI,
and gets it going on every platform there certainly is.  Besides the
fact that despite a complete rewrite the current EFI BIOSes are the
slowest I have seen.  With linuxBIOS I can load a kernel image over
the network before my hard drives spin up to speed.  This isn't always
the right thing to do.  But waiting 45 seconds before you start
booting your operating system is equally insane.

Any bootloader interface that requires a callback for more then a
fatal abort of the loaded image will be misimplemented someday.  Even
on the alpha with it's small selection of Firmware this problem
persists.  So the interface to the firmware needs to be as trivially
simple as possible.

Eric

Re: [ANNOUNCE] Gujin graphical bootloader 0.4

[Eric W. Biederman]

Etienne Lorrain <etienne_lorrain@yahoo.fr> writes:

> > This is indeed a good structure, but this wide interface is a pain to
> > keep stable, and having bootloaders call it directly is a genuinely
> > bad idea.  It will lock us into an interface, or cause major breakage,
> > when we have to do necessary revving of this interface.
> 
>  Note that this interface is so stable that the structure did not change
>  for a long time. If someone wants to change it completely, he will
>  have to rewrite tools which are accessing this structure (rdev) and
>  also the bootloaders which are setting up fields into it already.
>  This will involve re-coding real-mode i8086 assembly, and there is less
>  and less people knowing how to do it.

True.  However for the moment I am up for making this kind of change.
rdev needs to be recoded anyway as it is not portable.  We need a rdev
like tool to attach a command line and a ramdisk to arbitrary linux
kernels. 

> > Instead, the proper time to deal with this is at kernel link time.
> > The PC-BIOS stuff should go in, say arch/i386/pcbios, and you then can
> > have other platforms (say, for example, arch/i386/linuxbios) which has
> > its own setup code.  You then link a kernel image which has the
> > appropriate code for the platform you're running on, and you're set.
> 
>  I can see your point about keeping functions which set up the structure
>  and function which uses it in a single kernel release.
>  In fact, all the functions setting this structure are in the same
>  file in Gujin, vmlinuz.[ch]. This file could one day go into the Linux
>  kernel, but we are no more speaking of compatibility.
> 
>  My main problem is the order the things are done: First load compressed
>  files at defined addresses, then call a kernel function which callback the
>  BIOS, then uncompress files.

How is this a problem.  Is it the error handling or something else?
I like the fact that the kernel decompresses itself.  This is one less
thing that the bootloader doesn't have to do.  And allows the
compression scheme to be changed at least theoretically.

>  Once Gujin has started, I have a complete C environment so I can load
>  files, treat errors, display messages. I can do this either from cold
>  boot or from DOS (think of the first install of Linux on a system).

This definentily sounds nice.  Do you envision the setup code
returning to the bootloader if a fatal error occurs?
 
>  A good solution would be to have the kernel being two (or three) GZIP
>  files concatenated, the first would be the real-mode code to setup
>  the structure only, the second would be the protected-mode code of the
>  kernel (and the third the initrd). The first part would be a position
>  independant function getting some parameters (address/max size of the
>  structure to fill in) and returning information like microprocessor
>  minimum requirement, video mode supported (number of BPP, or text only),
>  address the kernel has been linked (to load a kernel at 16 Mb), ...
> 
>  Then I would call this setup function before doing invalid things like
>  writing in the "DOS=HIGH" area. Note also that Gujin do not keep the
>  compressed kernel/initrd files, it reads a block and uncompress it
>  immediately because of the 64Kb limit on the data section.

Why does Gujin need to do the decompression?  Is this simply for
better error checking, and error handling?
 
>  This interface would still not handle a distribution where there is
>  few kernel files:
>  /boot/Linux-2.4.8-SMP
>  /boot/Linux-2.4.8-UP
>  /boot/Linux-2.4.8-386
>  /boot/Linux-2.4.8-Pentium
>  And the bootloader should just select automagically the right kernel.

At this port I have linux booting from linux so what I would do is
load /boot/Linux-2.4.8-i386 and then from linux select the right
kernel modify the bootloader parameters, and boot the optimal kernel.

>  I have to say that I simply do not have time to do such a thing,
>  because I have a lot of other problems in Gujin: it is already
>  a Linux-3.0 bootloader, not Linux-2.5 .
>  Moreover, going from a simple solution (loading the binary image of
>  an ELF file) to a complex one (as described) to solve problem which
>  may appear in the future is not my way of thinking: it is already
>  complex enought to do simple software.

I believe it can be done much more simply.  And I think I can make the
time to get it done, over the long term.   I can do with an ELF
binary, not it's binary ala objdump -O binary I can do what you
propose to do with concatenated GZIP files.  And with that I don't
need decompression code in the bootloader. 

Eric

Re: [ANNOUNCE] Gujin graphical bootloader 0.4

[Etienne Lorrain]

 --- "Eric W. Biederman" <ebiederm@xmission.com> a écrit :
> Etienne Lorrain <etienne_lorrain@yahoo.fr> writes:
> >  If someone wants to change it completely, he will
> >  have to rewrite tools which are accessing this structure (rdev) and
> >  also the bootloaders which are setting up fields into it already.
> >  This will involve re-coding real-mode i8086 assembly, and there is less
> >  and less people knowing how to do it.
> 
> True.  However for the moment I am up for making this kind of change.
> rdev needs to be recoded anyway as it is not portable.  We need a rdev
> like tool to attach a command line and a ramdisk to arbitrary linux
> kernels.

  I am from the old Unix school, I like text - so IMHO this could go in
 a string, like "root=/dev/hda3 initrd=initrd-3.gz". That is why I proposed
 to put that string in the "comment" field of the gzip file (see RFC).
 Unfortunately right now there is no tool to set up this field, but all the
 uncompression tools seems to treat correctly (skip) this field if present.
 Is there someone against it? Is there a software (like the "znote"
 for pkzip) available on Internet to set the comment field of a gz file?

> >  My main problem is the order the things are done: First load compressed
> >  files at defined addresses, then call a kernel function which callback
> >  the BIOS, then uncompress files.
> 
> How is this a problem.  Is it the error handling or something else?

 I hate to destroy random memory content and then ask to the lower
 layers to behave correctly. For instance I used a DOS debugger to find
 some of my bugs - it is not possible if the memory at 0x100000 has been
 overwritten. Even simply my logger/debugger (writing information in a
 log file under DOS) can not work.

> I like the fact that the kernel decompresses itself.  This is one less
> thing that the bootloader doesn't have to do.  And allows the
> compression scheme to be changed at least theoretically.

 I need the decompressor. People will want icons to clic on for Linux,
 and I plan to uncompress them with my gunzip. Maybe one day I will even
 compress a part of my bootloader, for instance the different language
 messages or even part of the code. Note that this ungzip-er is very small,
 5 Kb for the one in 0.4, 4.5 Kb for the current one - and in some cases
 do not even need a 32 Kb memory window, unlike the real zlib.
 Keep in mind that this bootloader may one day go to the EEPROM of a
 network card (32 Kb max) or in the empty FLASH area of the BIOS...
 Also have a look at the memory requirement of bzip2, few Mb...

> Why does Gujin need to do the decompression?  Is this simply for
> better error checking, and error handling?

 Also, the CRC32 calculation is nearly the only way to guaranty that you
 have a real kernel - and did not do any error in loading the file. The
 usual problems I am waiting here is to load wrong sectors because I have
 a bug in the CHS description of the disk, or I have read a corrupted sector
 because the floppy had bad sectors and the ECC correction did a bad job.

> >  Once Gujin has started, I have a complete C environment so I can load
> >  files, treat errors, display messages. I can do this either from cold
> >  boot or from DOS (think of the first install of Linux on a system).
> 
> This definentily sounds nice.  Do you envision the setup code
> returning to the bootloader if a fatal error occurs?

  Even more, I envision running GPL maintenance software - and return to
 the bootloader at the end because this software did take care to not
 modify memory below 1Mb.
 For instance I have corrected a small bug in the bootloader (first stage)
 because I had a sector to read on the HD - and this sector cannot be read
 at the first time (needs a 0x20 - read with retry - IDE command,
 a 0x21 - read without retry - return "uncorrected ECC error") and I was
 not waiting enough for the ECC correction to finnish its task.
 Because one of the main feature of e2fs is to not fragment files, you
 will never run an unfragmenter, so there will (probably) be more and
 more sectors which will need ECC correction each time they are read.
 What is then needed is a small application displaying the ratio of
 block needing ECC correction and maybe just read and rewrite them
 to optimise the system.

> >  This interface would still not handle a distribution where there is
> >  few kernel files:
> >  /boot/Linux-2.4.8-SMP
> >  /boot/Linux-2.4.8-UP
> >  /boot/Linux-2.4.8-386
> >  /boot/Linux-2.4.8-Pentium
> >  And the bootloader should just select automagically the right kernel.
> 
> At this port I have linux booting from linux so what I would do is
> load /boot/Linux-2.4.8-i386 and then from linux select the right
> kernel modify the bootloader parameters, and boot the optimal kernel.

  IHMO a bit complex, managing the command line parameters for the two
 kernels (like ide addresses, scsi disk order...)

> >  Moreover, going from a simple solution (loading the binary image of
> >  an ELF file) to a complex one (as described) to solve problem which
> >  may appear in the future is not my way of thinking: it is already
> >  complex enought to do simple software.
> 
> I believe it can be done much more simply.  And I think I can make the
> time to get it done, over the long term.   I can do with an ELF
> binary, not it's binary ala objdump -O binary I can do what you
> propose to do with concatenated GZIP files.  And with that I don't
> need decompression code in the bootloader. 

  For me, I choose to load a GZIP because its format is trivial enough
 to extract, and a "objdump -O binary" because I do not want to do the
 loading part of an ELF file. GZIP is also used for initrd, so you need
 it if you want to check the integrity what you have loaded.
  You can try any other method you want, I can't say I never did a
 wrong choice.
 
  Etienne.


___________________________________________________________
Do You Yahoo!? -- Vos albums photos en ligne, 
Yahoo! Photos : http://fr.photos.yahoo.com

Re: [ANNOUNCE] Gujin graphical bootloader 0.4

[Eric W. Biederman]

Etienne Lorrain <etienne_lorrain@yahoo.fr> writes:

>  --- "Eric W. Biederman" <ebiederm@xmission.com> a écrit :
> > Etienne Lorrain <etienne_lorrain@yahoo.fr> writes:
> > >  If someone wants to change it completely, he will
> > >  have to rewrite tools which are accessing this structure (rdev) and
> > >  also the bootloaders which are setting up fields into it already.
> > >  This will involve re-coding real-mode i8086 assembly, and there is less
> > >  and less people knowing how to do it.
> > 
> > True.  However for the moment I am up for making this kind of change.
> > rdev needs to be recoded anyway as it is not portable.  We need a rdev
> > like tool to attach a command line and a ramdisk to arbitrary linux
> > kernels.
> 
>   I am from the old Unix school, I like text - so IMHO this could go in
>  a string, like "root=/dev/hda3 initrd=initrd-3.gz". That is why I proposed
>  to put that string in the "comment" field of the gzip file (see RFC).
>  Unfortunately right now there is no tool to set up this field, but all the
>  uncompression tools seems to treat correctly (skip) this field if present.
>  Is there someone against it? Is there a software (like the "znote"
>  for pkzip) available on Internet to set the comment field of a gz file?

If you are going with gzip.  That definentily sounds like a good strategy.
 
> > >  My main problem is the order the things are done: First load compressed
> > >  files at defined addresses, then call a kernel function which callback
> > >  the BIOS, then uncompress files.
> > 
> > How is this a problem.  Is it the error handling or something else?
> 
>  I hate to destroy random memory content and then ask to the lower
>  layers to behave correctly. For instance I used a DOS debugger to find
>  some of my bugs - it is not possible if the memory at 0x100000 has been
>  overwritten. Even simply my logger/debugger (writing information in a
>  log file under DOS) can not work.

There is sense it that.  Having a way to do all reasonable of the
error checking up front and then going forward with running the kernel.

> > I like the fact that the kernel decompresses itself.  This is one less
> > thing that the bootloader doesn't have to do.  And allows the
> > compression scheme to be changed at least theoretically.
> 
>  I need the decompressor. People will want icons to clic on for Linux,
>  and I plan to uncompress them with my gunzip. Maybe one day I will even
>  compress a part of my bootloader, for instance the different language
>  messages or even part of the code. Note that this ungzip-er is very small,
>  5 Kb for the one in 0.4, 4.5 Kb for the current one - and in some cases
>  do not even need a 32 Kb memory window, unlike the real zlib.
>  Keep in mind that this bootloader may one day go to the EEPROM of a
>  network card (32 Kb max) or in the empty FLASH area of the BIOS...
>  Also have a look at the memory requirement of bzip2, few Mb...

O.k.  I'll definentily have to have a look at it then.  The gunzip code
I have seems to have been larger.
 
> > Why does Gujin need to do the decompression?  Is this simply for
> > better error checking, and error handling?
> 
>  Also, the CRC32 calculation is nearly the only way to guaranty that you
>  have a real kernel - and did not do any error in loading the file. The
>  usual problems I am waiting here is to load wrong sectors because I have
>  a bug in the CHS description of the disk, or I have read a corrupted sector
>  because the floppy had bad sectors and the ECC correction did a bad job.

O.k.  That is one area where you are definentily ahead of me.  I have
been playing with some kind of checksum but I haven't yet.
 
> > >  Once Gujin has started, I have a complete C environment so I can load
> > >  files, treat errors, display messages. I can do this either from cold
> > >  boot or from DOS (think of the first install of Linux on a system).
> > 
> > This definentily sounds nice.  Do you envision the setup code
> > returning to the bootloader if a fatal error occurs?
> 
>   Even more, I envision running GPL maintenance software - and return to
>  the bootloader at the end because this software did take care to not
>  modify memory below 1Mb.
>  For instance I have corrected a small bug in the bootloader (first stage)
>  because I had a sector to read on the HD - and this sector cannot be read
>  at the first time (needs a 0x20 - read with retry - IDE command,
>  a 0x21 - read without retry - return "uncorrected ECC error") and I was
>  not waiting enough for the ECC correction to finnish its task.
>  Because one of the main feature of e2fs is to not fragment files, you
>  will never run an unfragmenter, so there will (probably) be more and
>  more sectors which will need ECC correction each time they are read.
>  What is then needed is a small application displaying the ratio of
>  block needing ECC correction and maybe just read and rewrite them
>  to optimise the system.
> 
> > >  This interface would still not handle a distribution where there is
> > >  few kernel files:
> > >  /boot/Linux-2.4.8-SMP
> > >  /boot/Linux-2.4.8-UP
> > >  /boot/Linux-2.4.8-386
> > >  /boot/Linux-2.4.8-Pentium
> > >  And the bootloader should just select automagically the right kernel.
> > 
> > At this port I have linux booting from linux so what I would do is
> > load /boot/Linux-2.4.8-i386 and then from linux select the right
> > kernel modify the bootloader parameters, and boot the optimal kernel.
> 
>   IHMO a bit complex, managing the command line parameters for the two
>  kernels (like ide addresses, scsi disk order...)

Oh it can get complex, but if all you want to do is to invoke another
kernel with exactly the same options you were invoked with it is a
simple thing to do.

> > >  Moreover, going from a simple solution (loading the binary image of
> > >  an ELF file) to a complex one (as described) to solve problem which
> > >  may appear in the future is not my way of thinking: it is already
> > >  complex enought to do simple software.
> > 
> > I believe it can be done much more simply.  And I think I can make the
> > time to get it done, over the long term.   I can do with an ELF
> > binary, not it's binary ala objdump -O binary I can do what you
> > propose to do with concatenated GZIP files.  And with that I don't
> > need decompression code in the bootloader. 
> 
>   For me, I choose to load a GZIP because its format is trivial enough
>  to extract, and a "objdump -O binary" because I do not want to do the
>  loading part of an ELF file. 

For purposes of a loading a binary the ELF program header simply says load
this chunk of file to this chunk of memory.  So while it is a touch more
complex then a GZIP header it is simpler than a whole gzip file.

With ELF I have so far implemented 3 bootloaders, that can handle it
which allows me to boot arbitrary kernels.

> GZIP is also used for initrd, so you need
>  it if you want to check the integrity what you have loaded.
>   You can try any other method you want, I can't say I never did a
>  wrong choice.

Where I have been focusing is on network booting, and booting in small
environments without a BIOS.  And for those having the option to
package the kernel and ramdisk, and the command line together is
important, as is the option to reduce size by using a different
compression algorithm.  Which is my primary reason for wanting to
include the decompressor with the kernel, and require the bootloader
to use it.


That isn't all of my comments but I need to run off to work now.

Later,
Eric

Re: [ANNOUNCE] Gujin graphical bootloader 0.4

[Eric W. Biederman]

"H. Peter Anvin" <hpa@zytor.com> writes:

> The point is that this belongs in the kernel image, so that it can be evolved,
> not in the boot loaders, where it becomes static.  These kinds of things will in
> 
> practice change too quickly to be frozen into boot loaders.

In principle I agree.  But until I have thought out all of the angles
I'll play with just about every idea.

I will say though that having a 32bit entry point in a bootloader is
fully reasonable as, if you are doing anything more than zImage the
bootloader needs to switch into protected mode anyway.  However having
a the ability to switch back into realmode to do BIOS calls is also
reasonable. The interesting case there is loadlin.

But I fully agree that bootloaders should be as simple as possible
with respect to the kernel.  But I also have a major issue with 
the fact that rdev only works on x86.  And even with linuxBIOS around
it would be nice to compile a kernel that is x86-generic.  That is one
of the neater ideas of the alpha port.  I know on the ppc port it
splits up per bootloader.  I don't have any problems with compiling
for a specific environment but I don't want to make it mandatory.

Eric

Re: [ANNOUNCE] Gujin graphical bootloader 0.4

[H. Peter Anvin]

Eric W. Biederman wrote:
> 
>>From your reaction I'm not explaining myself well.  And since I'm
> working with a work in progress that isn't too much of a suprise.
> 
> The basic rule is that nothing that can be queried from the hardware
> directly should be passed to the kernel.  However we do need to have
> an interface to describe the hardware that we can't do a
> PCI/ISAPNP/USB/etc bus scan to get.  To a certain extent the
> information is optional because sometimes we cannot get it.  But if we
> can it is good to have.  
> 
> That is all I intend to pass to the linux kernel besides a command
> line the unprobeable hardware details.  If something becomes probeable
> in the future that wasn't in the past, I'd spec it as optional to pass
> that information.  
> 
> For the kernel loaders I'd definentily have a standard probe routine
> that would query the traditional BIOS, and then package the results in
> the format I'm suggesting.  Because of working around BUGS sometimes
> you need extra information that gets lost in translation, so I'm not
> 100% certain that is the best way to go.  However it is possible to
> turn things on their heads and share the same code between multiple
> operating systems at which point it makes real sense to move that code
> into a bootloader.  This is one of those questions worth looking very
> closely at.
> 

The point is that this belongs in the kernel image, so that it can be 
evolved, not in the boot loaders, where it becomes static.  These kinds 
of things will in practice change too quickly to be frozen into boot 
loaders.

It's still a bad idea.

	-hpa

Re: [ANNOUNCE] Gujin graphical bootloader 0.4

[Eric W. Biederman]

"H. Peter Anvin" <hpa@zytor.com> writes:

> This is indeed a good structure, but this wide interface is a pain to
> keep stable, and having bootloaders call it directly is a genuinely
> bad idea.  It will lock us into an interface, or cause major breakage,
> when we have to do necessary revving of this interface.
> 
> Instead, the proper time to deal with this is at kernel link time.
> The PC-BIOS stuff should go in, say arch/i386/pcbios, and you then can
> have other platforms (say, for example, arch/i386/linuxbios) which has
> its own setup code.  You then link a kernel image which has the
> appropriate code for the platform you're running on, and you're set.

Probably.  My hope is that with linuxbios I can come up with a format
that is sufficient to be an internal interface.  There is also another
major reason for looking at this.  We don't have the same kind of
interface on any port.  For example rdev only exists on x86.  Making
the whole bootloader process a major pain everytime you swich a linux
hardware platform. 

> You're now
> encoding a ton of assumptions about what the kernel needs in each and
> every bootloader, which is bound to cause a major headache not too
> long down the road.  For example, the stuff you describe above may
> very well be obsolete in 2 years with HyperTransport, Infiniband and
> 3GIO on the very near horizon.  Now you have to suffer dealing with
> lots and lots of compatibility logic to make things work, which may
> not be possible, or we're going to have frequent breakage.
> 
> I do not believe this is a good idea.  This kind of information
> belongs in the kernel image, although it should be abstracted out
> within the kernel tree.

>From your reaction I'm not explaining myself well.  And since I'm
working with a work in progress that isn't too much of a suprise.

The basic rule is that nothing that can be queried from the hardware
directly should be passed to the kernel.  However we do need to have
an interface to describe the hardware that we can't do a
PCI/ISAPNP/USB/etc bus scan to get.  To a certain extent the
information is optional because sometimes we cannot get it.  But if we
can it is good to have.  

That is all I intend to pass to the linux kernel besides a command
line the unprobeable hardware details.  If something becomes probeable
in the future that wasn't in the past, I'd spec it as optional to pass
that information.  

For the kernel loaders I'd definentily have a standard probe routine
that would query the traditional BIOS, and then package the results in
the format I'm suggesting.  Because of working around BUGS sometimes
you need extra information that gets lost in translation, so I'm not
100% certain that is the best way to go.  However it is possible to
turn things on their heads and share the same code between multiple
operating systems at which point it makes real sense to move that code
into a bootloader.  This is one of those questions worth looking very
closely at.

For the unprobeable hardware what I want to do is pass in what amounts
to a struct pci_device tree.  But with struct pci_device generalized
to struct device as has been proposed for 2.5.  A preseeded tree
shouldn't be to hard to check up on, to see if it is accurate.
Verifing that I can handle resources like interrupts, and structures
that are not trees is my current challenge, with this.  Things like
interrupts tend to at least be DAGs.  I think the interfaces will need
some small extensions to handle 3GIO, Hypertransport, and Infiniband,
and other bus technologies to come.  Sort of like you need another RFC
when you encode another protocol on top of IP.  Just something to
specify exactly how the encoding will work.

I'll fight this in a month or two when I start integrating this into
2.5 and have had a chance to start integrating this in.  I'll see if I
can have this as a seperate of startup files in early 2.5 so that this
code can be tested seperately.  But there is real virtue in putting
all of your software eggs in one basket so long as it is a very good
basket.

Eric

Re: [ANNOUNCE] Gujin graphical bootloader 0.4

[H. Peter Anvin]

Followup to:  <m1n157rrpk.fsf@frodo.biederman.org>
By author:    ebiederm@xmission.com (Eric W. Biederman)
In newsgroup: linux.dev.kernel
> 
> There are good reasons to use the kernel 32 bit entry point.  In
> particular I routinely run linux on systems with exactly 10 16bit
> instructions.   On one of them I don't even have normal memory between
> 640KB and 1MB. The only real parameter the kernel needs from the BIOS
> is memory size, and I could probably steel code from memtest86 so it
> wouldn't even need that.  
> 
> However I do understand the instabilities, and the deficiets of the
> current 32bit entry point, and why having a standard 16bit entry point
> is a good thing.  Which is why in 2.5 (if it ever starts) I intend to
> do the work required so we can have a standard cross platform native
> mode entry point to the kernel.  
> 
> To keep linux portable we should never assume that on a given platform
> there is a specific kind of BIOS.  Alpha-linux at least is nasty
> because of this.  x86 linux is very nice because all you need to do on
> platforms that don't do support the classic BIOS interface is to drop
> the 16bit header.  That is definentily a structure worth keeping.
> 

This is indeed a good structure, but this wide interface is a pain to
keep stable, and having bootloaders call it directly is a genuinely
bad idea.  It will lock us into an interface, or cause major breakage,
when we have to do necessary revving of this interface.

Instead, the proper time to deal with this is at kernel link time.
The PC-BIOS stuff should go in, say arch/i386/pcbios, and you then can
have other platforms (say, for example, arch/i386/linuxbios) which has
its own setup code.  You then link a kernel image which has the
appropriate code for the platform you're running on, and you're set.

> Currently I have a stable easy to use structure that isn't even linux
> specific with just a few more details on how to encode parameters to
> work out.  The structure is the ELF format for static executables,
> with a specific implementation of how parameters will be passed to it
> from the bootloader, before the bootloader goes away.  In particular
> how to specify things like onboard ISA devices so we don't even have
> to assume what is or is not present on a motherboard for those devices
> that don't support probing and there is a firmware interface for
> finding them.  The interesting case there is not so much how to encode
> the device but instead on how to represent the location of devices,
> and the connections between devices.   
> 
> Being able to describe how an interrupt goes from a pci slot to an irq
> router to a legacy interrupt controller to a local apic and to the
> cpu, and simultanesly goes from the pci slot to an ioapic to a local
> apic to the cpu.   And saying that pci slot is behind a PCIX<->pci
> bridge.  Is an interesting question.  Especially in data structures
> that have very few special cases.  We are close in the kernel with
> struct resource, and struct pci_device but we haven't gone all of the
> way yet. 
> 
> So no I don't think it is right to blast someone for using the 32bit
> entry point, while at the same time I do agree that right now it is a
> very questionable things to do.

I haven't blasted anyone; I said it is a bad idea.  You're now
encoding a ton of assumptions about what the kernel needs in each and
every bootloader, which is bound to cause a major headache not too
long down the road.  For example, the stuff you describe above may
very well be obsolete in 2 years with HyperTransport, Infiniband and
3GIO on the very near horizon.  Now you have to suffer dealing with
lots and lots of compatibility logic to make things work, which may
not be possible, or we're going to have frequent breakage.

I do not believe this is a good idea.  This kind of information
belongs in the kernel image, although it should be abstracted out
within the kernel tree.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: [ANNOUNCE] Gujin graphical bootloader 0.4

[Eric W. Biederman]

"H. Peter Anvin" <hpa@zytor.com> writes:

> Followup to:  <Pine.LNX.4.33.0108060228220.10664-100000@hp.masroudeau.com>
> By author:    Etienne Lorrain <etienne@masroudeau.com>
> In newsgroup: linux.dev.kernel
> > 
> >  - Loads Linux kernel images (zImage and bZimage) without the help
> >    of LILO, and short-circuit all real-mode code in the kernel to
> >    start at the first protected mode instruction of the kernel.
> > 
> 
> This is a very bad idea.  The kernel entry point is in real mode for a
> reason: it means that the kernel doesn't have to rely on the boot
> loader to provide the services it needs from real mode before
> entering protected mode once and for all.  The interface to the real
> mode entry point is narrow and stable, the protected mode entrypoint
> is a kernel internal and doesn't have an interface that is guaranteed
> to be stable -- again, for good reason.

There are good reasons to use the kernel 32 bit entry point.  In
particular I routinely run linux on systems with exactly 10 16bit
instructions.   On one of them I don't even have normal memory between
640KB and 1MB. The only real parameter the kernel needs from the BIOS
is memory size, and I could probably steel code from memtest86 so it
wouldn't even need that.  

However I do understand the instabilities, and the deficiets of the
current 32bit entry point, and why having a standard 16bit entry point
is a good thing.  Which is why in 2.5 (if it ever starts) I intend to
do the work required so we can have a standard cross platform native
mode entry point to the kernel.  

To keep linux portable we should never assume that on a given platform
there is a specific kind of BIOS.  Alpha-linux at least is nasty
because of this.  x86 linux is very nice because all you need to do on
platforms that don't do support the classic BIOS interface is to drop
the 16bit header.  That is definentily a structure worth keeping.

Currently I have a stable easy to use structure that isn't even linux
specific with just a few more details on how to encode parameters to
work out.  The structure is the ELF format for static executables,
with a specific implementation of how parameters will be passed to it
from the bootloader, before the bootloader goes away.  In particular
how to specify things like onboard ISA devices so we don't even have
to assume what is or is not present on a motherboard for those devices
that don't support probing and there is a firmware interface for
finding them.  The interesting case there is not so much how to encode
the device but instead on how to represent the location of devices,
and the connections between devices.   

Being able to describe how an interrupt goes from a pci slot to an irq
router to a legacy interrupt controller to a local apic and to the
cpu, and simultanesly goes from the pci slot to an ioapic to a local
apic to the cpu.   And saying that pci slot is behind a PCIX<->pci
bridge.  Is an interesting question.  Especially in data structures
that have very few special cases.  We are close in the kernel with
struct resource, and struct pci_device but we haven't gone all of the
way yet. 

So no I don't think it is right to blast someone for using the 32bit
entry point, while at the same time I do agree that right now it is a
very questionable things to do.

Eric

Re: [ANNOUNCE] Gujin graphical bootloader 0.4

[Etienne Lorrain]

 --- Giacomo Catenazzi <cate@math.ethz.ch> a écrit :
> Etienne Lorrain wrote:
> >  These two files in memory have also to be at fixed linear
> >  addresses in real mode - and if you have a memory manager
> >  (himem.sys) loaded, these address may not be free. Usually
> >  you will find at the bottom of the himem memory the smartdrv
> >  (disk cache) data. It is then impossible to load a file at a random
> >  memory address and stay in real mode to do futher processing.
> >  In this case, Gujin is just malloc'ing the memory (using himem.sys),
> >  loading and decompressing this file (checking its CRC32), and
> >  only then disable interrupts, switch to protected mode, copy
> >  the file at its intended linear address and jump to the kernel
> >  code.
> > 
> 
> hmm.
> 
> You say: BIOS/hardware can be broken, let use DOS do load Linux!

  There are ISA/PCI cards which are initialised before the first sector
 of the HD/FD is loaded (SCSI, network boot ROM or special devices), but
 it is true that they are not usually using memory at 0x100000.
  I have just done this relocation thinking that it may be nice to be
 able to load a kernel over the 16 Mbytes limit - to free a maximum
 of space to the 16 Mb DMA-able area (if DMA is 16 Mb only) - but that
 is stuff for the future because the kernel would be linked at another
 address. I applied this late-relocation in case of himem.

> IMHO using DOS (himem.sys and letting DOS to setting our
> hardware in ibmbio.com and ibmdos.com) give us more problem
> that solution!

  I am not able to open the A20 gate in all the ways himem.sys can,
 objdump will be your friend here. Note that Linux has never booted
 on such a special machine - using I/O ports 0x329 or 0x320 to open A20 -
 I just leave the door open to run from DOS/win, even for beginner
 who do not know what is himem/emm386 - or people wanting to remote
 boot Linux from a DOS/win machine getting vmlinuz from the network.

> We should have complete control to hardware, not letting DOS
> to hide/modify the BIOS segment 0040:0000 and some other
> hardware setting.

  So Gujin will boot from an IDE HD not supported by the BIOS,
 on a VGA card without BIOS (or nearly: see VGA_MEMORY compile switch).
 It can also do what loadlin does - I do not want to maintain two
 identical software.

> Thus we should (if possible) use only BIOS call (or directly hardware),
> but forget DOS. (BTW you know what DOS makes before himem.sys ?
> Do we have the sources?)

  Forget DOS/Win - what a dream... Can I join your dream?

  Etienne.

___________________________________________________________
Do You Yahoo!? -- Vos albums photos en ligne, 
Yahoo! Photos : http://fr.photos.yahoo.com

Re: [ANNOUNCE] Gujin graphical bootloader 0.4

[Giacomo Catenazzi]

Etienne Lorrain wrote:
 
>  These two files in memory have also to be at fixed linear
>  addresses in real mode - and if you have a memory manager
>  (himem.sys) loaded, these address may not be free. Usually
>  you will find at the bottom of the himem memory the smartdrv
>  (disk cache) data. It is then impossible to load a file at a random
>  memory address and stay in real mode to do futher processing.
>  In this case, Gujin is just malloc'ing the memory (using himem.sys),
>  loading and decompressing this file (checking its CRC32), and
>  only then disable interrupts, switch to protected mode, copy
>  the file at its intended linear address and jump to the kernel
>  code.
> 

hmm.

You say: BIOS/hardware can be broken, let use DOS do load Linux!

IMHO using DOS (himem.sys and letting DOS to setting our
hardware in ibmbio.com and ibmdos.com) give us more problem
that solution!

We should have complete control to hardware, not letting DOS
to hide/modify the BIOS segment 0040:0000 and some other
hardware setting.
Thus we should (if possible) use only BIOS call (or directly hardware),
but forget DOS. (BTW you know what DOS makes before himem.sys ?
Do we have the sources?)


	giacomo

Re: [ANNOUNCE] Gujin graphical bootloader 0.4

[Etienne Lorrain]

hpa wrote:
> Etienne wrote:
> >  - Loads Linux kernel images (zImage and bZimage) without the help
> >    of LILO, and short-circuit all real-mode code in the kernel to
> >    start at the first protected mode instruction of the kernel.
>
> This is a very bad idea.  The kernel entry point is in real mode for a
> reason: it means that the kernel doesn't have to rely on the boot
> loader to provide the services it needs from real mode before
> entering protected mode once and for all.  The interface to the real
> mode entry point is narrow and stable, the protected mode entrypoint
> is a kernel internal and doesn't have an interface that is guaranteed
> to be stable -- again, for good reason.

 Thanks for the constructive criticism, now it is time to explain
 why I did it this way - it is far to be the simplest solution.

 First, I need to say that the Linux current interface is really
 stable: if you define "BOOT_KERNEL_BEFORE_2_0" in vmlinuz.[ch],
 Gujin will be able to boot pre- 1.0 Linux versions. Also, Gujin
 has always booted kernels this way.

 Considering the kernel interface to the bootloader, I find it is
 IMHO a bit complex: you not only have to give parameters in
 memory (at 0x9000:0000 or at %esi on 2.4.1+) but also provide
 callbacks (BIOS calls) for quite a lot of things.

 Alan Cox described it (incompletely) in his message in june,
 subject "draft BIOS use document for the kernel" at:
http://marc.theaimsgroup.com/?l=linux-kernel&m=99322719013363&w=2

 You also have to put in memory one or two compressed files
 (kernel and initrd) without any way to check if those hundred
 of Kbytes have been correctly read and are not corrupted.
 There will be no way - if an error is detected by the legacy
 loader at the decompression stage - to return to the bootloader
 saying: Fatal error, please select another kernel.

 These two files in memory have also to be at fixed linear
 addresses in real mode - and if you have a memory manager
 (himem.sys) loaded, these address may not be free. Usually
 you will find at the bottom of the himem memory the smartdrv
 (disk cache) data. It is then impossible to load a file at a random
 memory address and stay in real mode to do futher processing.
 In this case, Gujin is just malloc'ing the memory (using himem.sys),
 loading and decompressing this file (checking its CRC32), and
 only then disable interrupts, switch to protected mode, copy
 the file at its intended linear address and jump to the kernel
 code.

 Then, with the "old" method to load the kernel, you have the video
 selection menu. Well, it is a bit complex a thing to do in
 a bootloader, and its interface is complex (not speaking of the
 user interface), considering that you have already written everywhere
 in memory without knowing if (for instance) a video driver was loaded.
 All the video selection stuff has be to handled before selecting
 a kernel to load, IMO.

 I could probably find other problem, but I do not want to do
 destructive criticism.

 What do I propose? Lets put is simply:

 - The kernel file is a simple binary image to load at 0x100000,
 so you get it from the linux ELF file by just doing:
  objcopy -O binary -R .note -R .comment -S /usr/src/linux/vmlinux kernel
  gzip -9 kernel
 There is absolutely no limit on its size.

 - The initrd (if present) is a gzip of a filesystem image.

 - All information needed by the kernel is set in memory. It is
 clearly described on a C structure (vmlinuz.h::struct linux_param).
 This structure contains also (for reference) old fields which
 are no more used, or fields which were used only by previous
 bootloader. The is _no_ BIOS callback.

 - The hardware is setup correctly, for instance for the current
 video mode - if you are in graphic 8 BPP - the 256 color palette
 is set up with reasonnable values independant of the video card.
 Basically, hardware is immediately useable by the kernel.

 - There is a compatibility mode to load vmlinuz/zImage/bzImage.

 Note that all this is coded and working (I hope so). I know that
 with this solution, the bootloader and the kernel are more
 linked together, but I know also that the bootloader has to
 be a lot more intelligent (considering the number of related
 messages in the Linux lists). It should not try to load a Pentium
 compiled kernel on a 486. It should setup the video card (so
 be ready to get "invalid instruction exceptions" while executing
 BIOS calls). It should not try to run a corrupted kernel. It should
 not crash if a kernel/initrd file has been moved or two hard disks
 have been exchanged. It should display clear messages on what is
 wrong.
 It is up to the bootloader to crash if one BIOS call modify one
 register which is documented as constant (if it did not take care)
 - and in DEBUG mode send some informations to a serial/parallel port.
 When a new buggy BIOS appear, it is up to the bootloader to be upgraded,
 not to the Linux kernel.
 In short it is a real and complete software which handle all/most
 of the (buggy) BIOSes in their natural environment: the i386 real mode.
 If there is few fields to add to "struct linux_param" (really
 unusual from history), then Gujin will be upgraded. Anyway it is
 GPL - and written in C so there is a lot of people around able to
 change it, unlike assembly code.

 It is not as clean as described because of the APM / ACPI /
 PCI configuration BIOS32 calls; but that did not change by the
 Gujin bootloader. Note also that the SMP tables are passed in
 memory.

 One last thing I have to add (for people who have read up to here),
 is that having removed the "header" of the vmlinuz file, I lack the
 only important parameter in this header: the root device (rdev command).
 Right now, I am guessing it in a lot of configurations, but that
 is not a perfect solution. I think one of the simplest way is
 to add a "root=/dev/hd**" in the described comment field of the GZIP
 header; that is still not coded.
 From the same area, there is no way to know if a kernel will be able
 to boot in graphic mode (support of VESA framebuffer and which BPP
 are available). If vesafb is not compiled in and you start the
 kernel in graphic, the kernel boots but the display is like a
 crash... Right now the dirty way is the write-protect bit of the
 vmlinuz file - not a long term solution.

  Comment anyone?

  Etienne.

Re: [ANNOUNCE] Gujin graphical bootloader 0.4

[H. Peter Anvin]

Followup to:  <Pine.LNX.4.33.0108060228220.10664-100000@hp.masroudeau.com>
By author:    Etienne Lorrain <etienne@masroudeau.com>
In newsgroup: linux.dev.kernel
> 
>  - Loads Linux kernel images (zImage and bZimage) without the help
>    of LILO, and short-circuit all real-mode code in the kernel to
>    start at the first protected mode instruction of the kernel.
> 

This is a very bad idea.  The kernel entry point is in real mode for a
reason: it means that the kernel doesn't have to rely on the boot
loader to provide the services it needs from real mode before
entering protected mode once and for all.  The interface to the real
mode entry point is narrow and stable, the protected mode entrypoint
is a kernel internal and doesn't have an interface that is guaranteed
to be stable -- again, for good reason.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: [ANNOUNCE] Gujin graphical bootloader 0.4

[Etienne Lorrain]

On Thu, 9 Aug 2001, Matthias Andree wrote:
> On Mon, 06 Aug 2001, Etienne Lorrain wrote:
> >   I released the 0.4 version of the GPL Gujin bootloader.
> >
> >  Its homepage (with screenshoots) is at:
> > http://gujin.sourceforge.net/
>
> Given that I need to use rdev: is there any way to tell gujin to pass
> parameters to the kernel? I use ide0=ata66 on my 2.2 kernels, and other
> options on some other machines.

  Right now, it is only possible at the installation of Gujin on the
 floppy or at the boot.exe generation, by using the "--cmdline=" option
 of "./instboot" - so it will append the parameters to all kernel booted.
 So you will need to recompile from the source.

 It is still not possible to have different options for different kernels.

  --> Have a look at "./instboot --help".
 The maximum size of this string is 64 bytes (compile time: see boot.h,
 struct gujin_param_str->extra_cmdline).

  The target is to have a setup screen where this field can be modified,
 but before a more generic menu system is needed, and also a way to
 select the keyboard language.

  Note that you need to use rdev only if you have your kernel on a scsi
 disk or in a "small" partition at the beginning of the disk.
  If the kernel is /boot/vmlinuz*, and /boot is not a mount point,
 the root partition is surely the /

 If /boot is a mount point, there is also the trick:
cp /boot/vmlinuz* /tmp
umount /boot
cp /tmp/vmlinuz* /boot
mount /boot
 So you can select the right file at next reboot.


> Do you plan to set up a gujin mailing list on sourceforge to pull
> discussions off here?

 I was thinking the forum would be enought, but if you want it,
 and considering sourceforge says it is better to create those three:

gujin-devel@lists.sourceforge.net
gujin-users@lists.sourceforge.net
gujin-announce@lists.sourceforge.net

 Sourceforge says:
It will take 6-24 Hours for your list to be created.

 See you there,
 Etienne.

Re: [ANNOUNCE] Gujin graphical bootloader 0.4

[Matthias Andree]

On Mon, 06 Aug 2001, Etienne Lorrain wrote:

>   I released the 0.4 version of the GPL Gujin bootloader.
> 
>  Its homepage (with screenshoots) is at:
> http://gujin.sourceforge.net/

Given that I need to use rdev: is there any way to tell gujin to pass
parameters to the kernel? I use ide0=ata66 on my 2.2 kernels, and other
options on some other machines.

Do you plan to set up a gujin mailing list on sourceforge to pull
discussions off here?

-- 
Matthias Andree

[ANNOUNCE] Gujin graphical bootloader 0.4

[Etienne Lorrain]

  Hello there,

  I released the 0.4 version of the GPL Gujin bootloader.

 Its homepage (with screenshoots) is at:
http://gujin.sourceforge.net/
 Its documentation (FAQ, HowTo) is at:
http://sourceforge.net/docman/display_doc.php?docid=1989&group_id=15465
 All other information and source/precompiled downloads at:
http://sourceforge.net/projects/gujin/

 In short, this bootloader:
 - Is for PCs with at least an 80386 and a VGA, VESA1 or VESA2+
   graphic card.
 - Loads Linux kernel images (zImage and bZimage) without the help
   of LILO, and short-circuit all real-mode code in the kernel to
   start at the first protected mode instruction of the kernel.
   You can use it to load unlimited size kernels and initrd, and
   easily select the text/graphic mode for running the kernel.
 - is nearly completely written in C using GCC, so it is easy to
   modify/improve it. It uses e2fsprogs-1.22 to understand E2FS
   filesystems and a completely rewritten zlib to decompress the
   kernel. FAT12/FAT16 support is working.
 - stay in real mode for the complete boot process, so should be
   compatible with all PCs.
 - Can be installed as a bootloader (only on a floppy right now)
   or as a DOS/Win boot.exe
 - 80 % of the code is already written, so I just need to write
   the last 80 % to have the 1.0 version -:) It is IHMO already
   a good rescue floppy disk or a good single floppy system
   loader (minimum configuration around 20 Kb).

 Have fun!
 Etienne.