Request to revert superpage adjustments

Request to revert superpage adjustments

[Andrew Cooper]

Sadly, c/s cf393624 "x86: use 2M superpages for text/data/bss mappings"
exposes a bug in all Syslinux variants, including ISOlinux and
PXELinux,  causing a failure to boot.

Xen currently requires its bootloader to decompress it, and place it at
the 1MB physical boundary.  The alignment adjustments changed the size
of the decompressed (post mkelf32) image from 2.2MB to 7.1MB.  These
restrictions should be fixed independently of this exposed bug.  The
physical range between 0x100000 and 0x10fffe is prime clobbering space
for buggy BIOSes once the A20 line has been disabled (see c/s 1ed76797),
and if any reserved memory exists between 1MB and 1MB+sizeof(xen), the
bootloader wont be able to place Xen at its linked address.

Grub and iPXE work perfectly well when booting Xen, which is why this is
now clearly a Syslinux issue (all versions I cared to test, including
4.x and 6.3 are broken).  However, it clobbers any ability for XenServer
to do testing, as we PXEBoot our servers for install.  I expect a lot of
other people will encounter issues once the 4.7 RCs get tested.

Please revert c/s cf393624 and the following change (c/s 53aa3dde) which
depends upon the former, until I can work around the existing
restrictions.  After the restrictions are resolved, the patches can go
back in, but I am fairly sure I will not have time to resolve the issues
in the 4.7 timeframe.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Request to revert superpage adjustments

[Jan Beulich]

>>> On 11.03.16 at 10:22, <andrew.cooper3@citrix.com> wrote:
> Sadly, c/s cf393624 "x86: use 2M superpages for text/data/bss mappings"
> exposes a bug in all Syslinux variants, including ISOlinux and
> PXELinux,  causing a failure to boot.
> 
> Xen currently requires its bootloader to decompress it, and place it at
> the 1MB physical boundary.  The alignment adjustments changed the size
> of the decompressed (post mkelf32) image from 2.2MB to 7.1MB.  These
> restrictions should be fixed independently of this exposed bug.  The
> physical range between 0x100000 and 0x10fffe is prime clobbering space
> for buggy BIOSes once the A20 line has been disabled (see c/s 1ed76797),
> and if any reserved memory exists between 1MB and 1MB+sizeof(xen), the
> bootloader wont be able to place Xen at its linked address.
> 
> Grub and iPXE work perfectly well when booting Xen, which is why this is
> now clearly a Syslinux issue (all versions I cared to test, including
> 4.x and 6.3 are broken).  However, it clobbers any ability for XenServer
> to do testing, as we PXEBoot our servers for install.  I expect a lot of
> other people will encounter issues once the 4.7 RCs get tested.
> 
> Please revert c/s cf393624 and the following change (c/s 53aa3dde) which
> depends upon the former, until I can work around the existing
> restrictions.  After the restrictions are resolved, the patches can go
> back in, but I am fairly sure I will not have time to resolve the issues
> in the 4.7 timeframe.

I'm kind of hesitant to do a wholesale revert, for two reasons:

1) The change would still be useful for xen.efi, which is relocatable
already anyway.

2) I cannot currently see how you mean to address the issue:
Even if you made our binary decompress itself, that would only
paper over the issue, and (based on your description) only until
even the compressed image exceeds a certain size. Nor would
that deal with avoiding reserved regions not too far above 1Mb,
since at best the main payload can be relocatable (but certainly
not the binary seen by the boot loader, as at least multiboot1
doesn't support anything like that).

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Request to revert superpage adjustments

[Andrew Cooper]

On 11/03/16 12:06, Jan Beulich wrote:
>>>> On 11.03.16 at 10:22, <andrew.cooper3@citrix.com> wrote:
>> Sadly, c/s cf393624 "x86: use 2M superpages for text/data/bss mappings"
>> exposes a bug in all Syslinux variants, including ISOlinux and
>> PXELinux,  causing a failure to boot.
>>
>> Xen currently requires its bootloader to decompress it, and place it at
>> the 1MB physical boundary.  The alignment adjustments changed the size
>> of the decompressed (post mkelf32) image from 2.2MB to 7.1MB.  These
>> restrictions should be fixed independently of this exposed bug.  The
>> physical range between 0x100000 and 0x10fffe is prime clobbering space
>> for buggy BIOSes once the A20 line has been disabled (see c/s 1ed76797),
>> and if any reserved memory exists between 1MB and 1MB+sizeof(xen), the
>> bootloader wont be able to place Xen at its linked address.
>>
>> Grub and iPXE work perfectly well when booting Xen, which is why this is
>> now clearly a Syslinux issue (all versions I cared to test, including
>> 4.x and 6.3 are broken).  However, it clobbers any ability for XenServer
>> to do testing, as we PXEBoot our servers for install.  I expect a lot of
>> other people will encounter issues once the 4.7 RCs get tested.
>>
>> Please revert c/s cf393624 and the following change (c/s 53aa3dde) which
>> depends upon the former, until I can work around the existing
>> restrictions.  After the restrictions are resolved, the patches can go
>> back in, but I am fairly sure I will not have time to resolve the issues
>> in the 4.7 timeframe.
> I'm kind of hesitant to do a wholesale revert, for two reasons:
>
> 1) The change would still be useful for xen.efi, which is relocatable
> already anyway.

The latter change strictly depends on .init having 2M alignment, so
needs to go either way.  As for making a split, I am already out of time
which is why I opted for the plain revert.

>
> 2) I cannot currently see how you mean to address the issue:
> Even if you made our binary decompress itself, that would only
> paper over the issue, and (based on your description) only until
> even the compressed image exceeds a certain size.

Compressed, Xen is currently 700k and Linux weighs in at ~4MB, which is
half the size of the uncompressed Xen with superpages.

>  Nor would
> that deal with avoiding reserved regions not too far above 1Mb,
> since at best the main payload can be relocatable (but certainly
> not the binary seen by the boot loader, as at least multiboot1
> doesn't support anything like that).

The only reason Xen sits at the 1MB boundary is because of its ELF header.

A plain binary with a multiboot header has no such restriction, although
we flag an interested to have 4k alignment using the multiboot flags. 
There is no technical limitation causing Xen to be linked to run at 1MB;
its just the way its alway been.  There is nothing preventing the entry
point becoming properly relocatable.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Request to revert superpage adjustments

[Jan Beulich]

>>> On 11.03.16 at 14:21, <andrew.cooper3@citrix.com> wrote:
> On 11/03/16 12:06, Jan Beulich wrote:
>>>>> On 11.03.16 at 10:22, <andrew.cooper3@citrix.com> wrote:
>>> Sadly, c/s cf393624 "x86: use 2M superpages for text/data/bss mappings"
>>> exposes a bug in all Syslinux variants, including ISOlinux and
>>> PXELinux,  causing a failure to boot.
>>>
>>> Xen currently requires its bootloader to decompress it, and place it at
>>> the 1MB physical boundary.  The alignment adjustments changed the size
>>> of the decompressed (post mkelf32) image from 2.2MB to 7.1MB.  These
>>> restrictions should be fixed independently of this exposed bug.  The
>>> physical range between 0x100000 and 0x10fffe is prime clobbering space
>>> for buggy BIOSes once the A20 line has been disabled (see c/s 1ed76797),
>>> and if any reserved memory exists between 1MB and 1MB+sizeof(xen), the
>>> bootloader wont be able to place Xen at its linked address.
>>>
>>> Grub and iPXE work perfectly well when booting Xen, which is why this is
>>> now clearly a Syslinux issue (all versions I cared to test, including
>>> 4.x and 6.3 are broken).  However, it clobbers any ability for XenServer
>>> to do testing, as we PXEBoot our servers for install.  I expect a lot of
>>> other people will encounter issues once the 4.7 RCs get tested.
>>>
>>> Please revert c/s cf393624 and the following change (c/s 53aa3dde) which
>>> depends upon the former, until I can work around the existing
>>> restrictions.  After the restrictions are resolved, the patches can go
>>> back in, but I am fairly sure I will not have time to resolve the issues
>>> in the 4.7 timeframe.
>> I'm kind of hesitant to do a wholesale revert, for two reasons:
>>
>> 1) The change would still be useful for xen.efi, which is relocatable
>> already anyway.
> 
> The latter change strictly depends on .init having 2M alignment, so
> needs to go either way.  As for making a split, I am already out of time
> which is why I opted for the plain revert.

How about I try to find time to put together a partial revert
(hopefully early) next week?

>> 2) I cannot currently see how you mean to address the issue:
>> Even if you made our binary decompress itself, that would only
>> paper over the issue, and (based on your description) only until
>> even the compressed image exceeds a certain size.
> 
> Compressed, Xen is currently 700k and Linux weighs in at ~4MB, which is
> half the size of the uncompressed Xen with superpages.

Well, okay, there's some headroom (albeit my xen.gz binaries all are
above 900k).

>>  Nor would
>> that deal with avoiding reserved regions not too far above 1Mb,
>> since at best the main payload can be relocatable (but certainly
>> not the binary seen by the boot loader, as at least multiboot1
>> doesn't support anything like that).
> 
> The only reason Xen sits at the 1MB boundary is because of its ELF header.
> 
> A plain binary with a multiboot header has no such restriction, although
> we flag an interested to have 4k alignment using the multiboot flags. 
> There is no technical limitation causing Xen to be linked to run at 1MB;
> its just the way its alway been.  There is nothing preventing the entry
> point becoming properly relocatable.

Without proper container format, how would the boot loader
know where to place the binary, or how to relocate it if it doesn't
get placed at its linked address? The only alternatives I see in
grub1 are a.out and a kludge of a.out, both of which - at the first
glance - also don't appear to do any relocation. And for the Linux
variant, as said, it doesn't look like it's compatible with us needing
multiple modules.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Request to revert superpage adjustments

[Konrad Rzeszutek Wilk]

> >>  Nor would
> >> that deal with avoiding reserved regions not too far above 1Mb,
> >> since at best the main payload can be relocatable (but certainly
> >> not the binary seen by the boot loader, as at least multiboot1
> >> doesn't support anything like that).
> > 
> > The only reason Xen sits at the 1MB boundary is because of its ELF header.
> > 
> > A plain binary with a multiboot header has no such restriction, although
> > we flag an interested to have 4k alignment using the multiboot flags. 
> > There is no technical limitation causing Xen to be linked to run at 1MB;
> > its just the way its alway been.  There is nothing preventing the entry
> > point becoming properly relocatable.

And one can make it be at 2MB - which is what Daniel's patches did.
(CC-ing Daniel in case I got it wrong). We also needed GRUB2 patches
to take into account relocating Xen at a different location.

> 
> Without proper container format, how would the boot loader
> know where to place the binary, or how to relocate it if it doesn't
> get placed at its linked address? The only alternatives I see in
> grub1 are a.out and a kludge of a.out, both of which - at the first
> glance - also don't appear to do any relocation. And for the Linux
> variant, as said, it doesn't look like it's compatible with us needing
> multiple modules.
> 
> Jan
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Request to revert superpage adjustments

[Daniel Kiper]

On Fri, Mar 11, 2016 at 09:35:19AM -0500, Konrad Rzeszutek Wilk wrote:
> > >>  Nor would
> > >> that deal with avoiding reserved regions not too far above 1Mb,
> > >> since at best the main payload can be relocatable (but certainly
> > >> not the binary seen by the boot loader, as at least multiboot1
> > >> doesn't support anything like that).
> > >
> > > The only reason Xen sits at the 1MB boundary is because of its ELF header.
> > >
> > > A plain binary with a multiboot header has no such restriction, although
> > > we flag an interested to have 4k alignment using the multiboot flags.
> > > There is no technical limitation causing Xen to be linked to run at 1MB;
> > > its just the way its alway been.  There is nothing preventing the entry
> > > point becoming properly relocatable.
>
> And one can make it be at 2MB - which is what Daniel's patches did.

IIRC, with my patches it could be anything which is multiple of 2 MiB.

> (CC-ing Daniel in case I got it wrong). We also needed GRUB2 patches
> to take into account relocating Xen at a different location.

Konrad is correct.

By the way, I would like to mention that probably next week all required
multiboot2 functionality will be available in upstream git tree and later
in GRUB2 2.02 release. This means that new multiboot2 features are almost
set in stone.

> > Without proper container format, how would the boot loader
> > know where to place the binary, or how to relocate it if it doesn't
> > get placed at its linked address? The only alternatives I see in
> > grub1 are a.out and a kludge of a.out, both of which - at the first
> > glance - also don't appear to do any relocation. And for the Linux
> > variant, as said, it doesn't look like it's compatible with us needing
> > multiple modules.

multiboot and multiboot2 (without my extensions) put image exactly at address
specified in ELF or multiboot2 header. This means that boot loader is not able
to relocate image during load. I do not mention that current version of Xen
(without my EFI/multiboot2 patches) is not able to understand relocation
(if it would be provided/executed by loader) too.

Daniel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Request to revert superpage adjustments

[Andrew Cooper]

On 11/03/16 14:13, Jan Beulich wrote:
>>>> On 11.03.16 at 14:21, <andrew.cooper3@citrix.com> wrote:
>> On 11/03/16 12:06, Jan Beulich wrote:
>>>>>> On 11.03.16 at 10:22, <andrew.cooper3@citrix.com> wrote:
>>>> Sadly, c/s cf393624 "x86: use 2M superpages for text/data/bss mappings"
>>>> exposes a bug in all Syslinux variants, including ISOlinux and
>>>> PXELinux,  causing a failure to boot.
>>>>
>>>> Xen currently requires its bootloader to decompress it, and place it at
>>>> the 1MB physical boundary.  The alignment adjustments changed the size
>>>> of the decompressed (post mkelf32) image from 2.2MB to 7.1MB.  These
>>>> restrictions should be fixed independently of this exposed bug.  The
>>>> physical range between 0x100000 and 0x10fffe is prime clobbering space
>>>> for buggy BIOSes once the A20 line has been disabled (see c/s 1ed76797),
>>>> and if any reserved memory exists between 1MB and 1MB+sizeof(xen), the
>>>> bootloader wont be able to place Xen at its linked address.
>>>>
>>>> Grub and iPXE work perfectly well when booting Xen, which is why this is
>>>> now clearly a Syslinux issue (all versions I cared to test, including
>>>> 4.x and 6.3 are broken).  However, it clobbers any ability for XenServer
>>>> to do testing, as we PXEBoot our servers for install.  I expect a lot of
>>>> other people will encounter issues once the 4.7 RCs get tested.
>>>>
>>>> Please revert c/s cf393624 and the following change (c/s 53aa3dde) which
>>>> depends upon the former, until I can work around the existing
>>>> restrictions.  After the restrictions are resolved, the patches can go
>>>> back in, but I am fairly sure I will not have time to resolve the issues
>>>> in the 4.7 timeframe.
>>> I'm kind of hesitant to do a wholesale revert, for two reasons:
>>>
>>> 1) The change would still be useful for xen.efi, which is relocatable
>>> already anyway.
>> The latter change strictly depends on .init having 2M alignment, so
>> needs to go either way.  As for making a split, I am already out of time
>> which is why I opted for the plain revert.
> How about I try to find time to put together a partial revert
> (hopefully early) next week?
>
>>> 2) I cannot currently see how you mean to address the issue:
>>> Even if you made our binary decompress itself, that would only
>>> paper over the issue, and (based on your description) only until
>>> even the compressed image exceeds a certain size.
>> Compressed, Xen is currently 700k and Linux weighs in at ~4MB, which is
>> half the size of the uncompressed Xen with superpages.
> Well, okay, there's some headroom (albeit my xen.gz binaries all are
> above 900k).
>
>>>  Nor would
>>> that deal with avoiding reserved regions not too far above 1Mb,
>>> since at best the main payload can be relocatable (but certainly
>>> not the binary seen by the boot loader, as at least multiboot1
>>> doesn't support anything like that).
>> The only reason Xen sits at the 1MB boundary is because of its ELF header.
>>
>> A plain binary with a multiboot header has no such restriction, although
>> we flag an interested to have 4k alignment using the multiboot flags. 
>> There is no technical limitation causing Xen to be linked to run at 1MB;
>> its just the way its alway been.  There is nothing preventing the entry
>> point becoming properly relocatable.
> Without proper container format, how would the boot loader
> know where to place the binary, or how to relocate it if it doesn't
> get placed at its linked address? The only alternatives I see in
> grub1 are a.out and a kludge of a.out, both of which - at the first
> glance - also don't appear to do any relocation. And for the Linux
> variant, as said, it doesn't look like it's compatible with us needing
> multiple modules.

There is no need for the loader to know or care about the linked address
of the binary.

32bit segment bases are a very easy way around this problem while the
binary locates a suitable region to extract itself into.  The entry %eip
can be inspected to generate a suitable segment base.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel