Re: [BUG 1747]Guest could't find bootable device with memory more than 3600M

[Stefano Stabellini <stefano.stabellini@eu.citrix.com>]

On Fri, 7 Jun 2013, George Dunlap wrote:
> On 06/07/2013 01:15 PM, Stefano Stabellini wrote:
> > On Fri, 7 Jun 2013, Xu, YongweiX wrote:
> > > Hi Stefano Stabellini,
> > > 
> > > I found a new bug "Guest could't find bootable device with memory more
> > > than 3600M".
> > > Attach the link:
> > > http://bugzilla-archived.xenproject.org//bugzilla/show_bug.cgi?id=1857
> > > 
> > > When booting up guest(include Linux&Windows guest) with memory more than
> > > 3600M,the guest will show "No bootable device" and could not boot up. Then
> > > the guest will continuous reboot automatically and never found bootable
> > > device. But with guest memory less than or equal to 3600M, boot up
> > > successfully.
> > > 
> > > I found this is the qemu(qemu-upstream-unstable) issue, the latest version
> > > (commit:4597594c61add43725bd207bb498268a058f9cfb) caused this issue:
> > > xen: start PCI hole at 0xe0000000 (same as pc_init1 and
> > > qemu-xen-traditional)
> > > author      Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> > >             Wed, 5 Jun 2013 19:36:10 +0800 (11:36 +0000)
> > > committer   Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> > >             Wed, 5 Jun 2013 19:36:10 +0800 (11:36 +0000)
> > > commit     4597594c61add43725bd207bb498268a058f9cfb
> > > tree        d6831f75f4a7d4ad7a94bd4e33584ac358808ee6
> > > parent      25adf763933faddcc6a62bf55e1c52909a9bafbb
> > > 
> > > Can you debug this issue soon? Thanks.
> > 
> > Thank you very much for testing and bisecting the issue!
> > 
> > The problem is that by default hvmloader sets PCI_MEM_START to
> > 0xf0000000, then dynamically expands it backwards if needed.
> > It works with qemu-xen-traditional, because it doesn't do any checking
> > when registering memory regions.
> > It doesn't work with qemu-xen because it needs to know where the ram and
> > where the pci hole are when building the emulated machine.
> > We can't have hvmloader increasing the pci hole in a way that overlaps
> > with the guest ram (or where qemu thinks that the guest ram is).
> > It is also worth noting thet seabios has its own view of where the pci
> > hole starts and at the moment is set to 0xe0000000 at build time.
> > 
> > I can see two ways of fixing this now:
> > 
> > 1) have everybody agree about the pci hole starting at 0xe0000000
> > 
> > - set PCI_MEM_START in hvmloader to 0xe0000000 to match qemu's view of
> > the pci hole and have a bigger pci hole by default
> > 
> > - remove the loop in hvmloader to increase the pci hole
> > 
> > - set HVM_BELOW_4G_RAM_END to 0xe0000000, so that low_mem_pgend is set
> > accordingly in tools/libxc/xc_hvm_build_x86.c:build_hvm_info
> > 
> > 
> > 2) have everybody agree about the pci hole starting at 0xf0000000
> > 
> > - revert 4597594c61add43725bd207bb498268a058f9cfb in qemu-xen
> > 
> > - set BUILD_PCIMEM_START to 0xf0000000 in seabios
> > 
> > - remove the loop in hvmloader to increase the pci hole
> > 
> > 
> > Given that in both cases we need to remove the loop to increase the pci
> > hole in hvmloader I would rather go with 1) and have a bigger pci hole
> > by default to avoid problems with the pci hole being too small.
> > 
> 
> Did we ever figure out what actually happens on real hardware?  Or on KVM?  I
> have a hard time believing that real hardware is hard-coded -- who would be
> the person to ask about that, do you reckon?

For KVM I am CC'ing Paolo, for real hardware I'll let the Intel guys
speak.
To be clear the question is:

What happens on real hardware when the BIOS boots and finds out that the
PCI hole is too small to contain all the MMIO regions of the PCI devices?