* Re: [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
@ 2013-06-13 13:54 ` George Dunlap
0 siblings, 0 replies; 82+ messages in thread
From: George Dunlap @ 2013-06-13 13:54 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Tim Deegan, Yongjie Ren, xen-devel, Keir Fraser, Ian Campbell,
hanweidong, Xudong Hao, yanqiangjun, luonengjun, qemu-devel,
wangzhenguo, xiaowei.yang, arei.gonglei, Jan Beulich,
Paolo Bonzini, YongweiX Xu, SongtaoX Liu
On 13/06/13 14:44, Stefano Stabellini wrote:
> On Wed, 12 Jun 2013, George Dunlap wrote:
>> On 12/06/13 08:25, Jan Beulich wrote:
>>>>>> On 11.06.13 at 19:26, Stefano Stabellini
>>>>>> <stefano.stabellini@eu.citrix.com> wrote:
>>>> I went through the code that maps the PCI MMIO regions in hvmloader
>>>> (tools/firmware/hvmloader/pci.c:pci_setup) and it looks like it already
>>>> maps the PCI region to high memory if the PCI bar is 64-bit and the MMIO
>>>> region is larger than 512MB.
>>>>
>>>> Maybe we could just relax this condition and map the device memory to
>>>> high memory no matter the size of the MMIO region if the PCI bar is
>>>> 64-bit?
>>> I can only recommend not to: For one, guests not using PAE or
>>> PSE-36 can't map such space at all (and older OSes may not
>>> properly deal with 64-bit BARs at all). And then one would generally
>>> expect this allocation to be done top down (to minimize risk of
>>> running into RAM), and doing so is going to present further risks of
>>> incompatibilities with guest OSes (Linux for example learned only in
>>> 2.6.36 that PFNs in ioremap() can exceed 32 bits, but even in
>>> 3.10-rc5 ioremap_pte_range(), while using "u64 pfn", passes the
>>> PFN to pfn_pte(), the respective parameter of which is
>>> "unsigned long").
>>>
>>> I think this ought to be done in an iterative process - if all MMIO
>>> regions together don't fit below 4G, the biggest one should be
>>> moved up beyond 4G first, followed by the next to biggest one
>>> etc.
>> First of all, the proposal to move the PCI BAR up to the 64-bit range is a
>> temporary work-around. It should only be done if a device doesn't fit in the
>> current MMIO range.
>>
>> We have three options here:
>> 1. Don't do anything
>> 2. Have hvmloader move PCI devices up to the 64-bit MMIO hole if they don't
>> fit
>> 3. Convince qemu to allow MMIO regions to mask memory (or what it thinks is
>> memory).
>> 4. Add a mechanism to tell qemu that memory is being relocated.
>>
>> Number 4 is definitely the right answer long-term, but we just don't have time
>> to do that before the 4.3 release. We're not sure yet if #3 is possible; even
>> if it is, it may have unpredictable knock-on effects.
>>
>> Doing #2, it is true that many guests will be unable to access the device
>> because of 32-bit limitations. However, in #1, *no* guests will be able to
>> access the device. At least in #2, *many* guests will be able to do so. In
>> any case, apparently #2 is what KVM does, so having the limitation on guests
>> is not without precedent. It's also likely to be a somewhat tested
>> configuration (unlike #3, for example).
> I would avoid #3, because I don't think is a good idea to rely on that
> behaviour.
> I would also avoid #4, because having seen QEMU's code, it's wouldn't be
> easy and certainly not doable in time for 4.3.
>
> So we are left to play with the PCI MMIO region size and location in
> hvmloader.
>
> I agree with Jan that we shouldn't relocate unconditionally all the
> devices to the region above 4G. I meant to say that we should relocate
> only the ones that don't fit. And we shouldn't try to dynamically
> increase the PCI hole below 4G because clearly that doesn't work.
> However we could still increase the size of the PCI hole below 4G by
> default from start at 0xf0000000 to starting at 0xe0000000.
> Why do we know that is safe? Because in the current configuration
> hvmloader *already* increases the PCI hole size by decreasing the start
> address every time a device doesn't fit.
> So it's already common for hvmloader to set pci_mem_start to
> 0xe0000000, you just need to assign a device with a PCI hole size big
> enough.
>
>
> My proposed solution is:
>
> - set 0xe0000000 as the default PCI hole start for everybody, including
> qemu-xen-traditional
> - move above 4G everything that doesn't fit and support 64-bit bars
> - print an error if the device doesn't fit and doesn't support 64-bit
> bars
Also, as I understand it, at the moment:
1. Some operating systems (32-bit XP) won't be able to use relocated devices
2. Some devices (without 64-bit BARs) can't be relocated
3. qemu-traditional is fine with a resized <4GiB MMIO hole.
So if we have #1 or #2, at the moment an option for a work-around is to
use qemu-traditional.
However, if we add your "print an error if the device doesn't fit", then
this option will go away -- this will be a regression in functionality
from 4.2.
I thought that what we had proposed was to have an option in xenstore,
that libxl would set, which would instruct hvmloader whether to expand
the MMIO hole and whether to relocate devices above 64-bit?
-George
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [Qemu-devel] [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
2013-06-13 13:54 ` George Dunlap
@ 2013-06-13 14:50 ` Stefano Stabellini
-1 siblings, 0 replies; 82+ messages in thread
From: Stefano Stabellini @ 2013-06-13 14:50 UTC (permalink / raw)
To: George Dunlap
Cc: Tim Deegan, Yongjie Ren, yanqiangjun, Keir Fraser, Ian Campbell,
hanweidong, Xudong Hao, Stefano Stabellini, luonengjun,
qemu-devel, wangzhenguo, xiaowei.yang, arei.gonglei, Jan Beulich,
Paolo Bonzini, YongweiX Xu, SongtaoX Liu, xen-devel
On Thu, 13 Jun 2013, George Dunlap wrote:
> On 13/06/13 14:44, Stefano Stabellini wrote:
> > On Wed, 12 Jun 2013, George Dunlap wrote:
> > > On 12/06/13 08:25, Jan Beulich wrote:
> > > > > > > On 11.06.13 at 19:26, Stefano Stabellini
> > > > > > > <stefano.stabellini@eu.citrix.com> wrote:
> > > > > I went through the code that maps the PCI MMIO regions in hvmloader
> > > > > (tools/firmware/hvmloader/pci.c:pci_setup) and it looks like it
> > > > > already
> > > > > maps the PCI region to high memory if the PCI bar is 64-bit and the
> > > > > MMIO
> > > > > region is larger than 512MB.
> > > > >
> > > > > Maybe we could just relax this condition and map the device memory to
> > > > > high memory no matter the size of the MMIO region if the PCI bar is
> > > > > 64-bit?
> > > > I can only recommend not to: For one, guests not using PAE or
> > > > PSE-36 can't map such space at all (and older OSes may not
> > > > properly deal with 64-bit BARs at all). And then one would generally
> > > > expect this allocation to be done top down (to minimize risk of
> > > > running into RAM), and doing so is going to present further risks of
> > > > incompatibilities with guest OSes (Linux for example learned only in
> > > > 2.6.36 that PFNs in ioremap() can exceed 32 bits, but even in
> > > > 3.10-rc5 ioremap_pte_range(), while using "u64 pfn", passes the
> > > > PFN to pfn_pte(), the respective parameter of which is
> > > > "unsigned long").
> > > >
> > > > I think this ought to be done in an iterative process - if all MMIO
> > > > regions together don't fit below 4G, the biggest one should be
> > > > moved up beyond 4G first, followed by the next to biggest one
> > > > etc.
> > > First of all, the proposal to move the PCI BAR up to the 64-bit range is a
> > > temporary work-around. It should only be done if a device doesn't fit in
> > > the
> > > current MMIO range.
> > >
> > > We have three options here:
> > > 1. Don't do anything
> > > 2. Have hvmloader move PCI devices up to the 64-bit MMIO hole if they
> > > don't
> > > fit
> > > 3. Convince qemu to allow MMIO regions to mask memory (or what it thinks
> > > is
> > > memory).
> > > 4. Add a mechanism to tell qemu that memory is being relocated.
> > >
> > > Number 4 is definitely the right answer long-term, but we just don't have
> > > time
> > > to do that before the 4.3 release. We're not sure yet if #3 is possible;
> > > even
> > > if it is, it may have unpredictable knock-on effects.
> > >
> > > Doing #2, it is true that many guests will be unable to access the device
> > > because of 32-bit limitations. However, in #1, *no* guests will be able
> > > to
> > > access the device. At least in #2, *many* guests will be able to do so.
> > > In
> > > any case, apparently #2 is what KVM does, so having the limitation on
> > > guests
> > > is not without precedent. It's also likely to be a somewhat tested
> > > configuration (unlike #3, for example).
> > I would avoid #3, because I don't think is a good idea to rely on that
> > behaviour.
> > I would also avoid #4, because having seen QEMU's code, it's wouldn't be
> > easy and certainly not doable in time for 4.3.
> >
> > So we are left to play with the PCI MMIO region size and location in
> > hvmloader.
> >
> > I agree with Jan that we shouldn't relocate unconditionally all the
> > devices to the region above 4G. I meant to say that we should relocate
> > only the ones that don't fit. And we shouldn't try to dynamically
> > increase the PCI hole below 4G because clearly that doesn't work.
> > However we could still increase the size of the PCI hole below 4G by
> > default from start at 0xf0000000 to starting at 0xe0000000.
> > Why do we know that is safe? Because in the current configuration
> > hvmloader *already* increases the PCI hole size by decreasing the start
> > address every time a device doesn't fit.
> > So it's already common for hvmloader to set pci_mem_start to
> > 0xe0000000, you just need to assign a device with a PCI hole size big
> > enough.
> >
> >
> > My proposed solution is:
> >
> > - set 0xe0000000 as the default PCI hole start for everybody, including
> > qemu-xen-traditional
> > - move above 4G everything that doesn't fit and support 64-bit bars
> > - print an error if the device doesn't fit and doesn't support 64-bit
> > bars
>
> Also, as I understand it, at the moment:
> 1. Some operating systems (32-bit XP) won't be able to use relocated devices
> 2. Some devices (without 64-bit BARs) can't be relocated
> 3. qemu-traditional is fine with a resized <4GiB MMIO hole.
>
> So if we have #1 or #2, at the moment an option for a work-around is to use
> qemu-traditional.
>
> However, if we add your "print an error if the device doesn't fit", then this
> option will go away -- this will be a regression in functionality from 4.2.
Keep in mind that if we start the pci hole at 0xe0000000, the number of
cases for which any workarounds are needed is going to be dramatically
decreased to the point that I don't think we need a workaround anymore.
The algorithm is going to work like this in details:
- the pci hole size is set to 0xfc000000-0xe0000000 = 448MB
- we calculate the total mmio size, if it's bigger than the pci hole we
raise a 64 bit relocation flag
- if the 64 bit relocation is enabled, we relocate above 4G the first
device that is 64-bit capable and has an MMIO size greater or equal to
512MB
- if the pci hole size is now big enough for the remaining devices we
stop the above 4G relocation, otherwise keep relocating devices that are
64 bit capable and have an MMIO size greater or equal to 512MB
- if one or more devices don't fit we print an error and continue (it's
not a critical failure, one device won't be used)
We could have a xenstore flag somewhere that enables the old behaviour
so that people can revert back to qemu-xen-traditional and make the pci
hole below 4G even bigger than 448MB, but I think that keeping the old
behaviour around is going to make the code more difficult to maintain.
Also it's difficult for people to realize that they need the workaround
because hvmloader logs aren't enabled by default and only go to the Xen
serial console. The value of this workaround pretty low in my view.
Finally it's worth noting that Windows XP is going EOL in less than an
year.
> I thought that what we had proposed was to have an option in xenstore, that
> libxl would set, which would instruct hvmloader whether to expand the MMIO
> hole and whether to relocate devices above 64-bit?
I think it's right to have this discussion in public on the mailing
list, rather than behind closed doors.
Also I don't agree on the need for a workaround, as explained above.
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [BUG 1747]Guest could't find bootable device with memory more than 3600M
@ 2013-06-13 14:50 ` Stefano Stabellini
0 siblings, 0 replies; 82+ messages in thread
From: Stefano Stabellini @ 2013-06-13 14:50 UTC (permalink / raw)
To: George Dunlap
Cc: Tim Deegan, Yongjie Ren, yanqiangjun, Keir Fraser, Ian Campbell,
hanweidong, Xudong Hao, Stefano Stabellini, luonengjun,
qemu-devel, wangzhenguo, xiaowei.yang, arei.gonglei, Jan Beulich,
Paolo Bonzini, YongweiX Xu, SongtaoX Liu, xen-devel
On Thu, 13 Jun 2013, George Dunlap wrote:
> On 13/06/13 14:44, Stefano Stabellini wrote:
> > On Wed, 12 Jun 2013, George Dunlap wrote:
> > > On 12/06/13 08:25, Jan Beulich wrote:
> > > > > > > On 11.06.13 at 19:26, Stefano Stabellini
> > > > > > > <stefano.stabellini@eu.citrix.com> wrote:
> > > > > I went through the code that maps the PCI MMIO regions in hvmloader
> > > > > (tools/firmware/hvmloader/pci.c:pci_setup) and it looks like it
> > > > > already
> > > > > maps the PCI region to high memory if the PCI bar is 64-bit and the
> > > > > MMIO
> > > > > region is larger than 512MB.
> > > > >
> > > > > Maybe we could just relax this condition and map the device memory to
> > > > > high memory no matter the size of the MMIO region if the PCI bar is
> > > > > 64-bit?
> > > > I can only recommend not to: For one, guests not using PAE or
> > > > PSE-36 can't map such space at all (and older OSes may not
> > > > properly deal with 64-bit BARs at all). And then one would generally
> > > > expect this allocation to be done top down (to minimize risk of
> > > > running into RAM), and doing so is going to present further risks of
> > > > incompatibilities with guest OSes (Linux for example learned only in
> > > > 2.6.36 that PFNs in ioremap() can exceed 32 bits, but even in
> > > > 3.10-rc5 ioremap_pte_range(), while using "u64 pfn", passes the
> > > > PFN to pfn_pte(), the respective parameter of which is
> > > > "unsigned long").
> > > >
> > > > I think this ought to be done in an iterative process - if all MMIO
> > > > regions together don't fit below 4G, the biggest one should be
> > > > moved up beyond 4G first, followed by the next to biggest one
> > > > etc.
> > > First of all, the proposal to move the PCI BAR up to the 64-bit range is a
> > > temporary work-around. It should only be done if a device doesn't fit in
> > > the
> > > current MMIO range.
> > >
> > > We have three options here:
> > > 1. Don't do anything
> > > 2. Have hvmloader move PCI devices up to the 64-bit MMIO hole if they
> > > don't
> > > fit
> > > 3. Convince qemu to allow MMIO regions to mask memory (or what it thinks
> > > is
> > > memory).
> > > 4. Add a mechanism to tell qemu that memory is being relocated.
> > >
> > > Number 4 is definitely the right answer long-term, but we just don't have
> > > time
> > > to do that before the 4.3 release. We're not sure yet if #3 is possible;
> > > even
> > > if it is, it may have unpredictable knock-on effects.
> > >
> > > Doing #2, it is true that many guests will be unable to access the device
> > > because of 32-bit limitations. However, in #1, *no* guests will be able
> > > to
> > > access the device. At least in #2, *many* guests will be able to do so.
> > > In
> > > any case, apparently #2 is what KVM does, so having the limitation on
> > > guests
> > > is not without precedent. It's also likely to be a somewhat tested
> > > configuration (unlike #3, for example).
> > I would avoid #3, because I don't think is a good idea to rely on that
> > behaviour.
> > I would also avoid #4, because having seen QEMU's code, it's wouldn't be
> > easy and certainly not doable in time for 4.3.
> >
> > So we are left to play with the PCI MMIO region size and location in
> > hvmloader.
> >
> > I agree with Jan that we shouldn't relocate unconditionally all the
> > devices to the region above 4G. I meant to say that we should relocate
> > only the ones that don't fit. And we shouldn't try to dynamically
> > increase the PCI hole below 4G because clearly that doesn't work.
> > However we could still increase the size of the PCI hole below 4G by
> > default from start at 0xf0000000 to starting at 0xe0000000.
> > Why do we know that is safe? Because in the current configuration
> > hvmloader *already* increases the PCI hole size by decreasing the start
> > address every time a device doesn't fit.
> > So it's already common for hvmloader to set pci_mem_start to
> > 0xe0000000, you just need to assign a device with a PCI hole size big
> > enough.
> >
> >
> > My proposed solution is:
> >
> > - set 0xe0000000 as the default PCI hole start for everybody, including
> > qemu-xen-traditional
> > - move above 4G everything that doesn't fit and support 64-bit bars
> > - print an error if the device doesn't fit and doesn't support 64-bit
> > bars
>
> Also, as I understand it, at the moment:
> 1. Some operating systems (32-bit XP) won't be able to use relocated devices
> 2. Some devices (without 64-bit BARs) can't be relocated
> 3. qemu-traditional is fine with a resized <4GiB MMIO hole.
>
> So if we have #1 or #2, at the moment an option for a work-around is to use
> qemu-traditional.
>
> However, if we add your "print an error if the device doesn't fit", then this
> option will go away -- this will be a regression in functionality from 4.2.
Keep in mind that if we start the pci hole at 0xe0000000, the number of
cases for which any workarounds are needed is going to be dramatically
decreased to the point that I don't think we need a workaround anymore.
The algorithm is going to work like this in details:
- the pci hole size is set to 0xfc000000-0xe0000000 = 448MB
- we calculate the total mmio size, if it's bigger than the pci hole we
raise a 64 bit relocation flag
- if the 64 bit relocation is enabled, we relocate above 4G the first
device that is 64-bit capable and has an MMIO size greater or equal to
512MB
- if the pci hole size is now big enough for the remaining devices we
stop the above 4G relocation, otherwise keep relocating devices that are
64 bit capable and have an MMIO size greater or equal to 512MB
- if one or more devices don't fit we print an error and continue (it's
not a critical failure, one device won't be used)
We could have a xenstore flag somewhere that enables the old behaviour
so that people can revert back to qemu-xen-traditional and make the pci
hole below 4G even bigger than 448MB, but I think that keeping the old
behaviour around is going to make the code more difficult to maintain.
Also it's difficult for people to realize that they need the workaround
because hvmloader logs aren't enabled by default and only go to the Xen
serial console. The value of this workaround pretty low in my view.
Finally it's worth noting that Windows XP is going EOL in less than an
year.
> I thought that what we had proposed was to have an option in xenstore, that
> libxl would set, which would instruct hvmloader whether to expand the MMIO
> hole and whether to relocate devices above 64-bit?
I think it's right to have this discussion in public on the mailing
list, rather than behind closed doors.
Also I don't agree on the need for a workaround, as explained above.
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [Qemu-devel] [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
2013-06-13 14:50 ` Stefano Stabellini
@ 2013-06-13 15:06 ` Jan Beulich
-1 siblings, 0 replies; 82+ messages in thread
From: Jan Beulich @ 2013-06-13 15:06 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Tim Deegan, Yongjie Ren, xen-devel, Keir Fraser, Ian Campbell,
hanweidong, George Dunlap, Xudong Hao, yanqiangjun, luonengjun,
qemu-devel, wangzhenguo, xiaowei.yang, arei.gonglei,
Paolo Bonzini, YongweiX Xu, SongtaoX Liu
>>> On 13.06.13 at 16:50, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:
> The algorithm is going to work like this in details:
>
> - the pci hole size is set to 0xfc000000-0xe0000000 = 448MB
> - we calculate the total mmio size, if it's bigger than the pci hole we
> raise a 64 bit relocation flag
> - if the 64 bit relocation is enabled, we relocate above 4G the first
> device that is 64-bit capable and has an MMIO size greater or equal to
> 512MB
> - if the pci hole size is now big enough for the remaining devices we
> stop the above 4G relocation, otherwise keep relocating devices that are
> 64 bit capable and have an MMIO size greater or equal to 512MB
> - if one or more devices don't fit we print an error and continue (it's
> not a critical failure, one device won't be used)
Devices with 512Mb BARs won't fit in a 448Mb hole in any case,
so there's no point in trying. Any such BARs need to be relocated.
Then for 256Mb BARs, you could see whether it's just one and fits.
Else relocate it, and all others (except for perhaps one). Then
halve the size again and start over, etc.
Jan
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [BUG 1747]Guest could't find bootable device with memory more than 3600M
@ 2013-06-13 15:06 ` Jan Beulich
0 siblings, 0 replies; 82+ messages in thread
From: Jan Beulich @ 2013-06-13 15:06 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Tim Deegan, Yongjie Ren, xen-devel, Keir Fraser, Ian Campbell,
hanweidong, George Dunlap, Xudong Hao, yanqiangjun, luonengjun,
qemu-devel, wangzhenguo, xiaowei.yang, arei.gonglei,
Paolo Bonzini, YongweiX Xu, SongtaoX Liu
>>> On 13.06.13 at 16:50, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:
> The algorithm is going to work like this in details:
>
> - the pci hole size is set to 0xfc000000-0xe0000000 = 448MB
> - we calculate the total mmio size, if it's bigger than the pci hole we
> raise a 64 bit relocation flag
> - if the 64 bit relocation is enabled, we relocate above 4G the first
> device that is 64-bit capable and has an MMIO size greater or equal to
> 512MB
> - if the pci hole size is now big enough for the remaining devices we
> stop the above 4G relocation, otherwise keep relocating devices that are
> 64 bit capable and have an MMIO size greater or equal to 512MB
> - if one or more devices don't fit we print an error and continue (it's
> not a critical failure, one device won't be used)
Devices with 512Mb BARs won't fit in a 448Mb hole in any case,
so there's no point in trying. Any such BARs need to be relocated.
Then for 256Mb BARs, you could see whether it's just one and fits.
Else relocate it, and all others (except for perhaps one). Then
halve the size again and start over, etc.
Jan
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [Qemu-devel] [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
2013-06-13 14:50 ` Stefano Stabellini
@ 2013-06-13 15:29 ` George Dunlap
-1 siblings, 0 replies; 82+ messages in thread
From: George Dunlap @ 2013-06-13 15:29 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Tim Deegan, Yongjie Ren, xen-devel, Keir Fraser, Ian Campbell,
hanweidong, Xudong Hao, yanqiangjun, luonengjun, qemu-devel,
wangzhenguo, xiaowei.yang, arei.gonglei, Jan Beulich,
Paolo Bonzini, YongweiX Xu, SongtaoX Liu
On 13/06/13 15:50, Stefano Stabellini wrote:
> Keep in mind that if we start the pci hole at 0xe0000000, the number of
> cases for which any workarounds are needed is going to be dramatically
> decreased to the point that I don't think we need a workaround anymore.
You don't think anyone is going to want to pass through a card with
1GiB+ of RAM?
>
> The algorithm is going to work like this in details:
>
> - the pci hole size is set to 0xfc000000-0xe0000000 = 448MB
> - we calculate the total mmio size, if it's bigger than the pci hole we
> raise a 64 bit relocation flag
> - if the 64 bit relocation is enabled, we relocate above 4G the first
> device that is 64-bit capable and has an MMIO size greater or equal to
> 512MB
> - if the pci hole size is now big enough for the remaining devices we
> stop the above 4G relocation, otherwise keep relocating devices that are
> 64 bit capable and have an MMIO size greater or equal to 512MB
> - if one or more devices don't fit we print an error and continue (it's
> not a critical failure, one device won't be used)
>
> We could have a xenstore flag somewhere that enables the old behaviour
> so that people can revert back to qemu-xen-traditional and make the pci
> hole below 4G even bigger than 448MB, but I think that keeping the old
> behaviour around is going to make the code more difficult to maintain.
We'll only need to do that for one release, until we have a chance to
fix it properly.
>
> Also it's difficult for people to realize that they need the workaround
> because hvmloader logs aren't enabled by default and only go to the Xen
> serial console.
Well if key people know about it (Pasi, David Techer, &c), and we put it
on the wikis related to VGA pass-through, I think information will get
around.
> The value of this workaround pretty low in my view.
> Finally it's worth noting that Windows XP is going EOL in less than an
> year.
That's 1 year that a configuration with a currently-supported OS won't
work for Xen 4.3 that worked for 4.2. Apart from that, one of the
reasons for doing virtualization in the first place is to be able to run
older, unsupported OSes on current hardware; so "XP isn't important"
doesn't really cut it for me. :-)
>
>
>> I thought that what we had proposed was to have an option in xenstore, that
>> libxl would set, which would instruct hvmloader whether to expand the MMIO
>> hole and whether to relocate devices above 64-bit?
> I think it's right to have this discussion in public on the mailing
> list, rather than behind closed doors.
> Also I don't agree on the need for a workaround, as explained above.
I see -- you thought it was a bad idea and so were letting someone else
bring it up -- or maybe hoping no one would remember to bring it up. :-)
(Obviously the decision needs to be made in public, but sometimes having
technical solutions hashed out in a face-to-face meeting is more efficient.)
-George
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
@ 2013-06-13 15:29 ` George Dunlap
0 siblings, 0 replies; 82+ messages in thread
From: George Dunlap @ 2013-06-13 15:29 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Tim Deegan, Yongjie Ren, xen-devel, Keir Fraser, Ian Campbell,
hanweidong, Xudong Hao, yanqiangjun, luonengjun, qemu-devel,
wangzhenguo, xiaowei.yang, arei.gonglei, Jan Beulich,
Paolo Bonzini, YongweiX Xu, SongtaoX Liu
On 13/06/13 15:50, Stefano Stabellini wrote:
> Keep in mind that if we start the pci hole at 0xe0000000, the number of
> cases for which any workarounds are needed is going to be dramatically
> decreased to the point that I don't think we need a workaround anymore.
You don't think anyone is going to want to pass through a card with
1GiB+ of RAM?
>
> The algorithm is going to work like this in details:
>
> - the pci hole size is set to 0xfc000000-0xe0000000 = 448MB
> - we calculate the total mmio size, if it's bigger than the pci hole we
> raise a 64 bit relocation flag
> - if the 64 bit relocation is enabled, we relocate above 4G the first
> device that is 64-bit capable and has an MMIO size greater or equal to
> 512MB
> - if the pci hole size is now big enough for the remaining devices we
> stop the above 4G relocation, otherwise keep relocating devices that are
> 64 bit capable and have an MMIO size greater or equal to 512MB
> - if one or more devices don't fit we print an error and continue (it's
> not a critical failure, one device won't be used)
>
> We could have a xenstore flag somewhere that enables the old behaviour
> so that people can revert back to qemu-xen-traditional and make the pci
> hole below 4G even bigger than 448MB, but I think that keeping the old
> behaviour around is going to make the code more difficult to maintain.
We'll only need to do that for one release, until we have a chance to
fix it properly.
>
> Also it's difficult for people to realize that they need the workaround
> because hvmloader logs aren't enabled by default and only go to the Xen
> serial console.
Well if key people know about it (Pasi, David Techer, &c), and we put it
on the wikis related to VGA pass-through, I think information will get
around.
> The value of this workaround pretty low in my view.
> Finally it's worth noting that Windows XP is going EOL in less than an
> year.
That's 1 year that a configuration with a currently-supported OS won't
work for Xen 4.3 that worked for 4.2. Apart from that, one of the
reasons for doing virtualization in the first place is to be able to run
older, unsupported OSes on current hardware; so "XP isn't important"
doesn't really cut it for me. :-)
>
>
>> I thought that what we had proposed was to have an option in xenstore, that
>> libxl would set, which would instruct hvmloader whether to expand the MMIO
>> hole and whether to relocate devices above 64-bit?
> I think it's right to have this discussion in public on the mailing
> list, rather than behind closed doors.
> Also I don't agree on the need for a workaround, as explained above.
I see -- you thought it was a bad idea and so were letting someone else
bring it up -- or maybe hoping no one would remember to bring it up. :-)
(Obviously the decision needs to be made in public, but sometimes having
technical solutions hashed out in a face-to-face meeting is more efficient.)
-George
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [Qemu-devel] [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
2013-06-13 15:29 ` George Dunlap
@ 2013-06-13 16:13 ` Stefano Stabellini
-1 siblings, 0 replies; 82+ messages in thread
From: Stefano Stabellini @ 2013-06-13 16:13 UTC (permalink / raw)
To: George Dunlap
Cc: Tim Deegan, Yongjie Ren, yanqiangjun, Keir Fraser, Ian Campbell,
hanweidong, Xudong Hao, Stefano Stabellini, luonengjun,
qemu-devel, wangzhenguo, xiaowei.yang, arei.gonglei, Jan Beulich,
Paolo Bonzini, YongweiX Xu, SongtaoX Liu, xen-devel
On Thu, 13 Jun 2013, George Dunlap wrote:
> On 13/06/13 15:50, Stefano Stabellini wrote:
> > Keep in mind that if we start the pci hole at 0xe0000000, the number of
> > cases for which any workarounds are needed is going to be dramatically
> > decreased to the point that I don't think we need a workaround anymore.
>
> You don't think anyone is going to want to pass through a card with 1GiB+ of
> RAM?
Yes, but as Paolo pointed out, those devices are going to be 64-bit
capable so they'll relocate above 4G just fine.
> > The algorithm is going to work like this in details:
> >
> > - the pci hole size is set to 0xfc000000-0xe0000000 = 448MB
> > - we calculate the total mmio size, if it's bigger than the pci hole we
> > raise a 64 bit relocation flag
> > - if the 64 bit relocation is enabled, we relocate above 4G the first
> > device that is 64-bit capable and has an MMIO size greater or equal to
> > 512MB
> > - if the pci hole size is now big enough for the remaining devices we
> > stop the above 4G relocation, otherwise keep relocating devices that are
> > 64 bit capable and have an MMIO size greater or equal to 512MB
> > - if one or more devices don't fit we print an error and continue (it's
> > not a critical failure, one device won't be used)
> >
> > We could have a xenstore flag somewhere that enables the old behaviour
> > so that people can revert back to qemu-xen-traditional and make the pci
> > hole below 4G even bigger than 448MB, but I think that keeping the old
> > behaviour around is going to make the code more difficult to maintain.
>
> We'll only need to do that for one release, until we have a chance to fix it
> properly.
There is nothing more lasting than a "temporary" workaround :-)
Also it's not very clear what the proper solution would be like in this
case.
However keeping the old behaviour is certainly possible. It would just
be a bit harder to also keep the old (smaller) default pci hole around.
> > Also it's difficult for people to realize that they need the workaround
> > because hvmloader logs aren't enabled by default and only go to the Xen
> > serial console.
>
> Well if key people know about it (Pasi, David Techer, &c), and we put it on
> the wikis related to VGA pass-through, I think information will get around.
It's not that I don't value documentation, but given that the average
user won't see any logs and the error is completely non-informative,
many people are going to be lost in a wild goose chase on google.
> > The value of this workaround pretty low in my view.
> > Finally it's worth noting that Windows XP is going EOL in less than an
> > year.
>
> That's 1 year that a configuration with a currently-supported OS won't work
> for Xen 4.3 that worked for 4.2. Apart from that, one of the reasons for
> doing virtualization in the first place is to be able to run older,
> unsupported OSes on current hardware; so "XP isn't important" doesn't really
> cut it for me. :-)
fair enough
> > > I thought that what we had proposed was to have an option in xenstore,
> > > that
> > > libxl would set, which would instruct hvmloader whether to expand the MMIO
> > > hole and whether to relocate devices above 64-bit?
> > I think it's right to have this discussion in public on the mailing
> > list, rather than behind closed doors.
> > Also I don't agree on the need for a workaround, as explained above.
>
> I see -- you thought it was a bad idea and so were letting someone else bring
> it up -- or maybe hoping no one would remember to bring it up. :-)
Nothing that Macchiavellian: I didn't consider all the implications at
the time and I thought I managed to come up with a better plan.
> (Obviously the decision needs to be made in public, but sometimes having
> technical solutions hashed out in a face-to-face meeting is more efficient.)
But it's also easier to overlook something, at least it is easier for
me.
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
@ 2013-06-13 16:13 ` Stefano Stabellini
0 siblings, 0 replies; 82+ messages in thread
From: Stefano Stabellini @ 2013-06-13 16:13 UTC (permalink / raw)
To: George Dunlap
Cc: Tim Deegan, Yongjie Ren, yanqiangjun, Keir Fraser, Ian Campbell,
hanweidong, Xudong Hao, Stefano Stabellini, luonengjun,
qemu-devel, wangzhenguo, xiaowei.yang, arei.gonglei, Jan Beulich,
Paolo Bonzini, YongweiX Xu, SongtaoX Liu, xen-devel
On Thu, 13 Jun 2013, George Dunlap wrote:
> On 13/06/13 15:50, Stefano Stabellini wrote:
> > Keep in mind that if we start the pci hole at 0xe0000000, the number of
> > cases for which any workarounds are needed is going to be dramatically
> > decreased to the point that I don't think we need a workaround anymore.
>
> You don't think anyone is going to want to pass through a card with 1GiB+ of
> RAM?
Yes, but as Paolo pointed out, those devices are going to be 64-bit
capable so they'll relocate above 4G just fine.
> > The algorithm is going to work like this in details:
> >
> > - the pci hole size is set to 0xfc000000-0xe0000000 = 448MB
> > - we calculate the total mmio size, if it's bigger than the pci hole we
> > raise a 64 bit relocation flag
> > - if the 64 bit relocation is enabled, we relocate above 4G the first
> > device that is 64-bit capable and has an MMIO size greater or equal to
> > 512MB
> > - if the pci hole size is now big enough for the remaining devices we
> > stop the above 4G relocation, otherwise keep relocating devices that are
> > 64 bit capable and have an MMIO size greater or equal to 512MB
> > - if one or more devices don't fit we print an error and continue (it's
> > not a critical failure, one device won't be used)
> >
> > We could have a xenstore flag somewhere that enables the old behaviour
> > so that people can revert back to qemu-xen-traditional and make the pci
> > hole below 4G even bigger than 448MB, but I think that keeping the old
> > behaviour around is going to make the code more difficult to maintain.
>
> We'll only need to do that for one release, until we have a chance to fix it
> properly.
There is nothing more lasting than a "temporary" workaround :-)
Also it's not very clear what the proper solution would be like in this
case.
However keeping the old behaviour is certainly possible. It would just
be a bit harder to also keep the old (smaller) default pci hole around.
> > Also it's difficult for people to realize that they need the workaround
> > because hvmloader logs aren't enabled by default and only go to the Xen
> > serial console.
>
> Well if key people know about it (Pasi, David Techer, &c), and we put it on
> the wikis related to VGA pass-through, I think information will get around.
It's not that I don't value documentation, but given that the average
user won't see any logs and the error is completely non-informative,
many people are going to be lost in a wild goose chase on google.
> > The value of this workaround pretty low in my view.
> > Finally it's worth noting that Windows XP is going EOL in less than an
> > year.
>
> That's 1 year that a configuration with a currently-supported OS won't work
> for Xen 4.3 that worked for 4.2. Apart from that, one of the reasons for
> doing virtualization in the first place is to be able to run older,
> unsupported OSes on current hardware; so "XP isn't important" doesn't really
> cut it for me. :-)
fair enough
> > > I thought that what we had proposed was to have an option in xenstore,
> > > that
> > > libxl would set, which would instruct hvmloader whether to expand the MMIO
> > > hole and whether to relocate devices above 64-bit?
> > I think it's right to have this discussion in public on the mailing
> > list, rather than behind closed doors.
> > Also I don't agree on the need for a workaround, as explained above.
>
> I see -- you thought it was a bad idea and so were letting someone else bring
> it up -- or maybe hoping no one would remember to bring it up. :-)
Nothing that Macchiavellian: I didn't consider all the implications at
the time and I thought I managed to come up with a better plan.
> (Obviously the decision needs to be made in public, but sometimes having
> technical solutions hashed out in a face-to-face meeting is more efficient.)
But it's also easier to overlook something, at least it is easier for
me.
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [Qemu-devel] [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
2013-06-13 14:50 ` Stefano Stabellini
@ 2013-06-13 15:34 ` Ian Campbell
-1 siblings, 0 replies; 82+ messages in thread
From: Ian Campbell @ 2013-06-13 15:34 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Tim Deegan, Yongjie Ren, xen-devel, Keir Fraser, hanweidong,
George Dunlap, Xudong Hao, yanqiangjun, luonengjun, qemu-devel,
wangzhenguo, xiaowei.yang, arei.gonglei, Jan Beulich,
Paolo Bonzini, YongweiX Xu, SongtaoX Liu
On Thu, 2013-06-13 at 15:50 +0100, Stefano Stabellini wrote:
> On Thu, 13 Jun 2013, George Dunlap wrote:
> > On 13/06/13 14:44, Stefano Stabellini wrote:
> > > On Wed, 12 Jun 2013, George Dunlap wrote:
> > > > On 12/06/13 08:25, Jan Beulich wrote:
> > > > > > > > On 11.06.13 at 19:26, Stefano Stabellini
> > > > > > > > <stefano.stabellini@eu.citrix.com> wrote:
> > > > > > I went through the code that maps the PCI MMIO regions in hvmloader
> > > > > > (tools/firmware/hvmloader/pci.c:pci_setup) and it looks like it
> > > > > > already
> > > > > > maps the PCI region to high memory if the PCI bar is 64-bit and the
> > > > > > MMIO
> > > > > > region is larger than 512MB.
> > > > > >
> > > > > > Maybe we could just relax this condition and map the device memory to
> > > > > > high memory no matter the size of the MMIO region if the PCI bar is
> > > > > > 64-bit?
> > > > > I can only recommend not to: For one, guests not using PAE or
> > > > > PSE-36 can't map such space at all (and older OSes may not
> > > > > properly deal with 64-bit BARs at all). And then one would generally
> > > > > expect this allocation to be done top down (to minimize risk of
> > > > > running into RAM), and doing so is going to present further risks of
> > > > > incompatibilities with guest OSes (Linux for example learned only in
> > > > > 2.6.36 that PFNs in ioremap() can exceed 32 bits, but even in
> > > > > 3.10-rc5 ioremap_pte_range(), while using "u64 pfn", passes the
> > > > > PFN to pfn_pte(), the respective parameter of which is
> > > > > "unsigned long").
> > > > >
> > > > > I think this ought to be done in an iterative process - if all MMIO
> > > > > regions together don't fit below 4G, the biggest one should be
> > > > > moved up beyond 4G first, followed by the next to biggest one
> > > > > etc.
> > > > First of all, the proposal to move the PCI BAR up to the 64-bit range is a
> > > > temporary work-around. It should only be done if a device doesn't fit in
> > > > the
> > > > current MMIO range.
> > > >
> > > > We have three options here:
> > > > 1. Don't do anything
> > > > 2. Have hvmloader move PCI devices up to the 64-bit MMIO hole if they
> > > > don't
> > > > fit
> > > > 3. Convince qemu to allow MMIO regions to mask memory (or what it thinks
> > > > is
> > > > memory).
> > > > 4. Add a mechanism to tell qemu that memory is being relocated.
> > > >
> > > > Number 4 is definitely the right answer long-term, but we just don't have
> > > > time
> > > > to do that before the 4.3 release. We're not sure yet if #3 is possible;
> > > > even
> > > > if it is, it may have unpredictable knock-on effects.
> > > >
> > > > Doing #2, it is true that many guests will be unable to access the device
> > > > because of 32-bit limitations. However, in #1, *no* guests will be able
> > > > to
> > > > access the device. At least in #2, *many* guests will be able to do so.
> > > > In
> > > > any case, apparently #2 is what KVM does, so having the limitation on
> > > > guests
> > > > is not without precedent. It's also likely to be a somewhat tested
> > > > configuration (unlike #3, for example).
> > > I would avoid #3, because I don't think is a good idea to rely on that
> > > behaviour.
> > > I would also avoid #4, because having seen QEMU's code, it's wouldn't be
> > > easy and certainly not doable in time for 4.3.
> > >
> > > So we are left to play with the PCI MMIO region size and location in
> > > hvmloader.
> > >
> > > I agree with Jan that we shouldn't relocate unconditionally all the
> > > devices to the region above 4G. I meant to say that we should relocate
> > > only the ones that don't fit. And we shouldn't try to dynamically
> > > increase the PCI hole below 4G because clearly that doesn't work.
> > > However we could still increase the size of the PCI hole below 4G by
> > > default from start at 0xf0000000 to starting at 0xe0000000.
> > > Why do we know that is safe? Because in the current configuration
> > > hvmloader *already* increases the PCI hole size by decreasing the start
> > > address every time a device doesn't fit.
> > > So it's already common for hvmloader to set pci_mem_start to
> > > 0xe0000000, you just need to assign a device with a PCI hole size big
> > > enough.
> > >
> > >
> > > My proposed solution is:
> > >
> > > - set 0xe0000000 as the default PCI hole start for everybody, including
> > > qemu-xen-traditional
> > > - move above 4G everything that doesn't fit and support 64-bit bars
> > > - print an error if the device doesn't fit and doesn't support 64-bit
> > > bars
> >
> > Also, as I understand it, at the moment:
> > 1. Some operating systems (32-bit XP) won't be able to use relocated devices
> > 2. Some devices (without 64-bit BARs) can't be relocated
> > 3. qemu-traditional is fine with a resized <4GiB MMIO hole.
> >
> > So if we have #1 or #2, at the moment an option for a work-around is to use
> > qemu-traditional.
> >
> > However, if we add your "print an error if the device doesn't fit", then this
> > option will go away -- this will be a regression in functionality from 4.2.
>
> Keep in mind that if we start the pci hole at 0xe0000000, the number of
> cases for which any workarounds are needed is going to be dramatically
> decreased to the point that I don't think we need a workaround anymore.
Starting at 0xe0000000 leaves, as you say a 448MB whole, with graphics
cards regularly having 512M+ of RAM on them that suggests the workaround
will be required in many cases.
> The algorithm is going to work like this in details:
>
> - the pci hole size is set to 0xfc000000-0xe0000000 = 448MB
> - we calculate the total mmio size, if it's bigger than the pci hole we
> raise a 64 bit relocation flag
> - if the 64 bit relocation is enabled, we relocate above 4G the first
> device that is 64-bit capable and has an MMIO size greater or equal to
> 512MB
Don't you mean the device with the largest MMIO size? Otherwise 2 256MB
devices would still break things.
Paulo's comment about large alignments first is also worth considering.
> - if the pci hole size is now big enough for the remaining devices we
> stop the above 4G relocation, otherwise keep relocating devices that are
> 64 bit capable and have an MMIO size greater or equal to 512MB
> - if one or more devices don't fit we print an error and continue (it's
> not a critical failure, one device won't be used)
This can result in a different device being broken to the one which
would previously have been broken, including on qemu-trad I think?
> We could have a xenstore flag somewhere that enables the old behaviour
> so that people can revert back to qemu-xen-traditional and make the pci
> hole below 4G even bigger than 448MB, but I think that keeping the old
> behaviour around is going to make the code more difficult to maintain.
The downside of that is that things which worked with the old scheme may
not work with the new one though. Early in a release cycle when we have
time to discover what has broken then that might be OK, but is post rc4
really the time to be risking it?
> Also it's difficult for people to realize that they need the workaround
> because hvmloader logs aren't enabled by default and only go to the Xen
> serial console. The value of this workaround pretty low in my view.
> Finally it's worth noting that Windows XP is going EOL in less than an
> year.
That's been true for something like 5 years...
Also, apart from XP, doesn't Windows still pick a HAL at install time,
so even a modern guest installed under the old scheme may not get a PAE
capable HAL. If you increase the amount of RAM I think Windows will
"upgrade" the HAL, but is changing the MMIO layout enough to trigger
this? Or maybe modern Windows all use PAE (or even 64 bit) anyway?
There are also performance implications of enabling PAE over 2 level
paging. Not sure how significant they are with HAP though. Made a big
difference with shadow IIRC.
Maybe I'm worrying about nothing but while all of these unknowns might
be OK towards the start of a release cycle rc4 seems awfully late in the
day to be risking it.
Ian.
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
@ 2013-06-13 15:34 ` Ian Campbell
0 siblings, 0 replies; 82+ messages in thread
From: Ian Campbell @ 2013-06-13 15:34 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Tim Deegan, Yongjie Ren, xen-devel, Keir Fraser, hanweidong,
George Dunlap, Xudong Hao, yanqiangjun, luonengjun, qemu-devel,
wangzhenguo, xiaowei.yang, arei.gonglei, Jan Beulich,
Paolo Bonzini, YongweiX Xu, SongtaoX Liu
On Thu, 2013-06-13 at 15:50 +0100, Stefano Stabellini wrote:
> On Thu, 13 Jun 2013, George Dunlap wrote:
> > On 13/06/13 14:44, Stefano Stabellini wrote:
> > > On Wed, 12 Jun 2013, George Dunlap wrote:
> > > > On 12/06/13 08:25, Jan Beulich wrote:
> > > > > > > > On 11.06.13 at 19:26, Stefano Stabellini
> > > > > > > > <stefano.stabellini@eu.citrix.com> wrote:
> > > > > > I went through the code that maps the PCI MMIO regions in hvmloader
> > > > > > (tools/firmware/hvmloader/pci.c:pci_setup) and it looks like it
> > > > > > already
> > > > > > maps the PCI region to high memory if the PCI bar is 64-bit and the
> > > > > > MMIO
> > > > > > region is larger than 512MB.
> > > > > >
> > > > > > Maybe we could just relax this condition and map the device memory to
> > > > > > high memory no matter the size of the MMIO region if the PCI bar is
> > > > > > 64-bit?
> > > > > I can only recommend not to: For one, guests not using PAE or
> > > > > PSE-36 can't map such space at all (and older OSes may not
> > > > > properly deal with 64-bit BARs at all). And then one would generally
> > > > > expect this allocation to be done top down (to minimize risk of
> > > > > running into RAM), and doing so is going to present further risks of
> > > > > incompatibilities with guest OSes (Linux for example learned only in
> > > > > 2.6.36 that PFNs in ioremap() can exceed 32 bits, but even in
> > > > > 3.10-rc5 ioremap_pte_range(), while using "u64 pfn", passes the
> > > > > PFN to pfn_pte(), the respective parameter of which is
> > > > > "unsigned long").
> > > > >
> > > > > I think this ought to be done in an iterative process - if all MMIO
> > > > > regions together don't fit below 4G, the biggest one should be
> > > > > moved up beyond 4G first, followed by the next to biggest one
> > > > > etc.
> > > > First of all, the proposal to move the PCI BAR up to the 64-bit range is a
> > > > temporary work-around. It should only be done if a device doesn't fit in
> > > > the
> > > > current MMIO range.
> > > >
> > > > We have three options here:
> > > > 1. Don't do anything
> > > > 2. Have hvmloader move PCI devices up to the 64-bit MMIO hole if they
> > > > don't
> > > > fit
> > > > 3. Convince qemu to allow MMIO regions to mask memory (or what it thinks
> > > > is
> > > > memory).
> > > > 4. Add a mechanism to tell qemu that memory is being relocated.
> > > >
> > > > Number 4 is definitely the right answer long-term, but we just don't have
> > > > time
> > > > to do that before the 4.3 release. We're not sure yet if #3 is possible;
> > > > even
> > > > if it is, it may have unpredictable knock-on effects.
> > > >
> > > > Doing #2, it is true that many guests will be unable to access the device
> > > > because of 32-bit limitations. However, in #1, *no* guests will be able
> > > > to
> > > > access the device. At least in #2, *many* guests will be able to do so.
> > > > In
> > > > any case, apparently #2 is what KVM does, so having the limitation on
> > > > guests
> > > > is not without precedent. It's also likely to be a somewhat tested
> > > > configuration (unlike #3, for example).
> > > I would avoid #3, because I don't think is a good idea to rely on that
> > > behaviour.
> > > I would also avoid #4, because having seen QEMU's code, it's wouldn't be
> > > easy and certainly not doable in time for 4.3.
> > >
> > > So we are left to play with the PCI MMIO region size and location in
> > > hvmloader.
> > >
> > > I agree with Jan that we shouldn't relocate unconditionally all the
> > > devices to the region above 4G. I meant to say that we should relocate
> > > only the ones that don't fit. And we shouldn't try to dynamically
> > > increase the PCI hole below 4G because clearly that doesn't work.
> > > However we could still increase the size of the PCI hole below 4G by
> > > default from start at 0xf0000000 to starting at 0xe0000000.
> > > Why do we know that is safe? Because in the current configuration
> > > hvmloader *already* increases the PCI hole size by decreasing the start
> > > address every time a device doesn't fit.
> > > So it's already common for hvmloader to set pci_mem_start to
> > > 0xe0000000, you just need to assign a device with a PCI hole size big
> > > enough.
> > >
> > >
> > > My proposed solution is:
> > >
> > > - set 0xe0000000 as the default PCI hole start for everybody, including
> > > qemu-xen-traditional
> > > - move above 4G everything that doesn't fit and support 64-bit bars
> > > - print an error if the device doesn't fit and doesn't support 64-bit
> > > bars
> >
> > Also, as I understand it, at the moment:
> > 1. Some operating systems (32-bit XP) won't be able to use relocated devices
> > 2. Some devices (without 64-bit BARs) can't be relocated
> > 3. qemu-traditional is fine with a resized <4GiB MMIO hole.
> >
> > So if we have #1 or #2, at the moment an option for a work-around is to use
> > qemu-traditional.
> >
> > However, if we add your "print an error if the device doesn't fit", then this
> > option will go away -- this will be a regression in functionality from 4.2.
>
> Keep in mind that if we start the pci hole at 0xe0000000, the number of
> cases for which any workarounds are needed is going to be dramatically
> decreased to the point that I don't think we need a workaround anymore.
Starting at 0xe0000000 leaves, as you say a 448MB whole, with graphics
cards regularly having 512M+ of RAM on them that suggests the workaround
will be required in many cases.
> The algorithm is going to work like this in details:
>
> - the pci hole size is set to 0xfc000000-0xe0000000 = 448MB
> - we calculate the total mmio size, if it's bigger than the pci hole we
> raise a 64 bit relocation flag
> - if the 64 bit relocation is enabled, we relocate above 4G the first
> device that is 64-bit capable and has an MMIO size greater or equal to
> 512MB
Don't you mean the device with the largest MMIO size? Otherwise 2 256MB
devices would still break things.
Paulo's comment about large alignments first is also worth considering.
> - if the pci hole size is now big enough for the remaining devices we
> stop the above 4G relocation, otherwise keep relocating devices that are
> 64 bit capable and have an MMIO size greater or equal to 512MB
> - if one or more devices don't fit we print an error and continue (it's
> not a critical failure, one device won't be used)
This can result in a different device being broken to the one which
would previously have been broken, including on qemu-trad I think?
> We could have a xenstore flag somewhere that enables the old behaviour
> so that people can revert back to qemu-xen-traditional and make the pci
> hole below 4G even bigger than 448MB, but I think that keeping the old
> behaviour around is going to make the code more difficult to maintain.
The downside of that is that things which worked with the old scheme may
not work with the new one though. Early in a release cycle when we have
time to discover what has broken then that might be OK, but is post rc4
really the time to be risking it?
> Also it's difficult for people to realize that they need the workaround
> because hvmloader logs aren't enabled by default and only go to the Xen
> serial console. The value of this workaround pretty low in my view.
> Finally it's worth noting that Windows XP is going EOL in less than an
> year.
That's been true for something like 5 years...
Also, apart from XP, doesn't Windows still pick a HAL at install time,
so even a modern guest installed under the old scheme may not get a PAE
capable HAL. If you increase the amount of RAM I think Windows will
"upgrade" the HAL, but is changing the MMIO layout enough to trigger
this? Or maybe modern Windows all use PAE (or even 64 bit) anyway?
There are also performance implications of enabling PAE over 2 level
paging. Not sure how significant they are with HAP though. Made a big
difference with shadow IIRC.
Maybe I'm worrying about nothing but while all of these unknowns might
be OK towards the start of a release cycle rc4 seems awfully late in the
day to be risking it.
Ian.
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [Qemu-devel] [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
2013-06-13 15:34 ` Ian Campbell
@ 2013-06-13 16:55 ` Stefano Stabellini
-1 siblings, 0 replies; 82+ messages in thread
From: Stefano Stabellini @ 2013-06-13 16:55 UTC (permalink / raw)
To: Ian Campbell
Cc: Tim Deegan, Yongjie Ren, yanqiangjun, Keir Fraser, hanweidong,
George Dunlap, Xudong Hao, Stefano Stabellini, luonengjun,
qemu-devel, wangzhenguo, xiaowei.yang, arei.gonglei, Jan Beulich,
Paolo Bonzini, YongweiX Xu, SongtaoX Liu, xen-devel
On Thu, 13 Jun 2013, Ian Campbell wrote:
> On Thu, 2013-06-13 at 15:50 +0100, Stefano Stabellini wrote:
> > On Thu, 13 Jun 2013, George Dunlap wrote:
> > > On 13/06/13 14:44, Stefano Stabellini wrote:
> > > > On Wed, 12 Jun 2013, George Dunlap wrote:
> > > > > On 12/06/13 08:25, Jan Beulich wrote:
> > > > > > > > > On 11.06.13 at 19:26, Stefano Stabellini
> > > > > > > > > <stefano.stabellini@eu.citrix.com> wrote:
> > > > > > > I went through the code that maps the PCI MMIO regions in hvmloader
> > > > > > > (tools/firmware/hvmloader/pci.c:pci_setup) and it looks like it
> > > > > > > already
> > > > > > > maps the PCI region to high memory if the PCI bar is 64-bit and the
> > > > > > > MMIO
> > > > > > > region is larger than 512MB.
> > > > > > >
> > > > > > > Maybe we could just relax this condition and map the device memory to
> > > > > > > high memory no matter the size of the MMIO region if the PCI bar is
> > > > > > > 64-bit?
> > > > > > I can only recommend not to: For one, guests not using PAE or
> > > > > > PSE-36 can't map such space at all (and older OSes may not
> > > > > > properly deal with 64-bit BARs at all). And then one would generally
> > > > > > expect this allocation to be done top down (to minimize risk of
> > > > > > running into RAM), and doing so is going to present further risks of
> > > > > > incompatibilities with guest OSes (Linux for example learned only in
> > > > > > 2.6.36 that PFNs in ioremap() can exceed 32 bits, but even in
> > > > > > 3.10-rc5 ioremap_pte_range(), while using "u64 pfn", passes the
> > > > > > PFN to pfn_pte(), the respective parameter of which is
> > > > > > "unsigned long").
> > > > > >
> > > > > > I think this ought to be done in an iterative process - if all MMIO
> > > > > > regions together don't fit below 4G, the biggest one should be
> > > > > > moved up beyond 4G first, followed by the next to biggest one
> > > > > > etc.
> > > > > First of all, the proposal to move the PCI BAR up to the 64-bit range is a
> > > > > temporary work-around. It should only be done if a device doesn't fit in
> > > > > the
> > > > > current MMIO range.
> > > > >
> > > > > We have three options here:
> > > > > 1. Don't do anything
> > > > > 2. Have hvmloader move PCI devices up to the 64-bit MMIO hole if they
> > > > > don't
> > > > > fit
> > > > > 3. Convince qemu to allow MMIO regions to mask memory (or what it thinks
> > > > > is
> > > > > memory).
> > > > > 4. Add a mechanism to tell qemu that memory is being relocated.
> > > > >
> > > > > Number 4 is definitely the right answer long-term, but we just don't have
> > > > > time
> > > > > to do that before the 4.3 release. We're not sure yet if #3 is possible;
> > > > > even
> > > > > if it is, it may have unpredictable knock-on effects.
> > > > >
> > > > > Doing #2, it is true that many guests will be unable to access the device
> > > > > because of 32-bit limitations. However, in #1, *no* guests will be able
> > > > > to
> > > > > access the device. At least in #2, *many* guests will be able to do so.
> > > > > In
> > > > > any case, apparently #2 is what KVM does, so having the limitation on
> > > > > guests
> > > > > is not without precedent. It's also likely to be a somewhat tested
> > > > > configuration (unlike #3, for example).
> > > > I would avoid #3, because I don't think is a good idea to rely on that
> > > > behaviour.
> > > > I would also avoid #4, because having seen QEMU's code, it's wouldn't be
> > > > easy and certainly not doable in time for 4.3.
> > > >
> > > > So we are left to play with the PCI MMIO region size and location in
> > > > hvmloader.
> > > >
> > > > I agree with Jan that we shouldn't relocate unconditionally all the
> > > > devices to the region above 4G. I meant to say that we should relocate
> > > > only the ones that don't fit. And we shouldn't try to dynamically
> > > > increase the PCI hole below 4G because clearly that doesn't work.
> > > > However we could still increase the size of the PCI hole below 4G by
> > > > default from start at 0xf0000000 to starting at 0xe0000000.
> > > > Why do we know that is safe? Because in the current configuration
> > > > hvmloader *already* increases the PCI hole size by decreasing the start
> > > > address every time a device doesn't fit.
> > > > So it's already common for hvmloader to set pci_mem_start to
> > > > 0xe0000000, you just need to assign a device with a PCI hole size big
> > > > enough.
> > > >
> > > >
> > > > My proposed solution is:
> > > >
> > > > - set 0xe0000000 as the default PCI hole start for everybody, including
> > > > qemu-xen-traditional
> > > > - move above 4G everything that doesn't fit and support 64-bit bars
> > > > - print an error if the device doesn't fit and doesn't support 64-bit
> > > > bars
> > >
> > > Also, as I understand it, at the moment:
> > > 1. Some operating systems (32-bit XP) won't be able to use relocated devices
> > > 2. Some devices (without 64-bit BARs) can't be relocated
> > > 3. qemu-traditional is fine with a resized <4GiB MMIO hole.
> > >
> > > So if we have #1 or #2, at the moment an option for a work-around is to use
> > > qemu-traditional.
> > >
> > > However, if we add your "print an error if the device doesn't fit", then this
> > > option will go away -- this will be a regression in functionality from 4.2.
> >
> > Keep in mind that if we start the pci hole at 0xe0000000, the number of
> > cases for which any workarounds are needed is going to be dramatically
> > decreased to the point that I don't think we need a workaround anymore.
>
> Starting at 0xe0000000 leaves, as you say a 448MB whole, with graphics
> cards regularly having 512M+ of RAM on them that suggests the workaround
> will be required in many cases.
http://www.nvidia.co.uk/object/graphics_cards_buy_now_uk.html
Actually more than half of the graphic cards sold today have >= 2G of
videoram so they wouldn't fit below 4G even in the old scheme that gives
at most 2G of PCI hole below 4G.
So the resulting configurations would be the same: the devices would be
located above 4G.
> > The algorithm is going to work like this in details:
> >
> > - the pci hole size is set to 0xfc000000-0xe0000000 = 448MB
> > - we calculate the total mmio size, if it's bigger than the pci hole we
> > raise a 64 bit relocation flag
> > - if the 64 bit relocation is enabled, we relocate above 4G the first
> > device that is 64-bit capable and has an MMIO size greater or equal to
> > 512MB
>
> Don't you mean the device with the largest MMIO size? Otherwise 2 256MB
> devices would still break things.
You are right that would be much better.
It's worth mentioning that the problem you have just identified exists
even in the current scheme. In fact you could reach a non-configurable
state by passing through 4 graphic cards with 512MB of video ram each.
> Paulo's comment about large alignments first is also worth considering.
>
> > - if the pci hole size is now big enough for the remaining devices we
> > stop the above 4G relocation, otherwise keep relocating devices that are
> > 64 bit capable and have an MMIO size greater or equal to 512MB
> > - if one or more devices don't fit we print an error and continue (it's
> > not a critical failure, one device won't be used)
>
> This can result in a different device being broken to the one which
> would previously have been broken, including on qemu-trad I think?
Previously the guest would fail to boot with qemu-xen. On
qemu-xen-traditional the configuration would be completely different:
fewer devices would be relocated above 4G and the pci hole would be
bigger. So you are right the devices being broken would be different.
> > We could have a xenstore flag somewhere that enables the old behaviour
> > so that people can revert back to qemu-xen-traditional and make the pci
> > hole below 4G even bigger than 448MB, but I think that keeping the old
> > behaviour around is going to make the code more difficult to maintain.
>
> The downside of that is that things which worked with the old scheme may
> not work with the new one though. Early in a release cycle when we have
> time to discover what has broken then that might be OK, but is post rc4
> really the time to be risking it?
Yes, you are right: there are some scenarios that would have worked
before that wouldn't work anymore with the new scheme.
Are they important enough to have a workaround, pretty difficult to
identify for a user?
> > Also it's difficult for people to realize that they need the workaround
> > because hvmloader logs aren't enabled by default and only go to the Xen
> > serial console. The value of this workaround pretty low in my view.
> > Finally it's worth noting that Windows XP is going EOL in less than an
> > year.
>
> That's been true for something like 5 years...
>
> Also, apart from XP, doesn't Windows still pick a HAL at install time,
> so even a modern guest installed under the old scheme may not get a PAE
> capable HAL. If you increase the amount of RAM I think Windows will
> "upgrade" the HAL, but is changing the MMIO layout enough to trigger
> this? Or maybe modern Windows all use PAE (or even 64 bit) anyway?
>
> There are also performance implications of enabling PAE over 2 level
> paging. Not sure how significant they are with HAP though. Made a big
> difference with shadow IIRC.
>
> Maybe I'm worrying about nothing but while all of these unknowns might
> be OK towards the start of a release cycle rc4 seems awfully late in the
> day to be risking it.
Keep in mind that all these configurations are perfectly valid even with
the code that we have out there today. We aren't doing anything new,
just modifying the default.
One just needs to assign a PCI device with more than 190MB to trigger it.
I am trusting the fact that given that we had this behaviour for many
years now, and it's pretty common to assign a device only some of the
times you are booting your guest, any problems would have already come
up.
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
@ 2013-06-13 16:55 ` Stefano Stabellini
0 siblings, 0 replies; 82+ messages in thread
From: Stefano Stabellini @ 2013-06-13 16:55 UTC (permalink / raw)
To: Ian Campbell
Cc: Tim Deegan, Yongjie Ren, yanqiangjun, Keir Fraser, hanweidong,
George Dunlap, Xudong Hao, Stefano Stabellini, luonengjun,
qemu-devel, wangzhenguo, xiaowei.yang, arei.gonglei, Jan Beulich,
Paolo Bonzini, YongweiX Xu, SongtaoX Liu, xen-devel
On Thu, 13 Jun 2013, Ian Campbell wrote:
> On Thu, 2013-06-13 at 15:50 +0100, Stefano Stabellini wrote:
> > On Thu, 13 Jun 2013, George Dunlap wrote:
> > > On 13/06/13 14:44, Stefano Stabellini wrote:
> > > > On Wed, 12 Jun 2013, George Dunlap wrote:
> > > > > On 12/06/13 08:25, Jan Beulich wrote:
> > > > > > > > > On 11.06.13 at 19:26, Stefano Stabellini
> > > > > > > > > <stefano.stabellini@eu.citrix.com> wrote:
> > > > > > > I went through the code that maps the PCI MMIO regions in hvmloader
> > > > > > > (tools/firmware/hvmloader/pci.c:pci_setup) and it looks like it
> > > > > > > already
> > > > > > > maps the PCI region to high memory if the PCI bar is 64-bit and the
> > > > > > > MMIO
> > > > > > > region is larger than 512MB.
> > > > > > >
> > > > > > > Maybe we could just relax this condition and map the device memory to
> > > > > > > high memory no matter the size of the MMIO region if the PCI bar is
> > > > > > > 64-bit?
> > > > > > I can only recommend not to: For one, guests not using PAE or
> > > > > > PSE-36 can't map such space at all (and older OSes may not
> > > > > > properly deal with 64-bit BARs at all). And then one would generally
> > > > > > expect this allocation to be done top down (to minimize risk of
> > > > > > running into RAM), and doing so is going to present further risks of
> > > > > > incompatibilities with guest OSes (Linux for example learned only in
> > > > > > 2.6.36 that PFNs in ioremap() can exceed 32 bits, but even in
> > > > > > 3.10-rc5 ioremap_pte_range(), while using "u64 pfn", passes the
> > > > > > PFN to pfn_pte(), the respective parameter of which is
> > > > > > "unsigned long").
> > > > > >
> > > > > > I think this ought to be done in an iterative process - if all MMIO
> > > > > > regions together don't fit below 4G, the biggest one should be
> > > > > > moved up beyond 4G first, followed by the next to biggest one
> > > > > > etc.
> > > > > First of all, the proposal to move the PCI BAR up to the 64-bit range is a
> > > > > temporary work-around. It should only be done if a device doesn't fit in
> > > > > the
> > > > > current MMIO range.
> > > > >
> > > > > We have three options here:
> > > > > 1. Don't do anything
> > > > > 2. Have hvmloader move PCI devices up to the 64-bit MMIO hole if they
> > > > > don't
> > > > > fit
> > > > > 3. Convince qemu to allow MMIO regions to mask memory (or what it thinks
> > > > > is
> > > > > memory).
> > > > > 4. Add a mechanism to tell qemu that memory is being relocated.
> > > > >
> > > > > Number 4 is definitely the right answer long-term, but we just don't have
> > > > > time
> > > > > to do that before the 4.3 release. We're not sure yet if #3 is possible;
> > > > > even
> > > > > if it is, it may have unpredictable knock-on effects.
> > > > >
> > > > > Doing #2, it is true that many guests will be unable to access the device
> > > > > because of 32-bit limitations. However, in #1, *no* guests will be able
> > > > > to
> > > > > access the device. At least in #2, *many* guests will be able to do so.
> > > > > In
> > > > > any case, apparently #2 is what KVM does, so having the limitation on
> > > > > guests
> > > > > is not without precedent. It's also likely to be a somewhat tested
> > > > > configuration (unlike #3, for example).
> > > > I would avoid #3, because I don't think is a good idea to rely on that
> > > > behaviour.
> > > > I would also avoid #4, because having seen QEMU's code, it's wouldn't be
> > > > easy and certainly not doable in time for 4.3.
> > > >
> > > > So we are left to play with the PCI MMIO region size and location in
> > > > hvmloader.
> > > >
> > > > I agree with Jan that we shouldn't relocate unconditionally all the
> > > > devices to the region above 4G. I meant to say that we should relocate
> > > > only the ones that don't fit. And we shouldn't try to dynamically
> > > > increase the PCI hole below 4G because clearly that doesn't work.
> > > > However we could still increase the size of the PCI hole below 4G by
> > > > default from start at 0xf0000000 to starting at 0xe0000000.
> > > > Why do we know that is safe? Because in the current configuration
> > > > hvmloader *already* increases the PCI hole size by decreasing the start
> > > > address every time a device doesn't fit.
> > > > So it's already common for hvmloader to set pci_mem_start to
> > > > 0xe0000000, you just need to assign a device with a PCI hole size big
> > > > enough.
> > > >
> > > >
> > > > My proposed solution is:
> > > >
> > > > - set 0xe0000000 as the default PCI hole start for everybody, including
> > > > qemu-xen-traditional
> > > > - move above 4G everything that doesn't fit and support 64-bit bars
> > > > - print an error if the device doesn't fit and doesn't support 64-bit
> > > > bars
> > >
> > > Also, as I understand it, at the moment:
> > > 1. Some operating systems (32-bit XP) won't be able to use relocated devices
> > > 2. Some devices (without 64-bit BARs) can't be relocated
> > > 3. qemu-traditional is fine with a resized <4GiB MMIO hole.
> > >
> > > So if we have #1 or #2, at the moment an option for a work-around is to use
> > > qemu-traditional.
> > >
> > > However, if we add your "print an error if the device doesn't fit", then this
> > > option will go away -- this will be a regression in functionality from 4.2.
> >
> > Keep in mind that if we start the pci hole at 0xe0000000, the number of
> > cases for which any workarounds are needed is going to be dramatically
> > decreased to the point that I don't think we need a workaround anymore.
>
> Starting at 0xe0000000 leaves, as you say a 448MB whole, with graphics
> cards regularly having 512M+ of RAM on them that suggests the workaround
> will be required in many cases.
http://www.nvidia.co.uk/object/graphics_cards_buy_now_uk.html
Actually more than half of the graphic cards sold today have >= 2G of
videoram so they wouldn't fit below 4G even in the old scheme that gives
at most 2G of PCI hole below 4G.
So the resulting configurations would be the same: the devices would be
located above 4G.
> > The algorithm is going to work like this in details:
> >
> > - the pci hole size is set to 0xfc000000-0xe0000000 = 448MB
> > - we calculate the total mmio size, if it's bigger than the pci hole we
> > raise a 64 bit relocation flag
> > - if the 64 bit relocation is enabled, we relocate above 4G the first
> > device that is 64-bit capable and has an MMIO size greater or equal to
> > 512MB
>
> Don't you mean the device with the largest MMIO size? Otherwise 2 256MB
> devices would still break things.
You are right that would be much better.
It's worth mentioning that the problem you have just identified exists
even in the current scheme. In fact you could reach a non-configurable
state by passing through 4 graphic cards with 512MB of video ram each.
> Paulo's comment about large alignments first is also worth considering.
>
> > - if the pci hole size is now big enough for the remaining devices we
> > stop the above 4G relocation, otherwise keep relocating devices that are
> > 64 bit capable and have an MMIO size greater or equal to 512MB
> > - if one or more devices don't fit we print an error and continue (it's
> > not a critical failure, one device won't be used)
>
> This can result in a different device being broken to the one which
> would previously have been broken, including on qemu-trad I think?
Previously the guest would fail to boot with qemu-xen. On
qemu-xen-traditional the configuration would be completely different:
fewer devices would be relocated above 4G and the pci hole would be
bigger. So you are right the devices being broken would be different.
> > We could have a xenstore flag somewhere that enables the old behaviour
> > so that people can revert back to qemu-xen-traditional and make the pci
> > hole below 4G even bigger than 448MB, but I think that keeping the old
> > behaviour around is going to make the code more difficult to maintain.
>
> The downside of that is that things which worked with the old scheme may
> not work with the new one though. Early in a release cycle when we have
> time to discover what has broken then that might be OK, but is post rc4
> really the time to be risking it?
Yes, you are right: there are some scenarios that would have worked
before that wouldn't work anymore with the new scheme.
Are they important enough to have a workaround, pretty difficult to
identify for a user?
> > Also it's difficult for people to realize that they need the workaround
> > because hvmloader logs aren't enabled by default and only go to the Xen
> > serial console. The value of this workaround pretty low in my view.
> > Finally it's worth noting that Windows XP is going EOL in less than an
> > year.
>
> That's been true for something like 5 years...
>
> Also, apart from XP, doesn't Windows still pick a HAL at install time,
> so even a modern guest installed under the old scheme may not get a PAE
> capable HAL. If you increase the amount of RAM I think Windows will
> "upgrade" the HAL, but is changing the MMIO layout enough to trigger
> this? Or maybe modern Windows all use PAE (or even 64 bit) anyway?
>
> There are also performance implications of enabling PAE over 2 level
> paging. Not sure how significant they are with HAP though. Made a big
> difference with shadow IIRC.
>
> Maybe I'm worrying about nothing but while all of these unknowns might
> be OK towards the start of a release cycle rc4 seems awfully late in the
> day to be risking it.
Keep in mind that all these configurations are perfectly valid even with
the code that we have out there today. We aren't doing anything new,
just modifying the default.
One just needs to assign a PCI device with more than 190MB to trigger it.
I am trusting the fact that given that we had this behaviour for many
years now, and it's pretty common to assign a device only some of the
times you are booting your guest, any problems would have already come
up.
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [Qemu-devel] [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
2013-06-13 16:55 ` Stefano Stabellini
@ 2013-06-13 17:22 ` Ian Campbell
-1 siblings, 0 replies; 82+ messages in thread
From: Ian Campbell @ 2013-06-13 17:22 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Tim Deegan, Yongjie Ren, xen-devel, Keir Fraser, hanweidong,
George Dunlap, Xudong Hao, yanqiangjun, luonengjun, qemu-devel,
wangzhenguo, xiaowei.yang, arei.gonglei, Jan Beulich,
Paolo Bonzini, YongweiX Xu, SongtaoX Liu
On Thu, 2013-06-13 at 17:55 +0100, Stefano Stabellini wrote:
> > > We could have a xenstore flag somewhere that enables the old behaviour
> > > so that people can revert back to qemu-xen-traditional and make the pci
> > > hole below 4G even bigger than 448MB, but I think that keeping the old
> > > behaviour around is going to make the code more difficult to maintain.
> >
> > The downside of that is that things which worked with the old scheme may
> > not work with the new one though. Early in a release cycle when we have
> > time to discover what has broken then that might be OK, but is post rc4
> > really the time to be risking it?
>
> Yes, you are right: there are some scenarios that would have worked
> before that wouldn't work anymore with the new scheme.
> Are they important enough to have a workaround, pretty difficult to
> identify for a user?
That question would be reasonable early in the development cycle. At rc4
the question should be: do we think this problem is so critical that we
want to risk breaking something else which currently works for people.
Remember that we are invalidating whatever passthrough testing people
have already done up to this point of the release.
It is also worth noting that the things which this change ends up
breaking may for all we know be equally difficult for a user to identify
(they are after all approximately the same class of issue).
The problem here is that the risk is difficult to evaluate, we just
don't know what will break with this change, and we don't know therefore
if the cure is worse than the disease. The conservative approach at this
point in the release would be to not change anything, or to change the
minimal possible number of things (which would preclude changes which
impact qemu-trad IMHO).
WRT pretty difficult to identify -- the root of this thread suggests the
guest entered a reboot loop with "No bootable device", that sounds
eminently release notable to me. I also not that it was changing the
size of the PCI hole which caused the issue -- which does somewhat
underscore the risks involved in this sort of change.
> > > Also it's difficult for people to realize that they need the workaround
> > > because hvmloader logs aren't enabled by default and only go to the Xen
> > > serial console. The value of this workaround pretty low in my view.
> > > Finally it's worth noting that Windows XP is going EOL in less than an
> > > year.
> >
> > That's been true for something like 5 years...
> >
> > Also, apart from XP, doesn't Windows still pick a HAL at install time,
> > so even a modern guest installed under the old scheme may not get a PAE
> > capable HAL. If you increase the amount of RAM I think Windows will
> > "upgrade" the HAL, but is changing the MMIO layout enough to trigger
> > this? Or maybe modern Windows all use PAE (or even 64 bit) anyway?
> >
> > There are also performance implications of enabling PAE over 2 level
> > paging. Not sure how significant they are with HAP though. Made a big
> > difference with shadow IIRC.
> >
> > Maybe I'm worrying about nothing but while all of these unknowns might
> > be OK towards the start of a release cycle rc4 seems awfully late in the
> > day to be risking it.
>
> Keep in mind that all these configurations are perfectly valid even with
> the code that we have out there today. We aren't doing anything new,
> just modifying the default.
I don't think that is true. We are changing the behaviour, calling it
"just" a default doesn't make it any less worrying or any less of a
change.
> One just needs to assign a PCI device with more than 190MB to trigger it.
> I am trusting the fact that given that we had this behaviour for many
> years now, and it's pretty common to assign a device only some of the
> times you are booting your guest, any problems would have already come
> up.
With qemu-trad perhaps, although that's not completely obvious TBH. In
any case should we really be crossing our fingers and "trusting" that
it'll be ok at rc4?
Ian.
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
@ 2013-06-13 17:22 ` Ian Campbell
0 siblings, 0 replies; 82+ messages in thread
From: Ian Campbell @ 2013-06-13 17:22 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Tim Deegan, Yongjie Ren, xen-devel, Keir Fraser, hanweidong,
George Dunlap, Xudong Hao, yanqiangjun, luonengjun, qemu-devel,
wangzhenguo, xiaowei.yang, arei.gonglei, Jan Beulich,
Paolo Bonzini, YongweiX Xu, SongtaoX Liu
On Thu, 2013-06-13 at 17:55 +0100, Stefano Stabellini wrote:
> > > We could have a xenstore flag somewhere that enables the old behaviour
> > > so that people can revert back to qemu-xen-traditional and make the pci
> > > hole below 4G even bigger than 448MB, but I think that keeping the old
> > > behaviour around is going to make the code more difficult to maintain.
> >
> > The downside of that is that things which worked with the old scheme may
> > not work with the new one though. Early in a release cycle when we have
> > time to discover what has broken then that might be OK, but is post rc4
> > really the time to be risking it?
>
> Yes, you are right: there are some scenarios that would have worked
> before that wouldn't work anymore with the new scheme.
> Are they important enough to have a workaround, pretty difficult to
> identify for a user?
That question would be reasonable early in the development cycle. At rc4
the question should be: do we think this problem is so critical that we
want to risk breaking something else which currently works for people.
Remember that we are invalidating whatever passthrough testing people
have already done up to this point of the release.
It is also worth noting that the things which this change ends up
breaking may for all we know be equally difficult for a user to identify
(they are after all approximately the same class of issue).
The problem here is that the risk is difficult to evaluate, we just
don't know what will break with this change, and we don't know therefore
if the cure is worse than the disease. The conservative approach at this
point in the release would be to not change anything, or to change the
minimal possible number of things (which would preclude changes which
impact qemu-trad IMHO).
WRT pretty difficult to identify -- the root of this thread suggests the
guest entered a reboot loop with "No bootable device", that sounds
eminently release notable to me. I also not that it was changing the
size of the PCI hole which caused the issue -- which does somewhat
underscore the risks involved in this sort of change.
> > > Also it's difficult for people to realize that they need the workaround
> > > because hvmloader logs aren't enabled by default and only go to the Xen
> > > serial console. The value of this workaround pretty low in my view.
> > > Finally it's worth noting that Windows XP is going EOL in less than an
> > > year.
> >
> > That's been true for something like 5 years...
> >
> > Also, apart from XP, doesn't Windows still pick a HAL at install time,
> > so even a modern guest installed under the old scheme may not get a PAE
> > capable HAL. If you increase the amount of RAM I think Windows will
> > "upgrade" the HAL, but is changing the MMIO layout enough to trigger
> > this? Or maybe modern Windows all use PAE (or even 64 bit) anyway?
> >
> > There are also performance implications of enabling PAE over 2 level
> > paging. Not sure how significant they are with HAP though. Made a big
> > difference with shadow IIRC.
> >
> > Maybe I'm worrying about nothing but while all of these unknowns might
> > be OK towards the start of a release cycle rc4 seems awfully late in the
> > day to be risking it.
>
> Keep in mind that all these configurations are perfectly valid even with
> the code that we have out there today. We aren't doing anything new,
> just modifying the default.
I don't think that is true. We are changing the behaviour, calling it
"just" a default doesn't make it any less worrying or any less of a
change.
> One just needs to assign a PCI device with more than 190MB to trigger it.
> I am trusting the fact that given that we had this behaviour for many
> years now, and it's pretty common to assign a device only some of the
> times you are booting your guest, any problems would have already come
> up.
With qemu-trad perhaps, although that's not completely obvious TBH. In
any case should we really be crossing our fingers and "trusting" that
it'll be ok at rc4?
Ian.
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [Qemu-devel] [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
2013-06-13 17:22 ` Ian Campbell
@ 2013-06-14 10:53 ` George Dunlap
-1 siblings, 0 replies; 82+ messages in thread
From: George Dunlap @ 2013-06-14 10:53 UTC (permalink / raw)
To: Ian Campbell
Cc: Yongjie Ren, xen-devel, Keir Fraser, Hanweidong, Xudong Hao,
Stefano Stabellini, Tim Deegan, qemu-devel, Yanqiangjun,
Wangzhenguo, YangXiaowei, Gonglei (Arei),
Jan Beulich, YongweiX Xu, Luonengjun, Paolo Bonzini,
SongtaoX Liu
On Thu, Jun 13, 2013 at 6:22 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Thu, 2013-06-13 at 17:55 +0100, Stefano Stabellini wrote:
>
>> > > We could have a xenstore flag somewhere that enables the old behaviour
>> > > so that people can revert back to qemu-xen-traditional and make the pci
>> > > hole below 4G even bigger than 448MB, but I think that keeping the old
>> > > behaviour around is going to make the code more difficult to maintain.
>> >
>> > The downside of that is that things which worked with the old scheme may
>> > not work with the new one though. Early in a release cycle when we have
>> > time to discover what has broken then that might be OK, but is post rc4
>> > really the time to be risking it?
>>
>> Yes, you are right: there are some scenarios that would have worked
>> before that wouldn't work anymore with the new scheme.
>> Are they important enough to have a workaround, pretty difficult to
>> identify for a user?
>
> That question would be reasonable early in the development cycle. At rc4
> the question should be: do we think this problem is so critical that we
> want to risk breaking something else which currently works for people.
>
> Remember that we are invalidating whatever passthrough testing people
> have already done up to this point of the release.
>
> It is also worth noting that the things which this change ends up
> breaking may for all we know be equally difficult for a user to identify
> (they are after all approximately the same class of issue).
>
> The problem here is that the risk is difficult to evaluate, we just
> don't know what will break with this change, and we don't know therefore
> if the cure is worse than the disease. The conservative approach at this
> point in the release would be to not change anything, or to change the
> minimal possible number of things (which would preclude changes which
> impact qemu-trad IMHO).
>
> WRT pretty difficult to identify -- the root of this thread suggests the
> guest entered a reboot loop with "No bootable device", that sounds
> eminently release notable to me. I also not that it was changing the
> size of the PCI hole which caused the issue -- which does somewhat
> underscore the risks involved in this sort of change.
But that bug was a bug in the first attempt to fix the root problem.
The root problem shows up as qemu crashing at some point because it
tried to access invalid guest gpfn space; see
http://lists.xen.org/archives/html/xen-devel/2013-03/msg00559.html.
Stefano tried to fix it with the above patch, just changing the hole
to start at 0xe; but that was incomplete, as it didn't match with
hvmloader and seabios's view of the world. That's what this bug
report is about. This thread is an attempt to find a better fix.
So the root problem is that if we revert this patch, and someone
passes through a pci device using qemu-xen (the default) and the MMIO
hole is resized, at some point in the future qemu will randomly die.
If it's a choice between users experiencing, "My VM randomly crashes"
and experiencing, "I tried to pass through this device but the guest
OS doesn't see it", I'd rather choose the latter.
-George
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [BUG 1747]Guest could't find bootable device with memory more than 3600M
@ 2013-06-14 10:53 ` George Dunlap
0 siblings, 0 replies; 82+ messages in thread
From: George Dunlap @ 2013-06-14 10:53 UTC (permalink / raw)
To: Ian Campbell
Cc: Yongjie Ren, xen-devel, Keir Fraser, Hanweidong, Xudong Hao,
Stefano Stabellini, Tim Deegan, qemu-devel, Yanqiangjun,
Wangzhenguo, YangXiaowei, Gonglei (Arei),
Jan Beulich, YongweiX Xu, Luonengjun, Paolo Bonzini,
SongtaoX Liu
On Thu, Jun 13, 2013 at 6:22 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Thu, 2013-06-13 at 17:55 +0100, Stefano Stabellini wrote:
>
>> > > We could have a xenstore flag somewhere that enables the old behaviour
>> > > so that people can revert back to qemu-xen-traditional and make the pci
>> > > hole below 4G even bigger than 448MB, but I think that keeping the old
>> > > behaviour around is going to make the code more difficult to maintain.
>> >
>> > The downside of that is that things which worked with the old scheme may
>> > not work with the new one though. Early in a release cycle when we have
>> > time to discover what has broken then that might be OK, but is post rc4
>> > really the time to be risking it?
>>
>> Yes, you are right: there are some scenarios that would have worked
>> before that wouldn't work anymore with the new scheme.
>> Are they important enough to have a workaround, pretty difficult to
>> identify for a user?
>
> That question would be reasonable early in the development cycle. At rc4
> the question should be: do we think this problem is so critical that we
> want to risk breaking something else which currently works for people.
>
> Remember that we are invalidating whatever passthrough testing people
> have already done up to this point of the release.
>
> It is also worth noting that the things which this change ends up
> breaking may for all we know be equally difficult for a user to identify
> (they are after all approximately the same class of issue).
>
> The problem here is that the risk is difficult to evaluate, we just
> don't know what will break with this change, and we don't know therefore
> if the cure is worse than the disease. The conservative approach at this
> point in the release would be to not change anything, or to change the
> minimal possible number of things (which would preclude changes which
> impact qemu-trad IMHO).
>
> WRT pretty difficult to identify -- the root of this thread suggests the
> guest entered a reboot loop with "No bootable device", that sounds
> eminently release notable to me. I also not that it was changing the
> size of the PCI hole which caused the issue -- which does somewhat
> underscore the risks involved in this sort of change.
But that bug was a bug in the first attempt to fix the root problem.
The root problem shows up as qemu crashing at some point because it
tried to access invalid guest gpfn space; see
http://lists.xen.org/archives/html/xen-devel/2013-03/msg00559.html.
Stefano tried to fix it with the above patch, just changing the hole
to start at 0xe; but that was incomplete, as it didn't match with
hvmloader and seabios's view of the world. That's what this bug
report is about. This thread is an attempt to find a better fix.
So the root problem is that if we revert this patch, and someone
passes through a pci device using qemu-xen (the default) and the MMIO
hole is resized, at some point in the future qemu will randomly die.
If it's a choice between users experiencing, "My VM randomly crashes"
and experiencing, "I tried to pass through this device but the guest
OS doesn't see it", I'd rather choose the latter.
-George
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [Qemu-devel] [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
2013-06-14 10:53 ` George Dunlap
@ 2013-06-14 11:34 ` Ian Campbell
-1 siblings, 0 replies; 82+ messages in thread
From: Ian Campbell @ 2013-06-14 11:34 UTC (permalink / raw)
To: George Dunlap
Cc: Yongjie Ren, xen-devel, Keir Fraser, Hanweidong, Xudong Hao,
Stefano Stabellini, Tim Deegan, qemu-devel, Yanqiangjun,
Wangzhenguo, YangXiaowei, Gonglei (Arei),
Jan Beulich, YongweiX Xu, Luonengjun, Paolo Bonzini,
SongtaoX Liu
On Fri, 2013-06-14 at 11:53 +0100, George Dunlap wrote:
> On Thu, Jun 13, 2013 at 6:22 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > On Thu, 2013-06-13 at 17:55 +0100, Stefano Stabellini wrote:
> >
> >> > > We could have a xenstore flag somewhere that enables the old behaviour
> >> > > so that people can revert back to qemu-xen-traditional and make the pci
> >> > > hole below 4G even bigger than 448MB, but I think that keeping the old
> >> > > behaviour around is going to make the code more difficult to maintain.
> >> >
> >> > The downside of that is that things which worked with the old scheme may
> >> > not work with the new one though. Early in a release cycle when we have
> >> > time to discover what has broken then that might be OK, but is post rc4
> >> > really the time to be risking it?
> >>
> >> Yes, you are right: there are some scenarios that would have worked
> >> before that wouldn't work anymore with the new scheme.
> >> Are they important enough to have a workaround, pretty difficult to
> >> identify for a user?
> >
> > That question would be reasonable early in the development cycle. At rc4
> > the question should be: do we think this problem is so critical that we
> > want to risk breaking something else which currently works for people.
> >
> > Remember that we are invalidating whatever passthrough testing people
> > have already done up to this point of the release.
> >
> > It is also worth noting that the things which this change ends up
> > breaking may for all we know be equally difficult for a user to identify
> > (they are after all approximately the same class of issue).
> >
> > The problem here is that the risk is difficult to evaluate, we just
> > don't know what will break with this change, and we don't know therefore
> > if the cure is worse than the disease. The conservative approach at this
> > point in the release would be to not change anything, or to change the
> > minimal possible number of things (which would preclude changes which
> > impact qemu-trad IMHO).
> >
>
>
> > WRT pretty difficult to identify -- the root of this thread suggests the
> > guest entered a reboot loop with "No bootable device", that sounds
> > eminently release notable to me. I also not that it was changing the
> > size of the PCI hole which caused the issue -- which does somewhat
> > underscore the risks involved in this sort of change.
>
> But that bug was a bug in the first attempt to fix the root problem.
> The root problem shows up as qemu crashing at some point because it
> tried to access invalid guest gpfn space; see
> http://lists.xen.org/archives/html/xen-devel/2013-03/msg00559.html.
>
> Stefano tried to fix it with the above patch, just changing the hole
> to start at 0xe; but that was incomplete, as it didn't match with
> hvmloader and seabios's view of the world. That's what this bug
> report is about. This thread is an attempt to find a better fix.
>
> So the root problem is that if we revert this patch, and someone
> passes through a pci device using qemu-xen (the default) and the MMIO
> hole is resized, at some point in the future qemu will randomly die.
Right, I see, thanks for explaining.
> If it's a choice between users experiencing, "My VM randomly crashes"
> and experiencing, "I tried to pass through this device but the guest
> OS doesn't see it", I'd rather choose the latter.
All other things being equal, obviously we all would. But the point I've
been trying to make is that we don't know the other consequences of
making that fix -- e.g. on existing working configurations. So the
choice is "some VMs randomly crash, but other stuff works fine and we
have had a reasonable amount of user testing" and "those particular VMs
don't crash any more, but we don't know what other stuff no longer works
and the existing test base has been at least partially invalidated".
I think that at post rc4 in a release we ought to be being pretty
conservative about the risks of this sort of change, especially wrt
invalidating testing and the unknowns involved.
Aren't the configurations which might trip over this issue are going to
be in the minority compared to those which we risk breaking?
Ian.
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
@ 2013-06-14 11:34 ` Ian Campbell
0 siblings, 0 replies; 82+ messages in thread
From: Ian Campbell @ 2013-06-14 11:34 UTC (permalink / raw)
To: George Dunlap
Cc: Yongjie Ren, xen-devel, Keir Fraser, Hanweidong, Xudong Hao,
Stefano Stabellini, Tim Deegan, qemu-devel, Yanqiangjun,
Wangzhenguo, YangXiaowei, Gonglei (Arei),
Jan Beulich, YongweiX Xu, Luonengjun, Paolo Bonzini,
SongtaoX Liu
On Fri, 2013-06-14 at 11:53 +0100, George Dunlap wrote:
> On Thu, Jun 13, 2013 at 6:22 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> > On Thu, 2013-06-13 at 17:55 +0100, Stefano Stabellini wrote:
> >
> >> > > We could have a xenstore flag somewhere that enables the old behaviour
> >> > > so that people can revert back to qemu-xen-traditional and make the pci
> >> > > hole below 4G even bigger than 448MB, but I think that keeping the old
> >> > > behaviour around is going to make the code more difficult to maintain.
> >> >
> >> > The downside of that is that things which worked with the old scheme may
> >> > not work with the new one though. Early in a release cycle when we have
> >> > time to discover what has broken then that might be OK, but is post rc4
> >> > really the time to be risking it?
> >>
> >> Yes, you are right: there are some scenarios that would have worked
> >> before that wouldn't work anymore with the new scheme.
> >> Are they important enough to have a workaround, pretty difficult to
> >> identify for a user?
> >
> > That question would be reasonable early in the development cycle. At rc4
> > the question should be: do we think this problem is so critical that we
> > want to risk breaking something else which currently works for people.
> >
> > Remember that we are invalidating whatever passthrough testing people
> > have already done up to this point of the release.
> >
> > It is also worth noting that the things which this change ends up
> > breaking may for all we know be equally difficult for a user to identify
> > (they are after all approximately the same class of issue).
> >
> > The problem here is that the risk is difficult to evaluate, we just
> > don't know what will break with this change, and we don't know therefore
> > if the cure is worse than the disease. The conservative approach at this
> > point in the release would be to not change anything, or to change the
> > minimal possible number of things (which would preclude changes which
> > impact qemu-trad IMHO).
> >
>
>
> > WRT pretty difficult to identify -- the root of this thread suggests the
> > guest entered a reboot loop with "No bootable device", that sounds
> > eminently release notable to me. I also not that it was changing the
> > size of the PCI hole which caused the issue -- which does somewhat
> > underscore the risks involved in this sort of change.
>
> But that bug was a bug in the first attempt to fix the root problem.
> The root problem shows up as qemu crashing at some point because it
> tried to access invalid guest gpfn space; see
> http://lists.xen.org/archives/html/xen-devel/2013-03/msg00559.html.
>
> Stefano tried to fix it with the above patch, just changing the hole
> to start at 0xe; but that was incomplete, as it didn't match with
> hvmloader and seabios's view of the world. That's what this bug
> report is about. This thread is an attempt to find a better fix.
>
> So the root problem is that if we revert this patch, and someone
> passes through a pci device using qemu-xen (the default) and the MMIO
> hole is resized, at some point in the future qemu will randomly die.
Right, I see, thanks for explaining.
> If it's a choice between users experiencing, "My VM randomly crashes"
> and experiencing, "I tried to pass through this device but the guest
> OS doesn't see it", I'd rather choose the latter.
All other things being equal, obviously we all would. But the point I've
been trying to make is that we don't know the other consequences of
making that fix -- e.g. on existing working configurations. So the
choice is "some VMs randomly crash, but other stuff works fine and we
have had a reasonable amount of user testing" and "those particular VMs
don't crash any more, but we don't know what other stuff no longer works
and the existing test base has been at least partially invalidated".
I think that at post rc4 in a release we ought to be being pretty
conservative about the risks of this sort of change, especially wrt
invalidating testing and the unknowns involved.
Aren't the configurations which might trip over this issue are going to
be in the minority compared to those which we risk breaking?
Ian.
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [Qemu-devel] [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
2013-06-14 11:34 ` Ian Campbell
@ 2013-06-14 14:14 ` George Dunlap
-1 siblings, 0 replies; 82+ messages in thread
From: George Dunlap @ 2013-06-14 14:14 UTC (permalink / raw)
To: Ian Campbell
Cc: Yongjie Ren, xen-devel, Keir Fraser, Hanweidong, Xudong Hao,
Stefano Stabellini, Tim Deegan, qemu-devel, Yanqiangjun,
Wangzhenguo, YangXiaowei, Gonglei (Arei),
Jan Beulich, YongweiX Xu, Luonengjun, Paolo Bonzini,
SongtaoX Liu
On 14/06/13 12:34, Ian Campbell wrote:
> On Fri, 2013-06-14 at 11:53 +0100, George Dunlap wrote:
>> On Thu, Jun 13, 2013 at 6:22 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>>> On Thu, 2013-06-13 at 17:55 +0100, Stefano Stabellini wrote:
>>>
>>>>>> We could have a xenstore flag somewhere that enables the old behaviour
>>>>>> so that people can revert back to qemu-xen-traditional and make the pci
>>>>>> hole below 4G even bigger than 448MB, but I think that keeping the old
>>>>>> behaviour around is going to make the code more difficult to maintain.
>>>>> The downside of that is that things which worked with the old scheme may
>>>>> not work with the new one though. Early in a release cycle when we have
>>>>> time to discover what has broken then that might be OK, but is post rc4
>>>>> really the time to be risking it?
>>>> Yes, you are right: there are some scenarios that would have worked
>>>> before that wouldn't work anymore with the new scheme.
>>>> Are they important enough to have a workaround, pretty difficult to
>>>> identify for a user?
>>> That question would be reasonable early in the development cycle. At rc4
>>> the question should be: do we think this problem is so critical that we
>>> want to risk breaking something else which currently works for people.
>>>
>>> Remember that we are invalidating whatever passthrough testing people
>>> have already done up to this point of the release.
>>>
>>> It is also worth noting that the things which this change ends up
>>> breaking may for all we know be equally difficult for a user to identify
>>> (they are after all approximately the same class of issue).
>>>
>>> The problem here is that the risk is difficult to evaluate, we just
>>> don't know what will break with this change, and we don't know therefore
>>> if the cure is worse than the disease. The conservative approach at this
>>> point in the release would be to not change anything, or to change the
>>> minimal possible number of things (which would preclude changes which
>>> impact qemu-trad IMHO).
>>>
>>
>>> WRT pretty difficult to identify -- the root of this thread suggests the
>>> guest entered a reboot loop with "No bootable device", that sounds
>>> eminently release notable to me. I also not that it was changing the
>>> size of the PCI hole which caused the issue -- which does somewhat
>>> underscore the risks involved in this sort of change.
>> But that bug was a bug in the first attempt to fix the root problem.
>> The root problem shows up as qemu crashing at some point because it
>> tried to access invalid guest gpfn space; see
>> http://lists.xen.org/archives/html/xen-devel/2013-03/msg00559.html.
>>
>> Stefano tried to fix it with the above patch, just changing the hole
>> to start at 0xe; but that was incomplete, as it didn't match with
>> hvmloader and seabios's view of the world. That's what this bug
>> report is about. This thread is an attempt to find a better fix.
>>
>> So the root problem is that if we revert this patch, and someone
>> passes through a pci device using qemu-xen (the default) and the MMIO
>> hole is resized, at some point in the future qemu will randomly die.
> Right, I see, thanks for explaining.
>
>> If it's a choice between users experiencing, "My VM randomly crashes"
>> and experiencing, "I tried to pass through this device but the guest
>> OS doesn't see it", I'd rather choose the latter.
> All other things being equal, obviously we all would. But the point I've
> been trying to make is that we don't know the other consequences of
> making that fix -- e.g. on existing working configurations. So the
> choice is "some VMs randomly crash, but other stuff works fine and we
> have had a reasonable amount of user testing" and "those particular VMs
> don't crash any more, but we don't know what other stuff no longer works
> and the existing test base has been at least partially invalidated".
>
> I think that at post rc4 in a release we ought to be being pretty
> conservative about the risks of this sort of change, especially wrt
> invalidating testing and the unknowns involved.
>
> Aren't the configurations which might trip over this issue are going to
> be in the minority compared to those which we risk breaking?
So there are the technical proposals we've been discussing, each of
which has different risks.
1. Set the default MMIO hole size to 0xe0000000.
2. If possible, relocate PCI devices that don't fit in the hole to the
64-bit hole.
- Here "if possible" will mean a) the device has a 64-bit BAR, and b)
this hasn't been disabled by libxl (probably via a xenstore key).
3. If possible, resize the MMIO hole; otherwise refuse to map the device
- Currently "if possible" is always true; the new thing here would be
making it possible for libxl to disable this, probably via a xenstore key.
Each of these will have different risks for qemu-traditional and qemu-xen.
Implementing #3 would have no risk for qemu-traditional, because we
won't be changing the way anything works; what works will still work,
what is broken (if anything) will still be broken.
Implementing #3 for qemu-xen only changes one kind of failure for
another. If you resize the MMIO hole for qemu-xen, then you *will*
eventually crash. So this will not break existing working
configurations -- it will only change the failure from "qemu crashes at
some point" to "the guest OS cannot see the device". This is a uniform
improvement.
So #3 is very low risk, as far as I can tell, and has a solid benefit.
I think we should definitely implement it.
I think #2 should have no impact on qemu-traditional, because xl should
disable it by default.
For qemu-xen, the only devices that are relocated are devices that would
otherwise be disabled by #3; so remember that the alternate for this one
(assuming we implement #3) is "not visible to OS". There are several
potential sub-sets here:
a. Guest OSes that can't access the 64-bit region. In that case the
device will not be visible, which is the same failure as not
implementing this at all.
b. I think most devices and operating systems will just work; this
codepath in QEMU and guest OSes is fairly well tested with KVM.
c. It may be that there are devices that would have worked with
qemu-traditional and placed in a resized MMIO hole, but that will break
with qemu-xen and and placed in the 64-bit MMIO hole. For these
devices, the failure mode will change from "not visible to guest OS" to
"fails in some other unforseen way".
The the main risk I see from this one is c. However, I think from a
cost-benefits, it's still pretty low -- we get the benefit of most
people transparently being able to just use pci-passthrough, at the
potential cost of a handful of people having "weird crash" failures
instead of "can't use the device" failures.
I suppose that there is a potential risk for b as well, in that we
haven't tested relocating to the 64-bit MMIO hole *with Xen*. There may
be assumptions about 32-bit paddrs baked in somewhere that we don't know
about. If we implement this change on top of #3, and there are problems
with b, then it will change "device not visible to guest OS" failures
back into "may crash in a weird way" failures.
On the whole I would be inclined to implement #2 if it's not too
difficult, but I can certainly see the point of saying that it's too
risky and that we shouldn't do it.
Then there's #1. This should in theory be low-risk, because in theory
hvmloader might have chosen 0xe as the start of the MMIO hole anyway.
For qemu-traditional, this change has no benefits. The benefits for
qemu-xen depend on what else gets implemented. If nothing else is
implemented, this changes some "random qemu crash" failures into
successes (while leaving other ones to keep crashing). If #3 is
implemented, then it changes some "guest OS can't see device" failures
into successes. If Both #3 and #2 are implemented, then some of the
"guest OS can't see 64-bit MMIO hole" failures will change into successes.
However, as you say, the number of times hvmloader *actually* choses
that at the moment is fairly low. The vast majority of VMs and
configurations at the moment will *not* use 0xe as a base; at most only
a handful of people will have tested that configuration. So there is a
fairly significant risk that there *is* some configuration for which,
*had* hvmloader chosen 0xe, then it *would* have caused a problem. For
those configurations, #1 will change "will break but only if you have a
very large PCI device" to "will break no matter what".
This would be fine if we hadn't started RCs; but we have. Our most
important userbase is people who are *not* doing pci pass-through; so I
think I agree with you, that this introduces a probably unacceptable
risk for very little gain -- particularly if we implement #3.
So my recommendation is:
* Implement #3
* Consider implementing #2
* Don't implement #1.
Thoughts?
-George
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
@ 2013-06-14 14:14 ` George Dunlap
0 siblings, 0 replies; 82+ messages in thread
From: George Dunlap @ 2013-06-14 14:14 UTC (permalink / raw)
To: Ian Campbell
Cc: Yongjie Ren, xen-devel, Keir Fraser, Hanweidong, Xudong Hao,
Stefano Stabellini, Tim Deegan, qemu-devel, Yanqiangjun,
Wangzhenguo, YangXiaowei, Gonglei (Arei),
Jan Beulich, YongweiX Xu, Luonengjun, Paolo Bonzini,
SongtaoX Liu
On 14/06/13 12:34, Ian Campbell wrote:
> On Fri, 2013-06-14 at 11:53 +0100, George Dunlap wrote:
>> On Thu, Jun 13, 2013 at 6:22 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>>> On Thu, 2013-06-13 at 17:55 +0100, Stefano Stabellini wrote:
>>>
>>>>>> We could have a xenstore flag somewhere that enables the old behaviour
>>>>>> so that people can revert back to qemu-xen-traditional and make the pci
>>>>>> hole below 4G even bigger than 448MB, but I think that keeping the old
>>>>>> behaviour around is going to make the code more difficult to maintain.
>>>>> The downside of that is that things which worked with the old scheme may
>>>>> not work with the new one though. Early in a release cycle when we have
>>>>> time to discover what has broken then that might be OK, but is post rc4
>>>>> really the time to be risking it?
>>>> Yes, you are right: there are some scenarios that would have worked
>>>> before that wouldn't work anymore with the new scheme.
>>>> Are they important enough to have a workaround, pretty difficult to
>>>> identify for a user?
>>> That question would be reasonable early in the development cycle. At rc4
>>> the question should be: do we think this problem is so critical that we
>>> want to risk breaking something else which currently works for people.
>>>
>>> Remember that we are invalidating whatever passthrough testing people
>>> have already done up to this point of the release.
>>>
>>> It is also worth noting that the things which this change ends up
>>> breaking may for all we know be equally difficult for a user to identify
>>> (they are after all approximately the same class of issue).
>>>
>>> The problem here is that the risk is difficult to evaluate, we just
>>> don't know what will break with this change, and we don't know therefore
>>> if the cure is worse than the disease. The conservative approach at this
>>> point in the release would be to not change anything, or to change the
>>> minimal possible number of things (which would preclude changes which
>>> impact qemu-trad IMHO).
>>>
>>
>>> WRT pretty difficult to identify -- the root of this thread suggests the
>>> guest entered a reboot loop with "No bootable device", that sounds
>>> eminently release notable to me. I also not that it was changing the
>>> size of the PCI hole which caused the issue -- which does somewhat
>>> underscore the risks involved in this sort of change.
>> But that bug was a bug in the first attempt to fix the root problem.
>> The root problem shows up as qemu crashing at some point because it
>> tried to access invalid guest gpfn space; see
>> http://lists.xen.org/archives/html/xen-devel/2013-03/msg00559.html.
>>
>> Stefano tried to fix it with the above patch, just changing the hole
>> to start at 0xe; but that was incomplete, as it didn't match with
>> hvmloader and seabios's view of the world. That's what this bug
>> report is about. This thread is an attempt to find a better fix.
>>
>> So the root problem is that if we revert this patch, and someone
>> passes through a pci device using qemu-xen (the default) and the MMIO
>> hole is resized, at some point in the future qemu will randomly die.
> Right, I see, thanks for explaining.
>
>> If it's a choice between users experiencing, "My VM randomly crashes"
>> and experiencing, "I tried to pass through this device but the guest
>> OS doesn't see it", I'd rather choose the latter.
> All other things being equal, obviously we all would. But the point I've
> been trying to make is that we don't know the other consequences of
> making that fix -- e.g. on existing working configurations. So the
> choice is "some VMs randomly crash, but other stuff works fine and we
> have had a reasonable amount of user testing" and "those particular VMs
> don't crash any more, but we don't know what other stuff no longer works
> and the existing test base has been at least partially invalidated".
>
> I think that at post rc4 in a release we ought to be being pretty
> conservative about the risks of this sort of change, especially wrt
> invalidating testing and the unknowns involved.
>
> Aren't the configurations which might trip over this issue are going to
> be in the minority compared to those which we risk breaking?
So there are the technical proposals we've been discussing, each of
which has different risks.
1. Set the default MMIO hole size to 0xe0000000.
2. If possible, relocate PCI devices that don't fit in the hole to the
64-bit hole.
- Here "if possible" will mean a) the device has a 64-bit BAR, and b)
this hasn't been disabled by libxl (probably via a xenstore key).
3. If possible, resize the MMIO hole; otherwise refuse to map the device
- Currently "if possible" is always true; the new thing here would be
making it possible for libxl to disable this, probably via a xenstore key.
Each of these will have different risks for qemu-traditional and qemu-xen.
Implementing #3 would have no risk for qemu-traditional, because we
won't be changing the way anything works; what works will still work,
what is broken (if anything) will still be broken.
Implementing #3 for qemu-xen only changes one kind of failure for
another. If you resize the MMIO hole for qemu-xen, then you *will*
eventually crash. So this will not break existing working
configurations -- it will only change the failure from "qemu crashes at
some point" to "the guest OS cannot see the device". This is a uniform
improvement.
So #3 is very low risk, as far as I can tell, and has a solid benefit.
I think we should definitely implement it.
I think #2 should have no impact on qemu-traditional, because xl should
disable it by default.
For qemu-xen, the only devices that are relocated are devices that would
otherwise be disabled by #3; so remember that the alternate for this one
(assuming we implement #3) is "not visible to OS". There are several
potential sub-sets here:
a. Guest OSes that can't access the 64-bit region. In that case the
device will not be visible, which is the same failure as not
implementing this at all.
b. I think most devices and operating systems will just work; this
codepath in QEMU and guest OSes is fairly well tested with KVM.
c. It may be that there are devices that would have worked with
qemu-traditional and placed in a resized MMIO hole, but that will break
with qemu-xen and and placed in the 64-bit MMIO hole. For these
devices, the failure mode will change from "not visible to guest OS" to
"fails in some other unforseen way".
The the main risk I see from this one is c. However, I think from a
cost-benefits, it's still pretty low -- we get the benefit of most
people transparently being able to just use pci-passthrough, at the
potential cost of a handful of people having "weird crash" failures
instead of "can't use the device" failures.
I suppose that there is a potential risk for b as well, in that we
haven't tested relocating to the 64-bit MMIO hole *with Xen*. There may
be assumptions about 32-bit paddrs baked in somewhere that we don't know
about. If we implement this change on top of #3, and there are problems
with b, then it will change "device not visible to guest OS" failures
back into "may crash in a weird way" failures.
On the whole I would be inclined to implement #2 if it's not too
difficult, but I can certainly see the point of saying that it's too
risky and that we shouldn't do it.
Then there's #1. This should in theory be low-risk, because in theory
hvmloader might have chosen 0xe as the start of the MMIO hole anyway.
For qemu-traditional, this change has no benefits. The benefits for
qemu-xen depend on what else gets implemented. If nothing else is
implemented, this changes some "random qemu crash" failures into
successes (while leaving other ones to keep crashing). If #3 is
implemented, then it changes some "guest OS can't see device" failures
into successes. If Both #3 and #2 are implemented, then some of the
"guest OS can't see 64-bit MMIO hole" failures will change into successes.
However, as you say, the number of times hvmloader *actually* choses
that at the moment is fairly low. The vast majority of VMs and
configurations at the moment will *not* use 0xe as a base; at most only
a handful of people will have tested that configuration. So there is a
fairly significant risk that there *is* some configuration for which,
*had* hvmloader chosen 0xe, then it *would* have caused a problem. For
those configurations, #1 will change "will break but only if you have a
very large PCI device" to "will break no matter what".
This would be fine if we hadn't started RCs; but we have. Our most
important userbase is people who are *not* doing pci pass-through; so I
think I agree with you, that this introduces a probably unacceptable
risk for very little gain -- particularly if we implement #3.
So my recommendation is:
* Implement #3
* Consider implementing #2
* Don't implement #1.
Thoughts?
-George
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [Qemu-devel] [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
2013-06-14 14:14 ` George Dunlap
@ 2013-06-14 14:36 ` George Dunlap
-1 siblings, 0 replies; 82+ messages in thread
From: George Dunlap @ 2013-06-14 14:36 UTC (permalink / raw)
To: Ian Campbell
Cc: Yongjie Ren, xen-devel, Keir Fraser, Hanweidong, Xudong Hao,
Stefano Stabellini, Tim Deegan, qemu-devel, Yanqiangjun,
Wangzhenguo, YangXiaowei, Gonglei (Arei),
Jan Beulich, YongweiX Xu, Luonengjun, Paolo Bonzini,
SongtaoX Liu
On 14/06/13 15:14, George Dunlap wrote:
> On 14/06/13 12:34, Ian Campbell wrote:
>> On Fri, 2013-06-14 at 11:53 +0100, George Dunlap wrote:
>>> On Thu, Jun 13, 2013 at 6:22 PM, Ian Campbell
>>> <Ian.Campbell@citrix.com> wrote:
>>>> On Thu, 2013-06-13 at 17:55 +0100, Stefano Stabellini wrote:
>>>>
>>>>>>> We could have a xenstore flag somewhere that enables the old
>>>>>>> behaviour
>>>>>>> so that people can revert back to qemu-xen-traditional and make
>>>>>>> the pci
>>>>>>> hole below 4G even bigger than 448MB, but I think that keeping
>>>>>>> the old
>>>>>>> behaviour around is going to make the code more difficult to
>>>>>>> maintain.
>>>>>> The downside of that is that things which worked with the old
>>>>>> scheme may
>>>>>> not work with the new one though. Early in a release cycle when
>>>>>> we have
>>>>>> time to discover what has broken then that might be OK, but is
>>>>>> post rc4
>>>>>> really the time to be risking it?
>>>>> Yes, you are right: there are some scenarios that would have worked
>>>>> before that wouldn't work anymore with the new scheme.
>>>>> Are they important enough to have a workaround, pretty difficult to
>>>>> identify for a user?
>>>> That question would be reasonable early in the development cycle.
>>>> At rc4
>>>> the question should be: do we think this problem is so critical
>>>> that we
>>>> want to risk breaking something else which currently works for people.
>>>>
>>>> Remember that we are invalidating whatever passthrough testing people
>>>> have already done up to this point of the release.
>>>>
>>>> It is also worth noting that the things which this change ends up
>>>> breaking may for all we know be equally difficult for a user to
>>>> identify
>>>> (they are after all approximately the same class of issue).
>>>>
>>>> The problem here is that the risk is difficult to evaluate, we just
>>>> don't know what will break with this change, and we don't know
>>>> therefore
>>>> if the cure is worse than the disease. The conservative approach at
>>>> this
>>>> point in the release would be to not change anything, or to change the
>>>> minimal possible number of things (which would preclude changes which
>>>> impact qemu-trad IMHO).
>>>>
>>>
>>>> WRT pretty difficult to identify -- the root of this thread
>>>> suggests the
>>>> guest entered a reboot loop with "No bootable device", that sounds
>>>> eminently release notable to me. I also not that it was changing the
>>>> size of the PCI hole which caused the issue -- which does somewhat
>>>> underscore the risks involved in this sort of change.
>>> But that bug was a bug in the first attempt to fix the root problem.
>>> The root problem shows up as qemu crashing at some point because it
>>> tried to access invalid guest gpfn space; see
>>> http://lists.xen.org/archives/html/xen-devel/2013-03/msg00559.html.
>>>
>>> Stefano tried to fix it with the above patch, just changing the hole
>>> to start at 0xe; but that was incomplete, as it didn't match with
>>> hvmloader and seabios's view of the world. That's what this bug
>>> report is about. This thread is an attempt to find a better fix.
>>>
>>> So the root problem is that if we revert this patch, and someone
>>> passes through a pci device using qemu-xen (the default) and the MMIO
>>> hole is resized, at some point in the future qemu will randomly die.
>> Right, I see, thanks for explaining.
>>
>>> If it's a choice between users experiencing, "My VM randomly crashes"
>>> and experiencing, "I tried to pass through this device but the guest
>>> OS doesn't see it", I'd rather choose the latter.
>> All other things being equal, obviously we all would. But the point I've
>> been trying to make is that we don't know the other consequences of
>> making that fix -- e.g. on existing working configurations. So the
>> choice is "some VMs randomly crash, but other stuff works fine and we
>> have had a reasonable amount of user testing" and "those particular VMs
>> don't crash any more, but we don't know what other stuff no longer works
>> and the existing test base has been at least partially invalidated".
>>
>> I think that at post rc4 in a release we ought to be being pretty
>> conservative about the risks of this sort of change, especially wrt
>> invalidating testing and the unknowns involved.
>>
>> Aren't the configurations which might trip over this issue are going to
>> be in the minority compared to those which we risk breaking?
>
> So there are the technical proposals we've been discussing, each of
> which has different risks.
>
> 1. Set the default MMIO hole size to 0xe0000000.
> 2. If possible, relocate PCI devices that don't fit in the hole to the
> 64-bit hole.
> - Here "if possible" will mean a) the device has a 64-bit BAR, and b)
> this hasn't been disabled by libxl (probably via a xenstore key).
> 3. If possible, resize the MMIO hole; otherwise refuse to map the device
> - Currently "if possible" is always true; the new thing here would be
> making it possible for libxl to disable this, probably via a xenstore
> key.
>
> Each of these will have different risks for qemu-traditional and
> qemu-xen.
>
> Implementing #3 would have no risk for qemu-traditional, because we
> won't be changing the way anything works; what works will still work,
> what is broken (if anything) will still be broken.
>
> Implementing #3 for qemu-xen only changes one kind of failure for
> another. If you resize the MMIO hole for qemu-xen, then you *will*
> eventually crash. So this will not break existing working
> configurations -- it will only change the failure from "qemu crashes
> at some point" to "the guest OS cannot see the device". This is a
> uniform improvement.
I suppose this is not strictly true. If you resize the MMIO hole *such
that it overlaps what was originally guest memory*, then it will crash.
If you have a smaller guest with say, only 1 or 2GiB of RAM, then you
can probably resize the MMIO hole arbitrarily on qemu-xen and have no
ill effects. So as stated ("never resize MMIO hole"), this would cause
some successes into "guest can't see the device" failures.
(Stefano, correct me if I'm wrong here.)
But hvmloader should know whether this is the case, however, because if
there is memory there it has to relocate it. So we should change "is
possible" to mean, "if we don't need to relocate memory, or if
relocating memory has been enabled by libxl".
-George
^ permalink raw reply [flat|nested] 82+ messages in thread* Re: [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
@ 2013-06-14 14:36 ` George Dunlap
0 siblings, 0 replies; 82+ messages in thread
From: George Dunlap @ 2013-06-14 14:36 UTC (permalink / raw)
To: Ian Campbell
Cc: Yongjie Ren, xen-devel, Keir Fraser, Hanweidong, Xudong Hao,
Stefano Stabellini, Tim Deegan, qemu-devel, Yanqiangjun,
Wangzhenguo, YangXiaowei, Gonglei (Arei),
Jan Beulich, YongweiX Xu, Luonengjun, Paolo Bonzini,
SongtaoX Liu
On 14/06/13 15:14, George Dunlap wrote:
> On 14/06/13 12:34, Ian Campbell wrote:
>> On Fri, 2013-06-14 at 11:53 +0100, George Dunlap wrote:
>>> On Thu, Jun 13, 2013 at 6:22 PM, Ian Campbell
>>> <Ian.Campbell@citrix.com> wrote:
>>>> On Thu, 2013-06-13 at 17:55 +0100, Stefano Stabellini wrote:
>>>>
>>>>>>> We could have a xenstore flag somewhere that enables the old
>>>>>>> behaviour
>>>>>>> so that people can revert back to qemu-xen-traditional and make
>>>>>>> the pci
>>>>>>> hole below 4G even bigger than 448MB, but I think that keeping
>>>>>>> the old
>>>>>>> behaviour around is going to make the code more difficult to
>>>>>>> maintain.
>>>>>> The downside of that is that things which worked with the old
>>>>>> scheme may
>>>>>> not work with the new one though. Early in a release cycle when
>>>>>> we have
>>>>>> time to discover what has broken then that might be OK, but is
>>>>>> post rc4
>>>>>> really the time to be risking it?
>>>>> Yes, you are right: there are some scenarios that would have worked
>>>>> before that wouldn't work anymore with the new scheme.
>>>>> Are they important enough to have a workaround, pretty difficult to
>>>>> identify for a user?
>>>> That question would be reasonable early in the development cycle.
>>>> At rc4
>>>> the question should be: do we think this problem is so critical
>>>> that we
>>>> want to risk breaking something else which currently works for people.
>>>>
>>>> Remember that we are invalidating whatever passthrough testing people
>>>> have already done up to this point of the release.
>>>>
>>>> It is also worth noting that the things which this change ends up
>>>> breaking may for all we know be equally difficult for a user to
>>>> identify
>>>> (they are after all approximately the same class of issue).
>>>>
>>>> The problem here is that the risk is difficult to evaluate, we just
>>>> don't know what will break with this change, and we don't know
>>>> therefore
>>>> if the cure is worse than the disease. The conservative approach at
>>>> this
>>>> point in the release would be to not change anything, or to change the
>>>> minimal possible number of things (which would preclude changes which
>>>> impact qemu-trad IMHO).
>>>>
>>>
>>>> WRT pretty difficult to identify -- the root of this thread
>>>> suggests the
>>>> guest entered a reboot loop with "No bootable device", that sounds
>>>> eminently release notable to me. I also not that it was changing the
>>>> size of the PCI hole which caused the issue -- which does somewhat
>>>> underscore the risks involved in this sort of change.
>>> But that bug was a bug in the first attempt to fix the root problem.
>>> The root problem shows up as qemu crashing at some point because it
>>> tried to access invalid guest gpfn space; see
>>> http://lists.xen.org/archives/html/xen-devel/2013-03/msg00559.html.
>>>
>>> Stefano tried to fix it with the above patch, just changing the hole
>>> to start at 0xe; but that was incomplete, as it didn't match with
>>> hvmloader and seabios's view of the world. That's what this bug
>>> report is about. This thread is an attempt to find a better fix.
>>>
>>> So the root problem is that if we revert this patch, and someone
>>> passes through a pci device using qemu-xen (the default) and the MMIO
>>> hole is resized, at some point in the future qemu will randomly die.
>> Right, I see, thanks for explaining.
>>
>>> If it's a choice between users experiencing, "My VM randomly crashes"
>>> and experiencing, "I tried to pass through this device but the guest
>>> OS doesn't see it", I'd rather choose the latter.
>> All other things being equal, obviously we all would. But the point I've
>> been trying to make is that we don't know the other consequences of
>> making that fix -- e.g. on existing working configurations. So the
>> choice is "some VMs randomly crash, but other stuff works fine and we
>> have had a reasonable amount of user testing" and "those particular VMs
>> don't crash any more, but we don't know what other stuff no longer works
>> and the existing test base has been at least partially invalidated".
>>
>> I think that at post rc4 in a release we ought to be being pretty
>> conservative about the risks of this sort of change, especially wrt
>> invalidating testing and the unknowns involved.
>>
>> Aren't the configurations which might trip over this issue are going to
>> be in the minority compared to those which we risk breaking?
>
> So there are the technical proposals we've been discussing, each of
> which has different risks.
>
> 1. Set the default MMIO hole size to 0xe0000000.
> 2. If possible, relocate PCI devices that don't fit in the hole to the
> 64-bit hole.
> - Here "if possible" will mean a) the device has a 64-bit BAR, and b)
> this hasn't been disabled by libxl (probably via a xenstore key).
> 3. If possible, resize the MMIO hole; otherwise refuse to map the device
> - Currently "if possible" is always true; the new thing here would be
> making it possible for libxl to disable this, probably via a xenstore
> key.
>
> Each of these will have different risks for qemu-traditional and
> qemu-xen.
>
> Implementing #3 would have no risk for qemu-traditional, because we
> won't be changing the way anything works; what works will still work,
> what is broken (if anything) will still be broken.
>
> Implementing #3 for qemu-xen only changes one kind of failure for
> another. If you resize the MMIO hole for qemu-xen, then you *will*
> eventually crash. So this will not break existing working
> configurations -- it will only change the failure from "qemu crashes
> at some point" to "the guest OS cannot see the device". This is a
> uniform improvement.
I suppose this is not strictly true. If you resize the MMIO hole *such
that it overlaps what was originally guest memory*, then it will crash.
If you have a smaller guest with say, only 1 or 2GiB of RAM, then you
can probably resize the MMIO hole arbitrarily on qemu-xen and have no
ill effects. So as stated ("never resize MMIO hole"), this would cause
some successes into "guest can't see the device" failures.
(Stefano, correct me if I'm wrong here.)
But hvmloader should know whether this is the case, however, because if
there is memory there it has to relocate it. So we should change "is
possible" to mean, "if we don't need to relocate memory, or if
relocating memory has been enabled by libxl".
-George
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [Qemu-devel] [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
2013-06-13 13:54 ` George Dunlap
@ 2013-06-13 14:54 ` Paolo Bonzini
-1 siblings, 0 replies; 82+ messages in thread
From: Paolo Bonzini @ 2013-06-13 14:54 UTC (permalink / raw)
To: George Dunlap
Cc: Yongjie Ren, xen-devel, Keir Fraser, Ian Campbell,
Stefano Stabellini, Xudong Hao, hanweidong, Tim Deegan,
qemu-devel, yanqiangjun, wangzhenguo, xiaowei.yang, arei.gonglei,
Jan Beulich, luonengjun, YongweiX Xu, SongtaoX Liu
Il 13/06/2013 09:54, George Dunlap ha scritto:
>
> Also, as I understand it, at the moment:
> 1. Some operating systems (32-bit XP) won't be able to use relocated
> devices
> 2. Some devices (without 64-bit BARs) can't be relocated
Are there really devices with huge 32-bit BARs? I think #1 is the only
real problem, so far though it never was for KVM.
SeaBIOS sorts the BARs from smallest to largest alignment, and then from
smallest to largest size. Typically only the GPU will be relocated.
Paolo
> 3. qemu-traditional is fine with a resized <4GiB MMIO hole.
>
> So if we have #1 or #2, at the moment an option for a work-around is to
> use qemu-traditional.
>
> However, if we add your "print an error if the device doesn't fit", then
> this option will go away -- this will be a regression in functionality
> from 4.2.
>
> I thought that what we had proposed was to have an option in xenstore,
> that libxl would set, which would instruct hvmloader whether to expand
> the MMIO hole and whether to relocate devices above 64-bit?
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [BUG 1747]Guest could't find bootable device with memory more than 3600M
@ 2013-06-13 14:54 ` Paolo Bonzini
0 siblings, 0 replies; 82+ messages in thread
From: Paolo Bonzini @ 2013-06-13 14:54 UTC (permalink / raw)
To: George Dunlap
Cc: Yongjie Ren, xen-devel, Keir Fraser, Ian Campbell,
Stefano Stabellini, Xudong Hao, hanweidong, Tim Deegan,
qemu-devel, yanqiangjun, wangzhenguo, xiaowei.yang, arei.gonglei,
Jan Beulich, luonengjun, YongweiX Xu, SongtaoX Liu
Il 13/06/2013 09:54, George Dunlap ha scritto:
>
> Also, as I understand it, at the moment:
> 1. Some operating systems (32-bit XP) won't be able to use relocated
> devices
> 2. Some devices (without 64-bit BARs) can't be relocated
Are there really devices with huge 32-bit BARs? I think #1 is the only
real problem, so far though it never was for KVM.
SeaBIOS sorts the BARs from smallest to largest alignment, and then from
smallest to largest size. Typically only the GPU will be relocated.
Paolo
> 3. qemu-traditional is fine with a resized <4GiB MMIO hole.
>
> So if we have #1 or #2, at the moment an option for a work-around is to
> use qemu-traditional.
>
> However, if we add your "print an error if the device doesn't fit", then
> this option will go away -- this will be a regression in functionality
> from 4.2.
>
> I thought that what we had proposed was to have an option in xenstore,
> that libxl would set, which would instruct hvmloader whether to expand
> the MMIO hole and whether to relocate devices above 64-bit?
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [Qemu-devel] [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
2013-06-13 13:54 ` George Dunlap
@ 2013-06-13 15:16 ` Ian Campbell
-1 siblings, 0 replies; 82+ messages in thread
From: Ian Campbell @ 2013-06-13 15:16 UTC (permalink / raw)
To: George Dunlap
Cc: Tim Deegan, Yongjie Ren, yanqiangjun, Keir Fraser, hanweidong,
Xudong Hao, Stefano Stabellini, luonengjun, qemu-devel,
wangzhenguo, xiaowei.yang, arei.gonglei, Jan Beulich,
Paolo Bonzini, YongweiX Xu, SongtaoX Liu, xen-devel
On Thu, 2013-06-13 at 14:54 +0100, George Dunlap wrote:
> On 13/06/13 14:44, Stefano Stabellini wrote:
> > On Wed, 12 Jun 2013, George Dunlap wrote:
> >> On 12/06/13 08:25, Jan Beulich wrote:
> >>>>>> On 11.06.13 at 19:26, Stefano Stabellini
> >>>>>> <stefano.stabellini@eu.citrix.com> wrote:
> >>>> I went through the code that maps the PCI MMIO regions in hvmloader
> >>>> (tools/firmware/hvmloader/pci.c:pci_setup) and it looks like it already
> >>>> maps the PCI region to high memory if the PCI bar is 64-bit and the MMIO
> >>>> region is larger than 512MB.
> >>>>
> >>>> Maybe we could just relax this condition and map the device memory to
> >>>> high memory no matter the size of the MMIO region if the PCI bar is
> >>>> 64-bit?
> >>> I can only recommend not to: For one, guests not using PAE or
> >>> PSE-36 can't map such space at all (and older OSes may not
> >>> properly deal with 64-bit BARs at all). And then one would generally
> >>> expect this allocation to be done top down (to minimize risk of
> >>> running into RAM), and doing so is going to present further risks of
> >>> incompatibilities with guest OSes (Linux for example learned only in
> >>> 2.6.36 that PFNs in ioremap() can exceed 32 bits, but even in
> >>> 3.10-rc5 ioremap_pte_range(), while using "u64 pfn", passes the
> >>> PFN to pfn_pte(), the respective parameter of which is
> >>> "unsigned long").
> >>>
> >>> I think this ought to be done in an iterative process - if all MMIO
> >>> regions together don't fit below 4G, the biggest one should be
> >>> moved up beyond 4G first, followed by the next to biggest one
> >>> etc.
> >> First of all, the proposal to move the PCI BAR up to the 64-bit range is a
> >> temporary work-around. It should only be done if a device doesn't fit in the
> >> current MMIO range.
> >>
> >> We have three options here:
> >> 1. Don't do anything
> >> 2. Have hvmloader move PCI devices up to the 64-bit MMIO hole if they don't
> >> fit
> >> 3. Convince qemu to allow MMIO regions to mask memory (or what it thinks is
> >> memory).
> >> 4. Add a mechanism to tell qemu that memory is being relocated.
> >>
> >> Number 4 is definitely the right answer long-term, but we just don't have time
> >> to do that before the 4.3 release. We're not sure yet if #3 is possible; even
> >> if it is, it may have unpredictable knock-on effects.
> >>
> >> Doing #2, it is true that many guests will be unable to access the device
> >> because of 32-bit limitations. However, in #1, *no* guests will be able to
> >> access the device. At least in #2, *many* guests will be able to do so. In
> >> any case, apparently #2 is what KVM does, so having the limitation on guests
> >> is not without precedent. It's also likely to be a somewhat tested
> >> configuration (unlike #3, for example).
> > I would avoid #3, because I don't think is a good idea to rely on that
> > behaviour.
> > I would also avoid #4, because having seen QEMU's code, it's wouldn't be
> > easy and certainly not doable in time for 4.3.
> >
> > So we are left to play with the PCI MMIO region size and location in
> > hvmloader.
> >
> > I agree with Jan that we shouldn't relocate unconditionally all the
> > devices to the region above 4G. I meant to say that we should relocate
> > only the ones that don't fit. And we shouldn't try to dynamically
> > increase the PCI hole below 4G because clearly that doesn't work.
> > However we could still increase the size of the PCI hole below 4G by
> > default from start at 0xf0000000 to starting at 0xe0000000.
> > Why do we know that is safe? Because in the current configuration
> > hvmloader *already* increases the PCI hole size by decreasing the start
> > address every time a device doesn't fit.
> > So it's already common for hvmloader to set pci_mem_start to
> > 0xe0000000, you just need to assign a device with a PCI hole size big
> > enough.
Isn't this the exact case which is broken? And therefore not known safe
at all?
> >
> >
> > My proposed solution is:
> >
> > - set 0xe0000000 as the default PCI hole start for everybody, including
> > qemu-xen-traditional
What is the impact on existing qemu-trad guests?
It does mean that guest which were installed with a bit less than 4GB
RAM may now find a little bit of RAM moves above 4GB to make room for
the bigger whole. If they can dynamically enable PAE that might be ok.
Does this have any impact on Windows activation?
> > - move above 4G everything that doesn't fit and support 64-bit bars
> > - print an error if the device doesn't fit and doesn't support 64-bit
> > bars
>
> Also, as I understand it, at the moment:
> 1. Some operating systems (32-bit XP) won't be able to use relocated devices
> 2. Some devices (without 64-bit BARs) can't be relocated
> 3. qemu-traditional is fine with a resized <4GiB MMIO hole.
>
> So if we have #1 or #2, at the moment an option for a work-around is to
> use qemu-traditional.
>
> However, if we add your "print an error if the device doesn't fit", then
> this option will go away -- this will be a regression in functionality
> from 4.2.
Only if print an error also involves aborting. It could print an error
(lets call it a warning) and continue, which would leave the workaround
viable.
> I thought that what we had proposed was to have an option in xenstore,
> that libxl would set, which would instruct hvmloader whether to expand
> the MMIO hole and whether to relocate devices above 64-bit?
>
> -George
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [BUG 1747]Guest could't find bootable device with memory more than 3600M
@ 2013-06-13 15:16 ` Ian Campbell
0 siblings, 0 replies; 82+ messages in thread
From: Ian Campbell @ 2013-06-13 15:16 UTC (permalink / raw)
To: George Dunlap
Cc: Tim Deegan, Yongjie Ren, yanqiangjun, Keir Fraser, hanweidong,
Xudong Hao, Stefano Stabellini, luonengjun, qemu-devel,
wangzhenguo, xiaowei.yang, arei.gonglei, Jan Beulich,
Paolo Bonzini, YongweiX Xu, SongtaoX Liu, xen-devel
On Thu, 2013-06-13 at 14:54 +0100, George Dunlap wrote:
> On 13/06/13 14:44, Stefano Stabellini wrote:
> > On Wed, 12 Jun 2013, George Dunlap wrote:
> >> On 12/06/13 08:25, Jan Beulich wrote:
> >>>>>> On 11.06.13 at 19:26, Stefano Stabellini
> >>>>>> <stefano.stabellini@eu.citrix.com> wrote:
> >>>> I went through the code that maps the PCI MMIO regions in hvmloader
> >>>> (tools/firmware/hvmloader/pci.c:pci_setup) and it looks like it already
> >>>> maps the PCI region to high memory if the PCI bar is 64-bit and the MMIO
> >>>> region is larger than 512MB.
> >>>>
> >>>> Maybe we could just relax this condition and map the device memory to
> >>>> high memory no matter the size of the MMIO region if the PCI bar is
> >>>> 64-bit?
> >>> I can only recommend not to: For one, guests not using PAE or
> >>> PSE-36 can't map such space at all (and older OSes may not
> >>> properly deal with 64-bit BARs at all). And then one would generally
> >>> expect this allocation to be done top down (to minimize risk of
> >>> running into RAM), and doing so is going to present further risks of
> >>> incompatibilities with guest OSes (Linux for example learned only in
> >>> 2.6.36 that PFNs in ioremap() can exceed 32 bits, but even in
> >>> 3.10-rc5 ioremap_pte_range(), while using "u64 pfn", passes the
> >>> PFN to pfn_pte(), the respective parameter of which is
> >>> "unsigned long").
> >>>
> >>> I think this ought to be done in an iterative process - if all MMIO
> >>> regions together don't fit below 4G, the biggest one should be
> >>> moved up beyond 4G first, followed by the next to biggest one
> >>> etc.
> >> First of all, the proposal to move the PCI BAR up to the 64-bit range is a
> >> temporary work-around. It should only be done if a device doesn't fit in the
> >> current MMIO range.
> >>
> >> We have three options here:
> >> 1. Don't do anything
> >> 2. Have hvmloader move PCI devices up to the 64-bit MMIO hole if they don't
> >> fit
> >> 3. Convince qemu to allow MMIO regions to mask memory (or what it thinks is
> >> memory).
> >> 4. Add a mechanism to tell qemu that memory is being relocated.
> >>
> >> Number 4 is definitely the right answer long-term, but we just don't have time
> >> to do that before the 4.3 release. We're not sure yet if #3 is possible; even
> >> if it is, it may have unpredictable knock-on effects.
> >>
> >> Doing #2, it is true that many guests will be unable to access the device
> >> because of 32-bit limitations. However, in #1, *no* guests will be able to
> >> access the device. At least in #2, *many* guests will be able to do so. In
> >> any case, apparently #2 is what KVM does, so having the limitation on guests
> >> is not without precedent. It's also likely to be a somewhat tested
> >> configuration (unlike #3, for example).
> > I would avoid #3, because I don't think is a good idea to rely on that
> > behaviour.
> > I would also avoid #4, because having seen QEMU's code, it's wouldn't be
> > easy and certainly not doable in time for 4.3.
> >
> > So we are left to play with the PCI MMIO region size and location in
> > hvmloader.
> >
> > I agree with Jan that we shouldn't relocate unconditionally all the
> > devices to the region above 4G. I meant to say that we should relocate
> > only the ones that don't fit. And we shouldn't try to dynamically
> > increase the PCI hole below 4G because clearly that doesn't work.
> > However we could still increase the size of the PCI hole below 4G by
> > default from start at 0xf0000000 to starting at 0xe0000000.
> > Why do we know that is safe? Because in the current configuration
> > hvmloader *already* increases the PCI hole size by decreasing the start
> > address every time a device doesn't fit.
> > So it's already common for hvmloader to set pci_mem_start to
> > 0xe0000000, you just need to assign a device with a PCI hole size big
> > enough.
Isn't this the exact case which is broken? And therefore not known safe
at all?
> >
> >
> > My proposed solution is:
> >
> > - set 0xe0000000 as the default PCI hole start for everybody, including
> > qemu-xen-traditional
What is the impact on existing qemu-trad guests?
It does mean that guest which were installed with a bit less than 4GB
RAM may now find a little bit of RAM moves above 4GB to make room for
the bigger whole. If they can dynamically enable PAE that might be ok.
Does this have any impact on Windows activation?
> > - move above 4G everything that doesn't fit and support 64-bit bars
> > - print an error if the device doesn't fit and doesn't support 64-bit
> > bars
>
> Also, as I understand it, at the moment:
> 1. Some operating systems (32-bit XP) won't be able to use relocated devices
> 2. Some devices (without 64-bit BARs) can't be relocated
> 3. qemu-traditional is fine with a resized <4GiB MMIO hole.
>
> So if we have #1 or #2, at the moment an option for a work-around is to
> use qemu-traditional.
>
> However, if we add your "print an error if the device doesn't fit", then
> this option will go away -- this will be a regression in functionality
> from 4.2.
Only if print an error also involves aborting. It could print an error
(lets call it a warning) and continue, which would leave the workaround
viable.
> I thought that what we had proposed was to have an option in xenstore,
> that libxl would set, which would instruct hvmloader whether to expand
> the MMIO hole and whether to relocate devices above 64-bit?
>
> -George
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [Qemu-devel] [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
2013-06-13 15:16 ` Ian Campbell
@ 2013-06-13 15:30 ` George Dunlap
-1 siblings, 0 replies; 82+ messages in thread
From: George Dunlap @ 2013-06-13 15:30 UTC (permalink / raw)
To: Ian Campbell
Cc: Tim Deegan, Yongjie Ren, yanqiangjun, Keir Fraser, hanweidong,
Xudong Hao, Stefano Stabellini, luonengjun, qemu-devel,
wangzhenguo, xiaowei.yang, arei.gonglei, Jan Beulich,
Paolo Bonzini, YongweiX Xu, SongtaoX Liu, xen-devel
On 13/06/13 16:16, Ian Campbell wrote:
> On Thu, 2013-06-13 at 14:54 +0100, George Dunlap wrote:
>> On 13/06/13 14:44, Stefano Stabellini wrote:
>>> On Wed, 12 Jun 2013, George Dunlap wrote:
>>>> On 12/06/13 08:25, Jan Beulich wrote:
>>>>>>>> On 11.06.13 at 19:26, Stefano Stabellini
>>>>>>>> <stefano.stabellini@eu.citrix.com> wrote:
>>>>>> I went through the code that maps the PCI MMIO regions in hvmloader
>>>>>> (tools/firmware/hvmloader/pci.c:pci_setup) and it looks like it already
>>>>>> maps the PCI region to high memory if the PCI bar is 64-bit and the MMIO
>>>>>> region is larger than 512MB.
>>>>>>
>>>>>> Maybe we could just relax this condition and map the device memory to
>>>>>> high memory no matter the size of the MMIO region if the PCI bar is
>>>>>> 64-bit?
>>>>> I can only recommend not to: For one, guests not using PAE or
>>>>> PSE-36 can't map such space at all (and older OSes may not
>>>>> properly deal with 64-bit BARs at all). And then one would generally
>>>>> expect this allocation to be done top down (to minimize risk of
>>>>> running into RAM), and doing so is going to present further risks of
>>>>> incompatibilities with guest OSes (Linux for example learned only in
>>>>> 2.6.36 that PFNs in ioremap() can exceed 32 bits, but even in
>>>>> 3.10-rc5 ioremap_pte_range(), while using "u64 pfn", passes the
>>>>> PFN to pfn_pte(), the respective parameter of which is
>>>>> "unsigned long").
>>>>>
>>>>> I think this ought to be done in an iterative process - if all MMIO
>>>>> regions together don't fit below 4G, the biggest one should be
>>>>> moved up beyond 4G first, followed by the next to biggest one
>>>>> etc.
>>>> First of all, the proposal to move the PCI BAR up to the 64-bit range is a
>>>> temporary work-around. It should only be done if a device doesn't fit in the
>>>> current MMIO range.
>>>>
>>>> We have three options here:
>>>> 1. Don't do anything
>>>> 2. Have hvmloader move PCI devices up to the 64-bit MMIO hole if they don't
>>>> fit
>>>> 3. Convince qemu to allow MMIO regions to mask memory (or what it thinks is
>>>> memory).
>>>> 4. Add a mechanism to tell qemu that memory is being relocated.
>>>>
>>>> Number 4 is definitely the right answer long-term, but we just don't have time
>>>> to do that before the 4.3 release. We're not sure yet if #3 is possible; even
>>>> if it is, it may have unpredictable knock-on effects.
>>>>
>>>> Doing #2, it is true that many guests will be unable to access the device
>>>> because of 32-bit limitations. However, in #1, *no* guests will be able to
>>>> access the device. At least in #2, *many* guests will be able to do so. In
>>>> any case, apparently #2 is what KVM does, so having the limitation on guests
>>>> is not without precedent. It's also likely to be a somewhat tested
>>>> configuration (unlike #3, for example).
>>> I would avoid #3, because I don't think is a good idea to rely on that
>>> behaviour.
>>> I would also avoid #4, because having seen QEMU's code, it's wouldn't be
>>> easy and certainly not doable in time for 4.3.
>>>
>>> So we are left to play with the PCI MMIO region size and location in
>>> hvmloader.
>>>
>>> I agree with Jan that we shouldn't relocate unconditionally all the
>>> devices to the region above 4G. I meant to say that we should relocate
>>> only the ones that don't fit. And we shouldn't try to dynamically
>>> increase the PCI hole below 4G because clearly that doesn't work.
>>> However we could still increase the size of the PCI hole below 4G by
>>> default from start at 0xf0000000 to starting at 0xe0000000.
>>> Why do we know that is safe? Because in the current configuration
>>> hvmloader *already* increases the PCI hole size by decreasing the start
>>> address every time a device doesn't fit.
>>> So it's already common for hvmloader to set pci_mem_start to
>>> 0xe0000000, you just need to assign a device with a PCI hole size big
>>> enough.
> Isn't this the exact case which is broken? And therefore not known safe
> at all?
>
>>>
>>> My proposed solution is:
>>>
>>> - set 0xe0000000 as the default PCI hole start for everybody, including
>>> qemu-xen-traditional
> What is the impact on existing qemu-trad guests?
>
> It does mean that guest which were installed with a bit less than 4GB
> RAM may now find a little bit of RAM moves above 4GB to make room for
> the bigger whole. If they can dynamically enable PAE that might be ok.
>
> Does this have any impact on Windows activation?
>
>>> - move above 4G everything that doesn't fit and support 64-bit bars
>>> - print an error if the device doesn't fit and doesn't support 64-bit
>>> bars
>> Also, as I understand it, at the moment:
>> 1. Some operating systems (32-bit XP) won't be able to use relocated devices
>> 2. Some devices (without 64-bit BARs) can't be relocated
>> 3. qemu-traditional is fine with a resized <4GiB MMIO hole.
>>
>> So if we have #1 or #2, at the moment an option for a work-around is to
>> use qemu-traditional.
>>
>> However, if we add your "print an error if the device doesn't fit", then
>> this option will go away -- this will be a regression in functionality
>> from 4.2.
> Only if print an error also involves aborting. It could print an error
> (lets call it a warning) and continue, which would leave the workaround
> viable.\
No, because if hvmloader doesn't increase the size of the MMIO hole,
then the device won't actually work. The guest will boot, but the OS
will not be able to use it.
-George
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
@ 2013-06-13 15:30 ` George Dunlap
0 siblings, 0 replies; 82+ messages in thread
From: George Dunlap @ 2013-06-13 15:30 UTC (permalink / raw)
To: Ian Campbell
Cc: Tim Deegan, Yongjie Ren, yanqiangjun, Keir Fraser, hanweidong,
Xudong Hao, Stefano Stabellini, luonengjun, qemu-devel,
wangzhenguo, xiaowei.yang, arei.gonglei, Jan Beulich,
Paolo Bonzini, YongweiX Xu, SongtaoX Liu, xen-devel
On 13/06/13 16:16, Ian Campbell wrote:
> On Thu, 2013-06-13 at 14:54 +0100, George Dunlap wrote:
>> On 13/06/13 14:44, Stefano Stabellini wrote:
>>> On Wed, 12 Jun 2013, George Dunlap wrote:
>>>> On 12/06/13 08:25, Jan Beulich wrote:
>>>>>>>> On 11.06.13 at 19:26, Stefano Stabellini
>>>>>>>> <stefano.stabellini@eu.citrix.com> wrote:
>>>>>> I went through the code that maps the PCI MMIO regions in hvmloader
>>>>>> (tools/firmware/hvmloader/pci.c:pci_setup) and it looks like it already
>>>>>> maps the PCI region to high memory if the PCI bar is 64-bit and the MMIO
>>>>>> region is larger than 512MB.
>>>>>>
>>>>>> Maybe we could just relax this condition and map the device memory to
>>>>>> high memory no matter the size of the MMIO region if the PCI bar is
>>>>>> 64-bit?
>>>>> I can only recommend not to: For one, guests not using PAE or
>>>>> PSE-36 can't map such space at all (and older OSes may not
>>>>> properly deal with 64-bit BARs at all). And then one would generally
>>>>> expect this allocation to be done top down (to minimize risk of
>>>>> running into RAM), and doing so is going to present further risks of
>>>>> incompatibilities with guest OSes (Linux for example learned only in
>>>>> 2.6.36 that PFNs in ioremap() can exceed 32 bits, but even in
>>>>> 3.10-rc5 ioremap_pte_range(), while using "u64 pfn", passes the
>>>>> PFN to pfn_pte(), the respective parameter of which is
>>>>> "unsigned long").
>>>>>
>>>>> I think this ought to be done in an iterative process - if all MMIO
>>>>> regions together don't fit below 4G, the biggest one should be
>>>>> moved up beyond 4G first, followed by the next to biggest one
>>>>> etc.
>>>> First of all, the proposal to move the PCI BAR up to the 64-bit range is a
>>>> temporary work-around. It should only be done if a device doesn't fit in the
>>>> current MMIO range.
>>>>
>>>> We have three options here:
>>>> 1. Don't do anything
>>>> 2. Have hvmloader move PCI devices up to the 64-bit MMIO hole if they don't
>>>> fit
>>>> 3. Convince qemu to allow MMIO regions to mask memory (or what it thinks is
>>>> memory).
>>>> 4. Add a mechanism to tell qemu that memory is being relocated.
>>>>
>>>> Number 4 is definitely the right answer long-term, but we just don't have time
>>>> to do that before the 4.3 release. We're not sure yet if #3 is possible; even
>>>> if it is, it may have unpredictable knock-on effects.
>>>>
>>>> Doing #2, it is true that many guests will be unable to access the device
>>>> because of 32-bit limitations. However, in #1, *no* guests will be able to
>>>> access the device. At least in #2, *many* guests will be able to do so. In
>>>> any case, apparently #2 is what KVM does, so having the limitation on guests
>>>> is not without precedent. It's also likely to be a somewhat tested
>>>> configuration (unlike #3, for example).
>>> I would avoid #3, because I don't think is a good idea to rely on that
>>> behaviour.
>>> I would also avoid #4, because having seen QEMU's code, it's wouldn't be
>>> easy and certainly not doable in time for 4.3.
>>>
>>> So we are left to play with the PCI MMIO region size and location in
>>> hvmloader.
>>>
>>> I agree with Jan that we shouldn't relocate unconditionally all the
>>> devices to the region above 4G. I meant to say that we should relocate
>>> only the ones that don't fit. And we shouldn't try to dynamically
>>> increase the PCI hole below 4G because clearly that doesn't work.
>>> However we could still increase the size of the PCI hole below 4G by
>>> default from start at 0xf0000000 to starting at 0xe0000000.
>>> Why do we know that is safe? Because in the current configuration
>>> hvmloader *already* increases the PCI hole size by decreasing the start
>>> address every time a device doesn't fit.
>>> So it's already common for hvmloader to set pci_mem_start to
>>> 0xe0000000, you just need to assign a device with a PCI hole size big
>>> enough.
> Isn't this the exact case which is broken? And therefore not known safe
> at all?
>
>>>
>>> My proposed solution is:
>>>
>>> - set 0xe0000000 as the default PCI hole start for everybody, including
>>> qemu-xen-traditional
> What is the impact on existing qemu-trad guests?
>
> It does mean that guest which were installed with a bit less than 4GB
> RAM may now find a little bit of RAM moves above 4GB to make room for
> the bigger whole. If they can dynamically enable PAE that might be ok.
>
> Does this have any impact on Windows activation?
>
>>> - move above 4G everything that doesn't fit and support 64-bit bars
>>> - print an error if the device doesn't fit and doesn't support 64-bit
>>> bars
>> Also, as I understand it, at the moment:
>> 1. Some operating systems (32-bit XP) won't be able to use relocated devices
>> 2. Some devices (without 64-bit BARs) can't be relocated
>> 3. qemu-traditional is fine with a resized <4GiB MMIO hole.
>>
>> So if we have #1 or #2, at the moment an option for a work-around is to
>> use qemu-traditional.
>>
>> However, if we add your "print an error if the device doesn't fit", then
>> this option will go away -- this will be a regression in functionality
>> from 4.2.
> Only if print an error also involves aborting. It could print an error
> (lets call it a warning) and continue, which would leave the workaround
> viable.\
No, because if hvmloader doesn't increase the size of the MMIO hole,
then the device won't actually work. The guest will boot, but the OS
will not be able to use it.
-George
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [Qemu-devel] [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
2013-06-13 15:30 ` George Dunlap
@ 2013-06-13 15:36 ` Ian Campbell
-1 siblings, 0 replies; 82+ messages in thread
From: Ian Campbell @ 2013-06-13 15:36 UTC (permalink / raw)
To: George Dunlap
Cc: Tim Deegan, Yongjie Ren, yanqiangjun, Keir Fraser, hanweidong,
Xudong Hao, Stefano Stabellini, luonengjun, qemu-devel,
wangzhenguo, xiaowei.yang, arei.gonglei, Jan Beulich,
Paolo Bonzini, YongweiX Xu, SongtaoX Liu, xen-devel
On Thu, 2013-06-13 at 16:30 +0100, George Dunlap wrote:
> On 13/06/13 16:16, Ian Campbell wrote:
> > On Thu, 2013-06-13 at 14:54 +0100, George Dunlap wrote:
> >> On 13/06/13 14:44, Stefano Stabellini wrote:
> >>> On Wed, 12 Jun 2013, George Dunlap wrote:
> >>>> On 12/06/13 08:25, Jan Beulich wrote:
> >>>>>>>> On 11.06.13 at 19:26, Stefano Stabellini
> >>>>>>>> <stefano.stabellini@eu.citrix.com> wrote:
> >>>>>> I went through the code that maps the PCI MMIO regions in hvmloader
> >>>>>> (tools/firmware/hvmloader/pci.c:pci_setup) and it looks like it already
> >>>>>> maps the PCI region to high memory if the PCI bar is 64-bit and the MMIO
> >>>>>> region is larger than 512MB.
> >>>>>>
> >>>>>> Maybe we could just relax this condition and map the device memory to
> >>>>>> high memory no matter the size of the MMIO region if the PCI bar is
> >>>>>> 64-bit?
> >>>>> I can only recommend not to: For one, guests not using PAE or
> >>>>> PSE-36 can't map such space at all (and older OSes may not
> >>>>> properly deal with 64-bit BARs at all). And then one would generally
> >>>>> expect this allocation to be done top down (to minimize risk of
> >>>>> running into RAM), and doing so is going to present further risks of
> >>>>> incompatibilities with guest OSes (Linux for example learned only in
> >>>>> 2.6.36 that PFNs in ioremap() can exceed 32 bits, but even in
> >>>>> 3.10-rc5 ioremap_pte_range(), while using "u64 pfn", passes the
> >>>>> PFN to pfn_pte(), the respective parameter of which is
> >>>>> "unsigned long").
> >>>>>
> >>>>> I think this ought to be done in an iterative process - if all MMIO
> >>>>> regions together don't fit below 4G, the biggest one should be
> >>>>> moved up beyond 4G first, followed by the next to biggest one
> >>>>> etc.
> >>>> First of all, the proposal to move the PCI BAR up to the 64-bit range is a
> >>>> temporary work-around. It should only be done if a device doesn't fit in the
> >>>> current MMIO range.
> >>>>
> >>>> We have three options here:
> >>>> 1. Don't do anything
> >>>> 2. Have hvmloader move PCI devices up to the 64-bit MMIO hole if they don't
> >>>> fit
> >>>> 3. Convince qemu to allow MMIO regions to mask memory (or what it thinks is
> >>>> memory).
> >>>> 4. Add a mechanism to tell qemu that memory is being relocated.
> >>>>
> >>>> Number 4 is definitely the right answer long-term, but we just don't have time
> >>>> to do that before the 4.3 release. We're not sure yet if #3 is possible; even
> >>>> if it is, it may have unpredictable knock-on effects.
> >>>>
> >>>> Doing #2, it is true that many guests will be unable to access the device
> >>>> because of 32-bit limitations. However, in #1, *no* guests will be able to
> >>>> access the device. At least in #2, *many* guests will be able to do so. In
> >>>> any case, apparently #2 is what KVM does, so having the limitation on guests
> >>>> is not without precedent. It's also likely to be a somewhat tested
> >>>> configuration (unlike #3, for example).
> >>> I would avoid #3, because I don't think is a good idea to rely on that
> >>> behaviour.
> >>> I would also avoid #4, because having seen QEMU's code, it's wouldn't be
> >>> easy and certainly not doable in time for 4.3.
> >>>
> >>> So we are left to play with the PCI MMIO region size and location in
> >>> hvmloader.
> >>>
> >>> I agree with Jan that we shouldn't relocate unconditionally all the
> >>> devices to the region above 4G. I meant to say that we should relocate
> >>> only the ones that don't fit. And we shouldn't try to dynamically
> >>> increase the PCI hole below 4G because clearly that doesn't work.
> >>> However we could still increase the size of the PCI hole below 4G by
> >>> default from start at 0xf0000000 to starting at 0xe0000000.
> >>> Why do we know that is safe? Because in the current configuration
> >>> hvmloader *already* increases the PCI hole size by decreasing the start
> >>> address every time a device doesn't fit.
> >>> So it's already common for hvmloader to set pci_mem_start to
> >>> 0xe0000000, you just need to assign a device with a PCI hole size big
> >>> enough.
> > Isn't this the exact case which is broken? And therefore not known safe
> > at all?
> >
> >>>
> >>> My proposed solution is:
> >>>
> >>> - set 0xe0000000 as the default PCI hole start for everybody, including
> >>> qemu-xen-traditional
> > What is the impact on existing qemu-trad guests?
> >
> > It does mean that guest which were installed with a bit less than 4GB
> > RAM may now find a little bit of RAM moves above 4GB to make room for
> > the bigger whole. If they can dynamically enable PAE that might be ok.
> >
> > Does this have any impact on Windows activation?
> >
> >>> - move above 4G everything that doesn't fit and support 64-bit bars
> >>> - print an error if the device doesn't fit and doesn't support 64-bit
> >>> bars
> >> Also, as I understand it, at the moment:
> >> 1. Some operating systems (32-bit XP) won't be able to use relocated devices
> >> 2. Some devices (without 64-bit BARs) can't be relocated
> >> 3. qemu-traditional is fine with a resized <4GiB MMIO hole.
> >>
> >> So if we have #1 or #2, at the moment an option for a work-around is to
> >> use qemu-traditional.
> >>
> >> However, if we add your "print an error if the device doesn't fit", then
> >> this option will go away -- this will be a regression in functionality
> >> from 4.2.
> > Only if print an error also involves aborting. It could print an error
> > (lets call it a warning) and continue, which would leave the workaround
> > viable.\
>
> No, because if hvmloader doesn't increase the size of the MMIO hole,
> then the device won't actually work. The guest will boot, but the OS
> will not be able to use it.
I meant continue as in increasing the hole too, although rereading the
thread maybe that's not what everyone else was talking about ;-)
>
> -George
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
@ 2013-06-13 15:36 ` Ian Campbell
0 siblings, 0 replies; 82+ messages in thread
From: Ian Campbell @ 2013-06-13 15:36 UTC (permalink / raw)
To: George Dunlap
Cc: Tim Deegan, Yongjie Ren, yanqiangjun, Keir Fraser, hanweidong,
Xudong Hao, Stefano Stabellini, luonengjun, qemu-devel,
wangzhenguo, xiaowei.yang, arei.gonglei, Jan Beulich,
Paolo Bonzini, YongweiX Xu, SongtaoX Liu, xen-devel
On Thu, 2013-06-13 at 16:30 +0100, George Dunlap wrote:
> On 13/06/13 16:16, Ian Campbell wrote:
> > On Thu, 2013-06-13 at 14:54 +0100, George Dunlap wrote:
> >> On 13/06/13 14:44, Stefano Stabellini wrote:
> >>> On Wed, 12 Jun 2013, George Dunlap wrote:
> >>>> On 12/06/13 08:25, Jan Beulich wrote:
> >>>>>>>> On 11.06.13 at 19:26, Stefano Stabellini
> >>>>>>>> <stefano.stabellini@eu.citrix.com> wrote:
> >>>>>> I went through the code that maps the PCI MMIO regions in hvmloader
> >>>>>> (tools/firmware/hvmloader/pci.c:pci_setup) and it looks like it already
> >>>>>> maps the PCI region to high memory if the PCI bar is 64-bit and the MMIO
> >>>>>> region is larger than 512MB.
> >>>>>>
> >>>>>> Maybe we could just relax this condition and map the device memory to
> >>>>>> high memory no matter the size of the MMIO region if the PCI bar is
> >>>>>> 64-bit?
> >>>>> I can only recommend not to: For one, guests not using PAE or
> >>>>> PSE-36 can't map such space at all (and older OSes may not
> >>>>> properly deal with 64-bit BARs at all). And then one would generally
> >>>>> expect this allocation to be done top down (to minimize risk of
> >>>>> running into RAM), and doing so is going to present further risks of
> >>>>> incompatibilities with guest OSes (Linux for example learned only in
> >>>>> 2.6.36 that PFNs in ioremap() can exceed 32 bits, but even in
> >>>>> 3.10-rc5 ioremap_pte_range(), while using "u64 pfn", passes the
> >>>>> PFN to pfn_pte(), the respective parameter of which is
> >>>>> "unsigned long").
> >>>>>
> >>>>> I think this ought to be done in an iterative process - if all MMIO
> >>>>> regions together don't fit below 4G, the biggest one should be
> >>>>> moved up beyond 4G first, followed by the next to biggest one
> >>>>> etc.
> >>>> First of all, the proposal to move the PCI BAR up to the 64-bit range is a
> >>>> temporary work-around. It should only be done if a device doesn't fit in the
> >>>> current MMIO range.
> >>>>
> >>>> We have three options here:
> >>>> 1. Don't do anything
> >>>> 2. Have hvmloader move PCI devices up to the 64-bit MMIO hole if they don't
> >>>> fit
> >>>> 3. Convince qemu to allow MMIO regions to mask memory (or what it thinks is
> >>>> memory).
> >>>> 4. Add a mechanism to tell qemu that memory is being relocated.
> >>>>
> >>>> Number 4 is definitely the right answer long-term, but we just don't have time
> >>>> to do that before the 4.3 release. We're not sure yet if #3 is possible; even
> >>>> if it is, it may have unpredictable knock-on effects.
> >>>>
> >>>> Doing #2, it is true that many guests will be unable to access the device
> >>>> because of 32-bit limitations. However, in #1, *no* guests will be able to
> >>>> access the device. At least in #2, *many* guests will be able to do so. In
> >>>> any case, apparently #2 is what KVM does, so having the limitation on guests
> >>>> is not without precedent. It's also likely to be a somewhat tested
> >>>> configuration (unlike #3, for example).
> >>> I would avoid #3, because I don't think is a good idea to rely on that
> >>> behaviour.
> >>> I would also avoid #4, because having seen QEMU's code, it's wouldn't be
> >>> easy and certainly not doable in time for 4.3.
> >>>
> >>> So we are left to play with the PCI MMIO region size and location in
> >>> hvmloader.
> >>>
> >>> I agree with Jan that we shouldn't relocate unconditionally all the
> >>> devices to the region above 4G. I meant to say that we should relocate
> >>> only the ones that don't fit. And we shouldn't try to dynamically
> >>> increase the PCI hole below 4G because clearly that doesn't work.
> >>> However we could still increase the size of the PCI hole below 4G by
> >>> default from start at 0xf0000000 to starting at 0xe0000000.
> >>> Why do we know that is safe? Because in the current configuration
> >>> hvmloader *already* increases the PCI hole size by decreasing the start
> >>> address every time a device doesn't fit.
> >>> So it's already common for hvmloader to set pci_mem_start to
> >>> 0xe0000000, you just need to assign a device with a PCI hole size big
> >>> enough.
> > Isn't this the exact case which is broken? And therefore not known safe
> > at all?
> >
> >>>
> >>> My proposed solution is:
> >>>
> >>> - set 0xe0000000 as the default PCI hole start for everybody, including
> >>> qemu-xen-traditional
> > What is the impact on existing qemu-trad guests?
> >
> > It does mean that guest which were installed with a bit less than 4GB
> > RAM may now find a little bit of RAM moves above 4GB to make room for
> > the bigger whole. If they can dynamically enable PAE that might be ok.
> >
> > Does this have any impact on Windows activation?
> >
> >>> - move above 4G everything that doesn't fit and support 64-bit bars
> >>> - print an error if the device doesn't fit and doesn't support 64-bit
> >>> bars
> >> Also, as I understand it, at the moment:
> >> 1. Some operating systems (32-bit XP) won't be able to use relocated devices
> >> 2. Some devices (without 64-bit BARs) can't be relocated
> >> 3. qemu-traditional is fine with a resized <4GiB MMIO hole.
> >>
> >> So if we have #1 or #2, at the moment an option for a work-around is to
> >> use qemu-traditional.
> >>
> >> However, if we add your "print an error if the device doesn't fit", then
> >> this option will go away -- this will be a regression in functionality
> >> from 4.2.
> > Only if print an error also involves aborting. It could print an error
> > (lets call it a warning) and continue, which would leave the workaround
> > viable.\
>
> No, because if hvmloader doesn't increase the size of the MMIO hole,
> then the device won't actually work. The guest will boot, but the OS
> will not be able to use it.
I meant continue as in increasing the hole too, although rereading the
thread maybe that's not what everyone else was talking about ;-)
>
> -George
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [Qemu-devel] [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
2013-06-13 15:36 ` Ian Campbell
@ 2013-06-13 15:40 ` George Dunlap
-1 siblings, 0 replies; 82+ messages in thread
From: George Dunlap @ 2013-06-13 15:40 UTC (permalink / raw)
To: Ian Campbell
Cc: Tim Deegan, Yongjie Ren, yanqiangjun, Keir Fraser, hanweidong,
Xudong Hao, Stefano Stabellini, luonengjun, qemu-devel,
wangzhenguo, xiaowei.yang, arei.gonglei, Jan Beulich,
Paolo Bonzini, YongweiX Xu, SongtaoX Liu, xen-devel
On 13/06/13 16:36, Ian Campbell wrote:
> On Thu, 2013-06-13 at 16:30 +0100, George Dunlap wrote:
>> On 13/06/13 16:16, Ian Campbell wrote:
>>> On Thu, 2013-06-13 at 14:54 +0100, George Dunlap wrote:
>>>> On 13/06/13 14:44, Stefano Stabellini wrote:
>>>>> On Wed, 12 Jun 2013, George Dunlap wrote:
>>>>>> On 12/06/13 08:25, Jan Beulich wrote:
>>>>>>>>>> On 11.06.13 at 19:26, Stefano Stabellini
>>>>>>>>>> <stefano.stabellini@eu.citrix.com> wrote:
>>>>>>>> I went through the code that maps the PCI MMIO regions in hvmloader
>>>>>>>> (tools/firmware/hvmloader/pci.c:pci_setup) and it looks like it already
>>>>>>>> maps the PCI region to high memory if the PCI bar is 64-bit and the MMIO
>>>>>>>> region is larger than 512MB.
>>>>>>>>
>>>>>>>> Maybe we could just relax this condition and map the device memory to
>>>>>>>> high memory no matter the size of the MMIO region if the PCI bar is
>>>>>>>> 64-bit?
>>>>>>> I can only recommend not to: For one, guests not using PAE or
>>>>>>> PSE-36 can't map such space at all (and older OSes may not
>>>>>>> properly deal with 64-bit BARs at all). And then one would generally
>>>>>>> expect this allocation to be done top down (to minimize risk of
>>>>>>> running into RAM), and doing so is going to present further risks of
>>>>>>> incompatibilities with guest OSes (Linux for example learned only in
>>>>>>> 2.6.36 that PFNs in ioremap() can exceed 32 bits, but even in
>>>>>>> 3.10-rc5 ioremap_pte_range(), while using "u64 pfn", passes the
>>>>>>> PFN to pfn_pte(), the respective parameter of which is
>>>>>>> "unsigned long").
>>>>>>>
>>>>>>> I think this ought to be done in an iterative process - if all MMIO
>>>>>>> regions together don't fit below 4G, the biggest one should be
>>>>>>> moved up beyond 4G first, followed by the next to biggest one
>>>>>>> etc.
>>>>>> First of all, the proposal to move the PCI BAR up to the 64-bit range is a
>>>>>> temporary work-around. It should only be done if a device doesn't fit in the
>>>>>> current MMIO range.
>>>>>>
>>>>>> We have three options here:
>>>>>> 1. Don't do anything
>>>>>> 2. Have hvmloader move PCI devices up to the 64-bit MMIO hole if they don't
>>>>>> fit
>>>>>> 3. Convince qemu to allow MMIO regions to mask memory (or what it thinks is
>>>>>> memory).
>>>>>> 4. Add a mechanism to tell qemu that memory is being relocated.
>>>>>>
>>>>>> Number 4 is definitely the right answer long-term, but we just don't have time
>>>>>> to do that before the 4.3 release. We're not sure yet if #3 is possible; even
>>>>>> if it is, it may have unpredictable knock-on effects.
>>>>>>
>>>>>> Doing #2, it is true that many guests will be unable to access the device
>>>>>> because of 32-bit limitations. However, in #1, *no* guests will be able to
>>>>>> access the device. At least in #2, *many* guests will be able to do so. In
>>>>>> any case, apparently #2 is what KVM does, so having the limitation on guests
>>>>>> is not without precedent. It's also likely to be a somewhat tested
>>>>>> configuration (unlike #3, for example).
>>>>> I would avoid #3, because I don't think is a good idea to rely on that
>>>>> behaviour.
>>>>> I would also avoid #4, because having seen QEMU's code, it's wouldn't be
>>>>> easy and certainly not doable in time for 4.3.
>>>>>
>>>>> So we are left to play with the PCI MMIO region size and location in
>>>>> hvmloader.
>>>>>
>>>>> I agree with Jan that we shouldn't relocate unconditionally all the
>>>>> devices to the region above 4G. I meant to say that we should relocate
>>>>> only the ones that don't fit. And we shouldn't try to dynamically
>>>>> increase the PCI hole below 4G because clearly that doesn't work.
>>>>> However we could still increase the size of the PCI hole below 4G by
>>>>> default from start at 0xf0000000 to starting at 0xe0000000.
>>>>> Why do we know that is safe? Because in the current configuration
>>>>> hvmloader *already* increases the PCI hole size by decreasing the start
>>>>> address every time a device doesn't fit.
>>>>> So it's already common for hvmloader to set pci_mem_start to
>>>>> 0xe0000000, you just need to assign a device with a PCI hole size big
>>>>> enough.
>>> Isn't this the exact case which is broken? And therefore not known safe
>>> at all?
>>>
>>>>> My proposed solution is:
>>>>>
>>>>> - set 0xe0000000 as the default PCI hole start for everybody, including
>>>>> qemu-xen-traditional
>>> What is the impact on existing qemu-trad guests?
>>>
>>> It does mean that guest which were installed with a bit less than 4GB
>>> RAM may now find a little bit of RAM moves above 4GB to make room for
>>> the bigger whole. If they can dynamically enable PAE that might be ok.
>>>
>>> Does this have any impact on Windows activation?
>>>
>>>>> - move above 4G everything that doesn't fit and support 64-bit bars
>>>>> - print an error if the device doesn't fit and doesn't support 64-bit
>>>>> bars
>>>> Also, as I understand it, at the moment:
>>>> 1. Some operating systems (32-bit XP) won't be able to use relocated devices
>>>> 2. Some devices (without 64-bit BARs) can't be relocated
>>>> 3. qemu-traditional is fine with a resized <4GiB MMIO hole.
>>>>
>>>> So if we have #1 or #2, at the moment an option for a work-around is to
>>>> use qemu-traditional.
>>>>
>>>> However, if we add your "print an error if the device doesn't fit", then
>>>> this option will go away -- this will be a regression in functionality
>>>> from 4.2.
>>> Only if print an error also involves aborting. It could print an error
>>> (lets call it a warning) and continue, which would leave the workaround
>>> viable.\
>> No, because if hvmloader doesn't increase the size of the MMIO hole,
>> then the device won't actually work. The guest will boot, but the OS
>> will not be able to use it.
> I meant continue as in increasing the hole too, although rereading the
> thread maybe that's not what everyone else was talking about ;-)
Well if you continue increasing the hole, then it works on
qemu-traditional but on qemu-xen you have weird crashes and guest hangs
at some point in the future when qemu tries to map a non-existent guest
memory address -- that's much worse than the device just not being
visible to the OS.
That's the point -- current behavior on qemu-xen causes weird hangs; but
the simple way of preventing those hangs (just not increasing the MMIO
hole size) removes functionality from both qemu-xen and
qemu-traditional, even though qemu-traditional doesn't have any problems
with the resized MMIO hole.
So there's no simple way to avoid random crashes while keeping the
work-around functional; that's why someone suggested adding a xenstore
key to tell hvmloader what to do.
At least, that's what I understood the situation to be -- someone
correct me if I'm wrong. :-)
-George
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
@ 2013-06-13 15:40 ` George Dunlap
0 siblings, 0 replies; 82+ messages in thread
From: George Dunlap @ 2013-06-13 15:40 UTC (permalink / raw)
To: Ian Campbell
Cc: Tim Deegan, Yongjie Ren, yanqiangjun, Keir Fraser, hanweidong,
Xudong Hao, Stefano Stabellini, luonengjun, qemu-devel,
wangzhenguo, xiaowei.yang, arei.gonglei, Jan Beulich,
Paolo Bonzini, YongweiX Xu, SongtaoX Liu, xen-devel
On 13/06/13 16:36, Ian Campbell wrote:
> On Thu, 2013-06-13 at 16:30 +0100, George Dunlap wrote:
>> On 13/06/13 16:16, Ian Campbell wrote:
>>> On Thu, 2013-06-13 at 14:54 +0100, George Dunlap wrote:
>>>> On 13/06/13 14:44, Stefano Stabellini wrote:
>>>>> On Wed, 12 Jun 2013, George Dunlap wrote:
>>>>>> On 12/06/13 08:25, Jan Beulich wrote:
>>>>>>>>>> On 11.06.13 at 19:26, Stefano Stabellini
>>>>>>>>>> <stefano.stabellini@eu.citrix.com> wrote:
>>>>>>>> I went through the code that maps the PCI MMIO regions in hvmloader
>>>>>>>> (tools/firmware/hvmloader/pci.c:pci_setup) and it looks like it already
>>>>>>>> maps the PCI region to high memory if the PCI bar is 64-bit and the MMIO
>>>>>>>> region is larger than 512MB.
>>>>>>>>
>>>>>>>> Maybe we could just relax this condition and map the device memory to
>>>>>>>> high memory no matter the size of the MMIO region if the PCI bar is
>>>>>>>> 64-bit?
>>>>>>> I can only recommend not to: For one, guests not using PAE or
>>>>>>> PSE-36 can't map such space at all (and older OSes may not
>>>>>>> properly deal with 64-bit BARs at all). And then one would generally
>>>>>>> expect this allocation to be done top down (to minimize risk of
>>>>>>> running into RAM), and doing so is going to present further risks of
>>>>>>> incompatibilities with guest OSes (Linux for example learned only in
>>>>>>> 2.6.36 that PFNs in ioremap() can exceed 32 bits, but even in
>>>>>>> 3.10-rc5 ioremap_pte_range(), while using "u64 pfn", passes the
>>>>>>> PFN to pfn_pte(), the respective parameter of which is
>>>>>>> "unsigned long").
>>>>>>>
>>>>>>> I think this ought to be done in an iterative process - if all MMIO
>>>>>>> regions together don't fit below 4G, the biggest one should be
>>>>>>> moved up beyond 4G first, followed by the next to biggest one
>>>>>>> etc.
>>>>>> First of all, the proposal to move the PCI BAR up to the 64-bit range is a
>>>>>> temporary work-around. It should only be done if a device doesn't fit in the
>>>>>> current MMIO range.
>>>>>>
>>>>>> We have three options here:
>>>>>> 1. Don't do anything
>>>>>> 2. Have hvmloader move PCI devices up to the 64-bit MMIO hole if they don't
>>>>>> fit
>>>>>> 3. Convince qemu to allow MMIO regions to mask memory (or what it thinks is
>>>>>> memory).
>>>>>> 4. Add a mechanism to tell qemu that memory is being relocated.
>>>>>>
>>>>>> Number 4 is definitely the right answer long-term, but we just don't have time
>>>>>> to do that before the 4.3 release. We're not sure yet if #3 is possible; even
>>>>>> if it is, it may have unpredictable knock-on effects.
>>>>>>
>>>>>> Doing #2, it is true that many guests will be unable to access the device
>>>>>> because of 32-bit limitations. However, in #1, *no* guests will be able to
>>>>>> access the device. At least in #2, *many* guests will be able to do so. In
>>>>>> any case, apparently #2 is what KVM does, so having the limitation on guests
>>>>>> is not without precedent. It's also likely to be a somewhat tested
>>>>>> configuration (unlike #3, for example).
>>>>> I would avoid #3, because I don't think is a good idea to rely on that
>>>>> behaviour.
>>>>> I would also avoid #4, because having seen QEMU's code, it's wouldn't be
>>>>> easy and certainly not doable in time for 4.3.
>>>>>
>>>>> So we are left to play with the PCI MMIO region size and location in
>>>>> hvmloader.
>>>>>
>>>>> I agree with Jan that we shouldn't relocate unconditionally all the
>>>>> devices to the region above 4G. I meant to say that we should relocate
>>>>> only the ones that don't fit. And we shouldn't try to dynamically
>>>>> increase the PCI hole below 4G because clearly that doesn't work.
>>>>> However we could still increase the size of the PCI hole below 4G by
>>>>> default from start at 0xf0000000 to starting at 0xe0000000.
>>>>> Why do we know that is safe? Because in the current configuration
>>>>> hvmloader *already* increases the PCI hole size by decreasing the start
>>>>> address every time a device doesn't fit.
>>>>> So it's already common for hvmloader to set pci_mem_start to
>>>>> 0xe0000000, you just need to assign a device with a PCI hole size big
>>>>> enough.
>>> Isn't this the exact case which is broken? And therefore not known safe
>>> at all?
>>>
>>>>> My proposed solution is:
>>>>>
>>>>> - set 0xe0000000 as the default PCI hole start for everybody, including
>>>>> qemu-xen-traditional
>>> What is the impact on existing qemu-trad guests?
>>>
>>> It does mean that guest which were installed with a bit less than 4GB
>>> RAM may now find a little bit of RAM moves above 4GB to make room for
>>> the bigger whole. If they can dynamically enable PAE that might be ok.
>>>
>>> Does this have any impact on Windows activation?
>>>
>>>>> - move above 4G everything that doesn't fit and support 64-bit bars
>>>>> - print an error if the device doesn't fit and doesn't support 64-bit
>>>>> bars
>>>> Also, as I understand it, at the moment:
>>>> 1. Some operating systems (32-bit XP) won't be able to use relocated devices
>>>> 2. Some devices (without 64-bit BARs) can't be relocated
>>>> 3. qemu-traditional is fine with a resized <4GiB MMIO hole.
>>>>
>>>> So if we have #1 or #2, at the moment an option for a work-around is to
>>>> use qemu-traditional.
>>>>
>>>> However, if we add your "print an error if the device doesn't fit", then
>>>> this option will go away -- this will be a regression in functionality
>>>> from 4.2.
>>> Only if print an error also involves aborting. It could print an error
>>> (lets call it a warning) and continue, which would leave the workaround
>>> viable.\
>> No, because if hvmloader doesn't increase the size of the MMIO hole,
>> then the device won't actually work. The guest will boot, but the OS
>> will not be able to use it.
> I meant continue as in increasing the hole too, although rereading the
> thread maybe that's not what everyone else was talking about ;-)
Well if you continue increasing the hole, then it works on
qemu-traditional but on qemu-xen you have weird crashes and guest hangs
at some point in the future when qemu tries to map a non-existent guest
memory address -- that's much worse than the device just not being
visible to the OS.
That's the point -- current behavior on qemu-xen causes weird hangs; but
the simple way of preventing those hangs (just not increasing the MMIO
hole size) removes functionality from both qemu-xen and
qemu-traditional, even though qemu-traditional doesn't have any problems
with the resized MMIO hole.
So there's no simple way to avoid random crashes while keeping the
work-around functional; that's why someone suggested adding a xenstore
key to tell hvmloader what to do.
At least, that's what I understood the situation to be -- someone
correct me if I'm wrong. :-)
-George
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [Qemu-devel] [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
2013-06-13 15:40 ` George Dunlap
@ 2013-06-13 15:42 ` Ian Campbell
-1 siblings, 0 replies; 82+ messages in thread
From: Ian Campbell @ 2013-06-13 15:42 UTC (permalink / raw)
To: George Dunlap
Cc: Tim Deegan, Yongjie Ren, yanqiangjun, Keir Fraser, hanweidong,
Xudong Hao, Stefano Stabellini, luonengjun, qemu-devel,
wangzhenguo, xiaowei.yang, arei.gonglei, Jan Beulich,
Paolo Bonzini, YongweiX Xu, SongtaoX Liu, xen-devel
On Thu, 2013-06-13 at 16:40 +0100, George Dunlap wrote:
> On 13/06/13 16:36, Ian Campbell wrote:
> > On Thu, 2013-06-13 at 16:30 +0100, George Dunlap wrote:
> >> On 13/06/13 16:16, Ian Campbell wrote:
> >>> On Thu, 2013-06-13 at 14:54 +0100, George Dunlap wrote:
> >>>> On 13/06/13 14:44, Stefano Stabellini wrote:
> >>>>> On Wed, 12 Jun 2013, George Dunlap wrote:
> >>>>>> On 12/06/13 08:25, Jan Beulich wrote:
> >>>>>>>>>> On 11.06.13 at 19:26, Stefano Stabellini
> >>>>>>>>>> <stefano.stabellini@eu.citrix.com> wrote:
> >>>>>>>> I went through the code that maps the PCI MMIO regions in hvmloader
> >>>>>>>> (tools/firmware/hvmloader/pci.c:pci_setup) and it looks like it already
> >>>>>>>> maps the PCI region to high memory if the PCI bar is 64-bit and the MMIO
> >>>>>>>> region is larger than 512MB.
> >>>>>>>>
> >>>>>>>> Maybe we could just relax this condition and map the device memory to
> >>>>>>>> high memory no matter the size of the MMIO region if the PCI bar is
> >>>>>>>> 64-bit?
> >>>>>>> I can only recommend not to: For one, guests not using PAE or
> >>>>>>> PSE-36 can't map such space at all (and older OSes may not
> >>>>>>> properly deal with 64-bit BARs at all). And then one would generally
> >>>>>>> expect this allocation to be done top down (to minimize risk of
> >>>>>>> running into RAM), and doing so is going to present further risks of
> >>>>>>> incompatibilities with guest OSes (Linux for example learned only in
> >>>>>>> 2.6.36 that PFNs in ioremap() can exceed 32 bits, but even in
> >>>>>>> 3.10-rc5 ioremap_pte_range(), while using "u64 pfn", passes the
> >>>>>>> PFN to pfn_pte(), the respective parameter of which is
> >>>>>>> "unsigned long").
> >>>>>>>
> >>>>>>> I think this ought to be done in an iterative process - if all MMIO
> >>>>>>> regions together don't fit below 4G, the biggest one should be
> >>>>>>> moved up beyond 4G first, followed by the next to biggest one
> >>>>>>> etc.
> >>>>>> First of all, the proposal to move the PCI BAR up to the 64-bit range is a
> >>>>>> temporary work-around. It should only be done if a device doesn't fit in the
> >>>>>> current MMIO range.
> >>>>>>
> >>>>>> We have three options here:
> >>>>>> 1. Don't do anything
> >>>>>> 2. Have hvmloader move PCI devices up to the 64-bit MMIO hole if they don't
> >>>>>> fit
> >>>>>> 3. Convince qemu to allow MMIO regions to mask memory (or what it thinks is
> >>>>>> memory).
> >>>>>> 4. Add a mechanism to tell qemu that memory is being relocated.
> >>>>>>
> >>>>>> Number 4 is definitely the right answer long-term, but we just don't have time
> >>>>>> to do that before the 4.3 release. We're not sure yet if #3 is possible; even
> >>>>>> if it is, it may have unpredictable knock-on effects.
> >>>>>>
> >>>>>> Doing #2, it is true that many guests will be unable to access the device
> >>>>>> because of 32-bit limitations. However, in #1, *no* guests will be able to
> >>>>>> access the device. At least in #2, *many* guests will be able to do so. In
> >>>>>> any case, apparently #2 is what KVM does, so having the limitation on guests
> >>>>>> is not without precedent. It's also likely to be a somewhat tested
> >>>>>> configuration (unlike #3, for example).
> >>>>> I would avoid #3, because I don't think is a good idea to rely on that
> >>>>> behaviour.
> >>>>> I would also avoid #4, because having seen QEMU's code, it's wouldn't be
> >>>>> easy and certainly not doable in time for 4.3.
> >>>>>
> >>>>> So we are left to play with the PCI MMIO region size and location in
> >>>>> hvmloader.
> >>>>>
> >>>>> I agree with Jan that we shouldn't relocate unconditionally all the
> >>>>> devices to the region above 4G. I meant to say that we should relocate
> >>>>> only the ones that don't fit. And we shouldn't try to dynamically
> >>>>> increase the PCI hole below 4G because clearly that doesn't work.
> >>>>> However we could still increase the size of the PCI hole below 4G by
> >>>>> default from start at 0xf0000000 to starting at 0xe0000000.
> >>>>> Why do we know that is safe? Because in the current configuration
> >>>>> hvmloader *already* increases the PCI hole size by decreasing the start
> >>>>> address every time a device doesn't fit.
> >>>>> So it's already common for hvmloader to set pci_mem_start to
> >>>>> 0xe0000000, you just need to assign a device with a PCI hole size big
> >>>>> enough.
> >>> Isn't this the exact case which is broken? And therefore not known safe
> >>> at all?
> >>>
> >>>>> My proposed solution is:
> >>>>>
> >>>>> - set 0xe0000000 as the default PCI hole start for everybody, including
> >>>>> qemu-xen-traditional
> >>> What is the impact on existing qemu-trad guests?
> >>>
> >>> It does mean that guest which were installed with a bit less than 4GB
> >>> RAM may now find a little bit of RAM moves above 4GB to make room for
> >>> the bigger whole. If they can dynamically enable PAE that might be ok.
> >>>
> >>> Does this have any impact on Windows activation?
> >>>
> >>>>> - move above 4G everything that doesn't fit and support 64-bit bars
> >>>>> - print an error if the device doesn't fit and doesn't support 64-bit
> >>>>> bars
> >>>> Also, as I understand it, at the moment:
> >>>> 1. Some operating systems (32-bit XP) won't be able to use relocated devices
> >>>> 2. Some devices (without 64-bit BARs) can't be relocated
> >>>> 3. qemu-traditional is fine with a resized <4GiB MMIO hole.
> >>>>
> >>>> So if we have #1 or #2, at the moment an option for a work-around is to
> >>>> use qemu-traditional.
> >>>>
> >>>> However, if we add your "print an error if the device doesn't fit", then
> >>>> this option will go away -- this will be a regression in functionality
> >>>> from 4.2.
> >>> Only if print an error also involves aborting. It could print an error
> >>> (lets call it a warning) and continue, which would leave the workaround
> >>> viable.\
> >> No, because if hvmloader doesn't increase the size of the MMIO hole,
> >> then the device won't actually work. The guest will boot, but the OS
> >> will not be able to use it.
> > I meant continue as in increasing the hole too, although rereading the
> > thread maybe that's not what everyone else was talking about ;-)
>
> Well if you continue increasing the hole, then it works on
> qemu-traditional but on qemu-xen you have weird crashes and guest hangs
> at some point in the future when qemu tries to map a non-existent guest
> memory address -- that's much worse than the device just not being
> visible to the OS.
I thought the point of the print was simply to give us something to spot
in the logs in this latter case.
> That's the point -- current behavior on qemu-xen causes weird hangs; but
> the simple way of preventing those hangs (just not increasing the MMIO
> hole size) removes functionality from both qemu-xen and
> qemu-traditional, even though qemu-traditional doesn't have any problems
> with the resized MMIO hole.
>
> So there's no simple way to avoid random crashes while keeping the
> work-around functional; that's why someone suggested adding a xenstore
> key to tell hvmloader what to do.
>
> At least, that's what I understood the situation to be -- someone
> correct me if I'm wrong. :-)
>
> -George
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
@ 2013-06-13 15:42 ` Ian Campbell
0 siblings, 0 replies; 82+ messages in thread
From: Ian Campbell @ 2013-06-13 15:42 UTC (permalink / raw)
To: George Dunlap
Cc: Tim Deegan, Yongjie Ren, yanqiangjun, Keir Fraser, hanweidong,
Xudong Hao, Stefano Stabellini, luonengjun, qemu-devel,
wangzhenguo, xiaowei.yang, arei.gonglei, Jan Beulich,
Paolo Bonzini, YongweiX Xu, SongtaoX Liu, xen-devel
On Thu, 2013-06-13 at 16:40 +0100, George Dunlap wrote:
> On 13/06/13 16:36, Ian Campbell wrote:
> > On Thu, 2013-06-13 at 16:30 +0100, George Dunlap wrote:
> >> On 13/06/13 16:16, Ian Campbell wrote:
> >>> On Thu, 2013-06-13 at 14:54 +0100, George Dunlap wrote:
> >>>> On 13/06/13 14:44, Stefano Stabellini wrote:
> >>>>> On Wed, 12 Jun 2013, George Dunlap wrote:
> >>>>>> On 12/06/13 08:25, Jan Beulich wrote:
> >>>>>>>>>> On 11.06.13 at 19:26, Stefano Stabellini
> >>>>>>>>>> <stefano.stabellini@eu.citrix.com> wrote:
> >>>>>>>> I went through the code that maps the PCI MMIO regions in hvmloader
> >>>>>>>> (tools/firmware/hvmloader/pci.c:pci_setup) and it looks like it already
> >>>>>>>> maps the PCI region to high memory if the PCI bar is 64-bit and the MMIO
> >>>>>>>> region is larger than 512MB.
> >>>>>>>>
> >>>>>>>> Maybe we could just relax this condition and map the device memory to
> >>>>>>>> high memory no matter the size of the MMIO region if the PCI bar is
> >>>>>>>> 64-bit?
> >>>>>>> I can only recommend not to: For one, guests not using PAE or
> >>>>>>> PSE-36 can't map such space at all (and older OSes may not
> >>>>>>> properly deal with 64-bit BARs at all). And then one would generally
> >>>>>>> expect this allocation to be done top down (to minimize risk of
> >>>>>>> running into RAM), and doing so is going to present further risks of
> >>>>>>> incompatibilities with guest OSes (Linux for example learned only in
> >>>>>>> 2.6.36 that PFNs in ioremap() can exceed 32 bits, but even in
> >>>>>>> 3.10-rc5 ioremap_pte_range(), while using "u64 pfn", passes the
> >>>>>>> PFN to pfn_pte(), the respective parameter of which is
> >>>>>>> "unsigned long").
> >>>>>>>
> >>>>>>> I think this ought to be done in an iterative process - if all MMIO
> >>>>>>> regions together don't fit below 4G, the biggest one should be
> >>>>>>> moved up beyond 4G first, followed by the next to biggest one
> >>>>>>> etc.
> >>>>>> First of all, the proposal to move the PCI BAR up to the 64-bit range is a
> >>>>>> temporary work-around. It should only be done if a device doesn't fit in the
> >>>>>> current MMIO range.
> >>>>>>
> >>>>>> We have three options here:
> >>>>>> 1. Don't do anything
> >>>>>> 2. Have hvmloader move PCI devices up to the 64-bit MMIO hole if they don't
> >>>>>> fit
> >>>>>> 3. Convince qemu to allow MMIO regions to mask memory (or what it thinks is
> >>>>>> memory).
> >>>>>> 4. Add a mechanism to tell qemu that memory is being relocated.
> >>>>>>
> >>>>>> Number 4 is definitely the right answer long-term, but we just don't have time
> >>>>>> to do that before the 4.3 release. We're not sure yet if #3 is possible; even
> >>>>>> if it is, it may have unpredictable knock-on effects.
> >>>>>>
> >>>>>> Doing #2, it is true that many guests will be unable to access the device
> >>>>>> because of 32-bit limitations. However, in #1, *no* guests will be able to
> >>>>>> access the device. At least in #2, *many* guests will be able to do so. In
> >>>>>> any case, apparently #2 is what KVM does, so having the limitation on guests
> >>>>>> is not without precedent. It's also likely to be a somewhat tested
> >>>>>> configuration (unlike #3, for example).
> >>>>> I would avoid #3, because I don't think is a good idea to rely on that
> >>>>> behaviour.
> >>>>> I would also avoid #4, because having seen QEMU's code, it's wouldn't be
> >>>>> easy and certainly not doable in time for 4.3.
> >>>>>
> >>>>> So we are left to play with the PCI MMIO region size and location in
> >>>>> hvmloader.
> >>>>>
> >>>>> I agree with Jan that we shouldn't relocate unconditionally all the
> >>>>> devices to the region above 4G. I meant to say that we should relocate
> >>>>> only the ones that don't fit. And we shouldn't try to dynamically
> >>>>> increase the PCI hole below 4G because clearly that doesn't work.
> >>>>> However we could still increase the size of the PCI hole below 4G by
> >>>>> default from start at 0xf0000000 to starting at 0xe0000000.
> >>>>> Why do we know that is safe? Because in the current configuration
> >>>>> hvmloader *already* increases the PCI hole size by decreasing the start
> >>>>> address every time a device doesn't fit.
> >>>>> So it's already common for hvmloader to set pci_mem_start to
> >>>>> 0xe0000000, you just need to assign a device with a PCI hole size big
> >>>>> enough.
> >>> Isn't this the exact case which is broken? And therefore not known safe
> >>> at all?
> >>>
> >>>>> My proposed solution is:
> >>>>>
> >>>>> - set 0xe0000000 as the default PCI hole start for everybody, including
> >>>>> qemu-xen-traditional
> >>> What is the impact on existing qemu-trad guests?
> >>>
> >>> It does mean that guest which were installed with a bit less than 4GB
> >>> RAM may now find a little bit of RAM moves above 4GB to make room for
> >>> the bigger whole. If they can dynamically enable PAE that might be ok.
> >>>
> >>> Does this have any impact on Windows activation?
> >>>
> >>>>> - move above 4G everything that doesn't fit and support 64-bit bars
> >>>>> - print an error if the device doesn't fit and doesn't support 64-bit
> >>>>> bars
> >>>> Also, as I understand it, at the moment:
> >>>> 1. Some operating systems (32-bit XP) won't be able to use relocated devices
> >>>> 2. Some devices (without 64-bit BARs) can't be relocated
> >>>> 3. qemu-traditional is fine with a resized <4GiB MMIO hole.
> >>>>
> >>>> So if we have #1 or #2, at the moment an option for a work-around is to
> >>>> use qemu-traditional.
> >>>>
> >>>> However, if we add your "print an error if the device doesn't fit", then
> >>>> this option will go away -- this will be a regression in functionality
> >>>> from 4.2.
> >>> Only if print an error also involves aborting. It could print an error
> >>> (lets call it a warning) and continue, which would leave the workaround
> >>> viable.\
> >> No, because if hvmloader doesn't increase the size of the MMIO hole,
> >> then the device won't actually work. The guest will boot, but the OS
> >> will not be able to use it.
> > I meant continue as in increasing the hole too, although rereading the
> > thread maybe that's not what everyone else was talking about ;-)
>
> Well if you continue increasing the hole, then it works on
> qemu-traditional but on qemu-xen you have weird crashes and guest hangs
> at some point in the future when qemu tries to map a non-existent guest
> memory address -- that's much worse than the device just not being
> visible to the OS.
I thought the point of the print was simply to give us something to spot
in the logs in this latter case.
> That's the point -- current behavior on qemu-xen causes weird hangs; but
> the simple way of preventing those hangs (just not increasing the MMIO
> hole size) removes functionality from both qemu-xen and
> qemu-traditional, even though qemu-traditional doesn't have any problems
> with the resized MMIO hole.
>
> So there's no simple way to avoid random crashes while keeping the
> work-around functional; that's why someone suggested adding a xenstore
> key to tell hvmloader what to do.
>
> At least, that's what I understood the situation to be -- someone
> correct me if I'm wrong. :-)
>
> -George
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [Qemu-devel] [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
2013-06-13 15:16 ` Ian Campbell
@ 2013-06-13 15:40 ` Stefano Stabellini
-1 siblings, 0 replies; 82+ messages in thread
From: Stefano Stabellini @ 2013-06-13 15:40 UTC (permalink / raw)
To: Ian Campbell
Cc: Tim Deegan, Yongjie Ren, yanqiangjun, Keir Fraser, hanweidong,
George Dunlap, Xudong Hao, Stefano Stabellini, luonengjun,
qemu-devel, wangzhenguo, xiaowei.yang, arei.gonglei, Jan Beulich,
Paolo Bonzini, YongweiX Xu, SongtaoX Liu, xen-devel
On Thu, 13 Jun 2013, Ian Campbell wrote:
> On Thu, 2013-06-13 at 14:54 +0100, George Dunlap wrote:
> > On 13/06/13 14:44, Stefano Stabellini wrote:
> > > On Wed, 12 Jun 2013, George Dunlap wrote:
> > >> On 12/06/13 08:25, Jan Beulich wrote:
> > >>>>>> On 11.06.13 at 19:26, Stefano Stabellini
> > >>>>>> <stefano.stabellini@eu.citrix.com> wrote:
> > >>>> I went through the code that maps the PCI MMIO regions in hvmloader
> > >>>> (tools/firmware/hvmloader/pci.c:pci_setup) and it looks like it already
> > >>>> maps the PCI region to high memory if the PCI bar is 64-bit and the MMIO
> > >>>> region is larger than 512MB.
> > >>>>
> > >>>> Maybe we could just relax this condition and map the device memory to
> > >>>> high memory no matter the size of the MMIO region if the PCI bar is
> > >>>> 64-bit?
> > >>> I can only recommend not to: For one, guests not using PAE or
> > >>> PSE-36 can't map such space at all (and older OSes may not
> > >>> properly deal with 64-bit BARs at all). And then one would generally
> > >>> expect this allocation to be done top down (to minimize risk of
> > >>> running into RAM), and doing so is going to present further risks of
> > >>> incompatibilities with guest OSes (Linux for example learned only in
> > >>> 2.6.36 that PFNs in ioremap() can exceed 32 bits, but even in
> > >>> 3.10-rc5 ioremap_pte_range(), while using "u64 pfn", passes the
> > >>> PFN to pfn_pte(), the respective parameter of which is
> > >>> "unsigned long").
> > >>>
> > >>> I think this ought to be done in an iterative process - if all MMIO
> > >>> regions together don't fit below 4G, the biggest one should be
> > >>> moved up beyond 4G first, followed by the next to biggest one
> > >>> etc.
> > >> First of all, the proposal to move the PCI BAR up to the 64-bit range is a
> > >> temporary work-around. It should only be done if a device doesn't fit in the
> > >> current MMIO range.
> > >>
> > >> We have three options here:
> > >> 1. Don't do anything
> > >> 2. Have hvmloader move PCI devices up to the 64-bit MMIO hole if they don't
> > >> fit
> > >> 3. Convince qemu to allow MMIO regions to mask memory (or what it thinks is
> > >> memory).
> > >> 4. Add a mechanism to tell qemu that memory is being relocated.
> > >>
> > >> Number 4 is definitely the right answer long-term, but we just don't have time
> > >> to do that before the 4.3 release. We're not sure yet if #3 is possible; even
> > >> if it is, it may have unpredictable knock-on effects.
> > >>
> > >> Doing #2, it is true that many guests will be unable to access the device
> > >> because of 32-bit limitations. However, in #1, *no* guests will be able to
> > >> access the device. At least in #2, *many* guests will be able to do so. In
> > >> any case, apparently #2 is what KVM does, so having the limitation on guests
> > >> is not without precedent. It's also likely to be a somewhat tested
> > >> configuration (unlike #3, for example).
> > > I would avoid #3, because I don't think is a good idea to rely on that
> > > behaviour.
> > > I would also avoid #4, because having seen QEMU's code, it's wouldn't be
> > > easy and certainly not doable in time for 4.3.
> > >
> > > So we are left to play with the PCI MMIO region size and location in
> > > hvmloader.
> > >
> > > I agree with Jan that we shouldn't relocate unconditionally all the
> > > devices to the region above 4G. I meant to say that we should relocate
> > > only the ones that don't fit. And we shouldn't try to dynamically
> > > increase the PCI hole below 4G because clearly that doesn't work.
> > > However we could still increase the size of the PCI hole below 4G by
> > > default from start at 0xf0000000 to starting at 0xe0000000.
> > > Why do we know that is safe? Because in the current configuration
> > > hvmloader *already* increases the PCI hole size by decreasing the start
> > > address every time a device doesn't fit.
> > > So it's already common for hvmloader to set pci_mem_start to
> > > 0xe0000000, you just need to assign a device with a PCI hole size big
> > > enough.
>
> Isn't this the exact case which is broken? And therefore not known safe
> at all?
hvmloader sets pci_mem_start to 0xe0000000 and works with
qemu-xen-traditional but it doesn't with qemu-xen (before the patch that
increases the default pci hole size in QEMU).
What I was trying to say is: "it's already common for hvmloader to set
pci_mem_start to 0xe0000000 with qemu-xen-traditional".
> > > My proposed solution is:
> > >
> > > - set 0xe0000000 as the default PCI hole start for everybody, including
> > > qemu-xen-traditional
>
> What is the impact on existing qemu-trad guests?
>
> It does mean that guest which were installed with a bit less than 4GB
> RAM may now find a little bit of RAM moves above 4GB to make room for
> the bigger whole. If they can dynamically enable PAE that might be ok.
Yes, the amount of below 4G ram is going to be a bit less.
> Does this have any impact on Windows activation?
I don't think so: I assigned graphic cards with less than 512MB of
videoram to Windows guests before without compromising the Windows
license. I'll get more info on this.
> > > - move above 4G everything that doesn't fit and support 64-bit bars
> > > - print an error if the device doesn't fit and doesn't support 64-bit
> > > bars
> >
> > Also, as I understand it, at the moment:
> > 1. Some operating systems (32-bit XP) won't be able to use relocated devices
> > 2. Some devices (without 64-bit BARs) can't be relocated
> > 3. qemu-traditional is fine with a resized <4GiB MMIO hole.
> >
> > So if we have #1 or #2, at the moment an option for a work-around is to
> > use qemu-traditional.
> >
> > However, if we add your "print an error if the device doesn't fit", then
> > this option will go away -- this will be a regression in functionality
> > from 4.2.
>
> Only if print an error also involves aborting. It could print an error
> (lets call it a warning) and continue, which would leave the workaround
> viable.
Regardless of the warning we print and the strategy we decide to have to
fix the problem, we can always have a manually selectable workaround to
switch to the old behaviour if we want to (aside from the minimum PCI
hole size that might be harder to make selectable). It just
means more code in hvmloader.
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [Xen-devel] [BUG 1747]Guest could't find bootable device with memory more than 3600M
@ 2013-06-13 15:40 ` Stefano Stabellini
0 siblings, 0 replies; 82+ messages in thread
From: Stefano Stabellini @ 2013-06-13 15:40 UTC (permalink / raw)
To: Ian Campbell
Cc: Tim Deegan, Yongjie Ren, yanqiangjun, Keir Fraser, hanweidong,
George Dunlap, Xudong Hao, Stefano Stabellini, luonengjun,
qemu-devel, wangzhenguo, xiaowei.yang, arei.gonglei, Jan Beulich,
Paolo Bonzini, YongweiX Xu, SongtaoX Liu, xen-devel
On Thu, 13 Jun 2013, Ian Campbell wrote:
> On Thu, 2013-06-13 at 14:54 +0100, George Dunlap wrote:
> > On 13/06/13 14:44, Stefano Stabellini wrote:
> > > On Wed, 12 Jun 2013, George Dunlap wrote:
> > >> On 12/06/13 08:25, Jan Beulich wrote:
> > >>>>>> On 11.06.13 at 19:26, Stefano Stabellini
> > >>>>>> <stefano.stabellini@eu.citrix.com> wrote:
> > >>>> I went through the code that maps the PCI MMIO regions in hvmloader
> > >>>> (tools/firmware/hvmloader/pci.c:pci_setup) and it looks like it already
> > >>>> maps the PCI region to high memory if the PCI bar is 64-bit and the MMIO
> > >>>> region is larger than 512MB.
> > >>>>
> > >>>> Maybe we could just relax this condition and map the device memory to
> > >>>> high memory no matter the size of the MMIO region if the PCI bar is
> > >>>> 64-bit?
> > >>> I can only recommend not to: For one, guests not using PAE or
> > >>> PSE-36 can't map such space at all (and older OSes may not
> > >>> properly deal with 64-bit BARs at all). And then one would generally
> > >>> expect this allocation to be done top down (to minimize risk of
> > >>> running into RAM), and doing so is going to present further risks of
> > >>> incompatibilities with guest OSes (Linux for example learned only in
> > >>> 2.6.36 that PFNs in ioremap() can exceed 32 bits, but even in
> > >>> 3.10-rc5 ioremap_pte_range(), while using "u64 pfn", passes the
> > >>> PFN to pfn_pte(), the respective parameter of which is
> > >>> "unsigned long").
> > >>>
> > >>> I think this ought to be done in an iterative process - if all MMIO
> > >>> regions together don't fit below 4G, the biggest one should be
> > >>> moved up beyond 4G first, followed by the next to biggest one
> > >>> etc.
> > >> First of all, the proposal to move the PCI BAR up to the 64-bit range is a
> > >> temporary work-around. It should only be done if a device doesn't fit in the
> > >> current MMIO range.
> > >>
> > >> We have three options here:
> > >> 1. Don't do anything
> > >> 2. Have hvmloader move PCI devices up to the 64-bit MMIO hole if they don't
> > >> fit
> > >> 3. Convince qemu to allow MMIO regions to mask memory (or what it thinks is
> > >> memory).
> > >> 4. Add a mechanism to tell qemu that memory is being relocated.
> > >>
> > >> Number 4 is definitely the right answer long-term, but we just don't have time
> > >> to do that before the 4.3 release. We're not sure yet if #3 is possible; even
> > >> if it is, it may have unpredictable knock-on effects.
> > >>
> > >> Doing #2, it is true that many guests will be unable to access the device
> > >> because of 32-bit limitations. However, in #1, *no* guests will be able to
> > >> access the device. At least in #2, *many* guests will be able to do so. In
> > >> any case, apparently #2 is what KVM does, so having the limitation on guests
> > >> is not without precedent. It's also likely to be a somewhat tested
> > >> configuration (unlike #3, for example).
> > > I would avoid #3, because I don't think is a good idea to rely on that
> > > behaviour.
> > > I would also avoid #4, because having seen QEMU's code, it's wouldn't be
> > > easy and certainly not doable in time for 4.3.
> > >
> > > So we are left to play with the PCI MMIO region size and location in
> > > hvmloader.
> > >
> > > I agree with Jan that we shouldn't relocate unconditionally all the
> > > devices to the region above 4G. I meant to say that we should relocate
> > > only the ones that don't fit. And we shouldn't try to dynamically
> > > increase the PCI hole below 4G because clearly that doesn't work.
> > > However we could still increase the size of the PCI hole below 4G by
> > > default from start at 0xf0000000 to starting at 0xe0000000.
> > > Why do we know that is safe? Because in the current configuration
> > > hvmloader *already* increases the PCI hole size by decreasing the start
> > > address every time a device doesn't fit.
> > > So it's already common for hvmloader to set pci_mem_start to
> > > 0xe0000000, you just need to assign a device with a PCI hole size big
> > > enough.
>
> Isn't this the exact case which is broken? And therefore not known safe
> at all?
hvmloader sets pci_mem_start to 0xe0000000 and works with
qemu-xen-traditional but it doesn't with qemu-xen (before the patch that
increases the default pci hole size in QEMU).
What I was trying to say is: "it's already common for hvmloader to set
pci_mem_start to 0xe0000000 with qemu-xen-traditional".
> > > My proposed solution is:
> > >
> > > - set 0xe0000000 as the default PCI hole start for everybody, including
> > > qemu-xen-traditional
>
> What is the impact on existing qemu-trad guests?
>
> It does mean that guest which were installed with a bit less than 4GB
> RAM may now find a little bit of RAM moves above 4GB to make room for
> the bigger whole. If they can dynamically enable PAE that might be ok.
Yes, the amount of below 4G ram is going to be a bit less.
> Does this have any impact on Windows activation?
I don't think so: I assigned graphic cards with less than 512MB of
videoram to Windows guests before without compromising the Windows
license. I'll get more info on this.
> > > - move above 4G everything that doesn't fit and support 64-bit bars
> > > - print an error if the device doesn't fit and doesn't support 64-bit
> > > bars
> >
> > Also, as I understand it, at the moment:
> > 1. Some operating systems (32-bit XP) won't be able to use relocated devices
> > 2. Some devices (without 64-bit BARs) can't be relocated
> > 3. qemu-traditional is fine with a resized <4GiB MMIO hole.
> >
> > So if we have #1 or #2, at the moment an option for a work-around is to
> > use qemu-traditional.
> >
> > However, if we add your "print an error if the device doesn't fit", then
> > this option will go away -- this will be a regression in functionality
> > from 4.2.
>
> Only if print an error also involves aborting. It could print an error
> (lets call it a warning) and continue, which would leave the workaround
> viable.
Regardless of the warning we print and the strategy we decide to have to
fix the problem, we can always have a manually selectable workaround to
switch to the old behaviour if we want to (aside from the minimum PCI
hole size that might be harder to make selectable). It just
means more code in hvmloader.
^ permalink raw reply [flat|nested] 82+ messages in thread