[PATCH for-4.11] libxl: fix memory map reported to PVH guests

[PATCH for-4.11] libxl: fix memory map reported to PVH guests

[Roger Pau Monne]

PVH guests with 4GB of RAM or more get a memory map like the
following:

0x00000000000000 - 0x000000fee00000 RAM
0x000000fee00000 - 0x00000100000000 RESERVED
0x000000fc009000 - 0x000000fc009040 ACPI
0x000000fc000000 - 0x000000fc001000 ACPI
0x000000fc001000 - 0x000000fc009000 ACPI
0x00000100000000 - 0x000001fb200400 RAM

This is wrong because ACPI regions are also reported as RAM regions.
The cause of this issue is not setting a big enough MMIO hole, current
libxl code only takes into account the address of the local APIC page
and the reserved pages in order to set the size of the MMIO hole, when
it should also take into account the location of the ACPI tables.

After the fix the layout reported for the same guest is:

0x00000000000000 - 0x000000fc000000 RAM
0x000000fc000000 - 0x00000100000000 RESERVED
0x000000fc009000 - 0x000000fc009040 ACPI
0x000000fc000000 - 0x000000fc001000 ACPI
0x000000fc001000 - 0x000000fc009000 ACPI
0x00000100000000 - 0x000001fe000400 RAM

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Wei Liu <wei.liu2@citrix.com>
Cc: Juergen Gross <jgross@suse.com>
---
 tools/libxl/libxl_arch.h     |  1 +
 tools/libxl/libxl_dom.c      | 20 +++++++++++---------
 tools/libxl/libxl_x86_acpi.c |  1 -
 3 files changed, 12 insertions(+), 10 deletions(-)

diff --git a/tools/libxl/libxl_arch.h b/tools/libxl/libxl_arch.h
index 318c111bb4..74a5af3cf3 100644
--- a/tools/libxl/libxl_arch.h
+++ b/tools/libxl/libxl_arch.h
@@ -76,6 +76,7 @@ int libxl__arch_extra_memory(libxl__gc *gc,
 #if defined(__i386__) || defined(__x86_64__)
 
 #define LAPIC_BASE_ADDRESS  0xfee00000
+#define ACPI_INFO_PHYSICAL_ADDRESS 0xfc000000
 
 int libxl__dom_load_acpi(libxl__gc *gc,
                          const libxl_domain_build_info *b_info,
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index 8c3607b0f5..f0fd5fd3a3 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -1229,15 +1229,17 @@ int libxl__build_hvm(libxl__gc *gc, uint32_t domid,
         dom->mmio_size = HVM_BELOW_4G_MMIO_LENGTH;
     else if (dom->mmio_size == 0 && !device_model) {
 #if defined(__i386__) || defined(__x86_64__)
-        if (libxl_defbool_val(info->apic)) {
-            /* Make sure LAPIC_BASE_ADDRESS is below special pages */
-            assert(((((X86_HVM_END_SPECIAL_REGION - X86_HVM_NR_SPECIAL_PAGES)
-                      << XC_PAGE_SHIFT) - LAPIC_BASE_ADDRESS)) >= XC_PAGE_SIZE);
-            dom->mmio_size = GB(4) - LAPIC_BASE_ADDRESS;
-        } else
-            dom->mmio_size = GB(4) -
-                ((X86_HVM_END_SPECIAL_REGION - X86_HVM_NR_SPECIAL_PAGES)
-                 << XC_PAGE_SHIFT);
+        /*
+         * Make sure the local APIC page, the ACPI tables and the special pages
+         * are inside the MMIO hole.
+         */
+        xen_paddr_t start =
+            (X86_HVM_END_SPECIAL_REGION - X86_HVM_NR_SPECIAL_PAGES) <<
+            XC_PAGE_SHIFT;
+
+        start = min_t(xen_paddr_t, start, LAPIC_BASE_ADDRESS);
+        start = min_t(xen_paddr_t, start, ACPI_INFO_PHYSICAL_ADDRESS);
+        dom->mmio_size = GB(4) - start;
 #else
         assert(1);
 #endif
diff --git a/tools/libxl/libxl_x86_acpi.c b/tools/libxl/libxl_x86_acpi.c
index 143ce66644..ed6610c84e 100644
--- a/tools/libxl/libxl_x86_acpi.c
+++ b/tools/libxl/libxl_x86_acpi.c
@@ -22,7 +22,6 @@
 
  /* Number of pages holding ACPI tables */
 #define NUM_ACPI_PAGES 16
-#define ACPI_INFO_PHYSICAL_ADDRESS 0xfc000000
 
 struct libxl_acpi_ctxt {
     struct acpi_ctxt c;
-- 
2.17.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-4.11] libxl: fix memory map reported to PVH guests

[Juergen Gross]

On 20/04/18 15:52, Roger Pau Monne wrote:
> PVH guests with 4GB of RAM or more get a memory map like the
> following:
> 
> 0x00000000000000 - 0x000000fee00000 RAM
> 0x000000fee00000 - 0x00000100000000 RESERVED
> 0x000000fc009000 - 0x000000fc009040 ACPI
> 0x000000fc000000 - 0x000000fc001000 ACPI
> 0x000000fc001000 - 0x000000fc009000 ACPI
> 0x00000100000000 - 0x000001fb200400 RAM
> 
> This is wrong because ACPI regions are also reported as RAM regions.
> The cause of this issue is not setting a big enough MMIO hole, current
> libxl code only takes into account the address of the local APIC page
> and the reserved pages in order to set the size of the MMIO hole, when
> it should also take into account the location of the ACPI tables.
> 
> After the fix the layout reported for the same guest is:
> 
> 0x00000000000000 - 0x000000fc000000 RAM
> 0x000000fc000000 - 0x00000100000000 RESERVED
> 0x000000fc009000 - 0x000000fc009040 ACPI
> 0x000000fc000000 - 0x000000fc001000 ACPI
> 0x000000fc001000 - 0x000000fc009000 ACPI
> 0x00000100000000 - 0x000001fe000400 RAM
> 
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>

Release-acked-by: Juergen Gross <jgross@suse.com>


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-4.11] libxl: fix memory map reported to PVH guests

[Jan Beulich]

>>> On 20.04.18 at 15:52, <roger.pau@citrix.com> wrote:
> PVH guests with 4GB of RAM or more get a memory map like the
> following:
> 
> 0x00000000000000 - 0x000000fee00000 RAM
> 0x000000fee00000 - 0x00000100000000 RESERVED
> 0x000000fc009000 - 0x000000fc009040 ACPI
> 0x000000fc000000 - 0x000000fc001000 ACPI
> 0x000000fc001000 - 0x000000fc009000 ACPI
> 0x00000100000000 - 0x000001fb200400 RAM
> 
> This is wrong because ACPI regions are also reported as RAM regions.
> The cause of this issue is not setting a big enough MMIO hole, current
> libxl code only takes into account the address of the local APIC page
> and the reserved pages in order to set the size of the MMIO hole, when
> it should also take into account the location of the ACPI tables.
> 
> After the fix the layout reported for the same guest is:
> 
> 0x00000000000000 - 0x000000fc000000 RAM
> 0x000000fc000000 - 0x00000100000000 RESERVED
> 0x000000fc009000 - 0x000000fc009040 ACPI
> 0x000000fc000000 - 0x000000fc001000 ACPI
> 0x000000fc001000 - 0x000000fc009000 ACPI
> 0x00000100000000 - 0x000001fe000400 RAM

But this is still wrong - no two regions may overlap, regardless of type.
Also you appear to be losing all RAM in [fc009000, fee00000).

Jan



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-4.11] libxl: fix memory map reported to PVH guests

[Roger Pau Monné]

On Fri, Apr 20, 2018 at 08:01:35AM -0600, Jan Beulich wrote:
> >>> On 20.04.18 at 15:52, <roger.pau@citrix.com> wrote:
> > PVH guests with 4GB of RAM or more get a memory map like the
> > following:
> > 
> > 0x00000000000000 - 0x000000fee00000 RAM
> > 0x000000fee00000 - 0x00000100000000 RESERVED
> > 0x000000fc009000 - 0x000000fc009040 ACPI
> > 0x000000fc000000 - 0x000000fc001000 ACPI
> > 0x000000fc001000 - 0x000000fc009000 ACPI
> > 0x00000100000000 - 0x000001fb200400 RAM
> > 
> > This is wrong because ACPI regions are also reported as RAM regions.
> > The cause of this issue is not setting a big enough MMIO hole, current
> > libxl code only takes into account the address of the local APIC page
> > and the reserved pages in order to set the size of the MMIO hole, when
> > it should also take into account the location of the ACPI tables.
> > 
> > After the fix the layout reported for the same guest is:
> > 
> > 0x00000000000000 - 0x000000fc000000 RAM
> > 0x000000fc000000 - 0x00000100000000 RESERVED
> > 0x000000fc009000 - 0x000000fc009040 ACPI
> > 0x000000fc000000 - 0x000000fc001000 ACPI
> > 0x000000fc001000 - 0x000000fc009000 ACPI
> > 0x00000100000000 - 0x000001fe000400 RAM
> 
> But this is still wrong - no two regions may overlap, regardless of type.

It's going to be more complicated to fix that. I can give it a try,
but I think this is strictly better than what we do now.

Maybe instead of marking the whole MMIO hole as reserved we should
only mark as reserved the lapic page and the special pages? That
should avoid any overlaps.

> Also you appear to be losing all RAM in [fc009000, fee00000).

This is not populated. Note how:

0x000001fe000400 - 0x000001fb200400 == 0x000000fee00000 - 0x000000fc000000

The memory is shifted towards the end.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-4.11] libxl: fix memory map reported to PVH guests

[Jan Beulich]

>>> On 20.04.18 at 16:15, <roger.pau@citrix.com> wrote:
> On Fri, Apr 20, 2018 at 08:01:35AM -0600, Jan Beulich wrote:
>> >>> On 20.04.18 at 15:52, <roger.pau@citrix.com> wrote:
>> > PVH guests with 4GB of RAM or more get a memory map like the
>> > following:
>> > 
>> > 0x00000000000000 - 0x000000fee00000 RAM
>> > 0x000000fee00000 - 0x00000100000000 RESERVED
>> > 0x000000fc009000 - 0x000000fc009040 ACPI
>> > 0x000000fc000000 - 0x000000fc001000 ACPI
>> > 0x000000fc001000 - 0x000000fc009000 ACPI
>> > 0x00000100000000 - 0x000001fb200400 RAM
>> > 
>> > This is wrong because ACPI regions are also reported as RAM regions.
>> > The cause of this issue is not setting a big enough MMIO hole, current
>> > libxl code only takes into account the address of the local APIC page
>> > and the reserved pages in order to set the size of the MMIO hole, when
>> > it should also take into account the location of the ACPI tables.
>> > 
>> > After the fix the layout reported for the same guest is:
>> > 
>> > 0x00000000000000 - 0x000000fc000000 RAM
>> > 0x000000fc000000 - 0x00000100000000 RESERVED
>> > 0x000000fc009000 - 0x000000fc009040 ACPI
>> > 0x000000fc000000 - 0x000000fc001000 ACPI
>> > 0x000000fc001000 - 0x000000fc009000 ACPI
>> > 0x00000100000000 - 0x000001fe000400 RAM
>> 
>> But this is still wrong - no two regions may overlap, regardless of type.
> 
> It's going to be more complicated to fix that. I can give it a try,
> but I think this is strictly better than what we do now.
> 
> Maybe instead of marking the whole MMIO hole as reserved we should
> only mark as reserved the lapic page and the special pages? That
> should avoid any overlaps.

Well, if nothing else is in that range (or can be placed there dynamically at
runtime) I don't see why all of it is reserved. Marking a range reserved
prevents, for example, PCI device BARs to be put there by the OS. The
way it looks there's no MMIO window left available at all below 4Gb ...

>> Also you appear to be losing all RAM in [fc009000, fee00000).
> 
> This is not populated. Note how:
> 
> 0x000001fe000400 - 0x000001fb200400 == 0x000000fee00000 - 0x000000fc000000
> 
> The memory is shifted towards the end.

Oh, I see - I didn't pay attention to the upper bound of that last entry,
I'm sorry.

Jan



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-4.11] libxl: fix memory map reported to PVH guests

[Roger Pau Monné]

On Fri, Apr 20, 2018 at 08:25:41AM -0600, Jan Beulich wrote:
> >>> On 20.04.18 at 16:15, <roger.pau@citrix.com> wrote:
> > On Fri, Apr 20, 2018 at 08:01:35AM -0600, Jan Beulich wrote:
> >> >>> On 20.04.18 at 15:52, <roger.pau@citrix.com> wrote:
> >> > PVH guests with 4GB of RAM or more get a memory map like the
> >> > following:
> >> > 
> >> > 0x00000000000000 - 0x000000fee00000 RAM
> >> > 0x000000fee00000 - 0x00000100000000 RESERVED
> >> > 0x000000fc009000 - 0x000000fc009040 ACPI
> >> > 0x000000fc000000 - 0x000000fc001000 ACPI
> >> > 0x000000fc001000 - 0x000000fc009000 ACPI
> >> > 0x00000100000000 - 0x000001fb200400 RAM
> >> > 
> >> > This is wrong because ACPI regions are also reported as RAM regions.
> >> > The cause of this issue is not setting a big enough MMIO hole, current
> >> > libxl code only takes into account the address of the local APIC page
> >> > and the reserved pages in order to set the size of the MMIO hole, when
> >> > it should also take into account the location of the ACPI tables.
> >> > 
> >> > After the fix the layout reported for the same guest is:
> >> > 
> >> > 0x00000000000000 - 0x000000fc000000 RAM
> >> > 0x000000fc000000 - 0x00000100000000 RESERVED
> >> > 0x000000fc009000 - 0x000000fc009040 ACPI
> >> > 0x000000fc000000 - 0x000000fc001000 ACPI
> >> > 0x000000fc001000 - 0x000000fc009000 ACPI
> >> > 0x00000100000000 - 0x000001fe000400 RAM
> >> 
> >> But this is still wrong - no two regions may overlap, regardless of type.
> > 
> > It's going to be more complicated to fix that. I can give it a try,
> > but I think this is strictly better than what we do now.
> > 
> > Maybe instead of marking the whole MMIO hole as reserved we should
> > only mark as reserved the lapic page and the special pages? That
> > should avoid any overlaps.
> 
> Well, if nothing else is in that range (or can be placed there dynamically at
> runtime) I don't see why all of it is reserved. Marking a range reserved
> prevents, for example, PCI device BARs to be put there by the OS. The
> way it looks there's no MMIO window left available at all below 4Gb ...

Right, this will change when PVH gets devices with BARs, then it's
going to need a proper MMIO hole below 4GB

AFAICT the only reserved pages for PVH are the HVM special pages, so
I've changed the memory map builder to only report those as reserved,
here is the resulting memory map:

0x00000000000000 - 0x000000fc000000 RAM
0x000000feff8000 - 0x000000ff000000 RESERVED
0x000000fc009000 - 0x000000fc009040 ACPI
0x000000fc000000 - 0x000000fc001000 ACPI
0x000000fc001000 - 0x000000fc009000 ACPI
0x00000100000000 - 0x000001fe000400 RAM

I see no reason to mark the local APIC page as reserved in the memory
map. If that seems like a sensible approach I can send the updated
patch.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-4.11] libxl: fix memory map reported to PVH guests

[Jan Beulich]

>>> On 20.04.18 at 16:31, <roger.pau@citrix.com> wrote:
> AFAICT the only reserved pages for PVH are the HVM special pages, so
> I've changed the memory map builder to only report those as reserved,
> here is the resulting memory map:
> 
> 0x00000000000000 - 0x000000fc000000 RAM
> 0x000000feff8000 - 0x000000ff000000 RESERVED
> 0x000000fc009000 - 0x000000fc009040 ACPI
> 0x000000fc000000 - 0x000000fc001000 ACPI
> 0x000000fc001000 - 0x000000fc009000 ACPI
> 0x00000100000000 - 0x000001fe000400 RAM

LGTM

> I see no reason to mark the local APIC page as reserved in the memory
> map. If that seems like a sensible approach I can send the updated
> patch.

Right - commonly bare hardware doesn't have this page marked reserved
either.

Jan



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-4.11] libxl: fix memory map reported to PVH guests

[Andrew Cooper]

On 20/04/18 15:31, Roger Pau Monné wrote:
> On Fri, Apr 20, 2018 at 08:25:41AM -0600, Jan Beulich wrote:
>>>>> On 20.04.18 at 16:15, <roger.pau@citrix.com> wrote:
>>> On Fri, Apr 20, 2018 at 08:01:35AM -0600, Jan Beulich wrote:
>>>>>>> On 20.04.18 at 15:52, <roger.pau@citrix.com> wrote:
>>>>> PVH guests with 4GB of RAM or more get a memory map like the
>>>>> following:
>>>>>
>>>>> 0x00000000000000 - 0x000000fee00000 RAM
>>>>> 0x000000fee00000 - 0x00000100000000 RESERVED
>>>>> 0x000000fc009000 - 0x000000fc009040 ACPI
>>>>> 0x000000fc000000 - 0x000000fc001000 ACPI
>>>>> 0x000000fc001000 - 0x000000fc009000 ACPI
>>>>> 0x00000100000000 - 0x000001fb200400 RAM
>>>>>
>>>>> This is wrong because ACPI regions are also reported as RAM regions.
>>>>> The cause of this issue is not setting a big enough MMIO hole, current
>>>>> libxl code only takes into account the address of the local APIC page
>>>>> and the reserved pages in order to set the size of the MMIO hole, when
>>>>> it should also take into account the location of the ACPI tables.
>>>>>
>>>>> After the fix the layout reported for the same guest is:
>>>>>
>>>>> 0x00000000000000 - 0x000000fc000000 RAM
>>>>> 0x000000fc000000 - 0x00000100000000 RESERVED
>>>>> 0x000000fc009000 - 0x000000fc009040 ACPI
>>>>> 0x000000fc000000 - 0x000000fc001000 ACPI
>>>>> 0x000000fc001000 - 0x000000fc009000 ACPI
>>>>> 0x00000100000000 - 0x000001fe000400 RAM
>>>> But this is still wrong - no two regions may overlap, regardless of type.
>>> It's going to be more complicated to fix that. I can give it a try,
>>> but I think this is strictly better than what we do now.
>>>
>>> Maybe instead of marking the whole MMIO hole as reserved we should
>>> only mark as reserved the lapic page and the special pages? That
>>> should avoid any overlaps.
>> Well, if nothing else is in that range (or can be placed there dynamically at
>> runtime) I don't see why all of it is reserved. Marking a range reserved
>> prevents, for example, PCI device BARs to be put there by the OS. The
>> way it looks there's no MMIO window left available at all below 4Gb ...
> Right, this will change when PVH gets devices with BARs, then it's
> going to need a proper MMIO hole below 4GB

Whomever wants to make PVH guests work properly fast:

The only way this work while retaining 1G host superpages is to
unilaterally split at the 3G mark and continue from the 4G boundary. 
The ACPI tables and other ram-based tables can grow down from 3G, and
there is plenty of room for the LAPIC to work, a sensibly sized MMIO
hole, and gfn room to put mappings into without shattering superpages.

Then we've got to see about how to not use MTRRs...

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-4.11] libxl: fix memory map reported to PVH guests

[Juergen Gross]

On 20/04/18 17:09, Andrew Cooper wrote:
> On 20/04/18 15:31, Roger Pau Monné wrote:
>> On Fri, Apr 20, 2018 at 08:25:41AM -0600, Jan Beulich wrote:
>>>>>> On 20.04.18 at 16:15, <roger.pau@citrix.com> wrote:
>>>> On Fri, Apr 20, 2018 at 08:01:35AM -0600, Jan Beulich wrote:
>>>>>>>> On 20.04.18 at 15:52, <roger.pau@citrix.com> wrote:
>>>>>> PVH guests with 4GB of RAM or more get a memory map like the
>>>>>> following:
>>>>>>
>>>>>> 0x00000000000000 - 0x000000fee00000 RAM
>>>>>> 0x000000fee00000 - 0x00000100000000 RESERVED
>>>>>> 0x000000fc009000 - 0x000000fc009040 ACPI
>>>>>> 0x000000fc000000 - 0x000000fc001000 ACPI
>>>>>> 0x000000fc001000 - 0x000000fc009000 ACPI
>>>>>> 0x00000100000000 - 0x000001fb200400 RAM
>>>>>>
>>>>>> This is wrong because ACPI regions are also reported as RAM regions.
>>>>>> The cause of this issue is not setting a big enough MMIO hole, current
>>>>>> libxl code only takes into account the address of the local APIC page
>>>>>> and the reserved pages in order to set the size of the MMIO hole, when
>>>>>> it should also take into account the location of the ACPI tables.
>>>>>>
>>>>>> After the fix the layout reported for the same guest is:
>>>>>>
>>>>>> 0x00000000000000 - 0x000000fc000000 RAM
>>>>>> 0x000000fc000000 - 0x00000100000000 RESERVED
>>>>>> 0x000000fc009000 - 0x000000fc009040 ACPI
>>>>>> 0x000000fc000000 - 0x000000fc001000 ACPI
>>>>>> 0x000000fc001000 - 0x000000fc009000 ACPI
>>>>>> 0x00000100000000 - 0x000001fe000400 RAM
>>>>> But this is still wrong - no two regions may overlap, regardless of type.
>>>> It's going to be more complicated to fix that. I can give it a try,
>>>> but I think this is strictly better than what we do now.
>>>>
>>>> Maybe instead of marking the whole MMIO hole as reserved we should
>>>> only mark as reserved the lapic page and the special pages? That
>>>> should avoid any overlaps.
>>> Well, if nothing else is in that range (or can be placed there dynamically at
>>> runtime) I don't see why all of it is reserved. Marking a range reserved
>>> prevents, for example, PCI device BARs to be put there by the OS. The
>>> way it looks there's no MMIO window left available at all below 4Gb ...
>> Right, this will change when PVH gets devices with BARs, then it's
>> going to need a proper MMIO hole below 4GB
> 
> Whomever wants to make PVH guests work properly fast:
> 
> The only way this work while retaining 1G host superpages is to
> unilaterally split at the 3G mark and continue from the 4G boundary. 
> The ACPI tables and other ram-based tables can grow down from 3G, and
> there is plenty of room for the LAPIC to work, a sensibly sized MMIO
> hole, and gfn room to put mappings into without shattering superpages.

Hmm, I'd rather put the ACPI tables between 3G and 4G. They are not
accessed that often. The guest has then no need to use smaller pages
for direct RAM mappings due to ACPI RO-mappings.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH for-4.11] libxl: fix memory map reported to PVH guests

[Roger Pau Monné]

On Fri, Apr 20, 2018 at 04:09:46PM +0100, Andrew Cooper wrote:
> On 20/04/18 15:31, Roger Pau Monné wrote:
> > On Fri, Apr 20, 2018 at 08:25:41AM -0600, Jan Beulich wrote:
> >>>>> On 20.04.18 at 16:15, <roger.pau@citrix.com> wrote:
> >>> On Fri, Apr 20, 2018 at 08:01:35AM -0600, Jan Beulich wrote:
> >>>>>>> On 20.04.18 at 15:52, <roger.pau@citrix.com> wrote:
> >>>>> PVH guests with 4GB of RAM or more get a memory map like the
> >>>>> following:
> >>>>>
> >>>>> 0x00000000000000 - 0x000000fee00000 RAM
> >>>>> 0x000000fee00000 - 0x00000100000000 RESERVED
> >>>>> 0x000000fc009000 - 0x000000fc009040 ACPI
> >>>>> 0x000000fc000000 - 0x000000fc001000 ACPI
> >>>>> 0x000000fc001000 - 0x000000fc009000 ACPI
> >>>>> 0x00000100000000 - 0x000001fb200400 RAM
> >>>>>
> >>>>> This is wrong because ACPI regions are also reported as RAM regions.
> >>>>> The cause of this issue is not setting a big enough MMIO hole, current
> >>>>> libxl code only takes into account the address of the local APIC page
> >>>>> and the reserved pages in order to set the size of the MMIO hole, when
> >>>>> it should also take into account the location of the ACPI tables.
> >>>>>
> >>>>> After the fix the layout reported for the same guest is:
> >>>>>
> >>>>> 0x00000000000000 - 0x000000fc000000 RAM
> >>>>> 0x000000fc000000 - 0x00000100000000 RESERVED
> >>>>> 0x000000fc009000 - 0x000000fc009040 ACPI
> >>>>> 0x000000fc000000 - 0x000000fc001000 ACPI
> >>>>> 0x000000fc001000 - 0x000000fc009000 ACPI
> >>>>> 0x00000100000000 - 0x000001fe000400 RAM
> >>>> But this is still wrong - no two regions may overlap, regardless of type.
> >>> It's going to be more complicated to fix that. I can give it a try,
> >>> but I think this is strictly better than what we do now.
> >>>
> >>> Maybe instead of marking the whole MMIO hole as reserved we should
> >>> only mark as reserved the lapic page and the special pages? That
> >>> should avoid any overlaps.
> >> Well, if nothing else is in that range (or can be placed there dynamically at
> >> runtime) I don't see why all of it is reserved. Marking a range reserved
> >> prevents, for example, PCI device BARs to be put there by the OS. The
> >> way it looks there's no MMIO window left available at all below 4Gb ...
> > Right, this will change when PVH gets devices with BARs, then it's
> > going to need a proper MMIO hole below 4GB
> 
> Whomever wants to make PVH guests work properly fast:
> 
> The only way this work while retaining 1G host superpages is to
> unilaterally split at the 3G mark and continue from the 4G boundary. 
> The ACPI tables and other ram-based tables can grow down from 3G, and
> there is plenty of room for the LAPIC to work, a sensibly sized MMIO
> hole, and gfn room to put mappings into without shattering superpages.

Just having the hole from 3G to 4G and the ACPI tables somewhere in
there seems good enough. At the end of day ACPI tables are not
accessed that often.

In any case, that's 4.12 material.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel