[PATCH RFC 0/4] x86/pvh: implement iommu_inclusive_mapping for PVH Dom0

[PATCH RFC 0/4] x86/pvh: implement iommu_inclusive_mapping for PVH Dom0

[Roger Pau Monne]

Hello,

Currently iommu_inclusive_mapping is not working for PVH Dom0, this patch
series allows using it for a PVH Dom0, which seems to be required in order to
boot on older boxes.

This series is RFC because IMHO in part it's just masking a more severe
underlying issue that I've seen while trying to boot a PVH Dom0 on some
(oldish) hardware. Using this patch series allows PVH Dom0 to boot, but I'm not
really sure how reliable that is (see patch 3 for more details), and whether
the IOMMUs on this hardware are actually functional.

This series is based on top of the vPCI one, which in turn is based on top of
the legacy PCI interrupt routing one.

Git branch can be found at:

git://xenbits.xen.org/people/royger/xen.git iommu_pvh_rfc

Thanks, Roger.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH RFC 1/4] x86/dom0: prevent access to MMCFG areas for PVH Dom0

[Roger Pau Monne]

They are emulated by Xen, so they must not be mapped into Dom0 p2m.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Cc: Jan Beulich <jbeulich@suse.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
---
 xen/arch/x86/dom0_build.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/xen/arch/x86/dom0_build.c b/xen/arch/x86/dom0_build.c
index 3996d9dd12..481a899afe 100644
--- a/xen/arch/x86/dom0_build.c
+++ b/xen/arch/x86/dom0_build.c
@@ -18,6 +18,8 @@
 #include <asm/p2m.h>
 #include <asm/setup.h>
 
+#include "x86_64/mmconfig.h"
+
 static long __initdata dom0_nrpages;
 static long __initdata dom0_min_nrpages;
 static long __initdata dom0_max_nrpages = LONG_MAX;
@@ -452,6 +454,22 @@ int __init dom0_setup_permissions(struct domain *d)
             rc |= rangeset_add_singleton(mmio_ro_ranges, mfn);
     }
 
+    /* For PVH prevent access to the MMCFG areas. */
+    if ( dom0_pvh && pci_mmcfg_config_num )
+    {
+        unsigned int i;
+
+        for ( i = 0; i < pci_mmcfg_config_num; i++ )
+        {
+            paddr_t addr = pci_mmcfg_config[i].address +
+                           (pci_mmcfg_config[i].start_bus_number << 20);
+            size_t size = (pci_mmcfg_config[i].end_bus_number -
+                           pci_mmcfg_config[i].start_bus_number + 1) << 20;
+
+            rc |= iomem_deny_access(d, PFN_DOWN(addr), PFN_UP(addr + size));
+        }
+    }
+
     return rc;
 }
 
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH RFC 2/4] x86/dom0: prevent PVH Dom0 from mapping read-only the IO APIC area

[Roger Pau Monne]

This is emulated by Xen and must not be mapped into PVH Dom0 p2m.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Cc: Jan Beulich <jbeulich@suse.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
---
 xen/arch/x86/dom0_build.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/xen/arch/x86/dom0_build.c b/xen/arch/x86/dom0_build.c
index 481a899afe..be3ba33a16 100644
--- a/xen/arch/x86/dom0_build.c
+++ b/xen/arch/x86/dom0_build.c
@@ -416,7 +416,7 @@ int __init dom0_setup_permissions(struct domain *d)
     for ( i = 0; i < nr_ioapics; i++ )
     {
         mfn = paddr_to_pfn(mp_ioapics[i].mpc_apicaddr);
-        if ( !rangeset_contains_singleton(mmio_ro_ranges, mfn) )
+        if ( dom0_pvh || !rangeset_contains_singleton(mmio_ro_ranges, mfn) )
             rc |= iomem_deny_access(d, mfn, mfn);
     }
     /* MSI range. */
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH RFC 3/4] x86/vtd: introduce a PVH implementation of iommu_inclusive_mapping

[Roger Pau Monne]

On certain Intel systems, as far as I can tell almost all pre-Haswell ones,
trying to boot a PVH Dom0 will freeze the box completely, up to the point that
not even the watchdog works. The freeze happens exactly when enabling the DMA
remapping in the IOMMU, the last line seen is:

(XEN) [VT-D]iommu_enable_translation: iommu->reg = ffff82c00021b000

In order to workaround this (which seems to be a lack of proper RMRR entries,
plus the IOMMU being unable to generate faults and freezing the entire system)
add a PVH specific implementation of iommu_inclusive_mapping, that maps
non-RAM, non-unusable regions into Dom0 p2m. Note that care is taken to not map
device MMIO regions that Xen is emulating, like the local APIC or the IO APIC.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Cc: Kevin Tian <kevin.tian@intel.com>
---
 xen/drivers/passthrough/vtd/extern.h  |  1 +
 xen/drivers/passthrough/vtd/iommu.c   |  2 ++
 xen/drivers/passthrough/vtd/x86/vtd.c | 39 +++++++++++++++++++++++++++++++++++
 3 files changed, 42 insertions(+)

diff --git a/xen/drivers/passthrough/vtd/extern.h b/xen/drivers/passthrough/vtd/extern.h
index fb7edfaef9..0eaf8956ff 100644
--- a/xen/drivers/passthrough/vtd/extern.h
+++ b/xen/drivers/passthrough/vtd/extern.h
@@ -100,5 +100,6 @@ bool_t platform_supports_intremap(void);
 bool_t platform_supports_x2apic(void);
 
 void vtd_set_hwdom_mapping(struct domain *d);
+void vtd_set_pvh_hwdom_mapping(struct domain *d);
 
 #endif // _VTD_EXTERN_H_
diff --git a/xen/drivers/passthrough/vtd/iommu.c b/xen/drivers/passthrough/vtd/iommu.c
index a5c61c6e21..6bbf319100 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -1293,6 +1293,8 @@ static void __hwdom_init intel_iommu_hwdom_init(struct domain *d)
         /* Set up 1:1 page table for hardware domain. */
         vtd_set_hwdom_mapping(d);
     }
+    else if ( is_hvm_domain(d) )
+        vtd_set_pvh_hwdom_mapping(d);
 
     setup_hwdom_pci_devices(d, setup_hwdom_device);
     setup_hwdom_rmrr(d);
diff --git a/xen/drivers/passthrough/vtd/x86/vtd.c b/xen/drivers/passthrough/vtd/x86/vtd.c
index 88a60b3307..79c9b0526f 100644
--- a/xen/drivers/passthrough/vtd/x86/vtd.c
+++ b/xen/drivers/passthrough/vtd/x86/vtd.c
@@ -21,10 +21,12 @@
 #include <xen/softirq.h>
 #include <xen/domain_page.h>
 #include <asm/paging.h>
+#include <xen/iocap.h>
 #include <xen/iommu.h>
 #include <xen/irq.h>
 #include <xen/numa.h>
 #include <asm/fixmap.h>
+#include <asm/p2m.h>
 #include <asm/setup.h>
 #include "../iommu.h"
 #include "../dmar.h"
@@ -159,3 +161,40 @@ void __hwdom_init vtd_set_hwdom_mapping(struct domain *d)
     }
 }
 
+void __hwdom_init vtd_set_pvh_hwdom_mapping(struct domain *d)
+{
+    unsigned long pfn;
+
+    BUG_ON(!is_hardware_domain(d));
+
+    if ( !iommu_inclusive_mapping )
+        return;
+
+    /* NB: the low 1MB is already mapped in pvh_setup_p2m. */
+    for ( pfn = PFN_DOWN(MB(1)); pfn < PFN_DOWN(GB(4)); pfn++ )
+    {
+        p2m_access_t a;
+        int rc;
+
+        if ( !(pfn & 0xfff) )
+            process_pending_softirqs();
+
+        /* Skip RAM, ACPI and unusable regions. */
+        if ( page_is_ram_type(pfn, RAM_TYPE_CONVENTIONAL) ||
+             page_is_ram_type(pfn, RAM_TYPE_UNUSABLE) ||
+             page_is_ram_type(pfn, RAM_TYPE_ACPI) ||
+             !iomem_access_permitted(d, pfn, pfn) )
+            continue;
+
+        ASSERT(!xen_in_range(pfn));
+
+        a = rangeset_contains_range(mmio_ro_ranges, pfn, pfn) ? p2m_access_r
+                                                              : p2m_access_rw;
+        rc = set_identity_p2m_entry(d, pfn, a, 0);
+        if ( rc )
+           printk(XENLOG_WARNING VTDPREFIX
+                  " d%d: IOMMU mapping failed pfn %#lx: %d\n",
+                  d->domain_id, pfn, rc);
+    }
+}
+
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH RFC 4/4] x86/dom0: re-order DMA remapping enabling for PVH Dom0

[Roger Pau Monne]

Make sure the reserved regions are setup before enabling the DMA remapping in
the IOMMU, by calling dom0_setup_permissions before iommu_hwdom_init. Also, in
order to workaround IOMMU issues seen on pre-Haswell Intel hardware, make sure
the DMA remapping is enabled after populating Dom0 p2m.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Cc: Jan Beulich <jbeulich@suse.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
---
 xen/arch/x86/hvm/dom0_build.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/xen/arch/x86/hvm/dom0_build.c b/xen/arch/x86/hvm/dom0_build.c
index 52eb738249..61073c66c6 100644
--- a/xen/arch/x86/hvm/dom0_build.c
+++ b/xen/arch/x86/hvm/dom0_build.c
@@ -587,13 +587,6 @@ static int __init pvh_setup_cpus(struct domain *d, paddr_t entry,
         return rc;
     }
 
-    rc = dom0_setup_permissions(d);
-    if ( rc )
-    {
-        panic("Unable to setup Dom0 permissions: %d\n", rc);
-        return rc;
-    }
-
     update_domain_wallclock_time(d);
 
     clear_bit(_VPF_down, &v->pause_flags);
@@ -1059,7 +1052,12 @@ int __init dom0_construct_pvh(struct domain *d, const module_t *image,
 
     printk("** Building a PVH Dom0 **\n");
 
-    iommu_hwdom_init(d);
+    rc = dom0_setup_permissions(d);
+    if ( rc )
+    {
+        printk("Unable to setup Dom0 permissions: %d\n", rc);
+        return rc;
+    }
 
     rc = pvh_setup_p2m(d);
     if ( rc )
@@ -1068,6 +1066,8 @@ int __init dom0_construct_pvh(struct domain *d, const module_t *image,
         return rc;
     }
 
+    iommu_hwdom_init(d);
+
     rc = pvh_load_kernel(d, image, image_headroom, initrd, bootstrap_map(image),
                          cmdline, &entry, &start_info);
     if ( rc )
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH RFC 1/4] x86/dom0: prevent access to MMCFG areas for PVH Dom0

[Jan Beulich]

>>> Roger Pau Monne <roger.pau@citrix.com> 04/24/17 1:52 PM >>>
>--- a/xen/arch/x86/dom0_build.c
>+++ b/xen/arch/x86/dom0_build.c
>@@ -18,6 +18,8 @@
 >#include <asm/p2m.h>
 >#include <asm/setup.h>
 >
>+#include "x86_64/mmconfig.h"

Not just but also because of this I'd prefer if this was taken care of in the
MMCFG code itself, also covering ranges which are being added post-
boot. Presumably in/from pci_mmcfg_arch_{en,dis}able().

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH RFC 2/4] x86/dom0: prevent PVH Dom0 from mapping read-only the IO APIC area

[Jan Beulich]

>>> Roger Pau Monne <roger.pau@citrix.com> 04/24/17 1:52 PM >>>
>This is emulated by Xen and must not be mapped into PVH Dom0 p2m.

Yes, but this reminds of of the reason we permit the r/o mapping in PV
Dom0, and I wonder how well the emulated variant is going to work with
ACPI methods accessing the IO-APICs. Let's hope there's not going to
be any problems.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH RFC 4/4] x86/dom0: re-order DMA remapping enabling for PVH Dom0

[Jan Beulich]

>>> Roger Pau Monne <roger.pau@citrix.com> 04/24/17 1:52 PM >>>
>Make sure the reserved regions are setup before enabling the DMA remapping in
>the IOMMU, by calling dom0_setup_permissions before iommu_hwdom_init. Also, in
>order to workaround IOMMU issues seen on pre-Haswell Intel hardware, make sure
>the DMA remapping is enabled after populating Dom0 p2m.

Looks reasonable (especially considering that iommu_hwdom_init() is being called
fairly late in the PV case as well), but please explain the "IOMMU issues seen on
pre-Haswell Intel hardware" a little here (or if it all is mean to refer to what patch 3
does, refer to it).

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH RFC 1/4] x86/dom0: prevent access to MMCFG areas for PVH Dom0

[Roger Pau Monné]

On Fri, Jun 30, 2017 at 05:27:06AM -0600, Jan Beulich wrote:
> >>> Roger Pau Monne <roger.pau@citrix.com> 04/24/17 1:52 PM >>>
> >--- a/xen/arch/x86/dom0_build.c
> >+++ b/xen/arch/x86/dom0_build.c
> >@@ -18,6 +18,8 @@
>  >#include <asm/p2m.h>
>  >#include <asm/setup.h>
>  >
> >+#include "x86_64/mmconfig.h"
> 
> Not just but also because of this I'd prefer if this was taken care of in the
> MMCFG code itself, also covering ranges which are being added post-
> boot. Presumably in/from pci_mmcfg_arch_{en,dis}able().

The problem with this approach is that at the point in the boot where
pci_mmcfg_arch_enable gets called (from acpi_mmcfg_init) the domain
has not yet been created, so it's not possible to call
iomem_deny_access, and in any case the iomem ranges are initialized in
dom0_setup_permissions, so that would get overwritten.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH RFC 2/4] x86/dom0: prevent PVH Dom0 from mapping read-only the IO APIC area

[Roger Pau Monné]

On Fri, Jun 30, 2017 at 05:31:36AM -0600, Jan Beulich wrote:
> >>> Roger Pau Monne <roger.pau@citrix.com> 04/24/17 1:52 PM >>>
> >This is emulated by Xen and must not be mapped into PVH Dom0 p2m.
> 
> Yes, but this reminds of of the reason we permit the r/o mapping in PV
> Dom0, and I wonder how well the emulated variant is going to work with
> ACPI methods accessing the IO-APICs. Let's hope there's not going to
> be any problems.

Won't ACPI be satisfied by accessing the emulated vIO APIC? That's
available, and should reflect the real IO APIC state in most cases
(except for GSIs in use by Xen).

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH RFC 4/4] x86/dom0: re-order DMA remapping enabling for PVH Dom0

[Roger Pau Monné]

On Fri, Jun 30, 2017 at 05:52:33AM -0600, Jan Beulich wrote:
> >>> Roger Pau Monne <roger.pau@citrix.com> 04/24/17 1:52 PM >>>
> >Make sure the reserved regions are setup before enabling the DMA remapping in
> >the IOMMU, by calling dom0_setup_permissions before iommu_hwdom_init. Also, in
> >order to workaround IOMMU issues seen on pre-Haswell Intel hardware, make sure
> >the DMA remapping is enabled after populating Dom0 p2m.
> 
> Looks reasonable (especially considering that iommu_hwdom_init() is being called
> fairly late in the PV case as well), but please explain the "IOMMU issues seen on
> pre-Haswell Intel hardware" a little here (or if it all is mean to refer to what patch 3
> does, refer to it).

Yes, it's the same issue as described in patch #3, I will add a
reference to the title of the patch if that's fine.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH RFC 1/4] x86/dom0: prevent access to MMCFG areas for PVH Dom0

[Jan Beulich]

>>> On 30.06.17 at 17:33, <roger.pau@citrix.com> wrote:
> On Fri, Jun 30, 2017 at 05:27:06AM -0600, Jan Beulich wrote:
>> >>> Roger Pau Monne <roger.pau@citrix.com> 04/24/17 1:52 PM >>>
>> >--- a/xen/arch/x86/dom0_build.c
>> >+++ b/xen/arch/x86/dom0_build.c
>> >@@ -18,6 +18,8 @@
>>  >#include <asm/p2m.h>
>>  >#include <asm/setup.h>
>>  >
>> >+#include "x86_64/mmconfig.h"
>> 
>> Not just but also because of this I'd prefer if this was taken care of in the
>> MMCFG code itself, also covering ranges which are being added post-
>> boot. Presumably in/from pci_mmcfg_arch_{en,dis}able().
> 
> The problem with this approach is that at the point in the boot where
> pci_mmcfg_arch_enable gets called (from acpi_mmcfg_init) the domain
> has not yet been created, so it's not possible to call
> iomem_deny_access, and in any case the iomem ranges are initialized in
> dom0_setup_permissions, so that would get overwritten.

I understand that; a new helper function would be needed.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH RFC 2/4] x86/dom0: prevent PVH Dom0 from mapping read-only the IO APIC area

[Jan Beulich]

>>> On 30.06.17 at 17:34, <roger.pau@citrix.com> wrote:
> On Fri, Jun 30, 2017 at 05:31:36AM -0600, Jan Beulich wrote:
>> >>> Roger Pau Monne <roger.pau@citrix.com> 04/24/17 1:52 PM >>>
>> >This is emulated by Xen and must not be mapped into PVH Dom0 p2m.
>> 
>> Yes, but this reminds of of the reason we permit the r/o mapping in PV
>> Dom0, and I wonder how well the emulated variant is going to work with
>> ACPI methods accessing the IO-APICs. Let's hope there's not going to
>> be any problems.
> 
> Won't ACPI be satisfied by accessing the emulated vIO APIC? That's
> available, and should reflect the real IO APIC state in most cases
> (except for GSIs in use by Xen).

This "most cases" is what I'm not sufficiently happy with; hence the
"let's hope ...".

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel