PCI passthrough performance loss with Skylake-SP

PCI passthrough performance loss with Skylake-SP

[Andreas Kinzler]

I am currently testing PCI passthrough on the Skylake-SP platform using a  
Supermicro X11SPi-TF mainboard. Using PCI passthrough (an LSI SAS HBA)  
causes severe performance loss on the Skylake-SP platform while Xeon E3 v5  
is not affected at all. The test uses standard iperf3 as a client - the  
passed PCI device is not used in the test - so that just the presence of  
the passed device will cause the iperf3 performance to drop from 6.5  
gbit/sec (no passthrough) to 4.5 gbit/sec.

Any explanation/fixes for that?

Below the first part of xl dmesg for both systems.

Regards Andreas

Xeon E3-1240v5:
	(XEN) Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB.
	(XEN) Intel VT-d Snoop Control enabled.
	(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
	(XEN) Intel VT-d Queued Invalidation enabled.
	(XEN) Intel VT-d Interrupt Remapping enabled.
	(XEN) Intel VT-d Posted Interrupt not enabled.
	(XEN) Intel VT-d Shared EPT tables enabled.
	(XEN) 0000:00:13.0: unknown type 0
	(XEN) I/O virtualisation enabled
	(XEN)  - Dom0 mode: Relaxed
	(XEN) Interrupt remapping enabled
	(XEN) Enabled directed EOI with ioapic_ack_old on!
	(XEN) ENABLING IO-APIC IRQs
	(XEN)  -> Using old ACK method
	(XEN) Allocated console ring of 16 KiB.
	(XEN) VMX: Supported advanced features:
	(XEN)  - APIC MMIO access virtualisation
	(XEN)  - APIC TPR shadow
	(XEN)  - Extended Page Tables (EPT)
	(XEN)  - Virtual-Processor Identifiers (VPID)
	(XEN)  - Virtual NMI
	(XEN)  - MSR direct-access bitmap
	(XEN)  - Unrestricted Guest
	(XEN)  - VMCS shadowing
	(XEN)  - VM Functions
	(XEN)  - Virtualisation Exceptions
	(XEN)  - Page Modification Logging
	(XEN) HVM: ASIDs enabled.
	(XEN) HVM: VMX enabled
	(XEN) HVM: Hardware Assisted Paging (HAP) detected
	(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB

Skylake-SP:
	(XEN) Intel VT-d iommu 2 supported page sizes: 4kB, 2MB, 1GB.
	(XEN) Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB.
	(XEN) Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB.
	(XEN) Intel VT-d iommu 3 supported page sizes: 4kB, 2MB, 1GB.
	(XEN) Intel VT-d Snoop Control enabled.
	(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
	(XEN) Intel VT-d Queued Invalidation enabled.
	(XEN) Intel VT-d Interrupt Remapping enabled.
	(XEN) Intel VT-d Posted Interrupt not enabled.
	(XEN) Intel VT-d Shared EPT tables enabled.
	(XEN) I/O virtualisation enabled
	(XEN)  - Dom0 mode: Relaxed
	(XEN) Interrupt remapping enabled
	(XEN) Enabled directed EOI with ioapic_ack_old on!
	(XEN) ENABLING IO-APIC IRQs
	(XEN)  -> Using old ACK method
	(XEN) Allocated console ring of 128 KiB.
	(XEN) VMX: Supported advanced features:
	(XEN)  - APIC MMIO access virtualisation
	(XEN)  - APIC TPR shadow
	(XEN)  - Extended Page Tables (EPT)
	(XEN)  - Virtual-Processor Identifiers (VPID)
	(XEN)  - Virtual NMI
	(XEN)  - MSR direct-access bitmap
	(XEN)  - Unrestricted Guest
	(XEN)  - APIC Register Virtualization
	(XEN)  - Virtual Interrupt Delivery
	(XEN)  - Posted Interrupt Processing
	(XEN)  - VMCS shadowing
	(XEN)  - VM Functions
	(XEN)  - Virtualisation Exceptions
	(XEN)  - Page Modification Logging
	(XEN)  - TSC Scaling

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PCI passthrough performance loss with Skylake-SP

[Roger Pau Monné]

On Mon, Jun 25, 2018 at 05:32:58PM +0200, Andreas Kinzler wrote:
> I am currently testing PCI passthrough on the Skylake-SP platform using a
> Supermicro X11SPi-TF mainboard. Using PCI passthrough (an LSI SAS HBA)
> causes severe performance loss on the Skylake-SP platform while Xeon E3 v5
> is not affected at all. The test uses standard iperf3 as a client - the
> passed PCI device is not used in the test - so that just the presence of the
> passed device will cause the iperf3 performance to drop from 6.5 gbit/sec
> (no passthrough) to 4.5 gbit/sec.

I assume that the network interface that you are testing is a PV
network interface?

> Any explanation/fixes for that?

Are both systems using the same version of Xen and Linux?

Paul discovered that (recent?) Linux versions the grant table is
allocated in a MMIO area, which has the memory cache attribute set to
uncached. This has a noticeable impact on performance when the guest
also has a passthrough device.

Note that this is not an issue without passthrough because Xen forces
the cache attributes of memory to WB in that case, regardless of the
guest choice.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PCI passthrough performance loss with Skylake-SP

[Paul Durrant]

> -----Original Message-----
> From: Roger Pau Monne
> Sent: 26 June 2018 07:57
> To: Andreas Kinzler <hfp@posteo.de>
> Cc: xen-devel@lists.xen.org; Paul Durrant <Paul.Durrant@citrix.com>
> Subject: Re: [Xen-devel] PCI passthrough performance loss with Skylake-SP
> 
> On Mon, Jun 25, 2018 at 05:32:58PM +0200, Andreas Kinzler wrote:
> > I am currently testing PCI passthrough on the Skylake-SP platform using a
> > Supermicro X11SPi-TF mainboard. Using PCI passthrough (an LSI SAS HBA)
> > causes severe performance loss on the Skylake-SP platform while Xeon E3
> v5
> > is not affected at all. The test uses standard iperf3 as a client - the
> > passed PCI device is not used in the test - so that just the presence of the
> > passed device will cause the iperf3 performance to drop from 6.5 gbit/sec
> > (no passthrough) to 4.5 gbit/sec.
> 
> I assume that the network interface that you are testing is a PV
> network interface?
> 
> > Any explanation/fixes for that?
> 
> Are both systems using the same version of Xen and Linux?
> 
> Paul discovered that (recent?) Linux versions the grant table is
> allocated in a MMIO area, which has the memory cache attribute set to
> uncached. This has a noticeable impact on performance when the guest
> also has a passthrough device.

I can't necessarily claim credit for the discovery but that is indeed the case, and the sort of performance drop seen is exactly what I'd expect. I recently put a change into the Windows PV drivers to use a ballooned-out region of the guest RAM to host the grant tables instead, which avoids this problem.

> 
> Note that this is not an issue without passthrough because Xen forces
> the cache attributes of memory to WB in that case, regardless of the
> guest choice.
> 

We run with this little hack in XenServer, which also 'fixes' things for guests OS that have not been modified:

diff --git a/xen/arch/x86/hvm/mtrr.c b/xen/arch/x86/hvm/mtrr.c
index 3c51244..d0126b4 100644
--- a/xen/arch/x86/hvm/mtrr.c
+++ b/xen/arch/x86/hvm/mtrr.c
@@ -811,7 +811,8 @@ int epte_get_entry_emt(struct domain *d, unsigned long gfn, mfn_t mfn,
         return MTRR_TYPE_UNCACHABLE;
     }
 
-    if ( !need_iommu(d) && !cache_flush_permitted(d) )
+    if ( (!need_iommu(d) && !cache_flush_permitted(d)) ||
+         is_xen_heap_mfn(mfn_x(mfn)) )
     {
         *ipat = 1;
         return MTRR_TYPE_WRBACK;

So you may want to give that a try.

  Paul

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PCI passthrough performance loss with Skylake-SP

[Andreas Kinzler]

On Tue, 26 Jun 2018 09:47:11 +0200, Paul Durrant <Paul.Durrant@citrix.com>  
wrote:

>> > is not affected at all. The test uses standard iperf3 as a client - >  
>> the passed PCI device is not used in the test - so that
>> > just the presence of the passed device will cause the iperf3>  
>> performance to drop from 6.5 gbit/sec (no passthrough)
>> > to 4.5 gbit/sec.
>> I assume that the network interface that you are testing is a PV
>> network interface?

Yes, win-pv.

>> > Any explanation/fixes for that?
>> Are both systems using the same version of Xen and Linux?

Yes, same SSD. Attaching it to different machines.

> I can't necessarily claim credit for the discovery but that is indeed  
> the case, and the sort of performance drop seen is exactly what I'd  
> expect. I recently put a change into the Windows PV drivers to use a  
> ballooned-out region of the guest RAM to host the grant tables instead,  
> which avoids this problem.
> We run with this little hack in XenServer, which also 'fixes' things for  
> guests OS that have not been modified:
> --- a/xen/arch/x86/hvm/mtrr.c
> +++ b/xen/arch/x86/hvm/mtrr.c

I tried the patch and it seems to solve the problem. Thanks.
Is the patch accepted by Xen devs as upstream patch?

Regards Andreas

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: PCI passthrough performance loss with Skylake-SP

[Paul Durrant]

> -----Original Message-----
> From: Andreas Kinzler [mailto:hfp@posteo.de]
> Sent: 01 July 2018 21:15
> To: Paul Durrant <Paul.Durrant@citrix.com>
> Cc: Roger Pau Monne <roger.pau@citrix.com>; xen-devel@lists.xen.org
> Subject: Re: [Xen-devel] PCI passthrough performance loss with Skylake-SP
> 
> On Tue, 26 Jun 2018 09:47:11 +0200, Paul Durrant <Paul.Durrant@citrix.com>
> wrote:
> 
> >> > is not affected at all. The test uses standard iperf3 as a client - >
> >> the passed PCI device is not used in the test - so that
> >> > just the presence of the passed device will cause the iperf3>
> >> performance to drop from 6.5 gbit/sec (no passthrough)
> >> > to 4.5 gbit/sec.
> >> I assume that the network interface that you are testing is a PV
> >> network interface?
> 
> Yes, win-pv.
> 
> >> > Any explanation/fixes for that?
> >> Are both systems using the same version of Xen and Linux?
> 
> Yes, same SSD. Attaching it to different machines.
> 
> > I can't necessarily claim credit for the discovery but that is indeed
> > the case, and the sort of performance drop seen is exactly what I'd
> > expect. I recently put a change into the Windows PV drivers to use a
> > ballooned-out region of the guest RAM to host the grant tables instead,
> > which avoids this problem.
> > We run with this little hack in XenServer, which also 'fixes' things for
> > guests OS that have not been modified:
> > --- a/xen/arch/x86/hvm/mtrr.c
> > +++ b/xen/arch/x86/hvm/mtrr.c
> 
> I tried the patch and it seems to solve the problem. Thanks.
> Is the patch accepted by Xen devs as upstream patch?
> 

Glad the patch solves your problem. I don't think I posted it to xen-devel... probably should get round to doing that :-)

  Cheers,

    Paul

> Regards Andreas

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel