linux-pci.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH] PCI/VMD: White list for fast interrupt handlers
@ 2018-05-08 16:00 Keith Busch
  2018-05-09  4:38 ` Christoph Hellwig
  0 siblings, 1 reply; 9+ messages in thread
From: Keith Busch @ 2018-05-08 16:00 UTC (permalink / raw)
  To: Linux PCI, Bjorn Helgaas; +Cc: Scott Bauer, Jonathan Derrick, Keith Busch

Devices with slow interrupt handlers are significantly harming performance
when their interrupt vector is shared with a fast device. This patch
creates a class code white list for devices with known fast interrupt
handlers, and all other devices will share a single vector so they don't
interfere with performance.

At the moment, only the NVM Express class code is on the list, but more
may be added if VMD users desire to use other low-latency devices in
these domains.

Signed-off-by: Keith Busch <keith.busch@intel.com>
---
 drivers/pci/host/vmd.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/drivers/pci/host/vmd.c b/drivers/pci/host/vmd.c
index 930a8fa08bd6..f94f3e1d3470 100644
--- a/drivers/pci/host/vmd.c
+++ b/drivers/pci/host/vmd.c
@@ -175,9 +175,20 @@ static struct vmd_irq_list *vmd_next_irq(struct vmd_dev *vmd, struct msi_desc *d
 	int i, best = 1;
 	unsigned long flags;
 
-	if (pci_is_bridge(msi_desc_to_pci_dev(desc)) || vmd->msix_count == 1)
+	if (vmd->msix_count == 1)
 		return &vmd->irqs[0];
 
+	/*
+	 * White list for fast-interrupt handlers. All others will share the
+	 * "slow" interrupt vector.
+	 */
+	switch (msi_desc_to_pci_dev(desc)->class) {
+	case PCI_CLASS_STORAGE_EXPRESS:
+		break;
+	default:
+		return &vmd->irqs[0];
+	}
+
 	raw_spin_lock_irqsave(&list_lock, flags);
 	for (i = 1; i < vmd->msix_count; i++)
 		if (vmd->irqs[i].count < vmd->irqs[best].count)
-- 
2.14.3

^ permalink raw reply related	[flat|nested] 9+ messages in thread

* Re: [PATCH] PCI/VMD: White list for fast interrupt handlers
  2018-05-08 16:00 [PATCH] PCI/VMD: White list for fast interrupt handlers Keith Busch
@ 2018-05-09  4:38 ` Christoph Hellwig
  2018-05-09 15:26   ` Keith Busch
  0 siblings, 1 reply; 9+ messages in thread
From: Christoph Hellwig @ 2018-05-09  4:38 UTC (permalink / raw)
  To: Keith Busch; +Cc: Linux PCI, Bjorn Helgaas, Scott Bauer, Jonathan Derrick

On Tue, May 08, 2018 at 10:00:22AM -0600, Keith Busch wrote:
> Devices with slow interrupt handlers are significantly harming performance
> when their interrupt vector is shared with a fast device. This patch
> creates a class code white list for devices with known fast interrupt
> handlers, and all other devices will share a single vector so they don't
> interfere with performance.
> 
> At the moment, only the NVM Express class code is on the list, but more
> may be added if VMD users desire to use other low-latency devices in
> these domains.

I think this is far too much of a hack.   Just don't use VMD if your
care about performance.

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH] PCI/VMD: White list for fast interrupt handlers
  2018-05-09  4:38 ` Christoph Hellwig
@ 2018-05-09 15:26   ` Keith Busch
  2018-05-11 15:39     ` Derrick, Jonathan
  2018-05-14 14:15     ` Christoph Hellwig
  0 siblings, 2 replies; 9+ messages in thread
From: Keith Busch @ 2018-05-09 15:26 UTC (permalink / raw)
  To: Christoph Hellwig
  Cc: Linux PCI, Bjorn Helgaas, Bauer, Scott, Derrick, Jonathan

On Tue, May 08, 2018 at 09:38:28PM -0700, Christoph Hellwig wrote:
> On Tue, May 08, 2018 at 10:00:22AM -0600, Keith Busch wrote:
> > Devices with slow interrupt handlers are significantly harming performance
> > when their interrupt vector is shared with a fast device. This patch
> > creates a class code white list for devices with known fast interrupt
> > handlers, and all other devices will share a single vector so they don't
> > interfere with performance.
> > 
> > At the moment, only the NVM Express class code is on the list, but more
> > may be added if VMD users desire to use other low-latency devices in
> > these domains.
> 
> I think this is far too much of a hack.   Just don't use VMD if your
> care about performance.

I'm not aware of an easier way you can direct-assign an entire PCIe domain
to a virtual machine. :)

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH] PCI/VMD: White list for fast interrupt handlers
  2018-05-09 15:26   ` Keith Busch
@ 2018-05-11 15:39     ` Derrick, Jonathan
  2018-05-25 19:11       ` Derrick, Jonathan
  2018-05-14 14:15     ` Christoph Hellwig
  1 sibling, 1 reply; 9+ messages in thread
From: Derrick, Jonathan @ 2018-05-11 15:39 UTC (permalink / raw)
  To: hch, Busch, Keith; +Cc: linux-pci, Bauer, Scott, bhelgaas

[-- Attachment #1: Type: text/plain, Size: 1069 bytes --]

On Wed, 2018-05-09 at 09:26 -0600, Keith Busch wrote:
> On Tue, May 08, 2018 at 09:38:28PM -0700, Christoph Hellwig wrote:
> > On Tue, May 08, 2018 at 10:00:22AM -0600, Keith Busch wrote:
> > > Devices with slow interrupt handlers are significantly harming
> > > performance
> > > when their interrupt vector is shared with a fast device. This
> > > patch
> > > creates a class code white list for devices with known fast
> > > interrupt
> > > handlers, and all other devices will share a single vector so
> > > they don't
> > > interfere with performance.
> > > 
> > > At the moment, only the NVM Express class code is on the list,
> > > but more
> > > may be added if VMD users desire to use other low-latency devices
> > > in
> > > these domains.
> > 
> > I think this is far too much of a hack.   Just don't use VMD if
> > your
> > care about performance.
> 
> I'm not aware of an easier way you can direct-assign an entire PCIe
> domain
> to a virtual machine. :)

It's fine with me

Acked-by: Jon Derrick: <jonathan.derrick@intel.com>

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 3278 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH] PCI/VMD: White list for fast interrupt handlers
  2018-05-09 15:26   ` Keith Busch
  2018-05-11 15:39     ` Derrick, Jonathan
@ 2018-05-14 14:15     ` Christoph Hellwig
  1 sibling, 0 replies; 9+ messages in thread
From: Christoph Hellwig @ 2018-05-14 14:15 UTC (permalink / raw)
  To: Keith Busch
  Cc: Christoph Hellwig, Linux PCI, Bjorn Helgaas, Bauer, Scott,
	Derrick, Jonathan

On Wed, May 09, 2018 at 09:26:53AM -0600, Keith Busch wrote:
> I'm not aware of an easier way you can direct-assign an entire PCIe domain
> to a virtual machine. :)

libvirt will handle it just fine for you.

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH] PCI/VMD: White list for fast interrupt handlers
  2018-05-11 15:39     ` Derrick, Jonathan
@ 2018-05-25 19:11       ` Derrick, Jonathan
  2018-06-28 11:22         ` Lorenzo Pieralisi
  0 siblings, 1 reply; 9+ messages in thread
From: Derrick, Jonathan @ 2018-05-25 19:11 UTC (permalink / raw)
  To: hch, Busch, Keith; +Cc: linux-pci, Bauer, Scott, bhelgaas, lorenzo.pieralisi

K0xvcmVuem8NCg0KT24gRnJpLCAyMDE4LTA1LTExIGF0IDA5OjM5IC0wNjAwLCBKb25hdGhhbiBE
ZXJyaWNrIHdyb3RlOg0KPiBPbiBXZWQsIDIwMTgtMDUtMDkgYXQgMDk6MjYgLTA2MDAsIEtlaXRo
IEJ1c2NoIHdyb3RlOg0KPiA+IE9uIFR1ZSwgTWF5IDA4LCAyMDE4IGF0IDA5OjM4OjI4UE0gLTA3
MDAsIENocmlzdG9waCBIZWxsd2lnIHdyb3RlOg0KPiA+ID4gT24gVHVlLCBNYXkgMDgsIDIwMTgg
YXQgMTA6MDA6MjJBTSAtMDYwMCwgS2VpdGggQnVzY2ggd3JvdGU6DQo+ID4gPiA+IERldmljZXMg
d2l0aCBzbG93IGludGVycnVwdCBoYW5kbGVycyBhcmUgc2lnbmlmaWNhbnRseSBoYXJtaW5nDQo+
ID4gPiA+IHBlcmZvcm1hbmNlDQo+ID4gPiA+IHdoZW4gdGhlaXIgaW50ZXJydXB0IHZlY3RvciBp
cyBzaGFyZWQgd2l0aCBhIGZhc3QgZGV2aWNlLiBUaGlzDQo+ID4gPiA+IHBhdGNoDQo+ID4gPiA+
IGNyZWF0ZXMgYSBjbGFzcyBjb2RlIHdoaXRlIGxpc3QgZm9yIGRldmljZXMgd2l0aCBrbm93biBm
YXN0DQo+ID4gPiA+IGludGVycnVwdA0KPiA+ID4gPiBoYW5kbGVycywgYW5kIGFsbCBvdGhlciBk
ZXZpY2VzIHdpbGwgc2hhcmUgYSBzaW5nbGUgdmVjdG9yIHNvDQo+ID4gPiA+IHRoZXkgZG9uJ3QN
Cj4gPiA+ID4gaW50ZXJmZXJlIHdpdGggcGVyZm9ybWFuY2UuDQo+ID4gPiA+IA0KPiA+ID4gPiBB
dCB0aGUgbW9tZW50LCBvbmx5IHRoZSBOVk0gRXhwcmVzcyBjbGFzcyBjb2RlIGlzIG9uIHRoZSBs
aXN0LA0KPiA+ID4gPiBidXQgbW9yZQ0KPiA+ID4gPiBtYXkgYmUgYWRkZWQgaWYgVk1EIHVzZXJz
IGRlc2lyZSB0byB1c2Ugb3RoZXIgbG93LWxhdGVuY3kNCj4gPiA+ID4gZGV2aWNlcw0KPiA+ID4g
PiBpbg0KPiA+ID4gPiB0aGVzZSBkb21haW5zLg0KPiA+ID4gDQo+ID4gPiBJIHRoaW5rIHRoaXMg
aXMgZmFyIHRvbyBtdWNoIG9mIGEgaGFjay4gICBKdXN0IGRvbid0IHVzZSBWTUQgaWYNCj4gPiA+
IHlvdXINCj4gPiA+IGNhcmUgYWJvdXQgcGVyZm9ybWFuY2UuDQo+ID4gDQo+ID4gSSdtIG5vdCBh
d2FyZSBvZiBhbiBlYXNpZXIgd2F5IHlvdSBjYW4gZGlyZWN0LWFzc2lnbiBhbiBlbnRpcmUgUENJ
ZQ0KPiA+IGRvbWFpbg0KPiA+IHRvIGEgdmlydHVhbCBtYWNoaW5lLiA6KQ0KPiANCj4gSXQncyBm
aW5lIHdpdGggbWUNCj4gDQo+IEFja2VkLWJ5OiBKb24gRGVycmljazogPGpvbmF0aGFuLmRlcnJp
Y2tAaW50ZWwuY29tPg0KDQoNCldlJ3ZlIHNlZW4gdGhpcyBhY3R1YWxseSBmaXggYW4gaXNzdWUg
d2l0aCBvbmUgdmVuZG9yJ3MgbXVsdGktZnVuY3Rpb24NCnN3aXRjaC4NCkknZCBsaWtlIHRvIHNl
ZSBpdCBnZXQgaW50byA0LjE4IGlmIHBvc3NpYmxl

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH] PCI/VMD: White list for fast interrupt handlers
  2018-05-25 19:11       ` Derrick, Jonathan
@ 2018-06-28 11:22         ` Lorenzo Pieralisi
  2018-06-28 14:02           ` Keith Busch
  0 siblings, 1 reply; 9+ messages in thread
From: Lorenzo Pieralisi @ 2018-06-28 11:22 UTC (permalink / raw)
  To: Derrick, Jonathan; +Cc: hch, Busch, Keith, linux-pci, Bauer, Scott, bhelgaas

On Fri, May 25, 2018 at 07:11:14PM +0000, Derrick, Jonathan wrote:
> +Lorenzo
> 
> On Fri, 2018-05-11 at 09:39 -0600, Jonathan Derrick wrote:
> > On Wed, 2018-05-09 at 09:26 -0600, Keith Busch wrote:
> > > On Tue, May 08, 2018 at 09:38:28PM -0700, Christoph Hellwig wrote:
> > > > On Tue, May 08, 2018 at 10:00:22AM -0600, Keith Busch wrote:
> > > > > Devices with slow interrupt handlers are significantly harming
> > > > > performance
> > > > > when their interrupt vector is shared with a fast device. This
> > > > > patch
> > > > > creates a class code white list for devices with known fast
> > > > > interrupt
> > > > > handlers, and all other devices will share a single vector so
> > > > > they don't
> > > > > interfere with performance.
> > > > > 
> > > > > At the moment, only the NVM Express class code is on the list,
> > > > > but more
> > > > > may be added if VMD users desire to use other low-latency
> > > > > devices
> > > > > in
> > > > > these domains.
> > > > 
> > > > I think this is far too much of a hack.   Just don't use VMD if
> > > > your
> > > > care about performance.
> > > 
> > > I'm not aware of an easier way you can direct-assign an entire PCIe
> > > domain
> > > to a virtual machine. :)
> > 
> > It's fine with me
> > 
> > Acked-by: Jon Derrick: <jonathan.derrick@intel.com>
> 
> 
> We've seen this actually fix an issue with one vendor's multi-function
> switch.
> I'd like to see it get into 4.18 if possible

Sorry for the delay in getting back to this.

It seems like Christoph is not too happy about this patch, it is your
code so I would apply it unless there is a cleaner alternative so please
do let me know.

Thanks,
Lorenzo

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH] PCI/VMD: White list for fast interrupt handlers
  2018-06-28 11:22         ` Lorenzo Pieralisi
@ 2018-06-28 14:02           ` Keith Busch
  2018-06-28 16:32             ` Lorenzo Pieralisi
  0 siblings, 1 reply; 9+ messages in thread
From: Keith Busch @ 2018-06-28 14:02 UTC (permalink / raw)
  To: Lorenzo Pieralisi
  Cc: Derrick, Jonathan, hch, linux-pci, Bauer, Scott, bhelgaas

On Thu, Jun 28, 2018 at 12:22:17PM +0100, Lorenzo Pieralisi wrote:
> On Fri, May 25, 2018 at 07:11:14PM +0000, Derrick, Jonathan wrote:
> > We've seen this actually fix an issue with one vendor's multi-function
> > switch. I'd like to see it get into 4.18 if possible
> 
> Sorry for the delay in getting back to this.
> 
> It seems like Christoph is not too happy about this patch, it is your
> code so I would apply it unless there is a cleaner alternative so please
> do let me know.

VMD hardware really relies on the host software to bring all the logic,
and it was intended to prioritize PCIe attached storage. The patch just
captures that intention, so yes please, let's queue this up.

I understands Christoph's lack of enthusiasm for this, but at least this
isn't a PCIe HBA with proprietary interfaces and firmware. :)

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH] PCI/VMD: White list for fast interrupt handlers
  2018-06-28 14:02           ` Keith Busch
@ 2018-06-28 16:32             ` Lorenzo Pieralisi
  0 siblings, 0 replies; 9+ messages in thread
From: Lorenzo Pieralisi @ 2018-06-28 16:32 UTC (permalink / raw)
  To: Keith Busch; +Cc: Derrick, Jonathan, hch, linux-pci, Bauer, Scott, bhelgaas

On Thu, Jun 28, 2018 at 08:02:40AM -0600, Keith Busch wrote:
> On Thu, Jun 28, 2018 at 12:22:17PM +0100, Lorenzo Pieralisi wrote:
> > On Fri, May 25, 2018 at 07:11:14PM +0000, Derrick, Jonathan wrote:
> > > We've seen this actually fix an issue with one vendor's multi-function
> > > switch. I'd like to see it get into 4.18 if possible
> > 
> > Sorry for the delay in getting back to this.
> > 
> > It seems like Christoph is not too happy about this patch, it is your
> > code so I would apply it unless there is a cleaner alternative so please
> > do let me know.
> 
> VMD hardware really relies on the host software to bring all the logic,
> and it was intended to prioritize PCIe attached storage. The patch just
> captures that intention, so yes please, let's queue this up.
> 
> I understands Christoph's lack of enthusiasm for this, but at least this
> isn't a PCIe HBA with proprietary interfaces and firmware. :)

Applied to pci/vmd with a slightly updated changelog, thanks.

Lorenzo

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2018-06-28 16:32 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-05-08 16:00 [PATCH] PCI/VMD: White list for fast interrupt handlers Keith Busch
2018-05-09  4:38 ` Christoph Hellwig
2018-05-09 15:26   ` Keith Busch
2018-05-11 15:39     ` Derrick, Jonathan
2018-05-25 19:11       ` Derrick, Jonathan
2018-06-28 11:22         ` Lorenzo Pieralisi
2018-06-28 14:02           ` Keith Busch
2018-06-28 16:32             ` Lorenzo Pieralisi
2018-05-14 14:15     ` Christoph Hellwig

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).