From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751944AbdKJIzO (ORCPT ); Fri, 10 Nov 2017 03:55:14 -0500 Received: from mail-sn1nam02on0111.outbound.protection.outlook.com ([104.47.36.111]:64928 "EHLO NAM02-SN1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751078AbdKJIzL (ORCPT ); Fri, 10 Nov 2017 03:55:11 -0500 From: "Adrian Suhov (Cloudbase Solutions SRL)" To: Bjorn Helgaas , Dexuan Cui CC: Bjorn Helgaas , "linux-pci@vger.kernel.org" , Jake Oshins , KY Srinivasan , Stephen Hemminger , "devel@linuxdriverproject.org" , "linux-kernel@vger.kernel.org" , Haiyang Zhang , Jork Loeser , "Chris Valean (Cloudbase Solutions SRL)" , Simon Xiao , "'Eyal Mizrachi'" , "Jack Morgenstein" , Armen Guezalian , Firas Mahameed , Tziporet Koren , Daniel Jurgens Subject: RE: [PATCH] PCI: hv: use effective affinity mask Thread-Topic: [PATCH] PCI: hv: use effective affinity mask Thread-Index: AdNTSQFbm8/mhC8xRuOwOLkoIvvyZgE5PVcAAEisVmA= Date: Fri, 10 Nov 2017 08:55:07 +0000 Message-ID: References: <20171108010736.GA28427@bhelgaas-glaptop.roam.corp.google.com> In-Reply-To: <20171108010736.GA28427@bhelgaas-glaptop.roam.corp.google.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=True; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Owner=v-adsuho@microsoft.com; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2017-11-09T11:50:28.2997453Z; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=General; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Application=Microsoft Azure Information Protection; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Extended_MSFT_Method=Automatic; Sensitivity=General x-originating-ip: [79.114.93.36] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;MWHPR21MB0781;6:Om/mFu7SDqAcvPe70piHGBd5/syUT4s3AINV1sawHBIO+kqbmfodAAmWZf1EWXv4aPCv05wqvC/GWhibZK7raS875cBDrIu2IanKUtDktMcm6inyUyhNJBm7z/92nagm8MtOGViDnY3M9p7OBbDcFth1TQb3sWRMt9m+mR2KqbZe327xRFrilaOZm5n3RdG9Q65vQJMM/wRkFwcKQD9gm/jSWLyuUnaUftDDLJfhrtcNjfPFLCZSdaC5WrNKlkLRmTMXKO1Yqn3pNQQSzVNosCp3v/InSYZ8lJyOjxOc9XRssRIvYNVeo6UCOmYW09oqPhsT4GIjjzToyRI+FB9211IYbf7lXA8kNufQDM0SWC8=;5:PQkCKty9yG72IqtLviWd90D4IoTJ0uHCqHnBYDaxEzIuaFry3pCmmy3tnQcyLwshbDQHoUdUBN0SR+mTpWdL8Ab6WSuDD1WgFD/m9zpvohvnKfU7+qu1KWruggWx9ms61U3KZQOU+M14R8YVqno490TYuhsbnVVFHvBqLWBT12U=;24:TdwQwXcnU8JbADemcEQSZhov9TSLk/iijKhykaEArAin/AZUBfFtx7XeYBHcYI6iTKF81PlAnVpevdV20v1kbN4bPaxXzb/mCK3hd5RK2pU=;7:B8lAdUQIEIey6R6FGhhU7TKNFsM80/LcSjb4udoESMXUBJ+b1/ieYThCvZNRbc7kp26ZbtZgFXdxBCr/4QxnihCFy2EejnGB2jK2fj/goMR3cPvh007Lo7PmMHoTTiSAftM8NGvACxVjuv2K8CBDkv13cL3RoXu4HDyBhpPaZkr1DELv82wQxjoIdyFIilr2k8ysP7vtj6vcG24OS5siS8LueBAA/zXU3Z3wFHxB8xnlidYmLf4emYa5gc1v54dk x-ms-exchange-antispam-srfa-diagnostics: SSOS; x-ms-office365-filtering-correlation-id: 53b08a2f-c3c8-4318-4859-08d52818be24 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(48565401081)(4534020)(4602075)(4627115)(201703031133081)(201702281549075)(2017052603258);SRVR:MWHPR21MB0781; x-ms-traffictypediagnostic: MWHPR21MB0781: authentication-results: spf=none (sender IP is ) smtp.mailfrom=v-adsuho@microsoft.com; x-ld-processed: 72f988bf-86f1-41af-91ab-2d7cd011db47,ExtAddr x-exchange-antispam-report-test: UriScan:(89211679590171)(9452136761055)(211936372134217)(153496737603132); x-microsoft-antispam-prvs: x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(61425038)(6040450)(2401047)(5005006)(8121501046)(3002001)(100000703101)(100105400095)(10201501046)(93006095)(93001095)(3231021)(920507027)(6055026)(61426038)(61427038)(6041248)(20161123558100)(20161123564025)(20161123562025)(20161123555025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123560025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095);SRVR:MWHPR21MB0781;BCL:0;PCL:0;RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095);SRVR:MWHPR21MB0781; x-forefront-prvs: 0487C0DB7E x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(6009001)(39860400002)(376002)(346002)(47760400005)(13464003)(24454002)(199003)(189002)(105586002)(3846002)(8990500004)(9686003)(34040400001)(55016002)(2421001)(33656002)(53546010)(305945005)(478600001)(7736002)(74316002)(10290500003)(8676002)(8936002)(102836003)(2561002)(6116002)(76176999)(101416001)(81156014)(81166006)(54356999)(189998001)(99286004)(50986999)(2950100002)(106356001)(7696004)(22452003)(7416002)(77096006)(66066001)(1511001)(2900100001)(229853002)(5660300001)(6636002)(53936002)(3280700002)(6436002)(6506006)(3660700001)(25786009)(6246003)(68736007)(86612001)(110136005)(14454004)(86362001)(4326008)(2906002)(316002)(10090500001)(54906003)(97736004);DIR:OUT;SFP:1102;SCL:1;SRVR:MWHPR21MB0781;H:MWHPR21MB0287.namprd21.prod.outlook.com;FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-Network-Message-Id: 53b08a2f-c3c8-4318-4859-08d52818be24 X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Nov 2017 08:55:07.5761 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR21MB0781 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by nfs id vAA8tJTR026031 Hi, I've also tested this and it's working good. Kernels tested: - next-20171109 on top of Ubuntu 16.04 - MSFT kernel - 4.14.0-rc5 with patch applied - on top of RHEL 7.3 Adrian -----Original Message----- From: Bjorn Helgaas [mailto:helgaas@kernel.org] Sent: Wednesday, November 8, 2017 3:08 AM To: Dexuan Cui Cc: Bjorn Helgaas ; linux-pci@vger.kernel.org; Jake Oshins ; KY Srinivasan ; Stephen Hemminger ; devel@linuxdriverproject.org; linux-kernel@vger.kernel.org; Haiyang Zhang ; Jork Loeser ; Chris Valean (Cloudbase Solutions SRL) ; Adrian Suhov (Cloudbase Solutions SRL) ; Simon Xiao ; 'Eyal Mizrachi' ; Jack Morgenstein ; Armen Guezalian ; Firas Mahameed ; Tziporet Koren ; Daniel Jurgens Subject: Re: [PATCH] PCI: hv: use effective affinity mask On Wed, Nov 01, 2017 at 08:30:53PM +0000, Dexuan Cui wrote: > > The effective_affinity_mask is always set when an interrupt is > assigned in > __assign_irq_vector() -> apic->cpu_mask_to_apicid(), e.g. for struct > apic > apic_physflat: -> default_cpu_mask_to_apicid() -> > irq_data_update_effective_affinity(), but it looks d->common->affinity > remains all-1's before the user space or the kernel changes it later. > > In the early allocation/initialization phase of an irq, we should use > the effective_affinity_mask, otherwise Hyper-V may not deliver the > interrupt to the expected cpu. Without the patch, if we assign 7 > Mellanox ConnectX-3 VFs to a 32-vCPU VM, one of the VFs may fail to receive interrupts. > > Signed-off-by: Dexuan Cui > Cc: Jake Oshins > Cc: Jork Loeser > Cc: Stephen Hemminger > Cc: K. Y. Srinivasan > --- > > Please consider this for v4.14, if it's not too late. What would be the rationale for putting it in v4.14? After the merge window, I usually only merge fixes for problems introduced during the merge window, or for serious regressions. I can't tell if this fits into that or not. > drivers/pci/host/pci-hyperv.c | 8 +++++--- > 1 file changed, 5 insertions(+), 3 deletions(-) > > diff --git a/drivers/pci/host/pci-hyperv.c > b/drivers/pci/host/pci-hyperv.c index 5ccb47d..8b5f66d 100644 > --- a/drivers/pci/host/pci-hyperv.c > +++ b/drivers/pci/host/pci-hyperv.c > @@ -879,7 +879,7 @@ static void hv_irq_unmask(struct irq_data *data) > int cpu; > u64 res; > > - dest = irq_data_get_affinity_mask(data); > + dest = irq_data_get_effective_affinity_mask(data); > pdev = msi_desc_to_pci_dev(msi_desc); > pbus = pdev->bus; > hbus = container_of(pbus->sysdata, struct hv_pcibus_device, > sysdata); @@ -1042,6 +1042,7 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg) > struct hv_pci_dev *hpdev; > struct pci_bus *pbus; > struct pci_dev *pdev; > + struct cpumask *dest; > struct compose_comp_ctxt comp; > struct tran_int_desc *int_desc; > struct { > @@ -1056,6 +1057,7 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg) > int ret; > > pdev = msi_desc_to_pci_dev(irq_data_get_msi_desc(data)); > + dest = irq_data_get_effective_affinity_mask(data); > pbus = pdev->bus; > hbus = container_of(pbus->sysdata, struct hv_pcibus_device, sysdata); > hpdev = get_pcichild_wslot(hbus, devfn_to_wslot(pdev->devfn)); @@ > -1081,14 +1083,14 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg) > switch (pci_protocol_version) { > case PCI_PROTOCOL_VERSION_1_1: > size = hv_compose_msi_req_v1(&ctxt.int_pkts.v1, > - irq_data_get_affinity_mask(data), > + dest, > hpdev->desc.win_slot.slot, > cfg->vector); > break; > > case PCI_PROTOCOL_VERSION_1_2: > size = hv_compose_msi_req_v2(&ctxt.int_pkts.v2, > - irq_data_get_affinity_mask(data), > + dest, > hpdev->desc.win_slot.slot, > cfg->vector); > break; > -- > 2.7.4 > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: From: "Adrian Suhov (Cloudbase Solutions SRL)" To: Bjorn Helgaas , Dexuan Cui CC: Bjorn Helgaas , "linux-pci@vger.kernel.org" , Jake Oshins , KY Srinivasan , Stephen Hemminger , "devel@linuxdriverproject.org" , "linux-kernel@vger.kernel.org" , Haiyang Zhang , Jork Loeser , "Chris Valean (Cloudbase Solutions SRL)" , Simon Xiao , 'Eyal Mizrachi' , "Jack Morgenstein" , Armen Guezalian , Firas Mahameed , Tziporet Koren , Daniel Jurgens Subject: RE: [PATCH] PCI: hv: use effective affinity mask Date: Fri, 10 Nov 2017 08:55:07 +0000 Message-ID: References: <20171108010736.GA28427@bhelgaas-glaptop.roam.corp.google.com> In-Reply-To: <20171108010736.GA28427@bhelgaas-glaptop.roam.corp.google.com> Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org List-ID: Hi, I've also tested this and it's working good. Kernels tested: - next-20171109 on top of Ubuntu 16.04 - MSFT kernel - 4.14.0-rc5 with patch applied - on top of RHEL 7.3 Adrian -----Original Message----- From: Bjorn Helgaas [mailto:helgaas@kernel.org]=20 Sent: Wednesday, November 8, 2017 3:08 AM To: Dexuan Cui Cc: Bjorn Helgaas ; linux-pci@vger.kernel.org; Jake Os= hins ; KY Srinivasan ; Stephen Hemm= inger ; devel@linuxdriverproject.org; linux-kernel@= vger.kernel.org; Haiyang Zhang ; Jork Loeser ; Chris Valean (Cloudbase Solutions SRL) ; Adrian Suhov (Cloudbase Solutions SRL) ; Simon Xiao ; 'Eyal Mizrachi' = ; Jack Morgenstein ; Armen Guezalian ; Firas Mahameed ; Tziporet Koren ; Daniel Jurgens Subject: Re: [PATCH] PCI: hv: use effective affinity mask On Wed, Nov 01, 2017 at 08:30:53PM +0000, Dexuan Cui wrote: >=20 > The effective_affinity_mask is always set when an interrupt is=20 > assigned in > __assign_irq_vector() -> apic->cpu_mask_to_apicid(), e.g. for struct=20 > apic > apic_physflat: -> default_cpu_mask_to_apicid() ->=20 > irq_data_update_effective_affinity(), but it looks d->common->affinity=20 > remains all-1's before the user space or the kernel changes it later. >=20 > In the early allocation/initialization phase of an irq, we should use=20 > the effective_affinity_mask, otherwise Hyper-V may not deliver the=20 > interrupt to the expected cpu. Without the patch, if we assign 7=20 > Mellanox ConnectX-3 VFs to a 32-vCPU VM, one of the VFs may fail to recei= ve interrupts. >=20 > Signed-off-by: Dexuan Cui > Cc: Jake Oshins > Cc: Jork Loeser > Cc: Stephen Hemminger > Cc: K. Y. Srinivasan > --- >=20 > Please consider this for v4.14, if it's not too late. What would be the rationale for putting it in v4.14? After the merge windo= w, I usually only merge fixes for problems introduced during the merge wind= ow, or for serious regressions. I can't tell if this fits into that or not= . > drivers/pci/host/pci-hyperv.c | 8 +++++--- > 1 file changed, 5 insertions(+), 3 deletions(-) >=20 > diff --git a/drivers/pci/host/pci-hyperv.c=20 > b/drivers/pci/host/pci-hyperv.c index 5ccb47d..8b5f66d 100644 > --- a/drivers/pci/host/pci-hyperv.c > +++ b/drivers/pci/host/pci-hyperv.c > @@ -879,7 +879,7 @@ static void hv_irq_unmask(struct irq_data *data) > int cpu; > u64 res; > =20 > - dest =3D irq_data_get_affinity_mask(data); > + dest =3D irq_data_get_effective_affinity_mask(data); > pdev =3D msi_desc_to_pci_dev(msi_desc); > pbus =3D pdev->bus; > hbus =3D container_of(pbus->sysdata, struct hv_pcibus_device,=20 > sysdata); @@ -1042,6 +1042,7 @@ static void hv_compose_msi_msg(struct irq= _data *data, struct msi_msg *msg) > struct hv_pci_dev *hpdev; > struct pci_bus *pbus; > struct pci_dev *pdev; > + struct cpumask *dest; > struct compose_comp_ctxt comp; > struct tran_int_desc *int_desc; > struct { > @@ -1056,6 +1057,7 @@ static void hv_compose_msi_msg(struct irq_data *dat= a, struct msi_msg *msg) > int ret; > =20 > pdev =3D msi_desc_to_pci_dev(irq_data_get_msi_desc(data)); > + dest =3D irq_data_get_effective_affinity_mask(data); > pbus =3D pdev->bus; > hbus =3D container_of(pbus->sysdata, struct hv_pcibus_device, sysdata); > hpdev =3D get_pcichild_wslot(hbus, devfn_to_wslot(pdev->devfn)); @@=20 > -1081,14 +1083,14 @@ static void hv_compose_msi_msg(struct irq_data *data= , struct msi_msg *msg) > switch (pci_protocol_version) { > case PCI_PROTOCOL_VERSION_1_1: > size =3D hv_compose_msi_req_v1(&ctxt.int_pkts.v1, > - irq_data_get_affinity_mask(data), > + dest, > hpdev->desc.win_slot.slot, > cfg->vector); > break; > =20 > case PCI_PROTOCOL_VERSION_1_2: > size =3D hv_compose_msi_req_v2(&ctxt.int_pkts.v2, > - irq_data_get_affinity_mask(data), > + dest, > hpdev->desc.win_slot.slot, > cfg->vector); > break; > -- > 2.7.4 >=20