* [PATCH] PCI: hv: use effective affinity mask
@ 2017-11-01 20:30 ` Dexuan Cui
0 siblings, 0 replies; 14+ messages in thread
From: Dexuan Cui @ 2017-11-01 20:30 UTC (permalink / raw)
To: Bjorn Helgaas, linux-pci, Jake Oshins, KY Srinivasan, Stephen Hemminger
Cc: devel, linux-kernel, Haiyang Zhang, Jork Loeser,
Chris Valean (Cloudbase Solutions SRL),
Adrian Suhov (Cloudbase Solutions SRL),
Simon Xiao, 'Eyal Mizrachi',
Jack Morgenstein, Armen Guezalian, Firas Mahameed,
Tziporet Koren, Daniel Jurgens
The effective_affinity_mask is always set when an interrupt is assigned in
__assign_irq_vector() -> apic->cpu_mask_to_apicid(), e.g. for struct apic
apic_physflat: -> default_cpu_mask_to_apicid() ->
irq_data_update_effective_affinity(), but it looks d->common->affinity
remains all-1's before the user space or the kernel changes it later.
In the early allocation/initialization phase of an irq, we should use the
effective_affinity_mask, otherwise Hyper-V may not deliver the interrupt
to the expected cpu. Without the patch, if we assign 7 Mellanox ConnectX-3
VFs to a 32-vCPU VM, one of the VFs may fail to receive interrupts.
Signed-off-by: Dexuan Cui <decui@microsoft.com>
Cc: Jake Oshins <jakeo@microsoft.com>
Cc: Jork Loeser <jloeser@microsoft.com>
Cc: Stephen Hemminger <sthemmin@microsoft.com>
Cc: K. Y. Srinivasan <kys@microsoft.com>
---
Please consider this for v4.14, if it's not too late.
drivers/pci/host/pci-hyperv.c | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/drivers/pci/host/pci-hyperv.c b/drivers/pci/host/pci-hyperv.c
index 5ccb47d..8b5f66d 100644
--- a/drivers/pci/host/pci-hyperv.c
+++ b/drivers/pci/host/pci-hyperv.c
@@ -879,7 +879,7 @@ static void hv_irq_unmask(struct irq_data *data)
int cpu;
u64 res;
=20
- dest =3D irq_data_get_affinity_mask(data);
+ dest =3D irq_data_get_effective_affinity_mask(data);
pdev =3D msi_desc_to_pci_dev(msi_desc);
pbus =3D pdev->bus;
hbus =3D container_of(pbus->sysdata, struct hv_pcibus_device, sysdata);
@@ -1042,6 +1042,7 @@ static void hv_compose_msi_msg(struct irq_data *data,=
struct msi_msg *msg)
struct hv_pci_dev *hpdev;
struct pci_bus *pbus;
struct pci_dev *pdev;
+ struct cpumask *dest;
struct compose_comp_ctxt comp;
struct tran_int_desc *int_desc;
struct {
@@ -1056,6 +1057,7 @@ static void hv_compose_msi_msg(struct irq_data *data,=
struct msi_msg *msg)
int ret;
=20
pdev =3D msi_desc_to_pci_dev(irq_data_get_msi_desc(data));
+ dest =3D irq_data_get_effective_affinity_mask(data);
pbus =3D pdev->bus;
hbus =3D container_of(pbus->sysdata, struct hv_pcibus_device, sysdata);
hpdev =3D get_pcichild_wslot(hbus, devfn_to_wslot(pdev->devfn));
@@ -1081,14 +1083,14 @@ static void hv_compose_msi_msg(struct irq_data *dat=
a, struct msi_msg *msg)
switch (pci_protocol_version) {
case PCI_PROTOCOL_VERSION_1_1:
size =3D hv_compose_msi_req_v1(&ctxt.int_pkts.v1,
- irq_data_get_affinity_mask(data),
+ dest,
hpdev->desc.win_slot.slot,
cfg->vector);
break;
=20
case PCI_PROTOCOL_VERSION_1_2:
size =3D hv_compose_msi_req_v2(&ctxt.int_pkts.v2,
- irq_data_get_affinity_mask(data),
+ dest,
hpdev->desc.win_slot.slot,
cfg->vector);
break;
--=20
2.7.4
^ permalink raw reply related [flat|nested] 14+ messages in thread* RE: [PATCH] PCI: hv: use effective affinity mask
2017-11-01 20:30 ` Dexuan Cui
@ 2017-11-01 20:52 ` Jake Oshins
-1 siblings, 0 replies; 14+ messages in thread
From: Jake Oshins @ 2017-11-01 20:52 UTC (permalink / raw)
To: Dexuan Cui, Bjorn Helgaas, linux-pci, KY Srinivasan, Stephen Hemminger
Cc: devel, linux-kernel, Haiyang Zhang, Jork Loeser,
Chris Valean (Cloudbase Solutions SRL),
Adrian Suhov (Cloudbase Solutions SRL),
Simon Xiao, 'Eyal Mizrachi',
Jack Morgenstein, Armen Guezalian, Firas Mahameed,
Tziporet Koren, Daniel Jurgens
> -----Original Message-----
> From: Dexuan Cui
> Sent: Wednesday, November 1, 2017 1:31 PM
> To: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; Jake
> Oshins <jakeo@microsoft.com>; KY Srinivasan <kys@microsoft.com>;
> Stephen Hemminger <sthemmin@microsoft.com>
> Cc: devel@linuxdriverproject.org; linux-kernel@vger.kernel.org; Haiyang
> Zhang <haiyangz@microsoft.com>; Jork Loeser
> <Jork.Loeser@microsoft.com>; Chris Valean (Cloudbase Solutions SRL) <v-
> chvale@microsoft.com>; Adrian Suhov (Cloudbase Solutions SRL) <v-
> adsuho@microsoft.com>; Simon Xiao <sixiao@microsoft.com>; 'Eyal
> Mizrachi' <eyalmi@mellanox.com>; Jack Morgenstein
> <jackm@mellanox.com>; Armen Guezalian <armeng@mellanox.com>; Firas
> Mahameed <firas@mellanox.com>; Tziporet Koren
> <tziporet@mellanox.com>; Daniel Jurgens <danielj@mellanox.com>
> Subject: [PATCH] PCI: hv: use effective affinity mask
>
>
> The effective_affinity_mask is always set when an interrupt is assigned in
> __assign_irq_vector() -> apic->cpu_mask_to_apicid(), e.g. for struct apic
> apic_physflat: -> default_cpu_mask_to_apicid() ->
> irq_data_update_effective_affinity(), but it looks d->common->affinity
> remains all-1's before the user space or the kernel changes it later.
>
> In the early allocation/initialization phase of an irq, we should use the
> effective_affinity_mask, otherwise Hyper-V may not deliver the interrupt to
> the expected cpu. Without the patch, if we assign 7 Mellanox ConnectX-3
> VFs to a 32-vCPU VM, one of the VFs may fail to receive interrupts.
>
> Signed-off-by: Dexuan Cui <decui@microsoft.com>
> Cc: Jake Oshins <jakeo@microsoft.com>
> Cc: Jork Loeser <jloeser@microsoft.com>
> Cc: Stephen Hemminger <sthemmin@microsoft.com>
> Cc: K. Y. Srinivasan <kys@microsoft.com>
> ---
>
> Please consider this for v4.14, if it's not too late.
>
> drivers/pci/host/pci-hyperv.c | 8 +++++---
> 1 file changed, 5 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/pci/host/pci-hyperv.c b/drivers/pci/host/pci-hyperv.c
> index 5ccb47d..8b5f66d 100644
> --- a/drivers/pci/host/pci-hyperv.c
> +++ b/drivers/pci/host/pci-hyperv.c
> @@ -879,7 +879,7 @@ static void hv_irq_unmask(struct irq_data *data)
> int cpu;
> u64 res;
>
> - dest = irq_data_get_affinity_mask(data);
> + dest = irq_data_get_effective_affinity_mask(data);
> pdev = msi_desc_to_pci_dev(msi_desc);
> pbus = pdev->bus;
> hbus = container_of(pbus->sysdata, struct hv_pcibus_device,
> sysdata); @@ -1042,6 +1042,7 @@ static void hv_compose_msi_msg(struct
> irq_data *data, struct msi_msg *msg)
> struct hv_pci_dev *hpdev;
> struct pci_bus *pbus;
> struct pci_dev *pdev;
> + struct cpumask *dest;
> struct compose_comp_ctxt comp;
> struct tran_int_desc *int_desc;
> struct {
> @@ -1056,6 +1057,7 @@ static void hv_compose_msi_msg(struct irq_data
> *data, struct msi_msg *msg)
> int ret;
>
> pdev = msi_desc_to_pci_dev(irq_data_get_msi_desc(data));
> + dest = irq_data_get_effective_affinity_mask(data);
> pbus = pdev->bus;
> hbus = container_of(pbus->sysdata, struct hv_pcibus_device,
> sysdata);
> hpdev = get_pcichild_wslot(hbus, devfn_to_wslot(pdev->devfn));
> @@ -1081,14 +1083,14 @@ static void hv_compose_msi_msg(struct irq_data
> *data, struct msi_msg *msg)
> switch (pci_protocol_version) {
> case PCI_PROTOCOL_VERSION_1_1:
> size = hv_compose_msi_req_v1(&ctxt.int_pkts.v1,
> - irq_data_get_affinity_mask(data),
> + dest,
> hpdev->desc.win_slot.slot,
> cfg->vector);
> break;
>
> case PCI_PROTOCOL_VERSION_1_2:
> size = hv_compose_msi_req_v2(&ctxt.int_pkts.v2,
> - irq_data_get_affinity_mask(data),
> + dest,
> hpdev->desc.win_slot.slot,
> cfg->vector);
> break;
> --
> 2.7.4
Signed-off-by: Jake Oshins <jakeo@microsoft.com>
^ permalink raw reply [flat|nested] 14+ messages in thread* RE: [PATCH] PCI: hv: use effective affinity mask
@ 2017-11-01 20:52 ` Jake Oshins
0 siblings, 0 replies; 14+ messages in thread
From: Jake Oshins @ 2017-11-01 20:52 UTC (permalink / raw)
To: Dexuan Cui, Bjorn Helgaas, linux-pci, KY Srinivasan, Stephen Hemminger
Cc: devel, linux-kernel, Haiyang Zhang, Jork Loeser,
Chris Valean (Cloudbase Solutions SRL),
Adrian Suhov (Cloudbase Solutions SRL),
Simon Xiao, 'Eyal Mizrachi',
Jack Morgenstein, Armen Guezalian, Firas Mahameed,
Tziporet Koren, Daniel Jurgens
> -----Original Message-----
> From: Dexuan Cui
> Sent: Wednesday, November 1, 2017 1:31 PM
> To: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; Jake
> Oshins <jakeo@microsoft.com>; KY Srinivasan <kys@microsoft.com>;
> Stephen Hemminger <sthemmin@microsoft.com>
> Cc: devel@linuxdriverproject.org; linux-kernel@vger.kernel.org; Haiyang
> Zhang <haiyangz@microsoft.com>; Jork Loeser
> <Jork.Loeser@microsoft.com>; Chris Valean (Cloudbase Solutions SRL) <v-
> chvale@microsoft.com>; Adrian Suhov (Cloudbase Solutions SRL) <v-
> adsuho@microsoft.com>; Simon Xiao <sixiao@microsoft.com>; 'Eyal
> Mizrachi' <eyalmi@mellanox.com>; Jack Morgenstein
> <jackm@mellanox.com>; Armen Guezalian <armeng@mellanox.com>; Firas
> Mahameed <firas@mellanox.com>; Tziporet Koren
> <tziporet@mellanox.com>; Daniel Jurgens <danielj@mellanox.com>
> Subject: [PATCH] PCI: hv: use effective affinity mask
>=20
>=20
> The effective_affinity_mask is always set when an interrupt is assigned i=
n
> __assign_irq_vector() -> apic->cpu_mask_to_apicid(), e.g. for struct apic
> apic_physflat: -> default_cpu_mask_to_apicid() ->
> irq_data_update_effective_affinity(), but it looks d->common->affinity
> remains all-1's before the user space or the kernel changes it later.
>=20
> In the early allocation/initialization phase of an irq, we should use the
> effective_affinity_mask, otherwise Hyper-V may not deliver the interrupt =
to
> the expected cpu. Without the patch, if we assign 7 Mellanox ConnectX-3
> VFs to a 32-vCPU VM, one of the VFs may fail to receive interrupts.
>=20
> Signed-off-by: Dexuan Cui <decui@microsoft.com>
> Cc: Jake Oshins <jakeo@microsoft.com>
> Cc: Jork Loeser <jloeser@microsoft.com>
> Cc: Stephen Hemminger <sthemmin@microsoft.com>
> Cc: K. Y. Srinivasan <kys@microsoft.com>
> ---
>=20
> Please consider this for v4.14, if it's not too late.
>=20
> drivers/pci/host/pci-hyperv.c | 8 +++++---
> 1 file changed, 5 insertions(+), 3 deletions(-)
>=20
> diff --git a/drivers/pci/host/pci-hyperv.c b/drivers/pci/host/pci-hyperv.=
c
> index 5ccb47d..8b5f66d 100644
> --- a/drivers/pci/host/pci-hyperv.c
> +++ b/drivers/pci/host/pci-hyperv.c
> @@ -879,7 +879,7 @@ static void hv_irq_unmask(struct irq_data *data)
> int cpu;
> u64 res;
>=20
> - dest =3D irq_data_get_affinity_mask(data);
> + dest =3D irq_data_get_effective_affinity_mask(data);
> pdev =3D msi_desc_to_pci_dev(msi_desc);
> pbus =3D pdev->bus;
> hbus =3D container_of(pbus->sysdata, struct hv_pcibus_device,
> sysdata); @@ -1042,6 +1042,7 @@ static void hv_compose_msi_msg(struct
> irq_data *data, struct msi_msg *msg)
> struct hv_pci_dev *hpdev;
> struct pci_bus *pbus;
> struct pci_dev *pdev;
> + struct cpumask *dest;
> struct compose_comp_ctxt comp;
> struct tran_int_desc *int_desc;
> struct {
> @@ -1056,6 +1057,7 @@ static void hv_compose_msi_msg(struct irq_data
> *data, struct msi_msg *msg)
> int ret;
>=20
> pdev =3D msi_desc_to_pci_dev(irq_data_get_msi_desc(data));
> + dest =3D irq_data_get_effective_affinity_mask(data);
> pbus =3D pdev->bus;
> hbus =3D container_of(pbus->sysdata, struct hv_pcibus_device,
> sysdata);
> hpdev =3D get_pcichild_wslot(hbus, devfn_to_wslot(pdev->devfn));
> @@ -1081,14 +1083,14 @@ static void hv_compose_msi_msg(struct irq_data
> *data, struct msi_msg *msg)
> switch (pci_protocol_version) {
> case PCI_PROTOCOL_VERSION_1_1:
> size =3D hv_compose_msi_req_v1(&ctxt.int_pkts.v1,
> - irq_data_get_affinity_mask(data),
> + dest,
> hpdev->desc.win_slot.slot,
> cfg->vector);
> break;
>=20
> case PCI_PROTOCOL_VERSION_1_2:
> size =3D hv_compose_msi_req_v2(&ctxt.int_pkts.v2,
> - irq_data_get_affinity_mask(data),
> + dest,
> hpdev->desc.win_slot.slot,
> cfg->vector);
> break;
> --
> 2.7.4
Signed-off-by: Jake Oshins <jakeo@microsoft.com>
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [PATCH] PCI: hv: use effective affinity mask
2017-11-01 20:52 ` Jake Oshins
(?)
@ 2017-11-08 0:15 ` Bjorn Helgaas
2017-11-08 0:51 ` Jake Oshins
-1 siblings, 1 reply; 14+ messages in thread
From: Bjorn Helgaas @ 2017-11-08 0:15 UTC (permalink / raw)
To: Jake Oshins
Cc: Dexuan Cui, Bjorn Helgaas, linux-pci, KY Srinivasan,
Stephen Hemminger, devel, linux-kernel, Haiyang Zhang,
Jork Loeser, Chris Valean (Cloudbase Solutions SRL),
Adrian Suhov (Cloudbase Solutions SRL),
Simon Xiao, 'Eyal Mizrachi',
Jack Morgenstein, Armen Guezalian, Firas Mahameed,
Tziporet Koren, Daniel Jurgens
On Wed, Nov 01, 2017 at 08:52:56PM +0000, Jake Oshins wrote:
> > -----Original Message-----
> > From: Dexuan Cui
> > Sent: Wednesday, November 1, 2017 1:31 PM
> > To: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; Jake
> > Oshins <jakeo@microsoft.com>; KY Srinivasan <kys@microsoft.com>;
> > Stephen Hemminger <sthemmin@microsoft.com>
> > Cc: devel@linuxdriverproject.org; linux-kernel@vger.kernel.org; Haiyang
> > Zhang <haiyangz@microsoft.com>; Jork Loeser
> > <Jork.Loeser@microsoft.com>; Chris Valean (Cloudbase Solutions SRL) <v-
> > chvale@microsoft.com>; Adrian Suhov (Cloudbase Solutions SRL) <v-
> > adsuho@microsoft.com>; Simon Xiao <sixiao@microsoft.com>; 'Eyal
> > Mizrachi' <eyalmi@mellanox.com>; Jack Morgenstein
> > <jackm@mellanox.com>; Armen Guezalian <armeng@mellanox.com>; Firas
> > Mahameed <firas@mellanox.com>; Tziporet Koren
> > <tziporet@mellanox.com>; Daniel Jurgens <danielj@mellanox.com>
> > Subject: [PATCH] PCI: hv: use effective affinity mask
> >
> >
> > The effective_affinity_mask is always set when an interrupt is assigned in
> > __assign_irq_vector() -> apic->cpu_mask_to_apicid(), e.g. for struct apic
> > apic_physflat: -> default_cpu_mask_to_apicid() ->
> > irq_data_update_effective_affinity(), but it looks d->common->affinity
> > remains all-1's before the user space or the kernel changes it later.
> >
> > In the early allocation/initialization phase of an irq, we should use the
> > effective_affinity_mask, otherwise Hyper-V may not deliver the interrupt to
> > the expected cpu. Without the patch, if we assign 7 Mellanox ConnectX-3
> > VFs to a 32-vCPU VM, one of the VFs may fail to receive interrupts.
> >
> > Signed-off-by: Dexuan Cui <decui@microsoft.com>
> > Cc: Jake Oshins <jakeo@microsoft.com>
> > Cc: Jork Loeser <jloeser@microsoft.com>
> > Cc: Stephen Hemminger <sthemmin@microsoft.com>
> > Cc: K. Y. Srinivasan <kys@microsoft.com>
> > ---
> >
> > Please consider this for v4.14, if it's not too late.
> >
> > drivers/pci/host/pci-hyperv.c | 8 +++++---
> > 1 file changed, 5 insertions(+), 3 deletions(-)
> >
> > diff --git a/drivers/pci/host/pci-hyperv.c b/drivers/pci/host/pci-hyperv.c
> > index 5ccb47d..8b5f66d 100644
> > --- a/drivers/pci/host/pci-hyperv.c
> > +++ b/drivers/pci/host/pci-hyperv.c
> > @@ -879,7 +879,7 @@ static void hv_irq_unmask(struct irq_data *data)
> > int cpu;
> > u64 res;
> >
> > - dest = irq_data_get_affinity_mask(data);
> > + dest = irq_data_get_effective_affinity_mask(data);
> > pdev = msi_desc_to_pci_dev(msi_desc);
> > pbus = pdev->bus;
> > hbus = container_of(pbus->sysdata, struct hv_pcibus_device,
> > sysdata); @@ -1042,6 +1042,7 @@ static void hv_compose_msi_msg(struct
> > irq_data *data, struct msi_msg *msg)
> > struct hv_pci_dev *hpdev;
> > struct pci_bus *pbus;
> > struct pci_dev *pdev;
> > + struct cpumask *dest;
> > struct compose_comp_ctxt comp;
> > struct tran_int_desc *int_desc;
> > struct {
> > @@ -1056,6 +1057,7 @@ static void hv_compose_msi_msg(struct irq_data
> > *data, struct msi_msg *msg)
> > int ret;
> >
> > pdev = msi_desc_to_pci_dev(irq_data_get_msi_desc(data));
> > + dest = irq_data_get_effective_affinity_mask(data);
> > pbus = pdev->bus;
> > hbus = container_of(pbus->sysdata, struct hv_pcibus_device,
> > sysdata);
> > hpdev = get_pcichild_wslot(hbus, devfn_to_wslot(pdev->devfn));
> > @@ -1081,14 +1083,14 @@ static void hv_compose_msi_msg(struct irq_data
> > *data, struct msi_msg *msg)
> > switch (pci_protocol_version) {
> > case PCI_PROTOCOL_VERSION_1_1:
> > size = hv_compose_msi_req_v1(&ctxt.int_pkts.v1,
> > - irq_data_get_affinity_mask(data),
> > + dest,
> > hpdev->desc.win_slot.slot,
> > cfg->vector);
> > break;
> >
> > case PCI_PROTOCOL_VERSION_1_2:
> > size = hv_compose_msi_req_v2(&ctxt.int_pkts.v2,
> > - irq_data_get_affinity_mask(data),
> > + dest,
> > hpdev->desc.win_slot.slot,
> > cfg->vector);
> > break;
> > --
> > 2.7.4
>
> Signed-off-by: Jake Oshins <jakeo@microsoft.com>
I'm not sure what this means.
Per Documentation/process/submitting-patches.rst, "Signed-off-by"
means you were "involved in the development of the patch, or that
he/she was in the patch's delivery path." You weren't in the delivery
path (I got it from Dexuan), and if you were involved in development,
your Signed-off-by would normally appear in the original posting.
Should this be a Reviewed-by tag?
^ permalink raw reply [flat|nested] 14+ messages in thread* RE: [PATCH] PCI: hv: use effective affinity mask
2017-11-08 0:15 ` Bjorn Helgaas
@ 2017-11-08 0:51 ` Jake Oshins
0 siblings, 0 replies; 14+ messages in thread
From: Jake Oshins @ 2017-11-08 0:51 UTC (permalink / raw)
To: Bjorn Helgaas
Cc: Dexuan Cui, Bjorn Helgaas, linux-pci, KY Srinivasan,
Stephen Hemminger, devel, linux-kernel, Haiyang Zhang,
Jork Loeser, Chris Valean (Cloudbase Solutions SRL),
Adrian Suhov (Cloudbase Solutions SRL),
Simon Xiao, 'Eyal Mizrachi',
Jack Morgenstein, Armen Guezalian, Firas Mahameed,
Tziporet Koren, Daniel Jurgens
> -----Original Message-----
> From: Bjorn Helgaas [mailto:helgaas@kernel.org]
> Sent: Tuesday, November 7, 2017 4:15 PM
> To: Jake Oshins <jakeo@microsoft.com>
> Cc: Dexuan Cui <decui@microsoft.com>; Bjorn Helgaas
> <bhelgaas@google.com>; linux-pci@vger.kernel.org; KY Srinivasan
> <kys@microsoft.com>; Stephen Hemminger <sthemmin@microsoft.com>;
> devel@linuxdriverproject.org; linux-kernel@vger.kernel.org; Haiyang Zhang
> <haiyangz@microsoft.com>; Jork Loeser <Jork.Loeser@microsoft.com>;
> Chris Valean (Cloudbase Solutions SRL) <v-chvale@microsoft.com>; Adrian
> Suhov (Cloudbase Solutions SRL) <v-adsuho@microsoft.com>; Simon Xiao
> <sixiao@microsoft.com>; 'Eyal Mizrachi' <eyalmi@mellanox.com>; Jack
> Morgenstein <jackm@mellanox.com>; Armen Guezalian
> <armeng@mellanox.com>; Firas Mahameed <firas@mellanox.com>;
> Tziporet Koren <tziporet@mellanox.com>; Daniel Jurgens
> <danielj@mellanox.com>
> Subject: Re: [PATCH] PCI: hv: use effective affinity mask
>
> On Wed, Nov 01, 2017 at 08:52:56PM +0000, Jake Oshins wrote:
> > > -----Original Message-----
> > > From: Dexuan Cui
> > > Sent: Wednesday, November 1, 2017 1:31 PM
> > > To: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org;
> > > Jake Oshins <jakeo@microsoft.com>; KY Srinivasan
> > > <kys@microsoft.com>; Stephen Hemminger
> <sthemmin@microsoft.com>
> > > Cc: devel@linuxdriverproject.org; linux-kernel@vger.kernel.org;
> > > Haiyang Zhang <haiyangz@microsoft.com>; Jork Loeser
> > > <Jork.Loeser@microsoft.com>; Chris Valean (Cloudbase Solutions SRL)
> > > <v- chvale@microsoft.com>; Adrian Suhov (Cloudbase Solutions SRL)
> > > <v- adsuho@microsoft.com>; Simon Xiao <sixiao@microsoft.com>; 'Eyal
> > > Mizrachi' <eyalmi@mellanox.com>; Jack Morgenstein
> > > <jackm@mellanox.com>; Armen Guezalian <armeng@mellanox.com>;
> Firas
> > > Mahameed <firas@mellanox.com>; Tziporet Koren
> > > <tziporet@mellanox.com>; Daniel Jurgens <danielj@mellanox.com>
> > > Subject: [PATCH] PCI: hv: use effective affinity mask
> > >
> > >
> > > The effective_affinity_mask is always set when an interrupt is
> > > assigned in
> > > __assign_irq_vector() -> apic->cpu_mask_to_apicid(), e.g. for struct
> > > apic
> > > apic_physflat: -> default_cpu_mask_to_apicid() ->
> > > irq_data_update_effective_affinity(), but it looks
> > > d->common->affinity remains all-1's before the user space or the kernel
> changes it later.
> > >
> > > In the early allocation/initialization phase of an irq, we should
> > > use the effective_affinity_mask, otherwise Hyper-V may not deliver
> > > the interrupt to the expected cpu. Without the patch, if we assign 7
> > > Mellanox ConnectX-3 VFs to a 32-vCPU VM, one of the VFs may fail to
> receive interrupts.
> > >
> > > Signed-off-by: Dexuan Cui <decui@microsoft.com>
> > > Cc: Jake Oshins <jakeo@microsoft.com>
> > > Cc: Jork Loeser <jloeser@microsoft.com>
> > > Cc: Stephen Hemminger <sthemmin@microsoft.com>
> > > Cc: K. Y. Srinivasan <kys@microsoft.com>
> > > ---
> > >
> > > Please consider this for v4.14, if it's not too late.
> > >
> > > drivers/pci/host/pci-hyperv.c | 8 +++++---
> > > 1 file changed, 5 insertions(+), 3 deletions(-)
> > >
> > > diff --git a/drivers/pci/host/pci-hyperv.c
> > > b/drivers/pci/host/pci-hyperv.c index 5ccb47d..8b5f66d 100644
> > > --- a/drivers/pci/host/pci-hyperv.c
> > > +++ b/drivers/pci/host/pci-hyperv.c
> > > @@ -879,7 +879,7 @@ static void hv_irq_unmask(struct irq_data *data)
> > > int cpu;
> > > u64 res;
> > >
> > > - dest = irq_data_get_affinity_mask(data);
> > > + dest = irq_data_get_effective_affinity_mask(data);
> > > pdev = msi_desc_to_pci_dev(msi_desc);
> > > pbus = pdev->bus;
> > > hbus = container_of(pbus->sysdata, struct hv_pcibus_device,
> > > sysdata); @@ -1042,6 +1042,7 @@ static void
> > > hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
> > > struct hv_pci_dev *hpdev;
> > > struct pci_bus *pbus;
> > > struct pci_dev *pdev;
> > > + struct cpumask *dest;
> > > struct compose_comp_ctxt comp;
> > > struct tran_int_desc *int_desc;
> > > struct {
> > > @@ -1056,6 +1057,7 @@ static void hv_compose_msi_msg(struct
> irq_data
> > > *data, struct msi_msg *msg)
> > > int ret;
> > >
> > > pdev = msi_desc_to_pci_dev(irq_data_get_msi_desc(data));
> > > + dest = irq_data_get_effective_affinity_mask(data);
> > > pbus = pdev->bus;
> > > hbus = container_of(pbus->sysdata, struct hv_pcibus_device,
> > > sysdata);
> > > hpdev = get_pcichild_wslot(hbus, devfn_to_wslot(pdev->devfn));
> @@
> > > -1081,14 +1083,14 @@ static void hv_compose_msi_msg(struct irq_data
> > > *data, struct msi_msg *msg)
> > > switch (pci_protocol_version) {
> > > case PCI_PROTOCOL_VERSION_1_1:
> > > size = hv_compose_msi_req_v1(&ctxt.int_pkts.v1,
> > > - irq_data_get_affinity_mask(data),
> > > + dest,
> > > hpdev->desc.win_slot.slot,
> > > cfg->vector);
> > > break;
> > >
> > > case PCI_PROTOCOL_VERSION_1_2:
> > > size = hv_compose_msi_req_v2(&ctxt.int_pkts.v2,
> > > - irq_data_get_affinity_mask(data),
> > > + dest,
> > > hpdev->desc.win_slot.slot,
> > > cfg->vector);
> > > break;
> > > --
> > > 2.7.4
> >
> > Signed-off-by: Jake Oshins <jakeo@microsoft.com>
>
> I'm not sure what this means.
>
> Per Documentation/process/submitting-patches.rst, "Signed-off-by"
> means you were "involved in the development of the patch, or that he/she
> was in the patch's delivery path." You weren't in the delivery path (I got it
> from Dexuan), and if you were involved in development, your Signed-off-by
> would normally appear in the original posting.
>
> Should this be a Reviewed-by tag?
Sorry. I forgot the protocol. I'll fix that.
Reviewed-by: Jake Oshins <jakeo@microsoft.com>
^ permalink raw reply [flat|nested] 14+ messages in thread* RE: [PATCH] PCI: hv: use effective affinity mask
@ 2017-11-08 0:51 ` Jake Oshins
0 siblings, 0 replies; 14+ messages in thread
From: Jake Oshins @ 2017-11-08 0:51 UTC (permalink / raw)
To: Bjorn Helgaas
Cc: Dexuan Cui, Bjorn Helgaas, linux-pci, KY Srinivasan,
Stephen Hemminger, devel, linux-kernel, Haiyang Zhang,
Jork Loeser, Chris Valean (Cloudbase Solutions SRL),
Adrian Suhov (Cloudbase Solutions SRL),
Simon Xiao, 'Eyal Mizrachi',
Jack Morgenstein, Armen Guezalian, Firas Mahameed,
Tziporet Koren, Daniel Jurgens
> -----Original Message-----
> From: Bjorn Helgaas [mailto:helgaas@kernel.org]
> Sent: Tuesday, November 7, 2017 4:15 PM
> To: Jake Oshins <jakeo@microsoft.com>
> Cc: Dexuan Cui <decui@microsoft.com>; Bjorn Helgaas
> <bhelgaas@google.com>; linux-pci@vger.kernel.org; KY Srinivasan
> <kys@microsoft.com>; Stephen Hemminger <sthemmin@microsoft.com>;
> devel@linuxdriverproject.org; linux-kernel@vger.kernel.org; Haiyang Zhang
> <haiyangz@microsoft.com>; Jork Loeser <Jork.Loeser@microsoft.com>;
> Chris Valean (Cloudbase Solutions SRL) <v-chvale@microsoft.com>; Adrian
> Suhov (Cloudbase Solutions SRL) <v-adsuho@microsoft.com>; Simon Xiao
> <sixiao@microsoft.com>; 'Eyal Mizrachi' <eyalmi@mellanox.com>; Jack
> Morgenstein <jackm@mellanox.com>; Armen Guezalian
> <armeng@mellanox.com>; Firas Mahameed <firas@mellanox.com>;
> Tziporet Koren <tziporet@mellanox.com>; Daniel Jurgens
> <danielj@mellanox.com>
> Subject: Re: [PATCH] PCI: hv: use effective affinity mask
>=20
> On Wed, Nov 01, 2017 at 08:52:56PM +0000, Jake Oshins wrote:
> > > -----Original Message-----
> > > From: Dexuan Cui
> > > Sent: Wednesday, November 1, 2017 1:31 PM
> > > To: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org;
> > > Jake Oshins <jakeo@microsoft.com>; KY Srinivasan
> > > <kys@microsoft.com>; Stephen Hemminger
> <sthemmin@microsoft.com>
> > > Cc: devel@linuxdriverproject.org; linux-kernel@vger.kernel.org;
> > > Haiyang Zhang <haiyangz@microsoft.com>; Jork Loeser
> > > <Jork.Loeser@microsoft.com>; Chris Valean (Cloudbase Solutions SRL)
> > > <v- chvale@microsoft.com>; Adrian Suhov (Cloudbase Solutions SRL)
> > > <v- adsuho@microsoft.com>; Simon Xiao <sixiao@microsoft.com>; 'Eyal
> > > Mizrachi' <eyalmi@mellanox.com>; Jack Morgenstein
> > > <jackm@mellanox.com>; Armen Guezalian <armeng@mellanox.com>;
> Firas
> > > Mahameed <firas@mellanox.com>; Tziporet Koren
> > > <tziporet@mellanox.com>; Daniel Jurgens <danielj@mellanox.com>
> > > Subject: [PATCH] PCI: hv: use effective affinity mask
> > >
> > >
> > > The effective_affinity_mask is always set when an interrupt is
> > > assigned in
> > > __assign_irq_vector() -> apic->cpu_mask_to_apicid(), e.g. for struct
> > > apic
> > > apic_physflat: -> default_cpu_mask_to_apicid() ->
> > > irq_data_update_effective_affinity(), but it looks
> > > d->common->affinity remains all-1's before the user space or the kern=
el
> changes it later.
> > >
> > > In the early allocation/initialization phase of an irq, we should
> > > use the effective_affinity_mask, otherwise Hyper-V may not deliver
> > > the interrupt to the expected cpu. Without the patch, if we assign 7
> > > Mellanox ConnectX-3 VFs to a 32-vCPU VM, one of the VFs may fail to
> receive interrupts.
> > >
> > > Signed-off-by: Dexuan Cui <decui@microsoft.com>
> > > Cc: Jake Oshins <jakeo@microsoft.com>
> > > Cc: Jork Loeser <jloeser@microsoft.com>
> > > Cc: Stephen Hemminger <sthemmin@microsoft.com>
> > > Cc: K. Y. Srinivasan <kys@microsoft.com>
> > > ---
> > >
> > > Please consider this for v4.14, if it's not too late.
> > >
> > > drivers/pci/host/pci-hyperv.c | 8 +++++---
> > > 1 file changed, 5 insertions(+), 3 deletions(-)
> > >
> > > diff --git a/drivers/pci/host/pci-hyperv.c
> > > b/drivers/pci/host/pci-hyperv.c index 5ccb47d..8b5f66d 100644
> > > --- a/drivers/pci/host/pci-hyperv.c
> > > +++ b/drivers/pci/host/pci-hyperv.c
> > > @@ -879,7 +879,7 @@ static void hv_irq_unmask(struct irq_data *data)
> > > int cpu;
> > > u64 res;
> > >
> > > - dest =3D irq_data_get_affinity_mask(data);
> > > + dest =3D irq_data_get_effective_affinity_mask(data);
> > > pdev =3D msi_desc_to_pci_dev(msi_desc);
> > > pbus =3D pdev->bus;
> > > hbus =3D container_of(pbus->sysdata, struct hv_pcibus_device,
> > > sysdata); @@ -1042,6 +1042,7 @@ static void
> > > hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
> > > struct hv_pci_dev *hpdev;
> > > struct pci_bus *pbus;
> > > struct pci_dev *pdev;
> > > + struct cpumask *dest;
> > > struct compose_comp_ctxt comp;
> > > struct tran_int_desc *int_desc;
> > > struct {
> > > @@ -1056,6 +1057,7 @@ static void hv_compose_msi_msg(struct
> irq_data
> > > *data, struct msi_msg *msg)
> > > int ret;
> > >
> > > pdev =3D msi_desc_to_pci_dev(irq_data_get_msi_desc(data));
> > > + dest =3D irq_data_get_effective_affinity_mask(data);
> > > pbus =3D pdev->bus;
> > > hbus =3D container_of(pbus->sysdata, struct hv_pcibus_device,
> > > sysdata);
> > > hpdev =3D get_pcichild_wslot(hbus, devfn_to_wslot(pdev->devfn));
> @@
> > > -1081,14 +1083,14 @@ static void hv_compose_msi_msg(struct irq_data
> > > *data, struct msi_msg *msg)
> > > switch (pci_protocol_version) {
> > > case PCI_PROTOCOL_VERSION_1_1:
> > > size =3D hv_compose_msi_req_v1(&ctxt.int_pkts.v1,
> > > - irq_data_get_affinity_mask(data),
> > > + dest,
> > > hpdev->desc.win_slot.slot,
> > > cfg->vector);
> > > break;
> > >
> > > case PCI_PROTOCOL_VERSION_1_2:
> > > size =3D hv_compose_msi_req_v2(&ctxt.int_pkts.v2,
> > > - irq_data_get_affinity_mask(data),
> > > + dest,
> > > hpdev->desc.win_slot.slot,
> > > cfg->vector);
> > > break;
> > > --
> > > 2.7.4
> >
> > Signed-off-by: Jake Oshins <jakeo@microsoft.com>
>=20
> I'm not sure what this means.
>=20
> Per Documentation/process/submitting-patches.rst, "Signed-off-by"
> means you were "involved in the development of the patch, or that he/she
> was in the patch's delivery path." You weren't in the delivery path (I g=
ot it
> from Dexuan), and if you were involved in development, your Signed-off-by
> would normally appear in the original posting.
>=20
> Should this be a Reviewed-by tag?
Sorry. I forgot the protocol. I'll fix that.
Reviewed-by: Jake Oshins <jakeo@microsoft.com>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] PCI: hv: use effective affinity mask
2017-11-01 20:30 ` Dexuan Cui
(?)
(?)
@ 2017-11-08 1:07 ` Bjorn Helgaas
2017-11-08 1:27 ` Dexuan Cui
2017-11-10 8:55 ` Adrian Suhov (Cloudbase Solutions SRL)
-1 siblings, 2 replies; 14+ messages in thread
From: Bjorn Helgaas @ 2017-11-08 1:07 UTC (permalink / raw)
To: Dexuan Cui
Cc: Bjorn Helgaas, linux-pci, Jake Oshins, KY Srinivasan,
Stephen Hemminger, devel, linux-kernel, Haiyang Zhang,
Jork Loeser, Chris Valean (Cloudbase Solutions SRL),
Adrian Suhov (Cloudbase Solutions SRL),
Simon Xiao, 'Eyal Mizrachi',
Jack Morgenstein, Armen Guezalian, Firas Mahameed,
Tziporet Koren, Daniel Jurgens
On Wed, Nov 01, 2017 at 08:30:53PM +0000, Dexuan Cui wrote:
>
> The effective_affinity_mask is always set when an interrupt is assigned in
> __assign_irq_vector() -> apic->cpu_mask_to_apicid(), e.g. for struct apic
> apic_physflat: -> default_cpu_mask_to_apicid() ->
> irq_data_update_effective_affinity(), but it looks d->common->affinity
> remains all-1's before the user space or the kernel changes it later.
>
> In the early allocation/initialization phase of an irq, we should use the
> effective_affinity_mask, otherwise Hyper-V may not deliver the interrupt
> to the expected cpu. Without the patch, if we assign 7 Mellanox ConnectX-3
> VFs to a 32-vCPU VM, one of the VFs may fail to receive interrupts.
>
> Signed-off-by: Dexuan Cui <decui@microsoft.com>
> Cc: Jake Oshins <jakeo@microsoft.com>
> Cc: Jork Loeser <jloeser@microsoft.com>
> Cc: Stephen Hemminger <sthemmin@microsoft.com>
> Cc: K. Y. Srinivasan <kys@microsoft.com>
> ---
>
> Please consider this for v4.14, if it's not too late.
What would be the rationale for putting it in v4.14? After the merge
window, I usually only merge fixes for problems introduced during the
merge window, or for serious regressions. I can't tell if this fits
into that or not.
> drivers/pci/host/pci-hyperv.c | 8 +++++---
> 1 file changed, 5 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/pci/host/pci-hyperv.c b/drivers/pci/host/pci-hyperv.c
> index 5ccb47d..8b5f66d 100644
> --- a/drivers/pci/host/pci-hyperv.c
> +++ b/drivers/pci/host/pci-hyperv.c
> @@ -879,7 +879,7 @@ static void hv_irq_unmask(struct irq_data *data)
> int cpu;
> u64 res;
>
> - dest = irq_data_get_affinity_mask(data);
> + dest = irq_data_get_effective_affinity_mask(data);
> pdev = msi_desc_to_pci_dev(msi_desc);
> pbus = pdev->bus;
> hbus = container_of(pbus->sysdata, struct hv_pcibus_device, sysdata);
> @@ -1042,6 +1042,7 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
> struct hv_pci_dev *hpdev;
> struct pci_bus *pbus;
> struct pci_dev *pdev;
> + struct cpumask *dest;
> struct compose_comp_ctxt comp;
> struct tran_int_desc *int_desc;
> struct {
> @@ -1056,6 +1057,7 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
> int ret;
>
> pdev = msi_desc_to_pci_dev(irq_data_get_msi_desc(data));
> + dest = irq_data_get_effective_affinity_mask(data);
> pbus = pdev->bus;
> hbus = container_of(pbus->sysdata, struct hv_pcibus_device, sysdata);
> hpdev = get_pcichild_wslot(hbus, devfn_to_wslot(pdev->devfn));
> @@ -1081,14 +1083,14 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
> switch (pci_protocol_version) {
> case PCI_PROTOCOL_VERSION_1_1:
> size = hv_compose_msi_req_v1(&ctxt.int_pkts.v1,
> - irq_data_get_affinity_mask(data),
> + dest,
> hpdev->desc.win_slot.slot,
> cfg->vector);
> break;
>
> case PCI_PROTOCOL_VERSION_1_2:
> size = hv_compose_msi_req_v2(&ctxt.int_pkts.v2,
> - irq_data_get_affinity_mask(data),
> + dest,
> hpdev->desc.win_slot.slot,
> cfg->vector);
> break;
> --
> 2.7.4
>
^ permalink raw reply [flat|nested] 14+ messages in thread* RE: [PATCH] PCI: hv: use effective affinity mask
2017-11-08 1:07 ` Bjorn Helgaas
@ 2017-11-08 1:27 ` Dexuan Cui
2017-11-10 8:55 ` Adrian Suhov (Cloudbase Solutions SRL)
1 sibling, 0 replies; 14+ messages in thread
From: Dexuan Cui @ 2017-11-08 1:27 UTC (permalink / raw)
To: Bjorn Helgaas
Cc: Bjorn Helgaas, linux-pci, Jake Oshins, KY Srinivasan,
Stephen Hemminger, devel, linux-kernel, Haiyang Zhang,
Jork Loeser, Chris Valean (Cloudbase Solutions SRL),
Adrian Suhov (Cloudbase Solutions SRL),
Simon Xiao, 'Eyal Mizrachi',
Jack Morgenstein, Armen Guezalian, Firas Mahameed,
Tziporet Koren, Daniel Jurgens
> From: Bjorn Helgaas [mailto:helgaas@kernel.org]
> Sent: Tuesday, November 7, 2017 5:08 PM
> On Wed, Nov 01, 2017 at 08:30:53PM +0000, Dexuan Cui wrote:
> >
> > Please consider this for v4.14, if it's not too late.
>
> What would be the rationale for putting it in v4.14? After the merge
> window, I usually only merge fixes for problems introduced during the
> merge window, or for serious regressions. I can't tell if this fits
> into that or not.
The patch was sent last Wednesday and I hoped it could catch the
merge window to be in v4.14 so we won't have to request a backport
for the v4.14 stable kernel in future. And I was not exactly sure when
the merge window was. Sorry :-)
The patch is not fixing a serious regression. It just fixes a long standing
issue from the beginning of the Hyper-V vPCI driver: 1 of 7 Mellanox
VFs of a 32-vCPU VM running on Hyper-V may not receive interrupts.
Thanks,
-- Dexuan
^ permalink raw reply [flat|nested] 14+ messages in thread
* RE: [PATCH] PCI: hv: use effective affinity mask
@ 2017-11-08 1:27 ` Dexuan Cui
0 siblings, 0 replies; 14+ messages in thread
From: Dexuan Cui @ 2017-11-08 1:27 UTC (permalink / raw)
To: Bjorn Helgaas
Cc: Bjorn Helgaas, linux-pci, Jake Oshins, KY Srinivasan,
Stephen Hemminger, devel, linux-kernel, Haiyang Zhang,
Jork Loeser, Chris Valean (Cloudbase Solutions SRL),
Adrian Suhov (Cloudbase Solutions SRL),
Simon Xiao, 'Eyal Mizrachi',
Jack Morgenstein, Armen Guezalian, Firas Mahameed,
Tziporet Koren, Daniel Jurgens
> From: Bjorn Helgaas [mailto:helgaas@kernel.org]
> Sent: Tuesday, November 7, 2017 5:08 PM
> On Wed, Nov 01, 2017 at 08:30:53PM +0000, Dexuan Cui wrote:
> >
> > Please consider this for v4.14, if it's not too late.
>=20
> What would be the rationale for putting it in v4.14? After the merge
> window, I usually only merge fixes for problems introduced during the
> merge window, or for serious regressions. I can't tell if this fits
> into that or not.
The patch was sent last Wednesday and I hoped it could catch the=20
merge window to be in v4.14 so we won't have to request a backport
for the v4.14 stable kernel in future. And I was not exactly sure when
the merge window was. Sorry :-)
The patch is not fixing a serious regression. It just fixes a long standing
issue from the beginning of the Hyper-V vPCI driver: 1 of 7 Mellanox=20
VFs of a 32-vCPU VM running on Hyper-V may not receive interrupts.
Thanks,
-- Dexuan
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] PCI: hv: use effective affinity mask
2017-11-08 1:27 ` Dexuan Cui
(?)
@ 2017-11-08 3:01 ` Bjorn Helgaas
-1 siblings, 0 replies; 14+ messages in thread
From: Bjorn Helgaas @ 2017-11-08 3:01 UTC (permalink / raw)
To: Dexuan Cui
Cc: Bjorn Helgaas, linux-pci, Jake Oshins, KY Srinivasan,
Stephen Hemminger, devel, linux-kernel, Haiyang Zhang,
Jork Loeser, Chris Valean (Cloudbase Solutions SRL),
Adrian Suhov (Cloudbase Solutions SRL),
Simon Xiao, 'Eyal Mizrachi',
Jack Morgenstein, Armen Guezalian, Firas Mahameed,
Tziporet Koren, Daniel Jurgens
On Wed, Nov 08, 2017 at 01:27:02AM +0000, Dexuan Cui wrote:
> > From: Bjorn Helgaas [mailto:helgaas@kernel.org]
> > Sent: Tuesday, November 7, 2017 5:08 PM
> > On Wed, Nov 01, 2017 at 08:30:53PM +0000, Dexuan Cui wrote:
> > >
> > > Please consider this for v4.14, if it's not too late.
> >
> > What would be the rationale for putting it in v4.14? After the merge
> > window, I usually only merge fixes for problems introduced during the
> > merge window, or for serious regressions. I can't tell if this fits
> > into that or not.
>
> The patch was sent last Wednesday and I hoped it could catch the
> merge window to be in v4.14 so we won't have to request a backport
> for the v4.14 stable kernel in future. And I was not exactly sure when
> the merge window was. Sorry :-)
>
> The patch is not fixing a serious regression. It just fixes a long standing
> issue from the beginning of the Hyper-V vPCI driver: 1 of 7 Mellanox
> VFs of a 32-vCPU VM running on Hyper-V may not receive interrupts.
No problem. I added a stable tag so the backport should happen
automatically and put it on my pci/host-hv branch for v4.15, with Jake's
Reviewed-by. Thanks!
Bjorn
^ permalink raw reply [flat|nested] 14+ messages in thread
* RE: [PATCH] PCI: hv: use effective affinity mask
2017-11-08 1:07 ` Bjorn Helgaas
@ 2017-11-10 8:55 ` Adrian Suhov (Cloudbase Solutions SRL)
2017-11-10 8:55 ` Adrian Suhov (Cloudbase Solutions SRL)
1 sibling, 0 replies; 14+ messages in thread
From: Adrian Suhov (Cloudbase Solutions SRL) @ 2017-11-10 8:55 UTC (permalink / raw)
To: Bjorn Helgaas, Dexuan Cui
Cc: Bjorn Helgaas, linux-pci, Jake Oshins, KY Srinivasan,
Stephen Hemminger, devel, linux-kernel, Haiyang Zhang,
Jork Loeser, Chris Valean (Cloudbase Solutions SRL),
Simon Xiao, 'Eyal Mizrachi',
Jack Morgenstein, Armen Guezalian, Firas Mahameed,
Tziporet Koren, Daniel Jurgens
Hi,
I've also tested this and it's working good. Kernels tested:
- next-20171109 on top of Ubuntu 16.04
- MSFT kernel - 4.14.0-rc5 with patch applied - on top of RHEL 7.3
Adrian
-----Original Message-----
From: Bjorn Helgaas [mailto:helgaas@kernel.org]
Sent: Wednesday, November 8, 2017 3:08 AM
To: Dexuan Cui <decui@microsoft.com>
Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; Jake Oshins <jakeo@microsoft.com>; KY Srinivasan <kys@microsoft.com>; Stephen Hemminger <sthemmin@microsoft.com>; devel@linuxdriverproject.org; linux-kernel@vger.kernel.org; Haiyang Zhang <haiyangz@microsoft.com>; Jork Loeser <Jork.Loeser@microsoft.com>; Chris Valean (Cloudbase Solutions SRL) <v-chvale@microsoft.com>; Adrian Suhov (Cloudbase Solutions SRL) <v-adsuho@microsoft.com>; Simon Xiao <sixiao@microsoft.com>; 'Eyal Mizrachi' <eyalmi@mellanox.com>; Jack Morgenstein <jackm@mellanox.com>; Armen Guezalian <armeng@mellanox.com>; Firas Mahameed <firas@mellanox.com>; Tziporet Koren <tziporet@mellanox.com>; Daniel Jurgens <danielj@mellanox.com>
Subject: Re: [PATCH] PCI: hv: use effective affinity mask
On Wed, Nov 01, 2017 at 08:30:53PM +0000, Dexuan Cui wrote:
>
> The effective_affinity_mask is always set when an interrupt is
> assigned in
> __assign_irq_vector() -> apic->cpu_mask_to_apicid(), e.g. for struct
> apic
> apic_physflat: -> default_cpu_mask_to_apicid() ->
> irq_data_update_effective_affinity(), but it looks d->common->affinity
> remains all-1's before the user space or the kernel changes it later.
>
> In the early allocation/initialization phase of an irq, we should use
> the effective_affinity_mask, otherwise Hyper-V may not deliver the
> interrupt to the expected cpu. Without the patch, if we assign 7
> Mellanox ConnectX-3 VFs to a 32-vCPU VM, one of the VFs may fail to receive interrupts.
>
> Signed-off-by: Dexuan Cui <decui@microsoft.com>
> Cc: Jake Oshins <jakeo@microsoft.com>
> Cc: Jork Loeser <jloeser@microsoft.com>
> Cc: Stephen Hemminger <sthemmin@microsoft.com>
> Cc: K. Y. Srinivasan <kys@microsoft.com>
> ---
>
> Please consider this for v4.14, if it's not too late.
What would be the rationale for putting it in v4.14? After the merge window, I usually only merge fixes for problems introduced during the merge window, or for serious regressions. I can't tell if this fits into that or not.
> drivers/pci/host/pci-hyperv.c | 8 +++++---
> 1 file changed, 5 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/pci/host/pci-hyperv.c
> b/drivers/pci/host/pci-hyperv.c index 5ccb47d..8b5f66d 100644
> --- a/drivers/pci/host/pci-hyperv.c
> +++ b/drivers/pci/host/pci-hyperv.c
> @@ -879,7 +879,7 @@ static void hv_irq_unmask(struct irq_data *data)
> int cpu;
> u64 res;
>
> - dest = irq_data_get_affinity_mask(data);
> + dest = irq_data_get_effective_affinity_mask(data);
> pdev = msi_desc_to_pci_dev(msi_desc);
> pbus = pdev->bus;
> hbus = container_of(pbus->sysdata, struct hv_pcibus_device,
> sysdata); @@ -1042,6 +1042,7 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
> struct hv_pci_dev *hpdev;
> struct pci_bus *pbus;
> struct pci_dev *pdev;
> + struct cpumask *dest;
> struct compose_comp_ctxt comp;
> struct tran_int_desc *int_desc;
> struct {
> @@ -1056,6 +1057,7 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
> int ret;
>
> pdev = msi_desc_to_pci_dev(irq_data_get_msi_desc(data));
> + dest = irq_data_get_effective_affinity_mask(data);
> pbus = pdev->bus;
> hbus = container_of(pbus->sysdata, struct hv_pcibus_device, sysdata);
> hpdev = get_pcichild_wslot(hbus, devfn_to_wslot(pdev->devfn)); @@
> -1081,14 +1083,14 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
> switch (pci_protocol_version) {
> case PCI_PROTOCOL_VERSION_1_1:
> size = hv_compose_msi_req_v1(&ctxt.int_pkts.v1,
> - irq_data_get_affinity_mask(data),
> + dest,
> hpdev->desc.win_slot.slot,
> cfg->vector);
> break;
>
> case PCI_PROTOCOL_VERSION_1_2:
> size = hv_compose_msi_req_v2(&ctxt.int_pkts.v2,
> - irq_data_get_affinity_mask(data),
> + dest,
> hpdev->desc.win_slot.slot,
> cfg->vector);
> break;
> --
> 2.7.4
>
^ permalink raw reply [flat|nested] 14+ messages in thread* RE: [PATCH] PCI: hv: use effective affinity mask
@ 2017-11-10 8:55 ` Adrian Suhov (Cloudbase Solutions SRL)
0 siblings, 0 replies; 14+ messages in thread
From: Adrian Suhov (Cloudbase Solutions SRL) @ 2017-11-10 8:55 UTC (permalink / raw)
To: Bjorn Helgaas, Dexuan Cui
Cc: Bjorn Helgaas, linux-pci, Jake Oshins, KY Srinivasan,
Stephen Hemminger, devel, linux-kernel, Haiyang Zhang,
Jork Loeser, Chris Valean (Cloudbase Solutions SRL),
Simon Xiao, 'Eyal Mizrachi',
Jack Morgenstein, Armen Guezalian, Firas Mahameed,
Tziporet Koren, Daniel Jurgens
Hi,
I've also tested this and it's working good. Kernels tested:
- next-20171109 on top of Ubuntu 16.04
- MSFT kernel - 4.14.0-rc5 with patch applied - on top of RHEL 7.3
Adrian
-----Original Message-----
From: Bjorn Helgaas [mailto:helgaas@kernel.org]=20
Sent: Wednesday, November 8, 2017 3:08 AM
To: Dexuan Cui <decui@microsoft.com>
Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; Jake Os=
hins <jakeo@microsoft.com>; KY Srinivasan <kys@microsoft.com>; Stephen Hemm=
inger <sthemmin@microsoft.com>; devel@linuxdriverproject.org; linux-kernel@=
vger.kernel.org; Haiyang Zhang <haiyangz@microsoft.com>; Jork Loeser <Jork.=
Loeser@microsoft.com>; Chris Valean (Cloudbase Solutions SRL) <v-chvale@mic=
rosoft.com>; Adrian Suhov (Cloudbase Solutions SRL) <v-adsuho@microsoft.com=
>; Simon Xiao <sixiao@microsoft.com>; 'Eyal Mizrachi' <eyalmi@mellanox.com>=
; Jack Morgenstein <jackm@mellanox.com>; Armen Guezalian <armeng@mellanox.c=
om>; Firas Mahameed <firas@mellanox.com>; Tziporet Koren <tziporet@mellanox=
.com>; Daniel Jurgens <danielj@mellanox.com>
Subject: Re: [PATCH] PCI: hv: use effective affinity mask
On Wed, Nov 01, 2017 at 08:30:53PM +0000, Dexuan Cui wrote:
>=20
> The effective_affinity_mask is always set when an interrupt is=20
> assigned in
> __assign_irq_vector() -> apic->cpu_mask_to_apicid(), e.g. for struct=20
> apic
> apic_physflat: -> default_cpu_mask_to_apicid() ->=20
> irq_data_update_effective_affinity(), but it looks d->common->affinity=20
> remains all-1's before the user space or the kernel changes it later.
>=20
> In the early allocation/initialization phase of an irq, we should use=20
> the effective_affinity_mask, otherwise Hyper-V may not deliver the=20
> interrupt to the expected cpu. Without the patch, if we assign 7=20
> Mellanox ConnectX-3 VFs to a 32-vCPU VM, one of the VFs may fail to recei=
ve interrupts.
>=20
> Signed-off-by: Dexuan Cui <decui@microsoft.com>
> Cc: Jake Oshins <jakeo@microsoft.com>
> Cc: Jork Loeser <jloeser@microsoft.com>
> Cc: Stephen Hemminger <sthemmin@microsoft.com>
> Cc: K. Y. Srinivasan <kys@microsoft.com>
> ---
>=20
> Please consider this for v4.14, if it's not too late.
What would be the rationale for putting it in v4.14? After the merge windo=
w, I usually only merge fixes for problems introduced during the merge wind=
ow, or for serious regressions. I can't tell if this fits into that or not=
.
> drivers/pci/host/pci-hyperv.c | 8 +++++---
> 1 file changed, 5 insertions(+), 3 deletions(-)
>=20
> diff --git a/drivers/pci/host/pci-hyperv.c=20
> b/drivers/pci/host/pci-hyperv.c index 5ccb47d..8b5f66d 100644
> --- a/drivers/pci/host/pci-hyperv.c
> +++ b/drivers/pci/host/pci-hyperv.c
> @@ -879,7 +879,7 @@ static void hv_irq_unmask(struct irq_data *data)
> int cpu;
> u64 res;
> =20
> - dest =3D irq_data_get_affinity_mask(data);
> + dest =3D irq_data_get_effective_affinity_mask(data);
> pdev =3D msi_desc_to_pci_dev(msi_desc);
> pbus =3D pdev->bus;
> hbus =3D container_of(pbus->sysdata, struct hv_pcibus_device,=20
> sysdata); @@ -1042,6 +1042,7 @@ static void hv_compose_msi_msg(struct irq=
_data *data, struct msi_msg *msg)
> struct hv_pci_dev *hpdev;
> struct pci_bus *pbus;
> struct pci_dev *pdev;
> + struct cpumask *dest;
> struct compose_comp_ctxt comp;
> struct tran_int_desc *int_desc;
> struct {
> @@ -1056,6 +1057,7 @@ static void hv_compose_msi_msg(struct irq_data *dat=
a, struct msi_msg *msg)
> int ret;
> =20
> pdev =3D msi_desc_to_pci_dev(irq_data_get_msi_desc(data));
> + dest =3D irq_data_get_effective_affinity_mask(data);
> pbus =3D pdev->bus;
> hbus =3D container_of(pbus->sysdata, struct hv_pcibus_device, sysdata);
> hpdev =3D get_pcichild_wslot(hbus, devfn_to_wslot(pdev->devfn)); @@=20
> -1081,14 +1083,14 @@ static void hv_compose_msi_msg(struct irq_data *data=
, struct msi_msg *msg)
> switch (pci_protocol_version) {
> case PCI_PROTOCOL_VERSION_1_1:
> size =3D hv_compose_msi_req_v1(&ctxt.int_pkts.v1,
> - irq_data_get_affinity_mask(data),
> + dest,
> hpdev->desc.win_slot.slot,
> cfg->vector);
> break;
> =20
> case PCI_PROTOCOL_VERSION_1_2:
> size =3D hv_compose_msi_req_v2(&ctxt.int_pkts.v2,
> - irq_data_get_affinity_mask(data),
> + dest,
> hpdev->desc.win_slot.slot,
> cfg->vector);
> break;
> --
> 2.7.4
>=20
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [PATCH] PCI: hv: use effective affinity mask
2017-11-10 8:55 ` Adrian Suhov (Cloudbase Solutions SRL)
(?)
@ 2017-11-10 18:14 ` Bjorn Helgaas
-1 siblings, 0 replies; 14+ messages in thread
From: Bjorn Helgaas @ 2017-11-10 18:14 UTC (permalink / raw)
To: Adrian Suhov (Cloudbase Solutions SRL)
Cc: Dexuan Cui, Bjorn Helgaas, linux-pci, Jake Oshins, KY Srinivasan,
Stephen Hemminger, devel, linux-kernel, Haiyang Zhang,
Jork Loeser, Chris Valean (Cloudbase Solutions SRL),
Simon Xiao, 'Eyal Mizrachi',
Jack Morgenstein, Armen Guezalian, Firas Mahameed,
Tziporet Koren, Daniel Jurgens
On Fri, Nov 10, 2017 at 08:55:07AM +0000, Adrian Suhov (Cloudbase Solutions SRL) wrote:
> Hi,
>
> I've also tested this and it's working good. Kernels tested:
> - next-20171109 on top of Ubuntu 16.04
> - MSFT kernel - 4.14.0-rc5 with patch applied - on top of RHEL 7.3
>
> Adrian
Thanks, Adrian. I added this to the patch:
Tested-by: Adrian Suhov <v-adsuho@microsoft.com>
> -----Original Message-----
> From: Bjorn Helgaas [mailto:helgaas@kernel.org]
> Sent: Wednesday, November 8, 2017 3:08 AM
> To: Dexuan Cui <decui@microsoft.com>
> Cc: Bjorn Helgaas <bhelgaas@google.com>; linux-pci@vger.kernel.org; Jake Oshins <jakeo@microsoft.com>; KY Srinivasan <kys@microsoft.com>; Stephen Hemminger <sthemmin@microsoft.com>; devel@linuxdriverproject.org; linux-kernel@vger.kernel.org; Haiyang Zhang <haiyangz@microsoft.com>; Jork Loeser <Jork.Loeser@microsoft.com>; Chris Valean (Cloudbase Solutions SRL) <v-chvale@microsoft.com>; Adrian Suhov (Cloudbase Solutions SRL) <v-adsuho@microsoft.com>; Simon Xiao <sixiao@microsoft.com>; 'Eyal Mizrachi' <eyalmi@mellanox.com>; Jack Morgenstein <jackm@mellanox.com>; Armen Guezalian <armeng@mellanox.com>; Firas Mahameed <firas@mellanox.com>; Tziporet Koren <tziporet@mellanox.com>; Daniel Jurgens <danielj@mellanox.com>
> Subject: Re: [PATCH] PCI: hv: use effective affinity mask
>
> On Wed, Nov 01, 2017 at 08:30:53PM +0000, Dexuan Cui wrote:
> >
> > The effective_affinity_mask is always set when an interrupt is
> > assigned in
> > __assign_irq_vector() -> apic->cpu_mask_to_apicid(), e.g. for struct
> > apic
> > apic_physflat: -> default_cpu_mask_to_apicid() ->
> > irq_data_update_effective_affinity(), but it looks d->common->affinity
> > remains all-1's before the user space or the kernel changes it later.
> >
> > In the early allocation/initialization phase of an irq, we should use
> > the effective_affinity_mask, otherwise Hyper-V may not deliver the
> > interrupt to the expected cpu. Without the patch, if we assign 7
> > Mellanox ConnectX-3 VFs to a 32-vCPU VM, one of the VFs may fail to receive interrupts.
> >
> > Signed-off-by: Dexuan Cui <decui@microsoft.com>
> > Cc: Jake Oshins <jakeo@microsoft.com>
> > Cc: Jork Loeser <jloeser@microsoft.com>
> > Cc: Stephen Hemminger <sthemmin@microsoft.com>
> > Cc: K. Y. Srinivasan <kys@microsoft.com>
> > ---
> >
> > Please consider this for v4.14, if it's not too late.
>
> What would be the rationale for putting it in v4.14? After the merge window, I usually only merge fixes for problems introduced during the merge window, or for serious regressions. I can't tell if this fits into that or not.
>
> > drivers/pci/host/pci-hyperv.c | 8 +++++---
> > 1 file changed, 5 insertions(+), 3 deletions(-)
> >
> > diff --git a/drivers/pci/host/pci-hyperv.c
> > b/drivers/pci/host/pci-hyperv.c index 5ccb47d..8b5f66d 100644
> > --- a/drivers/pci/host/pci-hyperv.c
> > +++ b/drivers/pci/host/pci-hyperv.c
> > @@ -879,7 +879,7 @@ static void hv_irq_unmask(struct irq_data *data)
> > int cpu;
> > u64 res;
> >
> > - dest = irq_data_get_affinity_mask(data);
> > + dest = irq_data_get_effective_affinity_mask(data);
> > pdev = msi_desc_to_pci_dev(msi_desc);
> > pbus = pdev->bus;
> > hbus = container_of(pbus->sysdata, struct hv_pcibus_device,
> > sysdata); @@ -1042,6 +1042,7 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
> > struct hv_pci_dev *hpdev;
> > struct pci_bus *pbus;
> > struct pci_dev *pdev;
> > + struct cpumask *dest;
> > struct compose_comp_ctxt comp;
> > struct tran_int_desc *int_desc;
> > struct {
> > @@ -1056,6 +1057,7 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
> > int ret;
> >
> > pdev = msi_desc_to_pci_dev(irq_data_get_msi_desc(data));
> > + dest = irq_data_get_effective_affinity_mask(data);
> > pbus = pdev->bus;
> > hbus = container_of(pbus->sysdata, struct hv_pcibus_device, sysdata);
> > hpdev = get_pcichild_wslot(hbus, devfn_to_wslot(pdev->devfn)); @@
> > -1081,14 +1083,14 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
> > switch (pci_protocol_version) {
> > case PCI_PROTOCOL_VERSION_1_1:
> > size = hv_compose_msi_req_v1(&ctxt.int_pkts.v1,
> > - irq_data_get_affinity_mask(data),
> > + dest,
> > hpdev->desc.win_slot.slot,
> > cfg->vector);
> > break;
> >
> > case PCI_PROTOCOL_VERSION_1_2:
> > size = hv_compose_msi_req_v2(&ctxt.int_pkts.v2,
> > - irq_data_get_affinity_mask(data),
> > + dest,
> > hpdev->desc.win_slot.slot,
> > cfg->vector);
> > break;
> > --
> > 2.7.4
> >
^ permalink raw reply [flat|nested] 14+ messages in thread