[PATCH v2 0/2] PCI: hv: Refactor hv_irq_unmask() to use hv_vpset and cpumask_to_vpset()

[PATCH v2 0/2] PCI: hv: Refactor hv_irq_unmask() to use hv_vpset and cpumask_to_vpset()

[Maya Nakamura]

This patchset removes a duplicate definition of VP set (hv_vp_set) and
uses the common definition (hv_vpset) that is used in other places. It
changes the order of the members in struct hv_pcibus_device due to
flexible array in hv_vpset.

It also removes the duplicate implementation of cpumask_to_vpset(), uses
the shared implementation, and exports hv_max_vp_index, which is
required by cpumask_to_vpset().

Maya Nakamura (2):
  PCI: hv: Replace hv_vp_set with hv_vpset
  PCI: hv: Refactor hv_irq_unmask() to use cpumask_to_vpset()

 arch/x86/hyperv/hv_init.c           |  1 +
 drivers/pci/controller/pci-hyperv.c | 59 +++++++++++++----------------
 2 files changed, 28 insertions(+), 32 deletions(-)

-- 
2.17.1

[PATCH v2 1/2] PCI: hv: Replace hv_vp_set with hv_vpset

[Maya Nakamura]

Remove a duplicate definition of VP set (hv_vp_set) and use the common
definition (hv_vpset) that is used in other places.

Change the order of the members in struct hv_pcibus_device so that the
declaration of retarget_msi_interrupt_params is the last member. Struct
hv_vpset, which contains a flexible array, is nested two levels deep in
struct hv_pcibus_device via retarget_msi_interrupt_params.

Add a comment that retarget_msi_interrupt_params should be the last member
of struct hv_pcibus_device.

Signed-off-by: Maya Nakamura <m.maya.nakamura@gmail.com>
---
Change in v2:
- None

 drivers/pci/controller/pci-hyperv.c | 25 ++++++++++++-------------
 1 file changed, 12 insertions(+), 13 deletions(-)

diff --git a/drivers/pci/controller/pci-hyperv.c b/drivers/pci/controller/pci-hyperv.c
index 9ba4d12c179c..da8b58d8630d 100644
--- a/drivers/pci/controller/pci-hyperv.c
+++ b/drivers/pci/controller/pci-hyperv.c
@@ -393,12 +393,6 @@ struct hv_interrupt_entry {
 
 #define HV_VP_SET_BANK_COUNT_MAX	5 /* current implementation limit */
 
-struct hv_vp_set {
-	u64	format;			/* 0 (HvGenericSetSparse4k) */
-	u64	valid_banks;
-	u64	masks[HV_VP_SET_BANK_COUNT_MAX];
-};
-
 /*
  * flags for hv_device_interrupt_target.flags
  */
@@ -410,7 +404,7 @@ struct hv_device_interrupt_target {
 	u32	flags;
 	union {
 		u64		 vp_mask;
-		struct hv_vp_set vp_set;
+		struct hv_vpset vp_set;
 	};
 };
 
@@ -460,12 +454,16 @@ struct hv_pcibus_device {
 	struct msi_controller msi_chip;
 	struct irq_domain *irq_domain;
 
-	/* hypercall arg, must not cross page boundary */
-	struct retarget_msi_interrupt retarget_msi_interrupt_params;
-
 	spinlock_t retarget_msi_interrupt_lock;
 
 	struct workqueue_struct *wq;
+
+	/* hypercall arg, must not cross page boundary */
+	struct retarget_msi_interrupt retarget_msi_interrupt_params;
+
+	/*
+	 * Don't put anything here: retarget_msi_interrupt_params must be last
+	 */
 };
 
 /*
@@ -955,12 +953,13 @@ static void hv_irq_unmask(struct irq_data *data)
 		 */
 		params->int_target.flags |=
 			HV_DEVICE_INTERRUPT_TARGET_PROCESSOR_SET;
-		params->int_target.vp_set.valid_banks =
+		params->int_target.vp_set.valid_bank_mask =
 			(1ull << HV_VP_SET_BANK_COUNT_MAX) - 1;
 
 		/*
 		 * var-sized hypercall, var-size starts after vp_mask (thus
-		 * vp_set.format does not count, but vp_set.valid_banks does).
+		 * vp_set.format does not count, but vp_set.valid_bank_mask
+		 * does).
 		 */
 		var_size = 1 + HV_VP_SET_BANK_COUNT_MAX;
 
@@ -974,7 +973,7 @@ static void hv_irq_unmask(struct irq_data *data)
 				goto exit_unlock;
 			}
 
-			params->int_target.vp_set.masks[cpu_vmbus / 64] |=
+			params->int_target.vp_set.bank_contents[cpu_vmbus / 64]	|=
 				(1ULL << (cpu_vmbus & 63));
 		}
 	} else {
-- 
2.17.1

Re: [PATCH v2 1/2] PCI: hv: Replace hv_vp_set with hv_vpset

[Maya Nakamura]

On Sun, Jan 27, 2019 at 05:11:48AM +0000, Michael Kelley wrote:
> From: Maya Nakamura <m.maya.nakamura@gmail.com>  Sent: Saturday, January 26, 2019 12:52 AM
> > 
> > Remove a duplicate definition of VP set (hv_vp_set) and use the common
> > definition (hv_vpset) that is used in other places.
> > 
> > Change the order of the members in struct hv_pcibus_device so that the
> > declaration of retarget_msi_interrupt_params is the last member. Struct
> > hv_vpset, which contains a flexible array, is nested two levels deep in
> > struct hv_pcibus_device via retarget_msi_interrupt_params.
> > 
> > Add a comment that retarget_msi_interrupt_params should be the last member
> > of struct hv_pcibus_device.
> > 
> > Signed-off-by: Maya Nakamura <m.maya.nakamura@gmail.com>
> > ---
> > Change in v2:
> > - None
> > 
> 
> Right -- there was no code change.  But it's customary to note that
> you updated the commit message.
> 
Thank you for your feedback. I will edit the change log in v3.

> Reviewed-by:  Michael Kelley <mikelley@microsoft.com>

Re: [PATCH v2 1/2] PCI: hv: Replace hv_vp_set with hv_vpset

[Lorenzo Pieralisi]

On Mon, Jan 28, 2019 at 09:49:32PM -0800, Maya Nakamura wrote:
> On Sun, Jan 27, 2019 at 05:11:48AM +0000, Michael Kelley wrote:
> > From: Maya Nakamura <m.maya.nakamura@gmail.com>  Sent: Saturday, January 26, 2019 12:52 AM
> > > 
> > > Remove a duplicate definition of VP set (hv_vp_set) and use the common
> > > definition (hv_vpset) that is used in other places.
> > > 
> > > Change the order of the members in struct hv_pcibus_device so that the
> > > declaration of retarget_msi_interrupt_params is the last member. Struct
> > > hv_vpset, which contains a flexible array, is nested two levels deep in
> > > struct hv_pcibus_device via retarget_msi_interrupt_params.
> > > 
> > > Add a comment that retarget_msi_interrupt_params should be the last member
> > > of struct hv_pcibus_device.
> > > 
> > > Signed-off-by: Maya Nakamura <m.maya.nakamura@gmail.com>
> > > ---
> > > Change in v2:
> > > - None
> > > 
> > 
> > Right -- there was no code change.  But it's customary to note that
> > you updated the commit message.
> > 
> Thank you for your feedback. I will edit the change log in v3.
> 
> > Reviewed-by:  Michael Kelley <mikelley@microsoft.com>

Are you really sure there is no behavioural change ? What piece of
code allocates hv_vpset.bank_contents[] memory with this patch applied ?

I suspect the current code does not use hv_vpset for this specific
reason, ie allocate struct hv_vp_set.masks array memory statically.

Lorenzo

Re: [PATCH v2 1/2] PCI: hv: Replace hv_vp_set with hv_vpset

[Lorenzo Pieralisi]

On Wed, Feb 13, 2019 at 02:20:29AM +0000, Michael Kelley wrote:
> From: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>  Sent: Tuesday, February 12, 2019 8:35 AM
> >
> > On Mon, Jan 28, 2019 at 09:49:32PM -0800, Maya Nakamura wrote:
> > > On Sun, Jan 27, 2019 at 05:11:48AM +0000, Michael Kelley wrote:
> > > > From: Maya Nakamura <m.maya.nakamura@gmail.com>  Sent: Saturday, January 26,
> > 2019 12:52 AM
> > > > >
> > > > > Remove a duplicate definition of VP set (hv_vp_set) and use the common
> > > > > definition (hv_vpset) that is used in other places.
> > > > >
> > > > > Change the order of the members in struct hv_pcibus_device so that the
> > > > > declaration of retarget_msi_interrupt_params is the last member. Struct
> > > > > hv_vpset, which contains a flexible array, is nested two levels deep in
> > > > > struct hv_pcibus_device via retarget_msi_interrupt_params.
> > > > >
> > > > > Add a comment that retarget_msi_interrupt_params should be the last member
> > > > > of struct hv_pcibus_device.
> > > > >
> > > > > Signed-off-by: Maya Nakamura <m.maya.nakamura@gmail.com>
> > > > > ---
> > > > > Change in v2:
> > > > > - None
> > > > >
> > > >
> > > > Right -- there was no code change.  But it's customary to note that
> > > > you updated the commit message.
> > > >
> > > Thank you for your feedback. I will edit the change log in v3.
> > >
> > > > Reviewed-by:  Michael Kelley <mikelley@microsoft.com>
> > 
> > Are you really sure there is no behavioural change ? What piece of
> > code allocates hv_vpset.bank_contents[] memory with this patch applied ?
> > 
> > I suspect the current code does not use hv_vpset for this specific
> > reason, ie allocate struct hv_vp_set.masks array memory statically.
> > 
> 
> There is indeed no behavior change.   A full page of memory is
> allocated in hv_pci_probe() so that we can be sure that the Hyper-V
> hypercall arguments don't cross a page boundary.   This page allows
> more than enough space for the hv_vpset.bank_contents[] to grow
> as needed (with one bit allocated in the masks for up to the limit
> of 8192 CPUs allowed by Linux).   A flexible array is used because
> the hv_vpset structure is also used in some MMU hypercalls that
> have two variable size arrays.

I see, thanks for explaining.

Lorenzo

Re: [PATCH v2 1/2] PCI: hv: Replace hv_vp_set with hv_vpset

[Lorenzo Pieralisi]

On Mon, Jan 28, 2019 at 09:49:32PM -0800, Maya Nakamura wrote:
> On Sun, Jan 27, 2019 at 05:11:48AM +0000, Michael Kelley wrote:
> > From: Maya Nakamura <m.maya.nakamura@gmail.com>  Sent: Saturday, January 26, 2019 12:52 AM
> > > 
> > > Remove a duplicate definition of VP set (hv_vp_set) and use the common
> > > definition (hv_vpset) that is used in other places.
> > > 
> > > Change the order of the members in struct hv_pcibus_device so that the
> > > declaration of retarget_msi_interrupt_params is the last member. Struct
> > > hv_vpset, which contains a flexible array, is nested two levels deep in
> > > struct hv_pcibus_device via retarget_msi_interrupt_params.
> > > 
> > > Add a comment that retarget_msi_interrupt_params should be the last member
> > > of struct hv_pcibus_device.
> > > 
> > > Signed-off-by: Maya Nakamura <m.maya.nakamura@gmail.com>
> > > ---
> > > Change in v2:
> > > - None
> > > 
> > 
> > Right -- there was no code change.  But it's customary to note that
> > you updated the commit message.
> > 
> Thank you for your feedback. I will edit the change log in v3.
> 
> > Reviewed-by:  Michael Kelley <mikelley@microsoft.com>

I will add Michael's tag to v3 (unless Michael is not happy with that),
it is missing there.

Lorenzo

[PATCH v2 2/2] PCI: hv: Refactor hv_irq_unmask() to use cpumask_to_vpset()

[Maya Nakamura]

Remove the duplicate implementation of cpumask_to_vpset() and use the
shared implementation. Export hv_max_vp_index, which is required by
cpumask_to_vpset().

Apply changes to hv_irq_unmask() based on feedback.

Signed-off-by: Maya Nakamura <m.maya.nakamura@gmail.com>
---
Changes in v2:
 - Remove unnecessary nr_bank initialization.
 - Delete two unnecessary dev_err()'s.
 - Unlock before returning.

 arch/x86/hyperv/hv_init.c           |  1 +
 drivers/pci/controller/pci-hyperv.c | 38 +++++++++++++----------------
 2 files changed, 18 insertions(+), 21 deletions(-)

diff --git a/arch/x86/hyperv/hv_init.c b/arch/x86/hyperv/hv_init.c
index 7abb09e2eeb8..7f2eed1fc81b 100644
--- a/arch/x86/hyperv/hv_init.c
+++ b/arch/x86/hyperv/hv_init.c
@@ -96,6 +96,7 @@ void  __percpu **hyperv_pcpu_input_arg;
 EXPORT_SYMBOL_GPL(hyperv_pcpu_input_arg);
 
 u32 hv_max_vp_index;
+EXPORT_SYMBOL_GPL(hv_max_vp_index);
 
 static int hv_cpu_init(unsigned int cpu)
 {
diff --git a/drivers/pci/controller/pci-hyperv.c b/drivers/pci/controller/pci-hyperv.c
index da8b58d8630d..d74c0f111fd0 100644
--- a/drivers/pci/controller/pci-hyperv.c
+++ b/drivers/pci/controller/pci-hyperv.c
@@ -391,8 +391,6 @@ struct hv_interrupt_entry {
 	u32	data;
 };
 
-#define HV_VP_SET_BANK_COUNT_MAX	5 /* current implementation limit */
-
 /*
  * flags for hv_device_interrupt_target.flags
  */
@@ -908,12 +906,12 @@ static void hv_irq_unmask(struct irq_data *data)
 	struct retarget_msi_interrupt *params;
 	struct hv_pcibus_device *hbus;
 	struct cpumask *dest;
+	cpumask_var_t tmp;
 	struct pci_bus *pbus;
 	struct pci_dev *pdev;
 	unsigned long flags;
 	u32 var_size = 0;
-	int cpu_vmbus;
-	int cpu;
+	int cpu, nr_bank;
 	u64 res;
 
 	dest = irq_data_get_effective_affinity_mask(data);
@@ -953,29 +951,27 @@ static void hv_irq_unmask(struct irq_data *data)
 		 */
 		params->int_target.flags |=
 			HV_DEVICE_INTERRUPT_TARGET_PROCESSOR_SET;
-		params->int_target.vp_set.valid_bank_mask =
-			(1ull << HV_VP_SET_BANK_COUNT_MAX) - 1;
+
+		if (!alloc_cpumask_var(&tmp, GFP_KERNEL)) {
+			res = 1;
+			goto exit_unlock;
+		}
+
+		cpumask_and(tmp, dest, cpu_online_mask);
+		nr_bank = cpumask_to_vpset(&params->int_target.vp_set, tmp);
+		free_cpumask_var(tmp);
+
+		if (!nr_bank) {
+			res = 1;
+			goto exit_unlock;
+		}
 
 		/*
 		 * var-sized hypercall, var-size starts after vp_mask (thus
 		 * vp_set.format does not count, but vp_set.valid_bank_mask
 		 * does).
 		 */
-		var_size = 1 + HV_VP_SET_BANK_COUNT_MAX;
-
-		for_each_cpu_and(cpu, dest, cpu_online_mask) {
-			cpu_vmbus = hv_cpu_number_to_vp_number(cpu);
-
-			if (cpu_vmbus >= HV_VP_SET_BANK_COUNT_MAX * 64) {
-				dev_err(&hbus->hdev->device,
-					"too high CPU %d", cpu_vmbus);
-				res = 1;
-				goto exit_unlock;
-			}
-
-			params->int_target.vp_set.bank_contents[cpu_vmbus / 64]	|=
-				(1ULL << (cpu_vmbus & 63));
-		}
+		var_size = 1 + nr_bank;
 	} else {
 		for_each_cpu_and(cpu, dest, cpu_online_mask) {
 			params->int_target.vp_mask |=
-- 
2.17.1

Re: [PATCH v2 2/2] PCI: hv: Refactor hv_irq_unmask() to use cpumask_to_vpset()

[Maya Nakamura]

On Sun, Jan 27, 2019 at 05:22:06AM +0000, Michael Kelley wrote:
> From: Maya Nakamura <m.maya.nakamura@gmail.com>  Sent: Saturday, January 26, 2019 12:55 AM
> >
> > @@ -953,29 +951,27 @@ static void hv_irq_unmask(struct irq_data *data)
> >  		 */
> >  		params->int_target.flags |=
> >  			HV_DEVICE_INTERRUPT_TARGET_PROCESSOR_SET;
> > -		params->int_target.vp_set.valid_bank_mask =
> > -			(1ull << HV_VP_SET_BANK_COUNT_MAX) - 1;
> > +
> > +		if (!alloc_cpumask_var(&tmp, GFP_KERNEL)) {
> > +			res = 1;
> > +			goto exit_unlock;
> > +		}
> > +
> > +		cpumask_and(tmp, dest, cpu_online_mask);
> > +		nr_bank = cpumask_to_vpset(&params->int_target.vp_set, tmp);
> > +		free_cpumask_var(tmp);
> > +
> > +		if (!nr_bank) {
> 
> There are two failures cases in cpumask_to_vpset().  One case returns
> 0, and the other case returns -1.  The above test only catches the 0
> failure case.  Need to modify the test to catch both cases.
> 
> Michael
> 
Thank you for your feedback. I will correct it in v3.

Maya

> > +			res = 1;
> > +			goto exit_unlock;
> > +		}
> > 
>