Netdev Archive on lore.kernel.org
 help / color / Atom feed
* [PATCH tip:irq/core v1] genirq: remove auto-set of the mask when setting the hint
@ 2021-05-01  2:18 Jesse Brandeburg
  2021-05-04 12:15 ` Robin Murphy
  0 siblings, 1 reply; 4+ messages in thread
From: Jesse Brandeburg @ 2021-05-01  2:18 UTC (permalink / raw)
  To: Thomas Gleixner
  Cc: Ingo Molnar, linux-kernel, intel-wired-lan, jbrandeb, frederic,
	juri.lelli, Marcelo Tosatti, abelits, Robin Murphy, linux-api,
	bhelgaas, linux-pci, rostedt, peterz, davem, akpm, sfr, stephen,
	rppt, jinyuqi, zhangshaokun, netdev, chris.friesen,
	Jesse Brandeburg, Nitesh Lal

It was pointed out by Nitesh that the original work I did in 2014
to automatically set the interrupt affinity when requesting a
mask is no longer necessary. The kernel has moved on and no
longer has the original problem, BUT the original patch
introduced a subtle bug when booting a system with reserved or
excluded CPUs. Drivers calling this function with a mask value
that included a CPU that was currently or in the future
unavailable would generally not update the hint.

I'm sure there are a million ways to solve this, but the simplest
one is to just remove a little code that tries to force the
affinity, as Nitesh has shown it fixes the bug and doesn't seem
to introduce immediate side effects.

While I'm here, introduce a kernel-doc for the hint function.

Ref: https://lore.kernel.org/lkml/CAFki+L=_dd+JgAR12_eBPX0kZO2_6=1dGdgkwHE=u=K6chMeLQ@mail.gmail.com/
Cc: netdev@vger.kernel.org
Fixes: 4fe7ffb7e17c ("genirq: Fix null pointer reference in irq_set_affinity_hint()")
Fixes: e2e64a932556 ("genirq: Set initial affinity in irq_set_affinity_hint()")
Reported-by: Nitesh Lal <nilal@redhat.com>
Signed-off-by: Jesse Brandeburg <jesse.brandeburg@intel.com>
---

!!! NOTE: Compile tested only, would appreciate feedback

---
 kernel/irq/manage.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/kernel/irq/manage.c b/kernel/irq/manage.c
index e976c4927b25..a31df64662d5 100644
--- a/kernel/irq/manage.c
+++ b/kernel/irq/manage.c
@@ -456,6 +456,16 @@ int __irq_set_affinity(unsigned int irq, const struct cpumask *mask, bool force)
 	return ret;
 }
 
+/**
+ * 	irq_set_affinity_hint - set the hint for an irq
+ *	@irq:	Interrupt for which to set the hint
+ *	@m:	Mask to indicate which CPUs to suggest for the interrupt, use
+ *		NULL here to indicate to clear the value.
+ *
+ *	Use this function to recommend which CPU should handle the
+ *	interrupt to any userspace that uses /proc/irq/nn/smp_affinity_hint
+ *	in order to align interrupts. Pass NULL as the mask to clear the hint.
+ */
 int irq_set_affinity_hint(unsigned int irq, const struct cpumask *m)
 {
 	unsigned long flags;
@@ -465,9 +475,6 @@ int irq_set_affinity_hint(unsigned int irq, const struct cpumask *m)
 		return -EINVAL;
 	desc->affinity_hint = m;
 	irq_put_desc_unlock(desc, flags);
-	/* set the initial affinity to prevent every interrupt being on CPU0 */
-	if (m)
-		__irq_set_affinity(irq, m, false);
 	return 0;
 }
 EXPORT_SYMBOL_GPL(irq_set_affinity_hint);

base-commit: 765822e1569a37aab5e69736c52d4ad4a289eba6
-- 
2.30.2


^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH tip:irq/core v1] genirq: remove auto-set of the mask when setting the hint
  2021-05-01  2:18 [PATCH tip:irq/core v1] genirq: remove auto-set of the mask when setting the hint Jesse Brandeburg
@ 2021-05-04 12:15 ` Robin Murphy
  2021-05-04 14:29   ` Nitesh Lal
  2021-05-04 16:23   ` Jesse Brandeburg
  0 siblings, 2 replies; 4+ messages in thread
From: Robin Murphy @ 2021-05-04 12:15 UTC (permalink / raw)
  To: Jesse Brandeburg, Thomas Gleixner
  Cc: Ingo Molnar, linux-kernel, intel-wired-lan, jbrandeb, frederic,
	juri.lelli, Marcelo Tosatti, abelits, linux-api, bhelgaas,
	linux-pci, rostedt, peterz, davem, akpm, sfr, stephen, rppt,
	jinyuqi, zhangshaokun, netdev, chris.friesen, Nitesh Lal,
	Marc Zyngier

On 2021-05-01 03:18, Jesse Brandeburg wrote:
> It was pointed out by Nitesh that the original work I did in 2014
> to automatically set the interrupt affinity when requesting a
> mask is no longer necessary. The kernel has moved on and no
> longer has the original problem, BUT the original patch
> introduced a subtle bug when booting a system with reserved or
> excluded CPUs. Drivers calling this function with a mask value
> that included a CPU that was currently or in the future
> unavailable would generally not update the hint.
> 
> I'm sure there are a million ways to solve this, but the simplest
> one is to just remove a little code that tries to force the
> affinity, as Nitesh has shown it fixes the bug and doesn't seem
> to introduce immediate side effects.

Unfortunately, I think there are quite a few other drivers now relying 
on this behaviour, since they are really using irq_set_affinity_hint() 
as a proxy for irq_set_affinity(). Partly since the latter isn't 
exported to modules, but also I have a vague memory of it being said 
that it's nice to update the user-visible hint to match when the 
affinity does have to be forced to something specific.

Robin.

> While I'm here, introduce a kernel-doc for the hint function.
> 
> Ref: https://lore.kernel.org/lkml/CAFki+L=_dd+JgAR12_eBPX0kZO2_6=1dGdgkwHE=u=K6chMeLQ@mail.gmail.com/
> Cc: netdev@vger.kernel.org
> Fixes: 4fe7ffb7e17c ("genirq: Fix null pointer reference in irq_set_affinity_hint()")
> Fixes: e2e64a932556 ("genirq: Set initial affinity in irq_set_affinity_hint()")
> Reported-by: Nitesh Lal <nilal@redhat.com>
> Signed-off-by: Jesse Brandeburg <jesse.brandeburg@intel.com>
> ---
> 
> !!! NOTE: Compile tested only, would appreciate feedback
> 
> ---
>   kernel/irq/manage.c | 13 ++++++++++---
>   1 file changed, 10 insertions(+), 3 deletions(-)
> 
> diff --git a/kernel/irq/manage.c b/kernel/irq/manage.c
> index e976c4927b25..a31df64662d5 100644
> --- a/kernel/irq/manage.c
> +++ b/kernel/irq/manage.c
> @@ -456,6 +456,16 @@ int __irq_set_affinity(unsigned int irq, const struct cpumask *mask, bool force)
>   	return ret;
>   }
>   
> +/**
> + * 	irq_set_affinity_hint - set the hint for an irq
> + *	@irq:	Interrupt for which to set the hint
> + *	@m:	Mask to indicate which CPUs to suggest for the interrupt, use
> + *		NULL here to indicate to clear the value.
> + *
> + *	Use this function to recommend which CPU should handle the
> + *	interrupt to any userspace that uses /proc/irq/nn/smp_affinity_hint
> + *	in order to align interrupts. Pass NULL as the mask to clear the hint.
> + */
>   int irq_set_affinity_hint(unsigned int irq, const struct cpumask *m)
>   {
>   	unsigned long flags;
> @@ -465,9 +475,6 @@ int irq_set_affinity_hint(unsigned int irq, const struct cpumask *m)
>   		return -EINVAL;
>   	desc->affinity_hint = m;
>   	irq_put_desc_unlock(desc, flags);
> -	/* set the initial affinity to prevent every interrupt being on CPU0 */
> -	if (m)
> -		__irq_set_affinity(irq, m, false);
>   	return 0;
>   }
>   EXPORT_SYMBOL_GPL(irq_set_affinity_hint);
> 
> base-commit: 765822e1569a37aab5e69736c52d4ad4a289eba6
> 

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH tip:irq/core v1] genirq: remove auto-set of the mask when setting the hint
  2021-05-04 12:15 ` Robin Murphy
@ 2021-05-04 14:29   ` Nitesh Lal
  2021-05-04 16:23   ` Jesse Brandeburg
  1 sibling, 0 replies; 4+ messages in thread
From: Nitesh Lal @ 2021-05-04 14:29 UTC (permalink / raw)
  To: Robin Murphy
  Cc: Jesse Brandeburg, Thomas Gleixner, Ingo Molnar, linux-kernel,
	intel-wired-lan, jbrandeb, frederic, juri.lelli, Marcelo Tosatti,
	abelits, linux-api, bhelgaas, linux-pci, rostedt, peterz, davem,
	akpm, sfr, stephen, rppt, jinyuqi, zhangshaokun, netdev,
	chris.friesen, Marc Zyngier

On Tue, May 4, 2021 at 8:15 AM Robin Murphy <robin.murphy@arm.com> wrote:
>
> On 2021-05-01 03:18, Jesse Brandeburg wrote:
> > It was pointed out by Nitesh that the original work I did in 2014
> > to automatically set the interrupt affinity when requesting a
> > mask is no longer necessary. The kernel has moved on and no
> > longer has the original problem, BUT the original patch
> > introduced a subtle bug when booting a system with reserved or
> > excluded CPUs. Drivers calling this function with a mask value
> > that included a CPU that was currently or in the future
> > unavailable would generally not update the hint.
> >
> > I'm sure there are a million ways to solve this, but the simplest
> > one is to just remove a little code that tries to force the
> > affinity, as Nitesh has shown it fixes the bug and doesn't seem
> > to introduce immediate side effects.
>
> Unfortunately, I think there are quite a few other drivers now relying
> on this behaviour, since they are really using irq_set_affinity_hint()
> as a proxy for irq_set_affinity().

That's true.

> Partly since the latter isn't
> exported to modules, but also I have a vague memory of it being said
> that it's nice to update the user-visible hint to match when the
> affinity does have to be forced to something specific.

If you see the downside of it we are forcing the affinity to match the hint
mask without considering the default SMP affinity mask.

Also, we are repeating things here. First, we set certain mask for a device
IRQ via request_irq code path which does consider the default SMP mask but
then we are letting the driver over-write it.

If we want to set the IRQ mask in a certain way then it should be done at
the time of initial setup itself.

Do you know about a workload/use case that can show the benefit of
this behavior? As then we can try fixing it in the right way.

--
Thanks
Nitesh


^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH tip:irq/core v1] genirq: remove auto-set of the mask when setting the hint
  2021-05-04 12:15 ` Robin Murphy
  2021-05-04 14:29   ` Nitesh Lal
@ 2021-05-04 16:23   ` Jesse Brandeburg
  1 sibling, 0 replies; 4+ messages in thread
From: Jesse Brandeburg @ 2021-05-04 16:23 UTC (permalink / raw)
  To: Robin Murphy
  Cc: Thomas Gleixner, Ingo Molnar, linux-kernel, intel-wired-lan,
	jbrandeb, frederic, juri.lelli, Marcelo Tosatti, abelits,
	linux-api, bhelgaas, linux-pci, rostedt, peterz, davem, akpm,
	sfr, stephen, rppt, jinyuqi, zhangshaokun, netdev, chris.friesen,
	Nitesh Lal, Marc Zyngier

Robin Murphy wrote:

> On 2021-05-01 03:18, Jesse Brandeburg wrote:
> > It was pointed out by Nitesh that the original work I did in 2014
> > to automatically set the interrupt affinity when requesting a
> > mask is no longer necessary. The kernel has moved on and no
> > longer has the original problem, BUT the original patch
> > introduced a subtle bug when booting a system with reserved or
> > excluded CPUs. Drivers calling this function with a mask value
> > that included a CPU that was currently or in the future
> > unavailable would generally not update the hint.
> > 
> > I'm sure there are a million ways to solve this, but the simplest
> > one is to just remove a little code that tries to force the
> > affinity, as Nitesh has shown it fixes the bug and doesn't seem
> > to introduce immediate side effects.
> 
> Unfortunately, I think there are quite a few other drivers now relying 
> on this behaviour, since they are really using irq_set_affinity_hint() 
> as a proxy for irq_set_affinity(). Partly since the latter isn't 
> exported to modules, but also I have a vague memory of it being said 
> that it's nice to update the user-visible hint to match when the 
> affinity does have to be forced to something specific.
> 
> Robin.

Thanks for your feedback Robin, but there is definitely a bug here that
is being exposed by this code. The fact that people are using this
function means they're all exposed to this bug.

Not sure if you saw, but this analysis from Nitesh explains what
happened chronologically to the kernel w.r.t this code, it's a useful
analysis! [1]

I'd add in addition that irqbalance daemon *stopped* paying attention
to hints quite a while ago, so I'm not quite sure what purpose they
serve.

[1]
https://lore.kernel.org/lkml/CAFki+Lm0W_brLu31epqD3gAV+WNKOJfVDfX2M8ZM__aj3nv9uA@mail.gmail.com/


^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, back to index

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-05-01  2:18 [PATCH tip:irq/core v1] genirq: remove auto-set of the mask when setting the hint Jesse Brandeburg
2021-05-04 12:15 ` Robin Murphy
2021-05-04 14:29   ` Nitesh Lal
2021-05-04 16:23   ` Jesse Brandeburg

Netdev Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/netdev/0 netdev/git/0.git
	git clone --mirror https://lore.kernel.org/netdev/1 netdev/git/1.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 netdev netdev/ https://lore.kernel.org/netdev \
		netdev@vger.kernel.org
	public-inbox-index netdev

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.netdev


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git