[PATCH] irq/irq_sim: add locking

[PATCH] irq/irq_sim: add locking

[Bartosz Golaszewski]

Two threads can try to fire the irq_sim with different offsets and will
end up fighting for the irq_work asignment. To fix it: add a mutex and
lock it before firing.

Suggested-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de>
Signed-off-by: Bartosz Golaszewski <brgl@bgdev.pl>
---
 include/linux/irq_sim.h | 1 +
 kernel/irq/irq_sim.c    | 5 +++++
 2 files changed, 6 insertions(+)

diff --git a/include/linux/irq_sim.h b/include/linux/irq_sim.h
index 630a57e55db6..676bfa0c12b9 100644
--- a/include/linux/irq_sim.h
+++ b/include/linux/irq_sim.h
@@ -29,6 +29,7 @@ struct irq_sim {
 	int			irq_base;
 	unsigned int		irq_count;
 	struct irq_sim_irq_ctx	*irqs;
+	struct mutex		lock;
 };
 
 int irq_sim_init(struct irq_sim *sim, unsigned int num_irqs);
diff --git a/kernel/irq/irq_sim.c b/kernel/irq/irq_sim.c
index dd20d0d528d4..2f06c24b51a0 100644
--- a/kernel/irq/irq_sim.c
+++ b/kernel/irq/irq_sim.c
@@ -74,6 +74,7 @@ int irq_sim_init(struct irq_sim *sim, unsigned int num_irqs)
 	}
 
 	init_irq_work(&sim->work_ctx.work, irq_sim_handle_irq);
+	mutex_init(&sim->lock);
 	sim->irq_count = num_irqs;
 
 	return sim->irq_base;
@@ -142,10 +143,14 @@ EXPORT_SYMBOL_GPL(devm_irq_sim_init);
  */
 void irq_sim_fire(struct irq_sim *sim, unsigned int offset)
 {
+	mutex_lock(&sim->lock);
+
 	if (sim->irqs[offset].enabled) {
 		sim->work_ctx.irq = irq_sim_irqnum(sim, offset);
 		irq_work_queue(&sim->work_ctx.work);
 	}
+
+	mutex_unlock(&sim->lock);
 }
 EXPORT_SYMBOL_GPL(irq_sim_fire);
 
-- 
2.19.1

Re: [PATCH] irq/irq_sim: add locking

[Uwe Kleine-König]

Hello Bartosz,

On Thu, Nov 08, 2018 at 05:47:48PM +0100, Bartosz Golaszewski wrote:
> Two threads can try to fire the irq_sim with different offsets and will
> end up fighting for the irq_work asignment. To fix it: add a mutex and
> lock it before firing.
> 
> Suggested-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de>
> Signed-off-by: Bartosz Golaszewski <brgl@bgdev.pl>
> ---
>  include/linux/irq_sim.h | 1 +
>  kernel/irq/irq_sim.c    | 5 +++++
>  2 files changed, 6 insertions(+)
> 
> diff --git a/include/linux/irq_sim.h b/include/linux/irq_sim.h
> index 630a57e55db6..676bfa0c12b9 100644
> --- a/include/linux/irq_sim.h
> +++ b/include/linux/irq_sim.h
> @@ -29,6 +29,7 @@ struct irq_sim {
>  	int			irq_base;
>  	unsigned int		irq_count;
>  	struct irq_sim_irq_ctx	*irqs;
> +	struct mutex		lock;
>  };
>  
>  int irq_sim_init(struct irq_sim *sim, unsigned int num_irqs);
> diff --git a/kernel/irq/irq_sim.c b/kernel/irq/irq_sim.c
> index dd20d0d528d4..2f06c24b51a0 100644
> --- a/kernel/irq/irq_sim.c
> +++ b/kernel/irq/irq_sim.c
> @@ -74,6 +74,7 @@ int irq_sim_init(struct irq_sim *sim, unsigned int num_irqs)
>  	}
>  
>  	init_irq_work(&sim->work_ctx.work, irq_sim_handle_irq);
> +	mutex_init(&sim->lock);
>  	sim->irq_count = num_irqs;
>  
>  	return sim->irq_base;
> @@ -142,10 +143,14 @@ EXPORT_SYMBOL_GPL(devm_irq_sim_init);
>   */
>  void irq_sim_fire(struct irq_sim *sim, unsigned int offset)
>  {
> +	mutex_lock(&sim->lock);
> +
>  	if (sim->irqs[offset].enabled) {
>  		sim->work_ctx.irq = irq_sim_irqnum(sim, offset);
>  		irq_work_queue(&sim->work_ctx.work);
>  	}
> +
> +	mutex_unlock(&sim->lock);

This doesn't fix the issue I think. irq_work_queue() only schedules the
work function. If after irq_sim_fire() returned but before the worker
runs another irq_sim_fire() is issued the value is still overwritten.

Best regards
Uwe

-- 
Pengutronix e.K.                           | Uwe Kleine-König            |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |

Re: [PATCH] irq/irq_sim: add locking

[Bartosz Golaszewski]

czw., 8 lis 2018 o 20:41 Uwe Kleine-König
<u.kleine-koenig@pengutronix.de> napisał(a):
>
> Hello Bartosz,
>
> On Thu, Nov 08, 2018 at 05:47:48PM +0100, Bartosz Golaszewski wrote:
> > Two threads can try to fire the irq_sim with different offsets and will
> > end up fighting for the irq_work asignment. To fix it: add a mutex and
> > lock it before firing.
> >
> > Suggested-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de>
> > Signed-off-by: Bartosz Golaszewski <brgl@bgdev.pl>
> > ---
> >  include/linux/irq_sim.h | 1 +
> >  kernel/irq/irq_sim.c    | 5 +++++
> >  2 files changed, 6 insertions(+)
> >
> > diff --git a/include/linux/irq_sim.h b/include/linux/irq_sim.h
> > index 630a57e55db6..676bfa0c12b9 100644
> > --- a/include/linux/irq_sim.h
> > +++ b/include/linux/irq_sim.h
> > @@ -29,6 +29,7 @@ struct irq_sim {
> >       int                     irq_base;
> >       unsigned int            irq_count;
> >       struct irq_sim_irq_ctx  *irqs;
> > +     struct mutex            lock;
> >  };
> >
> >  int irq_sim_init(struct irq_sim *sim, unsigned int num_irqs);
> > diff --git a/kernel/irq/irq_sim.c b/kernel/irq/irq_sim.c
> > index dd20d0d528d4..2f06c24b51a0 100644
> > --- a/kernel/irq/irq_sim.c
> > +++ b/kernel/irq/irq_sim.c
> > @@ -74,6 +74,7 @@ int irq_sim_init(struct irq_sim *sim, unsigned int num_irqs)
> >       }
> >
> >       init_irq_work(&sim->work_ctx.work, irq_sim_handle_irq);
> > +     mutex_init(&sim->lock);
> >       sim->irq_count = num_irqs;
> >
> >       return sim->irq_base;
> > @@ -142,10 +143,14 @@ EXPORT_SYMBOL_GPL(devm_irq_sim_init);
> >   */
> >  void irq_sim_fire(struct irq_sim *sim, unsigned int offset)
> >  {
> > +     mutex_lock(&sim->lock);
> > +
> >       if (sim->irqs[offset].enabled) {
> >               sim->work_ctx.irq = irq_sim_irqnum(sim, offset);
> >               irq_work_queue(&sim->work_ctx.work);
> >       }
> > +
> > +     mutex_unlock(&sim->lock);
>
> This doesn't fix the issue I think. irq_work_queue() only schedules the
> work function. If after irq_sim_fire() returned but before the worker
> runs another irq_sim_fire() is issued the value is still overwritten.
>

Looking at irq_work_queue(): while there may be some arch-specific
details deeper down the stack, it seems that unless the work is
IRQ_WORK_LAZY, the handler should be executed immediately. I'll verify
tomorrow though.

Bart

Re: [PATCH] irq/irq_sim: add locking

[Uwe Kleine-König]

Hello,

On Thu, Nov 08, 2018 at 09:55:02PM +0100, Bartosz Golaszewski wrote:
> czw., 8 lis 2018 o 20:41 Uwe Kleine-König
> <u.kleine-koenig@pengutronix.de> napisał(a):
> > > @@ -142,10 +143,14 @@ EXPORT_SYMBOL_GPL(devm_irq_sim_init);
> > >   */
> > >  void irq_sim_fire(struct irq_sim *sim, unsigned int offset)
> > >  {
> > > +     mutex_lock(&sim->lock);
> > > +
> > >       if (sim->irqs[offset].enabled) {
> > >               sim->work_ctx.irq = irq_sim_irqnum(sim, offset);
> > >               irq_work_queue(&sim->work_ctx.work);
> > >       }
> > > +
> > > +     mutex_unlock(&sim->lock);
> >
> > This doesn't fix the issue I think. irq_work_queue() only schedules the
> > work function. If after irq_sim_fire() returned but before the worker
> > runs another irq_sim_fire() is issued the value is still overwritten.
> 
> Looking at irq_work_queue(): while there may be some arch-specific
> details deeper down the stack, it seems that unless the work is
> IRQ_WORK_LAZY, the handler should be executed immediately. I'll verify
> tomorrow though.

not considering the IRQ_WORK_LAZY case irq_work_queue adds the work
struct to a list and then calls arch_irq_work_raise(). The default
implementation for this function is empty. alpha, arm, arm64, powerpc,
sparc and x86 have alternative implementations. Quickly looking at the
arm one: It is only used on SMP. Also given that all relevant code of
irq_work_queue is protected by preempt_disable/preempt_enable this
cannot atomically call the work function, otherwise it would run with
preemption disabled which isn't the case AFAIK.

Best regards
Uwe

-- 
Pengutronix e.K.                           | Uwe Kleine-König            |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |

Re: [PATCH] irq/irq_sim: add locking

[Thomas Gleixner]

[-- Attachment #1: Type: text/plain, Size: 1494 bytes --]

On Thu, 8 Nov 2018, Uwe Kleine-König wrote:
> On Thu, Nov 08, 2018 at 05:47:48PM +0100, Bartosz Golaszewski wrote:
> > @@ -74,6 +74,7 @@ int irq_sim_init(struct irq_sim *sim, unsigned int num_irqs)
> >  	}
> >  
> >  	init_irq_work(&sim->work_ctx.work, irq_sim_handle_irq);
> > +	mutex_init(&sim->lock);
> >  	sim->irq_count = num_irqs;
> >  
> >  	return sim->irq_base;
> > @@ -142,10 +143,14 @@ EXPORT_SYMBOL_GPL(devm_irq_sim_init);
> >   */
> >  void irq_sim_fire(struct irq_sim *sim, unsigned int offset)
> >  {
> > +	mutex_lock(&sim->lock);
> > +
> >  	if (sim->irqs[offset].enabled) {
> >  		sim->work_ctx.irq = irq_sim_irqnum(sim, offset);
> >  		irq_work_queue(&sim->work_ctx.work);
> >  	}
> > +
> > +	mutex_unlock(&sim->lock);
> 
> This doesn't fix the issue I think. irq_work_queue() only schedules the
> work function. If after irq_sim_fire() returned but before the worker
> runs another irq_sim_fire() is issued the value is still overwritten.

Right. So the obvious solution is to avoid the irq number store and use a
bitfield instead.

struct irq_sim_work_ctx {
       ...
       unsigned long	pending;
};

fire(sim, offset)
{
	if (!sim->irqs[offset].enabled)
		return;

	set_bit(offset, &sim->work_ctx.pending);
	....

and in the work handler do:

handle(work)
{
	struct irq_sim_work_ctx *ctx = container_of(work,....);

	while (ctx->pending) {
		offs = ffs(ctx->pending);
		clr_bit(offs, &ctx->pending);
		handle_simple_irq(offs);
	}
}

Or something like that.

Thanks,

	tglx

Re: [PATCH] irq/irq_sim: add locking

[Bartosz Golaszewski]

pt., 9 lis 2018 o 11:19 Thomas Gleixner <tglx@linutronix.de> napisał(a):
>
> On Thu, 8 Nov 2018, Uwe Kleine-König wrote:
> > On Thu, Nov 08, 2018 at 05:47:48PM +0100, Bartosz Golaszewski wrote:
> > > @@ -74,6 +74,7 @@ int irq_sim_init(struct irq_sim *sim, unsigned int num_irqs)
> > >     }
> > >
> > >     init_irq_work(&sim->work_ctx.work, irq_sim_handle_irq);
> > > +   mutex_init(&sim->lock);
> > >     sim->irq_count = num_irqs;
> > >
> > >     return sim->irq_base;
> > > @@ -142,10 +143,14 @@ EXPORT_SYMBOL_GPL(devm_irq_sim_init);
> > >   */
> > >  void irq_sim_fire(struct irq_sim *sim, unsigned int offset)
> > >  {
> > > +   mutex_lock(&sim->lock);
> > > +
> > >     if (sim->irqs[offset].enabled) {
> > >             sim->work_ctx.irq = irq_sim_irqnum(sim, offset);
> > >             irq_work_queue(&sim->work_ctx.work);
> > >     }
> > > +
> > > +   mutex_unlock(&sim->lock);
> >
> > This doesn't fix the issue I think. irq_work_queue() only schedules the
> > work function. If after irq_sim_fire() returned but before the worker
> > runs another irq_sim_fire() is issued the value is still overwritten.
>
> Right. So the obvious solution is to avoid the irq number store and use a
> bitfield instead.
>
> struct irq_sim_work_ctx {
>        ...
>        unsigned long    pending;
> };
>
> fire(sim, offset)
> {
>         if (!sim->irqs[offset].enabled)
>                 return;
>
>         set_bit(offset, &sim->work_ctx.pending);
>         ....
>
> and in the work handler do:
>
> handle(work)
> {
>         struct irq_sim_work_ctx *ctx = container_of(work,....);
>
>         while (ctx->pending) {
>                 offs = ffs(ctx->pending);
>                 clr_bit(offs, &ctx->pending);
>                 handle_simple_irq(offs);
>         }
> }
>
> Or something like that.
>
> Thanks,
>
>         tglx

Actually on ARM and x86 with smp the handler is called between the
mutex lock and unlock alright but this solution looks much better.
I'll resend a v2.

Thanks!
Bartosz