[BUG] infiniband: sw: rdmavt: possible ABBA deadlocks in rvt_ruc_loopback()

[BUG] infiniband: sw: rdmavt: possible ABBA deadlocks in rvt_ruc_loopback()

[Jia-Ju Bai]

Hello,

My static analysis tool reports several possible ABBA deadlock in the 
rdmavt driver in Linux 5.18:

rvt_ruc_loopback()
   spin_lock_irqsave(&sqp->s_lock, flags); --> Line 3190 (Lock A)
   spin_lock(&sqp->r_lock); --> Line 3195 (Lock B)

rvt_qp_mr_clean()
   spin_lock_irq(&qp->r_lock); --> Line 698 (Lock B)
   spin_lock(&qp->s_lock); --> Line 700 (Lock A)

rvt_rc_timeout()
   spin_lock_irqsave(&qp->r_lock, flags); --> Line 2595 (Lock B)
   spin_lock(&qp->s_lock); --> Line 2596 (Lock A)

rvt_modify_qp()
   spin_lock_irq(&qp->r_lock); --> Line 1419 (Lock B)
   spin_lock(&qp->s_lock); --> Line 1421(Lock A)

_rvt_reset_qp()
   spin_lock_irq(&qp->r_lock); --> Line 907 (Lock B)
   spin_lock(&qp->s_lock); --> Line 909 (Lock A)

rvt_reset_qp()
   spin_lock_irq(&qp->r_lock); --> Line 936 (Lock B)
   spin_lock(&qp->s_lock); --> Line 938 (Lock A)

When rvt_ruc_loopback() is concurrently executed with rvt_qp_mr_clean(), 
rvt_rc_timeout(), rvt_modify_qp(), _rvt_reset_qp() or rvt_reset_qp(), 
the deadlocks can occur.

I am not quite sure whether these possible deadlocks are real and how to 
fix them if real.
Any feedback would be appreciated, thanks :)

Reported-by: TOTE Robot <oslab@tsinghua.edu.cn>


Best wishes,
Jia-Ju Bai

Re: [BUG] infiniband: sw: rdmavt: possible ABBA deadlocks in rvt_ruc_loopback()

[Dennis Dalessandro]

On 10/20/22 9:37 AM, Jia-Ju Bai wrote:
> Hello,
> 
> My static analysis tool reports several possible ABBA deadlock in the 
> rdmavt driver in Linux 5.18:
> 
> rvt_ruc_loopback()
>    spin_lock_irqsave(&sqp->s_lock, flags); --> Line 3190 (Lock A)
>    spin_lock(&sqp->r_lock); --> Line 3195 (Lock B)
> 
> rvt_qp_mr_clean()
>    spin_lock_irq(&qp->r_lock); --> Line 698 (Lock B)
>    spin_lock(&qp->s_lock); --> Line 700 (Lock A)
> 
> rvt_rc_timeout()
>    spin_lock_irqsave(&qp->r_lock, flags); --> Line 2595 (Lock B)
>    spin_lock(&qp->s_lock); --> Line 2596 (Lock A)
> 
> rvt_modify_qp()
>    spin_lock_irq(&qp->r_lock); --> Line 1419 (Lock B)
>    spin_lock(&qp->s_lock); --> Line 1421(Lock A)
> 
> _rvt_reset_qp()
>    spin_lock_irq(&qp->r_lock); --> Line 907 (Lock B)
>    spin_lock(&qp->s_lock); --> Line 909 (Lock A)
> 
> rvt_reset_qp()
>    spin_lock_irq(&qp->r_lock); --> Line 936 (Lock B)
>    spin_lock(&qp->s_lock); --> Line 938 (Lock A)
> 
> When rvt_ruc_loopback() is concurrently executed with rvt_qp_mr_clean(), 
> rvt_rc_timeout(), rvt_modify_qp(), _rvt_reset_qp() or rvt_reset_qp(), 
> the deadlocks can occur.
> 
> I am not quite sure whether these possible deadlocks are real and how to 
> fix them if real.
> Any feedback would be appreciated, thanks :)
> 
> Reported-by: TOTE Robot <oslab@tsinghua.edu.cn>

Thanks, we'll look into them.

-Denny

Re: [BUG] infiniband: sw: rdmavt: possible ABBA deadlocks in rvt_ruc_loopback()

[Niels Dossche]

On 20/10/2022 15:37, Jia-Ju Bai wrote:
> Hello,
> 
> My static analysis tool reports several possible ABBA deadlock in the rdmavt driver in Linux 5.18:
> 
> rvt_ruc_loopback()
>   spin_lock_irqsave(&sqp->s_lock, flags); --> Line 3190 (Lock A)
>   spin_lock(&sqp->r_lock); --> Line 3195 (Lock B)
> 
> rvt_qp_mr_clean()
>   spin_lock_irq(&qp->r_lock); --> Line 698 (Lock B)
>   spin_lock(&qp->s_lock); --> Line 700 (Lock A)
> 
> rvt_rc_timeout()
>   spin_lock_irqsave(&qp->r_lock, flags); --> Line 2595 (Lock B)
>   spin_lock(&qp->s_lock); --> Line 2596 (Lock A)
> 
> rvt_modify_qp()
>   spin_lock_irq(&qp->r_lock); --> Line 1419 (Lock B)
>   spin_lock(&qp->s_lock); --> Line 1421(Lock A)
> 
> _rvt_reset_qp()
>   spin_lock_irq(&qp->r_lock); --> Line 907 (Lock B)
>   spin_lock(&qp->s_lock); --> Line 909 (Lock A)
> 
> rvt_reset_qp()
>   spin_lock_irq(&qp->r_lock); --> Line 936 (Lock B)
>   spin_lock(&qp->s_lock); --> Line 938 (Lock A)
> 
> When rvt_ruc_loopback() is concurrently executed with rvt_qp_mr_clean(), rvt_rc_timeout(), rvt_modify_qp(), _rvt_reset_qp() or rvt_reset_qp(), the deadlocks can occur.
> 
> I am not quite sure whether these possible deadlocks are real and how to fix them if real.
> Any feedback would be appreciated, thanks :)
> 
> Reported-by: TOTE Robot <oslab@tsinghua.edu.cn>
> 
> 
> Best wishes,
> Jia-Ju Bai


Hi


I don't know whether this deadlock is real or not, but I wrote the offending code.
If it's a real bug, I'm sorry about that.
If that's the case the bug is also present in 2 other places of rvt_ruc_loopback() in v5.19+:

1)
spin_lock_irqsave(&sqp->s_lock, flags); --> Line 3133
spin_lock(&sqp->r_lock); --> Line 3137

2)
spin_lock_irqsave(&sqp->s_lock, flags); --> Line 3192
spin_lock(&sqp->r_lock); --> Line 3193


Kind regards
Niels