* Please add 2dce224f469f ("netns: protect netns ID lookups with RCU") to LTS
@ 2021-09-09 13:10 Haakon Bugge
2021-09-09 13:30 ` Greg KH
0 siblings, 1 reply; 5+ messages in thread
From: Haakon Bugge @ 2021-09-09 13:10 UTC (permalink / raw)
To: Greg KH, Sasha Levin
Cc: David S. Miller, Linux-Net, stable, Doug Ledford,
Jason Gunthorpe, OFED mailing list, LKML, Guillaume Nault
Hi Greg & Sasha,
tl;dr: Please add 2dce224f469f ("netns: protect netns ID lookups with RCU") to the stable releases from v5.4 and older. It fixes a spin_unlock_bh() in peernet2id() called with IRQs off. I think this neat side-effect of commit 2dce224f469f was quite un-intentional, hence no Fixes: tag or CC: stable.
The details:
From bugzilla.redhat.com/show_bug.cgi?id=1384179 (an ancient 4.9.0-0.rc0 kernel):
dump_stack+0x86/0xc3
__warn+0xcb/0xf0
warn_slowpath_null+0x1d/0x20
__local_bh_enable_ip+0x9d/0xc0
_raw_spin_unlock_bh+0x35/0x40
peernet2id+0x54/0x80
netlink_broadcast_filtered+0x220/0x3c0
netlink_broadcast+0x1d/0x20
audit_log+0x6a/0x90
security_set_bools+0xee/0x200
[]
Note, security_set_bools() calls write_lock_irq(). peernet2id() calls spin_unlock_bh().
From an internal (UEK) stack trace based on the v\x054.14.35 kernel (LTS 4.14.231):
queued_spin_lock_slowpath+0xb/0xf
_raw_spin_lock_irqsave+0x46/0x48
send_mad+0x3d2/0x590 [ib_core]
ib_sa_path_rec_get+0x223/0x4d0 [ib_core]
path_rec_start+0xa3/0x140 [ib_ipoib]
ipoib_start_xmit+0x2b0/0x6a0 [ib_ipoib]
dev_hard_start_xmit+0xb2/0x237
sch_direct_xmit+0x114/0x1bf
__dev_queue_xmit+0x592/0x818
dev_queue_xmit+0x10/0x12
arp_xmit+0x38/0xa6
arp_send_dst.part.16+0x61/0x84
arp_process+0x825/0x889
arp_rcv+0x140/0x1c9
__netif_receive_skb_core+0x401/0xb39
__netif_receive_skb+0x18/0x59
netif_receive_skb_internal+0x45/0x119
napi_gro_receive+0xd8/0xf6
ipoib_ib_handle_rx_wc+0x1ca/0x520 [ib_ipoib]
ipoib_poll+0xcd/0x150 [ib_ipoib]
net_rx_action+0x289/0x3f4
__do_softirq+0xe1/0x2b5
do_softirq_own_stack+0x2a/0x35
</IRQ>
do_softirq+0x4d/0x6a
__local_bh_enable_ip+0x57/0x59
_raw_spin_unlock_bh+0x23/0x25
peernet2id+0x51/0x73
netlink_broadcast_filtered+0x223/0x41b
netlink_broadcast+0x1d/0x1f
rdma_nl_multicast+0x22/0x30 [ib_core]
send_mad+0x3e5/0x590 [ib_core]
ib_sa_path_rec_get+0x223/0x4d0 [ib_core]
rdma_resolve_route+0x287/0x810 [rdma_cm]
rds_rdma_cm_event_handler_cmn+0x311/0x7d0 [rds_rdma]
rds_rdma_cm_event_handler_worker+0x22/0x30 [rds_rdma]
process_one_work+0x169/0x3a6
worker_thread+0x4d/0x3e5
kthread+0x105/0x138
ret_from_fork+0x24/0x49
Here, pay attention to ib_nl_make_request() which calls spin_lock_irqsave() on a global lock just before calling rdma_nl_multicast(). Thereafter, peernet2id() enables SoftIRQs, and ipoib starts and calls the same path and end up trying to acquire the same global lock again.
I have tried to repro this with no luck. But, stack traces seldom lies ;-)
Thxs, Håkon
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Please add 2dce224f469f ("netns: protect netns ID lookups with RCU") to LTS
2021-09-09 13:10 Please add 2dce224f469f ("netns: protect netns ID lookups with RCU") to LTS Haakon Bugge
@ 2021-09-09 13:30 ` Greg KH
2021-09-10 14:22 ` Haakon Bugge
0 siblings, 1 reply; 5+ messages in thread
From: Greg KH @ 2021-09-09 13:30 UTC (permalink / raw)
To: Haakon Bugge
Cc: Sasha Levin, David S. Miller, Linux-Net, stable, Doug Ledford,
Jason Gunthorpe, OFED mailing list, LKML, Guillaume Nault
On Thu, Sep 09, 2021 at 01:10:05PM +0000, Haakon Bugge wrote:
> Hi Greg & Sasha,
>
>
> tl;dr: Please add 2dce224f469f ("netns: protect netns ID lookups with
> RCU") to the stable releases from v5.4 and older. It fixes a
> spin_unlock_bh() in peernet2id() called with IRQs off. I think this
> neat side-effect of commit 2dce224f469f was quite un-intentional,
> hence no Fixes: tag or CC: stable.
Please provide a working backport for all of the relevant kernel
verisons, as it does not apply cleanly on it's own.
thanks,
greg k-h
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Please add 2dce224f469f ("netns: protect netns ID lookups with RCU") to LTS
2021-09-09 13:30 ` Greg KH
@ 2021-09-10 14:22 ` Haakon Bugge
2021-09-13 8:42 ` Greg KH
0 siblings, 1 reply; 5+ messages in thread
From: Haakon Bugge @ 2021-09-10 14:22 UTC (permalink / raw)
To: Greg KH
Cc: Sasha Levin, David S. Miller, Linux-Net, stable, Doug Ledford,
Jason Gunthorpe, OFED mailing list, LKML, Guillaume Nault
> On 9 Sep 2021, at 15:30, Greg KH <gregkh@linuxfoundation.org> wrote:
>
> On Thu, Sep 09, 2021 at 01:10:05PM +0000, Haakon Bugge wrote:
>> Hi Greg & Sasha,
>>
>>
>> tl;dr: Please add 2dce224f469f ("netns: protect netns ID lookups with
>> RCU") to the stable releases from v5.4 and older. It fixes a
>> spin_unlock_bh() in peernet2id() called with IRQs off. I think this
>> neat side-effect of commit 2dce224f469f was quite un-intentional,
>> hence no Fixes: tag or CC: stable.
>
> Please provide a working backport for all of the relevant kernel
> verisons, as it does not apply cleanly on it's own.
I've done the backports. 4.9 is actually not needed, because it uses spin_{lock,unlock}_irqsave() in peernet2id(). Hence, we have an "offending" commit which this one fixes:
fba143c66abb ("netns: avoid disabling irq for netns id")
Will get'm out during the weekend.
Thxs, Håkon
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Please add 2dce224f469f ("netns: protect netns ID lookups with RCU") to LTS
2021-09-10 14:22 ` Haakon Bugge
@ 2021-09-13 8:42 ` Greg KH
2021-09-14 8:58 ` Haakon Bugge
0 siblings, 1 reply; 5+ messages in thread
From: Greg KH @ 2021-09-13 8:42 UTC (permalink / raw)
To: Haakon Bugge
Cc: Sasha Levin, David S. Miller, Linux-Net, stable, Doug Ledford,
Jason Gunthorpe, OFED mailing list, LKML, Guillaume Nault
On Fri, Sep 10, 2021 at 02:22:32PM +0000, Haakon Bugge wrote:
>
>
> > On 9 Sep 2021, at 15:30, Greg KH <gregkh@linuxfoundation.org> wrote:
> >
> > On Thu, Sep 09, 2021 at 01:10:05PM +0000, Haakon Bugge wrote:
> >> Hi Greg & Sasha,
> >>
> >>
> >> tl;dr: Please add 2dce224f469f ("netns: protect netns ID lookups with
> >> RCU") to the stable releases from v5.4 and older. It fixes a
> >> spin_unlock_bh() in peernet2id() called with IRQs off. I think this
> >> neat side-effect of commit 2dce224f469f was quite un-intentional,
> >> hence no Fixes: tag or CC: stable.
> >
> > Please provide a working backport for all of the relevant kernel
> > verisons, as it does not apply cleanly on it's own.
>
> I've done the backports. 4.9 is actually not needed, because it uses spin_{lock,unlock}_irqsave() in peernet2id(). Hence, we have an "offending" commit which this one fixes:
>
> fba143c66abb ("netns: avoid disabling irq for netns id")
>
> Will get'm out during the weekend.
All now queued up, thanks.
greg k-h
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Please add 2dce224f469f ("netns: protect netns ID lookups with RCU") to LTS
2021-09-13 8:42 ` Greg KH
@ 2021-09-14 8:58 ` Haakon Bugge
0 siblings, 0 replies; 5+ messages in thread
From: Haakon Bugge @ 2021-09-14 8:58 UTC (permalink / raw)
To: Greg KH
Cc: Sasha Levin, David S. Miller, Linux-Net, stable, Doug Ledford,
Jason Gunthorpe, OFED mailing list, LKML, Guillaume Nault
> On 13 Sep 2021, at 10:42, Greg KH <gregkh@linuxfoundation.org> wrote:
>
> On Fri, Sep 10, 2021 at 02:22:32PM +0000, Haakon Bugge wrote:
>>
>>
>>> On 9 Sep 2021, at 15:30, Greg KH <gregkh@linuxfoundation.org> wrote:
>>>
>>> On Thu, Sep 09, 2021 at 01:10:05PM +0000, Haakon Bugge wrote:
>>>> Hi Greg & Sasha,
>>>>
>>>>
>>>> tl;dr: Please add 2dce224f469f ("netns: protect netns ID lookups with
>>>> RCU") to the stable releases from v5.4 and older. It fixes a
>>>> spin_unlock_bh() in peernet2id() called with IRQs off. I think this
>>>> neat side-effect of commit 2dce224f469f was quite un-intentional,
>>>> hence no Fixes: tag or CC: stable.
>>>
>>> Please provide a working backport for all of the relevant kernel
>>> verisons, as it does not apply cleanly on it's own.
>>
>> I've done the backports. 4.9 is actually not needed, because it uses spin_{lock,unlock}_irqsave() in peernet2id(). Hence, we have an "offending" commit which this one fixes:
>>
>> fba143c66abb ("netns: avoid disabling irq for netns id")
>>
>> Will get'm out during the weekend.
>
> All now queued up, thanks.
Thanks for helping me out here, Greg!
Appreciated, Håkon
>
> greg k-h
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-09-14 8:58 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-09 13:10 Please add 2dce224f469f ("netns: protect netns ID lookups with RCU") to LTS Haakon Bugge
2021-09-09 13:30 ` Greg KH
2021-09-10 14:22 ` Haakon Bugge
2021-09-13 8:42 ` Greg KH
2021-09-14 8:58 ` Haakon Bugge
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).