* Lockdep warning in iwl_pcie_rx_handle()
@ 2021-03-01 20:44 Jiri Kosina
2021-03-01 21:09 ` Jiri Kosina
2021-03-01 21:12 ` Johannes Berg
0 siblings, 2 replies; 14+ messages in thread
From: Jiri Kosina @ 2021-03-01 20:44 UTC (permalink / raw)
To: Luca Coelho; +Cc: Heiner Kallweit, linux-wireless, linux-kernel
Hi,
I am getting the splat below with Linus' tree as of today (5.11-rc1,
fe07bfda2fb). I haven't started to look into the code yet, but apparently
this has been already reported by Heiner here:
https://www.spinics.net/lists/linux-wireless/msg208353.html
so before I start digging deep into it (the previous kernel this
particular machine had is 5.9, so I'd rather avoid lenghty bisect for now
in case someone has already looked into it and has ideas where the problem
is), I thought I'd ask whether this has been root-caused elsewhere
already.
Thanks.
[ 9.970414] ========================================================
[ 9.973157] WARNING: possible irq lock inversion dependency detected
[ 9.976186] 5.12.0-rc1-00002-gbada49429032 #5 Not tainted
[ 9.979239] --------------------------------------------------------
[ 9.982223] irq/136-iwlwifi/565 just changed the state of lock:
[ 9.984904] ffff89f28433b0b0 (&rxq->lock){+.-.}-{2:2}, at: iwl_pcie_rx_handle+0x7f/0x960 [iwlwifi]
[ 9.987932] but this lock took another, SOFTIRQ-unsafe lock in the past:
[ 9.990980] (napi_hash_lock){+.+.}-{2:2}
[ 9.990986]
and interrupts could create inverse lock ordering between them.
[ 10.003955]
other info that might help us debug this:
[ 10.008989] Possible interrupt unsafe locking scenario:
[ 10.013966] CPU0 CPU1
[ 10.016404] ---- ----
[ 10.018976] lock(napi_hash_lock);
[ 10.021529] local_irq_disable();
[ 10.023985] lock(&rxq->lock);
[ 10.026340] lock(napi_hash_lock);
[ 10.028644] <Interrupt>
[ 10.030934] lock(&rxq->lock);
[ 10.033119]
*** DEADLOCK ***
[ 10.039456] 1 lock held by irq/136-iwlwifi/565:
[ 10.041549] #0: ffff89f2b1440170 (sync_cmd_lockdep_map){+.+.}-{0:0}, at: iwl_pcie_irq_handler+0x5/0xb30 [iwlwifi]
[ 10.043728]
the shortest dependencies between 2nd lock and 1st lock:
[ 10.047836] -> (napi_hash_lock){+.+.}-{2:2} {
[ 10.049845] HARDIRQ-ON-W at:
[ 10.051860] lock_acquire+0x277/0x3d0
[ 10.053881] _raw_spin_lock+0x2c/0x40
[ 10.056170] netif_napi_add+0x14b/0x270
[ 10.058349] e1000_probe+0x2fe/0xee0 [e1000e]
[ 10.060539] local_pci_probe+0x42/0x90
[ 10.062609] pci_device_probe+0x10b/0x1c0
[ 10.064579] really_probe+0xef/0x4b0
[ 10.066578] driver_probe_device+0xde/0x150
[ 10.068765] device_driver_attach+0x4f/0x60
[ 10.070890] __driver_attach+0x9c/0x140
[ 10.073115] bus_for_each_dev+0x79/0xc0
[ 10.075304] bus_add_driver+0x18d/0x220
[ 10.077464] driver_register+0x5b/0xf0
[ 10.079615] do_one_initcall+0x5b/0x300
[ 10.081686] do_init_module+0x5b/0x21c
[ 10.083730] load_module+0x1dae/0x22c0
[ 10.085751] __do_sys_finit_module+0xad/0x110
[ 10.087799] do_syscall_64+0x33/0x80
[ 10.089758] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 10.091747] SOFTIRQ-ON-W at:
[ 10.093634] lock_acquire+0x277/0x3d0
[ 10.095514] _raw_spin_lock+0x2c/0x40
[ 10.097353] netif_napi_add+0x14b/0x270
[ 10.099208] e1000_probe+0x2fe/0xee0 [e1000e]
[ 10.101032] local_pci_probe+0x42/0x90
[ 10.102835] pci_device_probe+0x10b/0x1c0
[ 10.104648] really_probe+0xef/0x4b0
[ 10.106439] driver_probe_device+0xde/0x150
[ 10.106443] device_driver_attach+0x4f/0x60
[ 10.106446] __driver_attach+0x9c/0x140
[ 10.111451] bus_for_each_dev+0x79/0xc0
[ 10.111454] bus_add_driver+0x18d/0x220
[ 10.111456] driver_register+0x5b/0xf0
[ 10.111459] do_one_initcall+0x5b/0x300
[ 10.111462] do_init_module+0x5b/0x21c
[ 10.111467] load_module+0x1dae/0x22c0
[ 10.111469] __do_sys_finit_module+0xad/0x110
[ 10.122761] do_syscall_64+0x33/0x80
[ 10.122768] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 10.122772] INITIAL USE at:
[ 10.122775] lock_acquire+0x277/0x3d0
[ 10.128672] _raw_spin_lock+0x2c/0x40
[ 10.130160] netif_napi_add+0x14b/0x270
[ 10.131725] e1000_probe+0x2fe/0xee0 [e1000e]
[ 10.133268] local_pci_probe+0x42/0x90
[ 10.134783] pci_device_probe+0x10b/0x1c0
[ 10.136325] really_probe+0xef/0x4b0
[ 10.137864] driver_probe_device+0xde/0x150
[ 10.139394] device_driver_attach+0x4f/0x60
[ 10.140908] __driver_attach+0x9c/0x140
[ 10.142429] bus_for_each_dev+0x79/0xc0
[ 10.143947] bus_add_driver+0x18d/0x220
[ 10.145490] driver_register+0x5b/0xf0
[ 10.147024] do_one_initcall+0x5b/0x300
[ 10.148566] do_init_module+0x5b/0x21c
[ 10.150013] load_module+0x1dae/0x22c0
[ 10.151293] __do_sys_finit_module+0xad/0x110
[ 10.152571] do_syscall_64+0x33/0x80
[ 10.153740] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 10.154889] }
[ 10.156023] ... key at: [<ffffffffae84ef38>] napi_hash_lock+0x18/0x40
[ 10.157191] ... acquired at:
[ 10.158318] _raw_spin_lock+0x2c/0x40
[ 10.159418] netif_napi_add+0x14b/0x270
[ 10.160500] _iwl_pcie_rx_init+0x1f4/0x710 [iwlwifi]
[ 10.161609] iwl_pcie_rx_init+0x1b/0x3b0 [iwlwifi]
[ 10.162704] iwl_trans_pcie_start_fw+0x2ac/0x6a0 [iwlwifi]
[ 10.163813] iwl_mvm_load_ucode_wait_alive+0x116/0x460 [iwlmvm]
[ 10.164955] iwl_run_init_mvm_ucode+0xa4/0x3a0 [iwlmvm]
[ 10.166079] iwl_op_mode_mvm_start+0x9ed/0xbf0 [iwlmvm]
[ 10.167206] _iwl_op_mode_start.isra.4+0x42/0x80 [iwlwifi]
[ 10.168326] iwl_opmode_register+0x71/0xe0 [iwlwifi]
[ 10.169451] iwl_mvm_init+0x34/0x1000 [iwlmvm]
[ 10.170585] do_one_initcall+0x5b/0x300
[ 10.171699] do_init_module+0x5b/0x21c
[ 10.172818] load_module+0x1dae/0x22c0
[ 10.173933] __do_sys_finit_module+0xad/0x110
[ 10.175048] do_syscall_64+0x33/0x80
[ 10.176164] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 10.178421] -> (&rxq->lock){+.-.}-{2:2} {
[ 10.179564] HARDIRQ-ON-W at:
[ 10.180709] lock_acquire+0x277/0x3d0
[ 10.181874] _raw_spin_lock_bh+0x34/0x40
[ 10.183049] _iwl_pcie_rx_init+0x130/0x710 [iwlwifi]
[ 10.184229] iwl_pcie_rx_init+0x1b/0x3b0 [iwlwifi]
[ 10.185397] iwl_trans_pcie_start_fw+0x2ac/0x6a0 [iwlwifi]
[ 10.186572] iwl_mvm_load_ucode_wait_alive+0x116/0x460 [iwlmvm]
[ 10.187759] iwl_run_init_mvm_ucode+0xa4/0x3a0 [iwlmvm]
[ 10.188950] iwl_op_mode_mvm_start+0x9ed/0xbf0 [iwlmvm]
[ 10.190132] _iwl_op_mode_start.isra.4+0x42/0x80 [iwlwifi]
[ 10.191312] iwl_opmode_register+0x71/0xe0 [iwlwifi]
[ 10.192497] iwl_mvm_init+0x34/0x1000 [iwlmvm]
[ 10.193685] do_one_initcall+0x5b/0x300
[ 10.194868] do_init_module+0x5b/0x21c
[ 10.196048] load_module+0x1dae/0x22c0
[ 10.197225] __do_sys_finit_module+0xad/0x110
[ 10.198409] do_syscall_64+0x33/0x80
[ 10.199601] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 10.200801] IN-SOFTIRQ-W at:
[ 10.202000] lock_acquire+0x277/0x3d0
[ 10.203202] _raw_spin_lock+0x2c/0x40
[ 10.204412] iwl_pcie_rx_handle+0x7f/0x960 [iwlwifi]
[ 10.205653] iwl_pcie_napi_poll+0x2c/0x100 [iwlwifi]
[ 10.206879] __napi_poll+0x2d/0x220
[ 10.208075] net_rx_action+0x106/0x280
[ 10.209276] __do_softirq+0xd4/0x461
[ 10.210486] do_softirq+0x63/0xa0
[ 10.211677] __local_bh_enable_ip+0xd1/0xf0
[ 10.212879] iwl_pcie_irq_handler+0x67d/0xb30 [iwlwifi]
[ 10.214092] irq_thread_fn+0x21/0x60
[ 10.215298] irq_thread+0x107/0x1a0
[ 10.216491] kthread+0x13e/0x160
[ 10.217670] ret_from_fork+0x22/0x30
[ 10.218855] INITIAL USE at:
[ 10.220030] lock_acquire+0x277/0x3d0
[ 10.221239] _raw_spin_lock_bh+0x34/0x40
[ 10.222438] _iwl_pcie_rx_init+0x130/0x710 [iwlwifi]
[ 10.223667] iwl_pcie_rx_init+0x1b/0x3b0 [iwlwifi]
[ 10.224894] iwl_trans_pcie_start_fw+0x2ac/0x6a0 [iwlwifi]
[ 10.226167] iwl_mvm_load_ucode_wait_alive+0x116/0x460 [iwlmvm]
[ 10.227448] iwl_run_init_mvm_ucode+0xa4/0x3a0 [iwlmvm]
[ 10.228718] iwl_op_mode_mvm_start+0x9ed/0xbf0 [iwlmvm]
[ 10.229977] _iwl_op_mode_start.isra.4+0x42/0x80 [iwlwifi]
[ 10.231236] iwl_opmode_register+0x71/0xe0 [iwlwifi]
[ 10.232492] iwl_mvm_init+0x34/0x1000 [iwlmvm]
[ 10.233758] do_one_initcall+0x5b/0x300
[ 10.235002] do_init_module+0x5b/0x21c
[ 10.236224] load_module+0x1dae/0x22c0
[ 10.237454] __do_sys_finit_module+0xad/0x110
[ 10.238679] do_syscall_64+0x33/0x80
[ 10.239906] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 10.241137] }
[ 10.242327] ... key at: [<ffffffffc111f0c0>] __key.69825+0x0/0xfffffffffffe2f40 [iwlwifi]
[ 10.243587] ... acquired at:
[ 10.244814] __lock_acquire+0x27d/0x1a50
[ 10.246055] lock_acquire+0x277/0x3d0
[ 10.247296] _raw_spin_lock+0x2c/0x40
[ 10.248518] iwl_pcie_rx_handle+0x7f/0x960 [iwlwifi]
[ 10.249789] iwl_pcie_napi_poll+0x2c/0x100 [iwlwifi]
[ 10.251028] __napi_poll+0x2d/0x220
[ 10.252254] net_rx_action+0x106/0x280
[ 10.253484] __do_softirq+0xd4/0x461
[ 10.254731] do_softirq+0x63/0xa0
[ 10.255946] __local_bh_enable_ip+0xd1/0xf0
[ 10.257157] iwl_pcie_irq_handler+0x67d/0xb30 [iwlwifi]
[ 10.258382] irq_thread_fn+0x21/0x60
[ 10.259608] irq_thread+0x107/0x1a0
[ 10.260819] kthread+0x13e/0x160
[ 10.262022] ret_from_fork+0x22/0x30
[ 10.264433]
stack backtrace:
[ 10.266755] CPU: 2 PID: 565 Comm: irq/136-iwlwifi Not tainted 5.12.0-rc1-00002-gbada49429032 #5
[ 10.267964] Hardware name: LENOVO 20K5S22R00/20K5S22R00, BIOS R0IET38W (1.16 ) 05/31/2017
[ 10.269175] Call Trace:
[ 10.270376] <IRQ>
[ 10.271566] dump_stack+0x7f/0xa1
[ 10.272750] mark_lock.part.49+0x8cb/0xc70
[ 10.273950] ? __lock_acquire+0x2e1/0x1a50
[ 10.275164] ? __lock_acquire+0x27d/0x1a50
[ 10.276372] __lock_acquire+0x27d/0x1a50
[ 10.277562] ? __lock_acquire+0x2e1/0x1a50
[ 10.278754] ? __lock_acquire+0x2e1/0x1a50
[ 10.279937] lock_acquire+0x277/0x3d0
[ 10.281110] ? iwl_pcie_rx_handle+0x7f/0x960 [iwlwifi]
[ 10.282317] ? lock_is_held_type+0xb4/0x120
[ 10.283483] _raw_spin_lock+0x2c/0x40
[ 10.284637] ? iwl_pcie_rx_handle+0x7f/0x960 [iwlwifi]
[ 10.285813] iwl_pcie_rx_handle+0x7f/0x960 [iwlwifi]
[ 10.286985] ? __lock_acquire+0x2e1/0x1a50
[ 10.288148] ? try_to_wake_up+0x41c/0x600
[ 10.289323] ? print_dl_rq+0xcc/0x160
[ 10.290490] iwl_pcie_napi_poll+0x2c/0x100 [iwlwifi]
[ 10.291684] __napi_poll+0x2d/0x220
[ 10.292862] net_rx_action+0x106/0x280
[ 10.294034] __do_softirq+0xd4/0x461
[ 10.295185] do_softirq+0x63/0xa0
[ 10.296349] </IRQ>
[ 10.297514] __local_bh_enable_ip+0xd1/0xf0
[ 10.298683] ? iwl_pcie_irq_handler+0x66c/0xb30 [iwlwifi]
[ 10.299882] iwl_pcie_irq_handler+0x67d/0xb30 [iwlwifi]
[ 10.301076] ? irq_forced_thread_fn+0x80/0x80
[ 10.302265] irq_thread_fn+0x21/0x60
[ 10.303434] ? irq_thread+0xa1/0x1a0
[ 10.304596] irq_thread+0x107/0x1a0
[ 10.305766] ? wake_threads_waitq+0x30/0x30
[ 10.306922] ? irq_thread_dtor+0xa0/0xa0
[ 10.308078] kthread+0x13e/0x160
[ 10.309246] ? kthread_create_worker_on_cpu+0x70/0x70
[ 10.310410] ret_from_fork+0x22/0x30
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Lockdep warning in iwl_pcie_rx_handle()
2021-03-01 20:44 Lockdep warning in iwl_pcie_rx_handle() Jiri Kosina
@ 2021-03-01 21:09 ` Jiri Kosina
2021-03-01 21:12 ` Johannes Berg
1 sibling, 0 replies; 14+ messages in thread
From: Jiri Kosina @ 2021-03-01 21:09 UTC (permalink / raw)
To: Luca Coelho; +Cc: Heiner Kallweit, linux-wireless, linux-kernel, Johannes Berg
On Mon, 1 Mar 2021, Jiri Kosina wrote:
> I am getting the splat below with Linus' tree as of today (5.11-rc1,
> fe07bfda2fb). I haven't started to look into the code yet, but apparently
> this has been already reported by Heiner here:
>
> https://www.spinics.net/lists/linux-wireless/msg208353.html
>
> so before I start digging deep into it (the previous kernel this
> particular machine had is 5.9, so I'd rather avoid lenghty bisect for now
> in case someone has already looked into it and has ideas where the problem
> is), I thought I'd ask whether this has been root-caused elsewhere
> already.
>
> Thanks.
After reverting 25edc8f259c7106 ("iwlwifi: pcie: properly implement
NAPI"), I don't see the lockdep warning any more (*), so it seems to be
culprit (or at least related). CCing Johannes.
Leaving the original mail below for reference.
(*) I see the warning below instead; but let's focus on fixing the lockdep
reported issue in the first place, I guess. It might be that I'd need
to revert more things in order to get to a fully consistent state.
[ 23.488194] WARNING: CPU: 1 PID: 1139 at kernel/softirq.c:178 __local_bh_enable_ip+0xa5/0xf0
[ 23.488203] Modules linked in: tun ip6table_mangle ip6table_filter ip6_tables iptable_mangle xt_DSCP xt_tcpudp xt_conntrack nf_conntrack nf_defrag_ipv6 libcrc32c nf_defrag_ipv4 iptable_filter ip_tables x_tables algif_skcipher af_alg bnep dm_crypt hid_generic snd_usb_audio snd_usbmidi_lib snd_rawmidi uvcvideo videobuf2_vmalloc btusb videobuf2_memops btrtl btbcm videobuf2_v4l2 btintel videodev videobuf2_common bluetooth ecdh_generic ecc iwlmvm mac80211 snd_hda_codec_hdmi libarc4 intel_rapl_msr snd_hda_codec_realtek snd_soc_skl iTCO_wdt iTCO_vendor_support snd_soc_sst_ipc snd_hda_codec_generic snd_soc_sst_dsp snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi wmi_bmof snd_soc_core intel_rapl_common x86_pkg_temp_thermal usbhid intel_powerclamp snd_compress coretemp snd_hda_intel snd_intel_dspcfg kvm_intel snd_hda_codec iwlwifi snd_hwdep e1000e kvm snd_hda_core ptp irqbypass joydev pcspkr snd_pcm i2c_i801 pps_core cfg80211 i2c_smbus snd_timer mei_me mei thermal intel_pch_thermal
[ 23.488334] thinkpad_acpi wmi battery ledtrig_audio platform_profile snd ac soundcore rfkill tpm_crb tpm_tis tpm_tis_core tpm acpi_pad button nls_iso8859_1 nls_cp437 vfat fat dm_mod fuse rtsx_pci_sdmmc mmc_core crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd i915 i2c_algo_bit drm_kms_helper syscopyarea xhci_pci serio_raw sysfillrect sysimgblt fb_sys_fops xhci_hcd rtsx_pci usbcore drm video sg msr efivarfs
[ 23.488404] CPU: 1 PID: 1139 Comm: NetworkManager Not tainted 5.12.0-rc1-00004-gb4ded168af79 #7
[ 23.488408] Hardware name: LENOVO 20K5S22R00/20K5S22R00, BIOS R0IET38W (1.16 ) 05/31/2017
[ 23.488411] RIP: 0010:__local_bh_enable_ip+0xa5/0xf0
[ 23.488416] Code: f7 69 e8 ee 23 14 00 fb 66 0f 1f 44 00 00 65 8b 05 f0 f4 f7 69 85 c0 74 3f 48 83 c4 08 5b c3 65 8b 05 9b fe f7 69 85 c0 75 8e <0f> 0b eb 8a 48 89 3c 24 e8 4e 20 14 00 48 8b 3c 24 eb 91 e8 13 4e
[ 23.488420] RSP: 0018:ffffafd580b13298 EFLAGS: 00010046
[ 23.488424] RAX: 0000000000000000 RBX: 0000000000000201 RCX: 0000000000000000
[ 23.488426] RDX: 0000000000000003 RSI: 0000000000000201 RDI: ffffffffc1272389
[ 23.488429] RBP: ffff96517ae4c018 R08: 0000000000000001 R09: 0000000000000000
[ 23.488431] R10: ffffafd580b13178 R11: 0000000000000001 R12: ffff96517b060000
[ 23.488433] R13: 0000000000000000 R14: ffffffff80000000 R15: 0000000000000001
[ 23.488436] FS: 00007fc604ebefc0(0000) GS:ffff965267480000(0000) knlGS:0000000000000000
[ 23.488439] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 23.488442] CR2: 000055fb3fef13b2 CR3: 0000000109112004 CR4: 00000000003706e0
[ 23.488444] Call Trace:
[ 23.488447] ? _raw_spin_unlock_bh+0x1f/0x30
[ 23.488453] iwl_pcie_enqueue_hcmd+0x5d9/0xa00 [iwlwifi]
[ 23.488499] iwl_trans_txq_send_hcmd+0x6c/0x430 [iwlwifi]
[ 23.488536] iwl_trans_send_cmd+0x88/0x170 [iwlwifi]
[ 23.488556] ? lock_acquire+0x277/0x3d0
[ 23.488564] iwl_mvm_send_cmd+0x32/0x80 [iwlmvm]
[ 23.488593] iwl_mvm_led_set+0xc2/0xe0 [iwlmvm]
[ 23.488629] ? led_trigger_event+0x46/0x70
[ 23.488633] led_trigger_event+0x46/0x70
[ 23.488643] ieee80211_do_open+0x5c5/0xa20 [mac80211]
[ 23.488719] ieee80211_open+0x67/0x90 [mac80211]
[ 23.488777] __dev_open+0xd4/0x150
[ 23.488790] __dev_change_flags+0x19e/0x1f0
[ 23.488805] dev_change_flags+0x23/0x60
[ 23.488813] do_setlink+0x30d/0x1230
[ 23.488829] ? lock_is_held_type+0xb4/0x120
[ 23.488841] ? __nla_validate_parse.part.7+0x57/0xcb0
[ 23.488855] ? __lock_acquire+0x2e1/0x1a50
[ 23.488870] __rtnl_newlink+0x560/0x910
[ 23.488890] ? __lock_acquire+0x2e1/0x1a50
[ 23.488893] ? __lock_acquire+0x2e1/0x1a50
[ 23.488915] ? lock_acquire+0x277/0x3d0
[ 23.488919] ? sock_def_readable+0x5/0x290
[ 23.488923] ? lock_is_held_type+0xb4/0x120
[ 23.488931] ? find_held_lock+0x2d/0x90
[ 23.488938] ? sock_def_readable+0xb3/0x290
[ 23.488944] ? lock_release+0x166/0x2a0
[ 23.489017] ? lock_is_held_type+0x90/0x120
[ 23.489037] rtnl_newlink+0x47/0x70
[ 23.489047] rtnetlink_rcv_msg+0x25c/0x470
[ 23.489057] ? netlink_deliver_tap+0x97/0x3e0
[ 23.489063] ? validate_linkmsg+0x350/0x350
[ 23.489072] netlink_rcv_skb+0x50/0x100
[ 23.489089] netlink_unicast+0x1b2/0x280
[ 23.489100] netlink_sendmsg+0x336/0x450
[ 23.489120] sock_sendmsg+0x5b/0x60
[ 23.489127] ____sys_sendmsg+0x1ed/0x250
[ 23.489131] ? copy_msghdr_from_user+0x5c/0x90
[ 23.489149] ___sys_sendmsg+0x88/0xd0
[ 23.489165] ? lock_is_held_type+0xb4/0x120
[ 23.489172] ? find_held_lock+0x2d/0x90
[ 23.489182] ? lock_release+0x166/0x2a0
[ 23.489194] ? __fget_files+0xfe/0x1d0
[ 23.489209] ? __sys_sendmsg+0x5e/0xa0
[ 23.489212] __sys_sendmsg+0x5e/0xa0
[ 23.489226] ? lockdep_hardirqs_on_prepare+0xd9/0x170
[ 23.489237] do_syscall_64+0x33/0x80
[ 23.489243] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 23.489247] RIP: 0033:0x7fc605c9572d
[ 23.489251] Code: 28 89 54 24 1c 48 89 74 24 10 89 7c 24 08 e8 da ee ff ff 8b 54 24 1c 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 33 44 89 c7 48 89 44 24 08 e8 2e ef ff ff 48
[ 23.489255] RSP: 002b:00007fffc83789f0 EFLAGS: 00000293 ORIG_RAX: 000000000000002e
[ 23.489259] RAX: ffffffffffffffda RBX: 000055ef468570c0 RCX: 00007fc605c9572d
[ 23.489261] RDX: 0000000000000000 RSI: 00007fffc8378a30 RDI: 000000000000000c
[ 23.489263] RBP: 0000000000000010 R08: 0000000000000000 R09: 0000000000000000
[ 23.489265] R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000000
[ 23.489267] R13: 00007fffc8378b80 R14: 00007fffc8378b7c R15: 0000000000000000
[ 23.489295] irq event stamp: 170785
[ 23.489297] hardirqs last enabled at (170783): [<ffffffff9609a8c2>] __local_bh_enable_ip+0x82/0xf0
[ 23.489303] hardirqs last disabled at (170784): [<ffffffff96a8613d>] _raw_read_lock_irqsave+0x8d/0x90
[ 23.489307] softirqs last enabled at (170782): [<ffffffffc1272389>] iwl_pcie_enqueue_hcmd+0x5d9/0xa00 [iwlwifi]
[ 23.489328] softirqs last disabled at (170785): [<ffffffffc1271ec6>] iwl_pcie_enqueue_hcmd+0x116/0xa00 [iwlwifi]
>
>
>
>
> [ 9.970414] ========================================================
> [ 9.973157] WARNING: possible irq lock inversion dependency detected
> [ 9.976186] 5.12.0-rc1-00002-gbada49429032 #5 Not tainted
> [ 9.979239] --------------------------------------------------------
> [ 9.982223] irq/136-iwlwifi/565 just changed the state of lock:
> [ 9.984904] ffff89f28433b0b0 (&rxq->lock){+.-.}-{2:2}, at: iwl_pcie_rx_handle+0x7f/0x960 [iwlwifi]
> [ 9.987932] but this lock took another, SOFTIRQ-unsafe lock in the past:
> [ 9.990980] (napi_hash_lock){+.+.}-{2:2}
> [ 9.990986]
>
> and interrupts could create inverse lock ordering between them.
>
> [ 10.003955]
> other info that might help us debug this:
> [ 10.008989] Possible interrupt unsafe locking scenario:
>
> [ 10.013966] CPU0 CPU1
> [ 10.016404] ---- ----
> [ 10.018976] lock(napi_hash_lock);
> [ 10.021529] local_irq_disable();
> [ 10.023985] lock(&rxq->lock);
> [ 10.026340] lock(napi_hash_lock);
> [ 10.028644] <Interrupt>
> [ 10.030934] lock(&rxq->lock);
> [ 10.033119]
> *** DEADLOCK ***
>
> [ 10.039456] 1 lock held by irq/136-iwlwifi/565:
> [ 10.041549] #0: ffff89f2b1440170 (sync_cmd_lockdep_map){+.+.}-{0:0}, at: iwl_pcie_irq_handler+0x5/0xb30 [iwlwifi]
> [ 10.043728]
> the shortest dependencies between 2nd lock and 1st lock:
> [ 10.047836] -> (napi_hash_lock){+.+.}-{2:2} {
> [ 10.049845] HARDIRQ-ON-W at:
> [ 10.051860] lock_acquire+0x277/0x3d0
> [ 10.053881] _raw_spin_lock+0x2c/0x40
> [ 10.056170] netif_napi_add+0x14b/0x270
> [ 10.058349] e1000_probe+0x2fe/0xee0 [e1000e]
> [ 10.060539] local_pci_probe+0x42/0x90
> [ 10.062609] pci_device_probe+0x10b/0x1c0
> [ 10.064579] really_probe+0xef/0x4b0
> [ 10.066578] driver_probe_device+0xde/0x150
> [ 10.068765] device_driver_attach+0x4f/0x60
> [ 10.070890] __driver_attach+0x9c/0x140
> [ 10.073115] bus_for_each_dev+0x79/0xc0
> [ 10.075304] bus_add_driver+0x18d/0x220
> [ 10.077464] driver_register+0x5b/0xf0
> [ 10.079615] do_one_initcall+0x5b/0x300
> [ 10.081686] do_init_module+0x5b/0x21c
> [ 10.083730] load_module+0x1dae/0x22c0
> [ 10.085751] __do_sys_finit_module+0xad/0x110
> [ 10.087799] do_syscall_64+0x33/0x80
> [ 10.089758] entry_SYSCALL_64_after_hwframe+0x44/0xae
> [ 10.091747] SOFTIRQ-ON-W at:
> [ 10.093634] lock_acquire+0x277/0x3d0
> [ 10.095514] _raw_spin_lock+0x2c/0x40
> [ 10.097353] netif_napi_add+0x14b/0x270
> [ 10.099208] e1000_probe+0x2fe/0xee0 [e1000e]
> [ 10.101032] local_pci_probe+0x42/0x90
> [ 10.102835] pci_device_probe+0x10b/0x1c0
> [ 10.104648] really_probe+0xef/0x4b0
> [ 10.106439] driver_probe_device+0xde/0x150
> [ 10.106443] device_driver_attach+0x4f/0x60
> [ 10.106446] __driver_attach+0x9c/0x140
> [ 10.111451] bus_for_each_dev+0x79/0xc0
> [ 10.111454] bus_add_driver+0x18d/0x220
> [ 10.111456] driver_register+0x5b/0xf0
> [ 10.111459] do_one_initcall+0x5b/0x300
> [ 10.111462] do_init_module+0x5b/0x21c
> [ 10.111467] load_module+0x1dae/0x22c0
> [ 10.111469] __do_sys_finit_module+0xad/0x110
> [ 10.122761] do_syscall_64+0x33/0x80
> [ 10.122768] entry_SYSCALL_64_after_hwframe+0x44/0xae
> [ 10.122772] INITIAL USE at:
> [ 10.122775] lock_acquire+0x277/0x3d0
> [ 10.128672] _raw_spin_lock+0x2c/0x40
> [ 10.130160] netif_napi_add+0x14b/0x270
> [ 10.131725] e1000_probe+0x2fe/0xee0 [e1000e]
> [ 10.133268] local_pci_probe+0x42/0x90
> [ 10.134783] pci_device_probe+0x10b/0x1c0
> [ 10.136325] really_probe+0xef/0x4b0
> [ 10.137864] driver_probe_device+0xde/0x150
> [ 10.139394] device_driver_attach+0x4f/0x60
> [ 10.140908] __driver_attach+0x9c/0x140
> [ 10.142429] bus_for_each_dev+0x79/0xc0
> [ 10.143947] bus_add_driver+0x18d/0x220
> [ 10.145490] driver_register+0x5b/0xf0
> [ 10.147024] do_one_initcall+0x5b/0x300
> [ 10.148566] do_init_module+0x5b/0x21c
> [ 10.150013] load_module+0x1dae/0x22c0
> [ 10.151293] __do_sys_finit_module+0xad/0x110
> [ 10.152571] do_syscall_64+0x33/0x80
> [ 10.153740] entry_SYSCALL_64_after_hwframe+0x44/0xae
> [ 10.154889] }
> [ 10.156023] ... key at: [<ffffffffae84ef38>] napi_hash_lock+0x18/0x40
> [ 10.157191] ... acquired at:
> [ 10.158318] _raw_spin_lock+0x2c/0x40
> [ 10.159418] netif_napi_add+0x14b/0x270
> [ 10.160500] _iwl_pcie_rx_init+0x1f4/0x710 [iwlwifi]
> [ 10.161609] iwl_pcie_rx_init+0x1b/0x3b0 [iwlwifi]
> [ 10.162704] iwl_trans_pcie_start_fw+0x2ac/0x6a0 [iwlwifi]
> [ 10.163813] iwl_mvm_load_ucode_wait_alive+0x116/0x460 [iwlmvm]
> [ 10.164955] iwl_run_init_mvm_ucode+0xa4/0x3a0 [iwlmvm]
> [ 10.166079] iwl_op_mode_mvm_start+0x9ed/0xbf0 [iwlmvm]
> [ 10.167206] _iwl_op_mode_start.isra.4+0x42/0x80 [iwlwifi]
> [ 10.168326] iwl_opmode_register+0x71/0xe0 [iwlwifi]
> [ 10.169451] iwl_mvm_init+0x34/0x1000 [iwlmvm]
> [ 10.170585] do_one_initcall+0x5b/0x300
> [ 10.171699] do_init_module+0x5b/0x21c
> [ 10.172818] load_module+0x1dae/0x22c0
> [ 10.173933] __do_sys_finit_module+0xad/0x110
> [ 10.175048] do_syscall_64+0x33/0x80
> [ 10.176164] entry_SYSCALL_64_after_hwframe+0x44/0xae
>
> [ 10.178421] -> (&rxq->lock){+.-.}-{2:2} {
> [ 10.179564] HARDIRQ-ON-W at:
> [ 10.180709] lock_acquire+0x277/0x3d0
> [ 10.181874] _raw_spin_lock_bh+0x34/0x40
> [ 10.183049] _iwl_pcie_rx_init+0x130/0x710 [iwlwifi]
> [ 10.184229] iwl_pcie_rx_init+0x1b/0x3b0 [iwlwifi]
> [ 10.185397] iwl_trans_pcie_start_fw+0x2ac/0x6a0 [iwlwifi]
> [ 10.186572] iwl_mvm_load_ucode_wait_alive+0x116/0x460 [iwlmvm]
> [ 10.187759] iwl_run_init_mvm_ucode+0xa4/0x3a0 [iwlmvm]
> [ 10.188950] iwl_op_mode_mvm_start+0x9ed/0xbf0 [iwlmvm]
> [ 10.190132] _iwl_op_mode_start.isra.4+0x42/0x80 [iwlwifi]
> [ 10.191312] iwl_opmode_register+0x71/0xe0 [iwlwifi]
> [ 10.192497] iwl_mvm_init+0x34/0x1000 [iwlmvm]
> [ 10.193685] do_one_initcall+0x5b/0x300
> [ 10.194868] do_init_module+0x5b/0x21c
> [ 10.196048] load_module+0x1dae/0x22c0
> [ 10.197225] __do_sys_finit_module+0xad/0x110
> [ 10.198409] do_syscall_64+0x33/0x80
> [ 10.199601] entry_SYSCALL_64_after_hwframe+0x44/0xae
> [ 10.200801] IN-SOFTIRQ-W at:
> [ 10.202000] lock_acquire+0x277/0x3d0
> [ 10.203202] _raw_spin_lock+0x2c/0x40
> [ 10.204412] iwl_pcie_rx_handle+0x7f/0x960 [iwlwifi]
> [ 10.205653] iwl_pcie_napi_poll+0x2c/0x100 [iwlwifi]
> [ 10.206879] __napi_poll+0x2d/0x220
> [ 10.208075] net_rx_action+0x106/0x280
> [ 10.209276] __do_softirq+0xd4/0x461
> [ 10.210486] do_softirq+0x63/0xa0
> [ 10.211677] __local_bh_enable_ip+0xd1/0xf0
> [ 10.212879] iwl_pcie_irq_handler+0x67d/0xb30 [iwlwifi]
> [ 10.214092] irq_thread_fn+0x21/0x60
> [ 10.215298] irq_thread+0x107/0x1a0
> [ 10.216491] kthread+0x13e/0x160
> [ 10.217670] ret_from_fork+0x22/0x30
> [ 10.218855] INITIAL USE at:
> [ 10.220030] lock_acquire+0x277/0x3d0
> [ 10.221239] _raw_spin_lock_bh+0x34/0x40
> [ 10.222438] _iwl_pcie_rx_init+0x130/0x710 [iwlwifi]
> [ 10.223667] iwl_pcie_rx_init+0x1b/0x3b0 [iwlwifi]
> [ 10.224894] iwl_trans_pcie_start_fw+0x2ac/0x6a0 [iwlwifi]
> [ 10.226167] iwl_mvm_load_ucode_wait_alive+0x116/0x460 [iwlmvm]
> [ 10.227448] iwl_run_init_mvm_ucode+0xa4/0x3a0 [iwlmvm]
> [ 10.228718] iwl_op_mode_mvm_start+0x9ed/0xbf0 [iwlmvm]
> [ 10.229977] _iwl_op_mode_start.isra.4+0x42/0x80 [iwlwifi]
> [ 10.231236] iwl_opmode_register+0x71/0xe0 [iwlwifi]
> [ 10.232492] iwl_mvm_init+0x34/0x1000 [iwlmvm]
> [ 10.233758] do_one_initcall+0x5b/0x300
> [ 10.235002] do_init_module+0x5b/0x21c
> [ 10.236224] load_module+0x1dae/0x22c0
> [ 10.237454] __do_sys_finit_module+0xad/0x110
> [ 10.238679] do_syscall_64+0x33/0x80
> [ 10.239906] entry_SYSCALL_64_after_hwframe+0x44/0xae
> [ 10.241137] }
> [ 10.242327] ... key at: [<ffffffffc111f0c0>] __key.69825+0x0/0xfffffffffffe2f40 [iwlwifi]
> [ 10.243587] ... acquired at:
> [ 10.244814] __lock_acquire+0x27d/0x1a50
> [ 10.246055] lock_acquire+0x277/0x3d0
> [ 10.247296] _raw_spin_lock+0x2c/0x40
> [ 10.248518] iwl_pcie_rx_handle+0x7f/0x960 [iwlwifi]
> [ 10.249789] iwl_pcie_napi_poll+0x2c/0x100 [iwlwifi]
> [ 10.251028] __napi_poll+0x2d/0x220
> [ 10.252254] net_rx_action+0x106/0x280
> [ 10.253484] __do_softirq+0xd4/0x461
> [ 10.254731] do_softirq+0x63/0xa0
> [ 10.255946] __local_bh_enable_ip+0xd1/0xf0
> [ 10.257157] iwl_pcie_irq_handler+0x67d/0xb30 [iwlwifi]
> [ 10.258382] irq_thread_fn+0x21/0x60
> [ 10.259608] irq_thread+0x107/0x1a0
> [ 10.260819] kthread+0x13e/0x160
> [ 10.262022] ret_from_fork+0x22/0x30
>
> [ 10.264433]
> stack backtrace:
> [ 10.266755] CPU: 2 PID: 565 Comm: irq/136-iwlwifi Not tainted 5.12.0-rc1-00002-gbada49429032 #5
> [ 10.267964] Hardware name: LENOVO 20K5S22R00/20K5S22R00, BIOS R0IET38W (1.16 ) 05/31/2017
> [ 10.269175] Call Trace:
> [ 10.270376] <IRQ>
> [ 10.271566] dump_stack+0x7f/0xa1
> [ 10.272750] mark_lock.part.49+0x8cb/0xc70
> [ 10.273950] ? __lock_acquire+0x2e1/0x1a50
> [ 10.275164] ? __lock_acquire+0x27d/0x1a50
> [ 10.276372] __lock_acquire+0x27d/0x1a50
> [ 10.277562] ? __lock_acquire+0x2e1/0x1a50
> [ 10.278754] ? __lock_acquire+0x2e1/0x1a50
> [ 10.279937] lock_acquire+0x277/0x3d0
> [ 10.281110] ? iwl_pcie_rx_handle+0x7f/0x960 [iwlwifi]
> [ 10.282317] ? lock_is_held_type+0xb4/0x120
> [ 10.283483] _raw_spin_lock+0x2c/0x40
> [ 10.284637] ? iwl_pcie_rx_handle+0x7f/0x960 [iwlwifi]
> [ 10.285813] iwl_pcie_rx_handle+0x7f/0x960 [iwlwifi]
> [ 10.286985] ? __lock_acquire+0x2e1/0x1a50
> [ 10.288148] ? try_to_wake_up+0x41c/0x600
> [ 10.289323] ? print_dl_rq+0xcc/0x160
> [ 10.290490] iwl_pcie_napi_poll+0x2c/0x100 [iwlwifi]
> [ 10.291684] __napi_poll+0x2d/0x220
> [ 10.292862] net_rx_action+0x106/0x280
> [ 10.294034] __do_softirq+0xd4/0x461
> [ 10.295185] do_softirq+0x63/0xa0
> [ 10.296349] </IRQ>
> [ 10.297514] __local_bh_enable_ip+0xd1/0xf0
> [ 10.298683] ? iwl_pcie_irq_handler+0x66c/0xb30 [iwlwifi]
> [ 10.299882] iwl_pcie_irq_handler+0x67d/0xb30 [iwlwifi]
> [ 10.301076] ? irq_forced_thread_fn+0x80/0x80
> [ 10.302265] irq_thread_fn+0x21/0x60
> [ 10.303434] ? irq_thread+0xa1/0x1a0
> [ 10.304596] irq_thread+0x107/0x1a0
> [ 10.305766] ? wake_threads_waitq+0x30/0x30
> [ 10.306922] ? irq_thread_dtor+0xa0/0xa0
> [ 10.308078] kthread+0x13e/0x160
> [ 10.309246] ? kthread_create_worker_on_cpu+0x70/0x70
> [ 10.310410] ret_from_fork+0x22/0x30
>
--
Jiri Kosina
SUSE Labs
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Lockdep warning in iwl_pcie_rx_handle()
2021-03-01 20:44 Lockdep warning in iwl_pcie_rx_handle() Jiri Kosina
2021-03-01 21:09 ` Jiri Kosina
@ 2021-03-01 21:12 ` Johannes Berg
2021-03-02 9:27 ` [PATCH] iwlwifi: don't call netif_napi_add() with rxq->lock held (was Re: Lockdep warning in iwl_pcie_rx_handle()) Jiri Kosina
1 sibling, 1 reply; 14+ messages in thread
From: Johannes Berg @ 2021-03-01 21:12 UTC (permalink / raw)
To: Jiri Kosina, Luca Coelho; +Cc: Heiner Kallweit, linux-wireless, linux-kernel
Hi Jiri,
> I am getting the splat below with Linus' tree as of today (5.11-rc1,
> fe07bfda2fb). I haven't started to look into the code yet, but apparently
> this has been already reported by Heiner here:
>
> https://www.spinics.net/lists/linux-wireless/msg208353.html
>
> so before I start digging deep into it (the previous kernel this
> particular machine had is 5.9, so I'd rather avoid lenghty bisect for now
> in case someone has already looked into it and has ideas where the problem
> is), I thought I'd ask whether this has been root-caused elsewhere
> already.
Yeah, I'm pretty sure we have a fix for this, though I'm not sure right
now where it is in the pipeline.
It's called "iwlwifi: pcie: don't add NAPI under rxq->lock" but right
now I can't find it in any of the public archives.
johannes
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH] iwlwifi: don't call netif_napi_add() with rxq->lock held (was Re: Lockdep warning in iwl_pcie_rx_handle())
2021-03-01 21:12 ` Johannes Berg
@ 2021-03-02 9:27 ` Jiri Kosina
2021-03-02 10:34 ` [PATCH v2] " Jiri Kosina
2021-03-02 10:48 ` [PATCH] " Coelho, Luciano
0 siblings, 2 replies; 14+ messages in thread
From: Jiri Kosina @ 2021-03-02 9:27 UTC (permalink / raw)
To: Johannes Berg; +Cc: Luca Coelho, Heiner Kallweit, linux-wireless, linux-kernel
On Mon, 1 Mar 2021, Johannes Berg wrote:
> > I am getting the splat below with Linus' tree as of today (5.11-rc1,
> > fe07bfda2fb). I haven't started to look into the code yet, but apparently
> > this has been already reported by Heiner here:
> >
> > https://www.spinics.net/lists/linux-wireless/msg208353.html
> >
> > so before I start digging deep into it (the previous kernel this
> > particular machine had is 5.9, so I'd rather avoid lenghty bisect for now
> > in case someone has already looked into it and has ideas where the problem
> > is), I thought I'd ask whether this has been root-caused elsewhere
> > already.
>
> Yeah, I'm pretty sure we have a fix for this, though I'm not sure right
> now where it is in the pipeline.
>
> It's called "iwlwifi: pcie: don't add NAPI under rxq->lock" but right
> now I can't find it in any of the public archives.
I was not able to find that patch anywhere indeed, but in the meantime I
fixed it by the patch below. Please consider merging either of the fixes.
Also I am still seeing
WARNING: CPU: 2 PID: 1138 at kernel/softirq.c:178 __local_bh_enable_ip+0xa5/0xf0
on the
[ 21.902173] iwl_pcie_enqueue_hcmd+0x5d9/0xa00 [iwlwifi]
[ 21.906445] iwl_trans_txq_send_hcmd+0x6c/0x430 [iwlwifi]
[ 21.910757] iwl_trans_send_cmd+0x88/0x170 [iwlwifi]
[ 21.915074] ? lock_acquire+0x277/0x3d0
[ 21.919327] iwl_mvm_send_cmd+0x32/0x80 [iwlmvm]
[ 21.923616] iwl_mvm_led_set+0xc2/0xe0 [iwlmvm]
codepath, but I'll look into it separately.
From: Jiri Kosina <jkosina@suse.cz>
Subject: [PATCH] iwlwifi: don't call netif_napi_add() with rxq->lock held
We can't call netif_napi_add() with rxq-lock held, as there is a potential
for deadlock as spotted by lockdep (see below). rxq->lock is not
protecting anything over the netif_napi_add() codepath anyway, so let's
drop it just before calling into NAPI.
========================================================
WARNING: possible irq lock inversion dependency detected
5.12.0-rc1-00002-gbada49429032 #5 Not tainted
--------------------------------------------------------
irq/136-iwlwifi/565 just changed the state of lock:
ffff89f28433b0b0 (&rxq->lock){+.-.}-{2:2}, at: iwl_pcie_rx_handle+0x7f/0x960 [iwlwifi]
but this lock took another, SOFTIRQ-unsafe lock in the past:
(napi_hash_lock){+.+.}-{2:2}
and interrupts could create inverse lock ordering between them.
other info that might help us debug this:
Possible interrupt unsafe locking scenario:
CPU0 CPU1
---- ----
lock(napi_hash_lock);
local_irq_disable();
lock(&rxq->lock);
lock(napi_hash_lock);
<Interrupt>
lock(&rxq->lock);
*** DEADLOCK ***
1 lock held by irq/136-iwlwifi/565:
#0: ffff89f2b1440170 (sync_cmd_lockdep_map){+.+.}-{0:0}, at: iwl_pcie_irq_handler+0x5/0xb30
the shortest dependencies between 2nd lock and 1st lock:
-> (napi_hash_lock){+.+.}-{2:2} {
HARDIRQ-ON-W at:
lock_acquire+0x277/0x3d0
_raw_spin_lock+0x2c/0x40
netif_napi_add+0x14b/0x270
e1000_probe+0x2fe/0xee0 [e1000e]
local_pci_probe+0x42/0x90
pci_device_probe+0x10b/0x1c0
really_probe+0xef/0x4b0
driver_probe_device+0xde/0x150
device_driver_attach+0x4f/0x60
__driver_attach+0x9c/0x140
bus_for_each_dev+0x79/0xc0
bus_add_driver+0x18d/0x220
driver_register+0x5b/0xf0
do_one_initcall+0x5b/0x300
do_init_module+0x5b/0x21c
load_module+0x1dae/0x22c0
__do_sys_finit_module+0xad/0x110
do_syscall_64+0x33/0x80
entry_SYSCALL_64_after_hwframe+0x44/0xae
SOFTIRQ-ON-W at:
lock_acquire+0x277/0x3d0
_raw_spin_lock+0x2c/0x40
netif_napi_add+0x14b/0x270
e1000_probe+0x2fe/0xee0 [e1000e]
local_pci_probe+0x42/0x90
pci_device_probe+0x10b/0x1c0
really_probe+0xef/0x4b0
driver_probe_device+0xde/0x150
device_driver_attach+0x4f/0x60
__driver_attach+0x9c/0x140
bus_for_each_dev+0x79/0xc0
bus_add_driver+0x18d/0x220
driver_register+0x5b/0xf0
do_one_initcall+0x5b/0x300
do_init_module+0x5b/0x21c
load_module+0x1dae/0x22c0
__do_sys_finit_module+0xad/0x110
do_syscall_64+0x33/0x80
entry_SYSCALL_64_after_hwframe+0x44/0xae
INITIAL USE at:
lock_acquire+0x277/0x3d0
_raw_spin_lock+0x2c/0x40
netif_napi_add+0x14b/0x270
e1000_probe+0x2fe/0xee0 [e1000e]
local_pci_probe+0x42/0x90
pci_device_probe+0x10b/0x1c0
really_probe+0xef/0x4b0
driver_probe_device+0xde/0x150
device_driver_attach+0x4f/0x60
__driver_attach+0x9c/0x140
bus_for_each_dev+0x79/0xc0
bus_add_driver+0x18d/0x220
driver_register+0x5b/0xf0
do_one_initcall+0x5b/0x300
do_init_module+0x5b/0x21c
load_module+0x1dae/0x22c0
__do_sys_finit_module+0xad/0x110
do_syscall_64+0x33/0x80
entry_SYSCALL_64_after_hwframe+0x44/0xae
}
... key at: [<ffffffffae84ef38>] napi_hash_lock+0x18/0x40
... acquired at:
_raw_spin_lock+0x2c/0x40
netif_napi_add+0x14b/0x270
_iwl_pcie_rx_init+0x1f4/0x710 [iwlwifi]
iwl_pcie_rx_init+0x1b/0x3b0 [iwlwifi]
iwl_trans_pcie_start_fw+0x2ac/0x6a0 [iwlwifi]
iwl_mvm_load_ucode_wait_alive+0x116/0x460 [iwlmvm]
iwl_run_init_mvm_ucode+0xa4/0x3a0 [iwlmvm]
iwl_op_mode_mvm_start+0x9ed/0xbf0 [iwlmvm]
_iwl_op_mode_start.isra.4+0x42/0x80 [iwlwifi]
iwl_opmode_register+0x71/0xe0 [iwlwifi]
iwl_mvm_init+0x34/0x1000 [iwlmvm]
do_one_initcall+0x5b/0x300
do_init_module+0x5b/0x21c
load_module+0x1dae/0x22c0
__do_sys_finit_module+0xad/0x110
do_syscall_64+0x33/0x80
entry_SYSCALL_64_after_hwframe+0x44/0xae
[ ... lockdep output trimmed .... ]
Fixes: 25edc8f259c7106 ("iwlwifi: pcie: properly implement NAPI")
Signed-off-by: Jiri Kosina <jkosina@suse.cz>
---
drivers/net/wireless/intel/iwlwifi/pcie/rx.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/net/wireless/intel/iwlwifi/pcie/rx.c b/drivers/net/wireless/intel/iwlwifi/pcie/rx.c
index 1bccaa034284..6262dd9043aa 100644
--- a/drivers/net/wireless/intel/iwlwifi/pcie/rx.c
+++ b/drivers/net/wireless/intel/iwlwifi/pcie/rx.c
@@ -1063,11 +1063,12 @@ static int _iwl_pcie_rx_init(struct iwl_trans *trans)
iwl_pcie_rx_init_rxb_lists(rxq);
+ spin_unlock_bh(&rxq->lock);
+
if (!rxq->napi.poll)
netif_napi_add(&trans_pcie->napi_dev, &rxq->napi,
iwl_pcie_dummy_napi_poll, 64);
- spin_unlock_bh(&rxq->lock);
}
/* move the pool to the default queue and allocator ownerships */
--
Jiri Kosina
SUSE Labs
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH v2] iwlwifi: don't call netif_napi_add() with rxq->lock held (was Re: Lockdep warning in iwl_pcie_rx_handle())
2021-03-02 9:27 ` [PATCH] iwlwifi: don't call netif_napi_add() with rxq->lock held (was Re: Lockdep warning in iwl_pcie_rx_handle()) Jiri Kosina
@ 2021-03-02 10:34 ` Jiri Kosina
2021-03-02 10:49 ` Coelho, Luciano
` (2 more replies)
2021-03-02 10:48 ` [PATCH] " Coelho, Luciano
1 sibling, 3 replies; 14+ messages in thread
From: Jiri Kosina @ 2021-03-02 10:34 UTC (permalink / raw)
To: Johannes Berg; +Cc: Luca Coelho, Heiner Kallweit, linux-wireless, linux-kernel
From: Jiri Kosina <jkosina@suse.cz>
We can't call netif_napi_add() with rxq-lock held, as there is a potential
for deadlock as spotted by lockdep (see below). rxq->lock is not
protecting anything over the netif_napi_add() codepath anyway, so let's
drop it just before calling into NAPI.
========================================================
WARNING: possible irq lock inversion dependency detected
5.12.0-rc1-00002-gbada49429032 #5 Not tainted
--------------------------------------------------------
irq/136-iwlwifi/565 just changed the state of lock:
ffff89f28433b0b0 (&rxq->lock){+.-.}-{2:2}, at: iwl_pcie_rx_handle+0x7f/0x960 [iwlwifi]
but this lock took another, SOFTIRQ-unsafe lock in the past:
(napi_hash_lock){+.+.}-{2:2}
and interrupts could create inverse lock ordering between them.
other info that might help us debug this:
Possible interrupt unsafe locking scenario:
CPU0 CPU1
---- ----
lock(napi_hash_lock);
local_irq_disable();
lock(&rxq->lock);
lock(napi_hash_lock);
<Interrupt>
lock(&rxq->lock);
*** DEADLOCK ***
1 lock held by irq/136-iwlwifi/565:
#0: ffff89f2b1440170 (sync_cmd_lockdep_map){+.+.}-{0:0}, at: iwl_pcie_irq_handler+0x5/0xb30
the shortest dependencies between 2nd lock and 1st lock:
-> (napi_hash_lock){+.+.}-{2:2} {
HARDIRQ-ON-W at:
lock_acquire+0x277/0x3d0
_raw_spin_lock+0x2c/0x40
netif_napi_add+0x14b/0x270
e1000_probe+0x2fe/0xee0 [e1000e]
local_pci_probe+0x42/0x90
pci_device_probe+0x10b/0x1c0
really_probe+0xef/0x4b0
driver_probe_device+0xde/0x150
device_driver_attach+0x4f/0x60
__driver_attach+0x9c/0x140
bus_for_each_dev+0x79/0xc0
bus_add_driver+0x18d/0x220
driver_register+0x5b/0xf0
do_one_initcall+0x5b/0x300
do_init_module+0x5b/0x21c
load_module+0x1dae/0x22c0
__do_sys_finit_module+0xad/0x110
do_syscall_64+0x33/0x80
entry_SYSCALL_64_after_hwframe+0x44/0xae
SOFTIRQ-ON-W at:
lock_acquire+0x277/0x3d0
_raw_spin_lock+0x2c/0x40
netif_napi_add+0x14b/0x270
e1000_probe+0x2fe/0xee0 [e1000e]
local_pci_probe+0x42/0x90
pci_device_probe+0x10b/0x1c0
really_probe+0xef/0x4b0
driver_probe_device+0xde/0x150
device_driver_attach+0x4f/0x60
__driver_attach+0x9c/0x140
bus_for_each_dev+0x79/0xc0
bus_add_driver+0x18d/0x220
driver_register+0x5b/0xf0
do_one_initcall+0x5b/0x300
do_init_module+0x5b/0x21c
load_module+0x1dae/0x22c0
__do_sys_finit_module+0xad/0x110
do_syscall_64+0x33/0x80
entry_SYSCALL_64_after_hwframe+0x44/0xae
INITIAL USE at:
lock_acquire+0x277/0x3d0
_raw_spin_lock+0x2c/0x40
netif_napi_add+0x14b/0x270
e1000_probe+0x2fe/0xee0 [e1000e]
local_pci_probe+0x42/0x90
pci_device_probe+0x10b/0x1c0
really_probe+0xef/0x4b0
driver_probe_device+0xde/0x150
device_driver_attach+0x4f/0x60
__driver_attach+0x9c/0x140
bus_for_each_dev+0x79/0xc0
bus_add_driver+0x18d/0x220
driver_register+0x5b/0xf0
do_one_initcall+0x5b/0x300
do_init_module+0x5b/0x21c
load_module+0x1dae/0x22c0
__do_sys_finit_module+0xad/0x110
do_syscall_64+0x33/0x80
entry_SYSCALL_64_after_hwframe+0x44/0xae
}
... key at: [<ffffffffae84ef38>] napi_hash_lock+0x18/0x40
... acquired at:
_raw_spin_lock+0x2c/0x40
netif_napi_add+0x14b/0x270
_iwl_pcie_rx_init+0x1f4/0x710 [iwlwifi]
iwl_pcie_rx_init+0x1b/0x3b0 [iwlwifi]
iwl_trans_pcie_start_fw+0x2ac/0x6a0 [iwlwifi]
iwl_mvm_load_ucode_wait_alive+0x116/0x460 [iwlmvm]
iwl_run_init_mvm_ucode+0xa4/0x3a0 [iwlmvm]
iwl_op_mode_mvm_start+0x9ed/0xbf0 [iwlmvm]
_iwl_op_mode_start.isra.4+0x42/0x80 [iwlwifi]
iwl_opmode_register+0x71/0xe0 [iwlwifi]
iwl_mvm_init+0x34/0x1000 [iwlmvm]
do_one_initcall+0x5b/0x300
do_init_module+0x5b/0x21c
load_module+0x1dae/0x22c0
__do_sys_finit_module+0xad/0x110
do_syscall_64+0x33/0x80
entry_SYSCALL_64_after_hwframe+0x44/0xae
[ ... lockdep output trimmed .... ]
Fixes: 25edc8f259c7106 ("iwlwifi: pcie: properly implement NAPI")
Signed-off-by: Jiri Kosina <jkosina@suse.cz>
---
v1->v2: Previous patch was not refreshed against current code-base, sorry.
drivers/net/wireless/intel/iwlwifi/pcie/rx.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/net/wireless/intel/iwlwifi/pcie/rx.c b/drivers/net/wireless/intel/iwlwifi/pcie/rx.c
index 42426e25cac6..2bec97133119 100644
--- a/drivers/net/wireless/intel/iwlwifi/pcie/rx.c
+++ b/drivers/net/wireless/intel/iwlwifi/pcie/rx.c
@@ -1129,6 +1129,8 @@ static int _iwl_pcie_rx_init(struct iwl_trans *trans)
iwl_pcie_rx_init_rxb_lists(rxq);
+ spin_unlock_bh(&rxq->lock);
+
if (!rxq->napi.poll) {
int (*poll)(struct napi_struct *, int) = iwl_pcie_napi_poll;
@@ -1149,7 +1151,6 @@ static int _iwl_pcie_rx_init(struct iwl_trans *trans)
napi_enable(&rxq->napi);
}
- spin_unlock_bh(&rxq->lock);
}
/* move the pool to the default queue and allocator ownerships */
--
Jiri Kosina
SUSE Labs
^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [PATCH] iwlwifi: don't call netif_napi_add() with rxq->lock held (was Re: Lockdep warning in iwl_pcie_rx_handle())
2021-03-02 9:27 ` [PATCH] iwlwifi: don't call netif_napi_add() with rxq->lock held (was Re: Lockdep warning in iwl_pcie_rx_handle()) Jiri Kosina
2021-03-02 10:34 ` [PATCH v2] " Jiri Kosina
@ 2021-03-02 10:48 ` Coelho, Luciano
1 sibling, 0 replies; 14+ messages in thread
From: Coelho, Luciano @ 2021-03-02 10:48 UTC (permalink / raw)
To: jikos, johannes, kvalo; +Cc: linux-wireless, hkallweit1, linux-kernel
On Tue, 2021-03-02 at 10:27 +0100, Jiri Kosina wrote:
> On Mon, 1 Mar 2021, Johannes Berg wrote:
>
> > > I am getting the splat below with Linus' tree as of today (5.11-rc1,
> > > fe07bfda2fb). I haven't started to look into the code yet, but apparently
> > > this has been already reported by Heiner here:
> > >
> > > https://www.spinics.net/lists/linux-wireless/msg208353.html
> > >
> > > so before I start digging deep into it (the previous kernel this
> > > particular machine had is 5.9, so I'd rather avoid lenghty bisect for now
> > > in case someone has already looked into it and has ideas where the problem
> > > is), I thought I'd ask whether this has been root-caused elsewhere
> > > already.
> >
> > Yeah, I'm pretty sure we have a fix for this, though I'm not sure right
> > now where it is in the pipeline.
> >
> > It's called "iwlwifi: pcie: don't add NAPI under rxq->lock" but right
> > now I can't find it in any of the public archives.
>
> I was not able to find that patch anywhere indeed, but in the meantime I
> fixed it by the patch below. Please consider merging either of the fixes.
I checked my queue and I realized that the patch Johannes mentioned was
marked as a fix for a fix and my script had issues with recursive
fixes. It normally squashes fixes to patches before the latter are
sent out. My bad.
Kalle, please take Jiri's v2 to wireless-drivers.git. I'll give my ack
separately in reply to v2.
--
Cheers,
Luca.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v2] iwlwifi: don't call netif_napi_add() with rxq->lock held (was Re: Lockdep warning in iwl_pcie_rx_handle())
2021-03-02 10:34 ` [PATCH v2] " Jiri Kosina
@ 2021-03-02 10:49 ` Coelho, Luciano
2021-03-02 12:17 ` Kalle Valo
2021-03-03 15:59 ` Kalle Valo
[not found] ` <20210303155941.25521C43463@smtp.codeaurora.org>
2 siblings, 1 reply; 14+ messages in thread
From: Coelho, Luciano @ 2021-03-02 10:49 UTC (permalink / raw)
To: jikos, johannes, kvalo; +Cc: linux-wireless, hkallweit1, linux-kernel
On Tue, 2021-03-02 at 11:34 +0100, Jiri Kosina wrote:
> From: Jiri Kosina <jkosina@suse.cz>
>
> We can't call netif_napi_add() with rxq-lock held, as there is a potential
> for deadlock as spotted by lockdep (see below). rxq->lock is not
> protecting anything over the netif_napi_add() codepath anyway, so let's
> drop it just before calling into NAPI.
>
> ========================================================
> WARNING: possible irq lock inversion dependency detected
> 5.12.0-rc1-00002-gbada49429032 #5 Not tainted
> --------------------------------------------------------
> irq/136-iwlwifi/565 just changed the state of lock:
> ffff89f28433b0b0 (&rxq->lock){+.-.}-{2:2}, at: iwl_pcie_rx_handle+0x7f/0x960 [iwlwifi]
> but this lock took another, SOFTIRQ-unsafe lock in the past:
> (napi_hash_lock){+.+.}-{2:2}
>
> and interrupts could create inverse lock ordering between them.
>
> other info that might help us debug this:
> Possible interrupt unsafe locking scenario:
>
> CPU0 CPU1
> ---- ----
> lock(napi_hash_lock);
> local_irq_disable();
> lock(&rxq->lock);
> lock(napi_hash_lock);
> <Interrupt>
> lock(&rxq->lock);
>
> *** DEADLOCK ***
>
> 1 lock held by irq/136-iwlwifi/565:
> #0: ffff89f2b1440170 (sync_cmd_lockdep_map){+.+.}-{0:0}, at: iwl_pcie_irq_handler+0x5/0xb30
>
> the shortest dependencies between 2nd lock and 1st lock:
> -> (napi_hash_lock){+.+.}-{2:2} {
> HARDIRQ-ON-W at:
> lock_acquire+0x277/0x3d0
> _raw_spin_lock+0x2c/0x40
> netif_napi_add+0x14b/0x270
> e1000_probe+0x2fe/0xee0 [e1000e]
> local_pci_probe+0x42/0x90
> pci_device_probe+0x10b/0x1c0
> really_probe+0xef/0x4b0
> driver_probe_device+0xde/0x150
> device_driver_attach+0x4f/0x60
> __driver_attach+0x9c/0x140
> bus_for_each_dev+0x79/0xc0
> bus_add_driver+0x18d/0x220
> driver_register+0x5b/0xf0
> do_one_initcall+0x5b/0x300
> do_init_module+0x5b/0x21c
> load_module+0x1dae/0x22c0
> __do_sys_finit_module+0xad/0x110
> do_syscall_64+0x33/0x80
> entry_SYSCALL_64_after_hwframe+0x44/0xae
> SOFTIRQ-ON-W at:
> lock_acquire+0x277/0x3d0
> _raw_spin_lock+0x2c/0x40
> netif_napi_add+0x14b/0x270
> e1000_probe+0x2fe/0xee0 [e1000e]
> local_pci_probe+0x42/0x90
> pci_device_probe+0x10b/0x1c0
> really_probe+0xef/0x4b0
> driver_probe_device+0xde/0x150
> device_driver_attach+0x4f/0x60
> __driver_attach+0x9c/0x140
> bus_for_each_dev+0x79/0xc0
> bus_add_driver+0x18d/0x220
> driver_register+0x5b/0xf0
> do_one_initcall+0x5b/0x300
> do_init_module+0x5b/0x21c
> load_module+0x1dae/0x22c0
> __do_sys_finit_module+0xad/0x110
> do_syscall_64+0x33/0x80
> entry_SYSCALL_64_after_hwframe+0x44/0xae
> INITIAL USE at:
> lock_acquire+0x277/0x3d0
> _raw_spin_lock+0x2c/0x40
> netif_napi_add+0x14b/0x270
> e1000_probe+0x2fe/0xee0 [e1000e]
> local_pci_probe+0x42/0x90
> pci_device_probe+0x10b/0x1c0
> really_probe+0xef/0x4b0
> driver_probe_device+0xde/0x150
> device_driver_attach+0x4f/0x60
> __driver_attach+0x9c/0x140
> bus_for_each_dev+0x79/0xc0
> bus_add_driver+0x18d/0x220
> driver_register+0x5b/0xf0
> do_one_initcall+0x5b/0x300
> do_init_module+0x5b/0x21c
> load_module+0x1dae/0x22c0
> __do_sys_finit_module+0xad/0x110
> do_syscall_64+0x33/0x80
> entry_SYSCALL_64_after_hwframe+0x44/0xae
> }
> ... key at: [<ffffffffae84ef38>] napi_hash_lock+0x18/0x40
> ... acquired at:
> _raw_spin_lock+0x2c/0x40
> netif_napi_add+0x14b/0x270
> _iwl_pcie_rx_init+0x1f4/0x710 [iwlwifi]
> iwl_pcie_rx_init+0x1b/0x3b0 [iwlwifi]
> iwl_trans_pcie_start_fw+0x2ac/0x6a0 [iwlwifi]
> iwl_mvm_load_ucode_wait_alive+0x116/0x460 [iwlmvm]
> iwl_run_init_mvm_ucode+0xa4/0x3a0 [iwlmvm]
> iwl_op_mode_mvm_start+0x9ed/0xbf0 [iwlmvm]
> _iwl_op_mode_start.isra.4+0x42/0x80 [iwlwifi]
> iwl_opmode_register+0x71/0xe0 [iwlwifi]
> iwl_mvm_init+0x34/0x1000 [iwlmvm]
> do_one_initcall+0x5b/0x300
> do_init_module+0x5b/0x21c
> load_module+0x1dae/0x22c0
> __do_sys_finit_module+0xad/0x110
> do_syscall_64+0x33/0x80
> entry_SYSCALL_64_after_hwframe+0x44/0xae
>
> [ ... lockdep output trimmed .... ]
>
> Fixes: 25edc8f259c7106 ("iwlwifi: pcie: properly implement NAPI")
> Signed-off-by: Jiri Kosina <jkosina@suse.cz>
> ---
>
> v1->v2: Previous patch was not refreshed against current code-base, sorry.
>
> drivers/net/wireless/intel/iwlwifi/pcie/rx.c | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
Thanks, Jiri! Let's take your patch since you already sent it out.
Kalle, can you please take this directly to wireless-drivers.git?
Acked-by: Luca Coelho <luciano.coelho@intel.com>
--
Cheers,
Luca.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v2] iwlwifi: don't call netif_napi_add() with rxq->lock held (was Re: Lockdep warning in iwl_pcie_rx_handle())
2021-03-02 10:49 ` Coelho, Luciano
@ 2021-03-02 12:17 ` Kalle Valo
2021-03-02 14:55 ` Jiri Kosina
2021-03-03 21:39 ` Sedat Dilek
0 siblings, 2 replies; 14+ messages in thread
From: Kalle Valo @ 2021-03-02 12:17 UTC (permalink / raw)
To: Coelho, Luciano; +Cc: jikos, johannes, linux-wireless, hkallweit1, linux-kernel
"Coelho, Luciano" <luciano.coelho@intel.com> writes:
> On Tue, 2021-03-02 at 11:34 +0100, Jiri Kosina wrote:
>> From: Jiri Kosina <jkosina@suse.cz>
>>
>> We can't call netif_napi_add() with rxq-lock held, as there is a potential
>> for deadlock as spotted by lockdep (see below). rxq->lock is not
>> protecting anything over the netif_napi_add() codepath anyway, so let's
>> drop it just before calling into NAPI.
>>
>> ========================================================
>> WARNING: possible irq lock inversion dependency detected
>> 5.12.0-rc1-00002-gbada49429032 #5 Not tainted
>> --------------------------------------------------------
>> irq/136-iwlwifi/565 just changed the state of lock:
>> ffff89f28433b0b0 (&rxq->lock){+.-.}-{2:2}, at:
>> iwl_pcie_rx_handle+0x7f/0x960 [iwlwifi]
>> but this lock took another, SOFTIRQ-unsafe lock in the past:
>> (napi_hash_lock){+.+.}-{2:2}
>>
>> and interrupts could create inverse lock ordering between them.
>>
>> other info that might help us debug this:
>> Possible interrupt unsafe locking scenario:
>>
>> CPU0 CPU1
>> ---- ----
>> lock(napi_hash_lock);
>> local_irq_disable();
>> lock(&rxq->lock);
>> lock(napi_hash_lock);
>> <Interrupt>
>> lock(&rxq->lock);
>>
>> *** DEADLOCK ***
>>
>> 1 lock held by irq/136-iwlwifi/565:
>> #0: ffff89f2b1440170 (sync_cmd_lockdep_map){+.+.}-{0:0}, at:
>> iwl_pcie_irq_handler+0x5/0xb30
>>
>> the shortest dependencies between 2nd lock and 1st lock:
>> -> (napi_hash_lock){+.+.}-{2:2} {
>> HARDIRQ-ON-W at:
>> lock_acquire+0x277/0x3d0
>> _raw_spin_lock+0x2c/0x40
>> netif_napi_add+0x14b/0x270
>> e1000_probe+0x2fe/0xee0 [e1000e]
>> local_pci_probe+0x42/0x90
>> pci_device_probe+0x10b/0x1c0
>> really_probe+0xef/0x4b0
>> driver_probe_device+0xde/0x150
>> device_driver_attach+0x4f/0x60
>> __driver_attach+0x9c/0x140
>> bus_for_each_dev+0x79/0xc0
>> bus_add_driver+0x18d/0x220
>> driver_register+0x5b/0xf0
>> do_one_initcall+0x5b/0x300
>> do_init_module+0x5b/0x21c
>> load_module+0x1dae/0x22c0
>> __do_sys_finit_module+0xad/0x110
>> do_syscall_64+0x33/0x80
>> entry_SYSCALL_64_after_hwframe+0x44/0xae
>> SOFTIRQ-ON-W at:
>> lock_acquire+0x277/0x3d0
>> _raw_spin_lock+0x2c/0x40
>> netif_napi_add+0x14b/0x270
>> e1000_probe+0x2fe/0xee0 [e1000e]
>> local_pci_probe+0x42/0x90
>> pci_device_probe+0x10b/0x1c0
>> really_probe+0xef/0x4b0
>> driver_probe_device+0xde/0x150
>> device_driver_attach+0x4f/0x60
>> __driver_attach+0x9c/0x140
>> bus_for_each_dev+0x79/0xc0
>> bus_add_driver+0x18d/0x220
>> driver_register+0x5b/0xf0
>> do_one_initcall+0x5b/0x300
>> do_init_module+0x5b/0x21c
>> load_module+0x1dae/0x22c0
>> __do_sys_finit_module+0xad/0x110
>> do_syscall_64+0x33/0x80
>> entry_SYSCALL_64_after_hwframe+0x44/0xae
>> INITIAL USE at:
>> lock_acquire+0x277/0x3d0
>> _raw_spin_lock+0x2c/0x40
>> netif_napi_add+0x14b/0x270
>> e1000_probe+0x2fe/0xee0 [e1000e]
>> local_pci_probe+0x42/0x90
>> pci_device_probe+0x10b/0x1c0
>> really_probe+0xef/0x4b0
>> driver_probe_device+0xde/0x150
>> device_driver_attach+0x4f/0x60
>> __driver_attach+0x9c/0x140
>> bus_for_each_dev+0x79/0xc0
>> bus_add_driver+0x18d/0x220
>> driver_register+0x5b/0xf0
>> do_one_initcall+0x5b/0x300
>> do_init_module+0x5b/0x21c
>> load_module+0x1dae/0x22c0
>> __do_sys_finit_module+0xad/0x110
>> do_syscall_64+0x33/0x80
>> entry_SYSCALL_64_after_hwframe+0x44/0xae
>> }
>> ... key at: [<ffffffffae84ef38>] napi_hash_lock+0x18/0x40
>> ... acquired at:
>> _raw_spin_lock+0x2c/0x40
>> netif_napi_add+0x14b/0x270
>> _iwl_pcie_rx_init+0x1f4/0x710 [iwlwifi]
>> iwl_pcie_rx_init+0x1b/0x3b0 [iwlwifi]
>> iwl_trans_pcie_start_fw+0x2ac/0x6a0 [iwlwifi]
>> iwl_mvm_load_ucode_wait_alive+0x116/0x460 [iwlmvm]
>> iwl_run_init_mvm_ucode+0xa4/0x3a0 [iwlmvm]
>> iwl_op_mode_mvm_start+0x9ed/0xbf0 [iwlmvm]
>> _iwl_op_mode_start.isra.4+0x42/0x80 [iwlwifi]
>> iwl_opmode_register+0x71/0xe0 [iwlwifi]
>> iwl_mvm_init+0x34/0x1000 [iwlmvm]
>> do_one_initcall+0x5b/0x300
>> do_init_module+0x5b/0x21c
>> load_module+0x1dae/0x22c0
>> __do_sys_finit_module+0xad/0x110
>> do_syscall_64+0x33/0x80
>> entry_SYSCALL_64_after_hwframe+0x44/0xae
>>
>> [ ... lockdep output trimmed .... ]
>>
>> Fixes: 25edc8f259c7106 ("iwlwifi: pcie: properly implement NAPI")
>> Signed-off-by: Jiri Kosina <jkosina@suse.cz>
>> ---
>>
>> v1->v2: Previous patch was not refreshed against current code-base, sorry.
>>
>> drivers/net/wireless/intel/iwlwifi/pcie/rx.c | 3 ++-
>> 1 file changed, 2 insertions(+), 1 deletion(-)
>
>
> Thanks, Jiri! Let's take your patch since you already sent it out.
>
> Kalle, can you please take this directly to wireless-drivers.git?
>
> Acked-by: Luca Coelho <luciano.coelho@intel.com>
Ok but I don't see this either in patchwork or lore, hopefully it shows
up later.
--
https://patchwork.kernel.org/project/linux-wireless/list/
https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v2] iwlwifi: don't call netif_napi_add() with rxq->lock held (was Re: Lockdep warning in iwl_pcie_rx_handle())
2021-03-02 12:17 ` Kalle Valo
@ 2021-03-02 14:55 ` Jiri Kosina
2021-03-03 21:39 ` Sedat Dilek
1 sibling, 0 replies; 14+ messages in thread
From: Jiri Kosina @ 2021-03-02 14:55 UTC (permalink / raw)
To: Kalle Valo
Cc: Coelho, Luciano, johannes, linux-wireless, hkallweit1, linux-kernel
On Tue, 2 Mar 2021, Kalle Valo wrote:
> > Thanks, Jiri! Let's take your patch since you already sent it out.
> >
> > Kalle, can you please take this directly to wireless-drivers.git?
> >
> > Acked-by: Luca Coelho <luciano.coelho@intel.com>
>
> Ok but I don't see this either in patchwork or lore, hopefully it shows
> up later.
Not sure about patchwork, but vger had hiccup (again) earlier today,
everything depending on the ML traffic is probably slower.
lore has it now though:
https://lore.kernel.org/lkml/nycvar.YFH.7.76.2103021134060.12405@cbobk.fhfr.pm/
Thanks,
--
Jiri Kosina
SUSE Labs
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v2] iwlwifi: don't call netif_napi_add() with rxq->lock held (was Re: Lockdep warning in iwl_pcie_rx_handle())
2021-03-02 10:34 ` [PATCH v2] " Jiri Kosina
2021-03-02 10:49 ` Coelho, Luciano
@ 2021-03-03 15:59 ` Kalle Valo
[not found] ` <20210303155941.25521C43463@smtp.codeaurora.org>
2 siblings, 0 replies; 14+ messages in thread
From: Kalle Valo @ 2021-03-03 15:59 UTC (permalink / raw)
To: Jiri Kosina
Cc: Johannes Berg, Luca Coelho, Heiner Kallweit, linux-wireless,
linux-kernel
Jiri Kosina <jikos@kernel.org> wrote:
> From: Jiri Kosina <jkosina@suse.cz>
>
> We can't call netif_napi_add() with rxq-lock held, as there is a potential
> for deadlock as spotted by lockdep (see below). rxq->lock is not
> protecting anything over the netif_napi_add() codepath anyway, so let's
> drop it just before calling into NAPI.
>
> ========================================================
> WARNING: possible irq lock inversion dependency detected
> 5.12.0-rc1-00002-gbada49429032 #5 Not tainted
> --------------------------------------------------------
> irq/136-iwlwifi/565 just changed the state of lock:
> ffff89f28433b0b0 (&rxq->lock){+.-.}-{2:2}, at: iwl_pcie_rx_handle+0x7f/0x960 [iwlwifi]
> but this lock took another, SOFTIRQ-unsafe lock in the past:
> (napi_hash_lock){+.+.}-{2:2}
>
> and interrupts could create inverse lock ordering between them.
>
> other info that might help us debug this:
> Possible interrupt unsafe locking scenario:
>
> CPU0 CPU1
> ---- ----
> lock(napi_hash_lock);
> local_irq_disable();
> lock(&rxq->lock);
> lock(napi_hash_lock);
> <Interrupt>
> lock(&rxq->lock);
>
> *** DEADLOCK ***
>
> 1 lock held by irq/136-iwlwifi/565:
> #0: ffff89f2b1440170 (sync_cmd_lockdep_map){+.+.}-{0:0}, at: iwl_pcie_irq_handler+0x5/0xb30
>
> the shortest dependencies between 2nd lock and 1st lock:
> -> (napi_hash_lock){+.+.}-{2:2} {
> HARDIRQ-ON-W at:
> lock_acquire+0x277/0x3d0
> _raw_spin_lock+0x2c/0x40
> netif_napi_add+0x14b/0x270
> e1000_probe+0x2fe/0xee0 [e1000e]
> local_pci_probe+0x42/0x90
> pci_device_probe+0x10b/0x1c0
> really_probe+0xef/0x4b0
> driver_probe_device+0xde/0x150
> device_driver_attach+0x4f/0x60
> __driver_attach+0x9c/0x140
> bus_for_each_dev+0x79/0xc0
> bus_add_driver+0x18d/0x220
> driver_register+0x5b/0xf0
> do_one_initcall+0x5b/0x300
> do_init_module+0x5b/0x21c
> load_module+0x1dae/0x22c0
> __do_sys_finit_module+0xad/0x110
> do_syscall_64+0x33/0x80
> entry_SYSCALL_64_after_hwframe+0x44/0xae
> SOFTIRQ-ON-W at:
> lock_acquire+0x277/0x3d0
> _raw_spin_lock+0x2c/0x40
> netif_napi_add+0x14b/0x270
> e1000_probe+0x2fe/0xee0 [e1000e]
> local_pci_probe+0x42/0x90
> pci_device_probe+0x10b/0x1c0
> really_probe+0xef/0x4b0
> driver_probe_device+0xde/0x150
> device_driver_attach+0x4f/0x60
> __driver_attach+0x9c/0x140
> bus_for_each_dev+0x79/0xc0
> bus_add_driver+0x18d/0x220
> driver_register+0x5b/0xf0
> do_one_initcall+0x5b/0x300
> do_init_module+0x5b/0x21c
> load_module+0x1dae/0x22c0
> __do_sys_finit_module+0xad/0x110
> do_syscall_64+0x33/0x80
> entry_SYSCALL_64_after_hwframe+0x44/0xae
> INITIAL USE at:
> lock_acquire+0x277/0x3d0
> _raw_spin_lock+0x2c/0x40
> netif_napi_add+0x14b/0x270
> e1000_probe+0x2fe/0xee0 [e1000e]
> local_pci_probe+0x42/0x90
> pci_device_probe+0x10b/0x1c0
> really_probe+0xef/0x4b0
> driver_probe_device+0xde/0x150
> device_driver_attach+0x4f/0x60
> __driver_attach+0x9c/0x140
> bus_for_each_dev+0x79/0xc0
> bus_add_driver+0x18d/0x220
> driver_register+0x5b/0xf0
> do_one_initcall+0x5b/0x300
> do_init_module+0x5b/0x21c
> load_module+0x1dae/0x22c0
> __do_sys_finit_module+0xad/0x110
> do_syscall_64+0x33/0x80
> entry_SYSCALL_64_after_hwframe+0x44/0xae
> }
> ... key at: [<ffffffffae84ef38>] napi_hash_lock+0x18/0x40
> ... acquired at:
> _raw_spin_lock+0x2c/0x40
> netif_napi_add+0x14b/0x270
> _iwl_pcie_rx_init+0x1f4/0x710 [iwlwifi]
> iwl_pcie_rx_init+0x1b/0x3b0 [iwlwifi]
> iwl_trans_pcie_start_fw+0x2ac/0x6a0 [iwlwifi]
> iwl_mvm_load_ucode_wait_alive+0x116/0x460 [iwlmvm]
> iwl_run_init_mvm_ucode+0xa4/0x3a0 [iwlmvm]
> iwl_op_mode_mvm_start+0x9ed/0xbf0 [iwlmvm]
> _iwl_op_mode_start.isra.4+0x42/0x80 [iwlwifi]
> iwl_opmode_register+0x71/0xe0 [iwlwifi]
> iwl_mvm_init+0x34/0x1000 [iwlmvm]
> do_one_initcall+0x5b/0x300
> do_init_module+0x5b/0x21c
> load_module+0x1dae/0x22c0
> __do_sys_finit_module+0xad/0x110
> do_syscall_64+0x33/0x80
> entry_SYSCALL_64_after_hwframe+0x44/0xae
>
> [ ... lockdep output trimmed .... ]
>
> Fixes: 25edc8f259c7106 ("iwlwifi: pcie: properly implement NAPI")
> Signed-off-by: Jiri Kosina <jkosina@suse.cz>
> Acked-by: Luca Coelho <luciano.coelho@intel.com>
Patch applied to wireless-drivers.git, thanks.
295d4cd82b01 iwlwifi: don't call netif_napi_add() with rxq->lock held (was Re: Lockdep warning in iwl_pcie_rx_handle())
--
https://patchwork.kernel.org/project/linux-wireless/patch/nycvar.YFH.7.76.2103021134060.12405@cbobk.fhfr.pm/
https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v2] iwlwifi: don't call netif_napi_add() with rxq->lock held (was Re: Lockdep warning in iwl_pcie_rx_handle())
[not found] ` <20210303155941.25521C43463@smtp.codeaurora.org>
@ 2021-03-03 18:01 ` Jiri Kosina
2021-03-03 18:33 ` Kalle Valo
0 siblings, 1 reply; 14+ messages in thread
From: Jiri Kosina @ 2021-03-03 18:01 UTC (permalink / raw)
To: Kalle Valo
Cc: Johannes Berg, Luca Coelho, Heiner Kallweit, linux-wireless,
linux-kernel
On Wed, 3 Mar 2021, Kalle Valo wrote:
> Patch applied to wireless-drivers.git, thanks.
Thanks, but ...
> 295d4cd82b01 iwlwifi: don't call netif_napi_add() with rxq->lock held (was Re: Lockdep warning in iwl_pcie_rx_handle())
... i believe you want to drop the "(was ...") part from the patch
subject.
--
Jiri Kosina
SUSE Labs
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v2] iwlwifi: don't call netif_napi_add() with rxq->lock held (was Re: Lockdep warning in iwl_pcie_rx_handle())
2021-03-03 18:01 ` Jiri Kosina
@ 2021-03-03 18:33 ` Kalle Valo
2021-03-03 19:19 ` Jiri Kosina
0 siblings, 1 reply; 14+ messages in thread
From: Kalle Valo @ 2021-03-03 18:33 UTC (permalink / raw)
To: Jiri Kosina
Cc: Johannes Berg, Luca Coelho, Heiner Kallweit, linux-wireless,
linux-kernel
Jiri Kosina <jikos@kernel.org> writes:
> On Wed, 3 Mar 2021, Kalle Valo wrote:
>
>> Patch applied to wireless-drivers.git, thanks.
>
> Thanks, but ...
>
>> 295d4cd82b01 iwlwifi: don't call netif_napi_add() with rxq->lock
>> held (was Re: Lockdep warning in iwl_pcie_rx_handle())
>
> ... i believe you want to drop the "(was ...") part from the patch
> subject.
Too late now, it's already applied and pull request sent. Why was it
there in the first place?
--
https://patchwork.kernel.org/project/linux-wireless/list/
https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v2] iwlwifi: don't call netif_napi_add() with rxq->lock held (was Re: Lockdep warning in iwl_pcie_rx_handle())
2021-03-03 18:33 ` Kalle Valo
@ 2021-03-03 19:19 ` Jiri Kosina
0 siblings, 0 replies; 14+ messages in thread
From: Jiri Kosina @ 2021-03-03 19:19 UTC (permalink / raw)
To: Kalle Valo
Cc: Johannes Berg, Luca Coelho, Heiner Kallweit, linux-wireless,
linux-kernel
On Wed, 3 Mar 2021, Kalle Valo wrote:
> > ... i believe you want to drop the "(was ...") part from the patch
> > subject.
>
> Too late now, it's already applied and pull request sent. Why was it
> there in the first place?
Yeah, it was, but I don't think it's a big issue :) So let it be.
BTW, how about the other fix I sent? It's also fixing a real functional
issue, so it IMHO is a -rc material
https://lore.kernel.org/linux-wireless/nycvar.YFH.7.76.2103021125430.12405@cbobk.fhfr.pm/
Thanks,
--
Jiri Kosina
SUSE Labs
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v2] iwlwifi: don't call netif_napi_add() with rxq->lock held (was Re: Lockdep warning in iwl_pcie_rx_handle())
2021-03-02 12:17 ` Kalle Valo
2021-03-02 14:55 ` Jiri Kosina
@ 2021-03-03 21:39 ` Sedat Dilek
1 sibling, 0 replies; 14+ messages in thread
From: Sedat Dilek @ 2021-03-03 21:39 UTC (permalink / raw)
To: Kalle Valo
Cc: Coelho, Luciano, jikos, johannes, linux-wireless, hkallweit1,
linux-kernel
On Wed, Mar 3, 2021 at 1:38 AM Kalle Valo <kvalo@codeaurora.org> wrote:
>
> "Coelho, Luciano" <luciano.coelho@intel.com> writes:
>
> > On Tue, 2021-03-02 at 11:34 +0100, Jiri Kosina wrote:
> >> From: Jiri Kosina <jkosina@suse.cz>
> >>
> >> We can't call netif_napi_add() with rxq-lock held, as there is a potential
> >> for deadlock as spotted by lockdep (see below). rxq->lock is not
> >> protecting anything over the netif_napi_add() codepath anyway, so let's
> >> drop it just before calling into NAPI.
> >>
> >> ========================================================
> >> WARNING: possible irq lock inversion dependency detected
> >> 5.12.0-rc1-00002-gbada49429032 #5 Not tainted
> >> --------------------------------------------------------
> >> irq/136-iwlwifi/565 just changed the state of lock:
> >> ffff89f28433b0b0 (&rxq->lock){+.-.}-{2:2}, at:
> >> iwl_pcie_rx_handle+0x7f/0x960 [iwlwifi]
> >> but this lock took another, SOFTIRQ-unsafe lock in the past:
> >> (napi_hash_lock){+.+.}-{2:2}
> >>
> >> and interrupts could create inverse lock ordering between them.
> >>
> >> other info that might help us debug this:
> >> Possible interrupt unsafe locking scenario:
> >>
> >> CPU0 CPU1
> >> ---- ----
> >> lock(napi_hash_lock);
> >> local_irq_disable();
> >> lock(&rxq->lock);
> >> lock(napi_hash_lock);
> >> <Interrupt>
> >> lock(&rxq->lock);
> >>
> >> *** DEADLOCK ***
> >>
> >> 1 lock held by irq/136-iwlwifi/565:
> >> #0: ffff89f2b1440170 (sync_cmd_lockdep_map){+.+.}-{0:0}, at:
> >> iwl_pcie_irq_handler+0x5/0xb30
> >>
> >> the shortest dependencies between 2nd lock and 1st lock:
> >> -> (napi_hash_lock){+.+.}-{2:2} {
> >> HARDIRQ-ON-W at:
> >> lock_acquire+0x277/0x3d0
> >> _raw_spin_lock+0x2c/0x40
> >> netif_napi_add+0x14b/0x270
> >> e1000_probe+0x2fe/0xee0 [e1000e]
> >> local_pci_probe+0x42/0x90
> >> pci_device_probe+0x10b/0x1c0
> >> really_probe+0xef/0x4b0
> >> driver_probe_device+0xde/0x150
> >> device_driver_attach+0x4f/0x60
> >> __driver_attach+0x9c/0x140
> >> bus_for_each_dev+0x79/0xc0
> >> bus_add_driver+0x18d/0x220
> >> driver_register+0x5b/0xf0
> >> do_one_initcall+0x5b/0x300
> >> do_init_module+0x5b/0x21c
> >> load_module+0x1dae/0x22c0
> >> __do_sys_finit_module+0xad/0x110
> >> do_syscall_64+0x33/0x80
> >> entry_SYSCALL_64_after_hwframe+0x44/0xae
> >> SOFTIRQ-ON-W at:
> >> lock_acquire+0x277/0x3d0
> >> _raw_spin_lock+0x2c/0x40
> >> netif_napi_add+0x14b/0x270
> >> e1000_probe+0x2fe/0xee0 [e1000e]
> >> local_pci_probe+0x42/0x90
> >> pci_device_probe+0x10b/0x1c0
> >> really_probe+0xef/0x4b0
> >> driver_probe_device+0xde/0x150
> >> device_driver_attach+0x4f/0x60
> >> __driver_attach+0x9c/0x140
> >> bus_for_each_dev+0x79/0xc0
> >> bus_add_driver+0x18d/0x220
> >> driver_register+0x5b/0xf0
> >> do_one_initcall+0x5b/0x300
> >> do_init_module+0x5b/0x21c
> >> load_module+0x1dae/0x22c0
> >> __do_sys_finit_module+0xad/0x110
> >> do_syscall_64+0x33/0x80
> >> entry_SYSCALL_64_after_hwframe+0x44/0xae
> >> INITIAL USE at:
> >> lock_acquire+0x277/0x3d0
> >> _raw_spin_lock+0x2c/0x40
> >> netif_napi_add+0x14b/0x270
> >> e1000_probe+0x2fe/0xee0 [e1000e]
> >> local_pci_probe+0x42/0x90
> >> pci_device_probe+0x10b/0x1c0
> >> really_probe+0xef/0x4b0
> >> driver_probe_device+0xde/0x150
> >> device_driver_attach+0x4f/0x60
> >> __driver_attach+0x9c/0x140
> >> bus_for_each_dev+0x79/0xc0
> >> bus_add_driver+0x18d/0x220
> >> driver_register+0x5b/0xf0
> >> do_one_initcall+0x5b/0x300
> >> do_init_module+0x5b/0x21c
> >> load_module+0x1dae/0x22c0
> >> __do_sys_finit_module+0xad/0x110
> >> do_syscall_64+0x33/0x80
> >> entry_SYSCALL_64_after_hwframe+0x44/0xae
> >> }
> >> ... key at: [<ffffffffae84ef38>] napi_hash_lock+0x18/0x40
> >> ... acquired at:
> >> _raw_spin_lock+0x2c/0x40
> >> netif_napi_add+0x14b/0x270
> >> _iwl_pcie_rx_init+0x1f4/0x710 [iwlwifi]
> >> iwl_pcie_rx_init+0x1b/0x3b0 [iwlwifi]
> >> iwl_trans_pcie_start_fw+0x2ac/0x6a0 [iwlwifi]
> >> iwl_mvm_load_ucode_wait_alive+0x116/0x460 [iwlmvm]
> >> iwl_run_init_mvm_ucode+0xa4/0x3a0 [iwlmvm]
> >> iwl_op_mode_mvm_start+0x9ed/0xbf0 [iwlmvm]
> >> _iwl_op_mode_start.isra.4+0x42/0x80 [iwlwifi]
> >> iwl_opmode_register+0x71/0xe0 [iwlwifi]
> >> iwl_mvm_init+0x34/0x1000 [iwlmvm]
> >> do_one_initcall+0x5b/0x300
> >> do_init_module+0x5b/0x21c
> >> load_module+0x1dae/0x22c0
> >> __do_sys_finit_module+0xad/0x110
> >> do_syscall_64+0x33/0x80
> >> entry_SYSCALL_64_after_hwframe+0x44/0xae
> >>
> >> [ ... lockdep output trimmed .... ]
> >>
> >> Fixes: 25edc8f259c7106 ("iwlwifi: pcie: properly implement NAPI")
> >> Signed-off-by: Jiri Kosina <jkosina@suse.cz>
> >> ---
> >>
> >> v1->v2: Previous patch was not refreshed against current code-base, sorry.
> >>
> >> drivers/net/wireless/intel/iwlwifi/pcie/rx.c | 3 ++-
> >> 1 file changed, 2 insertions(+), 1 deletion(-)
> >
> >
> > Thanks, Jiri! Let's take your patch since you already sent it out.
> >
> > Kalle, can you please take this directly to wireless-drivers.git?
> >
> > Acked-by: Luca Coelho <luciano.coelho@intel.com>
>
> Ok but I don't see this either in patchwork or lore, hopefully it shows
> up later.
>
Is that intended to have a subject like...?
iwlwifi: don't call netif_napi_add() with rxq->lock held (was Re:
Lockdep warning in iwl_pcie_rx_handle())
- Sedat -
[1] https://git.kernel.org/pub/scm/linux/kernel/git/kvalo/wireless-drivers.git/commit/?id=295d4cd82b0181dd36b145fd535c13d623d7a335
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2021-03-04 0:58 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-01 20:44 Lockdep warning in iwl_pcie_rx_handle() Jiri Kosina
2021-03-01 21:09 ` Jiri Kosina
2021-03-01 21:12 ` Johannes Berg
2021-03-02 9:27 ` [PATCH] iwlwifi: don't call netif_napi_add() with rxq->lock held (was Re: Lockdep warning in iwl_pcie_rx_handle()) Jiri Kosina
2021-03-02 10:34 ` [PATCH v2] " Jiri Kosina
2021-03-02 10:49 ` Coelho, Luciano
2021-03-02 12:17 ` Kalle Valo
2021-03-02 14:55 ` Jiri Kosina
2021-03-03 21:39 ` Sedat Dilek
2021-03-03 15:59 ` Kalle Valo
[not found] ` <20210303155941.25521C43463@smtp.codeaurora.org>
2021-03-03 18:01 ` Jiri Kosina
2021-03-03 18:33 ` Kalle Valo
2021-03-03 19:19 ` Jiri Kosina
2021-03-02 10:48 ` [PATCH] " Coelho, Luciano
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.