From: Sasha Levin <sashal@kernel.org>
To: linux-kernel@vger.kernel.org, stable@vger.kernel.org
Cc: Tsuchiya Yuto <kitakar@gmail.com>,
Maximilian Luz <luzmaximilian@gmail.com>,
Kalle Valo <kvalo@codeaurora.org>,
Sasha Levin <sashal@kernel.org>,
linux-wireless@vger.kernel.org, netdev@vger.kernel.org
Subject: [PATCH AUTOSEL 5.10 067/217] mwifiex: pcie: skip cancel_work_sync() on reset failure path
Date: Tue, 22 Dec 2020 21:13:56 -0500 [thread overview]
Message-ID: <20201223021626.2790791-67-sashal@kernel.org> (raw)
In-Reply-To: <20201223021626.2790791-1-sashal@kernel.org>
From: Tsuchiya Yuto <kitakar@gmail.com>
[ Upstream commit 4add4d988f95f47493500a7a19c623827061589b ]
If a reset is performed, but even the reset fails for some reasons (e.g.,
on Surface devices, the fw reset requires another quirks),
cancel_work_sync() hangs in mwifiex_cleanup_pcie().
# firmware went into a bad state
[...]
[ 1608.281690] mwifiex_pcie 0000:03:00.0: info: shutdown mwifiex...
[ 1608.282724] mwifiex_pcie 0000:03:00.0: rx_pending=0, tx_pending=1, cmd_pending=0
[ 1608.292400] mwifiex_pcie 0000:03:00.0: PREP_CMD: card is removed
[ 1608.292405] mwifiex_pcie 0000:03:00.0: PREP_CMD: card is removed
# reset performed after firmware went into a bad state
[ 1609.394320] mwifiex_pcie 0000:03:00.0: WLAN FW already running! Skip FW dnld
[ 1609.394335] mwifiex_pcie 0000:03:00.0: WLAN FW is active
# but even the reset failed
[ 1619.499049] mwifiex_pcie 0000:03:00.0: mwifiex_cmd_timeout_func: Timeout cmd id = 0xfa, act = 0xe000
[ 1619.499094] mwifiex_pcie 0000:03:00.0: num_data_h2c_failure = 0
[ 1619.499103] mwifiex_pcie 0000:03:00.0: num_cmd_h2c_failure = 0
[ 1619.499110] mwifiex_pcie 0000:03:00.0: is_cmd_timedout = 1
[ 1619.499117] mwifiex_pcie 0000:03:00.0: num_tx_timeout = 0
[ 1619.499124] mwifiex_pcie 0000:03:00.0: last_cmd_index = 0
[ 1619.499133] mwifiex_pcie 0000:03:00.0: last_cmd_id: fa 00 07 01 07 01 07 01 07 01
[ 1619.499140] mwifiex_pcie 0000:03:00.0: last_cmd_act: 00 e0 00 00 00 00 00 00 00 00
[ 1619.499147] mwifiex_pcie 0000:03:00.0: last_cmd_resp_index = 3
[ 1619.499155] mwifiex_pcie 0000:03:00.0: last_cmd_resp_id: 07 81 07 81 07 81 07 81 07 81
[ 1619.499162] mwifiex_pcie 0000:03:00.0: last_event_index = 2
[ 1619.499169] mwifiex_pcie 0000:03:00.0: last_event: 58 00 58 00 58 00 58 00 58 00
[ 1619.499177] mwifiex_pcie 0000:03:00.0: data_sent=0 cmd_sent=1
[ 1619.499185] mwifiex_pcie 0000:03:00.0: ps_mode=0 ps_state=0
[ 1619.499215] mwifiex_pcie 0000:03:00.0: info: _mwifiex_fw_dpc: unregister device
# mwifiex_pcie_work hang happening
[ 1823.233923] INFO: task kworker/3:1:44 blocked for more than 122 seconds.
[ 1823.233932] Tainted: G WC OE 5.10.0-rc1-1-mainline #1
[ 1823.233935] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1823.233940] task:kworker/3:1 state:D stack: 0 pid: 44 ppid: 2 flags:0x00004000
[ 1823.233960] Workqueue: events mwifiex_pcie_work [mwifiex_pcie]
[ 1823.233965] Call Trace:
[ 1823.233981] __schedule+0x292/0x820
[ 1823.233990] schedule+0x45/0xe0
[ 1823.233995] schedule_timeout+0x11c/0x160
[ 1823.234003] wait_for_completion+0x9e/0x100
[ 1823.234012] __flush_work.isra.0+0x156/0x210
[ 1823.234018] ? flush_workqueue_prep_pwqs+0x130/0x130
[ 1823.234026] __cancel_work_timer+0x11e/0x1a0
[ 1823.234035] mwifiex_cleanup_pcie+0x28/0xd0 [mwifiex_pcie]
[ 1823.234049] mwifiex_free_adapter+0x24/0xe0 [mwifiex]
[ 1823.234060] _mwifiex_fw_dpc+0x294/0x560 [mwifiex]
[ 1823.234074] mwifiex_reinit_sw+0x15d/0x300 [mwifiex]
[ 1823.234080] mwifiex_pcie_reset_done+0x50/0x80 [mwifiex_pcie]
[ 1823.234087] pci_try_reset_function+0x5c/0x90
[ 1823.234094] process_one_work+0x1d6/0x3a0
[ 1823.234100] worker_thread+0x4d/0x3d0
[ 1823.234107] ? rescuer_thread+0x410/0x410
[ 1823.234112] kthread+0x142/0x160
[ 1823.234117] ? __kthread_bind_mask+0x60/0x60
[ 1823.234124] ret_from_fork+0x22/0x30
[...]
This is a deadlock caused by calling cancel_work_sync() in
mwifiex_cleanup_pcie():
- Device resets are done via mwifiex_pcie_card_reset()
- which schedules card->work to call mwifiex_pcie_card_reset_work()
- which calls pci_try_reset_function().
- This leads to mwifiex_pcie_reset_done() be called on the same workqueue,
which in turn calls
- mwifiex_reinit_sw() and that calls
- _mwifiex_fw_dpc().
The problem is now that _mwifiex_fw_dpc() calls mwifiex_free_adapter()
in case firmware initialization fails. That ends up calling
mwifiex_cleanup_pcie().
Note that all those calls are still running on the workqueue. So when
mwifiex_cleanup_pcie() now calls cancel_work_sync(), it's really waiting
on itself to complete, causing a deadlock.
This commit fixes the deadlock by skipping cancel_work_sync() on a reset
failure path.
After this commit, when reset fails, the following output is
expected to be shown:
kernel: mwifiex_pcie 0000:03:00.0: info: _mwifiex_fw_dpc: unregister device
kernel: mwifiex: Failed to bring up adapter: -5
kernel: mwifiex_pcie 0000:03:00.0: reinit failed: -5
To reproduce this issue, for example, try putting the root port of wifi
into D3 (replace "00:1d.3" with your setup).
# put into D3 (root port)
sudo setpci -v -s 00:1d.3 CAP_PM+4.b=0b
Cc: Maximilian Luz <luzmaximilian@gmail.com>
Signed-off-by: Tsuchiya Yuto <kitakar@gmail.com>
Signed-off-by: Kalle Valo <kvalo@codeaurora.org>
Link: https://lore.kernel.org/r/20201028142346.18355-1-kitakar@gmail.com
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
drivers/net/wireless/marvell/mwifiex/pcie.c | 18 +++++++++++++++++-
drivers/net/wireless/marvell/mwifiex/pcie.h | 2 ++
2 files changed, 19 insertions(+), 1 deletion(-)
diff --git a/drivers/net/wireless/marvell/mwifiex/pcie.c b/drivers/net/wireless/marvell/mwifiex/pcie.c
index 6a10ff0377a24..33cf952cc01d3 100644
--- a/drivers/net/wireless/marvell/mwifiex/pcie.c
+++ b/drivers/net/wireless/marvell/mwifiex/pcie.c
@@ -526,6 +526,8 @@ static void mwifiex_pcie_reset_prepare(struct pci_dev *pdev)
clear_bit(MWIFIEX_IFACE_WORK_DEVICE_DUMP, &card->work_flags);
clear_bit(MWIFIEX_IFACE_WORK_CARD_RESET, &card->work_flags);
mwifiex_dbg(adapter, INFO, "%s, successful\n", __func__);
+
+ card->pci_reset_ongoing = true;
}
/*
@@ -554,6 +556,8 @@ static void mwifiex_pcie_reset_done(struct pci_dev *pdev)
dev_err(&pdev->dev, "reinit failed: %d\n", ret);
else
mwifiex_dbg(adapter, INFO, "%s, successful\n", __func__);
+
+ card->pci_reset_ongoing = false;
}
static const struct pci_error_handlers mwifiex_pcie_err_handler = {
@@ -3142,7 +3146,19 @@ static void mwifiex_cleanup_pcie(struct mwifiex_adapter *adapter)
int ret;
u32 fw_status;
- cancel_work_sync(&card->work);
+ /* Perform the cancel_work_sync() only when we're not resetting
+ * the card. It's because that function never returns if we're
+ * in reset path. If we're here when resetting the card, it means
+ * that we failed to reset the card (reset failure path).
+ */
+ if (!card->pci_reset_ongoing) {
+ mwifiex_dbg(adapter, MSG, "performing cancel_work_sync()...\n");
+ cancel_work_sync(&card->work);
+ mwifiex_dbg(adapter, MSG, "cancel_work_sync() done\n");
+ } else {
+ mwifiex_dbg(adapter, MSG,
+ "skipped cancel_work_sync() because we're in card reset failure path\n");
+ }
ret = mwifiex_read_reg(adapter, reg->fw_status, &fw_status);
if (fw_status == FIRMWARE_READY_PCIE) {
diff --git a/drivers/net/wireless/marvell/mwifiex/pcie.h b/drivers/net/wireless/marvell/mwifiex/pcie.h
index 843d57eda8201..5ed613d657094 100644
--- a/drivers/net/wireless/marvell/mwifiex/pcie.h
+++ b/drivers/net/wireless/marvell/mwifiex/pcie.h
@@ -242,6 +242,8 @@ struct pcie_service_card {
struct mwifiex_msix_context share_irq_ctx;
struct work_struct work;
unsigned long work_flags;
+
+ bool pci_reset_ongoing;
};
static inline int
--
2.27.0
next prev parent reply other threads:[~2020-12-23 3:07 UTC|newest]
Thread overview: 80+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-12-23 2:12 [PATCH AUTOSEL 5.10 001/217] soc: aspeed-lpc-ctrl: Fail probe of lpc-ctrl if reserved memory is not aligned Sasha Levin
2020-12-23 2:12 ` [PATCH AUTOSEL 5.10 002/217] drm/ingenic: Reset pixclock rate when parent clock rate changes Sasha Levin
2020-12-23 2:12 ` [PATCH AUTOSEL 5.10 003/217] drm/bridge: ti-sn65dsi86: Add retries for link training Sasha Levin
2020-12-23 2:12 ` [PATCH AUTOSEL 5.10 004/217] drm/amd/display: setup system context in dm_init Sasha Levin
2020-12-23 2:12 ` [PATCH AUTOSEL 5.10 005/217] drm/amd/display: Fix the display corruption issue on Navi10 Sasha Levin
2020-12-23 2:12 ` [PATCH AUTOSEL 5.10 006/217] locks: Fix UBSAN undefined behaviour in flock64_to_posix_lock Sasha Levin
2020-12-23 2:12 ` [PATCH AUTOSEL 5.10 007/217] platform/x86: asus-wmi: Add support for SW_TABLET_MODE on UX360 Sasha Levin
2020-12-23 2:12 ` [PATCH AUTOSEL 5.10 008/217] tomoyo: fix clang pointer arithmetic warning Sasha Levin
2020-12-23 2:12 ` [PATCH AUTOSEL 5.10 009/217] HID: hid-input: occasionally report stylus battery even if not changed Sasha Levin
2020-12-23 2:12 ` [PATCH AUTOSEL 5.10 010/217] drm/amdgpu: change to save bad pages in UMC error interrupt callback Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 011/217] crypto: omap-aes - fix the reference count leak of omap device Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 012/217] drm/amd/display: Do not silently accept DCC for multiplane formats Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 013/217] ASoC: intel: sof_rt5682: Add quirk for Dooly Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 014/217] bpf: Use separate lockdep class for each hashtab Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 015/217] drm/msm: Fix race condition in msm driver with async layer updates Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 016/217] staging: wimax: depends on NET Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 017/217] RDMA/siw: Fix typo of EAGAIN not -EAGAIN in siw_cm_work_handler() Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 018/217] drm/amd/display: Fix compilation error Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 019/217] drm/amd/display: Force prefetch mode to 0 Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 020/217] drm/amd/display: Keep GSL for full updates with planes that flip VSYNC Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 021/217] drm/amd/display: stop top_mgr when type change to non-MST during s3 Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 022/217] drm/amd/display: correct eDP T9 delay Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 023/217] drm/amd/display: Update connector on DSC property change Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 024/217] drm/amd/display: fix recout calculation for left side clip Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 025/217] drm/amdgpu: disable gfxoff if VCN is busy Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 026/217] net: ethernet: ti: am65-cpsw: fix tx csum offload for multi mac mode Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 027/217] net: mscc: ocelot: don't reset the pvid to 0 when deleting it Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 028/217] drm/amdgpu: set LDS_CONFIG=0x20 on Navy Flounder to fix a GPU hang (v2) Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 029/217] scsi: target: Fix cmd_count ref leak Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 030/217] scsi: pm80xx: Make mpi_build_cmd locking consistent Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 031/217] scsi: pm80xx: Make running_req atomic Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 032/217] scsi: pm80xx: Avoid busywait in FW ready check Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 033/217] scsi: pm80xx: Fix pm8001_mpi_get_nvmd_resp() race condition Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 034/217] scsi: ufs: Allow an error return value from ->device_reset() Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 035/217] fcntl: Fix potential deadlock in send_sig{io, urg}() Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 036/217] drm/amdgpu: add missing clock gating info in amdgpu_pm_info Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 037/217] mac80211: don't overwrite QoS TID of injected frames Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 038/217] staging: ks7010: fix missing destroy_workqueue() on error in ks7010_sdio_probe Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 039/217] staging: rtl8192u: fix wrong judgement in rtl8192_rx_isr Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 040/217] mips: ar7: add missing iounmap() on error in ar7_gpio_init Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 041/217] mips: cm: add missing iounmap() on error in mips_cm_probe() Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 042/217] usb: typec: ucsi: Work around PPM losing change information Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 043/217] rcuscale: Prevent hangs for invalid arguments Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 044/217] refscale: " Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 045/217] locktorture: " Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 046/217] torture: Prevent jitter processes from delaying failed run Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 047/217] rcutorture: Prevent hangs for invalid arguments Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 048/217] kcsan: Fix encoding masks and regain address bit Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 049/217] ath10k: fix compilation warning Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 050/217] rsi: Fix TX EAPOL packet handling against iwlwifi AP Sasha Levin
2021-03-02 19:25 ` Marek Vasut
2021-03-04 20:47 ` Sasha Levin
2021-03-04 21:07 ` Marek Vasut
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 051/217] thermal: intel: pch: fix S0ix failure due to PCH temperature above threshold Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 052/217] drm: panel: simple: add missing platform_driver_unregister() in panel_simple_init Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 053/217] drm/bridge: lvds-codec: Use dev_err_probe for error handling Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 054/217] drm/ast: Fixed 1920x1080 sync. polarity issue Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 055/217] s390: make sure vmemmap is top region table entry aligned Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 056/217] s390/pci: remove races against pte updates Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 057/217] s390/trng: set quality to 1024 Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 058/217] Bluetooth: btqca: Add valid le states quirk Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 059/217] Bluetooth: Resume advertising after LE connection Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 060/217] Bluetooth: Move force_bredr_smp debugfs into hci_debugfs_create_bredr Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 061/217] Bluetooth: hidp: use correct wait queue when removing ctrl_wait Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 062/217] binder: change error code from postive to negative in binder_transaction Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 063/217] net: skb_vlan_untag(): don't reset transport offset if set by GRO layer Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 064/217] drm/omap: Fix runtime PM imbalance on error Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 065/217] pinctrl: qcom: Kconfig: Rework PINCTRL_MSM to be a depenency rather then a selected config Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 066/217] ASoC: Fix vaud18 power leakage of mt6359 Sasha Levin
2020-12-23 2:13 ` Sasha Levin [this message]
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 068/217] drm/amd/amdgpu: Fix incorrect logic to increment VCN doorbell index Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 069/217] drm/amd/amdgpu: Add rev_id workaround logic for SRIOV setup Sasha Levin
2020-12-23 2:13 ` [PATCH AUTOSEL 5.10 070/217] ASoC: Intel: sof_sdw: add quirk for new TigerLake-SDCA device Sasha Levin
2020-12-23 2:14 ` [PATCH AUTOSEL 5.10 071/217] RDMA/core: Postpone uobject cleanup on failure till FD close Sasha Levin
2020-12-23 2:14 ` [PATCH AUTOSEL 5.10 072/217] MIPS: BMC47xx: fix kconfig dependency bug for BCM47XX_SSB Sasha Levin
2020-12-23 2:14 ` [PATCH AUTOSEL 5.10 073/217] net: ipconfig: Avoid spurious blank lines in boot log Sasha Levin
2020-12-23 2:14 ` [PATCH AUTOSEL 5.10 074/217] drm/amd/amdgpu: Update VCN initizalization behvaior Sasha Levin
2020-12-23 2:14 ` [PATCH AUTOSEL 5.10 075/217] drm/amdgpu: check hive pointer before access Sasha Levin
2020-12-23 2:14 ` [PATCH AUTOSEL 5.10 076/217] jfs: Fix memleak in dbAdjCtl Sasha Levin
2020-12-23 2:14 ` [PATCH AUTOSEL 5.10 077/217] r8169: use READ_ONCE in rtl_tx_slots_avail Sasha Levin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201223021626.2790791-67-sashal@kernel.org \
--to=sashal@kernel.org \
--cc=kitakar@gmail.com \
--cc=kvalo@codeaurora.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-wireless@vger.kernel.org \
--cc=luzmaximilian@gmail.com \
--cc=netdev@vger.kernel.org \
--cc=stable@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).