linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Sasha Levin <sashal@kernel.org>
To: linux-kernel@vger.kernel.org, stable@vger.kernel.org
Cc: Firo Yang <firo.yang@suse.com>,
	"David S . Miller" <davem@davemloft.net>,
	Sasha Levin <sashal@kernel.org>,
	netdev@vger.kernel.org
Subject: [PATCH AUTOSEL 5.4 40/50] enic: prevent waking up stopped tx queues over watchdog reset
Date: Sat, 22 Feb 2020 21:22:25 -0500	[thread overview]
Message-ID: <20200223022235.1404-40-sashal@kernel.org> (raw)
In-Reply-To: <20200223022235.1404-1-sashal@kernel.org>

From: Firo Yang <firo.yang@suse.com>

[ Upstream commit 0f90522591fd09dd201065c53ebefdfe3c6b55cb ]

Recent months, our customer reported several kernel crashes all
preceding with following message:
NETDEV WATCHDOG: eth2 (enic): transmit queue 0 timed out
Error message of one of those crashes:
BUG: unable to handle kernel paging request at ffffffffa007e090

After analyzing severl vmcores, I found that most of crashes are
caused by memory corruption. And all the corrupted memory areas
are overwritten by data of network packets. Moreover, I also found
that the tx queues were enabled over watchdog reset.

After going through the source code, I found that in enic_stop(),
the tx queues stopped by netif_tx_disable() could be woken up over
a small time window between netif_tx_disable() and the
napi_disable() by the following code path:
napi_poll->
  enic_poll_msix_wq->
     vnic_cq_service->
        enic_wq_service->
           netif_wake_subqueue(enic->netdev, q_number)->
              test_and_clear_bit(__QUEUE_STATE_DRV_XOFF, &txq->state)
In turn, upper netowrk stack could queue skb to ENIC NIC though
enic_hard_start_xmit(). And this might introduce some race condition.

Our customer comfirmed that this kind of kernel crash doesn't occur over
90 days since they applied this patch.

Signed-off-by: Firo Yang <firo.yang@suse.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
 drivers/net/ethernet/cisco/enic/enic_main.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/cisco/enic/enic_main.c b/drivers/net/ethernet/cisco/enic/enic_main.c
index acb2856936d20..6e2ab10ad2e6f 100644
--- a/drivers/net/ethernet/cisco/enic/enic_main.c
+++ b/drivers/net/ethernet/cisco/enic/enic_main.c
@@ -2013,10 +2013,10 @@ static int enic_stop(struct net_device *netdev)
 		napi_disable(&enic->napi[i]);
 
 	netif_carrier_off(netdev);
-	netif_tx_disable(netdev);
 	if (vnic_dev_get_intr_mode(enic->vdev) == VNIC_DEV_INTR_MODE_MSIX)
 		for (i = 0; i < enic->wq_count; i++)
 			napi_disable(&enic->napi[enic_cq_wq(enic, i)]);
+	netif_tx_disable(netdev);
 
 	if (!enic_is_dynamic(enic) && !enic_is_sriov_vf(enic))
 		enic_dev_del_station_addr(enic);
-- 
2.20.1


  parent reply	other threads:[~2020-02-23  2:31 UTC|newest]

Thread overview: 50+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-02-23  2:21 [PATCH AUTOSEL 5.4 01/50] ipmi:ssif: Handle a possible NULL pointer reference Sasha Levin
2020-02-23  2:21 ` [PATCH AUTOSEL 5.4 02/50] drm/msm: Set dma maximum segment size for mdss Sasha Levin
2020-02-23  2:21 ` [PATCH AUTOSEL 5.4 03/50] sched/core: Don't skip remote tick for idle CPUs Sasha Levin
2020-02-23  2:21 ` [PATCH AUTOSEL 5.4 04/50] timers/nohz: Update NOHZ load in remote tick Sasha Levin
2020-02-23  2:21 ` [PATCH AUTOSEL 5.4 05/50] sched/fair: Prevent unlimited runtime on throttled group Sasha Levin
2020-02-23  2:21 ` [PATCH AUTOSEL 5.4 06/50] dax: pass NOWAIT flag to iomap_apply Sasha Levin
2020-02-23  2:21 ` [PATCH AUTOSEL 5.4 07/50] mac80211: consider more elements in parsing CRC Sasha Levin
2020-02-23  2:21 ` [PATCH AUTOSEL 5.4 08/50] cfg80211: check wiphy driver existence for drvinfo report Sasha Levin
2020-02-23  2:21 ` [PATCH AUTOSEL 5.4 09/50] s390/zcrypt: fix card and queue total counter wrap Sasha Levin
2020-02-23  2:21 ` [PATCH AUTOSEL 5.4 10/50] qmi_wwan: re-add DW5821e pre-production variant Sasha Levin
2020-02-23  2:21 ` [PATCH AUTOSEL 5.4 11/50] qmi_wwan: unconditionally reject 2 ep interfaces Sasha Levin
2020-02-23  2:21 ` [PATCH AUTOSEL 5.4 12/50] NFSv4: Fix races between open and dentry revalidation Sasha Levin
2020-02-23  2:21 ` [PATCH AUTOSEL 5.4 13/50] perf/smmuv3: Use platform_get_irq_optional() for wired interrupt Sasha Levin
2020-02-23  2:21 ` [PATCH AUTOSEL 5.4 14/50] arm/ftrace: Fix BE text poking Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 15/50] sched/psi: Fix OOB write when writing 0 bytes to PSI files Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 16/50] perf/x86/intel: Add Elkhart Lake support Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 17/50] perf/x86/cstate: Add Tremont support Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 18/50] perf/x86/msr: " Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 19/50] ceph: do not execute direct write in parallel if O_APPEND is specified Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 20/50] ARM: dts: sti: fixup sound frame-inversion for stihxxx-b2120.dtsi Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 21/50] drm/amd/display: Do not set optimized_require to false after plane disable Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 22/50] RDMA/siw: Remove unwanted WARN_ON in siw_cm_llp_data_ready() Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 23/50] drm/amd/display: Check engine is not NULL before acquiring Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 24/50] drm/amd/display: Limit minimum DPPCLK to 100MHz Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 25/50] drm/amd/display: Add initialitions for PLL2 clock source Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 26/50] amdgpu: Prevent build errors regarding soft/hard-float FP ABI tags Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 27/50] soc/tegra: fuse: Fix build with Tegra194 configuration Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 28/50] i40e: Fix the conditional for i40e_vc_validate_vqs_bitmaps Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 29/50] net: ena: fix potential crash when rxfh key is NULL Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 30/50] net: ena: fix uses of round_jiffies() Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 31/50] net: ena: add missing ethtool TX timestamping indication Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 32/50] net: ena: fix incorrect default RSS key Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 33/50] net: ena: rss: do not allocate key when not supported Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 34/50] net: ena: rss: fix failure to get indirection table Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 35/50] net: ena: rss: store hash function as values and not bits Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 36/50] net: ena: fix incorrectly saving queue numbers when setting RSS indirection table Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 37/50] net: ena: fix corruption of dev_idx_to_host_tbl Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 38/50] net: ena: ethtool: use correct value for crc32 hash Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 39/50] net: ena: ena-com.c: prevent NULL pointer dereference Sasha Levin
2020-02-23  2:22 ` Sasha Levin [this message]
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 41/50] ice: update Unit Load Status bitmask to check after reset Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 42/50] cifs: Fix mode output in debugging statements Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 43/50] bcache: ignore pending signals when creating gc and allocator thread Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 44/50] cfg80211: add missing policy for NL80211_ATTR_STATUS_CODE Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 45/50] mac80211: fix wrong 160/80+80 MHz setting Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 46/50] net: hns3: add management table after IMP reset Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 47/50] net: hns3: fix a copying IPv6 address error in hclge_fd_get_flow_tuples() Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 48/50] nvme/tcp: fix bug on double requeue when send fails Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 49/50] nvme: prevent warning triggered by nvme_stop_keep_alive Sasha Levin
2020-02-23  2:22 ` [PATCH AUTOSEL 5.4 50/50] nvme/pci: move cqe check after device shutdown Sasha Levin

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200223022235.1404-40-sashal@kernel.org \
    --to=sashal@kernel.org \
    --cc=davem@davemloft.net \
    --cc=firo.yang@suse.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=netdev@vger.kernel.org \
    --cc=stable@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).