From mboxrd@z Thu Jan 1 00:00:00 1970 Return-path: Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:36282 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1753059AbcLRP7g (ORCPT ); Sun, 18 Dec 2016 10:59:36 -0500 Received: from pps.filterd (m0098420.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.17/8.16.0.17) with SMTP id uBIFx9O0037387 for ; Sun, 18 Dec 2016 10:59:35 -0500 Received: from e38.co.us.ibm.com (e38.co.us.ibm.com [32.97.110.159]) by mx0b-001b2d01.pphosted.com with ESMTP id 27dkuwqq1m-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Sun, 18 Dec 2016 10:59:35 -0500 Received: from localhost by e38.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Sun, 18 Dec 2016 08:59:34 -0700 Date: Sun, 18 Dec 2016 07:59:38 -0800 From: "Paul E. McKenney" To: Gabriel C Cc: lkml , ath9k-devel@qca.qualcomm.com, linux-wireless@vger.kernel.org, ath9k-devel@lists.ath9k.org, netdev@vger.kernel.org, nbd@nbd.name, kvalo@qca.qualcomm.com Subject: Re: regression: ath_tx_edma_tasklet() Illegal idle entry in RCU read-side critical section Reply-To: paulmck@linux.vnet.ibm.com References: <23a2a3ab-974a-ed26-6afa-aafab9bb972e@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii In-Reply-To: <23a2a3ab-974a-ed26-6afa-aafab9bb972e@gmail.com> Message-Id: <20161218155938.GP3924@linux.vnet.ibm.com> (sfid-20161218_170045_273068_3BF17D7C) Sender: linux-wireless-owner@vger.kernel.org List-ID: On Sun, Dec 18, 2016 at 02:52:48PM +0100, Gabriel C wrote: > Hello, > > while testing kernel 4.9 I run into a weird issue with the ath9k driver. > > I can boot the box in console mode and it stay up sometime but is not usable. Looks to me like someone forgot an rcu_read_unlock() somewhere. Given that the unmatched rcu_read_lock() appears in ath_tx_edma_tasklet(), perhaps that is also where the missing rcu_read_unlock() is. And sure enough, in the middle of this function we have the following: fifo_list = &txq->txq_fifo[txq->txq_tailidx]; if (list_empty(fifo_list)) { ath_txq_unlock(sc, txq); return; } This will of course return while still in an RCU read-side critical section. The caller cannot tell the difference between a return here and falling off the end of the function, so this is likely the bug. Or one of the bugs, anyway. Copying the author and committer for their thoughts. Please try the patch at the end of this email. Thanx, Paul > from dmesg : > > =============================== > [ INFO: suspicious RCU usage. ] > 4.9-fw1 #1 Tainted: G I > ------------------------------- > kernel/rcu/tree.c:705 Illegal idle entry in RCU read-side critical section.! > > other info that might help us debug this: > > > RCU used illegally from idle CPU! > rcu_scheduler_active = 1, debug_locks = 1 > RCU used illegally from extended quiescent state! > 1 lock held by swapper/0/0: > #0: (rcu_read_lock){......}, at: [] ath_tx_edma_tasklet+0x0/0x460 [ath9k] > > stack backtrace: > CPU: 0 PID: 0 Comm: swapper/0 Tainted: G I 4.9-fw1 #1 > Hardware name: FUJITSU PRIMERGY TX200 S5 /D2709, BIOS 6.00 Rev. 1.14.2709 02/04/2013 > ffff88043ee03f38 ffffffff812cf0f3 ffffffff81a11540 0000000000000001 > ffff88043ee03f68 ffffffff810b7865 ffffffff81a55d58 ffff88043efcedc0 > ffff88083cb1ca00 00000000000000d1 ffff88043ee03f88 ffffffff810dbfe8 > Call Trace: > > [] dump_stack+0x86/0xc3 > [] lockdep_rcu_suspicious+0xc5/0x100 > [] rcu_eqs_enter_common.constprop.62+0x128/0x130 > [] rcu_irq_exit+0x38/0x70 > [] irq_exit+0x74/0xd0 > [] do_IRQ+0x71/0x130 > [] common_interrupt+0x8c/0x8c > > [] ? cpuidle_enter_state+0x156/0x220 > [] cpuidle_enter+0x12/0x20 > [] call_cpuidle+0x1e/0x40 > [] cpu_startup_entry+0x11d/0x210 > [] rest_init+0x12c/0x140 > [] start_kernel+0x40f/0x41c > [] ? early_idt_handler_array+0x120/0x120 > [] x86_64_start_reservations+0x2a/0x2c > [] x86_64_start_kernel+0xeb/0xf8 ------------------------------------------------------------------------ commit 5a16fed76936184a7ac22e466cf39bd8bb5ee65e Author: Paul E. McKenney Date: Sun Dec 18 07:49:00 2016 -0800 drivers/ath: Add missing rcu_read_unlock() to ath_tx_edma_tasklet() Commit d94a461d7a7d ("ath9k: use ieee80211_tx_status_noskb where possible") added rcu_read_lock() and rcu_read_unlock() around the body of ath_tx_edma_tasklet(), but failed to add the needed rcu_read_unlock() before a "return" in the middle of this function. This commit therefore adds the missing rcu_read_unlock(). Reported-by: Gabriel C Signed-off-by: Paul E. McKenney Cc: Felix Fietkau Cc: Kalle Valo Cc: QCA ath9k Development Cc: diff --git a/drivers/net/wireless/ath/ath9k/xmit.c b/drivers/net/wireless/ath/ath9k/xmit.c index 52bfbb988611..857d5ae09a1d 100644 --- a/drivers/net/wireless/ath/ath9k/xmit.c +++ b/drivers/net/wireless/ath/ath9k/xmit.c @@ -2787,6 +2787,7 @@ void ath_tx_edma_tasklet(struct ath_softc *sc) fifo_list = &txq->txq_fifo[txq->txq_tailidx]; if (list_empty(fifo_list)) { ath_txq_unlock(sc, txq); + rcu_read_unlock(); return; } From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757795AbcLRP7h (ORCPT ); Sun, 18 Dec 2016 10:59:37 -0500 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:50457 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1754247AbcLRP7g (ORCPT ); Sun, 18 Dec 2016 10:59:36 -0500 Date: Sun, 18 Dec 2016 07:59:38 -0800 From: "Paul E. McKenney" To: Gabriel C Cc: lkml , ath9k-devel@qca.qualcomm.com, linux-wireless@vger.kernel.org, ath9k-devel@venema.h4ckr.net, netdev@vger.kernel.org, nbd@nbd.name, kvalo@qca.qualcomm.com Subject: Re: regression: ath_tx_edma_tasklet() Illegal idle entry in RCU read-side critical section Reply-To: paulmck@linux.vnet.ibm.com References: <23a2a3ab-974a-ed26-6afa-aafab9bb972e@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <23a2a3ab-974a-ed26-6afa-aafab9bb972e@gmail.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-GCONF: 00 X-Content-Scanned: Fidelis XPS MAILER x-cbid: 16121815-0028-0000-0000-000006487074 X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00006272; HX=3.00000240; KW=3.00000007; PH=3.00000004; SC=3.00000198; SDB=6.00795580; UDB=6.00386019; IPR=6.00573390; BA=6.00004980; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00013653; XFM=3.00000011; UTC=2016-12-18 15:59:34 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 16121815-0029-0000-0000-000031BEB1FE Message-Id: <20161218155938.GP3924@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2016-12-18_15:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1612050000 definitions=main-1612180284 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sun, Dec 18, 2016 at 02:52:48PM +0100, Gabriel C wrote: > Hello, > > while testing kernel 4.9 I run into a weird issue with the ath9k driver. > > I can boot the box in console mode and it stay up sometime but is not usable. Looks to me like someone forgot an rcu_read_unlock() somewhere. Given that the unmatched rcu_read_lock() appears in ath_tx_edma_tasklet(), perhaps that is also where the missing rcu_read_unlock() is. And sure enough, in the middle of this function we have the following: fifo_list = &txq->txq_fifo[txq->txq_tailidx]; if (list_empty(fifo_list)) { ath_txq_unlock(sc, txq); return; } This will of course return while still in an RCU read-side critical section. The caller cannot tell the difference between a return here and falling off the end of the function, so this is likely the bug. Or one of the bugs, anyway. Copying the author and committer for their thoughts. Please try the patch at the end of this email. Thanx, Paul > from dmesg : > > =============================== > [ INFO: suspicious RCU usage. ] > 4.9-fw1 #1 Tainted: G I > ------------------------------- > kernel/rcu/tree.c:705 Illegal idle entry in RCU read-side critical section.! > > other info that might help us debug this: > > > RCU used illegally from idle CPU! > rcu_scheduler_active = 1, debug_locks = 1 > RCU used illegally from extended quiescent state! > 1 lock held by swapper/0/0: > #0: (rcu_read_lock){......}, at: [] ath_tx_edma_tasklet+0x0/0x460 [ath9k] > > stack backtrace: > CPU: 0 PID: 0 Comm: swapper/0 Tainted: G I 4.9-fw1 #1 > Hardware name: FUJITSU PRIMERGY TX200 S5 /D2709, BIOS 6.00 Rev. 1.14.2709 02/04/2013 > ffff88043ee03f38 ffffffff812cf0f3 ffffffff81a11540 0000000000000001 > ffff88043ee03f68 ffffffff810b7865 ffffffff81a55d58 ffff88043efcedc0 > ffff88083cb1ca00 00000000000000d1 ffff88043ee03f88 ffffffff810dbfe8 > Call Trace: > > [] dump_stack+0x86/0xc3 > [] lockdep_rcu_suspicious+0xc5/0x100 > [] rcu_eqs_enter_common.constprop.62+0x128/0x130 > [] rcu_irq_exit+0x38/0x70 > [] irq_exit+0x74/0xd0 > [] do_IRQ+0x71/0x130 > [] common_interrupt+0x8c/0x8c > > [] ? cpuidle_enter_state+0x156/0x220 > [] cpuidle_enter+0x12/0x20 > [] call_cpuidle+0x1e/0x40 > [] cpu_startup_entry+0x11d/0x210 > [] rest_init+0x12c/0x140 > [] start_kernel+0x40f/0x41c > [] ? early_idt_handler_array+0x120/0x120 > [] x86_64_start_reservations+0x2a/0x2c > [] x86_64_start_kernel+0xeb/0xf8 ------------------------------------------------------------------------ commit 5a16fed76936184a7ac22e466cf39bd8bb5ee65e Author: Paul E. McKenney Date: Sun Dec 18 07:49:00 2016 -0800 drivers/ath: Add missing rcu_read_unlock() to ath_tx_edma_tasklet() Commit d94a461d7a7d ("ath9k: use ieee80211_tx_status_noskb where possible") added rcu_read_lock() and rcu_read_unlock() around the body of ath_tx_edma_tasklet(), but failed to add the needed rcu_read_unlock() before a "return" in the middle of this function. This commit therefore adds the missing rcu_read_unlock(). Reported-by: Gabriel C Signed-off-by: Paul E. McKenney Cc: Felix Fietkau Cc: Kalle Valo Cc: QCA ath9k Development Cc: diff --git a/drivers/net/wireless/ath/ath9k/xmit.c b/drivers/net/wireless/ath/ath9k/xmit.c index 52bfbb988611..857d5ae09a1d 100644 --- a/drivers/net/wireless/ath/ath9k/xmit.c +++ b/drivers/net/wireless/ath/ath9k/xmit.c @@ -2787,6 +2787,7 @@ void ath_tx_edma_tasklet(struct ath_softc *sc) fifo_list = &txq->txq_fifo[txq->txq_tailidx]; if (list_empty(fifo_list)) { ath_txq_unlock(sc, txq); + rcu_read_unlock(); return; } From mboxrd@z Thu Jan 1 00:00:00 1970 From: Paul E. McKenney Date: Sun, 18 Dec 2016 16:17:05 -0000 Subject: [ath9k-devel] regression: ath_tx_edma_tasklet() Illegal idle entry in RCU read-side critical section In-Reply-To: <23a2a3ab-974a-ed26-6afa-aafab9bb972e@gmail.com> References: <23a2a3ab-974a-ed26-6afa-aafab9bb972e@gmail.com> Message-ID: <20161218155938.GP3924@linux.vnet.ibm.com> List-Id: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: ath9k-devel@lists.ath9k.org On Sun, Dec 18, 2016 at 02:52:48PM +0100, Gabriel C wrote: > Hello, > > while testing kernel 4.9 I run into a weird issue with the ath9k driver. > > I can boot the box in console mode and it stay up sometime but is not usable. Looks to me like someone forgot an rcu_read_unlock() somewhere. Given that the unmatched rcu_read_lock() appears in ath_tx_edma_tasklet(), perhaps that is also where the missing rcu_read_unlock() is. And sure enough, in the middle of this function we have the following: fifo_list = &txq->txq_fifo[txq->txq_tailidx]; if (list_empty(fifo_list)) { ath_txq_unlock(sc, txq); return; } This will of course return while still in an RCU read-side critical section. The caller cannot tell the difference between a return here and falling off the end of the function, so this is likely the bug. Or one of the bugs, anyway. Copying the author and committer for their thoughts. Please try the patch at the end of this email. Thanx, Paul > from dmesg : > > =============================== > [ INFO: suspicious RCU usage. ] > 4.9-fw1 #1 Tainted: G I > ------------------------------- > kernel/rcu/tree.c:705 Illegal idle entry in RCU read-side critical section.! > > other info that might help us debug this: > > > RCU used illegally from idle CPU! > rcu_scheduler_active = 1, debug_locks = 1 > RCU used illegally from extended quiescent state! > 1 lock held by swapper/0/0: > #0: (rcu_read_lock){......}, at: [] ath_tx_edma_tasklet+0x0/0x460 [ath9k] > > stack backtrace: > CPU: 0 PID: 0 Comm: swapper/0 Tainted: G I 4.9-fw1 #1 > Hardware name: FUJITSU PRIMERGY TX200 S5 /D2709, BIOS 6.00 Rev. 1.14.2709 02/04/2013 > ffff88043ee03f38 ffffffff812cf0f3 ffffffff81a11540 0000000000000001 > ffff88043ee03f68 ffffffff810b7865 ffffffff81a55d58 ffff88043efcedc0 > ffff88083cb1ca00 00000000000000d1 ffff88043ee03f88 ffffffff810dbfe8 > Call Trace: > > [] dump_stack+0x86/0xc3 > [] lockdep_rcu_suspicious+0xc5/0x100 > [] rcu_eqs_enter_common.constprop.62+0x128/0x130 > [] rcu_irq_exit+0x38/0x70 > [] irq_exit+0x74/0xd0 > [] do_IRQ+0x71/0x130 > [] common_interrupt+0x8c/0x8c > > [] ? cpuidle_enter_state+0x156/0x220 > [] cpuidle_enter+0x12/0x20 > [] call_cpuidle+0x1e/0x40 > [] cpu_startup_entry+0x11d/0x210 > [] rest_init+0x12c/0x140 > [] start_kernel+0x40f/0x41c > [] ? early_idt_handler_array+0x120/0x120 > [] x86_64_start_reservations+0x2a/0x2c > [] x86_64_start_kernel+0xeb/0xf8 ------------------------------------------------------------------------ commit 5a16fed76936184a7ac22e466cf39bd8bb5ee65e Author: Paul E. McKenney Date: Sun Dec 18 07:49:00 2016 -0800 drivers/ath: Add missing rcu_read_unlock() to ath_tx_edma_tasklet() Commit d94a461d7a7d ("ath9k: use ieee80211_tx_status_noskb where possible") added rcu_read_lock() and rcu_read_unlock() around the body of ath_tx_edma_tasklet(), but failed to add the needed rcu_read_unlock() before a "return" in the middle of this function. This commit therefore adds the missing rcu_read_unlock(). Reported-by: Gabriel C Signed-off-by: Paul E. McKenney Cc: Felix Fietkau Cc: Kalle Valo Cc: QCA ath9k Development Cc: diff --git a/drivers/net/wireless/ath/ath9k/xmit.c b/drivers/net/wireless/ath/ath9k/xmit.c index 52bfbb988611..857d5ae09a1d 100644 --- a/drivers/net/wireless/ath/ath9k/xmit.c +++ b/drivers/net/wireless/ath/ath9k/xmit.c @@ -2787,6 +2787,7 @@ void ath_tx_edma_tasklet(struct ath_softc *sc) fifo_list = &txq->txq_fifo[txq->txq_tailidx]; if (list_empty(fifo_list)) { ath_txq_unlock(sc, txq); + rcu_read_unlock(); return; }