All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH] USB: usbtmc: Fix RCU stall warning
@ 2021-06-29  3:32 qiang.zhang
  2021-07-07  1:29 ` Zhang, Qiang
  2021-07-21  7:08 ` Greg KH
  0 siblings, 2 replies; 20+ messages in thread
From: qiang.zhang @ 2021-06-29  3:32 UTC (permalink / raw)
  To: gregkh, stern, dvyukov; +Cc: paulmck, dpenkler, guido.kiener, linux-usb

From: Zqiang <qiang.zhang@windriver.com>

rcu: INFO: rcu_preempt self-detected stall on CPU
rcu:    1-...!: (2 ticks this GP) idle=d92/1/0x4000000000000000
        softirq=25390/25392 fqs=3
        (t=12164 jiffies g=31645 q=43226)
rcu: rcu_preempt kthread starved for 12162 jiffies! g31645 f0x0
     RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0
rcu:    Unless rcu_preempt kthread gets sufficient CPU time,
        OOM is now expected behavior.
rcu: RCU grace-period kthread stack dump:
task:rcu_preempt     state:R  running task

In the case of system use dummy_hcd as usb controller, when the
usbtmc devices is disconnected, in usbtmc_interrupt(), if the urb
status is unknown, the urb will be resubmit, the urb may be insert
to dum_hcd->urbp_list again, this will cause the dummy_timer() not
to exit for a long time, beacause the dummy_timer() be called in
softirq and local_bh is disable, this not only causes the RCU reading
critical area to consume too much time but also makes the tasks in
the current CPU runq not run in time, and that triggered RCU stall.

return directly when find the urb status is not zero to fix it.

Reported-by: syzbot+e2eae5639e7203360018@syzkaller.appspotmail.com
Signed-off-by: Zqiang <qiang.zhang@windriver.com>
---
 drivers/usb/class/usbtmc.c | 12 ++----------
 1 file changed, 2 insertions(+), 10 deletions(-)

diff --git a/drivers/usb/class/usbtmc.c b/drivers/usb/class/usbtmc.c
index 74d5a9c5238a..c4e1a88fff78 100644
--- a/drivers/usb/class/usbtmc.c
+++ b/drivers/usb/class/usbtmc.c
@@ -2324,17 +2324,9 @@ static void usbtmc_interrupt(struct urb *urb)
 		dev_err(dev, "overflow with length %d, actual length is %d\n",
 			data->iin_wMaxPacketSize, urb->actual_length);
 		fallthrough;
-	case -ECONNRESET:
-	case -ENOENT:
-	case -ESHUTDOWN:
-	case -EILSEQ:
-	case -ETIME:
-	case -EPIPE:
-		/* urb terminated, clean up */
-		dev_dbg(dev, "urb terminated, status: %d\n", status);
-		return;
 	default:
-		dev_err(dev, "unknown status received: %d\n", status);
+		dev_err(dev, "error status received: %d\n", status);
+		return;
 	}
 exit:
 	rv = usb_submit_urb(urb, GFP_ATOMIC);
-- 
2.17.1


^ permalink raw reply related	[flat|nested] 20+ messages in thread
* Re: [PATCH] USB: usbtmc: Fix RCU stall warning
@ 2021-07-21 11:15 Guido Kiener
  0 siblings, 0 replies; 20+ messages in thread
From: Guido Kiener @ 2021-07-21 11:15 UTC (permalink / raw)
  To: Greg KH, Zhang, Qiang; +Cc: stern, dvyukov, paulmck, dpenkler, linux-usb

> -----Original Message-----
> From: Greg KH <gregkh@linuxfoundation.org>
> Sent: Wednesday, July 21, 2021 9:53 AM
> To: Zhang, Qiang <Qiang.Zhang@windriver.com>
> Cc: stern@rowland.harvard.edu; dvyukov@google.com; paulmck@kernel.org;
> dpenkler@gmail.com; Kiener Guido 14DS1 <Guido.Kiener@rohde-schwarz.com>;
> linux-usb@vger.kernel.org
> Subject: *EXT* Re: [PATCH] USB: usbtmc: Fix RCU stall warning
> 
> On Wed, Jul 21, 2021 at 07:30:39AM +0000, Zhang, Qiang wrote:
> >
> >
> > ________________________________________
> > From: Greg KH <gregkh@linuxfoundation.org>
> > Sent: Wednesday, 21 July 2021 15:08
> > To: Zhang, Qiang
> > Cc: stern@rowland.harvard.edu; dvyukov@google.com; paulmck@kernel.org;
> > dpenkler@gmail.com; guido.kiener@rohde-schwarz.com;
> > linux-usb@vger.kernel.org
> > Subject: Re: [PATCH] USB: usbtmc: Fix RCU stall warning
> >
> > [Please note: This e-mail is from an EXTERNAL e-mail address]
> >
> > On Tue, Jun 29, 2021 at 11:32:36AM +0800, qiang.zhang@windriver.com wrote:
> > > From: Zqiang <qiang.zhang@windriver.com>
> >
> > >I need a "full" name here, and in the signed-off-by line please.
> >
> > >
> > > rcu: INFO: rcu_preempt self-detected stall on CPU
> > > rcu:    1-...!: (2 ticks this GP) idle=d92/1/0x4000000000000000
> > >         softirq=25390/25392 fqs=3
> > >         (t=12164 jiffies g=31645 q=43226)
> > > rcu: rcu_preempt kthread starved for 12162 jiffies! g31645 f0x0
> > >      RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0
> > > rcu:    Unless rcu_preempt kthread gets sufficient CPU time,
> > >         OOM is now expected behavior.
> > > rcu: RCU grace-period kthread stack dump:
> > > task:rcu_preempt     state:R  running task
> > >
> > > In the case of system use dummy_hcd as usb controller, when the
> > > usbtmc devices is disconnected, in usbtmc_interrupt(), if the urb
> > > status is unknown, the urb will be resubmit, the urb may be insert
> > > to dum_hcd->urbp_list again, this will cause the dummy_timer() not
> > > to exit for a long time, beacause the dummy_timer() be called in
> > > softirq and local_bh is disable, this not only causes the RCU
> > > reading critical area to consume too much time but also makes the
> > > tasks in the current CPU runq not run in time, and that triggered RCU stall.
> > >
> > > return directly when find the urb status is not zero to fix it.
> > >
> > > Reported-by: syzbot+e2eae5639e7203360018@syzkaller.appspotmail.com
> > > Signed-off-by: Zqiang <qiang.zhang@windriver.com>
> >
> > >What commit does this fix?  Does it need to go to stable kernels?
> >
> >  I will add fix tags resend,   need to go to stable kernel
> >
> > >
> > >What about the usbtmc maintainers, what do they think about this?

I'm ok with the fix. It will make the syzbot and dummy_hcd controller happy when using
the usbtmc driver. Nevertheless there are many other usb kernel driver that resubmit
the urb when the callback handler detects the urb status = -EPROTO.
So I expect the issue will rehappen with other usb drivers again.
In "normal" environments the urb status = -EPROTO will mostly happen when the cable is
disconnected, but it does not freeze the kernel and shuts down the connection as usual.
Up to now we have no customer feedback which is blaming this issue. I hope this helps.

-Guido

^ permalink raw reply	[flat|nested] 20+ messages in thread
* Re: [PATCH] USB: usbtmc: Fix RCU stall warning
@ 2021-07-21 15:24 Guido Kiener
  2021-07-21 16:17 ` Alan Stern
  0 siblings, 1 reply; 20+ messages in thread
From: Guido Kiener @ 2021-07-21 15:24 UTC (permalink / raw)
  To: Alan Stern, dave penkler
  Cc: Greg KH, qiang.zhang, Dmitry Vyukov, paulmck, USB

> -----Original Message-----
> From: Alan Stern <stern@rowland.harvard.edu>
> Sent: Wednesday, July 21, 2021 4:22 PM
> To: dave penkler <dpenkler@gmail.com>
> Cc: Greg KH <gregkh@linuxfoundation.org>; qiang.zhang@windriver.com; Dmitry
> Vyukov <dvyukov@google.com>; paulmck@kernel.org; Kiener Guido 14DS1
> <Guido.Kiener@rohde-schwarz.com>; USB <linux-usb@vger.kernel.org>
> Subject: *EXT* Re: [PATCH] USB: usbtmc: Fix RCU stall warning
> 
> On Wed, Jul 21, 2021 at 11:44:23AM +0200, dave penkler wrote:
> > Sorry, the issue this patch is trying to fix occurs because the
> > current usbtmc driver resubmits the URB when it gets an EPROTO return.
> > The dummy usb host controller driver used in the syzbot tests keeps
> > returning the resubmitted URB with EPROTO causing a loop that starves
> > RCU. With an actual HCI driver it either recovers or returns an EPIPE.
> 
> Are you sure about that?  Have you ever observed a usbtmc device recovering and
> continuing to operate after an EPROTO error?

I can't speak for Dave and his investigations. However as you remember I did tests with
EPROTO errors, see thread: https://marc.info/?l=linux-usb&m=162163776930423&w=2
In the thread you can see the recovering.
Since no user blamed the usbtmc driver for system locks up to now, it's worth to think about
whether the problem is caused by the dummy_hcd driver.
I still have no time for further investigations and would agree to use the simple patch
to get rid of the topic for the usbtmc driver. Then the syzbot will maybe find another usb driver.

-Guido

> An EPIPE error also seems rather unlikely -- particularly if the device is not plugged
> into a high-speed hub.
> 
> Alan Stern
> 
> > In either case no loop occurs. So for my part as a user and maintainer
> > this patch is not ok.

^ permalink raw reply	[flat|nested] 20+ messages in thread
* Re: [PATCH] USB: usbtmc: Fix RCU stall warning
@ 2021-07-21 17:08 Guido Kiener
  2021-07-21 18:16 ` Alan Stern
  0 siblings, 1 reply; 20+ messages in thread
From: Guido Kiener @ 2021-07-21 17:08 UTC (permalink / raw)
  To: Alan Stern
  Cc: dave penkler, Greg KH, qiang.zhang, Dmitry Vyukov, paulmck, USB

> > > Subject: *EXT* Re: [PATCH] USB: usbtmc: Fix RCU stall warning
> > >
> > > On Wed, Jul 21, 2021 at 11:44:23AM +0200, dave penkler wrote:
> > > > Sorry, the issue this patch is trying to fix occurs because the
> > > > current usbtmc driver resubmits the URB when it gets an EPROTO return.
> > > > The dummy usb host controller driver used in the syzbot tests
> > > > keeps returning the resubmitted URB with EPROTO causing a loop
> > > > that starves RCU. With an actual HCI driver it either recovers or returns an
> EPIPE.
> > >
> > > Are you sure about that?  Have you ever observed a usbtmc device
> > > recovering and continuing to operate after an EPROTO error?
> >
> > I can't speak for Dave and his investigations. However as you remember
> > I did tests with EPROTO errors, see thread:
> > https://marc.info/?l=linux-usb&m=162163776930423&w=2
> > In the thread you can see the recovering.
> 
> Ah yes, now I remember.
> 
> That message doesn't show the _device_ recovering and continuing to operate,
> though.  It shows the _system_ recovering and realizing that the device has been
> disconnected.
> 
> What I was asking about was whether you knew of a case where there was an
> EPROTO error but afterward the usbtmc device continued to work -- no
> disconnection.  Assuming such cases are vanishingly rare, there's no harm in
> having the driver give up whenever it encounters EPROTO.

I have no idea how often the EPROTO error can happen during normal operation and believe you that it's vanishingly rare.
When it happens, does the USB hardware protocol automatically retransmit the lost/invalid packet?
If yes, we should think about an error counter.
If not, then we really can stop the INT pipe and the user will detect that something is wrong when reading the status.

> > Since no user blamed the usbtmc driver for system locks up to now,
> > it's worth to think about whether the problem is caused by the dummy_hcd driver.
> 
> Both drivers contributed to the lockup.  The question is: Which driver was doing the
> wrong thing?  (Or which was _more_ wrong?)  I believe the usbtmc driver was.
> 
> > I still have no time for further investigations and would agree to use
> > the simple patch to get rid of the topic for the usbtmc driver. Then the syzbot will
> maybe find another usb driver.
> 
> Agreed.  So Greg should go ahead and apply the $SUBJECT patch.
> 
> Alan Stern

^ permalink raw reply	[flat|nested] 20+ messages in thread
* Re: [PATCH] USB: usbtmc: Fix RCU stall warning
@ 2021-07-22 17:33 Guido Kiener
  2021-07-23  0:36 ` Zhang, Qiang
  0 siblings, 1 reply; 20+ messages in thread
From: Guido Kiener @ 2021-07-22 17:33 UTC (permalink / raw)
  To: Greg KH, dave penkler
  Cc: qiang.zhang, Alan Stern, Dmitry Vyukov, paulmck, USB

> From: Greg KH
> Sent: Wednesday, July 21, 2021 11:48 AM
> Subject: *EXT* Re: [PATCH] USB: usbtmc: Fix RCU stall warning
> 
> On Wed, Jul 21, 2021 at 11:44:23AM +0200, dave penkler wrote:
> > On Wed, 21 Jul 2021 at 09:52, Greg KH <gregkh@linuxfoundation.org> wrote:
> > >
> > > On Wed, Jul 21, 2021 at 09:41:15AM +0200, dave penkler wrote:
> > > > On Wed, 21 Jul 2021 at 09:08, Greg KH <gregkh@linuxfoundation.org>
> wrote:
> > > > >
> > > > > On Tue, Jun 29, 2021 at 11:32:36AM +0800, qiang.zhang@windriver.com
> wrote:
> > > > > > From: Zqiang <qiang.zhang@windriver.com>
> > > > >
> > > > > I need a "full" name here, and in the signed-off-by line please.
> > > > >
> > > > > >
> > > > > > rcu: INFO: rcu_preempt self-detected stall on CPU
> > > > > > rcu:    1-...!: (2 ticks this GP) idle=d92/1/0x4000000000000000
> > > > > >         softirq=25390/25392 fqs=3
> > > > > >         (t=12164 jiffies g=31645 q=43226)
> > > > > > rcu: rcu_preempt kthread starved for 12162 jiffies! g31645 f0x0
> > > > > >      RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0
> > > > > > rcu:    Unless rcu_preempt kthread gets sufficient CPU time,
> > > > > >         OOM is now expected behavior.
> > > > > > rcu: RCU grace-period kthread stack dump:
> > > > > > task:rcu_preempt     state:R  running task
> > > > > >
> > > > > > In the case of system use dummy_hcd as usb controller, when
> > > > > > the usbtmc devices is disconnected, in usbtmc_interrupt(), if
> > > > > > the urb status is unknown, the urb will be resubmit, the urb
> > > > > > may be insert to dum_hcd->urbp_list again, this will cause the
> > > > > > dummy_timer() not to exit for a long time, beacause the
> > > > > > dummy_timer() be called in softirq and local_bh is disable,
> > > > > > this not only causes the RCU reading critical area to consume
> > > > > > too much time but also makes the tasks in the current CPU runq not run
> in time, and that triggered RCU stall.
> > > > > >
> > > > > > return directly when find the urb status is not zero to fix it.
> > > > > >
> > > > > > Reported-by:
> > > > > > syzbot+e2eae5639e7203360018@syzkaller.appspotmail.com
> > > > > > Signed-off-by: Zqiang <qiang.zhang@windriver.com>
> > > > >
> > > > > What commit does this fix?  Does it need to go to stable kernels?
> > > > >
> > > > > What about the usbtmc maintainers, what do they think about this?
> > > >
> > > > This patch makes the babbling endpoint retry/recovery code in the
> > > > real world usb host controller drivers redundant and would prevent
> > > > usbtmc applications from benefiting from it.
> > >
> > > I do not understand, is this change ok or not?
> > >
> > > Why do usbtmc applications need to know if babbling happens or not?
> > >
> > > confused,
> > Sorry, the issue this patch is trying to fix occurs because the
> > current usbtmc driver resubmits the URB when it gets an EPROTO return.
> > The dummy usb host controller driver used in the syzbot tests keeps
> > returning the resubmitted URB with EPROTO causing a loop that starves
> > RCU. With an actual HCI driver it either recovers or returns an EPIPE.
> > In either case no loop occurs. So for my part as a user and maintainer
> > this patch is not ok.
> 
> Thanks for the review.
> 
> Zqiang, can you fix this patch up based on this please?
> 
> thanks,
> 
> greg k-h

Qiang,

After discussions with Alan and Dave we think that fixing the usbtmc driver is the best approach to fix the RCU stall warning.
Your first proposal was almost ok, but I think we should use dev_dbg() instead of dev_err() to avoid printing the EPROTO errors. See below:

Please feel free to add the following text to your patch description.

-Guido


The function usbtmc_interrupt() resubmits urbs when the error status
of an urb is -EPROTO. In systems using the dummy_hcd usb controller
this can result in endless interrupt loops when the usbtmc device is
disconnected from the host system.
    
Since host controller drivers already try to recover from transmission
errors, there is no need to resubmit the urb or try other solutions
to repair the error situation.
    
In case of errors the INT pipe just stops to wait for further packets.

Reviewed-by: Guido Kiener <guido.kiener@rohde-schwarz.com>

diff --git a/drivers/usb/class/usbtmc.c b/drivers/usb/class/usbtmc.c
index 74d5a9c5238a..73f419adce61 100644
--- a/drivers/usb/class/usbtmc.c
+++ b/drivers/usb/class/usbtmc.c
@@ -2324,17 +2324,10 @@ static void usbtmc_interrupt(struct urb *urb)
                dev_err(dev, "overflow with length %d, actual length is %d\n",
                        data->iin_wMaxPacketSize, urb->actual_length);
                fallthrough;
-       case -ECONNRESET:
-       case -ENOENT:
-       case -ESHUTDOWN:
-       case -EILSEQ:
-       case -ETIME:
-       case -EPIPE:
+       default:
                /* urb terminated, clean up */
                dev_dbg(dev, "urb terminated, status: %d\n", status);
                return;
-       default:
-               dev_err(dev, "unknown status received: %d\n", status);
        }
 exit:
        rv = usb_submit_urb(urb, GFP_ATOMIC);

^ permalink raw reply related	[flat|nested] 20+ messages in thread
end of thread, other threads:[~2021-07-23  0:36 UTC | newest]

Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-29  3:32 [PATCH] USB: usbtmc: Fix RCU stall warning qiang.zhang
2021-07-07  1:29 ` Zhang, Qiang
2021-07-07  5:58   ` gregkh
2021-07-21  7:08 ` Greg KH
2021-07-21  7:30   ` Zhang, Qiang
2021-07-21  7:52     ` Greg KH
2021-07-21  8:28       ` Zhang, Qiang
2021-07-21  7:41   ` dave penkler
2021-07-21  7:52     ` Greg KH
2021-07-21  9:44       ` dave penkler
2021-07-21  9:47         ` Greg KH
2021-07-21 14:22         ` Alan Stern
2021-07-21  8:34     ` Zhang, Qiang
2021-07-21 11:15 Guido Kiener
2021-07-21 15:24 Guido Kiener
2021-07-21 16:17 ` Alan Stern
2021-07-21 17:08 Guido Kiener
2021-07-21 18:16 ` Alan Stern
2021-07-22 17:33 Guido Kiener
2021-07-23  0:36 ` Zhang, Qiang

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.