Linux-USB Archive on lore.kernel.org
 help / color / Atom feed
* Regression: USB/xhci issues on some systems with newer kernel versions
@ 2019-10-02 12:28 Bernhard Gebetsberger
  2019-10-03 10:23 ` Mathias Nyman
  0 siblings, 1 reply; 5+ messages in thread
From: Bernhard Gebetsberger @ 2019-10-02 12:28 UTC (permalink / raw)
  To: linux-usb

Hi,

There has been a regression in the xhci driver since kernel version 4.20, on some systems some usb devices won't work until the system gets rebooted.
The error message in dmesg is "WARN Set TR Deq Ptr cmd failed due to incorrect slot or ep state", although for some reason there are some usb devices that are affected by this issue but don't throw the error message(including the device I'm using, I got the error in previous kernel versions though).
It seems like this bug can also lead to system instability, one user reported in the bug tracker(https://bugzilla.kernel.org/show_bug.cgi?id=202541#c58) that he got a system freeze because of this when using kernel 5.3.1.

When looking at the responses in the bug tracker, it looks like it mostly affects Ryzen based systems with 300 series motherboards, although there are some other affected systems as well. It doesn't only affect wifi/bluetooth sticks, some users even got this issue when connecting their smartphone or their external hard drive to their PC.

After enabling kernel debugging/tracing for xhci_hcd I got the following messages in dmesg(short version, link to the whole file below):
[  231.185635] xhci_hcd 0000:38:00.4: xhci_get_isoc_frame_id: index 0, reg 0x1b29 start_frame_id 0x366, end_frame_id 0x6e4, start_frame 0x372
[  231.185642] xhci_hcd 0000:38:00.4: xhci_get_isoc_frame_id: index 1, reg 0x1b29 start_frame_id 0x366, end_frame_id 0x6e4, start_frame 0x373
[  231.185646] xhci_hcd 0000:38:00.4: xhci_get_isoc_frame_id: index 2, reg 0x1b29 start_frame_id 0x366, end_frame_id 0x6e4, start_frame 0x374
......
[  231.887681] xhci_hcd 0000:38:00.4: xhci_get_isoc_frame_id: index 4, reg 0x3119 start_frame_id 0x624, end_frame_id 0x1a2, start_frame 0x633
[  231.887687] xhci_hcd 0000:38:00.4: xhci_get_isoc_frame_id: index 5, reg 0x3119 start_frame_id 0x624, end_frame_id 0x1a2, start_frame 0x634
[  231.892346] xhci_hcd 0000:38:00.4: Cancel URB 000000008599ca58, dev 1, ep 0x1, starting at offset 0xff388ea0
[  231.892355] xhci_hcd 0000:38:00.4: // Ding dong!
[  231.892363] xhci_hcd 0000:38:00.4: Cancel URB 000000000d35fd5d, dev 1, ep 0x1, starting at offset 0xff388ef0
[  231.892368] xhci_hcd 0000:38:00.4: Cancel URB 0000000074e3ee88, dev 1, ep 0x1, starting at offset 0xff388e40
[  231.892640] xhci_hcd 0000:38:00.4: Stopped on Transfer TRB for slot 1 ep 1
[  231.892647] xhci_hcd 0000:38:00.4: Removing canceled TD starting at 0xff388ea0 (dma).
[  231.892651] xhci_hcd 0000:38:00.4: Removing canceled TD starting at 0xff388eb0 (dma).
[  231.892653] xhci_hcd 0000:38:00.4: Removing canceled TD starting at 0xff388ec0 (dma).
[  231.892656] xhci_hcd 0000:38:00.4: Removing canceled TD starting at 0xff388ed0 (dma).
[  231.892658] xhci_hcd 0000:38:00.4: Removing canceled TD starting at 0xff388ee0 (dma).
[  231.892661] xhci_hcd 0000:38:00.4: Removing canceled TD starting at 0xff388ef0 (dma).
[  231.892663] xhci_hcd 0000:38:00.4: Removing canceled TD starting at 0xff388f00 (dma).
[  231.892666] xhci_hcd 0000:38:00.4: Removing canceled TD starting at 0xff388f10 (dma).
[  231.892668] xhci_hcd 0000:38:00.4: Removing canceled TD starting at 0xff388f20 (dma).
[  231.892670] xhci_hcd 0000:38:00.4: Removing canceled TD starting at 0xff388f30 (dma).
[  231.892672] xhci_hcd 0000:38:00.4: Removing canceled TD starting at 0xff388f40 (dma).
[  231.892675] xhci_hcd 0000:38:00.4: Removing canceled TD starting at 0xff388e90 (dma).
[  231.892677] xhci_hcd 0000:38:00.4: Finding endpoint context
[  231.892679] xhci_hcd 0000:38:00.4: Cycle state = 0x1
[  231.892682] xhci_hcd 0000:38:00.4: New dequeue segment = 000000005d174923 (virtual)
[  231.892685] xhci_hcd 0000:38:00.4: New dequeue pointer = 0xff388ea0 (DMA)
[  231.892688] xhci_hcd 0000:38:00.4: Set TR Deq Ptr cmd, new deq seg = 000000005d174923 (0xff388000 dma), new deq ptr = 00000000d5c5ed2a (0xff388ea0 dma), new cycle = 1
[  231.892693] xhci_hcd 0000:38:00.4: // Ding dong!
[  231.892728] xhci_hcd 0000:38:00.4: Successful Set TR Deq Ptr cmd, deq = @ff388ea0
[  231.897107] xhci_hcd 0000:38:00.4: xhci_drop_endpoint called for udev 0000000043fc1c1f
[  231.897126] xhci_hcd 0000:38:00.4: drop ep 0x1, slot id 1, new drop flags = 0x4, new add flags = 0x0
[  231.897129] xhci_hcd 0000:38:00.4: xhci_check_bandwidth called for udev 0000000043fc1c1f
[  231.897137] xhci_hcd 0000:38:00.4: // Ding dong!
[  231.898523] xhci_hcd 0000:38:00.4: Successful Endpoint Configure command

I have uploaded the whole dmesg file and the tracing file to transfer.sh: https://transfer.sh/zYohl/dmesg and https://transfer.sh/KNbFL/xhci-trace

The issues occur since commit f8f80be501aa2f10669585c3e328fad079d8cb3a "xhci: Use soft retry to recover faster from transaction errors". I think this commit should be reverted at least until a workaround has been found, especially since the next two kernel versions will be used by a lot of distributions(5.4 because it's a LTS kernel and 5.5 will probably be used in Ubuntu 20.04) so more users would be affected by this.

- Bernhard


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Regression: USB/xhci issues on some systems with newer kernel versions
  2019-10-02 12:28 Regression: USB/xhci issues on some systems with newer kernel versions Bernhard Gebetsberger
@ 2019-10-03 10:23 ` Mathias Nyman
  2019-10-03 15:13   ` Bernhard Gebetsberger
  0 siblings, 1 reply; 5+ messages in thread
From: Mathias Nyman @ 2019-10-03 10:23 UTC (permalink / raw)
  To: Bernhard Gebetsberger, linux-usb

On 2.10.2019 15.28, Bernhard Gebetsberger wrote:
> Hi,
> 
> There has been a regression in the xhci driver since kernel version 4.20, on some systems some usb devices won't work until the system gets rebooted.
> The error message in dmesg is "WARN Set TR Deq Ptr cmd failed due to incorrect slot or ep state", although for some reason there are some usb devices that are affected by this issue but don't throw the error message(including the device I'm using, I got the error in previous kernel versions though).
> It seems like this bug can also lead to system instability, one user reported in the bug tracker(https://bugzilla.kernel.org/show_bug.cgi?id=202541#c58) that he got a system freeze because of this when using kernel 5.3.1.
> 

Ok, lets take a look at this.
Some of the symptoms vary a bit in the report, so lets focus on ones that
show: "WARN Set TR Deq Ptr cmd failed due to incorrect slot or ep state"

> When looking at the responses in the bug tracker, it looks like it mostly affects Ryzen based systems with 300 series motherboards, although there are some other affected systems as well. It doesn't only affect wifi/bluetooth sticks, some users even got this issue when connecting their smartphone or their external hard drive to their PC.

> 
> I have uploaded the whole dmesg file and the tracing file to transfer.sh: https://transfer.sh/zYohl/dmesg and https://transfer.sh/KNbFL/xhci-trace

Hmm, trying to download these just shows "Not Found"

Could someone with a affected system enable tracing and dynamic debug on a
recent kernel, take logs and traces of one failing instance where the message
"WARN Set TR Deq Ptr cmd failed due to incorrect slot or ep state" is seen.

mount -t debugfs none /sys/kernel/debug
echo 'module xhci_hcd =p' >/sys/kernel/debug/dynamic_debug/control
echo 'module usbcore =p' >/sys/kernel/debug/dynamic_debug/control
echo 81920 > /sys/kernel/debug/tracing/buffer_size_kb
echo 1 > /sys/kernel/debug/tracing/events/xhci-hcd/enable

< Trigger the issue >

Send output of dmesg
Send content of /sys/kernel/debug/tracing/trace

> 
> The issues occur since commit f8f80be501aa2f10669585c3e328fad079d8cb3a "xhci: Use soft retry to recover faster from transaction errors". I think this commit should be reverted at least until a workaround has been found, especially since the next two kernel versions will be used by a lot of distributions(5.4 because it's a LTS kernel and 5.5 will probably be used in Ubuntu 20.04) so more users would be affected by this.
> 

There some time left before 5.4 is out, lets see if we can find the root cause first.

-Mathias


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Regression: USB/xhci issues on some systems with newer kernel versions
  2019-10-03 10:23 ` Mathias Nyman
@ 2019-10-03 15:13   ` Bernhard Gebetsberger
  2019-10-11  1:55     ` Bernhard Gebetsberger
  2019-10-14 13:03     ` Mathias Nyman
  0 siblings, 2 replies; 5+ messages in thread
From: Bernhard Gebetsberger @ 2019-10-03 15:13 UTC (permalink / raw)
  To: Mathias Nyman, linux-usb

I sent the instructions to one of the users in the bug tracker.
Here is the download link for his logs: https://www.sendspace.com/file/413hlj

- Bernhard

Am 03.10.19 um 12:23 schrieb Mathias Nyman:
> On 2.10.2019 15.28, Bernhard Gebetsberger wrote:
>> Hi,
>>
>> There has been a regression in the xhci driver since kernel version 4.20, on some systems some usb devices won't work until the system gets rebooted.
>> The error message in dmesg is "WARN Set TR Deq Ptr cmd failed due to incorrect slot or ep state", although for some reason there are some usb devices that are affected by this issue but don't throw the error message(including the device I'm using, I got the error in previous kernel versions though).
>> It seems like this bug can also lead to system instability, one user reported in the bug tracker(https://bugzilla.kernel.org/show_bug.cgi?id=202541#c58) that he got a system freeze because of this when using kernel 5.3.1.
>>
>
> Ok, lets take a look at this.
> Some of the symptoms vary a bit in the report, so lets focus on ones that
> show: "WARN Set TR Deq Ptr cmd failed due to incorrect slot or ep state"
>
>> When looking at the responses in the bug tracker, it looks like it mostly affects Ryzen based systems with 300 series motherboards, although there are some other affected systems as well. It doesn't only affect wifi/bluetooth sticks, some users even got this issue when connecting their smartphone or their external hard drive to their PC.
>
>>
>> I have uploaded the whole dmesg file and the tracing file to transfer.sh: https://transfer.sh/zYohl/dmesg and https://transfer.sh/KNbFL/xhci-trace
>
> Hmm, trying to download these just shows "Not Found"
>
> Could someone with a affected system enable tracing and dynamic debug on a
> recent kernel, take logs and traces of one failing instance where the message
> "WARN Set TR Deq Ptr cmd failed due to incorrect slot or ep state" is seen.
>
> mount -t debugfs none /sys/kernel/debug
> echo 'module xhci_hcd =p' >/sys/kernel/debug/dynamic_debug/control
> echo 'module usbcore =p' >/sys/kernel/debug/dynamic_debug/control
> echo 81920 > /sys/kernel/debug/tracing/buffer_size_kb
> echo 1 > /sys/kernel/debug/tracing/events/xhci-hcd/enable
>
> < Trigger the issue >
>
> Send output of dmesg
> Send content of /sys/kernel/debug/tracing/trace
>
>>
>> The issues occur since commit f8f80be501aa2f10669585c3e328fad079d8cb3a "xhci: Use soft retry to recover faster from transaction errors". I think this commit should be reverted at least until a workaround has been found, especially since the next two kernel versions will be used by a lot of distributions(5.4 because it's a LTS kernel and 5.5 will probably be used in Ubuntu 20.04) so more users would be affected by this.
>>
>
> There some time left before 5.4 is out, lets see if we can find the root cause first.
>
> -Mathias
>

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Regression: USB/xhci issues on some systems with newer kernel versions
  2019-10-03 15:13   ` Bernhard Gebetsberger
@ 2019-10-11  1:55     ` Bernhard Gebetsberger
  2019-10-14 13:03     ` Mathias Nyman
  1 sibling, 0 replies; 5+ messages in thread
From: Bernhard Gebetsberger @ 2019-10-11  1:55 UTC (permalink / raw)
  To: Mathias Nyman, linux-usb

I've just noticed that this problem also occurs when unplugging an affected device.
When unplugging the device the error
    "WARN Set TR Deq Ptr cmd failed due to incorrect slot or ep state"
gets shown, even though I don't get this error when plugging the device in.

Here is a link to the dmesg and trace logs:
https://gist.github.com/Brn9hrd7/011405276fdf7a699dcc5cb83c67d276
maybe there is something useful in there that was missing in the previous logs.

- Bernhard


Am 03.10.19 um 17:13 schrieb Bernhard Gebetsberger:
> I sent the instructions to one of the users in the bug tracker.
> Here is the download link for his logs: https://www.sendspace.com/file/413hlj
>
> - Bernhard
>
> Am 03.10.19 um 12:23 schrieb Mathias Nyman:
>> On 2.10.2019 15.28, Bernhard Gebetsberger wrote:
>>> Hi,
>>>
>>> There has been a regression in the xhci driver since kernel version 4.20, on some systems some usb devices won't work until the system gets rebooted.
>>> The error message in dmesg is "WARN Set TR Deq Ptr cmd failed due to incorrect slot or ep state", although for some reason there are some usb devices that are affected by this issue but don't throw the error message(including the device I'm using, I got the error in previous kernel versions though).
>>> It seems like this bug can also lead to system instability, one user reported in the bug tracker(https://bugzilla.kernel.org/show_bug.cgi?id=202541#c58) that he got a system freeze because of this when using kernel 5.3.1.
>>>
>> Ok, lets take a look at this.
>> Some of the symptoms vary a bit in the report, so lets focus on ones that
>> show: "WARN Set TR Deq Ptr cmd failed due to incorrect slot or ep state"
>>
>>> When looking at the responses in the bug tracker, it looks like it mostly affects Ryzen based systems with 300 series motherboards, although there are some other affected systems as well. It doesn't only affect wifi/bluetooth sticks, some users even got this issue when connecting their smartphone or their external hard drive to their PC.
>>> I have uploaded the whole dmesg file and the tracing file to transfer.sh: https://transfer.sh/zYohl/dmesg and https://transfer.sh/KNbFL/xhci-trace
>> Hmm, trying to download these just shows "Not Found"
>>
>> Could someone with a affected system enable tracing and dynamic debug on a
>> recent kernel, take logs and traces of one failing instance where the message
>> "WARN Set TR Deq Ptr cmd failed due to incorrect slot or ep state" is seen.
>>
>> mount -t debugfs none /sys/kernel/debug
>> echo 'module xhci_hcd =p' >/sys/kernel/debug/dynamic_debug/control
>> echo 'module usbcore =p' >/sys/kernel/debug/dynamic_debug/control
>> echo 81920 > /sys/kernel/debug/tracing/buffer_size_kb
>> echo 1 > /sys/kernel/debug/tracing/events/xhci-hcd/enable
>>
>> < Trigger the issue >
>>
>> Send output of dmesg
>> Send content of /sys/kernel/debug/tracing/trace
>>
>>> The issues occur since commit f8f80be501aa2f10669585c3e328fad079d8cb3a "xhci: Use soft retry to recover faster from transaction errors". I think this commit should be reverted at least until a workaround has been found, especially since the next two kernel versions will be used by a lot of distributions(5.4 because it's a LTS kernel and 5.5 will probably be used in Ubuntu 20.04) so more users would be affected by this.
>>>
>> There some time left before 5.4 is out, lets see if we can find the root cause first.
>>
>> -Mathias
>>

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Regression: USB/xhci issues on some systems with newer kernel versions
  2019-10-03 15:13   ` Bernhard Gebetsberger
  2019-10-11  1:55     ` Bernhard Gebetsberger
@ 2019-10-14 13:03     ` Mathias Nyman
  1 sibling, 0 replies; 5+ messages in thread
From: Mathias Nyman @ 2019-10-14 13:03 UTC (permalink / raw)
  To: Bernhard Gebetsberger, linux-usb

On 3.10.2019 18.13, Bernhard Gebetsberger wrote:
> I sent the instructions to one of the users in the bug tracker.
> Here is the download link for his logs: https://www.sendspace.com/file/413hlj
> 

Thanks.
Traces show driver handles the Transaction error normally by issuing a endpoint reset,
which is successful, but after that there is no activity on that endpoint even if there
are over 120 transfers requests (TRB) pending.
After over 40 seconds the class driver starts canceling the pending transfers.

after soft retry the xhci driver should ring the doorbell of the endpoint, and hardware
should start processing pending TRBs, but ring is not handling pending TRBs
Looks like either driver or hardware fails to start the endpoint ring again

I'll add some more tracing to check driver correctly rings the endpoint doorbell.


Details of trace:

-Several TRBs (over 120) queued for slot 4 ep 3 (ep1in-bulk), starting at 0xff2d1000, up to 0xff2d1800 (0x10 per TRB)

   164.884097: xhci_urb_enqueue: ep1in-bulk: urb 000000005ebe7973 pipe 3221259648 slot 4 length 0/3860 sgs 0/0 stream 0 flags 00010200
   164.884099: xhci_queue_trb: BULK: Buffer 00000000f9e2304c length 3860 TD size 0 intr 0 type 'Normal' flags b:i:I:c:s:I:e:c
   164.884101: xhci_inc_enq: BULK 00000000be510b60: enq 0x00000000ff2d1010(0x00000000ff2d1000) deq 0x00000000ff2d1000(0x00000000ff2d1000)
   ...
   164.884304: xhci_urb_enqueue: ep1in-bulk: urb 00000000fee4e260 pipe 3221259648 slot 4 length 0/3860 sgs 0/0 stream 0 flags 00010200
   164.884304: xhci_queue_trb: BULK: Buffer 00000000ff3a304c length 3860 TD size 0 intr 0 type 'Normal' flags b:i:I:c:s:I:e:c
   164.884304: xhci_inc_enq: BULK 00000000be510b60: enq 0x00000000ff2d1800(0x00000000ff2d1000) deq 0x00000000ff2d1000(0x00000000ff2d1000)

-Transaction error 3 seconds later for TRB at 0xff2d1000

   167.578273: xhci_handle_event: EVENT: TRB 00000000ff2d1000 status 'USB Transaction Error' len 3860 slot 4 ep 3 type 'Transfer Event' flags e:c
   167.578288: xhci_handle_transfer: BULK: Buffer 00000000f9e2304c length 3860 TD size 0 intr 0 type 'Normal' flags b:i:I:c:s:I:e:C

-Soft retry by issuing a host side reset endpoint command,

   167.578297: xhci_queue_trb: CMD: Reset Endpoint Command: ctx 0000000000000000 slot 4 ep 3 flags C
   167.578416: xhci_handle_event: EVENT: TRB 00000000ffefe440 status 'Success' len 0 slot 4 ep 0 type 'Command Completion Event' flags e:c

-Host side of endpoint reset successful, endpoint is in stopped state as it should

   167.578417: xhci_handle_command: CMD: Reset Endpoint Command: ctx 0000000000000000 slot 4 ep 3 flags C
   167.578419: xhci_handle_cmd_reset_ep: State stopped mult 1 max P. Streams 0 interval 125 us max ESIT payload 0 CErr 3 Type Bulk IN burst

-Driver should ring endpoint doorbell, and hardware should continue procressing TRBs
No activity at all on slot 4 ep 3, other endpoints continue running normally.
Check driver really rang ep doorbell

A lot later class driver asks to cancel pending tranfer:

   214.132531: xhci_urb_dequeue: ep1in-bulk: urb 000000005ebe7973 pipe 3221259648 slot 4 length 0/3860 sgs 0/0 stream 0 flags 00010200
   214.132548: xhci_dbg_cancel_urb: Cancel URB 000000005ebe7973, dev 2, ep 0x81, starting at offset 0xff2d1000

-xhci driver tries to stop endpoint to cancel transfer:

   214.132555: xhci_queue_trb: CMD: Stop Ring Command: slot 4 sp 0 ep 3 flags C

-but it fails as slot is not in a proper state to be stopped, ep is in halted state after failed stop attempt.

   214.132679: xhci_handle_event: EVENT: TRB 00000000ffefe450 status 'Context State Error' len 0 slot 4 ep 0 type 'Command Completion Event' flags e:C
   214.132680: xhci_handle_command: CMD: Stop Ring Command: slot 4 sp 0 ep 3 flags C
   214.132682: xhci_handle_cmd_stop_ep: State halted mult 1 max P. Streams 0 interval 125 us max ESIT payload 0 CErr 3 Type Bulk IN burst 0 maxp 512

-After this endpoint stays in halted state, xhci driver fails to recover from this while canceling the reset of the TRBs
  
-Mathias

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, back to index

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-10-02 12:28 Regression: USB/xhci issues on some systems with newer kernel versions Bernhard Gebetsberger
2019-10-03 10:23 ` Mathias Nyman
2019-10-03 15:13   ` Bernhard Gebetsberger
2019-10-11  1:55     ` Bernhard Gebetsberger
2019-10-14 13:03     ` Mathias Nyman

Linux-USB Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-usb/0 linux-usb/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-usb linux-usb/ https://lore.kernel.org/linux-usb \
		linux-usb@vger.kernel.org
	public-inbox-index linux-usb

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-usb


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git