Regression: "xhci: Fix command ring pointer corruption while aborting a command" breaks USB on Intel controllers

Regression: "xhci: Fix command ring pointer corruption while aborting a command" breaks USB on Intel controllers

[Hans de Goede]

Hi All,

There are several bug reports (arch and Fedora) about USB problems
starting with kernel 5.14.14 (5.14.13 is ok):

https://bugzilla.redhat.com/show_bug.cgi?id=2019788
https://bbs.archlinux.org/viewtopic.php?pid=2006862

And 5.16.6, which has the hub address0_mutex fixes does not
fix these problems for some users.

Looking at the history between those 2 commit ff0e50d3564f
("xhci: Fix command ring pointer corruption while aborting a command"):
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ff0e50d3564f

stood out to me as a possible cause of this. So I've build
a test 5.15.6 kernel for Fedora users with that commit reverted
and 2 users have now reported that this fixes things for them
(see https://bugzilla.redhat.com/show_bug.cgi?id=2019788).

The reason why this stood out to me is because doing a 32 bit
write over a possibly 64 bit databus to the xHCI controller may
result in the hw doing a 64 bit read + modify 32 bit + 64 bit write,
so I think that the following is happening after the commit:

sw: read 32 bit
hw: read 64 bit, return 32 bit
sw: modify it
sw: write 32 bit
hw: read 64 bit
hw: modify 32 bit of 64 bit wor5d
hw: write 64 bit

Which actually makes the chances of hitting the problem the commit
tries to fix larger on controllers using a 64 bit data bus.

Note this is just a theory, but it seems plausible to me.

All problem reports are people using integrated Intel xHCI controllers
which I believe are likely to use a 64 bit data-bus.

Regards,

Hans

Re: Regression: "xhci: Fix command ring pointer corruption while aborting a command" breaks USB on Intel controllers

[Thorsten Leemhuis]

Hi, this is your Linux kernel regression tracker speaking.

On 04.12.21 15:15, Hans de Goede wrote:
> Hi All,
> 
> There are several bug reports (arch and Fedora) about USB problems
> starting with kernel 5.14.14 (5.14.13 is ok):
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=2019788
> https://bbs.archlinux.org/viewtopic.php?pid=2006862
> 
> And 5.16.6, which has the hub address0_mutex fixes does not
> fix these problems for some users.
> 
> Looking at the history between those 2 commit ff0e50d3564f
> ("xhci: Fix command ring pointer corruption while aborting a command"):
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ff0e50d3564f
> 
> stood out to me as a possible cause of this. So I've build
> a test 5.15.6 kernel for Fedora users with that commit reverted
> and 2 users have now reported that this fixes things for them
> (see https://bugzilla.redhat.com/show_bug.cgi?id=2019788).
> 
> The reason why this stood out to me is because doing a 32 bit
> write over a possibly 64 bit databus to the xHCI controller may
> result in the hw doing a 64 bit read + modify 32 bit + 64 bit write,
> so I think that the following is happening after the commit:
> 
> sw: read 32 bit
> hw: read 64 bit, return 32 bit
> sw: modify it
> sw: write 32 bit
> hw: read 64 bit
> hw: modify 32 bit of 64 bit wor5d
> hw: write 64 bit
> 
> Which actually makes the chances of hitting the problem the commit
> tries to fix larger on controllers using a 64 bit data bus.
> 
> Note this is just a theory, but it seems plausible to me.
> 
> All problem reports are people using integrated Intel xHCI controllers
> which I believe are likely to use a 64 bit data-bus.

FWIW, the commit is known to cause problems, one of them is found here:

https://lore.kernel.org/lkml/35f7428b39f996c793f5b4a6a314772681c73d7a.camel@apache.org/

This commit should improve the situation (and afaics hopefully fix all
problems related to it):

https://lore.kernel.org/stable/20211126122340.1193239-2-mathias.nyman@linux.intel.com/

Greg wants to sent it upstream today:

https://lore.kernel.org/lkml/YatGfEzzniH%2FSrn4@kroah.com/

Guess he soon after will backport it to the stable trees as well.

HTH, Ciao, Thorsten

#regzbot ^backmonitor
https://lore.kernel.org/lkml/35f7428b39f996c793f5b4a6a314772681c73d7a.camel@apache.org/

Re: Regression: "xhci: Fix command ring pointer corruption while aborting a command" breaks USB on Intel controllers

[Greg Kroah-Hartman]

On Sat, Dec 04, 2021 at 03:15:15PM +0100, Hans de Goede wrote:
> Hi All,
> 
> There are several bug reports (arch and Fedora) about USB problems
> starting with kernel 5.14.14 (5.14.13 is ok):
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=2019788
> https://bbs.archlinux.org/viewtopic.php?pid=2006862
> 
> And 5.16.6, which has the hub address0_mutex fixes does not
> fix these problems for some users.
> 
> Looking at the history between those 2 commit ff0e50d3564f
> ("xhci: Fix command ring pointer corruption while aborting a command"):
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ff0e50d3564f
> 
> stood out to me as a possible cause of this. So I've build
> a test 5.15.6 kernel for Fedora users with that commit reverted
> and 2 users have now reported that this fixes things for them
> (see https://bugzilla.redhat.com/show_bug.cgi?id=2019788).
> 
> The reason why this stood out to me is because doing a 32 bit
> write over a possibly 64 bit databus to the xHCI controller may
> result in the hw doing a 64 bit read + modify 32 bit + 64 bit write,
> so I think that the following is happening after the commit:
> 
> sw: read 32 bit
> hw: read 64 bit, return 32 bit
> sw: modify it
> sw: write 32 bit
> hw: read 64 bit
> hw: modify 32 bit of 64 bit wor5d
> hw: write 64 bit
> 
> Which actually makes the chances of hitting the problem the commit
> tries to fix larger on controllers using a 64 bit data bus.
> 
> Note this is just a theory, but it seems plausible to me.
> 
> All problem reports are people using integrated Intel xHCI controllers
> which I believe are likely to use a 64 bit data-bus.

I think this is fixed by 09f736aa9547 ("xhci: Fix commad ring abort,
write all 64 bits to CRCR register.") in linux-next and I'll be sending
it to Linus in a few hours.

thanks,

greg k-h

Re: Regression: "xhci: Fix command ring pointer corruption while aborting a command" breaks USB on Intel controllers

[Greg Kroah-Hartman]

On Sun, Dec 05, 2021 at 07:46:39AM +0100, Thorsten Leemhuis wrote:
> Hi, this is your Linux kernel regression tracker speaking.
> 
> On 04.12.21 15:15, Hans de Goede wrote:
> > Hi All,
> > 
> > There are several bug reports (arch and Fedora) about USB problems
> > starting with kernel 5.14.14 (5.14.13 is ok):
> > 
> > https://bugzilla.redhat.com/show_bug.cgi?id=2019788
> > https://bbs.archlinux.org/viewtopic.php?pid=2006862
> > 
> > And 5.16.6, which has the hub address0_mutex fixes does not
> > fix these problems for some users.
> > 
> > Looking at the history between those 2 commit ff0e50d3564f
> > ("xhci: Fix command ring pointer corruption while aborting a command"):
> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ff0e50d3564f
> > 
> > stood out to me as a possible cause of this. So I've build
> > a test 5.15.6 kernel for Fedora users with that commit reverted
> > and 2 users have now reported that this fixes things for them
> > (see https://bugzilla.redhat.com/show_bug.cgi?id=2019788).
> > 
> > The reason why this stood out to me is because doing a 32 bit
> > write over a possibly 64 bit databus to the xHCI controller may
> > result in the hw doing a 64 bit read + modify 32 bit + 64 bit write,
> > so I think that the following is happening after the commit:
> > 
> > sw: read 32 bit
> > hw: read 64 bit, return 32 bit
> > sw: modify it
> > sw: write 32 bit
> > hw: read 64 bit
> > hw: modify 32 bit of 64 bit wor5d
> > hw: write 64 bit
> > 
> > Which actually makes the chances of hitting the problem the commit
> > tries to fix larger on controllers using a 64 bit data bus.
> > 
> > Note this is just a theory, but it seems plausible to me.
> > 
> > All problem reports are people using integrated Intel xHCI controllers
> > which I believe are likely to use a 64 bit data-bus.
> 
> FWIW, the commit is known to cause problems, one of them is found here:
> 
> https://lore.kernel.org/lkml/35f7428b39f996c793f5b4a6a314772681c73d7a.camel@apache.org/
> 
> This commit should improve the situation (and afaics hopefully fix all
> problems related to it):
> 
> https://lore.kernel.org/stable/20211126122340.1193239-2-mathias.nyman@linux.intel.com/
> 
> Greg wants to sent it upstream today:
> 
> https://lore.kernel.org/lkml/YatGfEzzniH%2FSrn4@kroah.com/
> 
> Guess he soon after will backport it to the stable trees as well.
> 
> HTH, Ciao, Thorsten
> 
> #regzbot ^backmonitor
> https://lore.kernel.org/lkml/35f7428b39f996c793f5b4a6a314772681c73d7a.camel@apache.org/

Ah, I should have read the whole thread, yes, this should be the fix for
this.

thanks,

greg k-h

Re: Regression: "xhci: Fix command ring pointer corruption while aborting a command" breaks USB on Intel controllers

[Greg Kroah-Hartman]

On Sun, Dec 05, 2021 at 09:20:00AM +0100, Greg Kroah-Hartman wrote:
> On Sat, Dec 04, 2021 at 03:15:15PM +0100, Hans de Goede wrote:
> > Hi All,
> > 
> > There are several bug reports (arch and Fedora) about USB problems
> > starting with kernel 5.14.14 (5.14.13 is ok):
> > 
> > https://bugzilla.redhat.com/show_bug.cgi?id=2019788
> > https://bbs.archlinux.org/viewtopic.php?pid=2006862
> > 
> > And 5.16.6, which has the hub address0_mutex fixes does not
> > fix these problems for some users.
> > 
> > Looking at the history between those 2 commit ff0e50d3564f
> > ("xhci: Fix command ring pointer corruption while aborting a command"):
> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ff0e50d3564f
> > 
> > stood out to me as a possible cause of this. So I've build
> > a test 5.15.6 kernel for Fedora users with that commit reverted
> > and 2 users have now reported that this fixes things for them
> > (see https://bugzilla.redhat.com/show_bug.cgi?id=2019788).
> > 
> > The reason why this stood out to me is because doing a 32 bit
> > write over a possibly 64 bit databus to the xHCI controller may
> > result in the hw doing a 64 bit read + modify 32 bit + 64 bit write,
> > so I think that the following is happening after the commit:
> > 
> > sw: read 32 bit
> > hw: read 64 bit, return 32 bit
> > sw: modify it
> > sw: write 32 bit
> > hw: read 64 bit
> > hw: modify 32 bit of 64 bit wor5d
> > hw: write 64 bit
> > 
> > Which actually makes the chances of hitting the problem the commit
> > tries to fix larger on controllers using a 64 bit data bus.
> > 
> > Note this is just a theory, but it seems plausible to me.
> > 
> > All problem reports are people using integrated Intel xHCI controllers
> > which I believe are likely to use a 64 bit data-bus.
> 
> I think this is fixed by 09f736aa9547 ("xhci: Fix commad ring abort,
> write all 64 bits to CRCR register.") in linux-next and I'll be sending
> it to Linus in a few hours.

Pull request is here:
	https://lore.kernel.org/r/Yayq/Xdb/pHSS7/l@kroah.com

Re: Regression: "xhci: Fix command ring pointer corruption while aborting a command" breaks USB on Intel controllers

[Tobias Bachmann]

On Sun, Dec 05, 2021, 13:48:18 CET Greg Kroah-Hartman wrote:
> > I think this is fixed by 09f736aa9547 ("xhci: Fix commad ring abort,
> > write all 64 bits to CRCR register.") in linux-next and I'll be sending
> > it to Linus in a few hours.
>
> Pull request is here:
> 	https://lore.kernel.org/r/Yayq/Xdb/pHSS7/l@kroah.com

As one of the affected users, I can confirm that my issue is resolved
in 5.16-rc4, which includes that pull request.