From: Greg KH <gregkh@linuxfoundation.org>
To: Thorsten Leemhuis <linux@leemhuis.info>
Cc: "regressions@lists.linux.dev" <regressions@lists.linux.dev>
Subject: Re: [REGRESSION] nvme: code command_id with a genctr for use-after-free validation crashes apple T2 SSD
Date: Mon, 4 Oct 2021 11:27:18 +0200 [thread overview]
Message-ID: <YVrI9rb4mnIs6RnF@kroah.com> (raw)
In-Reply-To: <637b0f3f-ac23-1e32-db7f-c696a04c1c73@leemhuis.info>
On Mon, Oct 04, 2021 at 11:17:21AM +0200, Thorsten Leemhuis wrote:
>
> On 26.09.21 07:59, Thorsten Leemhuis wrote:
> > On 25.09.21 15:10, Orlando Chamberlain wrote:
> >> Commit e7006de6c238 causes the SSD controller on Apple T2 computers to crash
> >> and prevents linux from booting.
> >>
> >> This commit implemented a counter that is stored within the NVMe command_id,
> >> however this counter makes the command_id higher than normal, causing a panic
> >> on the T2 security chip that functions as the SSD controller, which then
> >> causes the system to power off after a few seconds.
> >>
> >> This was reported on bugzilla here:
> >> https://bugzilla.kernel.org/show_bug.cgi?id=214509 but it was not originally
> >> classified as NVMe (when the report was created it was unknown what was
> >> causing it), so I don't know if it notified the NVMe mailing list when it
> >> was later reclassified to NVMe. Sorry if you've already seen this issue.
> >>
> >> The T2 security chip (which is the SSD) has this line in its crash log (the
> >> rest of this log is in an attachment on the bugzilla report):
> >>
> >> panic(cpu 1 caller 0xfffffff028d884ec): ANS2 Recoverable Panic - assert failed: [7447]:command id out of range error (cid = 4120), status_reg: 0x2000 - Null(2)
> >>
> >> This is the entry in lspci -nn for the ssd:
> >>
> >> 04:00.0 Mass storage controller [0180]: Apple Inc. ANS2 NVMe Controller [106b:2005] (rev 01)
> >>
> >> This commit was included in 5.14.6 and backported to 5.10.67, but does not
> >> occur in 5.14.5 and 5.10.66. I am on a MacBookPro16,1, the crash has been
> >> reproduced on a MacBookPro16,2 as well. I have been able to reproduce on Arch
> >> Linux with vanilla kernel 5.10.67 (others have gotten it on 5.14.6) with no
> >> DKMS modules, and I bisected it to that commit
> >> (e7006de6c23803799be000a5dcce4d916a36541a).
> >
> > Feel free to ignore this message. I write it to make regzbot track above
> > issue. Regzbot is the regression tracking bot I'm working on. It's still
> > in the early stages and this is still one of the first few regression I
> > make it track to get started and things tested in the field. That also
> > why I'm sending the mail just to the regressions list (it will do its
> > fully magic nevertheless). For details see:
> > https://linux-regtracking.leemhuis.info/post/inital-regzbot-running/
> > https://linux-regtracking.leemhuis.info/post/regzbot-approach/
> >
> > #regzbot ^introduced e7006de6c23803799be000a5dcce4d916a36541a
> >
> > #regzbot monitor
> > https://lore.kernel.org/lkml/CAHk-=wgML11x9afCvmg9yhVm9wi5mvnjBvmX+i7OfMA0Vd4FWA@mail.gmail.com/
>
> FWIW, this is just for the record: the fix for this landed in mainline,
> but didn't refer to this thread or the one montitored, hence I need to
> write this mail to make regzbot mark this regression as resolved:
>
> #regzbot monitor
> https://lore.kernel.org/all/20210927154306.387437-1-kbusch@kernel.org/
Thanks for tracking this, I'll go queue it up for 5.10.y and 5.14.y now.
greg k-h
next prev parent reply other threads:[~2021-10-04 9:27 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-09-25 13:10 [REGRESSION] nvme: code command_id with a genctr for use-after-free validation crashes apple T2 SSD Orlando Chamberlain
2021-09-25 17:16 ` Keith Busch
2021-09-25 23:40 ` Orlando Chamberlain
2021-09-26 2:08 ` Keith Busch
2021-09-26 3:53 ` Orlando Chamberlain
2021-09-26 4:35 ` Orlando Chamberlain
[not found] ` <PNZPR01MB4415801C6084E8CFD068A84AB8A69@PNZPR01MB4415.INDPRD01.PROD.OUTLOOK.COM>
2021-09-26 8:44 ` Sagi Grimberg
2021-09-27 4:22 ` Orlando Chamberlain
2021-09-27 4:51 ` Aditya Garg
2021-09-27 6:05 ` Sven Peter
2021-09-27 15:02 ` Keith Busch
2021-09-26 5:59 ` Thorsten Leemhuis
2021-10-04 9:17 ` Thorsten Leemhuis
2021-10-04 9:27 ` Greg KH [this message]
2021-10-04 10:11 ` Thorsten Leemhuis
2021-10-04 11:36 ` Greg KH
2021-10-05 5:50 ` Thorsten Leemhuis
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YVrI9rb4mnIs6RnF@kroah.com \
--to=gregkh@linuxfoundation.org \
--cc=linux@leemhuis.info \
--cc=regressions@lists.linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).