All of lore.kernel.org
 help / color / mirror / Atom feed
From: Orlando Chamberlain <redecorating@protonmail.com>
To: "linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>
Cc: "regressions@lists.linux.dev" <regressions@lists.linux.dev>,
	"gargaditya08@live.com" <gargaditya08@live.com>,
	"kbusch@kernel.org" <kbusch@kernel.org>,
	"sagi@grimberg.me" <sagi@grimberg.me>,
	"hare@suse.de" <hare@suse.de>,
	"dwagner@suse.de" <dwagner@suse.de>, "hch@lst.de" <hch@lst.de>
Subject: [REGRESSION] nvme: code command_id with a genctr for use-after-free validation crashes apple T2 SSD
Date: Sat, 25 Sep 2021 13:10:42 +0000	[thread overview]
Message-ID: <cjJiSFV77WM51ciS8EuBcdeBcv9T83PUB-Kw3yi8PuC_LwrrUUnQ3w5RC1PbKvSYE72KryXp3wOJhv4Ov_WWIe2gKWOOo5uwuUjbbFA8HDM=@protonmail.com> (raw)

[-- Attachment #1: Type: text/plain, Size: 2979 bytes --]

Commit e7006de6c238 causes the SSD controller on Apple T2 computers to crash
and prevents linux from booting.

This commit implemented a counter that is stored within the NVMe command_id,
however this counter makes the command_id higher than normal, causing a panic
on the T2 security chip that functions as the SSD controller, which then
causes the system to power off after a few seconds.

This was reported on bugzilla here:
https://bugzilla.kernel.org/show_bug.cgi?id=214509 but it was not originally
classified as NVMe (when the report was created it was unknown what was
causing it), so I don't know if it notified the NVMe mailing list when it
was later reclassified to NVMe. Sorry if you've already seen this issue.

The T2 security chip (which is the SSD) has this line in its crash log (the
rest of this log is in an attachment on the bugzilla report):

panic(cpu 1 caller 0xfffffff028d884ec): ANS2 Recoverable Panic - assert failed: [7447]:command id out of range error (cid = 4120), status_reg: 0x2000 - Null(2)

This is the entry in lspci -nn for the ssd:

04:00.0 Mass storage controller [0180]: Apple Inc. ANS2 NVMe Controller [106b:2005] (rev 01)

This commit was included in 5.14.6 and backported to 5.10.67, but does not
occur in 5.14.5 and 5.10.66. I am on a MacBookPro16,1, the crash has been
reproduced on a MacBookPro16,2 as well. I have been able to reproduce on Arch
Linux with vanilla kernel 5.10.67 (others have gotten it on 5.14.6) with no
DKMS modules, and I bisected it to that commit
(e7006de6c23803799be000a5dcce4d916a36541a).

I've tried to modify the genctr so that it is in the other side of the
command_id (which I thought might make the command_id's lower) with the patch
below, but it did not prevent the crash.

Regards,
Orlando Chamberlain

--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -489,17 +489,20 @@ struct nvme_ctrl_ops {

 /*
  * nvme command_id is constructed as such:
- * | xxxx | xxxxxxxxxxxx |
- *   gen    request tag
- */
+ * | xxxxxxxxxxxx | xxxx |
+ *   request tag    gen
+ *
+ * The gen is at the end as the SSD in Apple T2 Computers
+ * crashes when the command_id is too high.
+*/
 #define nvme_genctr_mask(gen)                  (gen & 0xf)
-#define nvme_cid_install_genctr(gen)           (nvme_genctr_mask(gen) << 12)
-#define nvme_genctr_from_cid(cid)              ((cid & 0xf000) >> 12)
-#define nvme_tag_from_cid(cid)                 (cid & 0xfff)
+#define nvme_cid_install_genctr(gen)           (nvme_genctr_mask(gen))
+#define nvme_genctr_from_cid(cid)              ((cid & 0x000f))
+#define nvme_tag_from_cid(cid)                 ((cid & 0xfff0) >> 4)

 static inline u16 nvme_cid(struct request *rq)
 {
-       return nvme_cid_install_genctr(nvme_req(rq)->genctr) | rq->tag;
+       return nvme_cid_install_genctr(nvme_req(rq)->genctr) | (rq->tag << 4);
 }

 static inline struct request *nvme_find_rq(struct blk_mq_tags *tags,

[-- Attachment #2: publickey - redecorating@protonmail.com - 0xEE1BCCD7.asc --]
[-- Type: application/pgp-keys, Size: 1873 bytes --]

[-- Attachment #3: Type: text/plain, Size: 158 bytes --]

_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

WARNING: multiple messages have this Message-ID
From: Orlando Chamberlain <redecorating@protonmail.com>
To: "linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>
Cc: "regressions@lists.linux.dev" <regressions@lists.linux.dev>,
	"gargaditya08@live.com" <gargaditya08@live.com>,
	"kbusch@kernel.org" <kbusch@kernel.org>,
	"sagi@grimberg.me" <sagi@grimberg.me>,
	"hare@suse.de" <hare@suse.de>,
	"dwagner@suse.de" <dwagner@suse.de>, "hch@lst.de" <hch@lst.de>
Subject: [REGRESSION] nvme: code command_id with a genctr for use-after-free validation crashes apple T2 SSD
Date: Sat, 25 Sep 2021 13:10:42 +0000	[thread overview]
Message-ID: <cjJiSFV77WM51ciS8EuBcdeBcv9T83PUB-Kw3yi8PuC_LwrrUUnQ3w5RC1PbKvSYE72KryXp3wOJhv4Ov_WWIe2gKWOOo5uwuUjbbFA8HDM=@protonmail.com> (raw)

[-- Attachment #1: Type: text/plain, Size: 2979 bytes --]

Commit e7006de6c238 causes the SSD controller on Apple T2 computers to crash
and prevents linux from booting.

This commit implemented a counter that is stored within the NVMe command_id,
however this counter makes the command_id higher than normal, causing a panic
on the T2 security chip that functions as the SSD controller, which then
causes the system to power off after a few seconds.

This was reported on bugzilla here:
https://bugzilla.kernel.org/show_bug.cgi?id=214509 but it was not originally
classified as NVMe (when the report was created it was unknown what was
causing it), so I don't know if it notified the NVMe mailing list when it
was later reclassified to NVMe. Sorry if you've already seen this issue.

The T2 security chip (which is the SSD) has this line in its crash log (the
rest of this log is in an attachment on the bugzilla report):

panic(cpu 1 caller 0xfffffff028d884ec): ANS2 Recoverable Panic - assert failed: [7447]:command id out of range error (cid = 4120), status_reg: 0x2000 - Null(2)

This is the entry in lspci -nn for the ssd:

04:00.0 Mass storage controller [0180]: Apple Inc. ANS2 NVMe Controller [106b:2005] (rev 01)

This commit was included in 5.14.6 and backported to 5.10.67, but does not
occur in 5.14.5 and 5.10.66. I am on a MacBookPro16,1, the crash has been
reproduced on a MacBookPro16,2 as well. I have been able to reproduce on Arch
Linux with vanilla kernel 5.10.67 (others have gotten it on 5.14.6) with no
DKMS modules, and I bisected it to that commit
(e7006de6c23803799be000a5dcce4d916a36541a).

I've tried to modify the genctr so that it is in the other side of the
command_id (which I thought might make the command_id's lower) with the patch
below, but it did not prevent the crash.

Regards,
Orlando Chamberlain

--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -489,17 +489,20 @@ struct nvme_ctrl_ops {

 /*
  * nvme command_id is constructed as such:
- * | xxxx | xxxxxxxxxxxx |
- *   gen    request tag
- */
+ * | xxxxxxxxxxxx | xxxx |
+ *   request tag    gen
+ *
+ * The gen is at the end as the SSD in Apple T2 Computers
+ * crashes when the command_id is too high.
+*/
 #define nvme_genctr_mask(gen)                  (gen & 0xf)
-#define nvme_cid_install_genctr(gen)           (nvme_genctr_mask(gen) << 12)
-#define nvme_genctr_from_cid(cid)              ((cid & 0xf000) >> 12)
-#define nvme_tag_from_cid(cid)                 (cid & 0xfff)
+#define nvme_cid_install_genctr(gen)           (nvme_genctr_mask(gen))
+#define nvme_genctr_from_cid(cid)              ((cid & 0x000f))
+#define nvme_tag_from_cid(cid)                 ((cid & 0xfff0) >> 4)

 static inline u16 nvme_cid(struct request *rq)
 {
-       return nvme_cid_install_genctr(nvme_req(rq)->genctr) | rq->tag;
+       return nvme_cid_install_genctr(nvme_req(rq)->genctr) | (rq->tag << 4);
 }

 static inline struct request *nvme_find_rq(struct blk_mq_tags *tags,

[-- Attachment #2: publickey - redecorating@protonmail.com - 0xEE1BCCD7.asc --]
[-- Type: application/pgp-keys, Size: 1873 bytes --]

             reply	other threads:[~2021-09-25 13:11 UTC|newest]

Thread overview: 28+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-09-25 13:10 Orlando Chamberlain [this message]
2021-09-25 13:10 ` Orlando Chamberlain
2021-09-25 17:16 ` Keith Busch
2021-09-25 17:16   ` Keith Busch
2021-09-25 23:40   ` Orlando Chamberlain
2021-09-25 23:40     ` Orlando Chamberlain
2021-09-26  2:08     ` Keith Busch
2021-09-26  2:08       ` Keith Busch
2021-09-26  3:53       ` Orlando Chamberlain
2021-09-26  3:53         ` Orlando Chamberlain
2021-09-26  4:35         ` Orlando Chamberlain
2021-09-26  4:35           ` Orlando Chamberlain
     [not found]           ` <PNZPR01MB4415801C6084E8CFD068A84AB8A69@PNZPR01MB4415.INDPRD01.PROD.OUTLOOK.COM>
2021-09-26  8:44             ` Sagi Grimberg
2021-09-26  8:44               ` Sagi Grimberg
2021-09-27  4:22               ` Orlando Chamberlain
2021-09-27  4:22                 ` Orlando Chamberlain
2021-09-27  4:51                 ` Aditya Garg
2021-09-27  4:51                   ` Aditya Garg
2021-09-27  6:05                   ` Sven Peter
2021-09-27  6:05                     ` Sven Peter
2021-09-27 15:02                     ` Keith Busch
2021-09-27 15:02                       ` Keith Busch
2021-09-26  5:59 ` Thorsten Leemhuis
2021-10-04  9:17   ` Thorsten Leemhuis
2021-10-04  9:27     ` Greg KH
2021-10-04 10:11       ` Thorsten Leemhuis
2021-10-04 11:36         ` Greg KH
2021-10-05  5:50           ` Thorsten Leemhuis

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='cjJiSFV77WM51ciS8EuBcdeBcv9T83PUB-Kw3yi8PuC_LwrrUUnQ3w5RC1PbKvSYE72KryXp3wOJhv4Ov_WWIe2gKWOOo5uwuUjbbFA8HDM=@protonmail.com' \
    --to=redecorating@protonmail.com \
    --cc=dwagner@suse.de \
    --cc=gargaditya08@live.com \
    --cc=hare@suse.de \
    --cc=hch@lst.de \
    --cc=kbusch@kernel.org \
    --cc=linux-nvme@lists.infradead.org \
    --cc=regressions@lists.linux.dev \
    --cc=sagi@grimberg.me \
    --subject='Re: [REGRESSION] nvme: code command_id with a genctr for use-after-free validation crashes apple T2 SSD' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.