From: Ben Widawsky <ben.widawsky@intel.com>
To: Dan Williams <dan.j.williams@intel.com>
Cc: David Rientjes <rientjes@google.com>,
linux-cxl@vger.kernel.org,
Linux ACPI <linux-acpi@vger.kernel.org>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
linux-nvdimm <linux-nvdimm@lists.01.org>,
Linux PCI <linux-pci@vger.kernel.org>,
Bjorn Helgaas <helgaas@kernel.org>,
Chris Browy <cbrowy@avery-design.com>,
Christoph Hellwig <hch@infradead.org>,
Ira Weiny <ira.weiny@intel.com>, Jon Masters <jcm@jonmasters.org>,
Jonathan Cameron <Jonathan.Cameron@huawei.com>,
Rafael Wysocki <rafael.j.wysocki@intel.com>,
Randy Dunlap <rdunlap@infradead.org>,
Vishal Verma <vishal.l.verma@intel.com>,
daniel.lll@alibaba-inc.com,
"John Groves (jgroves)" <jgroves@micron.com>,
"Kelley, Sean V" <sean.v.kelley@intel.com>
Subject: Re: [PATCH 04/14] cxl/mem: Implement polled mode mailbox
Date: Tue, 2 Feb 2021 16:54:40 -0800 [thread overview]
Message-ID: <20210203005440.yyyphe4yigf3fvkc@intel.com> (raw)
In-Reply-To: <CAPcyv4gPbN74wOzPH5-qC6a1V5ZwXUVedtr0ZJmfP8DA13YWnA@mail.gmail.com>
On 21-02-02 15:54:03, Dan Williams wrote:
> On Tue, Feb 2, 2021 at 2:57 PM Ben Widawsky <ben.widawsky@intel.com> wrote:
> >
> > On 21-02-01 12:00:18, Dan Williams wrote:
> > > On Sat, Jan 30, 2021 at 3:52 PM David Rientjes <rientjes@google.com> wrote:
> > > >
> > > > On Fri, 29 Jan 2021, Ben Widawsky wrote:
> > > >
> > > > > Provide enough functionality to utilize the mailbox of a memory device.
> > > > > The mailbox is used to interact with the firmware running on the memory
> > > > > device.
> > > > >
> > > > > The CXL specification defines separate capabilities for the mailbox and
> > > > > the memory device. The mailbox interface has a doorbell to indicate
> > > > > ready to accept commands and the memory device has a capability register
> > > > > that indicates the mailbox interface is ready. The expectation is that
> > > > > the doorbell-ready is always later than the memory-device-indication
> > > > > that the mailbox is ready.
> > > > >
> > > > > Create a function to handle sending a command, optionally with a
> > > > > payload, to the memory device, polling on a result, and then optionally
> > > > > copying out the payload. The algorithm for doing this comes straight out
> > > > > of the CXL 2.0 specification.
> > > > >
> > > > > Primary mailboxes are capable of generating an interrupt when submitting
> > > > > a command in the background. That implementation is saved for a later
> > > > > time.
> > > > >
> > > > > Secondary mailboxes aren't implemented at this time.
> > > > >
> > > > > The flow is proven with one implemented command, "identify". Because the
> > > > > class code has already told the driver this is a memory device and the
> > > > > identify command is mandatory.
> > > > >
> > > > > Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
> > > > > ---
> > > > > drivers/cxl/Kconfig | 14 ++
> > > > > drivers/cxl/cxl.h | 39 +++++
> > > > > drivers/cxl/mem.c | 342 +++++++++++++++++++++++++++++++++++++++++++-
> > > > > 3 files changed, 394 insertions(+), 1 deletion(-)
> > > > >
> > > > > diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig
> > > > > index 3b66b46af8a0..fe591f74af96 100644
> > > > > --- a/drivers/cxl/Kconfig
> > > > > +++ b/drivers/cxl/Kconfig
> > > > > @@ -32,4 +32,18 @@ config CXL_MEM
> > > > > Chapter 2.3 Type 3 CXL Device in the CXL 2.0 specification.
> > > > >
> > > > > If unsure say 'm'.
> > > > > +
> > > > > +config CXL_MEM_INSECURE_DEBUG
> > > > > + bool "CXL.mem debugging"
> > > > > + depends on CXL_MEM
> > > > > + help
> > > > > + Enable debug of all CXL command payloads.
> > > > > +
> > > > > + Some CXL devices and controllers support encryption and other
> > > > > + security features. The payloads for the commands that enable
> > > > > + those features may contain sensitive clear-text security
> > > > > + material. Disable debug of those command payloads by default.
> > > > > + If you are a kernel developer actively working on CXL
> > > > > + security enabling say Y, otherwise say N.
> > > >
> > > > Not specific to this patch, but the reference to encryption made me
> > > > curious about integrity: are all CXL.mem devices compatible with DIMP?
> > > > Some? None?
> > >
> > > The encryption here is "device passphrase" similar to the NVDIMM
> > > Security Management described here:
> > >
> > > https://pmem.io/documents/IntelOptanePMem_DSM_Interface-V2.0.pdf
> > >
> > > The LIBNVDIMM enabling wrapped this support with the Linux keys
> > > interface which among other things enforces wrapping the clear text
> > > passphrase with a Linux "trusted/encrypted" key.
> > >
> > > Additionally, the CXL.io interface optionally supports PCI IDE:
> > >
> > > https://www.intel.com/content/dam/www/public/us/en/documents/reference-guides/pcie-device-security-enhancements.pdf
> > >
> > > I'm otherwise not familiar with the DIMP acronym?
> > >
> > > > > +
> > > > > endif
> > > > > diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> > > > > index a3da7f8050c4..df3d97154b63 100644
> > > > > --- a/drivers/cxl/cxl.h
> > > > > +++ b/drivers/cxl/cxl.h
> > > > > @@ -31,9 +31,36 @@
> > > > > #define CXLDEV_MB_CAPS_OFFSET 0x00
> > > > > #define CXLDEV_MB_CAP_PAYLOAD_SIZE_MASK GENMASK(4, 0)
> > > > > #define CXLDEV_MB_CTRL_OFFSET 0x04
> > > > > +#define CXLDEV_MB_CTRL_DOORBELL BIT(0)
> > > > > #define CXLDEV_MB_CMD_OFFSET 0x08
> > > > > +#define CXLDEV_MB_CMD_COMMAND_OPCODE_MASK GENMASK(15, 0)
> > > > > +#define CXLDEV_MB_CMD_PAYLOAD_LENGTH_MASK GENMASK(36, 16)
> > > > > #define CXLDEV_MB_STATUS_OFFSET 0x10
> > > > > +#define CXLDEV_MB_STATUS_RET_CODE_MASK GENMASK(47, 32)
> > > > > #define CXLDEV_MB_BG_CMD_STATUS_OFFSET 0x18
> > > > > +#define CXLDEV_MB_PAYLOAD_OFFSET 0x20
> > > > > +
> > > > > +/* Memory Device (CXL 2.0 - 8.2.8.5.1.1) */
> > > > > +#define CXLMDEV_STATUS_OFFSET 0x0
> > > > > +#define CXLMDEV_DEV_FATAL BIT(0)
> > > > > +#define CXLMDEV_FW_HALT BIT(1)
> > > > > +#define CXLMDEV_STATUS_MEDIA_STATUS_MASK GENMASK(3, 2)
> > > > > +#define CXLMDEV_MS_NOT_READY 0
> > > > > +#define CXLMDEV_MS_READY 1
> > > > > +#define CXLMDEV_MS_ERROR 2
> > > > > +#define CXLMDEV_MS_DISABLED 3
> > > > > +#define CXLMDEV_READY(status) \
> > > > > + (CXL_GET_FIELD(status, CXLMDEV_STATUS_MEDIA_STATUS) == CXLMDEV_MS_READY)
> > > > > +#define CXLMDEV_MBOX_IF_READY BIT(4)
> > > > > +#define CXLMDEV_RESET_NEEDED_SHIFT 5
> > > > > +#define CXLMDEV_RESET_NEEDED_MASK GENMASK(7, 5)
> > > > > +#define CXLMDEV_RESET_NEEDED_NOT 0
> > > > > +#define CXLMDEV_RESET_NEEDED_COLD 1
> > > > > +#define CXLMDEV_RESET_NEEDED_WARM 2
> > > > > +#define CXLMDEV_RESET_NEEDED_HOT 3
> > > > > +#define CXLMDEV_RESET_NEEDED_CXL 4
> > > > > +#define CXLMDEV_RESET_NEEDED(status) \
> > > > > + (CXL_GET_FIELD(status, CXLMDEV_RESET_NEEDED) != CXLMDEV_RESET_NEEDED_NOT)
> > > > >
> > > > > /**
> > > > > * struct cxl_mem - A CXL memory device
> > > > > @@ -44,6 +71,16 @@ struct cxl_mem {
> > > > > struct pci_dev *pdev;
> > > > > void __iomem *regs;
> > > > >
> > > > > + struct {
> > > > > + struct range range;
> > > > > + } pmem;
> > > > > +
> > > > > + struct {
> > > > > + struct range range;
> > > > > + } ram;
> > > > > +
> > > > > + char firmware_version[0x10];
> > > > > +
> > > > > /* Cap 0001h - CXL_CAP_CAP_ID_DEVICE_STATUS */
> > > > > struct {
> > > > > void __iomem *regs;
> > > > > @@ -51,6 +88,7 @@ struct cxl_mem {
> > > > >
> > > > > /* Cap 0002h - CXL_CAP_CAP_ID_PRIMARY_MAILBOX */
> > > > > struct {
> > > > > + struct mutex mutex; /* Protects device mailbox and firmware */
> > > > > void __iomem *regs;
> > > > > size_t payload_size;
> > > > > } mbox;
> > > > > @@ -89,5 +127,6 @@ struct cxl_mem {
> > > > >
> > > > > cxl_reg(status);
> > > > > cxl_reg(mbox);
> > > > > +cxl_reg(mem);
> > > > >
> > > > > #endif /* __CXL_H__ */
> > > > > diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
> > > > > index fa14d51243ee..69ed15bfa5d4 100644
> > > > > --- a/drivers/cxl/mem.c
> > > > > +++ b/drivers/cxl/mem.c
> > > > > @@ -6,6 +6,270 @@
> > > > > #include "pci.h"
> > > > > #include "cxl.h"
> > > > >
> > > > > +#define cxl_doorbell_busy(cxlm) \
> > > > > + (cxl_read_mbox_reg32(cxlm, CXLDEV_MB_CTRL_OFFSET) & \
> > > > > + CXLDEV_MB_CTRL_DOORBELL)
> > > > > +
> > > > > +#define CXL_MAILBOX_TIMEOUT_US 2000
> > > >
> > > > This should be _MS?
> > > >
> > > > > +
> > > > > +enum opcode {
> > > > > + CXL_MBOX_OP_IDENTIFY = 0x4000,
> > > > > + CXL_MBOX_OP_MAX = 0x10000
> > > > > +};
> > > > > +
> > > > > +/**
> > > > > + * struct mbox_cmd - A command to be submitted to hardware.
> > > > > + * @opcode: (input) The command set and command submitted to hardware.
> > > > > + * @payload_in: (input) Pointer to the input payload.
> > > > > + * @payload_out: (output) Pointer to the output payload. Must be allocated by
> > > > > + * the caller.
> > > > > + * @size_in: (input) Number of bytes to load from @payload.
> > > > > + * @size_out: (output) Number of bytes loaded into @payload.
> > > > > + * @return_code: (output) Error code returned from hardware.
> > > > > + *
> > > > > + * This is the primary mechanism used to send commands to the hardware.
> > > > > + * All the fields except @payload_* correspond exactly to the fields described in
> > > > > + * Command Register section of the CXL 2.0 spec (8.2.8.4.5). @payload_in and
> > > > > + * @payload_out are written to, and read from the Command Payload Registers
> > > > > + * defined in (8.2.8.4.8).
> > > > > + */
> > > > > +struct mbox_cmd {
> > > > > + u16 opcode;
> > > > > + void *payload_in;
> > > > > + void *payload_out;
> > > > > + size_t size_in;
> > > > > + size_t size_out;
> > > > > + u16 return_code;
> > > > > +#define CXL_MBOX_SUCCESS 0
> > > > > +};
> > > > > +
> > > > > +static int cxl_mem_wait_for_doorbell(struct cxl_mem *cxlm)
> > > > > +{
> > > > > + const int timeout = msecs_to_jiffies(CXL_MAILBOX_TIMEOUT_US);
> > > > > + const unsigned long start = jiffies;
> > > > > + unsigned long end = start;
> > > > > +
> > > > > + while (cxl_doorbell_busy(cxlm)) {
> > > > > + end = jiffies;
> > > > > +
> > > > > + if (time_after(end, start + timeout)) {
> > > > > + /* Check again in case preempted before timeout test */
> > > > > + if (!cxl_doorbell_busy(cxlm))
> > > > > + break;
> > > > > + return -ETIMEDOUT;
> > > > > + }
> > > > > + cpu_relax();
> > > > > + }
> > > > > +
> > > > > + dev_dbg(&cxlm->pdev->dev, "Doorbell wait took %dms",
> > > > > + jiffies_to_msecs(end) - jiffies_to_msecs(start));
> > > > > + return 0;
> > > > > +}
> > > > > +
> > > > > +static void cxl_mem_mbox_timeout(struct cxl_mem *cxlm,
> > > > > + struct mbox_cmd *mbox_cmd)
> > > > > +{
> > > > > + dev_warn(&cxlm->pdev->dev, "Mailbox command timed out\n");
> > > > > + dev_info(&cxlm->pdev->dev,
> > > > > + "\topcode: 0x%04x\n"
> > > > > + "\tpayload size: %zub\n",
> > > > > + mbox_cmd->opcode, mbox_cmd->size_in);
> > > > > +
> > > > > + if (IS_ENABLED(CONFIG_CXL_MEM_INSECURE_DEBUG)) {
> > > > > + print_hex_dump_debug("Payload ", DUMP_PREFIX_OFFSET, 16, 1,
> > > > > + mbox_cmd->payload_in, mbox_cmd->size_in,
> > > > > + true);
> > > > > + }
> > > > > +
> > > > > + /* Here's a good place to figure out if a device reset is needed */
> > > >
> > > > What are the implications if we don't do a reset, as this implementation
> > > > does not? IOW, does a timeout require a device to be recovered through a
> > > > reset before it can receive additional commands, or is it safe to simply
> > > > drop the command that timed out on the floor and proceed?
> > >
> > > Not a satisfying answer, but "it depends". It's also complicated by
> > > the fact that a reset may need to be coordinated with other devices in
> > > the interleave-set as the HDM decoders may bounce.
> > >
> > > For comparison, to date there have been no problems with the "drop on
> > > the floor" policy of LIBNVDIMM command timeouts. At the same time
> > > there simply was not a software visible reset mechanism for those
> > > devices so this problem never came out. This mailbox isn't a fast
> > > path, so the device is likely completely dead if this timeout is ever
> > > violated, and the firmware reporting a timeout might as well assume
> > > that the OS gives up on the device.
> > >
> > > I'll let Ben chime in on the rest...
> >
> > Reset handling is next on the TODO list for the driver. I had two main reasons
> > for not even taking a stab at it.
> > 1. I have no good way to test it. We are working on adding some test conditions
> > to QEMU for it.
> > 2. The main difficulty in my mind with reset is you can't pull the memory out
> > from under the OS here. While the driver doesn't yet handle persistent memory
> > capacities, it may have volatile capacity configured by the BIOS. So the goal
> > was, get the bits of the driver in that would at least allow developers,
> > hardware vendors, and folks contributing to the spec a way to have basic
> > interaction with a CXL type 3 device.
>
> Honestly I think in most cases if the firmware decides to return a
> "reset required" status the Linux response will be "lol, no" because
> the firmware has no concept of the violence that would impose on the
> rest of the system.
How about UAPI to initiate a reset? I think a sysfs bool would do the trick.
Maybe sysfs file to display current error status, and one to reset?
next prev parent reply other threads:[~2021-02-03 0:55 UTC|newest]
Thread overview: 96+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-01-30 0:24 [PATCH 00/14] CXL 2.0 Support Ben Widawsky
2021-01-30 0:24 ` [PATCH 01/14] cxl/mem: Introduce a driver for CXL-2.0-Type-3 endpoints Ben Widawsky
2021-01-30 23:51 ` David Rientjes
2021-02-01 17:21 ` Jonathan Cameron
2021-02-01 17:34 ` Konrad Rzeszutek Wilk
2021-02-02 17:58 ` Christoph Hellwig
2021-02-02 18:00 ` Christoph Hellwig
2021-01-30 0:24 ` [PATCH 02/14] cxl/mem: Map memory device registers Ben Widawsky
2021-01-30 23:51 ` David Rientjes
2021-02-01 16:46 ` Ben Widawsky
2021-02-01 18:19 ` Jonathan Cameron
2021-02-01 17:36 ` Konrad Rzeszutek Wilk
2021-02-02 18:04 ` Christoph Hellwig
2021-02-02 18:31 ` Ben Widawsky
2021-02-03 17:12 ` Christoph Hellwig
2021-01-30 0:24 ` [PATCH 03/14] cxl/mem: Find device capabilities Ben Widawsky
2021-01-30 23:51 ` David Rientjes
2021-02-01 16:53 ` Ben Widawsky
2021-02-01 21:51 ` David Rientjes
2021-02-01 21:58 ` Ben Widawsky
2021-02-01 22:23 ` David Rientjes
2021-02-01 22:28 ` Ben Widawsky
2021-02-01 22:33 ` Ben Widawsky
2021-02-01 22:45 ` David Rientjes
2021-02-01 22:50 ` Ben Widawsky
2021-02-01 23:09 ` David Rientjes
2021-02-01 23:17 ` Ben Widawsky
2021-02-01 23:58 ` David Rientjes
2021-02-02 0:11 ` Ben Widawsky
2021-02-02 0:14 ` Dan Williams
2021-02-02 1:09 ` David Rientjes
2021-02-01 22:02 ` Dan Williams
2021-02-01 17:41 ` Konrad Rzeszutek Wilk
2021-02-01 17:50 ` Ben Widawsky
2021-02-01 18:08 ` Konrad Rzeszutek Wilk
2021-02-02 18:10 ` Christoph Hellwig
2021-02-02 18:24 ` Ben Widawsky
2021-02-03 17:15 ` Christoph Hellwig
2021-02-03 17:23 ` Ben Widawsky
2021-02-03 21:23 ` Dan Williams
2021-02-04 7:16 ` Christoph Hellwig
2021-02-04 15:29 ` Ben Widawsky
2021-01-30 0:24 ` [PATCH 04/14] cxl/mem: Implement polled mode mailbox Ben Widawsky
2021-01-30 23:51 ` David Rientjes
2021-02-01 20:00 ` Dan Williams
2021-02-02 22:57 ` Ben Widawsky
2021-02-02 23:54 ` Dan Williams
2021-02-03 0:54 ` Ben Widawsky [this message]
2021-02-02 22:50 ` Ben Widawsky
2021-02-01 17:54 ` Konrad Rzeszutek Wilk
2021-02-01 19:13 ` Ben Widawsky
2021-02-01 19:28 ` Dan Williams
[not found] ` <SN6PR08MB46052FE9BC20A747CACD8F50D1B39@SN6PR08MB4605.namprd08.prod.outlook.com>
2021-02-04 22:24 ` [EXT] " Ben Widawsky
2021-01-30 0:24 ` [PATCH 05/14] cxl/mem: Register CXL memX devices Ben Widawsky
2021-01-30 0:31 ` Dan Williams
2021-01-30 23:52 ` David Rientjes
2021-02-01 17:10 ` Ben Widawsky
2021-02-01 21:53 ` David Rientjes
2021-02-01 21:55 ` Dan Williams
2021-02-02 18:13 ` Christoph Hellwig
2021-01-30 0:24 ` [PATCH 06/14] cxl/mem: Add basic IOCTL interface Ben Widawsky
2021-02-02 18:15 ` Christoph Hellwig
2021-02-02 18:33 ` Ben Widawsky
2021-01-30 0:24 ` [PATCH 07/14] cxl/mem: Add send command Ben Widawsky
2021-02-01 18:15 ` Konrad Rzeszutek Wilk
2021-02-02 23:08 ` Ben Widawsky
2021-01-30 0:24 ` [PATCH 08/14] taint: add taint for direct hardware access Ben Widawsky
2021-02-01 18:18 ` Konrad Rzeszutek Wilk
2021-02-01 18:34 ` Ben Widawsky
2021-02-01 19:01 ` Dan Williams
2021-02-02 2:49 ` Konrad Rzeszutek Wilk
2021-02-02 17:46 ` Dan Williams
2021-02-08 22:00 ` Dan Williams
2021-02-08 22:09 ` Kees Cook
2021-02-08 23:05 ` Ben Widawsky
2021-02-08 23:36 ` Dan Williams
2021-02-09 1:03 ` Dan Williams
2021-02-09 3:36 ` Ben Widawsky
2021-01-30 0:24 ` [PATCH 09/14] cxl/mem: Add a "RAW" send command Ben Widawsky
2021-02-01 18:24 ` Konrad Rzeszutek Wilk
2021-02-01 19:27 ` Ben Widawsky
2021-02-01 19:34 ` Konrad Rzeszutek Wilk
2021-02-01 21:20 ` Dan Williams
2021-01-30 0:24 ` [PATCH 10/14] cxl/mem: Create concept of enabled commands Ben Widawsky
2021-01-30 0:24 ` [PATCH 11/14] cxl/mem: Use CEL for enabling commands Ben Widawsky
2021-01-30 0:24 ` [PATCH 12/14] cxl/mem: Add set of informational commands Ben Widawsky
2021-01-30 0:24 ` [PATCH 13/14] cxl/mem: Add limited Get Log command (0401h) Ben Widawsky
2021-02-01 18:28 ` Konrad Rzeszutek Wilk
2021-02-02 23:51 ` Ben Widawsky
2021-02-02 23:57 ` Dan Williams
2021-02-03 17:16 ` Ben Widawsky
2021-02-03 18:14 ` Konrad Rzeszutek Wilk
2021-02-03 20:31 ` Dan Williams
2021-02-04 18:55 ` Ben Widawsky
2021-02-04 21:01 ` Dan Williams
2021-01-30 0:24 ` [PATCH 14/14] MAINTAINERS: Add maintainers of the CXL driver Ben Widawsky
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210203005440.yyyphe4yigf3fvkc@intel.com \
--to=ben.widawsky@intel.com \
--cc=Jonathan.Cameron@huawei.com \
--cc=cbrowy@avery-design.com \
--cc=dan.j.williams@intel.com \
--cc=daniel.lll@alibaba-inc.com \
--cc=hch@infradead.org \
--cc=helgaas@kernel.org \
--cc=ira.weiny@intel.com \
--cc=jcm@jonmasters.org \
--cc=jgroves@micron.com \
--cc=linux-acpi@vger.kernel.org \
--cc=linux-cxl@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nvdimm@lists.01.org \
--cc=linux-pci@vger.kernel.org \
--cc=rafael.j.wysocki@intel.com \
--cc=rdunlap@infradead.org \
--cc=rientjes@google.com \
--cc=sean.v.kelley@intel.com \
--cc=vishal.l.verma@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).