linux-pci.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Ben Widawsky <ben.widawsky@intel.com>
To: linux-cxl@vger.kernel.org
Cc: Ben Widawsky <ben.widawsky@intel.com>,
	linux-acpi@vger.kernel.org, linux-kernel@vger.kernel.org,
	linux-nvdimm@lists.01.org, linux-pci@vger.kernel.org,
	Bjorn Helgaas <helgaas@kernel.org>,
	Chris Browy <cbrowy@avery-design.com>,
	Christoph Hellwig <hch@infradead.org>,
	Dan Williams <dan.j.williams@intel.com>,
	Ira Weiny <ira.weiny@intel.com>, Jon Masters <jcm@jonmasters.org>,
	Jonathan Cameron <Jonathan.Cameron@Huawei.com>,
	Rafael Wysocki <rafael.j.wysocki@intel.com>,
	Randy Dunlap <rdunlap@infradead.org>,
	Vishal Verma <vishal.l.verma@intel.com>,
	daniel.lll@alibaba-inc.com,
	"John Groves (jgroves)" <jgroves@micron.com>,
	"Kelley, Sean V" <sean.v.kelley@intel.com>
Subject: [PATCH 10/14] cxl/mem: Create concept of enabled commands
Date: Fri, 29 Jan 2021 16:24:34 -0800	[thread overview]
Message-ID: <20210130002438.1872527-11-ben.widawsky@intel.com> (raw)
In-Reply-To: <20210130002438.1872527-1-ben.widawsky@intel.com>

CXL devices must implement the Device Command Interface (described in
8.2.9 of the CXL 2.0 spec). While the driver already maintains a list of
commands it supports, there is still a need to be able to distinguish
between commands that the driver knows about from commands that may not
be supported by the hardware. No such commands currently are defined in
the driver.

The implementation leaves the statically defined table of commands and
supplements it with a bitmap to determine commands that are enabled.

There are multiple approaches that can be taken, but this is nice for a
few reasons.

Here are some of the other solutions:

Create a per instance table with only the supported commands.
1. Having a fixed command id -> command mapping is much easier to manage
   for development and debugging.
2. Dealing with dynamic memory allocation for the table adds unnecessary
   complexity.
3. Most tables for device types are likely to be quite similar.
4. Makes it difficult to implement helper macros like cxl_for_each_cmd()

If the per instance table did preserve ids, #1 above can be addressed.
However, as "enable" is currently the only mutable state for the
commands, it would yield a lot of overhead for not much gain.
Additionally, the other issues remain.

If "enable" remains the only mutable state, I believe this to be the
best solution. Once the number of mutable elements in a command grows,
it probably makes sense to move to per device instance state with a
fixed command ID mapping.

Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
 drivers/cxl/cxl.h |  4 ++++
 drivers/cxl/mem.c | 40 +++++++++++++++++++++++++++++++++++++++-
 2 files changed, 43 insertions(+), 1 deletion(-)

diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index b042eee7ee25..2d2f25065b81 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -17,6 +17,9 @@
 
 #define CXL_GET_FIELD(word, field) FIELD_GET(field##_MASK, word)
 
+/* XXX: Arbitrary max */
+#define CXL_MAX_COMMANDS 32
+
 /* Device Capabilities (CXL 2.0 - 8.2.8.1) */
 #define CXLDEV_CAP_ARRAY_OFFSET 0x0
 #define   CXLDEV_CAP_ARRAY_CAP_ID 0
@@ -83,6 +86,7 @@ struct cxl_mem {
 	} ram;
 
 	char firmware_version[0x10];
+	DECLARE_BITMAP(enabled_cmds, CXL_MAX_COMMANDS);
 
 	/* Cap 0001h - CXL_CAP_CAP_ID_DEVICE_STATUS */
 	struct {
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index 2942730dc967..d01c6ee32a6b 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -111,6 +111,8 @@ static bool raw_allow_all;
  *    would typically be used for deprecated commands.
  *  * %CXL_CMD_FLAG_MANDATORY: Hardware must support this command. This flag is
  *    only used internally by the driver for sanity checking.
+ *  * %CXL_CMD_INTERNAL_FLAG_PSEUDO: This is a pseudo command which doesn't have
+ *    a direct mapping to hardware. They are implicitly always enabled.
  *
  * The cxl_mem_command is the driver's internal representation of commands that
  * are supported by the driver. Some of these commands may not be supported by
@@ -126,6 +128,7 @@ struct cxl_mem_command {
 #define CXL_CMD_INTERNAL_FLAG_NONE 0
 #define CXL_CMD_INTERNAL_FLAG_HIDDEN BIT(0)
 #define CXL_CMD_INTERNAL_FLAG_MANDATORY BIT(1)
+#define CXL_CMD_INTERNAL_FLAG_PSEUDO BIT(2)
 };
 
 #define CXL_CMD(_id, _flags, sin, sout, f)                                     \
@@ -149,7 +152,7 @@ struct cxl_mem_command {
 static struct cxl_mem_command mem_commands[] = {
 	CXL_CMD(INVALID, KERNEL, 0, 0, HIDDEN),
 	CXL_CMD(IDENTIFY, NONE, 0, 0x43, MANDATORY),
-	CXL_CMD(RAW, NONE, ~0, ~0, MANDATORY),
+	CXL_CMD(RAW, NONE, ~0, ~0, PSEUDO),
 };
 
 /*
@@ -683,6 +686,10 @@ static int cxl_validate_cmd_from_user(struct cxl_mem *cxlm,
 	c = &mem_commands[send_cmd->id];
 	info = &c->info;
 
+	/* Check that the command is enabled for hardware */
+	if (!test_bit(info->id, cxlm->enabled_cmds))
+		return -ENOTTY;
+
 	if (info->flags & CXL_MEM_COMMAND_FLAG_KERNEL)
 		return -EPERM;
 
@@ -1161,6 +1168,33 @@ static int cxl_mem_add_memdev(struct cxl_mem *cxlm)
 	return rc;
 }
 
+/**
+ * cxl_mem_enumerate_cmds() - Enumerate commands for a device.
+ * @cxlm: The device.
+ *
+ * Returns 0 if enumerate completed successfully.
+ *
+ * CXL devices have optional support for certain commands. This function will
+ * determine the set of supported commands for the hardware and update the
+ * enabled_cmds bitmap in the @cxlm.
+ */
+static int cxl_mem_enumerate_cmds(struct cxl_mem *cxlm)
+{
+	struct cxl_mem_command *c;
+
+	BUILD_BUG_ON(ARRAY_SIZE(mem_commands) >= CXL_MAX_COMMANDS);
+
+	/* All commands are considered enabled for now (except INVALID). */
+	cxl_for_each_cmd(c) {
+		if (c->flags & CXL_CMD_INTERNAL_FLAG_HIDDEN)
+			continue;
+
+		set_bit(c->info.id, cxlm->enabled_cmds);
+	}
+
+	return 0;
+}
+
 /**
  * cxl_mem_identify() - Send the IDENTIFY command to the device.
  * @cxlm: The device to identify.
@@ -1280,6 +1314,10 @@ static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	if (rc)
 		return rc;
 
+	rc = cxl_mem_enumerate_cmds(cxlm);
+	if (rc)
+		return rc;
+
 	rc = cxl_mem_identify(cxlm);
 	if (rc)
 		return rc;
-- 
2.30.0


  parent reply	other threads:[~2021-01-30 10:20 UTC|newest]

Thread overview: 96+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-01-30  0:24 [PATCH 00/14] CXL 2.0 Support Ben Widawsky
2021-01-30  0:24 ` [PATCH 01/14] cxl/mem: Introduce a driver for CXL-2.0-Type-3 endpoints Ben Widawsky
2021-01-30 23:51   ` David Rientjes
2021-02-01 17:21   ` Jonathan Cameron
2021-02-01 17:34   ` Konrad Rzeszutek Wilk
2021-02-02 17:58     ` Christoph Hellwig
2021-02-02 18:00   ` Christoph Hellwig
2021-01-30  0:24 ` [PATCH 02/14] cxl/mem: Map memory device registers Ben Widawsky
2021-01-30 23:51   ` David Rientjes
2021-02-01 16:46     ` Ben Widawsky
2021-02-01 18:19       ` Jonathan Cameron
2021-02-01 17:36   ` Konrad Rzeszutek Wilk
2021-02-02 18:04   ` Christoph Hellwig
2021-02-02 18:31     ` Ben Widawsky
2021-02-03 17:12       ` Christoph Hellwig
2021-01-30  0:24 ` [PATCH 03/14] cxl/mem: Find device capabilities Ben Widawsky
2021-01-30 23:51   ` David Rientjes
2021-02-01 16:53     ` Ben Widawsky
2021-02-01 21:51       ` David Rientjes
2021-02-01 21:58         ` Ben Widawsky
2021-02-01 22:23           ` David Rientjes
2021-02-01 22:28             ` Ben Widawsky
2021-02-01 22:33               ` Ben Widawsky
2021-02-01 22:45                 ` David Rientjes
2021-02-01 22:50                   ` Ben Widawsky
2021-02-01 23:09                     ` David Rientjes
2021-02-01 23:17                       ` Ben Widawsky
2021-02-01 23:58                         ` David Rientjes
2021-02-02  0:11                           ` Ben Widawsky
2021-02-02  0:14                             ` Dan Williams
2021-02-02  1:09                               ` David Rientjes
2021-02-01 22:02         ` Dan Williams
2021-02-01 17:41   ` Konrad Rzeszutek Wilk
2021-02-01 17:50     ` Ben Widawsky
2021-02-01 18:08       ` Konrad Rzeszutek Wilk
2021-02-02 18:10   ` Christoph Hellwig
2021-02-02 18:24     ` Ben Widawsky
2021-02-03 17:15       ` Christoph Hellwig
2021-02-03 17:23         ` Ben Widawsky
2021-02-03 21:23           ` Dan Williams
2021-02-04  7:16             ` Christoph Hellwig
2021-02-04 15:29               ` Ben Widawsky
2021-01-30  0:24 ` [PATCH 04/14] cxl/mem: Implement polled mode mailbox Ben Widawsky
2021-01-30 23:51   ` David Rientjes
2021-02-01 20:00     ` Dan Williams
2021-02-02 22:57       ` Ben Widawsky
2021-02-02 23:54         ` Dan Williams
2021-02-03  0:54           ` Ben Widawsky
2021-02-02 22:50     ` Ben Widawsky
2021-02-01 17:54   ` Konrad Rzeszutek Wilk
2021-02-01 19:13     ` Ben Widawsky
2021-02-01 19:28       ` Dan Williams
     [not found]         ` <SN6PR08MB46052FE9BC20A747CACD8F50D1B39@SN6PR08MB4605.namprd08.prod.outlook.com>
2021-02-04 22:24           ` [EXT] " Ben Widawsky
2021-01-30  0:24 ` [PATCH 05/14] cxl/mem: Register CXL memX devices Ben Widawsky
2021-01-30  0:31   ` Dan Williams
2021-01-30 23:52   ` David Rientjes
2021-02-01 17:10     ` Ben Widawsky
2021-02-01 21:53       ` David Rientjes
2021-02-01 21:55         ` Dan Williams
2021-02-02 18:13   ` Christoph Hellwig
2021-01-30  0:24 ` [PATCH 06/14] cxl/mem: Add basic IOCTL interface Ben Widawsky
2021-02-02 18:15   ` Christoph Hellwig
2021-02-02 18:33     ` Ben Widawsky
2021-01-30  0:24 ` [PATCH 07/14] cxl/mem: Add send command Ben Widawsky
2021-02-01 18:15   ` Konrad Rzeszutek Wilk
2021-02-02 23:08     ` Ben Widawsky
2021-01-30  0:24 ` [PATCH 08/14] taint: add taint for direct hardware access Ben Widawsky
2021-02-01 18:18   ` Konrad Rzeszutek Wilk
2021-02-01 18:34     ` Ben Widawsky
2021-02-01 19:01       ` Dan Williams
2021-02-02  2:49         ` Konrad Rzeszutek Wilk
2021-02-02 17:46           ` Dan Williams
2021-02-08 22:00   ` Dan Williams
2021-02-08 22:09     ` Kees Cook
2021-02-08 23:05       ` Ben Widawsky
2021-02-08 23:36       ` Dan Williams
2021-02-09  1:03         ` Dan Williams
2021-02-09  3:36           ` Ben Widawsky
2021-01-30  0:24 ` [PATCH 09/14] cxl/mem: Add a "RAW" send command Ben Widawsky
2021-02-01 18:24   ` Konrad Rzeszutek Wilk
2021-02-01 19:27     ` Ben Widawsky
2021-02-01 19:34       ` Konrad Rzeszutek Wilk
2021-02-01 21:20         ` Dan Williams
2021-01-30  0:24 ` Ben Widawsky [this message]
2021-01-30  0:24 ` [PATCH 11/14] cxl/mem: Use CEL for enabling commands Ben Widawsky
2021-01-30  0:24 ` [PATCH 12/14] cxl/mem: Add set of informational commands Ben Widawsky
2021-01-30  0:24 ` [PATCH 13/14] cxl/mem: Add limited Get Log command (0401h) Ben Widawsky
2021-02-01 18:28   ` Konrad Rzeszutek Wilk
2021-02-02 23:51     ` Ben Widawsky
2021-02-02 23:57       ` Dan Williams
2021-02-03 17:16         ` Ben Widawsky
2021-02-03 18:14           ` Konrad Rzeszutek Wilk
2021-02-03 20:31             ` Dan Williams
2021-02-04 18:55               ` Ben Widawsky
2021-02-04 21:01                 ` Dan Williams
2021-01-30  0:24 ` [PATCH 14/14] MAINTAINERS: Add maintainers of the CXL driver Ben Widawsky

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210130002438.1872527-11-ben.widawsky@intel.com \
    --to=ben.widawsky@intel.com \
    --cc=Jonathan.Cameron@Huawei.com \
    --cc=cbrowy@avery-design.com \
    --cc=dan.j.williams@intel.com \
    --cc=daniel.lll@alibaba-inc.com \
    --cc=hch@infradead.org \
    --cc=helgaas@kernel.org \
    --cc=ira.weiny@intel.com \
    --cc=jcm@jonmasters.org \
    --cc=jgroves@micron.com \
    --cc=linux-acpi@vger.kernel.org \
    --cc=linux-cxl@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-nvdimm@lists.01.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=rafael.j.wysocki@intel.com \
    --cc=rdunlap@infradead.org \
    --cc=sean.v.kelley@intel.com \
    --cc=vishal.l.verma@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).