linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [RFC PATCH v2 00/14] CXL 2.0 Support
@ 2020-12-09  0:24 Ben Widawsky
  2020-12-09  0:24 ` [RFC PATCH v2 01/14] docs: cxl: Add basic documentation Ben Widawsky
                   ` (15 more replies)
  0 siblings, 16 replies; 35+ messages in thread
From: Ben Widawsky @ 2020-12-09  0:24 UTC (permalink / raw)
  To: linux-cxl
  Cc: Ben Widawsky, linux-kernel, linux-pci, linux-acpi, Ira Weiny,
	Dan Williams, Vishal Verma, Kelley, Sean V, Rafael Wysocki,
	Bjorn Helgaas, Jonathan Cameron, Jon Masters, Chris Browy,
	Randy Dunlap, Christoph Hellwig

Changes since v1 [1]

A few additions have been made:
 - IOCTL (UAPI) interface has been added with commands
 - Kernel docs have been created
 - A new debug macro is introduced and sprinkled throughout.

A deletion was made:
 - Removal of the non-standard _OSC UUID.

The detailed list of fixes is:
 - fix cxl_register() no previous prototype warning (0day robot)
 - s/REGLOG/REGLOC/ (Ben)
 - Wait for doorbell on cxl_mem_mbox_get() and add comment on why (Ben)
 - make "type-3" a proper adjective, add spec references, also did the same for
   the Kconfig (Bjorn)
 - align some defines (Bjorn)
 - s/bar/BAR (Bjorn)
 - rename cxl_bus_prepared() to cxl_bus_acquire() (Bjorn)
 - move definition of struct cxl_mem to "cxl/mem: Map memory device registers" (Bjorn)
 - use consistent hex/decimal (Bjorn)
 - use consistent upper/lower hex values (Bjorn)
 - get rid of READ_ONCE (Bjorn)
 - add offsets to debug messages (Bjorn)
 - cleanup SPDX comment style (Bjorn, Christoph)
 - change errors returned by case (Bjorn, Dan)
 - 80 character violation cleanups (Christoph)
 - cleanup CXL_BUS_PROVIDER dependencies (Christoph, Randy)
 - remove "raw" from mmio functions (Dan)
 - rename PCI_DVSEC_VENDOR_CXL to add _ID (Jonathan)
 - combine introduction of mbox infrastruct and cxl_mem_identify() (Jonathan)
 - add ABI documentation for sysfs attributes (Jonathan)
 - document scope of cxl_memdev_lock (Jonathan)
 - rework cxl_register() to have devm semantics (reaction to comments about
   cxl_mem_remove() and cxl_mem_add_memdev() semantics) (Jonathan)
 - fix cxl_mem_exit() ordering (Jonathan)
 - use GENMASK/GET_FIELD (Jonathan)
 - fix and add comments for cap ids (Jonathan)
 - use _OFFSET postfix in definitions (Jonathan)
 - save pci_set_drvdata for later (Jonathan)

[1]: https://lore.kernel.org/linux-cxl/20201111054356.793390-1-ben.widawsky@intel.com/

---

Introduce support for “type-3” memory devices defined in the recently released
Compute Express Link (CXL) 2.0 specification[2]. Specifically, these are the
memory devices defined by section 8.2.8.5 of the CXL 2.0 spec. A reference
implementation emulating these devices has been submitted to the QEMU mailing
list and is available on gitlab [3]. “Type-3” is a CXL device that acts as a
memory expander for RAM or PMEM.  It might be interleaved with other CXL devices
in a given physical address range.

These changes allow for foundational enumeration of CXL 2.0 memory devices as
well as basic userspace interaction. The functionality present is:
- Initial driver bring-up
- Device enumeration and an initial sysfs representation
- Submit a basic firmware command via ‘mailbox’ to an emulated memory device
  with non-volatile capacity.
- Provide an interface to send "raw" commands to the hardware.

Some of the functionality that is still missing includes:
- Memory interleaving at the host bridge, root port, or switch level
- CXL 1.1 Root Complex Integrated Endpoint Support
- CXL 2.0 Hot plug support
- A bevy of supported device commands

In addition to the core functionality of discovering the spec defined registers
and resources, introduce a CXL device model that will be the foundation for
translating CXL capabilities into existing Linux infrastructure for Persistent
Memory and other memory devices. For now, this only includes support for the
management command mailbox that type-3 devices surface. These control devices
fill the role of “DIMMs” / nmemX memory-devices in LIBNVDIMM terms.

Now, while implementing the driver some feedback for the specification was
generated to cover perceived gaps and address conflicts. The feedback is
presented as a reference implementation in the driver and QEMU emulation.
Specifically the following concepts are original to the Linux implementation and
feedback / collaboration is requested to develop these into specification
proposals:
1. Top level ACPI object (ACPI0017)
2. HW imposed address space and interleave constraints

ACPI0017
--------
Introduce a new ACPI namespace device with an _HID of ACPI0017. The purpose of
this object is twofold, support a legacy OS with a set of out-of-tree CXL
modules, and establish an attach point for a driver that knows about
interleaving. Both of these boil down to the same point, to centralize Operating
System support for resources described by the CXL Early Discovery Table (CEDT).

The legacy OS problem stems from the spec's description of a host bridge,
ACPI0016 is denoted as the _HID for host bridges, with a _CID of PNP0A08. In a
CXL unaware version of Linux, the core ACPI subsystem will bind a driver to
PNP0A08 and preclude a CXL-aware driver from binding to ACPI0016. An ACPI0017
device allows a standalone CXL-aware driver to register for handling /
coordinating CEDT and CXL-specific _OSC control.

Similarly when managing interleaving there needs to be some management layer
above the ACPI0016 device that is capable of assembling leaf nodes into
interleave sets. As is the case with ACPI0012 that does this central
coordination for NFIT defined resources, ACPI0017 does the same for CEDT
described resources.

Memory Windows
-------
For CXL.mem capable platforms, there is a need for a mechanism for platform
firmware to make the Operating System aware of any restrictions that hardware
might have in address space. For example, in a system with 4 host bridges all
participating in an interleave set, the firmware needs to provide some
description of this. That information is missing from the CXL 2.0 spec as of
today and it also is not implemented in the driver. A variety of ACPI based
mechanisms, for example _CRS fields on the ACPI0017 device, were considered.

Next steps after this basic foundation is expanded command support and LIBNVDIMM
integration. This is the initial “release early / release often” version of the
Linux CXL enabling.

[2]: https://www.computeexpresslink.org/
[3]: https://gitlab.com/bwidawsk/qemu/-/tree/cxl-2.0v2


Ben Widawsky (10):
  docs: cxl: Add basic documentation
  cxl/mem: Map memory device registers
  cxl/mem: Find device capabilities
  cxl/mem: Implement polled mode mailbox
  cxl/mem: Add basic IOCTL interface
  cxl/mem: Add send command
  cxl/mem: Add a "RAW" send command
  cxl: Add basic debugging
  MAINTAINERS: Add maintainers of the CXL driver
  WIP/cxl/mem: Add get firmware for testing

Dan Williams (2):
  cxl/mem: Introduce a driver for CXL-2.0-Type-3 endpoints
  cxl/mem: Register CXL memX devices

Vishal Verma (2):
  cxl/acpi: Add an acpi_cxl module for the CXL interconnect
  cxl/acpi: add OSC support

 Documentation/ABI/testing/sysfs-bus-cxl |   26 +
 Documentation/cxl/index.rst             |   12 +
 Documentation/cxl/memory-devices.rst    |   51 ++
 Documentation/index.rst                 |    1 +
 MAINTAINERS                             |   10 +
 drivers/Kconfig                         |    1 +
 drivers/Makefile                        |    1 +
 drivers/cxl/Kconfig                     |   58 ++
 drivers/cxl/Makefile                    |    9 +
 drivers/cxl/acpi.c                      |  352 ++++++++
 drivers/cxl/acpi.h                      |   35 +
 drivers/cxl/bus.c                       |   54 ++
 drivers/cxl/bus.h                       |    8 +
 drivers/cxl/cxl.h                       |  188 +++++
 drivers/cxl/mem.c                       | 1022 +++++++++++++++++++++++
 drivers/cxl/pci.h                       |   34 +
 include/acpi/actbl1.h                   |   51 ++
 include/uapi/linux/cxl_mem.h            |  148 ++++
 18 files changed, 2061 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-bus-cxl
 create mode 100644 Documentation/cxl/index.rst
 create mode 100644 Documentation/cxl/memory-devices.rst
 create mode 100644 drivers/cxl/Kconfig
 create mode 100644 drivers/cxl/Makefile
 create mode 100644 drivers/cxl/acpi.c
 create mode 100644 drivers/cxl/acpi.h
 create mode 100644 drivers/cxl/bus.c
 create mode 100644 drivers/cxl/bus.h
 create mode 100644 drivers/cxl/cxl.h
 create mode 100644 drivers/cxl/mem.c
 create mode 100644 drivers/cxl/pci.h
 create mode 100644 include/uapi/linux/cxl_mem.h

-- 
2.29.2


^ permalink raw reply	[flat|nested] 35+ messages in thread

* [RFC PATCH v2 01/14] docs: cxl: Add basic documentation
  2020-12-09  0:24 [RFC PATCH v2 00/14] CXL 2.0 Support Ben Widawsky
@ 2020-12-09  0:24 ` Ben Widawsky
  2020-12-09  0:24 ` [RFC PATCH v2 02/14] cxl/acpi: Add an acpi_cxl module for the CXL interconnect Ben Widawsky
                   ` (14 subsequent siblings)
  15 siblings, 0 replies; 35+ messages in thread
From: Ben Widawsky @ 2020-12-09  0:24 UTC (permalink / raw)
  To: linux-cxl
  Cc: Ben Widawsky, linux-kernel, linux-pci, linux-acpi, Ira Weiny,
	Dan Williams, Vishal Verma, Kelley, Sean V, Rafael Wysocki,
	Bjorn Helgaas, Jonathan Cameron, Jon Masters, Chris Browy,
	Randy Dunlap, Christoph Hellwig

This starts a new cxl directory because CXL is a new bus and it is
expected that the documentation will grow large over time. Currently
only memory devices are documented as they are all that's supported.

Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
 Documentation/cxl/index.rst          | 12 ++++++++++++
 Documentation/cxl/memory-devices.rst | 15 +++++++++++++++
 Documentation/index.rst              |  1 +
 3 files changed, 28 insertions(+)
 create mode 100644 Documentation/cxl/index.rst
 create mode 100644 Documentation/cxl/memory-devices.rst

diff --git a/Documentation/cxl/index.rst b/Documentation/cxl/index.rst
new file mode 100644
index 000000000000..036e49553542
--- /dev/null
+++ b/Documentation/cxl/index.rst
@@ -0,0 +1,12 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+====================
+Compute Express Link
+====================
+
+.. toctree::
+   :maxdepth: 1
+
+   memory-devices
+
+.. only::  subproject and html
diff --git a/Documentation/cxl/memory-devices.rst b/Documentation/cxl/memory-devices.rst
new file mode 100644
index 000000000000..aa4262280c67
--- /dev/null
+++ b/Documentation/cxl/memory-devices.rst
@@ -0,0 +1,15 @@
+.. SPDX-License-Identifier: GPL-2.0
+.. include:: <isonum.txt>
+
+===================================
+Compute Express Link Memory Devices
+===================================
+
+A Compute Express Link Memory Device is a CXL component that implements the
+CXL.mem protocol. It contains some amount of volatile memory, persistent memory,
+or both.
+
+Driver Infrastructure
+=====================
+
+This sections covers the driver infrastructure for a CXL memory device.
diff --git a/Documentation/index.rst b/Documentation/index.rst
index 57719744774c..38678f6c5676 100644
--- a/Documentation/index.rst
+++ b/Documentation/index.rst
@@ -137,6 +137,7 @@ needed).
    misc-devices/index
    scheduler/index
    mhi/index
+   cxl/index
 
 Architecture-agnostic documentation
 -----------------------------------
-- 
2.29.2


^ permalink raw reply related	[flat|nested] 35+ messages in thread

* [RFC PATCH v2 02/14] cxl/acpi: Add an acpi_cxl module for the CXL interconnect
  2020-12-09  0:24 [RFC PATCH v2 00/14] CXL 2.0 Support Ben Widawsky
  2020-12-09  0:24 ` [RFC PATCH v2 01/14] docs: cxl: Add basic documentation Ben Widawsky
@ 2020-12-09  0:24 ` Ben Widawsky
  2020-12-09  0:24 ` [RFC PATCH v2 03/14] cxl/acpi: add OSC support Ben Widawsky
                   ` (13 subsequent siblings)
  15 siblings, 0 replies; 35+ messages in thread
From: Ben Widawsky @ 2020-12-09  0:24 UTC (permalink / raw)
  To: linux-cxl
  Cc: Vishal Verma, Ben Widawsky, Dan Williams, linux-kernel,
	linux-pci, linux-acpi, Ira Weiny, Kelley, Sean V, Rafael Wysocki,
	Bjorn Helgaas, Jonathan Cameron, Jon Masters, Chris Browy,
	Randy Dunlap, Christoph Hellwig

From: Vishal Verma <vishal.l.verma@intel.com>

Add an acpi_cxl module to coordinate the ACPI portions of the CXL
(Compute eXpress Link) interconnect. This driver binds to ACPI0017
objects in the ACPI tree, and coordinates access to the resources
provided by the ACPI CEDT (CXL Early Discovery Table).

It also coordinates operations of the root port _OSC object to notify
platform firmware that the OS has native support for the CXL
capabilities of endpoints.

Note: the actbl1.h changes are speculative. The expectation is that they
will arrive through the ACPICA tree in due time.

Cc: Ben Widawsky <ben.widawsky@intel.com>
Cc: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Vishal Verma <vishal.l.verma@intel.com>
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
 drivers/Kconfig       |  1 +
 drivers/Makefile      |  1 +
 drivers/cxl/Kconfig   | 36 ++++++++++++++++
 drivers/cxl/Makefile  |  5 +++
 drivers/cxl/acpi.c    | 97 +++++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/acpi.h    | 15 +++++++
 include/acpi/actbl1.h | 51 +++++++++++++++++++++++
 7 files changed, 206 insertions(+)
 create mode 100644 drivers/cxl/Kconfig
 create mode 100644 drivers/cxl/Makefile
 create mode 100644 drivers/cxl/acpi.c
 create mode 100644 drivers/cxl/acpi.h

diff --git a/drivers/Kconfig b/drivers/Kconfig
index dcecc9f6e33f..62c753a73651 100644
--- a/drivers/Kconfig
+++ b/drivers/Kconfig
@@ -6,6 +6,7 @@ menu "Device Drivers"
 source "drivers/amba/Kconfig"
 source "drivers/eisa/Kconfig"
 source "drivers/pci/Kconfig"
+source "drivers/cxl/Kconfig"
 source "drivers/pcmcia/Kconfig"
 source "drivers/rapidio/Kconfig"
 
diff --git a/drivers/Makefile b/drivers/Makefile
index c0cd1b9075e3..5dad349de73b 100644
--- a/drivers/Makefile
+++ b/drivers/Makefile
@@ -73,6 +73,7 @@ obj-$(CONFIG_NVM)		+= lightnvm/
 obj-y				+= base/ block/ misc/ mfd/ nfc/
 obj-$(CONFIG_LIBNVDIMM)		+= nvdimm/
 obj-$(CONFIG_DAX)		+= dax/
+obj-$(CONFIG_CXL_BUS)		+= cxl/
 obj-$(CONFIG_DMA_SHARED_BUFFER) += dma-buf/
 obj-$(CONFIG_NUBUS)		+= nubus/
 obj-y				+= macintosh/
diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig
new file mode 100644
index 000000000000..68da926ba5b1
--- /dev/null
+++ b/drivers/cxl/Kconfig
@@ -0,0 +1,36 @@
+# SPDX-License-Identifier: GPL-2.0-only
+menuconfig CXL_BUS
+	tristate "CXL (Compute Express Link) Devices Support"
+	help
+	  CXL is a bus that is electrically compatible with PCI-E, but layers
+	  three protocols on that signalling (CXL.io, CXL.cache, and CXL.mem).
+	  The CXL.cache protocol allows devices to hold cachelines locally, the
+	  CXL.mem protocol allows devices to be fully coherent memory targets,
+	  the CXL.io protocol is equivalent to PCI-E. Say 'y' to enable support
+	  for the configuration and management of devices supporting these
+	  protocols.
+
+if CXL_BUS
+
+config CXL_BUS_PROVIDER
+	tristate
+
+config CXL_ACPI
+	tristate "CXL ACPI: Platform Support"
+	depends on ACPI
+	default CXL_BUS
+	select CXL_BUS_PROVIDER
+	help
+	  Say 'y/m' to enable a driver (named "cxl_acpi.ko" when built
+	  as a module) that will enable support for CXL.mem endpoint
+	  devices. In general, CXL Platform Support is a prerequisite
+	  for any CXL device driver that wants to claim ownership of a
+	  component register space. By default platform firmware assumes
+	  Linux is unaware of CXL capabilities and requires explicit
+	  opt-in. This platform component also mediates resources
+	  described by the CEDT (CXL Early Discovery Table).  See
+	  Chapter 9.14.1 CXL Early Discovery Table (CEDT) in the CXL 2.0
+	  specification.
+
+	  If unsure say 'm'
+endif
diff --git a/drivers/cxl/Makefile b/drivers/cxl/Makefile
new file mode 100644
index 000000000000..d38cd34a2582
--- /dev/null
+++ b/drivers/cxl/Makefile
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o
+
+ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL
+cxl_acpi-y := acpi.o
diff --git a/drivers/cxl/acpi.c b/drivers/cxl/acpi.c
new file mode 100644
index 000000000000..0f1ba9b3f1ed
--- /dev/null
+++ b/drivers/cxl/acpi.c
@@ -0,0 +1,97 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(c) 2020 Intel Corporation. All rights reserved. */
+#include <linux/platform_device.h>
+#include <linux/list_sort.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/sysfs.h>
+#include <linux/list.h>
+#include <linux/acpi.h>
+#include <linux/sort.h>
+#include <linux/pci.h>
+#include "acpi.h"
+
+/*
+ * If/when CXL support is defined by other platform firmware the kernel
+ * will need a mechanism to select between the platform specific version
+ * of this routine, until then, hard-code ACPI assumptions
+ */
+int cxl_bus_acquire(struct pci_dev *pdev)
+{
+	struct acpi_device *adev;
+	struct pci_dev *root_port;
+	struct device *root;
+
+	root_port = pcie_find_root_port(pdev);
+	if (!root_port)
+		return -ENXIO;
+
+	root = root_port->dev.parent;
+	if (!root)
+		return -ENXIO;
+
+	adev = ACPI_COMPANION(root);
+	if (!adev)
+		return -ENXIO;
+
+	/* TODO: OSC enabling */
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(cxl_bus_acquire);
+
+static void acpi_cedt_put_table(void *table)
+{
+	acpi_put_table(table);
+}
+
+static int cxl_acpi_probe(struct platform_device *pdev)
+{
+	struct acpi_device *adev = ACPI_COMPANION(&pdev->dev);
+	struct cxl_acpi_desc *acpi_desc;
+	struct device *dev = &adev->dev;
+	struct acpi_table_header *tbl;
+	acpi_status status;
+	acpi_size sz;
+	int rc;
+
+	status = acpi_get_table(ACPI_SIG_CEDT, 0, &tbl);
+	if (ACPI_FAILURE(status)) {
+		dev_err(dev, "failed to find CEDT at startup\n");
+		return 0;
+	}
+
+	rc = devm_add_action_or_reset(dev, acpi_cedt_put_table, tbl);
+	if (rc)
+		return rc;
+
+	sz = tbl->length;
+	dev_info(dev, "found CEDT at startup: %lld bytes\n", sz);
+
+	acpi_desc = devm_kzalloc(dev, sizeof(*acpi_desc), GFP_KERNEL);
+	if (!acpi_desc)
+		return -ENOMEM;
+
+	dev_set_drvdata(&adev->dev, acpi_desc);
+	acpi_desc->dev = &adev->dev;
+	acpi_desc->acpi_header = *tbl;
+
+	return 0;
+}
+
+static const struct acpi_device_id cxl_acpi_ids[] = {
+	{ "ACPI0017", 0 },
+	{ "", 0 },
+};
+MODULE_DEVICE_TABLE(acpi, cxl_acpi_ids);
+
+static struct platform_driver cxl_acpi_driver = {
+	.probe = cxl_acpi_probe,
+	.driver = {
+		.name = KBUILD_MODNAME,
+		.acpi_match_table = cxl_acpi_ids,
+	},
+};
+
+module_platform_driver(cxl_acpi_driver);
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/cxl/acpi.h b/drivers/cxl/acpi.h
new file mode 100644
index 000000000000..d638f8886ab7
--- /dev/null
+++ b/drivers/cxl/acpi.h
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(c) 2020 Intel Corporation. All rights reserved. */
+
+#ifndef __CXL_ACPI_H__
+#define __CXL_ACPI_H__
+#include <linux/acpi.h>
+
+struct cxl_acpi_desc {
+	struct acpi_table_header acpi_header;
+	struct device *dev;
+};
+
+int cxl_bus_acquire(struct pci_dev *pci_dev);
+
+#endif	/* __CXL_ACPI_H__ */
diff --git a/include/acpi/actbl1.h b/include/acpi/actbl1.h
index 43549547ed3e..aa22c648ce95 100644
--- a/include/acpi/actbl1.h
+++ b/include/acpi/actbl1.h
@@ -28,6 +28,7 @@
 #define ACPI_SIG_BERT           "BERT"	/* Boot Error Record Table */
 #define ACPI_SIG_BGRT           "BGRT"	/* Boot Graphics Resource Table */
 #define ACPI_SIG_BOOT           "BOOT"	/* Simple Boot Flag Table */
+#define ACPI_SIG_CEDT           "CEDT"	/* CXL Early Discovery Table */
 #define ACPI_SIG_CPEP           "CPEP"	/* Corrected Platform Error Polling table */
 #define ACPI_SIG_CSRT           "CSRT"	/* Core System Resource Table */
 #define ACPI_SIG_DBG2           "DBG2"	/* Debug Port table type 2 */
@@ -1624,6 +1625,56 @@ struct acpi_ibft_target {
 	u16 reverse_chap_secret_offset;
 };
 
+/*******************************************************************************
+ *
+ * CEDT - CXL Early Discovery Table (ACPI 6.4)
+ *        Version 1
+ *
+ ******************************************************************************/
+
+struct acpi_table_cedt {
+	struct acpi_table_header header;	/* Common ACPI table header */
+	u32 reserved;
+};
+
+/* Values for CEDT structure types */
+
+enum acpi_cedt_type {
+	ACPI_CEDT_TYPE_HOST_BRIDGE = 0, /* CHBS - CXL Host Bridge Structure */
+};
+
+struct acpi_cedt_structure {
+	u8 type;
+	u8 reserved;
+	u16 length;
+};
+
+/*
+ * CEDT Structures, correspond to Type in struct acpi_cedt_structure
+ */
+
+/* 0: CXL Host Bridge Structure */
+
+struct acpi_cedt_chbs {
+	struct acpi_cedt_structure header;
+	u32 uid;
+	u32 version;
+	u32 reserved1;
+	u64 base;
+	u32 length;
+	u32 reserved2;
+};
+
+/* Values for version field above */
+
+#define ACPI_CEDT_CHBS_VERSION_CXL11    (0)
+#define ACPI_CEDT_CHBS_VERSION_CXL20    (1)
+
+/* Values for length field above */
+
+#define ACPI_CEDT_CHBS_LENGTH_CXL11     (0x2000)
+#define ACPI_CEDT_CHBS_LENGTH_CXL20     (0x10000)
+
 /* Reset to default packing */
 
 #pragma pack()
-- 
2.29.2


^ permalink raw reply related	[flat|nested] 35+ messages in thread

* [RFC PATCH v2 03/14] cxl/acpi: add OSC support
  2020-12-09  0:24 [RFC PATCH v2 00/14] CXL 2.0 Support Ben Widawsky
  2020-12-09  0:24 ` [RFC PATCH v2 01/14] docs: cxl: Add basic documentation Ben Widawsky
  2020-12-09  0:24 ` [RFC PATCH v2 02/14] cxl/acpi: Add an acpi_cxl module for the CXL interconnect Ben Widawsky
@ 2020-12-09  0:24 ` Ben Widawsky
  2020-12-09  0:24 ` [RFC PATCH v2 04/14] cxl/mem: Introduce a driver for CXL-2.0-Type-3 endpoints Ben Widawsky
                   ` (12 subsequent siblings)
  15 siblings, 0 replies; 35+ messages in thread
From: Ben Widawsky @ 2020-12-09  0:24 UTC (permalink / raw)
  To: linux-cxl
  Cc: Vishal Verma, Ben Widawsky, linux-kernel, linux-pci, linux-acpi,
	Ira Weiny, Dan Williams, Kelley, Sean V, Rafael Wysocki,
	Bjorn Helgaas, Jonathan Cameron, Jon Masters, Chris Browy,
	Randy Dunlap, Christoph Hellwig

From: Vishal Verma <vishal.l.verma@intel.com>

Add support to advertise OS capabilities, and request OS control for CXL
features using the ACPI _OSC mechanism. Advertise support for all
possible CXL features, and attempt to request control for all possible
features.

Based on a patch by Sean Kelley.

Signed-off-by: Vishal Verma <vishal.l.verma@intel.com>
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
 Documentation/cxl/memory-devices.rst |  15 ++
 drivers/cxl/acpi.c                   | 261 ++++++++++++++++++++++++++-
 drivers/cxl/acpi.h                   |  20 ++
 3 files changed, 293 insertions(+), 3 deletions(-)

diff --git a/Documentation/cxl/memory-devices.rst b/Documentation/cxl/memory-devices.rst
index aa4262280c67..6ce88f9d5f4f 100644
--- a/Documentation/cxl/memory-devices.rst
+++ b/Documentation/cxl/memory-devices.rst
@@ -13,3 +13,18 @@ Driver Infrastructure
 =====================
 
 This sections covers the driver infrastructure for a CXL memory device.
+
+ACPI CXL
+--------
+
+.. kernel-doc:: drivers/cxl/acpi.c
+   :doc: cxl acpi
+
+.. kernel-doc:: drivers/cxl/acpi.c
+   :internal:
+
+External Interfaces
+===================
+
+.. kernel-doc:: drivers/cxl/acpi.c
+   :export:
diff --git a/drivers/cxl/acpi.c b/drivers/cxl/acpi.c
index 0f1ba9b3f1ed..42180c64cbee 100644
--- a/drivers/cxl/acpi.c
+++ b/drivers/cxl/acpi.c
@@ -11,7 +11,259 @@
 #include <linux/pci.h>
 #include "acpi.h"
 
-/*
+/**
+ * DOC: cxl acpi
+ *
+ * ACPI _OSC setup: The exported function cxl_bus_acquire() sets up ACPI
+ * Operating System Capabilities (_OSC) for the CXL bus. It declares support
+ * for all CXL capabilities, and attempts to request control for all possible
+ * capabilities. The resulting support and control sets are saved in global
+ * variables cxl_osc_support_set and cxl_osc_control_set. The internal
+ * functions cxl_osc_declare_support(), and cxl_osc_request_control() can be
+ * used to update the support and control sets in accordance with the ACPI
+ * rules for _OSC evaluation - most importantly, capabilities already granted
+ * should not be rescinded by either the OS or firmware.
+ */
+
+static u32 cxl_osc_support_set;
+static u32 cxl_osc_control_set;
+static DEFINE_MUTEX(acpi_desc_lock);
+
+struct pci_osc_bit_struct {
+	u32 bit;
+	char *desc;
+};
+
+static struct pci_osc_bit_struct cxl_osc_support_bit[] = {
+	{ CXL_OSC_PORT_REG_ACCESS_SUPPORT, "CXLPortRegAccess" },
+	{ CXL_OSC_PORT_DEV_REG_ACCESS_SUPPORT, "CXLPortDevRegAccess" },
+	{ CXL_OSC_PER_SUPPORT, "CXLProtocolErrorReporting" },
+	{ CXL_OSC_NATIVE_HP_SUPPORT, "CXLNativeHotPlug" },
+};
+
+static struct pci_osc_bit_struct cxl_osc_control_bit[] = {
+	{ CXL_OSC_MEM_ERROR_CONTROL, "CXLMemErrorReporting" },
+};
+
+static u8 cxl_osc_uuid_str[] = "68F2D50B-C469-4d8A-BD3D-941A103FD3FC";
+
+static void decode_osc_bits(struct device *dev, char *msg, u32 word,
+			    struct pci_osc_bit_struct *table, int size)
+{
+	char buf[80];
+	int i, len = 0;
+	struct pci_osc_bit_struct *entry;
+
+	buf[0] = '\0';
+	for (i = 0, entry = table; i < size; i++, entry++)
+		if (word & entry->bit)
+			len += scnprintf(buf + len, sizeof(buf) - len, "%s%s",
+					 len ? " " : "", entry->desc);
+
+	dev_info(dev, "_OSC: %s [%s]\n", msg, buf);
+}
+
+static void decode_cxl_osc_support(struct device *dev, char *msg, u32 word)
+{
+	decode_osc_bits(dev, msg, word, cxl_osc_support_bit,
+			ARRAY_SIZE(cxl_osc_support_bit));
+}
+
+static void decode_cxl_osc_control(struct device *dev, char *msg, u32 word)
+{
+	decode_osc_bits(dev, msg, word, cxl_osc_control_bit,
+			ARRAY_SIZE(cxl_osc_control_bit));
+}
+
+static acpi_status acpi_cap_run_osc(acpi_handle handle, const u32 *capbuf,
+				    u8 *uuid_str, u32 *retval)
+{
+	struct acpi_osc_context context = {
+		.uuid_str = uuid_str,
+		.rev = 1,
+		.cap.length = 20,
+		.cap.pointer = (void *)capbuf,
+	};
+	acpi_status status;
+
+	status = acpi_run_osc(handle, &context);
+	if (ACPI_SUCCESS(status)) {
+		/* pointer + offset to DWORD 5 */
+		*retval = *((u32 *)(context.ret.pointer + 16));
+		kfree(context.ret.pointer);
+	}
+	return status;
+}
+
+static acpi_status cxl_query_osc(acpi_handle handle, u32 support, u32 *control)
+{
+	struct acpi_pci_root *root;
+	acpi_status status;
+	u32 result, capbuf[5];
+
+	root = acpi_pci_find_root(handle);
+	if (!root)
+		return -ENXIO;
+
+	support &= CXL_OSC_SUPPORT_VALID_MASK;
+	support |= cxl_osc_support_set;
+
+	capbuf[OSC_QUERY_DWORD] = OSC_QUERY_ENABLE;
+	capbuf[PCI_OSC_SUPPORT_DWORD] = root->osc_support_set;
+	capbuf[PCI_OSC_CONTROL_DWORD] = root->osc_control_set;
+	capbuf[CXL_OSC_SUPPORT_DWORD] = support;
+	if (control) {
+		*control &= CXL_OSC_CONTROL_VALID_MASK;
+		capbuf[CXL_OSC_CONTROL_DWORD] = *control | cxl_osc_control_set;
+	} else {
+		/* Run _OSC query only with existing controls. */
+		capbuf[CXL_OSC_CONTROL_DWORD] = cxl_osc_control_set;
+	}
+
+	status = acpi_cap_run_osc(handle, capbuf, cxl_osc_uuid_str, &result);
+	if (ACPI_SUCCESS(status)) {
+		cxl_osc_support_set = support;
+		if (control)
+			*control = result;
+	}
+	return status;
+}
+
+/**
+ * cxl_osc_declare_support - Declare support for CXL root _OSC features.
+ * @handle: ACPI device handle for the PCI root bridge (or PCIe Root Complex).
+ * @flags: The _OSC bits to declare support for.
+ *
+ * The PCI specific DWORDS are obtained from the pci root port device, and
+ * used as-is. The resulting CXL support set is saved in cxl_osc_support_set.
+ * Any future calls to this are forced to be strictly incremental from the
+ * existing cxl_osc_support_set.
+ */
+static acpi_status cxl_osc_declare_support(acpi_handle handle, u32 flags)
+{
+	acpi_status status;
+
+	mutex_lock(&acpi_desc_lock);
+	status = cxl_query_osc(handle, flags, NULL);
+	mutex_unlock(&acpi_desc_lock);
+	return status;
+}
+
+/**
+ * cxl_osc_request_control - Request control of CXL root _OSC features.
+ * @adev: ACPI device for the PCI root bridge (or PCIe Root Complex).
+ * @mask: Mask of _OSC bits to request control of, place to store control mask.
+ * @req: Mask of _OSC bits the control of is essential to the caller.
+ *
+ * The PCI specific DWORDS are obtained from the pci root port device, and
+ * used as-is. The resulting CXL control set is saved in cxl_osc_control_set.
+ * Any future calls to this are forced to be strictly incremental from the
+ * existing cxl_osc_control_set.
+ */
+static acpi_status cxl_osc_request_control(struct acpi_device *adev, u32 *mask,
+					   u32 req)
+{
+	acpi_handle handle = adev->handle;
+	struct device *dev = &adev->dev;
+	struct acpi_pci_root *root;
+	acpi_status status = AE_OK;
+	u32 ctrl, capbuf[5];
+
+	if (!mask)
+		return AE_BAD_PARAMETER;
+
+	ctrl = *mask & CXL_OSC_MEM_ERROR_CONTROL;
+	if ((ctrl & req) != req)
+		return AE_TYPE;
+
+	root = acpi_pci_find_root(handle);
+	if (!root)
+		return -ENXIO;
+
+	mutex_lock(&acpi_desc_lock);
+
+	*mask = ctrl | cxl_osc_control_set;
+	/* No need to evaluate _OSC if the control was already granted. */
+	if ((cxl_osc_control_set & ctrl) == ctrl)
+		goto out;
+
+	/* Need to check the available controls bits before requesting them. */
+	while (*mask) {
+		status = cxl_query_osc(handle, cxl_osc_support_set, mask);
+		if (ACPI_FAILURE(status))
+			goto out;
+		if (ctrl == *mask)
+			break;
+		decode_cxl_osc_control(dev, "platform does not support",
+				       ctrl & ~(*mask));
+		ctrl = *mask;
+	}
+
+	if ((ctrl & req) != req) {
+		decode_cxl_osc_control(
+			dev,
+			"not requesting control; platform does not support",
+			req & ~(ctrl));
+		status = AE_SUPPORT;
+		goto out;
+	}
+
+	capbuf[OSC_QUERY_DWORD] = 0;
+	capbuf[PCI_OSC_SUPPORT_DWORD] = root->osc_support_set;
+	capbuf[PCI_OSC_CONTROL_DWORD] = root->osc_control_set;
+	capbuf[CXL_OSC_SUPPORT_DWORD] = cxl_osc_support_set;
+	capbuf[CXL_OSC_CONTROL_DWORD] = ctrl;
+	status = acpi_cap_run_osc(handle, capbuf, cxl_osc_uuid_str, mask);
+	if (ACPI_SUCCESS(status))
+		cxl_osc_control_set = *mask;
+out:
+	mutex_unlock(&acpi_desc_lock);
+	return status;
+}
+
+static int cxl_negotiate_osc(struct acpi_device *adev)
+{
+	u32 cxl_support, cxl_control, requested;
+	acpi_handle handle = adev->handle;
+	struct device *dev = &adev->dev;
+	acpi_status status;
+
+	/* Declare support for everything */
+	cxl_support = CXL_OSC_SUPPORT_VALID_MASK;
+	decode_cxl_osc_support(dev, "OS supports", cxl_support);
+	status = cxl_osc_declare_support(handle, cxl_support);
+	if (ACPI_FAILURE(status)) {
+		dev_info(dev, "CXL_OSC failed (%s)\n",
+			 acpi_format_exception(status));
+		return -ENXIO;
+	}
+
+	/* Request control for everything */
+	cxl_control = CXL_OSC_CONTROL_VALID_MASK;
+	requested = cxl_control;
+	status = cxl_osc_request_control(adev, &cxl_control,
+					 CXL_OSC_MEM_ERROR_CONTROL);
+	if (ACPI_SUCCESS(status)) {
+		decode_cxl_osc_control(dev, "OS now controls", cxl_control);
+	} else {
+		decode_cxl_osc_control(dev, "OS requested", requested);
+		decode_cxl_osc_control(dev, "platform willing to grant",
+				       cxl_control);
+		dev_info(dev, "_OSC failed (%s)\n",
+			 acpi_format_exception(status));
+	}
+	return 0;
+}
+
+/**
+ * cxl_bus_acquire - Perform platform-specific bus operations
+ * @pdev: pci_dev associated with the CXL device
+ *
+ * This performs bus-specific operations such as ACPI _OSC to ensure that
+ * the bus is 'prepared'. Since the CXL definition of _OSC depends on the
+ * existing PCI _OSC DWORDS in 'Arg3', pull those in from the pci root port
+ * device, and merge those into the new CXL-augmented _OSC calls.
+ *
  * If/when CXL support is defined by other platform firmware the kernel
  * will need a mechanism to select between the platform specific version
  * of this routine, until then, hard-code ACPI assumptions
@@ -21,6 +273,7 @@ int cxl_bus_acquire(struct pci_dev *pdev)
 	struct acpi_device *adev;
 	struct pci_dev *root_port;
 	struct device *root;
+	int rc;
 
 	root_port = pcie_find_root_port(pdev);
 	if (!root_port)
@@ -34,9 +287,11 @@ int cxl_bus_acquire(struct pci_dev *pdev)
 	if (!adev)
 		return -ENXIO;
 
-	/* TODO: OSC enabling */
+	rc = cxl_negotiate_osc(adev);
+	if (rc)
+		dev_err(&pdev->dev, "Failed to negotiate OSC\n");
 
-	return 0;
+	return rc;
 }
 EXPORT_SYMBOL_GPL(cxl_bus_acquire);
 
diff --git a/drivers/cxl/acpi.h b/drivers/cxl/acpi.h
index d638f8886ab7..6ef154021745 100644
--- a/drivers/cxl/acpi.h
+++ b/drivers/cxl/acpi.h
@@ -12,4 +12,24 @@ struct cxl_acpi_desc {
 
 int cxl_bus_acquire(struct pci_dev *pci_dev);
 
+/* Indexes into _OSC Capabilities Buffer */
+#define PCI_OSC_SUPPORT_DWORD			1	/* DWORD 2 */
+#define PCI_OSC_CONTROL_DWORD			2	/* DWORD 3 */
+#define CXL_OSC_SUPPORT_DWORD			3	/* DWORD 4 */
+#define CXL_OSC_CONTROL_DWORD			4	/* DWORD 5 */
+
+/* CXL Host Bridge _OSC: Capabilities DWORD 4: Support Field */
+#define CXL_OSC_PORT_REG_ACCESS_SUPPORT		0x00000001
+#define CXL_OSC_PORT_DEV_REG_ACCESS_SUPPORT	0x00000002
+#define CXL_OSC_PER_SUPPORT			0x00000004
+#define CXL_OSC_NATIVE_HP_SUPPORT		0x00000008
+#define CXL_OSC_SUPPORT_VALID_MASK		(CXL_OSC_PORT_REG_ACCESS_SUPPORT |	\
+						 CXL_OSC_PORT_DEV_REG_ACCESS_SUPPORT |	\
+						 CXL_OSC_PER_SUPPORT |			\
+						 CXL_OSC_NATIVE_HP_SUPPORT)
+
+/* CXL Host Bridge _OSC: Capabilities DWORD 5: Control Field */
+#define CXL_OSC_MEM_ERROR_CONTROL		0x00000001
+#define CXL_OSC_CONTROL_VALID_MASK		(CXL_OSC_MEM_ERROR_CONTROL)
+
 #endif	/* __CXL_ACPI_H__ */
-- 
2.29.2


^ permalink raw reply related	[flat|nested] 35+ messages in thread

* [RFC PATCH v2 04/14] cxl/mem: Introduce a driver for CXL-2.0-Type-3 endpoints
  2020-12-09  0:24 [RFC PATCH v2 00/14] CXL 2.0 Support Ben Widawsky
                   ` (2 preceding siblings ...)
  2020-12-09  0:24 ` [RFC PATCH v2 03/14] cxl/acpi: add OSC support Ben Widawsky
@ 2020-12-09  0:24 ` Ben Widawsky
  2020-12-09  0:24 ` [RFC PATCH v2 05/14] cxl/mem: Map memory device registers Ben Widawsky
                   ` (11 subsequent siblings)
  15 siblings, 0 replies; 35+ messages in thread
From: Ben Widawsky @ 2020-12-09  0:24 UTC (permalink / raw)
  To: linux-cxl
  Cc: Dan Williams, Ben Widawsky, linux-kernel, linux-pci, linux-acpi,
	Ira Weiny, Vishal Verma, Kelley, Sean V, Rafael Wysocki,
	Bjorn Helgaas, Jonathan Cameron, Jon Masters, Chris Browy,
	Randy Dunlap, Christoph Hellwig

From: Dan Williams <dan.j.williams@intel.com>

The CXL.mem protocol allows a device to act as a provider of "System
RAM" and/or "Persistent Memory" that is fully coherent as if the memory
was attached to the typical CPU memory controller.

With the CXL-2.0 specification a PCI endpoint can implement a "Type-3"
device interface and give the operating system control over "Host
Managed Device Memory". See section 2.3 Type 3 CXL Device.

The memory range exported by the device may optionally be described by
the platform firmware memory map, or by infrastructure like LIBNVDIMM to
provision persistent memory capacity from one, or more, CXL.mem devices.

A pre-requisite for Linux-managed memory-capacity provisioning is this
cxl_mem driver that can speak the mailbox protocol defined in section
8.2.8.4 Mailbox Registers.

For now just land the driver boiler-plate and fill it in with
functionality in subsequent commits.

Link: https://www.computeexpresslink.org/download-the-specification
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
 Documentation/cxl/memory-devices.rst |  9 ++++
 drivers/cxl/Kconfig                  | 22 +++++++++
 drivers/cxl/Makefile                 |  2 +
 drivers/cxl/mem.c                    | 69 ++++++++++++++++++++++++++++
 drivers/cxl/pci.h                    | 20 ++++++++
 5 files changed, 122 insertions(+)
 create mode 100644 drivers/cxl/mem.c
 create mode 100644 drivers/cxl/pci.h

diff --git a/Documentation/cxl/memory-devices.rst b/Documentation/cxl/memory-devices.rst
index 6ce88f9d5f4f..134c9b6b4ff4 100644
--- a/Documentation/cxl/memory-devices.rst
+++ b/Documentation/cxl/memory-devices.rst
@@ -23,6 +23,15 @@ ACPI CXL
 .. kernel-doc:: drivers/cxl/acpi.c
    :internal:
 
+CXL Memory Device
+-----------------
+
+.. kernel-doc:: drivers/cxl/mem.c
+   :doc: cxl mem
+
+.. kernel-doc:: drivers/cxl/mem.c
+   :internal:
+
 External Interfaces
 ===================
 
diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig
index 68da926ba5b1..0ac5080cd6e0 100644
--- a/drivers/cxl/Kconfig
+++ b/drivers/cxl/Kconfig
@@ -33,4 +33,26 @@ config CXL_ACPI
 	  specification.
 
 	  If unsure say 'm'
+
+config CXL_MEM
+        tristate "CXL.mem: Endpoint Support"
+        depends on PCI && CXL_BUS_PROVIDER
+        default CXL_BUS_PROVIDER
+        help
+          The CXL.mem protocol allows a device to act as a provider of
+          "System RAM" and/or "Persistent Memory" that is fully coherent
+          as if the memory was attached to the typical CPU memory
+          controller.
+
+	  Say 'y/m' to enable a driver (named "cxl_mem.ko" when built as
+	  a module) that will attach to CXL.mem devices for
+	  configuration, provisioning, and health monitoring. This
+	  driver is required for dynamic provisioning of CXL.mem
+	  attached memory which is a pre-requisite for persistent memory
+	  support. Typically volatile memory is mapped by platform
+	  firmware and included in the platform memory map, but in some
+	  cases the OS is responsible for mapping that memory. See
+	  Chapter 2.3 Type 3 CXL Device in the CXL 2.0 specification.
+
+          If unsure say 'm'.
 endif
diff --git a/drivers/cxl/Makefile b/drivers/cxl/Makefile
index d38cd34a2582..97fdffb00f2d 100644
--- a/drivers/cxl/Makefile
+++ b/drivers/cxl/Makefile
@@ -1,5 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o
+obj-$(CONFIG_CXL_MEM) += cxl_mem.o
 
 ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL
 cxl_acpi-y := acpi.o
+cxl_mem-y := mem.o
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
new file mode 100644
index 000000000000..005404888942
--- /dev/null
+++ b/drivers/cxl/mem.c
@@ -0,0 +1,69 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(c) 2020 Intel Corporation. All rights reserved. */
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/io.h>
+#include "acpi.h"
+#include "pci.h"
+
+static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec)
+{
+	int pos;
+
+	pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_DVSEC);
+	if (!pos)
+		return 0;
+
+	while (pos) {
+		u16 vendor, id;
+
+		pci_read_config_word(pdev, pos + PCI_DVSEC_VENDOR_ID_OFFSET,
+				     &vendor);
+		pci_read_config_word(pdev, pos + PCI_DVSEC_ID_OFFSET, &id);
+		if (vendor == PCI_DVSEC_VENDOR_ID_CXL && dvsec == id)
+			return pos;
+
+		pos = pci_find_next_ext_capability(pdev, pos,
+						   PCI_EXT_CAP_ID_DVSEC);
+	}
+
+	return 0;
+}
+
+static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
+{
+	struct device *dev = &pdev->dev;
+	int rc, regloc;
+
+	rc = cxl_bus_acquire(pdev);
+	if (rc != 0) {
+		dev_err(dev, "failed to acquire interface\n");
+		return rc;
+	}
+
+	regloc = cxl_mem_dvsec(pdev, PCI_DVSEC_ID_CXL_REGLOC);
+	if (!regloc) {
+		dev_err(dev, "register location dvsec not found\n");
+		return -ENXIO;
+	}
+
+	return 0;
+}
+
+static const struct pci_device_id cxl_mem_pci_tbl[] = {
+	/* PCI class code for CXL.mem Type-3 Devices */
+	{ PCI_ANY_ID, PCI_ANY_ID, PCI_ANY_ID, PCI_ANY_ID,
+	  PCI_CLASS_MEMORY_CXL, 0xffffff, 0 },
+	{ /* terminate list */ },
+};
+MODULE_DEVICE_TABLE(pci, cxl_mem_pci_tbl);
+
+static struct pci_driver cxl_mem_driver = {
+	.name			= KBUILD_MODNAME,
+	.id_table		= cxl_mem_pci_tbl,
+	.probe			= cxl_mem_probe,
+};
+
+MODULE_LICENSE("GPL v2");
+module_pci_driver(cxl_mem_driver);
+MODULE_IMPORT_NS(CXL);
diff --git a/drivers/cxl/pci.h b/drivers/cxl/pci.h
new file mode 100644
index 000000000000..a8a9935fa90b
--- /dev/null
+++ b/drivers/cxl/pci.h
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(c) 2020 Intel Corporation. All rights reserved. */
+#ifndef __CXL_PCI_H__
+#define __CXL_PCI_H__
+
+#define PCI_CLASS_MEMORY_CXL	0x050210
+
+/*
+ * See section 8.1 Configuration Space Registers in the CXL 2.0
+ * Specification
+ */
+#define PCI_EXT_CAP_ID_DVSEC		0x23
+#define PCI_DVSEC_VENDOR_ID_CXL		0x1E98
+#define PCI_DVSEC_VENDOR_ID_OFFSET	0x4
+#define PCI_DVSEC_ID_CXL		0x0
+#define PCI_DVSEC_ID_OFFSET		0x8
+
+#define PCI_DVSEC_ID_CXL_REGLOC		0x8
+
+#endif /* __CXL_PCI_H__ */
-- 
2.29.2


^ permalink raw reply related	[flat|nested] 35+ messages in thread

* [RFC PATCH v2 05/14] cxl/mem: Map memory device registers
  2020-12-09  0:24 [RFC PATCH v2 00/14] CXL 2.0 Support Ben Widawsky
                   ` (3 preceding siblings ...)
  2020-12-09  0:24 ` [RFC PATCH v2 04/14] cxl/mem: Introduce a driver for CXL-2.0-Type-3 endpoints Ben Widawsky
@ 2020-12-09  0:24 ` Ben Widawsky
  2020-12-09  0:24 ` [RFC PATCH v2 06/14] cxl/mem: Find device capabilities Ben Widawsky
                   ` (10 subsequent siblings)
  15 siblings, 0 replies; 35+ messages in thread
From: Ben Widawsky @ 2020-12-09  0:24 UTC (permalink / raw)
  To: linux-cxl
  Cc: Ben Widawsky, linux-kernel, linux-pci, linux-acpi, Ira Weiny,
	Dan Williams, Vishal Verma, Kelley, Sean V, Rafael Wysocki,
	Bjorn Helgaas, Jonathan Cameron, Jon Masters, Chris Browy,
	Randy Dunlap, Christoph Hellwig

All the necessary bits are initialized in order to find and map the
register space for CXL Memory Devices. This is accomplished by using the
Register Locator DVSEC (CXL 2.0 - 8.1.9.1) to determine which PCI BAR to
use, and how much of an offset from that BAR should be added.

If the memory device registers are found and mapped a new internal data
structure tracking device state is allocated.

Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
 drivers/cxl/cxl.h | 17 ++++++++++
 drivers/cxl/mem.c | 83 +++++++++++++++++++++++++++++++++++++++++++++--
 drivers/cxl/pci.h | 14 ++++++++
 3 files changed, 112 insertions(+), 2 deletions(-)
 create mode 100644 drivers/cxl/cxl.h

diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
new file mode 100644
index 000000000000..d81d0ba4617c
--- /dev/null
+++ b/drivers/cxl/cxl.h
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(c) 2020 Intel Corporation. */
+
+#ifndef __CXL_H__
+#define __CXL_H__
+
+/**
+ * struct cxl_mem - A CXL memory device
+ * @pdev: The PCI device associated with this CXL device.
+ * @regs: IO mappings to the device's MMIO
+ */
+struct cxl_mem {
+	struct pci_dev *pdev;
+	void __iomem *regs;
+};
+
+#endif
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index 005404888942..06113d306cd2 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -5,6 +5,58 @@
 #include <linux/io.h>
 #include "acpi.h"
 #include "pci.h"
+#include "cxl.h"
+
+/**
+ * cxl_mem_create() - Create a new &struct cxl_mem.
+ * @pdev: The pci device associated with the new &struct cxl_mem.
+ * @reg_lo: Lower 32b of the register locator
+ * @reg_hi: Upper 32b of the register locator.
+ *
+ * Return: The new &struct cxl_mem on success, NULL on failure.
+ *
+ * Map the BAR for a CXL memory device. This BAR has the memory devices
+ * registers for the device as specified in CXL specification.
+ */
+static struct cxl_mem *cxl_mem_create(struct pci_dev *pdev, u32 reg_lo,
+				      u32 reg_hi)
+{
+	struct device *dev = &pdev->dev;
+	struct cxl_mem *cxlm;
+	void __iomem *regs;
+	u64 offset;
+	u8 bar;
+	int rc;
+
+	offset = ((u64)reg_hi << 32) | (reg_lo & CXL_REGLOC_ADDR_MASK);
+	bar = (reg_lo >> CXL_REGLOC_BIR_SHIFT) & CXL_REGLOC_BIR_MASK;
+
+	/* Basic sanity check that BAR is big enough */
+	if (pci_resource_len(pdev, bar) < offset) {
+		dev_err(dev, "BAR%d: %pr: too small (offset: %#llx)\n", bar,
+			&pdev->resource[bar], (unsigned long long)offset);
+		return NULL;
+	}
+
+	rc = pcim_iomap_regions(pdev, 1 << bar, pci_name(pdev));
+	if (rc != 0) {
+		dev_err(dev, "failed to map registers\n");
+		return NULL;
+	}
+
+	cxlm = devm_kzalloc(&pdev->dev, sizeof(*cxlm), GFP_KERNEL);
+	if (!cxlm) {
+		dev_err(dev, "No memory available\n");
+		return NULL;
+	}
+
+	regs = pcim_iomap_table(pdev)[bar];
+	cxlm->pdev = pdev;
+	cxlm->regs = regs + offset;
+
+	dev_dbg(dev, "Mapped CXL Memory Device resource\n");
+	return cxlm;
+}
 
 static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec)
 {
@@ -33,7 +85,8 @@ static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec)
 static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 {
 	struct device *dev = &pdev->dev;
-	int rc, regloc;
+	struct cxl_mem *cxlm;
+	int rc, regloc, i;
 
 	rc = cxl_bus_acquire(pdev);
 	if (rc != 0) {
@@ -41,13 +94,39 @@ static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 		return rc;
 	}
 
+	rc = pcim_enable_device(pdev);
+	if (rc)
+		return rc;
+
 	regloc = cxl_mem_dvsec(pdev, PCI_DVSEC_ID_CXL_REGLOC);
 	if (!regloc) {
 		dev_err(dev, "register location dvsec not found\n");
 		return -ENXIO;
 	}
+	regloc += 0xc; /* Skip DVSEC + reserved fields */
 
-	return 0;
+	rc = -ENXIO;
+	for (i = regloc; i < regloc + 0x24; i += 8) {
+		u32 reg_lo, reg_hi;
+		u8 reg_type;
+
+		/* "register low and high" contain other bits */
+		pci_read_config_dword(pdev, i, &reg_lo);
+		pci_read_config_dword(pdev, i + 4, &reg_hi);
+
+		reg_type =
+			(reg_lo >> CXL_REGLOC_RBI_SHIFT) & CXL_REGLOC_RBI_MASK;
+
+		if (reg_type == CXL_REGLOC_RBI_MEMDEV) {
+			rc = 0;
+			cxlm = cxl_mem_create(pdev, reg_lo, reg_hi);
+			if (!cxlm)
+				rc = -ENODEV;
+			break;
+		}
+	}
+
+	return rc;
 }
 
 static const struct pci_device_id cxl_mem_pci_tbl[] = {
diff --git a/drivers/cxl/pci.h b/drivers/cxl/pci.h
index a8a9935fa90b..df222edb6ac3 100644
--- a/drivers/cxl/pci.h
+++ b/drivers/cxl/pci.h
@@ -17,4 +17,18 @@
 
 #define PCI_DVSEC_ID_CXL_REGLOC		0x8
 
+/* BAR Indicator Register (BIR) */
+#define CXL_REGLOC_BIR_SHIFT 0
+#define CXL_REGLOC_BIR_MASK 0x7
+
+/* Register Block Identifier (RBI) */
+#define CXL_REGLOC_RBI_SHIFT 8
+#define CXL_REGLOC_RBI_MASK 0xff
+#define CXL_REGLOC_RBI_EMPTY 0
+#define CXL_REGLOC_RBI_COMPONENT 1
+#define CXL_REGLOC_RBI_VIRT 2
+#define CXL_REGLOC_RBI_MEMDEV 3
+
+#define CXL_REGLOC_ADDR_MASK 0xffff0000
+
 #endif /* __CXL_PCI_H__ */
-- 
2.29.2


^ permalink raw reply related	[flat|nested] 35+ messages in thread

* [RFC PATCH v2 06/14] cxl/mem: Find device capabilities
  2020-12-09  0:24 [RFC PATCH v2 00/14] CXL 2.0 Support Ben Widawsky
                   ` (4 preceding siblings ...)
  2020-12-09  0:24 ` [RFC PATCH v2 05/14] cxl/mem: Map memory device registers Ben Widawsky
@ 2020-12-09  0:24 ` Ben Widawsky
  2020-12-09  0:24 ` [RFC PATCH v2 07/14] cxl/mem: Implement polled mode mailbox Ben Widawsky
                   ` (9 subsequent siblings)
  15 siblings, 0 replies; 35+ messages in thread
From: Ben Widawsky @ 2020-12-09  0:24 UTC (permalink / raw)
  To: linux-cxl
  Cc: Ben Widawsky, linux-kernel, linux-pci, linux-acpi, Ira Weiny,
	Dan Williams, Vishal Verma, Kelley, Sean V, Rafael Wysocki,
	Bjorn Helgaas, Jonathan Cameron, Jon Masters, Chris Browy,
	Randy Dunlap, Christoph Hellwig

CXL devices contain an array of capabilities that describe the
interactions software can interact with the device, or firmware running
on the device. A CXL compliant device must implement the device status
and the mailbox capability. A CXL compliant memory device must implement
the memory device capability.

Each of the capabilities can [will] provide an offset within the MMIO
region for interacting with the CXL device.

Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
 drivers/cxl/cxl.h | 92 ++++++++++++++++++++++++++++++++++++++++++++++-
 drivers/cxl/mem.c | 74 +++++++++++++++++++++++++++++++++++++-
 2 files changed, 164 insertions(+), 2 deletions(-)

diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index d81d0ba4617c..91bb40f73ddc 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -4,6 +4,38 @@
 #ifndef __CXL_H__
 #define __CXL_H__
 
+#include <linux/bitfield.h>
+#include <linux/bitops.h>
+#include <linux/io.h>
+
+#define CXL_SET_FIELD(value, field)                                            \
+	({                                                                     \
+		WARN_ON(!FIELD_FIT(field##_MASK, value));                      \
+		FIELD_PREP(field##_MASK, value);                               \
+	})
+
+#define CXL_GET_FIELD(word, field) FIELD_GET(field##_MASK, word)
+
+/* Device */
+#define CXLDEV_CAP_ARRAY_REG 0x0
+#define CXLDEV_CAP_ARRAY_CAP_ID 0
+#define CXLDEV_CAP_ARRAY_ID(x) ((x) & (0xffff))
+#define CXLDEV_CAP_ARRAY_COUNT(x) (((x) >> 32) & 0xffff)
+
+#define CXL_CAP_CAP_ID_DEVICE_STATUS 0x1
+#define CXL_CAP_CAP_ID_PRIMARY_MAILBOX 0x2
+#define CXL_CAP_CAP_ID_SECONDARY_MAILBOX 0x3
+#define CXL_CAP_CAP_ID_MEMDEV 0x4000
+
+/* Mailbox (CXL 2.0 - 8.2.8.4) */
+#define CXLDEV_MB_CAPS_OFFSET 0x00
+#define   CXLDEV_MB_CAP_PAYLOAD_SIZE_MASK GENMASK(5, 0)
+#define   CXLDEV_MB_CAP_PAYLOAD_SIZE_SHIFT 0
+#define CXLDEV_MB_CTRL_OFFSET 0x04
+#define CXLDEV_MB_CMD_OFFSET 0x08
+#define CXLDEV_MB_STATUS_OFFSET 0x10
+#define CXLDEV_MB_BG_CMD_STATUS_OFFSET 0x18
+
 /**
  * struct cxl_mem - A CXL memory device
  * @pdev: The PCI device associated with this CXL device.
@@ -12,6 +44,64 @@
 struct cxl_mem {
 	struct pci_dev *pdev;
 	void __iomem *regs;
+
+	/* Cap 0001h - CXL_CAP_CAP_ID_DEVICE_STATUS */
+	struct {
+		void __iomem *regs;
+	} status;
+
+	/* Cap 0002h - CXL_CAP_CAP_ID_PRIMARY_MAILBOX */
+	struct {
+		void __iomem *regs;
+		size_t payload_size;
+	} mbox;
+
+	/* Cap 4000h - CXL_CAP_CAP_ID_MEMDEV */
+	struct {
+		void __iomem *regs;
+	} mem;
 };
 
-#endif
+#define cxl_reg(type)                                                          \
+	static inline void cxl_write_##type##_reg32(struct cxl_mem *cxlm,      \
+						    u32 reg, u32 value)        \
+	{                                                                      \
+		void __iomem *reg_addr = cxlm->type.regs;                      \
+		writel(value, reg_addr + reg);                                 \
+	}                                                                      \
+	static inline void cxl_write_##type##_reg64(struct cxl_mem *cxlm,      \
+						    u32 reg, u64 value)        \
+	{                                                                      \
+		void __iomem *reg_addr = cxlm->type.regs;                      \
+		writeq(value, reg_addr + reg);                                 \
+	}                                                                      \
+	static inline u32 cxl_read_##type##_reg32(struct cxl_mem *cxlm,        \
+						  u32 reg)                     \
+	{                                                                      \
+		void __iomem *reg_addr = cxlm->type.regs;                      \
+		return readl(reg_addr + reg);                                  \
+	}                                                                      \
+	static inline u64 cxl_read_##type##_reg64(struct cxl_mem *cxlm,        \
+						  u32 reg)                     \
+	{                                                                      \
+		void __iomem *reg_addr = cxlm->type.regs;                      \
+		return readq(reg_addr + reg);                                  \
+	}
+
+cxl_reg(status);
+cxl_reg(mbox);
+
+static inline u32 __cxl_read_reg32(struct cxl_mem *cxlm, u32 reg)
+{
+	void __iomem *reg_addr = cxlm->regs;
+
+	return readl(reg_addr + reg);
+}
+
+static inline u64 __cxl_read_reg64(struct cxl_mem *cxlm, u32 reg)
+{
+	void __iomem *reg_addr = cxlm->regs;
+
+	return readq(reg_addr + reg);
+}
+#endif /* __CXL_H__ */
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index 06113d306cd2..8ea830ffdbba 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -7,6 +7,75 @@
 #include "pci.h"
 #include "cxl.h"
 
+/**
+ * cxl_mem_setup_regs() - Setup necessary MMIO.
+ * @cxlm: The CXL memory device to communicate with.
+ *
+ * Return: 0 if all necessary registers mapped.
+ *
+ * A memory device is required by spec to implement a certain set of MMIO
+ * regions. The purpose of this function is to enumerate and map those
+ * registers.
+ */
+static int cxl_mem_setup_regs(struct cxl_mem *cxlm)
+{
+	u64 cap_array;
+	int cap;
+
+	/*
+	 * Register accessors need the mappings set up by this function, so the
+	 * raw versions of the mmio accessors must be used instead.
+	 */
+	cap_array = __cxl_read_reg64(cxlm, CXLDEV_CAP_ARRAY_REG);
+	if (CXLDEV_CAP_ARRAY_ID(cap_array) != CXLDEV_CAP_ARRAY_CAP_ID)
+		return -ENODEV;
+
+	for (cap = 1; cap <= CXLDEV_CAP_ARRAY_COUNT(cap_array); cap++) {
+		void *__iomem register_block;
+		u32 offset;
+		u16 cap_id;
+
+		cap_id = __cxl_read_reg32(cxlm, cap * 0x10) & 0xffff;
+		offset = __cxl_read_reg32(cxlm, cap * 0x10 + 0x4);
+		register_block = cxlm->regs + offset;
+
+		switch (cap_id) {
+		case CXL_CAP_CAP_ID_DEVICE_STATUS:
+			dev_dbg(&cxlm->pdev->dev,
+				"found Status capability (0x%x)\n", offset);
+			cxlm->status.regs = register_block;
+			break;
+		case CXL_CAP_CAP_ID_PRIMARY_MAILBOX:
+			dev_dbg(&cxlm->pdev->dev,
+				"found Mailbox capability (0x%x)\n", offset);
+			cxlm->mbox.regs = register_block;
+			cxlm->mbox.payload_size =
+				CXL_GET_FIELD(cap, CXLDEV_MB_CAP_PAYLOAD_SIZE);
+			break;
+		case CXL_CAP_CAP_ID_SECONDARY_MAILBOX:
+			dev_dbg(&cxlm->pdev->dev,
+				"found Secondary Mailbox capability (0x%x)\n",
+				offset);
+			break;
+		case CXL_CAP_CAP_ID_MEMDEV:
+			dev_dbg(&cxlm->pdev->dev,
+				"found Memory Device capability (0x%x)\n",
+				offset);
+			cxlm->mem.regs = register_block;
+			break;
+		default:
+			dev_err(&cxlm->pdev->dev, "Unknown cap ID: %d (0x%x)\n",
+				cap_id, offset);
+			return -ENXIO;
+		}
+	}
+
+	if (!cxlm->status.regs || !cxlm->mbox.regs || !cxlm->mem.regs)
+		return -ENXIO;
+
+	return 0;
+}
+
 /**
  * cxl_mem_create() - Create a new &struct cxl_mem.
  * @pdev: The pci device associated with the new &struct cxl_mem.
@@ -126,7 +195,10 @@ static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 		}
 	}
 
-	return rc;
+	if (rc)
+		return rc;
+
+	return cxl_mem_setup_regs(cxlm);
 }
 
 static const struct pci_device_id cxl_mem_pci_tbl[] = {
-- 
2.29.2


^ permalink raw reply related	[flat|nested] 35+ messages in thread

* [RFC PATCH v2 07/14] cxl/mem: Implement polled mode mailbox
  2020-12-09  0:24 [RFC PATCH v2 00/14] CXL 2.0 Support Ben Widawsky
                   ` (5 preceding siblings ...)
  2020-12-09  0:24 ` [RFC PATCH v2 06/14] cxl/mem: Find device capabilities Ben Widawsky
@ 2020-12-09  0:24 ` Ben Widawsky
  2021-01-07 19:05   ` Ben Widawsky
  2020-12-09  0:24 ` [RFC PATCH v2 08/14] cxl/mem: Register CXL memX devices Ben Widawsky
                   ` (8 subsequent siblings)
  15 siblings, 1 reply; 35+ messages in thread
From: Ben Widawsky @ 2020-12-09  0:24 UTC (permalink / raw)
  To: linux-cxl
  Cc: Ben Widawsky, linux-kernel, linux-pci, linux-acpi, Ira Weiny,
	Dan Williams, Vishal Verma, Kelley, Sean V, Rafael Wysocki,
	Bjorn Helgaas, Jonathan Cameron, Jon Masters, Chris Browy,
	Randy Dunlap, Christoph Hellwig

Provide enough functionality to utilize the mailbox of a memory device.
The mailbox is used to interact with the firmware running on the memory
device.

The CXL specification defines separate capabilities for the mailbox and
the memory device. While we can confirm the mailbox is ready, in order
to actually interact with the memory device, you must also confirm the
device's firmware is ready.

Create a function to handle sending a command, optionally with a
payload, to the memory device, polling on a result, and then optionally
copying out the payload. The algorithm for doing this comes straight out
of the CXL 2.0 specification.

Primary mailboxes are capable of generating an interrupt when submitting
a command in the background. That implementation is saved for a later
time.

Secondary mailboxes aren't implemented at this time.

The flow is proven with one implemented command, "identify".

Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
 drivers/cxl/cxl.h |  75 +++++++++++++
 drivers/cxl/mem.c | 265 +++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 339 insertions(+), 1 deletion(-)

diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index 91bb40f73ddc..6d641309013d 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -32,9 +32,38 @@
 #define   CXLDEV_MB_CAP_PAYLOAD_SIZE_MASK GENMASK(5, 0)
 #define   CXLDEV_MB_CAP_PAYLOAD_SIZE_SHIFT 0
 #define CXLDEV_MB_CTRL_OFFSET 0x04
+#define   CXLDEV_MB_CTRL_DOORBELL BIT(0)
 #define CXLDEV_MB_CMD_OFFSET 0x08
+#define   CXLDEV_MB_CMD_PAYLOAD_LENGTH_SHIFT 16
+#define   CXLDEV_MB_CMD_PAYLOAD_LENGTH_MASK GENMASK(36, 16)
 #define CXLDEV_MB_STATUS_OFFSET 0x10
+#define   CXLDEV_MB_STATUS_RET_CODE_SHIFT 32
+#define   CXLDEV_MB_STATUS_RET_CODE_MASK GENMASK(47, 32)
 #define CXLDEV_MB_BG_CMD_STATUS_OFFSET 0x18
+#define CXLDEV_MB_PAYLOAD_OFFSET 0x20
+
+/* Memory Device (CXL 2.0 - 8.2.8.5.1.1) */
+#define CXLMDEV_STATUS_OFFSET 0x0
+#define   CXLMDEV_DEV_FATAL BIT(0)
+#define   CXLMDEV_FW_HALT BIT(1)
+#define   CXLMDEV_STATUS_MEDIA_STATUS_SHIFT 2
+#define   CXLMDEV_STATUS_MEDIA_STATUS_MASK GENMASK(3, 2)
+#define     CXLMDEV_MS_NOT_READY 0
+#define     CXLMDEV_MS_READY 1
+#define     CXLMDEV_MS_ERROR 2
+#define     CXLMDEV_MS_DISABLED 3
+#define   CXLMDEV_READY(status) \
+		(CXL_GET_FIELD(status, CXLMDEV_STATUS_MEDIA_STATUS) == CXLMDEV_MS_READY)
+#define   CXLMDEV_MBOX_IF_READY BIT(4)
+#define   CXLMDEV_RESET_NEEDED_SHIFT 5
+#define   CXLMDEV_RESET_NEEDED_MASK GENMASK(7, 5)
+#define     CXLMDEV_RESET_NEEDED_NOT 0
+#define     CXLMDEV_RESET_NEEDED_COLD 1
+#define     CXLMDEV_RESET_NEEDED_WARM 2
+#define     CXLMDEV_RESET_NEEDED_HOT 3
+#define     CXLMDEV_RESET_NEEDED_CXL 4
+#define   CXLMDEV_RESET_NEEDED(status) \
+		(CXL_GET_FIELD(status, CXLMDEV_RESET_NEEDED) != CXLMDEV_RESET_NEEDED_NOT)
 
 /**
  * struct cxl_mem - A CXL memory device
@@ -45,6 +74,18 @@ struct cxl_mem {
 	struct pci_dev *pdev;
 	void __iomem *regs;
 
+	spinlock_t mbox_lock; /* Protects device mailbox and firmware */
+
+	struct {
+		struct range range;
+	} pmem;
+
+	struct {
+		struct range range;
+	} ram;
+
+	char firmware_version[0x10];
+
 	/* Cap 0001h - CXL_CAP_CAP_ID_DEVICE_STATUS */
 	struct {
 		void __iomem *regs;
@@ -90,6 +131,7 @@ struct cxl_mem {
 
 cxl_reg(status);
 cxl_reg(mbox);
+cxl_reg(mem);
 
 static inline u32 __cxl_read_reg32(struct cxl_mem *cxlm, u32 reg)
 {
@@ -104,4 +146,37 @@ static inline u64 __cxl_read_reg64(struct cxl_mem *cxlm, u32 reg)
 
 	return readq(reg_addr + reg);
 }
+
+static inline void cxl_mbox_payload_fill(struct cxl_mem *cxlm, u8 *input,
+					 unsigned int length)
+{
+	memcpy_toio(cxlm->mbox.regs + CXLDEV_MB_PAYLOAD_OFFSET, input, length);
+}
+
+static inline void cxl_mbox_payload_drain(struct cxl_mem *cxlm, u8 *output,
+					  unsigned int length)
+{
+	memcpy_fromio(output, cxlm->mbox.regs + CXLDEV_MB_PAYLOAD_OFFSET,
+		      length);
+}
+
+#define CXL_MBOX_IDENTIFY 0x4000
+
+struct cxl_mbox_identify {
+	char fw_revision[0x10];
+	__le64 total_capacity;
+	__le64 volatile_capacity;
+	__le64 persistent_capacity;
+	__le64 partition_align;
+	__le16 info_event_log_size;
+	__le16 warning_event_log_size;
+	__le16 failure_event_log_size;
+	__le16 fatal_event_log_size;
+	__le32 lsa_size;
+	u8 poison_list_max_mer[3];
+	__le16 inject_poison_limit;
+	u8 poison_caps;
+	u8 qos_telemetry_caps;
+} __packed;
+
 #endif /* __CXL_H__ */
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index 8ea830ffdbba..82984bf58a8d 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -1,5 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0-only
 /* Copyright(c) 2020 Intel Corporation. All rights reserved. */
+#include <linux/sched/clock.h>
 #include <linux/module.h>
 #include <linux/pci.h>
 #include <linux/io.h>
@@ -7,6 +8,211 @@
 #include "pci.h"
 #include "cxl.h"
 
+#define cxl_doorbell_busy(cxlm)                                                \
+	(cxl_read_mbox_reg32(cxlm, CXLDEV_MB_CTRL_OFFSET) &                    \
+	 CXLDEV_MB_CTRL_DOORBELL)
+
+/**
+ * struct mbox_cmd - A command to be submitted to hardware.
+ * @opcode: (input) The command set and command submitted to hardware.
+ * @payload: (input/output) Pointer to the input and output payload.
+ * @size_in: (input) Number of bytes to load from @payload.
+ * @size_out:
+ *  - (input) Number of bytes allocated to load into @payload.
+ *  - (output) Number of bytes loaded into @payload.
+ * @return_code: (output) Error code returned from hardware.
+ *
+ * This is the primary mechanism used to send commands to the hardware.
+ * All the fields except @payload correspond exactly to the fields described in
+ * Command Register section of the CXL 2.0 spec (8.2.8.4.5). @payload
+ * corresponds to the Command Payload Registers (8.2.8.4.8).
+ */
+struct mbox_cmd {
+	u16 opcode;
+	u8 *payload;
+	size_t size_in;
+	size_t size_out;
+	u16 return_code;
+};
+
+static int cxl_mem_wait_for_doorbell(struct cxl_mem *cxlm)
+{
+	const int timeout = msecs_to_jiffies(2000);
+	const unsigned long start = jiffies;
+
+	while (cxl_doorbell_busy(cxlm)) {
+		if (time_after(jiffies, start + timeout)) {
+			/* Check again in case preempted before timeout test */
+			if (!cxl_doorbell_busy(cxlm))
+				break;
+			return -ETIMEDOUT;
+		}
+		cpu_relax();
+	}
+
+	return 0;
+}
+
+/**
+ * cxl_mem_mbox_send_cmd() - Send a mailbox command to a memory device.
+ * @cxlm: The CXL memory device to communicate with.
+ * @mbox_cmd: Command to send to the memory device.
+ *
+ * Context: Any context. Expects mbox_lock to be held.
+ * Return: 0 if the doorbell transaction was successful from a protocol level.
+ * Caller should check the return code in @mbox_cmd to make sure it succeeded.
+ */
+static int cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm,
+				 struct mbox_cmd *mbox_cmd)
+{
+	u64 cmd, status;
+	size_t out_len;
+	int rc;
+
+	lockdep_assert_held(&cxlm->mbox_lock);
+
+	/*
+	 * Here are the steps from 8.2.8.4 of the CXL 2.0 spec.
+	 *   1. Caller reads MB Control Register to verify doorbell is clear
+	 *   2. Caller writes Command Register
+	 *   3. Caller writes Command Payload Registers if input payload is non-empty
+	 *   4. Caller writes MB Control Register to set doorbell
+	 *   5. Caller either polls for doorbell to be clear or waits for interrupt if configured
+	 *   6. Caller reads MB Status Register to fetch Return code
+	 *   7. If command successful, Caller reads Command Register to get Payload Length
+	 *   8. If output payload is non-empty, host reads Command Payload Registers
+	 */
+
+	/* #1 */
+	WARN_ON(cxl_doorbell_busy(cxlm));
+
+	/* #2 */
+	cxl_write_mbox_reg64(cxlm, CXLDEV_MB_CMD_OFFSET, mbox_cmd->opcode);
+
+	if (mbox_cmd->size_in) {
+		/* #3 */
+		CXL_SET_FIELD(mbox_cmd->size_in, CXLDEV_MB_CMD_PAYLOAD_LENGTH);
+		cxl_mbox_payload_fill(cxlm, mbox_cmd->payload,
+				      mbox_cmd->size_in);
+	}
+
+	/* #4 */
+	cxl_write_mbox_reg32(cxlm, CXLDEV_MB_CTRL_OFFSET,
+			     CXLDEV_MB_CTRL_DOORBELL);
+
+	/* #5 */
+	rc = cxl_mem_wait_for_doorbell(cxlm);
+	if (rc == -ETIMEDOUT) {
+		dev_warn(&cxlm->pdev->dev, "Mailbox command timed out\n");
+		return rc;
+	}
+
+	/* #6 */
+	status = cxl_read_mbox_reg64(cxlm, CXLDEV_MB_STATUS_OFFSET);
+	cmd = cxl_read_mbox_reg64(cxlm, CXLDEV_MB_CMD_OFFSET);
+
+	mbox_cmd->return_code =
+		CXL_GET_FIELD(status, CXLDEV_MB_STATUS_RET_CODE);
+
+	/* There was a problem, let the caller deal with it */
+	if (mbox_cmd->return_code != 0)
+		return 0;
+
+	/* #7 */
+	out_len = CXL_GET_FIELD(cmd, CXLDEV_MB_CMD_PAYLOAD_LENGTH);
+	if (out_len > mbox_cmd->size_out)
+		return -EINVAL;
+
+	mbox_cmd->size_out = out_len;
+
+	/* #8 */
+	if (mbox_cmd->size_out)
+		cxl_mbox_payload_drain(cxlm, mbox_cmd->payload,
+				       mbox_cmd->size_out);
+
+	return 0;
+}
+
+/**
+ * cxl_mem_mbox_get() - Acquire exclusive access to the mailbox.
+ * @cxlm: The memory device to gain access to.
+ *
+ * Context: Any context. Takes the mbox_lock.
+ * Return: 0 if exclusive access was acquired.
+ */
+static int cxl_mem_mbox_get(struct cxl_mem *cxlm)
+{
+	u64 md_status;
+	int rc = -EBUSY;
+
+	spin_lock(&cxlm->mbox_lock);
+
+	/*
+	 * XXX: There is some amount of ambiguity in the 2.0 version of the spec
+	 * around the mailbox interface ready (8.2.8.5.1.1). The purpose of the
+	 * bit is to allow firmware running on the device to notify us that it's
+	 * ready to receive commands. It is unclear if the bit needs to be read
+	 * every time one tries to use the mailbox, ie. the firmware can switch
+	 * it on and off as needed. Second, there is no defined timeout for
+	 * mailbox ready, like there is for the doorbell interface.
+	 *
+	 * As such, we make the following assumptions:
+	 * 1. The firmware might toggle the Mailbox Interface Ready bit, and so
+	 *    we check it for every command.
+	 * 2. If the doorbell is clear, the firmware should have first set the
+	 *    Mailbox Interface Ready bit. Therefore, waiting for the doorbell
+	 *    to be ready is a sufficient amount of time.
+	 */
+	rc = cxl_mem_wait_for_doorbell(cxlm);
+	if (rc) {
+		dev_warn(&cxlm->pdev->dev, "Mailbox interface not ready\n");
+		goto out;
+	}
+
+	md_status = cxl_read_mem_reg64(cxlm, CXLMDEV_STATUS_OFFSET);
+	if (md_status & CXLMDEV_MBOX_IF_READY && CXLMDEV_READY(md_status)) {
+		/*
+		 * Hardware shouldn't allow a ready status but also have failure
+		 * bits set. Spit out an error, this should be a bug report
+		 */
+		if (md_status & CXLMDEV_DEV_FATAL) {
+			dev_err(&cxlm->pdev->dev,
+				"CXL device reporting ready and fatal\n");
+			rc = -EFAULT;
+			goto out;
+		}
+		if (md_status & CXLMDEV_FW_HALT) {
+			dev_err(&cxlm->pdev->dev,
+				"CXL device reporting ready and halted\n");
+			rc = -EFAULT;
+			goto out;
+		}
+		if (CXLMDEV_RESET_NEEDED(md_status)) {
+			dev_err(&cxlm->pdev->dev,
+				"CXL device reporting ready and reset needed\n");
+			rc = -EFAULT;
+			goto out;
+		}
+
+		return 0;
+	}
+
+out:
+	spin_unlock(&cxlm->mbox_lock);
+	return rc;
+}
+
+/**
+ * cxl_mem_mbox_put() - Release exclusive access to the mailbox.
+ * @cxlm: The CXL memory device to communicate with.
+ *
+ * Context: Any context. Expects mbox_lock to be held.
+ */
+static void cxl_mem_mbox_put(struct cxl_mem *cxlm)
+{
+	spin_unlock(&cxlm->mbox_lock);
+}
+
 /**
  * cxl_mem_setup_regs() - Setup necessary MMIO.
  * @cxlm: The CXL memory device to communicate with.
@@ -119,6 +325,8 @@ static struct cxl_mem *cxl_mem_create(struct pci_dev *pdev, u32 reg_lo,
 		return NULL;
 	}
 
+	spin_lock_init(&cxlm->mbox_lock);
+
 	regs = pcim_iomap_table(pdev)[bar];
 	cxlm->pdev = pdev;
 	cxlm->regs = regs + offset;
@@ -151,6 +359,57 @@ static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec)
 	return 0;
 }
 
+/**
+ * cxl_mem_identify() - Send the IDENTIFY command to the device.
+ * @cxlm: The device to identify.
+ *
+ * Return: 0 if identify was executed successfully.
+ *
+ * This will dispatch the identify command to the device and on success populate
+ * structures to be exported to sysfs.
+ */
+static int cxl_mem_identify(struct cxl_mem *cxlm)
+{
+	struct cxl_mbox_identify *id;
+	struct mbox_cmd mbox_cmd;
+	u8 pload[256];
+	int rc;
+
+	/* Retrieve initial device memory map */
+	rc = cxl_mem_mbox_get(cxlm);
+	if (rc)
+		return rc;
+
+	mbox_cmd = (struct mbox_cmd){
+		.opcode = CXL_MBOX_IDENTIFY,
+		.payload = pload,
+		.size_in = 0,
+		.size_out = sizeof(pload),
+	};
+	rc = cxl_mem_mbox_send_cmd(cxlm, &mbox_cmd);
+	if (rc)
+		goto out;
+
+	id = (struct cxl_mbox_identify *)mbox_cmd.payload;
+
+	/*
+	 * TODO: enumerate DPA map, as 'ram' and 'pmem' do not alias.
+	 * For now, only the capacity is exported in sysfs
+	 */
+	cxlm->ram.range.start = 0;
+	cxlm->ram.range.end = le64_to_cpu(id->volatile_capacity) - 1;
+
+	cxlm->pmem.range.start = 0;
+	cxlm->pmem.range.end = le64_to_cpu(id->persistent_capacity) - 1;
+
+	memcpy(cxlm->firmware_version, id->fw_revision,
+	       sizeof(id->fw_revision));
+out:
+	cxl_mem_mbox_put(cxlm);
+
+	return rc;
+}
+
 static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 {
 	struct device *dev = &pdev->dev;
@@ -198,7 +457,11 @@ static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	if (rc)
 		return rc;
 
-	return cxl_mem_setup_regs(cxlm);
+	rc = cxl_mem_setup_regs(cxlm);
+	if (rc)
+		return rc;
+
+	return cxl_mem_identify(cxlm);
 }
 
 static const struct pci_device_id cxl_mem_pci_tbl[] = {
-- 
2.29.2


^ permalink raw reply related	[flat|nested] 35+ messages in thread

* [RFC PATCH v2 08/14] cxl/mem: Register CXL memX devices
  2020-12-09  0:24 [RFC PATCH v2 00/14] CXL 2.0 Support Ben Widawsky
                   ` (6 preceding siblings ...)
  2020-12-09  0:24 ` [RFC PATCH v2 07/14] cxl/mem: Implement polled mode mailbox Ben Widawsky
@ 2020-12-09  0:24 ` Ben Widawsky
  2020-12-09  0:24 ` [RFC PATCH 09/14] cxl/mem: Add basic IOCTL interface Ben Widawsky
                   ` (7 subsequent siblings)
  15 siblings, 0 replies; 35+ messages in thread
From: Ben Widawsky @ 2020-12-09  0:24 UTC (permalink / raw)
  To: linux-cxl
  Cc: Dan Williams, Ben Widawsky, linux-kernel, linux-pci, linux-acpi,
	Ira Weiny, Vishal Verma, Kelley, Sean V, Rafael Wysocki,
	Bjorn Helgaas, Jonathan Cameron, Jon Masters, Chris Browy,
	Randy Dunlap, Christoph Hellwig

From: Dan Williams <dan.j.williams@intel.com>

Create the /sys/bus/cxl hierarchy to enumerate memory devices
(per-endpoint control devices), memory address space devices (platform
address ranges with interleaving, performance, and persistence
attributes), and memory regions (active provisioned memory from an
address space device that is in use as System RAM or delegated to
libnvdimm as Persistent Memory regions).

For now, only the per-endpoint control devices are registered on the
'cxl' bus.

Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
 Documentation/ABI/testing/sysfs-bus-cxl |  26 +++
 Documentation/cxl/memory-devices.rst    |   3 +
 drivers/cxl/Makefile                    |   2 +
 drivers/cxl/bus.c                       |  54 +++++
 drivers/cxl/bus.h                       |   8 +
 drivers/cxl/cxl.h                       |   3 +
 drivers/cxl/mem.c                       | 249 +++++++++++++++++++++++-
 7 files changed, 343 insertions(+), 2 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-bus-cxl
 create mode 100644 drivers/cxl/bus.c
 create mode 100644 drivers/cxl/bus.h

diff --git a/Documentation/ABI/testing/sysfs-bus-cxl b/Documentation/ABI/testing/sysfs-bus-cxl
new file mode 100644
index 000000000000..fe7b87eba988
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-bus-cxl
@@ -0,0 +1,26 @@
+What:		/sys/bus/cxl/devices/memX/firmware_version
+Date:		December, 2020
+KernelVersion:	v5.12
+Contact:	linux-cxl@vger.kernel.org
+Description:
+		(RO) "FW Revision" string as reported by the Identify
+		Memory Device Output Payload in the CXL-2.0
+		specification.
+
+What:		/sys/bus/cxl/devices/memX/ram/size
+Date:		December, 2020
+KernelVersion:	v5.12
+Contact:	linux-cxl@vger.kernel.org
+Description:
+		(RO) "Volatile Only Capacity" as reported by the
+		Identify Memory Device Output Payload in the CXL-2.0
+		specification.
+
+What:		/sys/bus/cxl/devices/memX/pmem/size
+Date:		December, 2020
+KernelVersion:	v5.12
+Contact:	linux-cxl@vger.kernel.org
+Description:
+		(RO) "Persistent Only Capacity" as reported by the
+		Identify Memory Device Output Payload in the CXL-2.0
+		specification.
diff --git a/Documentation/cxl/memory-devices.rst b/Documentation/cxl/memory-devices.rst
index 134c9b6b4ff4..5f723c25382b 100644
--- a/Documentation/cxl/memory-devices.rst
+++ b/Documentation/cxl/memory-devices.rst
@@ -37,3 +37,6 @@ External Interfaces
 
 .. kernel-doc:: drivers/cxl/acpi.c
    :export:
+
+.. kernel-doc:: drivers/cxl/bus.c
+   :export:
diff --git a/drivers/cxl/Makefile b/drivers/cxl/Makefile
index 97fdffb00f2d..f9baaeb94bda 100644
--- a/drivers/cxl/Makefile
+++ b/drivers/cxl/Makefile
@@ -1,7 +1,9 @@
 # SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_CXL_BUS_PROVIDER) += cxl_bus.o
 obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o
 obj-$(CONFIG_CXL_MEM) += cxl_mem.o
 
 ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL
+cxl_bus-y := bus.o
 cxl_acpi-y := acpi.o
 cxl_mem-y := mem.o
diff --git a/drivers/cxl/bus.c b/drivers/cxl/bus.c
new file mode 100644
index 000000000000..91fe765bea94
--- /dev/null
+++ b/drivers/cxl/bus.c
@@ -0,0 +1,54 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(c) 2020 Intel Corporation. All rights reserved. */
+#include <linux/device.h>
+#include <linux/module.h>
+#include "bus.h"
+
+static struct bus_type cxl_bus_type = {
+	.name = "cxl",
+};
+
+static void cxl_unregister(void *dev)
+{
+	device_unregister(dev);
+}
+
+/**
+ * cxl_register() - Register a device on the CXL bus.
+ * @dev: The device to register onto the CXL bus.
+ *
+ * A CXL device driver must call this in order to have the device be a part of
+ * the CXL bus. All endpoint device drivers should utilize this function.
+ *
+ * Return: 0 on success.
+ */
+int cxl_register(struct device *dev)
+{
+	int rc;
+
+	if (!dev->parent || !dev->parent->driver)
+		return -EINVAL;
+
+	dev->bus = &cxl_bus_type;
+	rc = device_add(dev);
+	if (rc)
+		put_device(dev);
+	else
+		rc = devm_add_action_or_reset(dev->parent, cxl_unregister, dev);
+	return rc;
+}
+EXPORT_SYMBOL_GPL(cxl_register);
+
+static __init int cxl_bus_init(void)
+{
+	return bus_register(&cxl_bus_type);
+}
+
+static void cxl_bus_exit(void)
+{
+	bus_unregister(&cxl_bus_type);
+}
+
+module_init(cxl_bus_init);
+module_exit(cxl_bus_exit);
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/cxl/bus.h b/drivers/cxl/bus.h
new file mode 100644
index 000000000000..5fc75ebd3650
--- /dev/null
+++ b/drivers/cxl/bus.h
@@ -0,0 +1,8 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(c) 2020 Intel Corporation. All rights reserved. */
+#ifndef __CXL_BUS_H__
+#define __CXL_BUS_H__
+
+int cxl_register(struct device *dev);
+
+#endif /* __CXL_BUS_H__ */
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index 6d641309013d..77c2dee6843c 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -3,6 +3,7 @@
 
 #ifndef __CXL_H__
 #define __CXL_H__
+#include <linux/range.h>
 
 #include <linux/bitfield.h>
 #include <linux/bitops.h>
@@ -65,6 +66,7 @@
 #define   CXLMDEV_RESET_NEEDED(status) \
 		(CXL_GET_FIELD(status, CXLMDEV_RESET_NEEDED) != CXLMDEV_RESET_NEEDED_NOT)
 
+struct cxl_memdev;
 /**
  * struct cxl_mem - A CXL memory device
  * @pdev: The PCI device associated with this CXL device.
@@ -73,6 +75,7 @@
 struct cxl_mem {
 	struct pci_dev *pdev;
 	void __iomem *regs;
+	struct cxl_memdev *cxlmd;
 
 	spinlock_t mbox_lock; /* Protects device mailbox and firmware */
 
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index 82984bf58a8d..bb6ea58f6c7b 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -2,11 +2,37 @@
 /* Copyright(c) 2020 Intel Corporation. All rights reserved. */
 #include <linux/sched/clock.h>
 #include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/cdev.h>
+#include <linux/idr.h>
 #include <linux/pci.h>
 #include <linux/io.h>
 #include "acpi.h"
 #include "pci.h"
 #include "cxl.h"
+#include "bus.h"
+
+/**
+ * DOC: cxl mem
+ *
+ * This implements a CXL memory device ("type-3") as it is defined by the
+ * Compute Express Link specification.
+ *
+ * The driver has several responsibilities, mainly:
+ *  - Create the memX device and register on the CXL bus.
+ *  - Enumerate device's register interface and map them.
+ *  - Probe the device attributes to establish sysfs interface.
+ *  - Provide an IOCTL interface to userspace to communicate with the device for
+ *    things like firmware update.
+ *  - Support management of interleave sets.
+ *  - Handle and manage error conditions.
+ */
+
+/*
+ * An entire PCI topology full of devices should be enough for any
+ * config
+ */
+#define CXL_MEM_MAX_DEVS 65536
 
 #define cxl_doorbell_busy(cxlm)                                                \
 	(cxl_read_mbox_reg32(cxlm, CXLDEV_MB_CTRL_OFFSET) &                    \
@@ -35,6 +61,18 @@ struct mbox_cmd {
 	u16 return_code;
 };
 
+struct cxl_memdev {
+	struct device dev;
+	struct cxl_mem *cxlm;
+	int id;
+};
+
+static int cxl_mem_major;
+static struct cdev cxl_mem_cdev;
+static DEFINE_IDR(cxl_mem_idr);
+/* protect cxl_mem_idr allocations */
+static DEFINE_MUTEX(cxl_memdev_lock);
+
 static int cxl_mem_wait_for_doorbell(struct cxl_mem *cxlm)
 {
 	const int timeout = msecs_to_jiffies(2000);
@@ -213,6 +251,36 @@ static void cxl_mem_mbox_put(struct cxl_mem *cxlm)
 	spin_unlock(&cxlm->mbox_lock);
 }
 
+static int cxl_mem_open(struct inode *inode, struct file *file)
+{
+	long minor = iminor(inode);
+	struct cxl_memdev *cxlmd;
+
+	rcu_read_lock();
+	cxlmd = idr_find(&cxl_mem_idr, minor);
+	rcu_read_unlock();
+
+	if (!cxlmd)
+		return -ENXIO;
+
+	file->private_data = cxlmd;
+
+	return 0;
+}
+
+static long cxl_mem_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
+{
+	return -ENOTTY;
+}
+
+static const struct file_operations cxl_mem_fops = {
+	.owner = THIS_MODULE,
+	.open = cxl_mem_open,
+	.unlocked_ioctl = cxl_mem_ioctl,
+	.compat_ioctl = compat_ptr_ioctl,
+	.llseek = noop_llseek,
+};
+
 /**
  * cxl_mem_setup_regs() - Setup necessary MMIO.
  * @cxlm: The CXL memory device to communicate with.
@@ -359,6 +427,141 @@ static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec)
 	return 0;
 }
 
+static struct cxl_memdev *to_cxl_memdev(struct device *dev)
+{
+	return container_of(dev, struct cxl_memdev, dev);
+}
+
+static void cxl_memdev_release(struct device *dev)
+{
+	struct cxl_memdev *cxlmd = to_cxl_memdev(dev);
+
+	mutex_lock(&cxl_memdev_lock);
+	idr_remove(&cxl_mem_idr, cxlmd->id);
+	mutex_unlock(&cxl_memdev_lock);
+
+	kfree(cxlmd);
+}
+
+static char *cxl_memdev_devnode(struct device *dev, umode_t *mode, kuid_t *uid,
+				kgid_t *gid)
+{
+	return kasprintf(GFP_KERNEL, "cxl/%s", dev_name(dev));
+}
+
+static ssize_t firmware_version_show(struct device *dev,
+				     struct device_attribute *attr, char *buf)
+{
+	struct cxl_memdev *cxlmd = to_cxl_memdev(dev);
+	struct cxl_mem *cxlm = cxlmd->cxlm;
+
+	return sprintf(buf, "%.16s\n", cxlm->firmware_version);
+}
+static DEVICE_ATTR_RO(firmware_version);
+
+static ssize_t ram_size_show(struct device *dev, struct device_attribute *attr,
+			     char *buf)
+{
+	struct cxl_memdev *cxlmd = to_cxl_memdev(dev);
+	struct cxl_mem *cxlm = cxlmd->cxlm;
+	unsigned long long len = range_len(&cxlm->ram.range);
+
+	return sprintf(buf, "%#llx\n", len);
+}
+
+static struct device_attribute dev_attr_ram_size =
+	__ATTR(size, 0444, ram_size_show, NULL);
+
+static ssize_t pmem_size_show(struct device *dev, struct device_attribute *attr,
+			      char *buf)
+{
+	struct cxl_memdev *cxlmd = to_cxl_memdev(dev);
+	struct cxl_mem *cxlm = cxlmd->cxlm;
+	unsigned long long len = range_len(&cxlm->pmem.range);
+
+	return sprintf(buf, "%#llx\n", len);
+}
+
+static struct device_attribute dev_attr_pmem_size =
+	__ATTR(size, 0444, pmem_size_show, NULL);
+
+static struct attribute *cxl_memdev_attributes[] = {
+	&dev_attr_firmware_version.attr,
+	NULL,
+};
+
+static struct attribute *cxl_memdev_pmem_attributes[] = {
+	&dev_attr_pmem_size.attr,
+	NULL,
+};
+
+static struct attribute *cxl_memdev_ram_attributes[] = {
+	&dev_attr_ram_size.attr,
+	NULL,
+};
+
+static struct attribute_group cxl_memdev_attribute_group = {
+	.attrs = cxl_memdev_attributes,
+};
+
+static struct attribute_group cxl_memdev_ram_attribute_group = {
+	.name = "ram",
+	.attrs = cxl_memdev_ram_attributes,
+};
+
+static struct attribute_group cxl_memdev_pmem_attribute_group = {
+	.name = "pmem",
+	.attrs = cxl_memdev_pmem_attributes,
+};
+
+static const struct attribute_group *cxl_memdev_attribute_groups[] = {
+	&cxl_memdev_attribute_group,
+	&cxl_memdev_ram_attribute_group,
+	&cxl_memdev_pmem_attribute_group,
+	NULL,
+};
+
+static const struct device_type cxl_memdev_type = {
+	.name = "cxl_memdev",
+	.release = cxl_memdev_release,
+	.devnode = cxl_memdev_devnode,
+	.groups = cxl_memdev_attribute_groups,
+};
+
+static int cxl_mem_add_memdev(struct cxl_mem *cxlm)
+{
+	struct pci_dev *pdev = cxlm->pdev;
+	struct cxl_memdev *cxlmd;
+	struct device *dev;
+	int rc;
+
+	cxlmd = kzalloc(sizeof(*cxlmd), GFP_KERNEL);
+	if (!cxlmd)
+		return -ENOMEM;
+
+	cxlmd->cxlm = cxlm;
+
+	mutex_lock(&cxl_memdev_lock);
+	rc = idr_alloc(&cxl_mem_idr, cxlmd, 0, CXL_MEM_MAX_DEVS, GFP_KERNEL);
+	mutex_unlock(&cxl_memdev_lock);
+	if (rc < 0) {
+		kfree(cxlmd);
+		return rc;
+	}
+
+	cxlmd->id = rc;
+
+	dev = &cxlmd->dev;
+
+	device_initialize(dev);
+	dev->parent = &pdev->dev;
+	dev->devt = MKDEV(cxl_mem_major, cxlmd->id);
+	dev->type = &cxl_memdev_type;
+	dev_set_name(dev, "mem%d", cxlmd->id);
+
+	return cxl_register(dev);
+}
+
 /**
  * cxl_mem_identify() - Send the IDENTIFY command to the device.
  * @cxlm: The device to identify.
@@ -461,7 +664,11 @@ static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	if (rc)
 		return rc;
 
-	return cxl_mem_identify(cxlm);
+	rc = cxl_mem_identify(cxlm);
+	if (rc)
+		return rc;
+
+	return cxl_mem_add_memdev(cxlm);
 }
 
 static const struct pci_device_id cxl_mem_pci_tbl[] = {
@@ -478,6 +685,44 @@ static struct pci_driver cxl_mem_driver = {
 	.probe			= cxl_mem_probe,
 };
 
+static __init int cxl_mem_init(void)
+{
+	int rc;
+	dev_t devt;
+
+	rc = alloc_chrdev_region(&devt, 0, CXL_MEM_MAX_DEVS, "cxl");
+	if (rc)
+		return rc;
+
+	cxl_mem_major = MAJOR(devt);
+
+	cdev_init(&cxl_mem_cdev, &cxl_mem_fops);
+	rc = cdev_add(&cxl_mem_cdev, MKDEV(cxl_mem_major, 0), CXL_MEM_MAX_DEVS);
+	if (rc)
+		goto err_cdev;
+
+	rc = pci_register_driver(&cxl_mem_driver);
+	if (rc)
+		goto err_driver;
+
+	return 0;
+
+err_driver:
+	cdev_del(&cxl_mem_cdev);
+err_cdev:
+	unregister_chrdev_region(MKDEV(cxl_mem_major, 0), CXL_MEM_MAX_DEVS);
+
+	return rc;
+}
+
+static __exit void cxl_mem_exit(void)
+{
+	pci_unregister_driver(&cxl_mem_driver);
+	cdev_del(&cxl_mem_cdev);
+	unregister_chrdev_region(MKDEV(cxl_mem_major, 0), CXL_MEM_MAX_DEVS);
+}
+
 MODULE_LICENSE("GPL v2");
-module_pci_driver(cxl_mem_driver);
+module_init(cxl_mem_init);
+module_exit(cxl_mem_exit);
 MODULE_IMPORT_NS(CXL);
-- 
2.29.2


^ permalink raw reply related	[flat|nested] 35+ messages in thread

* [RFC PATCH 09/14] cxl/mem: Add basic IOCTL interface
  2020-12-09  0:24 [RFC PATCH v2 00/14] CXL 2.0 Support Ben Widawsky
                   ` (7 preceding siblings ...)
  2020-12-09  0:24 ` [RFC PATCH v2 08/14] cxl/mem: Register CXL memX devices Ben Widawsky
@ 2020-12-09  0:24 ` Ben Widawsky
  2020-12-09  1:37   ` Dan Williams
  2020-12-10  3:32   ` Randy Dunlap
  2020-12-09  0:24 ` [RFC PATCH 10/14] cxl/mem: Add send command Ben Widawsky
                   ` (6 subsequent siblings)
  15 siblings, 2 replies; 35+ messages in thread
From: Ben Widawsky @ 2020-12-09  0:24 UTC (permalink / raw)
  To: linux-cxl
  Cc: Ben Widawsky, linux-kernel, linux-pci, linux-acpi, Ira Weiny,
	Dan Williams, Vishal Verma, Kelley, Sean V, Rafael Wysocki,
	Bjorn Helgaas, Jonathan Cameron, Jon Masters, Chris Browy,
	Randy Dunlap, Christoph Hellwig

Add a straightforward IOCTL that provides a mechanism for userspace to
query the supported memory device commands.

Memory device commands are specified in 8.2.9 of the CXL 2.0
specification. They are submitted through a mailbox mechanism specified
in 8.2.8.4.

Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>

---

I did attempt to use the same struct for querying commands as well as
sending commands (upcoming patch). The number of unused fields between
the two made for a bad fit IMO.

Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
 Documentation/cxl/memory-devices.rst |   9 +++
 drivers/cxl/mem.c                    |  89 +++++++++++++++++++++++
 include/uapi/linux/cxl_mem.h         | 102 +++++++++++++++++++++++++++
 3 files changed, 200 insertions(+)
 create mode 100644 include/uapi/linux/cxl_mem.h

diff --git a/Documentation/cxl/memory-devices.rst b/Documentation/cxl/memory-devices.rst
index 5f723c25382b..ec54674b3822 100644
--- a/Documentation/cxl/memory-devices.rst
+++ b/Documentation/cxl/memory-devices.rst
@@ -32,6 +32,15 @@ CXL Memory Device
 .. kernel-doc:: drivers/cxl/mem.c
    :internal:
 
+CXL IOCTL Interface
+-------------------
+
+.. kernel-doc:: include/uapi/linux/cxl_mem.h
+   :doc: UAPI
+
+.. kernel-doc:: include/uapi/linux/cxl_mem.h
+   :internal:
+
 External Interfaces
 ===================
 
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index bb6ea58f6c7b..2c4aadcea0e4 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -7,6 +7,7 @@
 #include <linux/idr.h>
 #include <linux/pci.h>
 #include <linux/io.h>
+#include <uapi/linux/cxl_mem.h>
 #include "acpi.h"
 #include "pci.h"
 #include "cxl.h"
@@ -73,6 +74,49 @@ static DEFINE_IDR(cxl_mem_idr);
 /* protect cxl_mem_idr allocations */
 static DEFINE_MUTEX(cxl_memdev_lock);
 
+/*
+ * This table defines the supported mailboxes commands for the driver. The id is
+ * ordinal and thus gaps in this table aren't allowed. This table is made up of
+ * a UAPI structure. Non-negative values in the table will be validated against
+ * the user's input. For example, if size_in is 0, and the user passed in 1, it
+ * is an error.
+ */
+#define CXL_CMD(_id, _flags, sin, sout, _name, _enable, op)                    \
+	{                                                                      \
+		{ .id = CXL_MEM_COMMAND_ID_##_id,                              \
+		  .flags = CXL_MEM_COMMAND_FLAG_##_flags,                      \
+		  .size_in = sin,                                              \
+		  .size_out = sout,                                            \
+		  .name = _name },                                             \
+			.enable = _enable, .opcode = op                        \
+	}
+
+/**
+ * struct cxl_mem_command - Driver representation of a memory device command
+ * @info: Command information as it exists for the UAPI
+ * @opcode: The actual bits used for the mailbox protocol
+ * @enable: Whether the command is enabled. The driver may support a large set
+ *	    of commands that may not be enabled. The primary reason a command
+ *	    would not be enabled is for commands that are specified as optional
+ *	    and the hardware doesn't support the command.
+ *
+ * The cxl_mem_command is the driver's internal representation of commands that
+ * are supported by the driver. Some of these commands may not be supported by
+ * the hardware (!@enable). The driver will use @info to validate the fields
+ * passed in by the user then submit the @opcode to the hardware.
+ *
+ * See struct cxl_command_info.
+ */
+struct cxl_mem_command {
+	const struct cxl_command_info info;
+	const u16 opcode;
+	bool enable;
+};
+
+static struct cxl_mem_command mem_commands[] = {
+	CXL_CMD(INVALID, NONE, 0, 0, "Reserved", false, 0),
+};
+
 static int cxl_mem_wait_for_doorbell(struct cxl_mem *cxlm)
 {
 	const int timeout = msecs_to_jiffies(2000);
@@ -268,8 +312,53 @@ static int cxl_mem_open(struct inode *inode, struct file *file)
 	return 0;
 }
 
+static int cxl_mem_count_commands(void)
+{
+	int i, n = 0;
+
+	for (i = 0; i < ARRAY_SIZE(mem_commands); i++) {
+		struct cxl_mem_command *c = &mem_commands[i];
+
+		if (c->enable)
+			n++;
+	}
+
+	return n;
+}
+
 static long cxl_mem_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
 {
+	if (cmd == CXL_MEM_QUERY_COMMANDS) {
+		struct cxl_mem_query_commands __user *q = (void __user *)arg;
+		u32 n_commands;
+		int i, j;
+
+		if (get_user(n_commands, (u32 __user *)arg))
+			return -EFAULT;
+
+		if (n_commands == 0)
+			return put_user(cxl_mem_count_commands(),
+					(u32 __user *)arg);
+
+		for (i = 0, j = 0;
+		     i < ARRAY_SIZE(mem_commands) && j < n_commands; i++) {
+			struct cxl_mem_command *c = &mem_commands[i];
+			const struct cxl_command_info *info = &c->info;
+
+			if (!c->enable)
+				continue;
+
+			if (copy_to_user(&q->commands[j], info, sizeof(*info)))
+				return -EFAULT;
+
+			if (copy_to_user(&q->commands[j].name, info->name,
+					 strlen(info->name)))
+				return -EFAULT;
+
+			j++;
+		}
+	}
+
 	return -ENOTTY;
 }
 
diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
new file mode 100644
index 000000000000..1d1e143f98ec
--- /dev/null
+++ b/include/uapi/linux/cxl_mem.h
@@ -0,0 +1,102 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * CXL IOCTLs for Memory Devices
+ */
+
+#ifndef _UAPI_CXL_MEM_H_
+#define _UAPI_CXL_MEM_H_
+
+#if defined(__cplusplus)
+extern "C" {
+#endif
+
+/**
+ * DOC: UAPI
+ *
+ * CXL memory devices expose UAPI to have a standard user interface.
+ * Userspace can refer to these structure definitions and UAPI formats
+ * to communicate to driver
+ */
+
+#define CXL_MEM_QUERY_COMMANDS _IOR('C', 1, struct cxl_mem_query_commands)
+
+#define CXL_MEM_COMMAND_NAME_LENGTH 32
+
+/**
+ * struct cxl_command_info - Command information returned from a query.
+ * @id: ID number for the command.
+ * @flags: Flags that specify command behavior.
+ *
+ *          - CXL_MEM_COMMAND_FLAG_TAINT: Using this command will taint the kernel.
+ * @size_in: Expected input size, or -1 if variable length.
+ * @size_out: Expected output size, or -1 if variable length.
+ * @name: Name describing the command.
+ *
+ * Represents a single command that is supported by both the driver and the
+ * hardware. The is returned as part of an array from the query ioctl. The
+ * following would be a command named "foobar" that takes a variable length
+ * input and returns 0 bytes of output.
+ *
+ *  - @id = 10
+ *  - @name = foobar
+ *  - @flags = 0
+ *  - @size_in = -1
+ *  - @size_out = 0
+ *
+ * See struct cxl_mem_query_commands.
+ */
+struct cxl_command_info {
+	__u32 id;
+#define CXL_MEM_COMMAND_ID_INVALID 0
+
+	__u32 flags;
+#define CXL_MEM_COMMAND_FLAG_NONE 0
+#define CXL_MEM_COMMAND_FLAG_TAINT BIT(0)
+
+	__s32 size_in;
+	__s32 size_out;
+
+	char name[32];
+};
+
+/**
+ * struct cxl_mem_query_commands - Query supported commands.
+ * @n_commands: In/out parameter. When @n_commands is > 0, the driver will
+ *		return min(num_support_commands, n_commands). When @n_commands
+ *		is 0, driver will return the number of total supported commands.
+ * @rsvd: Reserved for future use.
+ * @commands: Output array of supported commands. This array must be allocated
+ *            by userspace to be at least min(num_support_commands, @n_commands)
+ *
+ * Allow userspace to query the available commands supported by both the driver,
+ * and the hardware. Commands that aren't supported by either the driver, or the
+ * hardware are not returned in the query.
+ *
+ * Examples:
+ *
+ *  - { .n_commands = 0 } // Get number of supported commands
+ *  - { .n_commands = 15, .commands = buf } // Return first 15 (or less)
+ *    supported commands
+ *
+ *  See struct cxl_command_info.
+ */
+struct cxl_mem_query_commands {
+	/*
+	 * Input: Number of commands to return (space allocated by user)
+	 * Output: Number of commands supported by the driver/hardware
+	 *
+	 * If n_commands is 0, kernel will only return number of commands and
+	 * not try to populate commands[], thus allowing userspace to know how
+	 * much space to allocate
+	 */
+	__u32 n_commands;
+	__u32 rsvd;
+
+	struct cxl_command_info __user commands[]; /* out: supported commands */
+};
+
+#if defined(__cplusplus)
+}
+#endif
+
+#endif
-- 
2.29.2


^ permalink raw reply related	[flat|nested] 35+ messages in thread

* [RFC PATCH 10/14] cxl/mem: Add send command
  2020-12-09  0:24 [RFC PATCH v2 00/14] CXL 2.0 Support Ben Widawsky
                   ` (8 preceding siblings ...)
  2020-12-09  0:24 ` [RFC PATCH 09/14] cxl/mem: Add basic IOCTL interface Ben Widawsky
@ 2020-12-09  0:24 ` Ben Widawsky
  2020-12-09 22:06   ` Dan Williams
  2020-12-09  0:24 ` [RFC PATCH 11/14] cxl/mem: Add a "RAW" " Ben Widawsky
                   ` (5 subsequent siblings)
  15 siblings, 1 reply; 35+ messages in thread
From: Ben Widawsky @ 2020-12-09  0:24 UTC (permalink / raw)
  To: linux-cxl
  Cc: Ben Widawsky, linux-kernel, linux-pci, linux-acpi, Ira Weiny,
	Dan Williams, Vishal Verma, Kelley, Sean V, Rafael Wysocki,
	Bjorn Helgaas, Jonathan Cameron, Jon Masters, Chris Browy,
	Randy Dunlap, Christoph Hellwig

The send command allows userspace to issue mailbox commands directly to
the hardware. The driver will verify basic properties of the command but
otherwise pass any input payload untouched to the hardware, and return
the output payload to userspace.

The caller of this IOCTL is required to allocate enough space for
max(size_in, size_out) of the payload. The payload input data will be
wiped out if any output payload exists.

Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
 drivers/cxl/mem.c            | 127 +++++++++++++++++++++++++++++++++++
 include/uapi/linux/cxl_mem.h |  35 ++++++++++
 2 files changed, 162 insertions(+)

diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index 2c4aadcea0e4..0bf03afc0c80 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -324,6 +324,120 @@ static int cxl_mem_count_commands(void)
 	}
 
 	return n;
+};
+
+/**
+ * handle_mailbox_cmd_from_user() - Dispatch a mailbox command.
+ * @cxlmd: The CXL memory device to communicate with.
+ * @cmd: The validated command
+ * @u: The command submitted by userspace. Only useful for RAW commands.
+ *
+ * Return: 0 on success.
+ *
+ * This function packages up a &struct mbox_cmd on behalf of userspace,
+ * dispatches the command, and returns the results.
+ */
+static int handle_mailbox_cmd_from_user(struct cxl_memdev *cxlmd,
+					const struct cxl_mem_command *cmd,
+					struct cxl_send_command __user *u)
+{
+	struct mbox_cmd mbox_cmd;
+	ssize_t payload_size;
+	void *payload;
+	u32 size_out;
+	int rc;
+
+	if (get_user(size_out, &u->size_out))
+		return -EFAULT;
+
+	payload_size = max_t(ssize_t, cmd->info.size_in, size_out);
+	if (payload_size) {
+		payload =
+			memdup_user(u64_to_user_ptr(u->payload), payload_size);
+		if (IS_ERR(payload))
+			return PTR_ERR(payload);
+	}
+
+	rc = cxl_mem_mbox_get(cxlmd->cxlm);
+	if (rc)
+		return rc;
+
+	mbox_cmd = (struct mbox_cmd){
+		.opcode = cmd->opcode,
+		.payload = payload,
+		.size_in = cmd->info.size_in,
+		.size_out = size_out,
+	};
+	rc = cxl_mem_mbox_send_cmd(cxlmd->cxlm, &mbox_cmd);
+	cxl_mem_mbox_put(cxlmd->cxlm);
+	if (rc)
+		goto out;
+
+	rc = put_user(mbox_cmd.return_code, &u->retval);
+	if (rc)
+		goto out;
+
+	rc = put_user(mbox_cmd.size_out, &u->size_out);
+	if (rc)
+		goto out;
+
+	if (mbox_cmd.size_out)
+		if (copy_to_user(u64_to_user_ptr(u->payload), payload,
+				 mbox_cmd.size_out))
+			rc = -EFAULT;
+
+out:
+	if (payload_size)
+		kfree(payload);
+	return rc;
+}
+
+/**
+ * cxl_validate_cmd_from_user() - Check fields for CXL_MEM_SEND_COMMAND.
+ * @user_cmd: &struct cxl_send_command from userspace.
+ * @out_cmd: Sanitized and populared &struct cxl_mem_command.
+ *
+ * Return:
+ *  * %0	- Command dispatched successfully.
+ *  * %-EFAULT	- Something happened with copy_to/from_user.
+ *  * %-EINVAL	- Rerserved fields were used.
+ *  * %-EPERM	- Protected command used by the RAW interface.
+ *  * %-ENOMEM	- Input or output buffer wasn't large enough.
+ *
+ */
+static int cxl_validate_cmd_from_user(struct cxl_send_command __user *user_cmd,
+				      struct cxl_mem_command *out_cmd)
+{
+	const struct cxl_command_info *info;
+	struct cxl_send_command cmd;
+	struct cxl_mem_command *c;
+
+	if (copy_from_user(&cmd, user_cmd, sizeof(cmd)))
+		return -EFAULT;
+
+	if (cmd.id == 0 || cmd.id >= CXL_MEM_COMMAND_ID_MAX)
+		return -EINVAL;
+
+	c = &mem_commands[cmd.id];
+	info = &c->info;
+
+	if (cmd.flags & CXL_MEM_COMMAND_FLAG_MASK)
+		return -EINVAL;
+
+	if (cmd.rsvd)
+		return -EINVAL;
+
+	/* Check the input buffer is the expected size */
+	if (info->size_in >= 0 && info->size_in != cmd.size_in)
+		return -ENOMEM;
+
+	/* Check the output buffer is at least large enough */
+	if (info->size_out >= 0 && cmd.size_out < info->size_out)
+		return -ENOMEM;
+
+	memcpy(out_cmd, c, sizeof(*c));
+
+	return 0;
 }
 
 static long cxl_mem_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
@@ -357,6 +471,19 @@ static long cxl_mem_ioctl(struct file *file, unsigned int cmd, unsigned long arg
 
 			j++;
 		}
+
+		return 0;
+	} else if (cmd == CXL_MEM_SEND_COMMAND) {
+		struct cxl_send_command __user *u = (void __user *)arg;
+		struct cxl_memdev *cxlmd = file->private_data;
+		struct cxl_mem_command c;
+		int rc;
+
+		rc = cxl_validate_cmd_from_user(u, &c);
+		if (rc)
+			return rc;
+
+		return handle_mailbox_cmd_from_user(cxlmd, &c, u);
 	}
 
 	return -ENOTTY;
diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
index 1d1e143f98ec..189d86a13637 100644
--- a/include/uapi/linux/cxl_mem.h
+++ b/include/uapi/linux/cxl_mem.h
@@ -19,6 +19,7 @@ extern "C" {
  */
 
 #define CXL_MEM_QUERY_COMMANDS _IOR('C', 1, struct cxl_mem_query_commands)
+#define CXL_MEM_SEND_COMMAND _IOWR('C', 2, struct cxl_send_command)
 
 #define CXL_MEM_COMMAND_NAME_LENGTH 32
 
@@ -48,10 +49,12 @@ extern "C" {
 struct cxl_command_info {
 	__u32 id;
 #define CXL_MEM_COMMAND_ID_INVALID 0
+#define CXL_MEM_COMMAND_ID_MAX (CXL_MEM_COMMAND_ID_INVALID + 1)
 
 	__u32 flags;
 #define CXL_MEM_COMMAND_FLAG_NONE 0
 #define CXL_MEM_COMMAND_FLAG_TAINT BIT(0)
+#define CXL_MEM_COMMAND_FLAG_MASK ~BIT(0)
 
 	__s32 size_in;
 	__s32 size_out;
@@ -95,6 +98,38 @@ struct cxl_mem_query_commands {
 	struct cxl_command_info __user commands[]; /* out: supported commands */
 };
 
+/**
+ * struct cxl_send_command - Send a command to a memory device.
+ * @id: The command to send to the memory device. This must be one of the
+ *	commands returned by the query command.
+ * @flags: Flags for the command
+ * @rsvd: Reserved for future use.
+ * @retval: Return value from the memory device (output).
+ * @size_in: Size of the payload to provide to the device (input).
+ * @size_out: Size of the payload received from the device (input/output). This
+ *	      field is filled in my userspace to let the driver know how much
+ *	      space was allocated for output. It is populated by the driver to
+ *	      let userspace know how large the output payload actually was.
+ * @payload: Pointer to memory available for payload input/output.
+ *
+ * Mechanism for userspace to send a command to the hardware for processing. The
+ * driver will do basic validation on the command sizes, but the payload input
+ * and output are not introspected. Userspace is required to allocate large
+ * enough buffers for max(size_in, size_out).
+ */
+struct cxl_send_command {
+	__u32 id;
+	__u32 flags;
+	__u32 rsvd;
+	__u32 retval;
+
+	struct {
+		__s32 size_in;
+		__s32 size_out;
+		__u64 payload;
+	};
+};
+
 #if defined(__cplusplus)
 }
 #endif
-- 
2.29.2


^ permalink raw reply related	[flat|nested] 35+ messages in thread

* [RFC PATCH 11/14] cxl/mem: Add a "RAW" send command
  2020-12-09  0:24 [RFC PATCH v2 00/14] CXL 2.0 Support Ben Widawsky
                   ` (9 preceding siblings ...)
  2020-12-09  0:24 ` [RFC PATCH 10/14] cxl/mem: Add send command Ben Widawsky
@ 2020-12-09  0:24 ` Ben Widawsky
  2020-12-09 22:38   ` Dan Williams
  2020-12-09  0:24 ` [RFC PATCH 12/14] cxl: Add basic debugging Ben Widawsky
                   ` (4 subsequent siblings)
  15 siblings, 1 reply; 35+ messages in thread
From: Ben Widawsky @ 2020-12-09  0:24 UTC (permalink / raw)
  To: linux-cxl
  Cc: Ben Widawsky, linux-kernel, linux-pci, linux-acpi, Ira Weiny,
	Dan Williams, Vishal Verma, Kelley, Sean V, Rafael Wysocki,
	Bjorn Helgaas, Jonathan Cameron, Jon Masters, Chris Browy,
	Randy Dunlap, Christoph Hellwig

The CXL memory device send interface will have a number of supported
commands. The raw command is not such a command. Raw commands allow
userspace to send a specified opcode to the underlying hardware and
bypass all driver checks on the command. This is useful for a couple of
usecases, mainly:
1. Undocumented vendor specific hardware commands
2. Prototyping new hardware commands not yet supported by the driver

While this all sounds very powerful it comes with a couple of caveats:
1. Bug reports using raw commands will not get the same level of
   attention as bug reports using supported commands (via taint).
2. Supported commands will be rejected by the RAW command.

Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
 drivers/cxl/mem.c            | 32 ++++++++++++++++++++++++++++++++
 include/uapi/linux/cxl_mem.h | 14 ++++++++++++--
 2 files changed, 44 insertions(+), 2 deletions(-)

diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index 0bf03afc0c80..a2cea7ac7cc6 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -115,6 +115,7 @@ struct cxl_mem_command {
 
 static struct cxl_mem_command mem_commands[] = {
 	CXL_CMD(INVALID, NONE, 0, 0, "Reserved", false, 0),
+	CXL_CMD(RAW, TAINT, ~0, ~0, "Raw", true, 0),
 };
 
 static int cxl_mem_wait_for_doorbell(struct cxl_mem *cxlm)
@@ -326,6 +327,20 @@ static int cxl_mem_count_commands(void)
 	return n;
 };
 
+static struct cxl_mem_command *cxl_mem_find_command(u16 opcode)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(mem_commands); i++) {
+		struct cxl_mem_command *c = &mem_commands[i];
+
+		if (c->opcode == opcode)
+			return c;
+	}
+
+	return NULL;
+};
+
 /**
  * handle_mailbox_cmd_from_user() - Dispatch a mailbox command.
  * @cxlmd: The CXL memory device to communicate with.
@@ -421,6 +436,23 @@ static int cxl_validate_cmd_from_user(struct cxl_send_command __user *user_cmd,
 	c = &mem_commands[cmd.id];
 	info = &c->info;
 
+	/* Checks are bypassed for raw commands but along comes the taint! */
+	if (cmd.id == CXL_MEM_COMMAND_ID_RAW) {
+		struct cxl_mem_command temp =
+			CXL_CMD(RAW, NONE, cmd.size_in, cmd.size_out, "Raw",
+				true, cmd.raw.opcode);
+
+		if (cmd.raw.rsvd)
+			return -EINVAL;
+
+		if (cxl_mem_find_command(cmd.raw.opcode))
+			return -EPERM;
+
+		add_taint(TAINT_WARN, LOCKDEP_STILL_OK);
+		memcpy(out_cmd, &temp, sizeof(temp));
+		return 0;
+	}
+
 	if (cmd.flags & CXL_MEM_COMMAND_FLAG_MASK)
 		return -EINVAL;
 
diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
index 189d86a13637..f2fbb0dcda06 100644
--- a/include/uapi/linux/cxl_mem.h
+++ b/include/uapi/linux/cxl_mem.h
@@ -49,7 +49,8 @@ extern "C" {
 struct cxl_command_info {
 	__u32 id;
 #define CXL_MEM_COMMAND_ID_INVALID 0
-#define CXL_MEM_COMMAND_ID_MAX (CXL_MEM_COMMAND_ID_INVALID + 1)
+#define CXL_MEM_COMMAND_ID_RAW 1
+#define CXL_MEM_COMMAND_ID_MAX (CXL_MEM_COMMAND_ID_RAW + 1)
 
 	__u32 flags;
 #define CXL_MEM_COMMAND_FLAG_NONE 0
@@ -103,6 +104,9 @@ struct cxl_mem_query_commands {
  * @id: The command to send to the memory device. This must be one of the
  *	commands returned by the query command.
  * @flags: Flags for the command
+ * @raw: Special fields for raw commands
+ * @raw.opcode: Opcode passed to hardware when using the RAW command.
+ * @raw.rsvd: Reserved for future use.
  * @rsvd: Reserved for future use.
  * @retval: Return value from the memory device (output).
  * @size_in: Size of the payload to provide to the device (input).
@@ -120,7 +124,13 @@ struct cxl_mem_query_commands {
 struct cxl_send_command {
 	__u32 id;
 	__u32 flags;
-	__u32 rsvd;
+	union {
+		struct {
+			__u16 opcode;
+			__u16 rsvd;
+		} raw;
+		__u32 rsvd;
+	};
 	__u32 retval;
 
 	struct {
-- 
2.29.2


^ permalink raw reply related	[flat|nested] 35+ messages in thread

* [RFC PATCH 12/14] cxl: Add basic debugging
  2020-12-09  0:24 [RFC PATCH v2 00/14] CXL 2.0 Support Ben Widawsky
                   ` (10 preceding siblings ...)
  2020-12-09  0:24 ` [RFC PATCH 11/14] cxl/mem: Add a "RAW" " Ben Widawsky
@ 2020-12-09  0:24 ` Ben Widawsky
  2020-12-09  1:17   ` Dan Williams
  2020-12-09  0:24 ` [RFC PATCH 13/14] MAINTAINERS: Add maintainers of the CXL driver Ben Widawsky
                   ` (3 subsequent siblings)
  15 siblings, 1 reply; 35+ messages in thread
From: Ben Widawsky @ 2020-12-09  0:24 UTC (permalink / raw)
  To: linux-cxl
  Cc: Ben Widawsky, linux-kernel, linux-pci, linux-acpi, Ira Weiny,
	Dan Williams, Vishal Verma, Kelley, Sean V, Rafael Wysocki,
	Bjorn Helgaas, Jonathan Cameron, Jon Masters, Chris Browy,
	Randy Dunlap, Christoph Hellwig

Provide a standard debug function for use throughout the driver.

Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
 drivers/cxl/cxl.h |  3 +++
 drivers/cxl/mem.c | 26 +++++++++++++++++++++++++-
 2 files changed, 28 insertions(+), 1 deletion(-)

diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index 77c2dee6843c..e5afb89dab0b 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -9,6 +9,9 @@
 #include <linux/bitops.h>
 #include <linux/io.h>
 
+#define cxl_debug(fmt, ...)                                                    \
+	pr_debug("CXL DEBUG: %s: " fmt, __func__, ##__VA_ARGS__)
+
 #define CXL_SET_FIELD(value, field)                                            \
 	({                                                                     \
 		WARN_ON(!FIELD_FIT(field##_MASK, value));                      \
diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index a2cea7ac7cc6..6b2f8d3776b5 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -122,9 +122,12 @@ static int cxl_mem_wait_for_doorbell(struct cxl_mem *cxlm)
 {
 	const int timeout = msecs_to_jiffies(2000);
 	const unsigned long start = jiffies;
+	unsigned long end = start;
 
 	while (cxl_doorbell_busy(cxlm)) {
-		if (time_after(jiffies, start + timeout)) {
+		end = jiffies;
+
+		if (time_after(end, start + timeout)) {
 			/* Check again in case preempted before timeout test */
 			if (!cxl_doorbell_busy(cxlm))
 				break;
@@ -133,6 +136,8 @@ static int cxl_mem_wait_for_doorbell(struct cxl_mem *cxlm)
 		cpu_relax();
 	}
 
+	cxl_debug("Doorbell wait took %dms",
+		  jiffies_to_msecs(end) - jiffies_to_msecs(start));
 	return 0;
 }
 
@@ -180,6 +185,8 @@ static int cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm,
 	}
 
 	/* #4 */
+	cxl_debug("Sending command to %s\n",
+		  dev_driver_string(&cxlm->pdev->dev));
 	cxl_write_mbox_reg32(cxlm, CXLDEV_MB_CTRL_OFFSET,
 			     CXLDEV_MB_CTRL_DOORBELL);
 
@@ -308,6 +315,8 @@ static int cxl_mem_open(struct inode *inode, struct file *file)
 	if (!cxlmd)
 		return -ENXIO;
 
+	cxl_debug("Opened %pD\n", file);
+
 	file->private_data = cxlmd;
 
 	return 0;
@@ -383,6 +392,10 @@ static int handle_mailbox_cmd_from_user(struct cxl_memdev *cxlmd,
 		.size_in = cmd->info.size_in,
 		.size_out = size_out,
 	};
+	cxl_debug("Submitting command for user\n"
+		  "\topcode: %x\n"
+		  "\tsize: %zub/%zub\n",
+		  mbox_cmd.opcode, mbox_cmd.size_in, mbox_cmd.size_out);
 	rc = cxl_mem_mbox_send_cmd(cxlmd->cxlm, &mbox_cmd);
 	cxl_mem_mbox_put(cxlmd->cxlm);
 	if (rc)
@@ -479,6 +492,8 @@ static long cxl_mem_ioctl(struct file *file, unsigned int cmd, unsigned long arg
 		u32 n_commands;
 		int i, j;
 
+		cxl_debug("Query IOCTL\n");
+
 		if (get_user(n_commands, (u32 __user *)arg))
 			return -EFAULT;
 
@@ -511,6 +526,8 @@ static long cxl_mem_ioctl(struct file *file, unsigned int cmd, unsigned long arg
 		struct cxl_mem_command c;
 		int rc;
 
+		cxl_debug("Send IOCTL\n");
+
 		rc = cxl_validate_cmd_from_user(u, &c);
 		if (rc)
 			return rc;
@@ -843,6 +860,13 @@ static int cxl_mem_identify(struct cxl_mem *cxlm)
 
 	id = (struct cxl_mbox_identify *)mbox_cmd.payload;
 
+	cxl_debug("Driver identify command\n"
+		  "\tFirmware Version: %s\n"
+		  "\tTotal Capacity: %llu (%llu persistent)\n"
+		  "\tLSA size: %u\n",
+		  id->fw_revision, id->total_capacity, id->persistent_capacity,
+		  id->lsa_size);
+
 	/*
 	 * TODO: enumerate DPA map, as 'ram' and 'pmem' do not alias.
 	 * For now, only the capacity is exported in sysfs
-- 
2.29.2


^ permalink raw reply related	[flat|nested] 35+ messages in thread

* [RFC PATCH 13/14] MAINTAINERS: Add maintainers of the CXL driver
  2020-12-09  0:24 [RFC PATCH v2 00/14] CXL 2.0 Support Ben Widawsky
                   ` (11 preceding siblings ...)
  2020-12-09  0:24 ` [RFC PATCH 12/14] cxl: Add basic debugging Ben Widawsky
@ 2020-12-09  0:24 ` Ben Widawsky
  2020-12-09  0:24 ` [RFC PATCH 14/14] WIP/cxl/mem: Add get firmware for testing Ben Widawsky
                   ` (2 subsequent siblings)
  15 siblings, 0 replies; 35+ messages in thread
From: Ben Widawsky @ 2020-12-09  0:24 UTC (permalink / raw)
  To: linux-cxl
  Cc: Ben Widawsky, Dan Williams, Vishal Verma, Ira Weiny,
	linux-kernel, linux-pci, linux-acpi, Kelley, Sean V,
	Rafael Wysocki, Bjorn Helgaas, Jonathan Cameron, Jon Masters,
	Chris Browy, Randy Dunlap, Christoph Hellwig

Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Vishal Verma <vishal.l.verma@intel.com>
Cc: Ira Weiny <ira.weiny@intel.com>
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
 MAINTAINERS | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index b516bb34a8d5..d6aa4b684a1a 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4428,6 +4428,16 @@ F:	fs/configfs/
 F:	include/linux/configfs.h
 F:	samples/configfs/
 
+COMPUTE EXPRESS LINK (CXL)
+M:	Vishal Verma <vishal.l.verma@intel.com>
+M:	Ira Weiny <ira.weiny@intel.com>
+M:	Ben Widawsky <ben.widawsky@intel.com>
+M:	Dan Williams <dan.j.williams@intel.com>
+L:	linux-cxl@vger.kernel.org
+S:	Maintained
+F:	drivers/cxl/
+F:	include/uapi/linux/cxl_mem.h
+
 CONSOLE SUBSYSTEM
 M:	Greg Kroah-Hartman <gregkh@linuxfoundation.org>
 S:	Supported
-- 
2.29.2


^ permalink raw reply related	[flat|nested] 35+ messages in thread

* [RFC PATCH 14/14] WIP/cxl/mem: Add get firmware for testing
  2020-12-09  0:24 [RFC PATCH v2 00/14] CXL 2.0 Support Ben Widawsky
                   ` (12 preceding siblings ...)
  2020-12-09  0:24 ` [RFC PATCH 13/14] MAINTAINERS: Add maintainers of the CXL driver Ben Widawsky
@ 2020-12-09  0:24 ` Ben Widawsky
  2020-12-10  0:42   ` Dan Williams
  2020-12-09  0:33 ` [RFC PATCH v2 00/14] CXL 2.0 Support Verma, Vishal L
  2020-12-09  0:47 ` Ben Widawsky
  15 siblings, 1 reply; 35+ messages in thread
From: Ben Widawsky @ 2020-12-09  0:24 UTC (permalink / raw)
  To: linux-cxl
  Cc: Ben Widawsky, linux-kernel, linux-pci, linux-acpi, Ira Weiny,
	Dan Williams, Vishal Verma, Kelley, Sean V, Rafael Wysocki,
	Bjorn Helgaas, Jonathan Cameron, Jon Masters, Chris Browy,
	Randy Dunlap, Christoph Hellwig

This also serves as an example how to add a new command

Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
 drivers/cxl/mem.c            | 22 ++++++++++++++++++++++
 include/uapi/linux/cxl_mem.h |  3 ++-
 2 files changed, 24 insertions(+), 1 deletion(-)

diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
index 6b2f8d3776b5..76aa1e6e4117 100644
--- a/drivers/cxl/mem.c
+++ b/drivers/cxl/mem.c
@@ -116,6 +116,7 @@ struct cxl_mem_command {
 static struct cxl_mem_command mem_commands[] = {
 	CXL_CMD(INVALID, NONE, 0, 0, "Reserved", false, 0),
 	CXL_CMD(RAW, TAINT, ~0, ~0, "Raw", true, 0),
+	CXL_CMD(GET_FW_INFO, NONE, 0, 0x50, "Get FW Info", false, 0x0200),
 };
 
 static int cxl_mem_wait_for_doorbell(struct cxl_mem *cxlm)
@@ -827,6 +828,23 @@ static int cxl_mem_add_memdev(struct cxl_mem *cxlm)
 	return cxl_register(dev);
 }
 
+static int cxl_mem_enable_commands(struct cxl_mem *cxlm)
+{
+	struct cxl_mem_command *c;
+
+	/*
+	 * For now we pretend Get FW info is supported.
+	 *
+	 * FIXME: Invoke GET LOG to get the Command Effect Logs (CEL).
+	 */
+	c = cxl_mem_find_command(0x200);
+	if (!c)
+		return -ENOENT;
+
+	c->enable = true;
+	return 0;
+}
+
 /**
  * cxl_mem_identify() - Send the IDENTIFY command to the device.
  * @cxlm: The device to identify.
@@ -936,6 +954,10 @@ static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	if (rc)
 		return rc;
 
+	rc = cxl_mem_enable_commands(cxlm);
+	if (rc)
+		return rc;
+
 	rc = cxl_mem_identify(cxlm);
 	if (rc)
 		return rc;
diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
index f2fbb0dcda06..3ac39acf8fa7 100644
--- a/include/uapi/linux/cxl_mem.h
+++ b/include/uapi/linux/cxl_mem.h
@@ -50,7 +50,8 @@ struct cxl_command_info {
 	__u32 id;
 #define CXL_MEM_COMMAND_ID_INVALID 0
 #define CXL_MEM_COMMAND_ID_RAW 1
-#define CXL_MEM_COMMAND_ID_MAX (CXL_MEM_COMMAND_ID_RAW + 1)
+#define CXL_MEM_COMMAND_ID_GET_FW_INFO 2
+#define CXL_MEM_COMMAND_ID_MAX (CXL_MEM_COMMAND_ID_GET_FW_INFO + 1)
 
 	__u32 flags;
 #define CXL_MEM_COMMAND_FLAG_NONE 0
-- 
2.29.2


^ permalink raw reply related	[flat|nested] 35+ messages in thread

* Re: [RFC PATCH v2 00/14] CXL 2.0 Support
  2020-12-09  0:24 [RFC PATCH v2 00/14] CXL 2.0 Support Ben Widawsky
                   ` (13 preceding siblings ...)
  2020-12-09  0:24 ` [RFC PATCH 14/14] WIP/cxl/mem: Add get firmware for testing Ben Widawsky
@ 2020-12-09  0:33 ` Verma, Vishal L
  2020-12-09  0:47 ` Ben Widawsky
  15 siblings, 0 replies; 35+ messages in thread
From: Verma, Vishal L @ 2020-12-09  0:33 UTC (permalink / raw)
  To: Widawsky, Ben, linux-cxl
  Cc: rdunlap, hch, Kelley, Sean V, linux-acpi, jcm, Jonathan.Cameron,
	linux-kernel, Williams, Dan J, Wysocki, Rafael J, linux-pci,
	Weiny, Ira, cbrowy, helgaas

On Tue, 2020-12-08 at 16:24 -0800, Ben Widawsky wrote:
> Changes since v1 [1]
> 
> A few additions have been made:
>  - IOCTL (UAPI) interface has been added with commands
>  - Kernel docs have been created
>  - A new debug macro is introduced and sprinkled throughout.
> 
> A deletion was made:
>  - Removal of the non-standard _OSC UUID.
> 
> The detailed list of fixes is:
>  - fix cxl_register() no previous prototype warning (0day robot)
>  - s/REGLOG/REGLOC/ (Ben)
>  - Wait for doorbell on cxl_mem_mbox_get() and add comment on why (Ben)
>  - make "type-3" a proper adjective, add spec references, also did the same for
>    the Kconfig (Bjorn)
>  - align some defines (Bjorn)
>  - s/bar/BAR (Bjorn)
>  - rename cxl_bus_prepared() to cxl_bus_acquire() (Bjorn)
>  - move definition of struct cxl_mem to "cxl/mem: Map memory device registers" (Bjorn)
>  - use consistent hex/decimal (Bjorn)
>  - use consistent upper/lower hex values (Bjorn)
>  - get rid of READ_ONCE (Bjorn)
>  - add offsets to debug messages (Bjorn)
>  - cleanup SPDX comment style (Bjorn, Christoph)
>  - change errors returned by case (Bjorn, Dan)
>  - 80 character violation cleanups (Christoph)
>  - cleanup CXL_BUS_PROVIDER dependencies (Christoph, Randy)
>  - remove "raw" from mmio functions (Dan)
>  - rename PCI_DVSEC_VENDOR_CXL to add _ID (Jonathan)
>  - combine introduction of mbox infrastruct and cxl_mem_identify() (Jonathan)
>  - add ABI documentation for sysfs attributes (Jonathan)
>  - document scope of cxl_memdev_lock (Jonathan)
>  - rework cxl_register() to have devm semantics (reaction to comments about
>    cxl_mem_remove() and cxl_mem_add_memdev() semantics) (Jonathan)
>  - fix cxl_mem_exit() ordering (Jonathan)
>  - use GENMASK/GET_FIELD (Jonathan)
>  - fix and add comments for cap ids (Jonathan)
>  - use _OFFSET postfix in definitions (Jonathan)
>  - save pci_set_drvdata for later (Jonathan)

There are a few more change credits for the acpi patches:

- Remove unnecessary helpers and callbacks in drivers/cxl/acpi.c (Christoph, Jonathan)
- Remove some unnecessary variable initalizations (Bjorn, Jonathan)
- Convert to platform_driver (Rafael)

> 
> [1]: https://lore.kernel.org/linux-cxl/20201111054356.793390-1-ben.widawsky@intel.com/
> 
> ---
> 
> Introduce support for “type-3” memory devices defined in the recently released
> Compute Express Link (CXL) 2.0 specification[2]. Specifically, these are the
> memory devices defined by section 8.2.8.5 of the CXL 2.0 spec. A reference
> implementation emulating these devices has been submitted to the QEMU mailing
> list and is available on gitlab [3]. “Type-3” is a CXL device that acts as a
> memory expander for RAM or PMEM.  It might be interleaved with other CXL devices
> in a given physical address range.
> 
> These changes allow for foundational enumeration of CXL 2.0 memory devices as
> well as basic userspace interaction. The functionality present is:
> - Initial driver bring-up
> - Device enumeration and an initial sysfs representation
> - Submit a basic firmware command via ‘mailbox’ to an emulated memory device
>   with non-volatile capacity.
> - Provide an interface to send "raw" commands to the hardware.
> 
> Some of the functionality that is still missing includes:
> - Memory interleaving at the host bridge, root port, or switch level
> - CXL 1.1 Root Complex Integrated Endpoint Support
> - CXL 2.0 Hot plug support
> - A bevy of supported device commands
> 
> In addition to the core functionality of discovering the spec defined registers
> and resources, introduce a CXL device model that will be the foundation for
> translating CXL capabilities into existing Linux infrastructure for Persistent
> Memory and other memory devices. For now, this only includes support for the
> management command mailbox that type-3 devices surface. These control devices
> fill the role of “DIMMs” / nmemX memory-devices in LIBNVDIMM terms.
> 
> Now, while implementing the driver some feedback for the specification was
> generated to cover perceived gaps and address conflicts. The feedback is
> presented as a reference implementation in the driver and QEMU emulation.
> Specifically the following concepts are original to the Linux implementation and
> feedback / collaboration is requested to develop these into specification
> proposals:
> 1. Top level ACPI object (ACPI0017)
> 2. HW imposed address space and interleave constraints
> 
> ACPI0017
> --------
> Introduce a new ACPI namespace device with an _HID of ACPI0017. The purpose of
> this object is twofold, support a legacy OS with a set of out-of-tree CXL
> modules, and establish an attach point for a driver that knows about
> interleaving. Both of these boil down to the same point, to centralize Operating
> System support for resources described by the CXL Early Discovery Table (CEDT).
> 
> The legacy OS problem stems from the spec's description of a host bridge,
> ACPI0016 is denoted as the _HID for host bridges, with a _CID of PNP0A08. In a
> CXL unaware version of Linux, the core ACPI subsystem will bind a driver to
> PNP0A08 and preclude a CXL-aware driver from binding to ACPI0016. An ACPI0017
> device allows a standalone CXL-aware driver to register for handling /
> coordinating CEDT and CXL-specific _OSC control.
> 
> Similarly when managing interleaving there needs to be some management layer
> above the ACPI0016 device that is capable of assembling leaf nodes into
> interleave sets. As is the case with ACPI0012 that does this central
> coordination for NFIT defined resources, ACPI0017 does the same for CEDT
> described resources.
> 
> Memory Windows
> -------
> For CXL.mem capable platforms, there is a need for a mechanism for platform
> firmware to make the Operating System aware of any restrictions that hardware
> might have in address space. For example, in a system with 4 host bridges all
> participating in an interleave set, the firmware needs to provide some
> description of this. That information is missing from the CXL 2.0 spec as of
> today and it also is not implemented in the driver. A variety of ACPI based
> mechanisms, for example _CRS fields on the ACPI0017 device, were considered.
> 
> Next steps after this basic foundation is expanded command support and LIBNVDIMM
> integration. This is the initial “release early / release often” version of the
> Linux CXL enabling.
> 
> [2]: https://www.computeexpresslink.org/
> [3]: https://gitlab.com/bwidawsk/qemu/-/tree/cxl-2.0v2
> 
> 
> Ben Widawsky (10):
>   docs: cxl: Add basic documentation
>   cxl/mem: Map memory device registers
>   cxl/mem: Find device capabilities
>   cxl/mem: Implement polled mode mailbox
>   cxl/mem: Add basic IOCTL interface
>   cxl/mem: Add send command
>   cxl/mem: Add a "RAW" send command
>   cxl: Add basic debugging
>   MAINTAINERS: Add maintainers of the CXL driver
>   WIP/cxl/mem: Add get firmware for testing
> 
> Dan Williams (2):
>   cxl/mem: Introduce a driver for CXL-2.0-Type-3 endpoints
>   cxl/mem: Register CXL memX devices
> 
> Vishal Verma (2):
>   cxl/acpi: Add an acpi_cxl module for the CXL interconnect
>   cxl/acpi: add OSC support
> 
>  Documentation/ABI/testing/sysfs-bus-cxl |   26 +
>  Documentation/cxl/index.rst             |   12 +
>  Documentation/cxl/memory-devices.rst    |   51 ++
>  Documentation/index.rst                 |    1 +
>  MAINTAINERS                             |   10 +
>  drivers/Kconfig                         |    1 +
>  drivers/Makefile                        |    1 +
>  drivers/cxl/Kconfig                     |   58 ++
>  drivers/cxl/Makefile                    |    9 +
>  drivers/cxl/acpi.c                      |  352 ++++++++
>  drivers/cxl/acpi.h                      |   35 +
>  drivers/cxl/bus.c                       |   54 ++
>  drivers/cxl/bus.h                       |    8 +
>  drivers/cxl/cxl.h                       |  188 +++++
>  drivers/cxl/mem.c                       | 1022 +++++++++++++++++++++++
>  drivers/cxl/pci.h                       |   34 +
>  include/acpi/actbl1.h                   |   51 ++
>  include/uapi/linux/cxl_mem.h            |  148 ++++
>  18 files changed, 2061 insertions(+)
>  create mode 100644 Documentation/ABI/testing/sysfs-bus-cxl
>  create mode 100644 Documentation/cxl/index.rst
>  create mode 100644 Documentation/cxl/memory-devices.rst
>  create mode 100644 drivers/cxl/Kconfig
>  create mode 100644 drivers/cxl/Makefile
>  create mode 100644 drivers/cxl/acpi.c
>  create mode 100644 drivers/cxl/acpi.h
>  create mode 100644 drivers/cxl/bus.c
>  create mode 100644 drivers/cxl/bus.h
>  create mode 100644 drivers/cxl/cxl.h
>  create mode 100644 drivers/cxl/mem.c
>  create mode 100644 drivers/cxl/pci.h
>  create mode 100644 include/uapi/linux/cxl_mem.h
> 


^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [RFC PATCH v2 00/14] CXL 2.0 Support
  2020-12-09  0:24 [RFC PATCH v2 00/14] CXL 2.0 Support Ben Widawsky
                   ` (14 preceding siblings ...)
  2020-12-09  0:33 ` [RFC PATCH v2 00/14] CXL 2.0 Support Verma, Vishal L
@ 2020-12-09  0:47 ` Ben Widawsky
  15 siblings, 0 replies; 35+ messages in thread
From: Ben Widawsky @ 2020-12-09  0:47 UTC (permalink / raw)
  To: linux-cxl
  Cc: linux-kernel, linux-pci, linux-acpi, Ira Weiny, Dan Williams,
	Vishal Verma, Kelley, Sean V, Rafael Wysocki, Bjorn Helgaas,
	Jonathan Cameron, Jon Masters, Chris Browy, Randy Dunlap,
	Christoph Hellwig

On 20-12-08 16:24:04, Ben Widawsky wrote:

[snip]

This is available on gitlab here: https://gitlab.com/bwidawsk/linux/-/tree/cxl-2.0v2

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [RFC PATCH 12/14] cxl: Add basic debugging
  2020-12-09  0:24 ` [RFC PATCH 12/14] cxl: Add basic debugging Ben Widawsky
@ 2020-12-09  1:17   ` Dan Williams
  2020-12-09  2:04     ` Ben Widawsky
  0 siblings, 1 reply; 35+ messages in thread
From: Dan Williams @ 2020-12-09  1:17 UTC (permalink / raw)
  To: Ben Widawsky
  Cc: linux-cxl, Linux Kernel Mailing List, Linux PCI, Linux ACPI,
	Ira Weiny, Vishal Verma, Kelley, Sean V, Rafael Wysocki,
	Bjorn Helgaas, Jonathan Cameron, Jon Masters, Chris Browy,
	Randy Dunlap, Christoph Hellwig

On Tue, Dec 8, 2020 at 4:24 PM Ben Widawsky <ben.widawsky@intel.com> wrote:
>
> Provide a standard debug function for use throughout the driver.
>
> Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
> ---
>  drivers/cxl/cxl.h |  3 +++
>  drivers/cxl/mem.c | 26 +++++++++++++++++++++++++-
>  2 files changed, 28 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> index 77c2dee6843c..e5afb89dab0b 100644
> --- a/drivers/cxl/cxl.h
> +++ b/drivers/cxl/cxl.h
> @@ -9,6 +9,9 @@
>  #include <linux/bitops.h>
>  #include <linux/io.h>
>
> +#define cxl_debug(fmt, ...)                                                    \
> +       pr_debug("CXL DEBUG: %s: " fmt, __func__, ##__VA_ARGS__)
> +

This should be dev_dbg(), then you don't need the CXL DEBUG prefix. In
fact you don't need a cxl_debug() macro at all in that case. cxl_mem
might need a ->dev attribute for this purpose.

>  #define CXL_SET_FIELD(value, field)                                            \
>         ({                                                                     \
>                 WARN_ON(!FIELD_FIT(field##_MASK, value));                      \
> diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
> index a2cea7ac7cc6..6b2f8d3776b5 100644
> --- a/drivers/cxl/mem.c
> +++ b/drivers/cxl/mem.c
> @@ -122,9 +122,12 @@ static int cxl_mem_wait_for_doorbell(struct cxl_mem *cxlm)
>  {
>         const int timeout = msecs_to_jiffies(2000);
>         const unsigned long start = jiffies;
> +       unsigned long end = start;
>
>         while (cxl_doorbell_busy(cxlm)) {
> -               if (time_after(jiffies, start + timeout)) {
> +               end = jiffies;
> +
> +               if (time_after(end, start + timeout)) {
>                         /* Check again in case preempted before timeout test */
>                         if (!cxl_doorbell_busy(cxlm))
>                                 break;
> @@ -133,6 +136,8 @@ static int cxl_mem_wait_for_doorbell(struct cxl_mem *cxlm)
>                 cpu_relax();
>         }
>
> +       cxl_debug("Doorbell wait took %dms",
> +                 jiffies_to_msecs(end) - jiffies_to_msecs(start));
>         return 0;
>  }
>
> @@ -180,6 +185,8 @@ static int cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm,
>         }
>
>         /* #4 */
> +       cxl_debug("Sending command to %s\n",
> +                 dev_driver_string(&cxlm->pdev->dev));

dev_dbg() already includes dev_driver_string().

>         cxl_write_mbox_reg32(cxlm, CXLDEV_MB_CTRL_OFFSET,
>                              CXLDEV_MB_CTRL_DOORBELL);
>
> @@ -308,6 +315,8 @@ static int cxl_mem_open(struct inode *inode, struct file *file)
>         if (!cxlmd)
>                 return -ENXIO;
>
> +       cxl_debug("Opened %pD\n", file);
> +
>         file->private_data = cxlmd;
>
>         return 0;
> @@ -383,6 +392,10 @@ static int handle_mailbox_cmd_from_user(struct cxl_memdev *cxlmd,
>                 .size_in = cmd->info.size_in,
>                 .size_out = size_out,
>         };
> +       cxl_debug("Submitting command for user\n"
> +                 "\topcode: %x\n"
> +                 "\tsize: %zub/%zub\n",
> +                 mbox_cmd.opcode, mbox_cmd.size_in, mbox_cmd.size_out);
>         rc = cxl_mem_mbox_send_cmd(cxlmd->cxlm, &mbox_cmd);
>         cxl_mem_mbox_put(cxlmd->cxlm);
>         if (rc)
> @@ -479,6 +492,8 @@ static long cxl_mem_ioctl(struct file *file, unsigned int cmd, unsigned long arg
>                 u32 n_commands;
>                 int i, j;
>
> +               cxl_debug("Query IOCTL\n");
> +
>                 if (get_user(n_commands, (u32 __user *)arg))
>                         return -EFAULT;
>
> @@ -511,6 +526,8 @@ static long cxl_mem_ioctl(struct file *file, unsigned int cmd, unsigned long arg
>                 struct cxl_mem_command c;
>                 int rc;
>
> +               cxl_debug("Send IOCTL\n");
> +
>                 rc = cxl_validate_cmd_from_user(u, &c);
>                 if (rc)
>                         return rc;
> @@ -843,6 +860,13 @@ static int cxl_mem_identify(struct cxl_mem *cxlm)
>
>         id = (struct cxl_mbox_identify *)mbox_cmd.payload;
>
> +       cxl_debug("Driver identify command\n"
> +                 "\tFirmware Version: %s\n"
> +                 "\tTotal Capacity: %llu (%llu persistent)\n"
> +                 "\tLSA size: %u\n",
> +                 id->fw_revision, id->total_capacity, id->persistent_capacity,
> +                 id->lsa_size);
> +

Seems not necessary for details that are published in sysfs?

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [RFC PATCH 09/14] cxl/mem: Add basic IOCTL interface
  2020-12-09  0:24 ` [RFC PATCH 09/14] cxl/mem: Add basic IOCTL interface Ben Widawsky
@ 2020-12-09  1:37   ` Dan Williams
  2020-12-09  2:12     ` Ben Widawsky
  2020-12-10  3:32   ` Randy Dunlap
  1 sibling, 1 reply; 35+ messages in thread
From: Dan Williams @ 2020-12-09  1:37 UTC (permalink / raw)
  To: Ben Widawsky
  Cc: linux-cxl, Linux Kernel Mailing List, Linux PCI, Linux ACPI,
	Ira Weiny, Vishal Verma, Kelley, Sean V, Rafael Wysocki,
	Bjorn Helgaas, Jonathan Cameron, Jon Masters, Chris Browy,
	Randy Dunlap, Christoph Hellwig

On Tue, Dec 8, 2020 at 4:24 PM Ben Widawsky <ben.widawsky@intel.com> wrote:
>
> Add a straightforward IOCTL that provides a mechanism for userspace to
> query the supported memory device commands.
>
> Memory device commands are specified in 8.2.9 of the CXL 2.0
> specification. They are submitted through a mailbox mechanism specified
> in 8.2.8.4.
>
> Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
>
> ---
>
> I did attempt to use the same struct for querying commands as well as
> sending commands (upcoming patch). The number of unused fields between
> the two made for a bad fit IMO.
>
> Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
> ---
>  Documentation/cxl/memory-devices.rst |   9 +++
>  drivers/cxl/mem.c                    |  89 +++++++++++++++++++++++
>  include/uapi/linux/cxl_mem.h         | 102 +++++++++++++++++++++++++++
>  3 files changed, 200 insertions(+)
>  create mode 100644 include/uapi/linux/cxl_mem.h
>
> diff --git a/Documentation/cxl/memory-devices.rst b/Documentation/cxl/memory-devices.rst
> index 5f723c25382b..ec54674b3822 100644
> --- a/Documentation/cxl/memory-devices.rst
> +++ b/Documentation/cxl/memory-devices.rst
> @@ -32,6 +32,15 @@ CXL Memory Device
>  .. kernel-doc:: drivers/cxl/mem.c
>     :internal:
>
> +CXL IOCTL Interface
> +-------------------
> +
> +.. kernel-doc:: include/uapi/linux/cxl_mem.h
> +   :doc: UAPI
> +
> +.. kernel-doc:: include/uapi/linux/cxl_mem.h
> +   :internal:
> +
>  External Interfaces
>  ===================
>
> diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
> index bb6ea58f6c7b..2c4aadcea0e4 100644
> --- a/drivers/cxl/mem.c
> +++ b/drivers/cxl/mem.c
> @@ -7,6 +7,7 @@
>  #include <linux/idr.h>
>  #include <linux/pci.h>
>  #include <linux/io.h>
> +#include <uapi/linux/cxl_mem.h>
>  #include "acpi.h"
>  #include "pci.h"
>  #include "cxl.h"
> @@ -73,6 +74,49 @@ static DEFINE_IDR(cxl_mem_idr);
>  /* protect cxl_mem_idr allocations */
>  static DEFINE_MUTEX(cxl_memdev_lock);
>
> +/*
> + * This table defines the supported mailboxes commands for the driver. The id is
> + * ordinal and thus gaps in this table aren't allowed. This table is made up of
> + * a UAPI structure. Non-negative values in the table will be validated against
> + * the user's input. For example, if size_in is 0, and the user passed in 1, it
> + * is an error.
> + */
> +#define CXL_CMD(_id, _flags, sin, sout, _name, _enable, op)                    \
> +       {                                                                      \
> +               { .id = CXL_MEM_COMMAND_ID_##_id,                              \
> +                 .flags = CXL_MEM_COMMAND_FLAG_##_flags,                      \
> +                 .size_in = sin,                                              \
> +                 .size_out = sout,                                            \
> +                 .name = _name },                                             \
> +                       .enable = _enable, .opcode = op                        \
> +       }

Seems the ordinality requirement could be dropped if the definition was:

#define CXL_CMD(_id, _flags, sin, sout, _name, _enable, op)                    \
       [CXL_MEM_COMMAND_ID_##_id] = {
                             \
               { .id = CXL_MEM_COMMAND_ID_##_id,                              \
...

Then command 0 and 42 could be defined out of order in the table.
Especially if we need to config-disable or deprecate commands, I think
it would be useful if this table was tolerant to being sparse.

> +
> +/**
> + * struct cxl_mem_command - Driver representation of a memory device command
> + * @info: Command information as it exists for the UAPI
> + * @opcode: The actual bits used for the mailbox protocol
> + * @enable: Whether the command is enabled. The driver may support a large set
> + *         of commands that may not be enabled. The primary reason a command
> + *         would not be enabled is for commands that are specified as optional
> + *         and the hardware doesn't support the command.
> + *
> + * The cxl_mem_command is the driver's internal representation of commands that
> + * are supported by the driver. Some of these commands may not be supported by
> + * the hardware (!@enable). The driver will use @info to validate the fields
> + * passed in by the user then submit the @opcode to the hardware.
> + *
> + * See struct cxl_command_info.
> + */
> +struct cxl_mem_command {
> +       const struct cxl_command_info info;
> +       const u16 opcode;
> +       bool enable;
> +};
> +
> +static struct cxl_mem_command mem_commands[] = {
> +       CXL_CMD(INVALID, NONE, 0, 0, "Reserved", false, 0),
> +};
> +
>  static int cxl_mem_wait_for_doorbell(struct cxl_mem *cxlm)
>  {
>         const int timeout = msecs_to_jiffies(2000);
> @@ -268,8 +312,53 @@ static int cxl_mem_open(struct inode *inode, struct file *file)
>         return 0;
>  }
>
> +static int cxl_mem_count_commands(void)
> +{
> +       int i, n = 0;
> +
> +       for (i = 0; i < ARRAY_SIZE(mem_commands); i++) {
> +               struct cxl_mem_command *c = &mem_commands[i];
> +
> +               if (c->enable)
> +                       n++;
> +       }
> +
> +       return n;
> +}
> +
>  static long cxl_mem_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
>  {
> +       if (cmd == CXL_MEM_QUERY_COMMANDS) {
> +               struct cxl_mem_query_commands __user *q = (void __user *)arg;
> +               u32 n_commands;
> +               int i, j;
> +
> +               if (get_user(n_commands, (u32 __user *)arg))
> +                       return -EFAULT;
> +
> +               if (n_commands == 0)
> +                       return put_user(cxl_mem_count_commands(),
> +                                       (u32 __user *)arg);
> +
> +               for (i = 0, j = 0;
> +                    i < ARRAY_SIZE(mem_commands) && j < n_commands; i++) {
> +                       struct cxl_mem_command *c = &mem_commands[i];
> +                       const struct cxl_command_info *info = &c->info;
> +
> +                       if (!c->enable)
> +                               continue;
> +
> +                       if (copy_to_user(&q->commands[j], info, sizeof(*info)))
> +                               return -EFAULT;
> +
> +                       if (copy_to_user(&q->commands[j].name, info->name,
> +                                        strlen(info->name)))
> +                               return -EFAULT;

Not sure why this is needed, see comment below about @name in
cxl_mem_query_commands.

> +
> +                       j++;
> +               }
> +       }
> +
>         return -ENOTTY;
>  }
>
> diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
> new file mode 100644
> index 000000000000..1d1e143f98ec
> --- /dev/null
> +++ b/include/uapi/linux/cxl_mem.h
> @@ -0,0 +1,102 @@
> +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
> +/*
> + * CXL IOCTLs for Memory Devices
> + */
> +
> +#ifndef _UAPI_CXL_MEM_H_
> +#define _UAPI_CXL_MEM_H_
> +
> +#if defined(__cplusplus)
> +extern "C" {
> +#endif
> +
> +/**
> + * DOC: UAPI
> + *
> + * CXL memory devices expose UAPI to have a standard user interface.
> + * Userspace can refer to these structure definitions and UAPI formats
> + * to communicate to driver
> + */
> +
> +#define CXL_MEM_QUERY_COMMANDS _IOR('C', 1, struct cxl_mem_query_commands)
> +
> +#define CXL_MEM_COMMAND_NAME_LENGTH 32
> +
> +/**
> + * struct cxl_command_info - Command information returned from a query.
> + * @id: ID number for the command.
> + * @flags: Flags that specify command behavior.
> + *
> + *          - CXL_MEM_COMMAND_FLAG_TAINT: Using this command will taint the kernel.
> + * @size_in: Expected input size, or -1 if variable length.
> + * @size_out: Expected output size, or -1 if variable length.
> + * @name: Name describing the command.
> + *
> + * Represents a single command that is supported by both the driver and the
> + * hardware. The is returned as part of an array from the query ioctl. The
> + * following would be a command named "foobar" that takes a variable length
> + * input and returns 0 bytes of output.
> + *
> + *  - @id = 10
> + *  - @name = foobar
> + *  - @flags = 0
> + *  - @size_in = -1
> + *  - @size_out = 0
> + *
> + * See struct cxl_mem_query_commands.
> + */
> +struct cxl_command_info {
> +       __u32 id;
> +#define CXL_MEM_COMMAND_ID_INVALID 0
> +
> +       __u32 flags;
> +#define CXL_MEM_COMMAND_FLAG_NONE 0
> +#define CXL_MEM_COMMAND_FLAG_TAINT BIT(0)
> +
> +       __s32 size_in;
> +       __s32 size_out;
> +
> +       char name[32];

Why does the name for a command need to be shuffled back and forth
over the ioctl interface. Can't this be handled by a static lookup
table defined in the header?

> +};
> +
> +/**
> + * struct cxl_mem_query_commands - Query supported commands.
> + * @n_commands: In/out parameter. When @n_commands is > 0, the driver will
> + *             return min(num_support_commands, n_commands). When @n_commands
> + *             is 0, driver will return the number of total supported commands.
> + * @rsvd: Reserved for future use.
> + * @commands: Output array of supported commands. This array must be allocated
> + *            by userspace to be at least min(num_support_commands, @n_commands)
> + *
> + * Allow userspace to query the available commands supported by both the driver,
> + * and the hardware. Commands that aren't supported by either the driver, or the
> + * hardware are not returned in the query.
> + *
> + * Examples:
> + *
> + *  - { .n_commands = 0 } // Get number of supported commands
> + *  - { .n_commands = 15, .commands = buf } // Return first 15 (or less)
> + *    supported commands
> + *
> + *  See struct cxl_command_info.
> + */
> +struct cxl_mem_query_commands {
> +       /*
> +        * Input: Number of commands to return (space allocated by user)
> +        * Output: Number of commands supported by the driver/hardware
> +        *
> +        * If n_commands is 0, kernel will only return number of commands and
> +        * not try to populate commands[], thus allowing userspace to know how
> +        * much space to allocate
> +        */
> +       __u32 n_commands;
> +       __u32 rsvd;
> +
> +       struct cxl_command_info __user commands[]; /* out: supported commands */
> +};
> +
> +#if defined(__cplusplus)
> +}
> +#endif
> +
> +#endif
> --
> 2.29.2
>

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [RFC PATCH 12/14] cxl: Add basic debugging
  2020-12-09  1:17   ` Dan Williams
@ 2020-12-09  2:04     ` Ben Widawsky
  2020-12-09  3:06       ` Dan Williams
  0 siblings, 1 reply; 35+ messages in thread
From: Ben Widawsky @ 2020-12-09  2:04 UTC (permalink / raw)
  To: Dan Williams
  Cc: linux-cxl, Linux Kernel Mailing List, Linux PCI, Linux ACPI,
	Ira Weiny, Vishal Verma, Kelley, Sean V, Rafael Wysocki,
	Bjorn Helgaas, Jonathan Cameron, Jon Masters, Chris Browy,
	Randy Dunlap, Christoph Hellwig

On 20-12-08 17:17:36, Dan Williams wrote:
> On Tue, Dec 8, 2020 at 4:24 PM Ben Widawsky <ben.widawsky@intel.com> wrote:
> >
> > Provide a standard debug function for use throughout the driver.
> >
> > Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
> > ---
> >  drivers/cxl/cxl.h |  3 +++
> >  drivers/cxl/mem.c | 26 +++++++++++++++++++++++++-
> >  2 files changed, 28 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> > index 77c2dee6843c..e5afb89dab0b 100644
> > --- a/drivers/cxl/cxl.h
> > +++ b/drivers/cxl/cxl.h
> > @@ -9,6 +9,9 @@
> >  #include <linux/bitops.h>
> >  #include <linux/io.h>
> >
> > +#define cxl_debug(fmt, ...)                                                    \
> > +       pr_debug("CXL DEBUG: %s: " fmt, __func__, ##__VA_ARGS__)
> > +
> 
> This should be dev_dbg(), then you don't need the CXL DEBUG prefix. In
> fact you don't need a cxl_debug() macro at all in that case. cxl_mem
> might need a ->dev attribute for this purpose.
> 

I really like the ability to turn specific messages on and off at will. (FWIW,
__func__ is also redundant because pr_debug allows you to specify a flag to
always print the function name). While it's not very frequent events here, in
the future it likely will be and I think it can be really helpful to be able to
have that level of control.

If you want to avoid creating a new debug functionality, I'm okay with that, but
I'd really like to use pr_debug instead of dev_dbg for those messages going
forward. Once you take that step, it seems giving contributors a macro named
'cxl_debug' so they don't have to figure out when to use what, makes sense. My
mental separation is, dev_* is useful primarily for errors and initialization
debug messaging, pr_debug/trace_printk is for runtime things.

I probably should have put that in the commit message...

> >  #define CXL_SET_FIELD(value, field)                                            \
> >         ({                                                                     \
> >                 WARN_ON(!FIELD_FIT(field##_MASK, value));                      \
> > diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
> > index a2cea7ac7cc6..6b2f8d3776b5 100644
> > --- a/drivers/cxl/mem.c
> > +++ b/drivers/cxl/mem.c
> > @@ -122,9 +122,12 @@ static int cxl_mem_wait_for_doorbell(struct cxl_mem *cxlm)
> >  {
> >         const int timeout = msecs_to_jiffies(2000);
> >         const unsigned long start = jiffies;
> > +       unsigned long end = start;
> >
> >         while (cxl_doorbell_busy(cxlm)) {
> > -               if (time_after(jiffies, start + timeout)) {
> > +               end = jiffies;
> > +
> > +               if (time_after(end, start + timeout)) {
> >                         /* Check again in case preempted before timeout test */
> >                         if (!cxl_doorbell_busy(cxlm))
> >                                 break;
> > @@ -133,6 +136,8 @@ static int cxl_mem_wait_for_doorbell(struct cxl_mem *cxlm)
> >                 cpu_relax();
> >         }
> >
> > +       cxl_debug("Doorbell wait took %dms",
> > +                 jiffies_to_msecs(end) - jiffies_to_msecs(start));
> >         return 0;
> >  }
> >
> > @@ -180,6 +185,8 @@ static int cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm,
> >         }
> >
> >         /* #4 */
> > +       cxl_debug("Sending command to %s\n",
> > +                 dev_driver_string(&cxlm->pdev->dev));
> 
> dev_dbg() already includes dev_driver_string().
> 
> >         cxl_write_mbox_reg32(cxlm, CXLDEV_MB_CTRL_OFFSET,
> >                              CXLDEV_MB_CTRL_DOORBELL);
> >
> > @@ -308,6 +315,8 @@ static int cxl_mem_open(struct inode *inode, struct file *file)
> >         if (!cxlmd)
> >                 return -ENXIO;
> >
> > +       cxl_debug("Opened %pD\n", file);
> > +
> >         file->private_data = cxlmd;
> >
> >         return 0;
> > @@ -383,6 +392,10 @@ static int handle_mailbox_cmd_from_user(struct cxl_memdev *cxlmd,
> >                 .size_in = cmd->info.size_in,
> >                 .size_out = size_out,
> >         };
> > +       cxl_debug("Submitting command for user\n"
> > +                 "\topcode: %x\n"
> > +                 "\tsize: %zub/%zub\n",
> > +                 mbox_cmd.opcode, mbox_cmd.size_in, mbox_cmd.size_out);
> >         rc = cxl_mem_mbox_send_cmd(cxlmd->cxlm, &mbox_cmd);
> >         cxl_mem_mbox_put(cxlmd->cxlm);
> >         if (rc)
> > @@ -479,6 +492,8 @@ static long cxl_mem_ioctl(struct file *file, unsigned int cmd, unsigned long arg
> >                 u32 n_commands;
> >                 int i, j;
> >
> > +               cxl_debug("Query IOCTL\n");
> > +
> >                 if (get_user(n_commands, (u32 __user *)arg))
> >                         return -EFAULT;
> >
> > @@ -511,6 +526,8 @@ static long cxl_mem_ioctl(struct file *file, unsigned int cmd, unsigned long arg
> >                 struct cxl_mem_command c;
> >                 int rc;
> >
> > +               cxl_debug("Send IOCTL\n");
> > +
> >                 rc = cxl_validate_cmd_from_user(u, &c);
> >                 if (rc)
> >                         return rc;
> > @@ -843,6 +860,13 @@ static int cxl_mem_identify(struct cxl_mem *cxlm)
> >
> >         id = (struct cxl_mbox_identify *)mbox_cmd.payload;
> >
> > +       cxl_debug("Driver identify command\n"
> > +                 "\tFirmware Version: %s\n"
> > +                 "\tTotal Capacity: %llu (%llu persistent)\n"
> > +                 "\tLSA size: %u\n",
> > +                 id->fw_revision, id->total_capacity, id->persistent_capacity,
> > +                 id->lsa_size);
> > +
> 
> Seems not necessary for details that are published in sysfs?

I was thinking for cases where driver doesn't bind, seeing the identify
information could be useful. This is one case where dev_dbg would also serve
IMO.

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [RFC PATCH 09/14] cxl/mem: Add basic IOCTL interface
  2020-12-09  1:37   ` Dan Williams
@ 2020-12-09  2:12     ` Ben Widawsky
  2020-12-09  3:33       ` Dan Williams
  0 siblings, 1 reply; 35+ messages in thread
From: Ben Widawsky @ 2020-12-09  2:12 UTC (permalink / raw)
  To: Dan Williams
  Cc: linux-cxl, Linux Kernel Mailing List, Linux PCI, Linux ACPI,
	Ira Weiny, Vishal Verma, Kelley, Sean V, Rafael Wysocki,
	Bjorn Helgaas, Jonathan Cameron, Jon Masters, Chris Browy,
	Randy Dunlap, Christoph Hellwig

On 20-12-08 17:37:50, Dan Williams wrote:
> On Tue, Dec 8, 2020 at 4:24 PM Ben Widawsky <ben.widawsky@intel.com> wrote:
> >
> > Add a straightforward IOCTL that provides a mechanism for userspace to
> > query the supported memory device commands.
> >
> > Memory device commands are specified in 8.2.9 of the CXL 2.0
> > specification. They are submitted through a mailbox mechanism specified
> > in 8.2.8.4.
> >
> > Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
> >
> > ---
> >
> > I did attempt to use the same struct for querying commands as well as
> > sending commands (upcoming patch). The number of unused fields between
> > the two made for a bad fit IMO.
> >
> > Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
> > ---
> >  Documentation/cxl/memory-devices.rst |   9 +++
> >  drivers/cxl/mem.c                    |  89 +++++++++++++++++++++++
> >  include/uapi/linux/cxl_mem.h         | 102 +++++++++++++++++++++++++++
> >  3 files changed, 200 insertions(+)
> >  create mode 100644 include/uapi/linux/cxl_mem.h
> >
> > diff --git a/Documentation/cxl/memory-devices.rst b/Documentation/cxl/memory-devices.rst
> > index 5f723c25382b..ec54674b3822 100644
> > --- a/Documentation/cxl/memory-devices.rst
> > +++ b/Documentation/cxl/memory-devices.rst
> > @@ -32,6 +32,15 @@ CXL Memory Device
> >  .. kernel-doc:: drivers/cxl/mem.c
> >     :internal:
> >
> > +CXL IOCTL Interface
> > +-------------------
> > +
> > +.. kernel-doc:: include/uapi/linux/cxl_mem.h
> > +   :doc: UAPI
> > +
> > +.. kernel-doc:: include/uapi/linux/cxl_mem.h
> > +   :internal:
> > +
> >  External Interfaces
> >  ===================
> >
> > diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
> > index bb6ea58f6c7b..2c4aadcea0e4 100644
> > --- a/drivers/cxl/mem.c
> > +++ b/drivers/cxl/mem.c
> > @@ -7,6 +7,7 @@
> >  #include <linux/idr.h>
> >  #include <linux/pci.h>
> >  #include <linux/io.h>
> > +#include <uapi/linux/cxl_mem.h>
> >  #include "acpi.h"
> >  #include "pci.h"
> >  #include "cxl.h"
> > @@ -73,6 +74,49 @@ static DEFINE_IDR(cxl_mem_idr);
> >  /* protect cxl_mem_idr allocations */
> >  static DEFINE_MUTEX(cxl_memdev_lock);
> >
> > +/*
> > + * This table defines the supported mailboxes commands for the driver. The id is
> > + * ordinal and thus gaps in this table aren't allowed. This table is made up of
> > + * a UAPI structure. Non-negative values in the table will be validated against
> > + * the user's input. For example, if size_in is 0, and the user passed in 1, it
> > + * is an error.
> > + */
> > +#define CXL_CMD(_id, _flags, sin, sout, _name, _enable, op)                    \
> > +       {                                                                      \
> > +               { .id = CXL_MEM_COMMAND_ID_##_id,                              \
> > +                 .flags = CXL_MEM_COMMAND_FLAG_##_flags,                      \
> > +                 .size_in = sin,                                              \
> > +                 .size_out = sout,                                            \
> > +                 .name = _name },                                             \
> > +                       .enable = _enable, .opcode = op                        \
> > +       }
> 
> Seems the ordinality requirement could be dropped if the definition was:
> 
> #define CXL_CMD(_id, _flags, sin, sout, _name, _enable, op)                    \
>        [CXL_MEM_COMMAND_ID_##_id] = {
>                              \
>                { .id = CXL_MEM_COMMAND_ID_##_id,                              \
> ...
> 
> Then command 0 and 42 could be defined out of order in the table.
> Especially if we need to config-disable or deprecate commands, I think
> it would be useful if this table was tolerant to being sparse.
> 

How sparse are we talking? The current form does support sparseness, but
obviously gets quite large if the ID numbering is similar to random
distribution.

I think if we do see this being more like random distribution, it can be
supported, but I think it adds a decent amount of complexity for what I see as
not much reward - unless you know of a fairly simple way to create this data
structure with full sparse ID support?

> > +
> > +/**
> > + * struct cxl_mem_command - Driver representation of a memory device command
> > + * @info: Command information as it exists for the UAPI
> > + * @opcode: The actual bits used for the mailbox protocol
> > + * @enable: Whether the command is enabled. The driver may support a large set
> > + *         of commands that may not be enabled. The primary reason a command
> > + *         would not be enabled is for commands that are specified as optional
> > + *         and the hardware doesn't support the command.
> > + *
> > + * The cxl_mem_command is the driver's internal representation of commands that
> > + * are supported by the driver. Some of these commands may not be supported by
> > + * the hardware (!@enable). The driver will use @info to validate the fields
> > + * passed in by the user then submit the @opcode to the hardware.
> > + *
> > + * See struct cxl_command_info.
> > + */
> > +struct cxl_mem_command {
> > +       const struct cxl_command_info info;
> > +       const u16 opcode;
> > +       bool enable;
> > +};
> > +
> > +static struct cxl_mem_command mem_commands[] = {
> > +       CXL_CMD(INVALID, NONE, 0, 0, "Reserved", false, 0),
> > +};
> > +
> >  static int cxl_mem_wait_for_doorbell(struct cxl_mem *cxlm)
> >  {
> >         const int timeout = msecs_to_jiffies(2000);
> > @@ -268,8 +312,53 @@ static int cxl_mem_open(struct inode *inode, struct file *file)
> >         return 0;
> >  }
> >
> > +static int cxl_mem_count_commands(void)
> > +{
> > +       int i, n = 0;
> > +
> > +       for (i = 0; i < ARRAY_SIZE(mem_commands); i++) {
> > +               struct cxl_mem_command *c = &mem_commands[i];
> > +
> > +               if (c->enable)
> > +                       n++;
> > +       }
> > +
> > +       return n;
> > +}
> > +
> >  static long cxl_mem_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
> >  {
> > +       if (cmd == CXL_MEM_QUERY_COMMANDS) {
> > +               struct cxl_mem_query_commands __user *q = (void __user *)arg;
> > +               u32 n_commands;
> > +               int i, j;
> > +
> > +               if (get_user(n_commands, (u32 __user *)arg))
> > +                       return -EFAULT;
> > +
> > +               if (n_commands == 0)
> > +                       return put_user(cxl_mem_count_commands(),
> > +                                       (u32 __user *)arg);
> > +
> > +               for (i = 0, j = 0;
> > +                    i < ARRAY_SIZE(mem_commands) && j < n_commands; i++) {
> > +                       struct cxl_mem_command *c = &mem_commands[i];
> > +                       const struct cxl_command_info *info = &c->info;
> > +
> > +                       if (!c->enable)
> > +                               continue;
> > +
> > +                       if (copy_to_user(&q->commands[j], info, sizeof(*info)))
> > +                               return -EFAULT;
> > +
> > +                       if (copy_to_user(&q->commands[j].name, info->name,
> > +                                        strlen(info->name)))
> > +                               return -EFAULT;
> 
> Not sure why this is needed, see comment below about @name in
> cxl_mem_query_commands.
> 
> > +
> > +                       j++;
> > +               }
> > +       }
> > +
> >         return -ENOTTY;
> >  }
> >
> > diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
> > new file mode 100644
> > index 000000000000..1d1e143f98ec
> > --- /dev/null
> > +++ b/include/uapi/linux/cxl_mem.h
> > @@ -0,0 +1,102 @@
> > +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
> > +/*
> > + * CXL IOCTLs for Memory Devices
> > + */
> > +
> > +#ifndef _UAPI_CXL_MEM_H_
> > +#define _UAPI_CXL_MEM_H_
> > +
> > +#if defined(__cplusplus)
> > +extern "C" {
> > +#endif
> > +
> > +/**
> > + * DOC: UAPI
> > + *
> > + * CXL memory devices expose UAPI to have a standard user interface.
> > + * Userspace can refer to these structure definitions and UAPI formats
> > + * to communicate to driver
> > + */
> > +
> > +#define CXL_MEM_QUERY_COMMANDS _IOR('C', 1, struct cxl_mem_query_commands)
> > +
> > +#define CXL_MEM_COMMAND_NAME_LENGTH 32
> > +
> > +/**
> > + * struct cxl_command_info - Command information returned from a query.
> > + * @id: ID number for the command.
> > + * @flags: Flags that specify command behavior.
> > + *
> > + *          - CXL_MEM_COMMAND_FLAG_TAINT: Using this command will taint the kernel.
> > + * @size_in: Expected input size, or -1 if variable length.
> > + * @size_out: Expected output size, or -1 if variable length.
> > + * @name: Name describing the command.
> > + *
> > + * Represents a single command that is supported by both the driver and the
> > + * hardware. The is returned as part of an array from the query ioctl. The
> > + * following would be a command named "foobar" that takes a variable length
> > + * input and returns 0 bytes of output.
> > + *
> > + *  - @id = 10
> > + *  - @name = foobar
> > + *  - @flags = 0
> > + *  - @size_in = -1
> > + *  - @size_out = 0
> > + *
> > + * See struct cxl_mem_query_commands.
> > + */
> > +struct cxl_command_info {
> > +       __u32 id;
> > +#define CXL_MEM_COMMAND_ID_INVALID 0
> > +
> > +       __u32 flags;
> > +#define CXL_MEM_COMMAND_FLAG_NONE 0
> > +#define CXL_MEM_COMMAND_FLAG_TAINT BIT(0)
> > +
> > +       __s32 size_in;
> > +       __s32 size_out;
> > +
> > +       char name[32];
> 
> Why does the name for a command need to be shuffled back and forth
> over the ioctl interface. Can't this be handled by a static lookup
> table defined in the header?
> 

I was thinking of cases where the userspace application doesn't match the
current kernel's UAPI and giving the driver flexibility to return whatever.

OTTOMH, I also can't think of a way to do this if you want to do define the
table sparsely though. Do you have ideas for that?

> > +};
> > +
> > +/**
> > + * struct cxl_mem_query_commands - Query supported commands.
> > + * @n_commands: In/out parameter. When @n_commands is > 0, the driver will
> > + *             return min(num_support_commands, n_commands). When @n_commands
> > + *             is 0, driver will return the number of total supported commands.
> > + * @rsvd: Reserved for future use.
> > + * @commands: Output array of supported commands. This array must be allocated
> > + *            by userspace to be at least min(num_support_commands, @n_commands)
> > + *
> > + * Allow userspace to query the available commands supported by both the driver,
> > + * and the hardware. Commands that aren't supported by either the driver, or the
> > + * hardware are not returned in the query.
> > + *
> > + * Examples:
> > + *
> > + *  - { .n_commands = 0 } // Get number of supported commands
> > + *  - { .n_commands = 15, .commands = buf } // Return first 15 (or less)
> > + *    supported commands
> > + *
> > + *  See struct cxl_command_info.
> > + */
> > +struct cxl_mem_query_commands {
> > +       /*
> > +        * Input: Number of commands to return (space allocated by user)
> > +        * Output: Number of commands supported by the driver/hardware
> > +        *
> > +        * If n_commands is 0, kernel will only return number of commands and
> > +        * not try to populate commands[], thus allowing userspace to know how
> > +        * much space to allocate
> > +        */
> > +       __u32 n_commands;
> > +       __u32 rsvd;
> > +
> > +       struct cxl_command_info __user commands[]; /* out: supported commands */
> > +};
> > +
> > +#if defined(__cplusplus)
> > +}
> > +#endif
> > +
> > +#endif
> > --
> > 2.29.2
> >

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [RFC PATCH 12/14] cxl: Add basic debugging
  2020-12-09  2:04     ` Ben Widawsky
@ 2020-12-09  3:06       ` Dan Williams
  2020-12-16 21:02         ` Ben Widawsky
  0 siblings, 1 reply; 35+ messages in thread
From: Dan Williams @ 2020-12-09  3:06 UTC (permalink / raw)
  To: Ben Widawsky
  Cc: linux-cxl, Linux Kernel Mailing List, Linux PCI, Linux ACPI,
	Ira Weiny, Vishal Verma, Kelley, Sean V, Rafael Wysocki,
	Bjorn Helgaas, Jonathan Cameron, Jon Masters, Chris Browy,
	Randy Dunlap, Christoph Hellwig

On Tue, Dec 8, 2020 at 6:04 PM Ben Widawsky <ben.widawsky@intel.com> wrote:
>
> On 20-12-08 17:17:36, Dan Williams wrote:
> > On Tue, Dec 8, 2020 at 4:24 PM Ben Widawsky <ben.widawsky@intel.com> wrote:
> > >
> > > Provide a standard debug function for use throughout the driver.
> > >
> > > Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
> > > ---
> > >  drivers/cxl/cxl.h |  3 +++
> > >  drivers/cxl/mem.c | 26 +++++++++++++++++++++++++-
> > >  2 files changed, 28 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> > > index 77c2dee6843c..e5afb89dab0b 100644
> > > --- a/drivers/cxl/cxl.h
> > > +++ b/drivers/cxl/cxl.h
> > > @@ -9,6 +9,9 @@
> > >  #include <linux/bitops.h>
> > >  #include <linux/io.h>
> > >
> > > +#define cxl_debug(fmt, ...)                                                    \
> > > +       pr_debug("CXL DEBUG: %s: " fmt, __func__, ##__VA_ARGS__)
> > > +
> >
> > This should be dev_dbg(), then you don't need the CXL DEBUG prefix. In
> > fact you don't need a cxl_debug() macro at all in that case. cxl_mem
> > might need a ->dev attribute for this purpose.
> >
>
> I really like the ability to turn specific messages on and off at will. (FWIW,
> __func__ is also redundant because pr_debug allows you to specify a flag to
> always print the function name). While it's not very frequent events here, in
> the future it likely will be and I think it can be really helpful to be able to
> have that level of control.
>
> If you want to avoid creating a new debug functionality, I'm okay with that, but
> I'd really like to use pr_debug instead of dev_dbg for those messages going
> forward. Once you take that step, it seems giving contributors a macro named
> 'cxl_debug' so they don't have to figure out when to use what, makes sense. My
> mental separation is, dev_* is useful primarily for errors and initialization
> debug messaging, pr_debug/trace_printk is for runtime things.
>
> I probably should have put that in the commit message...

I suspect you haven't taken a look at the backed and of pr_debug() and
dev_dbg() in a while? They both use _dynamic_func_call and enjoy all
the benefits afforded by /sys/debug/dynamic_debug/control and the
"dyndbg" module option for adding __func__ and enable / disable by
line number or format message . pr_debug() and this cxl_debug() macro
are completely superseded by dev_dbg(). Even if a driver wanted a
common prefix on all prints there is the "#define dev_fmt ..."
mechanism for that.

>
> > >  #define CXL_SET_FIELD(value, field)                                            \
> > >         ({                                                                     \
> > >                 WARN_ON(!FIELD_FIT(field##_MASK, value));                      \
> > > diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
> > > index a2cea7ac7cc6..6b2f8d3776b5 100644
> > > --- a/drivers/cxl/mem.c
> > > +++ b/drivers/cxl/mem.c
> > > @@ -122,9 +122,12 @@ static int cxl_mem_wait_for_doorbell(struct cxl_mem *cxlm)
> > >  {
> > >         const int timeout = msecs_to_jiffies(2000);
> > >         const unsigned long start = jiffies;
> > > +       unsigned long end = start;
> > >
> > >         while (cxl_doorbell_busy(cxlm)) {
> > > -               if (time_after(jiffies, start + timeout)) {
> > > +               end = jiffies;
> > > +
> > > +               if (time_after(end, start + timeout)) {
> > >                         /* Check again in case preempted before timeout test */
> > >                         if (!cxl_doorbell_busy(cxlm))
> > >                                 break;
> > > @@ -133,6 +136,8 @@ static int cxl_mem_wait_for_doorbell(struct cxl_mem *cxlm)
> > >                 cpu_relax();
> > >         }
> > >
> > > +       cxl_debug("Doorbell wait took %dms",
> > > +                 jiffies_to_msecs(end) - jiffies_to_msecs(start));
> > >         return 0;
> > >  }
> > >
> > > @@ -180,6 +185,8 @@ static int cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm,
> > >         }
> > >
> > >         /* #4 */
> > > +       cxl_debug("Sending command to %s\n",
> > > +                 dev_driver_string(&cxlm->pdev->dev));
> >
> > dev_dbg() already includes dev_driver_string().
> >
> > >         cxl_write_mbox_reg32(cxlm, CXLDEV_MB_CTRL_OFFSET,
> > >                              CXLDEV_MB_CTRL_DOORBELL);
> > >
> > > @@ -308,6 +315,8 @@ static int cxl_mem_open(struct inode *inode, struct file *file)
> > >         if (!cxlmd)
> > >                 return -ENXIO;
> > >
> > > +       cxl_debug("Opened %pD\n", file);
> > > +
> > >         file->private_data = cxlmd;
> > >
> > >         return 0;
> > > @@ -383,6 +392,10 @@ static int handle_mailbox_cmd_from_user(struct cxl_memdev *cxlmd,
> > >                 .size_in = cmd->info.size_in,
> > >                 .size_out = size_out,
> > >         };
> > > +       cxl_debug("Submitting command for user\n"
> > > +                 "\topcode: %x\n"
> > > +                 "\tsize: %zub/%zub\n",
> > > +                 mbox_cmd.opcode, mbox_cmd.size_in, mbox_cmd.size_out);
> > >         rc = cxl_mem_mbox_send_cmd(cxlmd->cxlm, &mbox_cmd);
> > >         cxl_mem_mbox_put(cxlmd->cxlm);
> > >         if (rc)
> > > @@ -479,6 +492,8 @@ static long cxl_mem_ioctl(struct file *file, unsigned int cmd, unsigned long arg
> > >                 u32 n_commands;
> > >                 int i, j;
> > >
> > > +               cxl_debug("Query IOCTL\n");
> > > +
> > >                 if (get_user(n_commands, (u32 __user *)arg))
> > >                         return -EFAULT;
> > >
> > > @@ -511,6 +526,8 @@ static long cxl_mem_ioctl(struct file *file, unsigned int cmd, unsigned long arg
> > >                 struct cxl_mem_command c;
> > >                 int rc;
> > >
> > > +               cxl_debug("Send IOCTL\n");
> > > +
> > >                 rc = cxl_validate_cmd_from_user(u, &c);
> > >                 if (rc)
> > >                         return rc;
> > > @@ -843,6 +860,13 @@ static int cxl_mem_identify(struct cxl_mem *cxlm)
> > >
> > >         id = (struct cxl_mbox_identify *)mbox_cmd.payload;
> > >
> > > +       cxl_debug("Driver identify command\n"
> > > +                 "\tFirmware Version: %s\n"
> > > +                 "\tTotal Capacity: %llu (%llu persistent)\n"
> > > +                 "\tLSA size: %u\n",
> > > +                 id->fw_revision, id->total_capacity, id->persistent_capacity,
> > > +                 id->lsa_size);
> > > +
> >
> > Seems not necessary for details that are published in sysfs?
>
> I was thinking for cases where driver doesn't bind, seeing the identify
> information could be useful. This is one case where dev_dbg would also serve
> IMO.

There's very little that can cause bind failures after
cxl_mem_identify() succeeds. The bind failures themselves will
dev_err() the reason.

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [RFC PATCH 09/14] cxl/mem: Add basic IOCTL interface
  2020-12-09  2:12     ` Ben Widawsky
@ 2020-12-09  3:33       ` Dan Williams
  2020-12-09 16:23         ` Ben Widawsky
  0 siblings, 1 reply; 35+ messages in thread
From: Dan Williams @ 2020-12-09  3:33 UTC (permalink / raw)
  To: Ben Widawsky
  Cc: linux-cxl, Linux Kernel Mailing List, Linux PCI, Linux ACPI,
	Ira Weiny, Vishal Verma, Kelley, Sean V, Rafael Wysocki,
	Bjorn Helgaas, Jonathan Cameron, Jon Masters, Chris Browy,
	Randy Dunlap, Christoph Hellwig

On Tue, Dec 8, 2020 at 6:13 PM Ben Widawsky <ben.widawsky@intel.com> wrote:
>
> On 20-12-08 17:37:50, Dan Williams wrote:
> > On Tue, Dec 8, 2020 at 4:24 PM Ben Widawsky <ben.widawsky@intel.com> wrote:
> > >
> > > Add a straightforward IOCTL that provides a mechanism for userspace to
> > > query the supported memory device commands.
> > >
> > > Memory device commands are specified in 8.2.9 of the CXL 2.0
> > > specification. They are submitted through a mailbox mechanism specified
> > > in 8.2.8.4.
> > >
> > > Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
> > >
> > > ---
> > >
> > > I did attempt to use the same struct for querying commands as well as
> > > sending commands (upcoming patch). The number of unused fields between
> > > the two made for a bad fit IMO.
> > >
> > > Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
> > > ---
> > >  Documentation/cxl/memory-devices.rst |   9 +++
> > >  drivers/cxl/mem.c                    |  89 +++++++++++++++++++++++
> > >  include/uapi/linux/cxl_mem.h         | 102 +++++++++++++++++++++++++++
> > >  3 files changed, 200 insertions(+)
> > >  create mode 100644 include/uapi/linux/cxl_mem.h
> > >
> > > diff --git a/Documentation/cxl/memory-devices.rst b/Documentation/cxl/memory-devices.rst
> > > index 5f723c25382b..ec54674b3822 100644
> > > --- a/Documentation/cxl/memory-devices.rst
> > > +++ b/Documentation/cxl/memory-devices.rst
> > > @@ -32,6 +32,15 @@ CXL Memory Device
> > >  .. kernel-doc:: drivers/cxl/mem.c
> > >     :internal:
> > >
> > > +CXL IOCTL Interface
> > > +-------------------
> > > +
> > > +.. kernel-doc:: include/uapi/linux/cxl_mem.h
> > > +   :doc: UAPI
> > > +
> > > +.. kernel-doc:: include/uapi/linux/cxl_mem.h
> > > +   :internal:
> > > +
> > >  External Interfaces
> > >  ===================
> > >
> > > diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
> > > index bb6ea58f6c7b..2c4aadcea0e4 100644
> > > --- a/drivers/cxl/mem.c
> > > +++ b/drivers/cxl/mem.c
> > > @@ -7,6 +7,7 @@
> > >  #include <linux/idr.h>
> > >  #include <linux/pci.h>
> > >  #include <linux/io.h>
> > > +#include <uapi/linux/cxl_mem.h>
> > >  #include "acpi.h"
> > >  #include "pci.h"
> > >  #include "cxl.h"
> > > @@ -73,6 +74,49 @@ static DEFINE_IDR(cxl_mem_idr);
> > >  /* protect cxl_mem_idr allocations */
> > >  static DEFINE_MUTEX(cxl_memdev_lock);
> > >
> > > +/*
> > > + * This table defines the supported mailboxes commands for the driver. The id is
> > > + * ordinal and thus gaps in this table aren't allowed. This table is made up of
> > > + * a UAPI structure. Non-negative values in the table will be validated against
> > > + * the user's input. For example, if size_in is 0, and the user passed in 1, it
> > > + * is an error.
> > > + */
> > > +#define CXL_CMD(_id, _flags, sin, sout, _name, _enable, op)                    \
> > > +       {                                                                      \
> > > +               { .id = CXL_MEM_COMMAND_ID_##_id,                              \
> > > +                 .flags = CXL_MEM_COMMAND_FLAG_##_flags,                      \
> > > +                 .size_in = sin,                                              \
> > > +                 .size_out = sout,                                            \
> > > +                 .name = _name },                                             \
> > > +                       .enable = _enable, .opcode = op                        \
> > > +       }
> >
> > Seems the ordinality requirement could be dropped if the definition was:
> >
> > #define CXL_CMD(_id, _flags, sin, sout, _name, _enable, op)                    \
> >        [CXL_MEM_COMMAND_ID_##_id] = {
> >                              \
> >                { .id = CXL_MEM_COMMAND_ID_##_id,                              \
> > ...
> >
> > Then command 0 and 42 could be defined out of order in the table.
> > Especially if we need to config-disable or deprecate commands, I think
> > it would be useful if this table was tolerant to being sparse.
> >
>
> How sparse are we talking? The current form does support sparseness, but
> obviously gets quite large if the ID numbering is similar to random
> distribution.

"Sparse" may have been the wrong word to use. I was implying sparse
enough that if I add command N+1 I don't need to be careful where I
put it in mem_commands, but still be able to rely on lookups into
mem_commands being indexed by the command-id.

> I think if we do see this being more like random distribution, it can be
> supported, but I think it adds a decent amount of complexity for what I see as
> not much reward - unless you know of a fairly simple way to create this data
> structure with full sparse ID support?

I'm expecting the command distribution to be mostly uniform, it's more
of the lookup property that I think would be useful especially for the
dynamic case of walking mem_commands to update it relative to what the
hardware supports or other metadata. Speaking of which I think @enable
should be turned into @flags of which 'enable' is one, in case we want
to define more flags in the future.

>
> > > +
> > > +/**
> > > + * struct cxl_mem_command - Driver representation of a memory device command
> > > + * @info: Command information as it exists for the UAPI
> > > + * @opcode: The actual bits used for the mailbox protocol
> > > + * @enable: Whether the command is enabled. The driver may support a large set
> > > + *         of commands that may not be enabled. The primary reason a command
> > > + *         would not be enabled is for commands that are specified as optional
> > > + *         and the hardware doesn't support the command.
> > > + *
> > > + * The cxl_mem_command is the driver's internal representation of commands that
> > > + * are supported by the driver. Some of these commands may not be supported by
> > > + * the hardware (!@enable). The driver will use @info to validate the fields
> > > + * passed in by the user then submit the @opcode to the hardware.
> > > + *
> > > + * See struct cxl_command_info.
> > > + */
> > > +struct cxl_mem_command {
> > > +       const struct cxl_command_info info;
> > > +       const u16 opcode;
> > > +       bool enable;
> > > +};
> > > +
> > > +static struct cxl_mem_command mem_commands[] = {
> > > +       CXL_CMD(INVALID, NONE, 0, 0, "Reserved", false, 0),
> > > +};
> > > +
> > >  static int cxl_mem_wait_for_doorbell(struct cxl_mem *cxlm)
> > >  {
> > >         const int timeout = msecs_to_jiffies(2000);
> > > @@ -268,8 +312,53 @@ static int cxl_mem_open(struct inode *inode, struct file *file)
> > >         return 0;
> > >  }
> > >
> > > +static int cxl_mem_count_commands(void)
> > > +{
> > > +       int i, n = 0;
> > > +
> > > +       for (i = 0; i < ARRAY_SIZE(mem_commands); i++) {
> > > +               struct cxl_mem_command *c = &mem_commands[i];
> > > +
> > > +               if (c->enable)
> > > +                       n++;
> > > +       }
> > > +
> > > +       return n;
> > > +}
> > > +
> > >  static long cxl_mem_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
> > >  {
> > > +       if (cmd == CXL_MEM_QUERY_COMMANDS) {
> > > +               struct cxl_mem_query_commands __user *q = (void __user *)arg;
> > > +               u32 n_commands;
> > > +               int i, j;
> > > +
> > > +               if (get_user(n_commands, (u32 __user *)arg))
> > > +                       return -EFAULT;
> > > +
> > > +               if (n_commands == 0)
> > > +                       return put_user(cxl_mem_count_commands(),
> > > +                                       (u32 __user *)arg);
> > > +
> > > +               for (i = 0, j = 0;
> > > +                    i < ARRAY_SIZE(mem_commands) && j < n_commands; i++) {
> > > +                       struct cxl_mem_command *c = &mem_commands[i];
> > > +                       const struct cxl_command_info *info = &c->info;
> > > +
> > > +                       if (!c->enable)
> > > +                               continue;
> > > +
> > > +                       if (copy_to_user(&q->commands[j], info, sizeof(*info)))
> > > +                               return -EFAULT;
> > > +
> > > +                       if (copy_to_user(&q->commands[j].name, info->name,
> > > +                                        strlen(info->name)))
> > > +                               return -EFAULT;
> >
> > Not sure why this is needed, see comment below about @name in
> > cxl_mem_query_commands.
> >
> > > +
> > > +                       j++;
> > > +               }
> > > +       }
> > > +
> > >         return -ENOTTY;
> > >  }
> > >
> > > diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
> > > new file mode 100644
> > > index 000000000000..1d1e143f98ec
> > > --- /dev/null
> > > +++ b/include/uapi/linux/cxl_mem.h
> > > @@ -0,0 +1,102 @@
> > > +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
> > > +/*
> > > + * CXL IOCTLs for Memory Devices
> > > + */
> > > +
> > > +#ifndef _UAPI_CXL_MEM_H_
> > > +#define _UAPI_CXL_MEM_H_
> > > +
> > > +#if defined(__cplusplus)
> > > +extern "C" {
> > > +#endif
> > > +
> > > +/**
> > > + * DOC: UAPI
> > > + *
> > > + * CXL memory devices expose UAPI to have a standard user interface.
> > > + * Userspace can refer to these structure definitions and UAPI formats
> > > + * to communicate to driver
> > > + */
> > > +
> > > +#define CXL_MEM_QUERY_COMMANDS _IOR('C', 1, struct cxl_mem_query_commands)
> > > +
> > > +#define CXL_MEM_COMMAND_NAME_LENGTH 32
> > > +
> > > +/**
> > > + * struct cxl_command_info - Command information returned from a query.
> > > + * @id: ID number for the command.
> > > + * @flags: Flags that specify command behavior.
> > > + *
> > > + *          - CXL_MEM_COMMAND_FLAG_TAINT: Using this command will taint the kernel.
> > > + * @size_in: Expected input size, or -1 if variable length.
> > > + * @size_out: Expected output size, or -1 if variable length.
> > > + * @name: Name describing the command.
> > > + *
> > > + * Represents a single command that is supported by both the driver and the
> > > + * hardware. The is returned as part of an array from the query ioctl. The
> > > + * following would be a command named "foobar" that takes a variable length
> > > + * input and returns 0 bytes of output.
> > > + *
> > > + *  - @id = 10
> > > + *  - @name = foobar
> > > + *  - @flags = 0
> > > + *  - @size_in = -1
> > > + *  - @size_out = 0
> > > + *
> > > + * See struct cxl_mem_query_commands.
> > > + */
> > > +struct cxl_command_info {
> > > +       __u32 id;
> > > +#define CXL_MEM_COMMAND_ID_INVALID 0
> > > +
> > > +       __u32 flags;
> > > +#define CXL_MEM_COMMAND_FLAG_NONE 0
> > > +#define CXL_MEM_COMMAND_FLAG_TAINT BIT(0)
> > > +
> > > +       __s32 size_in;
> > > +       __s32 size_out;
> > > +
> > > +       char name[32];
> >
> > Why does the name for a command need to be shuffled back and forth
> > over the ioctl interface. Can't this be handled by a static lookup
> > table defined in the header?
> >
>
> I was thinking of cases where the userspace application doesn't match the
> current kernel's UAPI and giving the driver flexibility to return whatever.

How / why would the application by looking at @name for UAPI compatibility?

> OTTOMH, I also can't think of a way to do this if you want to do define the
> table sparsely though. Do you have ideas for that?

I don't think the name lookup would be sparse. i.e. it would be ok for
mem_commands to not have an entry for everything in the name lookup
table. As for defining the table it could use C preprocessor trick
popularized by Steven Rostedt:

#define CMDS                                                     \
        C(CMD1, "command one"),     \
        C(CMD2, "command two")     \
#undef C
#define C(a, b) a
enum commands_enum { CMDS };
#undef C
#define C(a, b) { b }
static struct {
        const char *name;
} commands[] = { CMDS };
#undef C

...then there's no way for the command ids to get out of sync with the names.

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [RFC PATCH 09/14] cxl/mem: Add basic IOCTL interface
  2020-12-09  3:33       ` Dan Williams
@ 2020-12-09 16:23         ` Ben Widawsky
  0 siblings, 0 replies; 35+ messages in thread
From: Ben Widawsky @ 2020-12-09 16:23 UTC (permalink / raw)
  To: Dan Williams
  Cc: linux-cxl, Linux Kernel Mailing List, Linux PCI, Linux ACPI,
	Ira Weiny, Vishal Verma, Kelley, Sean V, Rafael Wysocki,
	Bjorn Helgaas, Jonathan Cameron, Jon Masters, Chris Browy,
	Randy Dunlap, Christoph Hellwig

On 20-12-08 19:33:13, Dan Williams wrote:
> On Tue, Dec 8, 2020 at 6:13 PM Ben Widawsky <ben.widawsky@intel.com> wrote:
> >
> > On 20-12-08 17:37:50, Dan Williams wrote:
> > > On Tue, Dec 8, 2020 at 4:24 PM Ben Widawsky <ben.widawsky@intel.com> wrote:
> > > >
> > > > Add a straightforward IOCTL that provides a mechanism for userspace to
> > > > query the supported memory device commands.
> > > >
> > > > Memory device commands are specified in 8.2.9 of the CXL 2.0
> > > > specification. They are submitted through a mailbox mechanism specified
> > > > in 8.2.8.4.
> > > >
> > > > Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
> > > >
> > > > ---
> > > >
> > > > I did attempt to use the same struct for querying commands as well as
> > > > sending commands (upcoming patch). The number of unused fields between
> > > > the two made for a bad fit IMO.
> > > >
> > > > Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
> > > > ---
> > > >  Documentation/cxl/memory-devices.rst |   9 +++
> > > >  drivers/cxl/mem.c                    |  89 +++++++++++++++++++++++
> > > >  include/uapi/linux/cxl_mem.h         | 102 +++++++++++++++++++++++++++
> > > >  3 files changed, 200 insertions(+)
> > > >  create mode 100644 include/uapi/linux/cxl_mem.h
> > > >
> > > > diff --git a/Documentation/cxl/memory-devices.rst b/Documentation/cxl/memory-devices.rst
> > > > index 5f723c25382b..ec54674b3822 100644
> > > > --- a/Documentation/cxl/memory-devices.rst
> > > > +++ b/Documentation/cxl/memory-devices.rst
> > > > @@ -32,6 +32,15 @@ CXL Memory Device
> > > >  .. kernel-doc:: drivers/cxl/mem.c
> > > >     :internal:
> > > >
> > > > +CXL IOCTL Interface
> > > > +-------------------
> > > > +
> > > > +.. kernel-doc:: include/uapi/linux/cxl_mem.h
> > > > +   :doc: UAPI
> > > > +
> > > > +.. kernel-doc:: include/uapi/linux/cxl_mem.h
> > > > +   :internal:
> > > > +
> > > >  External Interfaces
> > > >  ===================
> > > >
> > > > diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
> > > > index bb6ea58f6c7b..2c4aadcea0e4 100644
> > > > --- a/drivers/cxl/mem.c
> > > > +++ b/drivers/cxl/mem.c
> > > > @@ -7,6 +7,7 @@
> > > >  #include <linux/idr.h>
> > > >  #include <linux/pci.h>
> > > >  #include <linux/io.h>
> > > > +#include <uapi/linux/cxl_mem.h>
> > > >  #include "acpi.h"
> > > >  #include "pci.h"
> > > >  #include "cxl.h"
> > > > @@ -73,6 +74,49 @@ static DEFINE_IDR(cxl_mem_idr);
> > > >  /* protect cxl_mem_idr allocations */
> > > >  static DEFINE_MUTEX(cxl_memdev_lock);
> > > >
> > > > +/*
> > > > + * This table defines the supported mailboxes commands for the driver. The id is
> > > > + * ordinal and thus gaps in this table aren't allowed. This table is made up of
> > > > + * a UAPI structure. Non-negative values in the table will be validated against
> > > > + * the user's input. For example, if size_in is 0, and the user passed in 1, it
> > > > + * is an error.
> > > > + */
> > > > +#define CXL_CMD(_id, _flags, sin, sout, _name, _enable, op)                    \
> > > > +       {                                                                      \
> > > > +               { .id = CXL_MEM_COMMAND_ID_##_id,                              \
> > > > +                 .flags = CXL_MEM_COMMAND_FLAG_##_flags,                      \
> > > > +                 .size_in = sin,                                              \
> > > > +                 .size_out = sout,                                            \
> > > > +                 .name = _name },                                             \
> > > > +                       .enable = _enable, .opcode = op                        \
> > > > +       }
> > >
> > > Seems the ordinality requirement could be dropped if the definition was:
> > >
> > > #define CXL_CMD(_id, _flags, sin, sout, _name, _enable, op)                    \
> > >        [CXL_MEM_COMMAND_ID_##_id] = {
> > >                              \
> > >                { .id = CXL_MEM_COMMAND_ID_##_id,                              \
> > > ...
> > >
> > > Then command 0 and 42 could be defined out of order in the table.
> > > Especially if we need to config-disable or deprecate commands, I think
> > > it would be useful if this table was tolerant to being sparse.
> > >
> >
> > How sparse are we talking? The current form does support sparseness, but
> > obviously gets quite large if the ID numbering is similar to random
> > distribution.
> 
> "Sparse" may have been the wrong word to use. I was implying sparse
> enough that if I add command N+1 I don't need to be careful where I
> put it in mem_commands, but still be able to rely on lookups into
> mem_commands being indexed by the command-id.
> 

I'm not sure I understand the issue then. It's already demonstrated via the
first command being reserved - ie. already sparse.

CXL_CMD(INVALID, NONE, 0, 0, "Reserved", false, 0)

As long as the command doesn't have @enable set, it's effectively ignored for
all user interactions.

If you look at the last patch in the series, WIP, there is an example for
enabling one.


> > I think if we do see this being more like random distribution, it can be
> > supported, but I think it adds a decent amount of complexity for what I see as
> > not much reward - unless you know of a fairly simple way to create this data
> > structure with full sparse ID support?
> 
> I'm expecting the command distribution to be mostly uniform, it's more
> of the lookup property that I think would be useful especially for the
> dynamic case of walking mem_commands to update it relative to what the
> hardware supports or other metadata. Speaking of which I think @enable
> should be turned into @flags of which 'enable' is one, in case we want
> to define more flags in the future.
> 

I like the idea of moving enable to flags. I still don't see a reason to change
how it's defined today, can you give me an example where what is there won't
work?

> >
> > > > +
> > > > +/**
> > > > + * struct cxl_mem_command - Driver representation of a memory device command
> > > > + * @info: Command information as it exists for the UAPI
> > > > + * @opcode: The actual bits used for the mailbox protocol
> > > > + * @enable: Whether the command is enabled. The driver may support a large set
> > > > + *         of commands that may not be enabled. The primary reason a command
> > > > + *         would not be enabled is for commands that are specified as optional
> > > > + *         and the hardware doesn't support the command.
> > > > + *
> > > > + * The cxl_mem_command is the driver's internal representation of commands that
> > > > + * are supported by the driver. Some of these commands may not be supported by
> > > > + * the hardware (!@enable). The driver will use @info to validate the fields
> > > > + * passed in by the user then submit the @opcode to the hardware.
> > > > + *
> > > > + * See struct cxl_command_info.
> > > > + */
> > > > +struct cxl_mem_command {
> > > > +       const struct cxl_command_info info;
> > > > +       const u16 opcode;
> > > > +       bool enable;
> > > > +};
> > > > +
> > > > +static struct cxl_mem_command mem_commands[] = {
> > > > +       CXL_CMD(INVALID, NONE, 0, 0, "Reserved", false, 0),
> > > > +};
> > > > +
> > > >  static int cxl_mem_wait_for_doorbell(struct cxl_mem *cxlm)
> > > >  {
> > > >         const int timeout = msecs_to_jiffies(2000);
> > > > @@ -268,8 +312,53 @@ static int cxl_mem_open(struct inode *inode, struct file *file)
> > > >         return 0;
> > > >  }
> > > >
> > > > +static int cxl_mem_count_commands(void)
> > > > +{
> > > > +       int i, n = 0;
> > > > +
> > > > +       for (i = 0; i < ARRAY_SIZE(mem_commands); i++) {
> > > > +               struct cxl_mem_command *c = &mem_commands[i];
> > > > +
> > > > +               if (c->enable)
> > > > +                       n++;
> > > > +       }
> > > > +
> > > > +       return n;
> > > > +}
> > > > +
> > > >  static long cxl_mem_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
> > > >  {
> > > > +       if (cmd == CXL_MEM_QUERY_COMMANDS) {
> > > > +               struct cxl_mem_query_commands __user *q = (void __user *)arg;
> > > > +               u32 n_commands;
> > > > +               int i, j;
> > > > +
> > > > +               if (get_user(n_commands, (u32 __user *)arg))
> > > > +                       return -EFAULT;
> > > > +
> > > > +               if (n_commands == 0)
> > > > +                       return put_user(cxl_mem_count_commands(),
> > > > +                                       (u32 __user *)arg);
> > > > +
> > > > +               for (i = 0, j = 0;
> > > > +                    i < ARRAY_SIZE(mem_commands) && j < n_commands; i++) {
> > > > +                       struct cxl_mem_command *c = &mem_commands[i];
> > > > +                       const struct cxl_command_info *info = &c->info;
> > > > +
> > > > +                       if (!c->enable)
> > > > +                               continue;
> > > > +
> > > > +                       if (copy_to_user(&q->commands[j], info, sizeof(*info)))
> > > > +                               return -EFAULT;
> > > > +
> > > > +                       if (copy_to_user(&q->commands[j].name, info->name,
> > > > +                                        strlen(info->name)))
> > > > +                               return -EFAULT;
> > >
> > > Not sure why this is needed, see comment below about @name in
> > > cxl_mem_query_commands.
> > >
> > > > +
> > > > +                       j++;
> > > > +               }
> > > > +       }
> > > > +
> > > >         return -ENOTTY;
> > > >  }
> > > >
> > > > diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
> > > > new file mode 100644
> > > > index 000000000000..1d1e143f98ec
> > > > --- /dev/null
> > > > +++ b/include/uapi/linux/cxl_mem.h
> > > > @@ -0,0 +1,102 @@
> > > > +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
> > > > +/*
> > > > + * CXL IOCTLs for Memory Devices
> > > > + */
> > > > +
> > > > +#ifndef _UAPI_CXL_MEM_H_
> > > > +#define _UAPI_CXL_MEM_H_
> > > > +
> > > > +#if defined(__cplusplus)
> > > > +extern "C" {
> > > > +#endif
> > > > +
> > > > +/**
> > > > + * DOC: UAPI
> > > > + *
> > > > + * CXL memory devices expose UAPI to have a standard user interface.
> > > > + * Userspace can refer to these structure definitions and UAPI formats
> > > > + * to communicate to driver
> > > > + */
> > > > +
> > > > +#define CXL_MEM_QUERY_COMMANDS _IOR('C', 1, struct cxl_mem_query_commands)
> > > > +
> > > > +#define CXL_MEM_COMMAND_NAME_LENGTH 32
> > > > +
> > > > +/**
> > > > + * struct cxl_command_info - Command information returned from a query.
> > > > + * @id: ID number for the command.
> > > > + * @flags: Flags that specify command behavior.
> > > > + *
> > > > + *          - CXL_MEM_COMMAND_FLAG_TAINT: Using this command will taint the kernel.
> > > > + * @size_in: Expected input size, or -1 if variable length.
> > > > + * @size_out: Expected output size, or -1 if variable length.
> > > > + * @name: Name describing the command.
> > > > + *
> > > > + * Represents a single command that is supported by both the driver and the
> > > > + * hardware. The is returned as part of an array from the query ioctl. The
> > > > + * following would be a command named "foobar" that takes a variable length
> > > > + * input and returns 0 bytes of output.
> > > > + *
> > > > + *  - @id = 10
> > > > + *  - @name = foobar
> > > > + *  - @flags = 0
> > > > + *  - @size_in = -1
> > > > + *  - @size_out = 0
> > > > + *
> > > > + * See struct cxl_mem_query_commands.
> > > > + */
> > > > +struct cxl_command_info {
> > > > +       __u32 id;
> > > > +#define CXL_MEM_COMMAND_ID_INVALID 0
> > > > +
> > > > +       __u32 flags;
> > > > +#define CXL_MEM_COMMAND_FLAG_NONE 0
> > > > +#define CXL_MEM_COMMAND_FLAG_TAINT BIT(0)
> > > > +
> > > > +       __s32 size_in;
> > > > +       __s32 size_out;
> > > > +
> > > > +       char name[32];
> > >
> > > Why does the name for a command need to be shuffled back and forth
> > > over the ioctl interface. Can't this be handled by a static lookup
> > > table defined in the header?
> > >
> >
> > I was thinking of cases where the userspace application doesn't match the
> > current kernel's UAPI and giving the driver flexibility to return whatever.
> 
> How / why would the application by looking at @name for UAPI compatibility?
> 
> > OTTOMH, I also can't think of a way to do this if you want to do define the
> > table sparsely though. Do you have ideas for that?
> 
> I don't think the name lookup would be sparse. i.e. it would be ok for
> mem_commands to not have an entry for everything in the name lookup
> table. As for defining the table it could use C preprocessor trick
> popularized by Steven Rostedt:
> 
> #define CMDS                                                     \
>         C(CMD1, "command one"),     \
>         C(CMD2, "command two")     \
> #undef C
> #define C(a, b) a
> enum commands_enum { CMDS };
> #undef C
> #define C(a, b) { b }
> static struct {
>         const char *name;
> } commands[] = { CMDS };
> #undef C
> 
> ...then there's no way for the command ids to get out of sync with the names.

I will move it to the header and drop name[32] from UAPI. My personal preference
is to have the driver fill in the field, but I have no objective reason for
that.

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [RFC PATCH 10/14] cxl/mem: Add send command
  2020-12-09  0:24 ` [RFC PATCH 10/14] cxl/mem: Add send command Ben Widawsky
@ 2020-12-09 22:06   ` Dan Williams
  2020-12-15 21:43     ` Ben Widawsky
  0 siblings, 1 reply; 35+ messages in thread
From: Dan Williams @ 2020-12-09 22:06 UTC (permalink / raw)
  To: Ben Widawsky
  Cc: linux-cxl, Linux Kernel Mailing List, Linux PCI, Linux ACPI,
	Ira Weiny, Vishal Verma, Kelley, Sean V, Rafael Wysocki,
	Bjorn Helgaas, Jonathan Cameron, Jon Masters, Chris Browy,
	Randy Dunlap, Christoph Hellwig

On Tue, Dec 8, 2020 at 4:24 PM Ben Widawsky <ben.widawsky@intel.com> wrote:
>
> The send command allows userspace to issue mailbox commands directly to
> the hardware. The driver will verify basic properties of the command but
> otherwise pass any input payload untouched to the hardware, and return
> the output payload to userspace.
>
> The caller of this IOCTL is required to allocate enough space for
> max(size_in, size_out) of the payload. The payload input data will be
> wiped out if any output payload exists.

Seems awkward for the kernel to overwrite input payloads. I can see
that potentially complicating userspace if it has the same payload to
send to multiple devices. What's the rationale for overwriting input?

>
> Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
> ---
>  drivers/cxl/mem.c            | 127 +++++++++++++++++++++++++++++++++++
>  include/uapi/linux/cxl_mem.h |  35 ++++++++++
>  2 files changed, 162 insertions(+)
>
> diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
> index 2c4aadcea0e4..0bf03afc0c80 100644
> --- a/drivers/cxl/mem.c
> +++ b/drivers/cxl/mem.c
> @@ -324,6 +324,120 @@ static int cxl_mem_count_commands(void)
>         }
>
>         return n;
> +};
> +
> +/**
> + * handle_mailbox_cmd_from_user() - Dispatch a mailbox command.
> + * @cxlmd: The CXL memory device to communicate with.
> + * @cmd: The validated command
> + * @u: The command submitted by userspace. Only useful for RAW commands.
> + *
> + * Return: 0 on success.
> + *
> + * This function packages up a &struct mbox_cmd on behalf of userspace,
> + * dispatches the command, and returns the results.
> + */
> +static int handle_mailbox_cmd_from_user(struct cxl_memdev *cxlmd,
> +                                       const struct cxl_mem_command *cmd,
> +                                       struct cxl_send_command __user *u)
> +{
> +       struct mbox_cmd mbox_cmd;
> +       ssize_t payload_size;
> +       void *payload;
> +       u32 size_out;
> +       int rc;
> +
> +       if (get_user(size_out, &u->size_out))
> +               return -EFAULT;
> +
> +       payload_size = max_t(ssize_t, cmd->info.size_in, size_out);
> +       if (payload_size) {
> +               payload =
> +                       memdup_user(u64_to_user_ptr(u->payload), payload_size);

Me thinks this should be vmemdup_user() for payloads that exceed the
kmalloc() max, and I think it would be worthwhile to clamp @size_out
to some maximum and not let userspace ask for gigantic payloads.
Return EINVAL for payloads greater than... 4MB? At least 4MB is the
arbitrary max that libnvdimm picked.

> +               if (IS_ERR(payload))
> +                       return PTR_ERR(payload);
> +       }
> +
> +       rc = cxl_mem_mbox_get(cxlmd->cxlm);
> +       if (rc)
> +               return rc;
> +
> +       mbox_cmd = (struct mbox_cmd){

I'd expect a space between: ){ ...looks like clang-format does not
have that rule.

> +               .opcode = cmd->opcode,
> +               .payload = payload,
> +               .size_in = cmd->info.size_in,
> +               .size_out = size_out,
> +       };
> +       rc = cxl_mem_mbox_send_cmd(cxlmd->cxlm, &mbox_cmd);
> +       cxl_mem_mbox_put(cxlmd->cxlm);
> +       if (rc)
> +               goto out;
> +
> +       rc = put_user(mbox_cmd.return_code, &u->retval);
> +       if (rc)
> +               goto out;
> +
> +       rc = put_user(mbox_cmd.size_out, &u->size_out);
> +       if (rc)
> +               goto out;
> +
> +       if (mbox_cmd.size_out)
> +               if (copy_to_user(u64_to_user_ptr(u->payload), payload,
> +                                mbox_cmd.size_out))
> +                       rc = -EFAULT;
> +
> +out:
> +       if (payload_size)
> +               kfree(payload);
> +       return rc;
> +}
> +
> +/**
> + * cxl_validate_cmd_from_user() - Check fields for CXL_MEM_SEND_COMMAND.
> + * @user_cmd: &struct cxl_send_command from userspace.

Ah cool, I did not realize kernel doc recognized type specifiers like
that, nice.

> + * @out_cmd: Sanitized and populared &struct cxl_mem_command.

s/populared/populated/

> + *
> + * Return:
> + *  * %0       - Command dispatched successfully.
> + *  * %-EFAULT - Something happened with copy_to/from_user.
> + *  * %-EINVAL - Rerserved fields were used.

s/Rerserved/Reserved/

> + *  * %-EPERM  - Protected command used by the RAW interface.
> + *  * %-ENOMEM - Input or output buffer wasn't large enough.
> + *
> + */
> +static int cxl_validate_cmd_from_user(struct cxl_send_command __user *user_cmd,
> +                                     struct cxl_mem_command *out_cmd)
> +{
> +       const struct cxl_command_info *info;
> +       struct cxl_send_command cmd;
> +       struct cxl_mem_command *c;
> +
> +       if (copy_from_user(&cmd, user_cmd, sizeof(cmd)))
> +               return -EFAULT;
> +
> +       if (cmd.id == 0 || cmd.id >= CXL_MEM_COMMAND_ID_MAX)
> +               return -EINVAL;

I wonder if the "cmd.id >= CXL_MEM_COMMAND_ID_MAX" case should return
-ENOTTY. The command might be perfectly valid, just the kernel does
not have that command specified which would be the case with older
kernel with newer userspace.

> +
> +       c = &mem_commands[cmd.id];
> +       info = &c->info;
> +
> +       if (cmd.flags & CXL_MEM_COMMAND_FLAG_MASK)
> +               return -EINVAL;
> +
> +       if (cmd.rsvd)
> +               return -EINVAL;
> +
> +       /* Check the input buffer is the expected size */
> +       if (info->size_in >= 0 && info->size_in != cmd.size_in)
> +               return -ENOMEM;
> +
> +       /* Check the output buffer is at least large enough */
> +       if (info->size_out >= 0 && cmd.size_out < info->size_out)
> +               return -ENOMEM;
> +
> +       memcpy(out_cmd, c, sizeof(*c));

Why not do the copy_from_user() directly into out_cmd to save a copy operation?

> +
> +       return 0;
>  }
>
>  static long cxl_mem_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
> @@ -357,6 +471,19 @@ static long cxl_mem_ioctl(struct file *file, unsigned int cmd, unsigned long arg
>
>                         j++;
>                 }
> +
> +               return 0;
> +       } else if (cmd == CXL_MEM_SEND_COMMAND) {
> +               struct cxl_send_command __user *u = (void __user *)arg;
> +               struct cxl_memdev *cxlmd = file->private_data;
> +               struct cxl_mem_command c;
> +               int rc;
> +
> +               rc = cxl_validate_cmd_from_user(u, &c);
> +               if (rc)
> +                       return rc;
> +
> +               return handle_mailbox_cmd_from_user(cxlmd, &c, u);
>         }
>
>         return -ENOTTY;
> diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
> index 1d1e143f98ec..189d86a13637 100644
> --- a/include/uapi/linux/cxl_mem.h
> +++ b/include/uapi/linux/cxl_mem.h
> @@ -19,6 +19,7 @@ extern "C" {
>   */
>
>  #define CXL_MEM_QUERY_COMMANDS _IOR('C', 1, struct cxl_mem_query_commands)
> +#define CXL_MEM_SEND_COMMAND _IOWR('C', 2, struct cxl_send_command)
>
>  #define CXL_MEM_COMMAND_NAME_LENGTH 32
>
> @@ -48,10 +49,12 @@ extern "C" {
>  struct cxl_command_info {
>         __u32 id;
>  #define CXL_MEM_COMMAND_ID_INVALID 0
> +#define CXL_MEM_COMMAND_ID_MAX (CXL_MEM_COMMAND_ID_INVALID + 1)
>
>         __u32 flags;
>  #define CXL_MEM_COMMAND_FLAG_NONE 0
>  #define CXL_MEM_COMMAND_FLAG_TAINT BIT(0)

I guess I missed this in the last patch, why would userspace specify a
taint flag? @flags is an output parameter?

> +#define CXL_MEM_COMMAND_FLAG_MASK ~BIT(0)
>
>         __s32 size_in;
>         __s32 size_out;
> @@ -95,6 +98,38 @@ struct cxl_mem_query_commands {
>         struct cxl_command_info __user commands[]; /* out: supported commands */
>  };
>
> +/**
> + * struct cxl_send_command - Send a command to a memory device.
> + * @id: The command to send to the memory device. This must be one of the
> + *     commands returned by the query command.
> + * @flags: Flags for the command

in / out field?

> + * @rsvd: Reserved for future use.

Document 'must be zero' to match the code?

> + * @retval: Return value from the memory device (output).

Is this a translated error code or the raw
> + * @size_in: Size of the payload to provide to the device (input).
> + * @size_out: Size of the payload received from the device (input/output). This
> + *           field is filled in my userspace to let the driver know how much

s/my/by/

> + *           space was allocated for output. It is populated by the driver to
> + *           let userspace know how large the output payload actually was.
> + * @payload: Pointer to memory available for payload input/output.
> + *
> + * Mechanism for userspace to send a command to the hardware for processing. The
> + * driver will do basic validation on the command sizes, but the payload input
> + * and output are not introspected. Userspace is required to allocate large

They might be introspected, at least there were some NVDIMM ioctls
like "clear error" where the driver in addition to telling the device
to clear the error also cleaned up the kernel's error tracking list.
I'd say one of the benefits of Linux-commands that get translated to
CXL-specific payloads is that they allow for kernel translation /
interpretation / quirking vs 'raw' where the kernel does not care.

> + * enough buffers for max(size_in, size_out).
> + */
> +struct cxl_send_command {
> +       __u32 id;
> +       __u32 flags;
> +       __u32 rsvd;
> +       __u32 retval;
> +
> +       struct {
> +               __s32 size_in;
> +               __s32 size_out;
> +               __u64 payload;

per the initial comment above a separate @payload_in and a
@payload_out pointer would seem to be more flexible and get out of the
situation where the kernel needs to clobber inputs.

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [RFC PATCH 11/14] cxl/mem: Add a "RAW" send command
  2020-12-09  0:24 ` [RFC PATCH 11/14] cxl/mem: Add a "RAW" " Ben Widawsky
@ 2020-12-09 22:38   ` Dan Williams
  2020-12-16 20:42     ` Ben Widawsky
  0 siblings, 1 reply; 35+ messages in thread
From: Dan Williams @ 2020-12-09 22:38 UTC (permalink / raw)
  To: Ben Widawsky
  Cc: linux-cxl, Linux Kernel Mailing List, Linux PCI, Linux ACPI,
	Ira Weiny, Vishal Verma, Kelley, Sean V, Rafael Wysocki,
	Bjorn Helgaas, Jonathan Cameron, Jon Masters, Chris Browy,
	Randy Dunlap, Christoph Hellwig

On Tue, Dec 8, 2020 at 4:24 PM Ben Widawsky <ben.widawsky@intel.com> wrote:
>
> The CXL memory device send interface will have a number of supported
> commands. The raw command is not such a command. Raw commands allow
> userspace to send a specified opcode to the underlying hardware and
> bypass all driver checks on the command. This is useful for a couple of
> usecases, mainly:
> 1. Undocumented vendor specific hardware commands
> 2. Prototyping new hardware commands not yet supported by the driver
>
> While this all sounds very powerful it comes with a couple of caveats:
> 1. Bug reports using raw commands will not get the same level of
>    attention as bug reports using supported commands (via taint).
> 2. Supported commands will be rejected by the RAW command.
>
> Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
> ---
>  drivers/cxl/mem.c            | 32 ++++++++++++++++++++++++++++++++
>  include/uapi/linux/cxl_mem.h | 14 ++++++++++++--
>  2 files changed, 44 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
> index 0bf03afc0c80..a2cea7ac7cc6 100644
> --- a/drivers/cxl/mem.c
> +++ b/drivers/cxl/mem.c
> @@ -115,6 +115,7 @@ struct cxl_mem_command {
>
>  static struct cxl_mem_command mem_commands[] = {
>         CXL_CMD(INVALID, NONE, 0, 0, "Reserved", false, 0),
> +       CXL_CMD(RAW, TAINT, ~0, ~0, "Raw", true, 0),

Why is the taint indication in the ABI? It seems like it only needs to
be documented.

>  };
>
>  static int cxl_mem_wait_for_doorbell(struct cxl_mem *cxlm)
> @@ -326,6 +327,20 @@ static int cxl_mem_count_commands(void)
>         return n;
>  };
>
> +static struct cxl_mem_command *cxl_mem_find_command(u16 opcode)
> +{
> +       int i;
> +
> +       for (i = 0; i < ARRAY_SIZE(mem_commands); i++) {
> +               struct cxl_mem_command *c = &mem_commands[i];
> +
> +               if (c->opcode == opcode)
> +                       return c;
> +       }
> +
> +       return NULL;
> +};
> +
>  /**
>   * handle_mailbox_cmd_from_user() - Dispatch a mailbox command.
>   * @cxlmd: The CXL memory device to communicate with.
> @@ -421,6 +436,23 @@ static int cxl_validate_cmd_from_user(struct cxl_send_command __user *user_cmd,
>         c = &mem_commands[cmd.id];
>         info = &c->info;
>
> +       /* Checks are bypassed for raw commands but along comes the taint! */
> +       if (cmd.id == CXL_MEM_COMMAND_ID_RAW) {
> +               struct cxl_mem_command temp =
> +                       CXL_CMD(RAW, NONE, cmd.size_in, cmd.size_out, "Raw",
> +                               true, cmd.raw.opcode);

Oh, I thought CXL_CMD() was only used to populate the mem_commands
array. Feels out of place to use it here when all it is doing is
updating the size_{in,out} and opcode fields. Mainly I'm interested in
CXL_CMD() enforcing that the command-id is the mem_commands index.

> +
> +               if (cmd.raw.rsvd)
> +                       return -EINVAL;
> +
> +               if (cxl_mem_find_command(cmd.raw.opcode))
> +                       return -EPERM;
> +
> +               add_taint(TAINT_WARN, LOCKDEP_STILL_OK);

TAINT_WARN seems the wrong value, especially since no WARN has
occurred. I feel that this is more in the spirit of
TAINT_PROPRIETARY_MODULE, TAINT_OVERRIDDEN_ACPI_TABLE, and
TAINT_OOT_MODULE. How about a new TAINT_RAW_PASSTHROUGH? I could use
this for the acpi/nfit driver as well to disclaim responsibility for
system errors that can result from not using the nominal
kernel-provided commands.

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [RFC PATCH 14/14] WIP/cxl/mem: Add get firmware for testing
  2020-12-09  0:24 ` [RFC PATCH 14/14] WIP/cxl/mem: Add get firmware for testing Ben Widawsky
@ 2020-12-10  0:42   ` Dan Williams
  0 siblings, 0 replies; 35+ messages in thread
From: Dan Williams @ 2020-12-10  0:42 UTC (permalink / raw)
  To: Ben Widawsky
  Cc: linux-cxl, Linux Kernel Mailing List, Linux PCI, Linux ACPI,
	Ira Weiny, Vishal Verma, Kelley, Sean V, Rafael Wysocki,
	Bjorn Helgaas, Jonathan Cameron, Jon Masters, Chris Browy,
	Randy Dunlap, Christoph Hellwig

On Tue, Dec 8, 2020 at 4:25 PM Ben Widawsky <ben.widawsky@intel.com> wrote:
>
> This also serves as an example how to add a new command
>
> Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
> ---
>  drivers/cxl/mem.c            | 22 ++++++++++++++++++++++
>  include/uapi/linux/cxl_mem.h |  3 ++-
>  2 files changed, 24 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
> index 6b2f8d3776b5..76aa1e6e4117 100644
> --- a/drivers/cxl/mem.c
> +++ b/drivers/cxl/mem.c
> @@ -116,6 +116,7 @@ struct cxl_mem_command {
>  static struct cxl_mem_command mem_commands[] = {
>         CXL_CMD(INVALID, NONE, 0, 0, "Reserved", false, 0),
>         CXL_CMD(RAW, TAINT, ~0, ~0, "Raw", true, 0),
> +       CXL_CMD(GET_FW_INFO, NONE, 0, 0x50, "Get FW Info", false, 0x0200),

I think CXL_CMD() arguments can be put on a diet if the
mem-command-id-to-name was moved to its own table, and the opcode was
defined as something like:

#define CXL_MBOX_OP_GET_FW_INFO 0x200

...so that CXL_CMD can just do:

    .opcode = CXL_MBOX_OP_##_id

That @_enable arg wants a flag #define like CMD_ENABLE which reads
better than 'true'. I would then add CMD_KERNEL_EXCL alongside that
flag to indicate the commands that may be exclusive to the kernel when
the device is active in a PMEM memory region, or ones that have an
alternate ABI wrapped around them like the keys subsystem for security
or the firwmare activation sysfs interface.

>  };
>
>  static int cxl_mem_wait_for_doorbell(struct cxl_mem *cxlm)
> @@ -827,6 +828,23 @@ static int cxl_mem_add_memdev(struct cxl_mem *cxlm)
>         return cxl_register(dev);
>  }
>
> +static int cxl_mem_enable_commands(struct cxl_mem *cxlm)
> +{
> +       struct cxl_mem_command *c;
> +
> +       /*
> +        * For now we pretend Get FW info is supported.
> +        *
> +        * FIXME: Invoke GET LOG to get the Command Effect Logs (CEL).
> +        */
> +       c = cxl_mem_find_command(0x200);
> +       if (!c)
> +               return -ENOENT;
> +
> +       c->enable = true;
> +       return 0;
> +}
> +
>  /**
>   * cxl_mem_identify() - Send the IDENTIFY command to the device.
>   * @cxlm: The device to identify.
> @@ -936,6 +954,10 @@ static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
>         if (rc)
>                 return rc;
>
> +       rc = cxl_mem_enable_commands(cxlm);
> +       if (rc)
> +               return rc;
> +
>         rc = cxl_mem_identify(cxlm);
>         if (rc)
>                 return rc;
> diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
> index f2fbb0dcda06..3ac39acf8fa7 100644
> --- a/include/uapi/linux/cxl_mem.h
> +++ b/include/uapi/linux/cxl_mem.h
> @@ -50,7 +50,8 @@ struct cxl_command_info {
>         __u32 id;
>  #define CXL_MEM_COMMAND_ID_INVALID 0
>  #define CXL_MEM_COMMAND_ID_RAW 1
> -#define CXL_MEM_COMMAND_ID_MAX (CXL_MEM_COMMAND_ID_RAW + 1)
> +#define CXL_MEM_COMMAND_ID_GET_FW_INFO 2
> +#define CXL_MEM_COMMAND_ID_MAX (CXL_MEM_COMMAND_ID_GET_FW_INFO + 1)

Seems like CXL_MEM_COMMAND_ID definitions want to be an enum so that
CXL_MEM_COMMAND_ID_MAX does not need to be touched every time.

>
>         __u32 flags;
>  #define CXL_MEM_COMMAND_FLAG_NONE 0
> --
> 2.29.2
>

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [RFC PATCH 09/14] cxl/mem: Add basic IOCTL interface
  2020-12-09  0:24 ` [RFC PATCH 09/14] cxl/mem: Add basic IOCTL interface Ben Widawsky
  2020-12-09  1:37   ` Dan Williams
@ 2020-12-10  3:32   ` Randy Dunlap
  2020-12-14 17:29     ` Ben Widawsky
  1 sibling, 1 reply; 35+ messages in thread
From: Randy Dunlap @ 2020-12-10  3:32 UTC (permalink / raw)
  To: Ben Widawsky, linux-cxl
  Cc: linux-kernel, linux-pci, linux-acpi, Ira Weiny, Dan Williams,
	Vishal Verma, Kelley, Sean V, Rafael Wysocki, Bjorn Helgaas,
	Jonathan Cameron, Jon Masters, Chris Browy, Christoph Hellwig

On 12/8/20 4:24 PM, Ben Widawsky wrote:
> +
> +#define CXL_MEM_QUERY_COMMANDS _IOR('C', 1, struct cxl_mem_query_commands)

Hi,
I could have missed it, but IOCTL major "numbers" (like 'C') should be
listed in Documentation/userspace-api/ioctl/ioctl-number.rst.


thanks.
-- 
~Randy


^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [RFC PATCH 09/14] cxl/mem: Add basic IOCTL interface
  2020-12-10  3:32   ` Randy Dunlap
@ 2020-12-14 17:29     ` Ben Widawsky
  0 siblings, 0 replies; 35+ messages in thread
From: Ben Widawsky @ 2020-12-14 17:29 UTC (permalink / raw)
  To: Randy Dunlap
  Cc: linux-cxl, linux-kernel, linux-pci, linux-acpi, Ira Weiny,
	Dan Williams, Vishal Verma, Kelley, Sean V, Rafael Wysocki,
	Bjorn Helgaas, Jonathan Cameron, Jon Masters, Chris Browy,
	Christoph Hellwig

On 20-12-09 19:32:19, Randy Dunlap wrote:
> On 12/8/20 4:24 PM, Ben Widawsky wrote:
> > +
> > +#define CXL_MEM_QUERY_COMMANDS _IOR('C', 1, struct cxl_mem_query_commands)
> 
> Hi,
> I could have missed it, but IOCTL major "numbers" (like 'C') should be
> listed in Documentation/userspace-api/ioctl/ioctl-number.rst.
> 
> 
> thanks.

I was unaware of this. Fixing for v3 without conflict.

Thanks.
Ben

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [RFC PATCH 10/14] cxl/mem: Add send command
  2020-12-09 22:06   ` Dan Williams
@ 2020-12-15 21:43     ` Ben Widawsky
  2020-12-15 22:03       ` Dan Williams
  0 siblings, 1 reply; 35+ messages in thread
From: Ben Widawsky @ 2020-12-15 21:43 UTC (permalink / raw)
  To: Dan Williams
  Cc: linux-cxl, Linux Kernel Mailing List, Linux PCI, Linux ACPI,
	Ira Weiny, Vishal Verma, Kelley, Sean V, Rafael Wysocki,
	Bjorn Helgaas, Jonathan Cameron, Jon Masters, Chris Browy,
	Randy Dunlap, Christoph Hellwig

On 20-12-09 14:06:22, Dan Williams wrote:
> On Tue, Dec 8, 2020 at 4:24 PM Ben Widawsky <ben.widawsky@intel.com> wrote:
> >
> > The send command allows userspace to issue mailbox commands directly to
> > the hardware. The driver will verify basic properties of the command but
> > otherwise pass any input payload untouched to the hardware, and return
> > the output payload to userspace.
> >
> > The caller of this IOCTL is required to allocate enough space for
> > max(size_in, size_out) of the payload. The payload input data will be
> > wiped out if any output payload exists.
> 
> Seems awkward for the kernel to overwrite input payloads. I can see
> that potentially complicating userspace if it has the same payload to
> send to multiple devices. What's the rationale for overwriting input?
> 

I couldn't think of a case where preserving the input payload would be beneficial
and I actually think it's a bit simpler with just one payload for both sides -
however, since you came up with a good usage, I will modify this.

> >
> > Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
> > ---
> >  drivers/cxl/mem.c            | 127 +++++++++++++++++++++++++++++++++++
> >  include/uapi/linux/cxl_mem.h |  35 ++++++++++
> >  2 files changed, 162 insertions(+)
> >
> > diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
> > index 2c4aadcea0e4..0bf03afc0c80 100644
> > --- a/drivers/cxl/mem.c
> > +++ b/drivers/cxl/mem.c
> > @@ -324,6 +324,120 @@ static int cxl_mem_count_commands(void)
> >         }
> >
> >         return n;
> > +};
> > +
> > +/**
> > + * handle_mailbox_cmd_from_user() - Dispatch a mailbox command.
> > + * @cxlmd: The CXL memory device to communicate with.
> > + * @cmd: The validated command
> > + * @u: The command submitted by userspace. Only useful for RAW commands.
> > + *
> > + * Return: 0 on success.
> > + *
> > + * This function packages up a &struct mbox_cmd on behalf of userspace,
> > + * dispatches the command, and returns the results.
> > + */
> > +static int handle_mailbox_cmd_from_user(struct cxl_memdev *cxlmd,
> > +                                       const struct cxl_mem_command *cmd,
> > +                                       struct cxl_send_command __user *u)
> > +{
> > +       struct mbox_cmd mbox_cmd;
> > +       ssize_t payload_size;
> > +       void *payload;
> > +       u32 size_out;
> > +       int rc;
> > +
> > +       if (get_user(size_out, &u->size_out))
> > +               return -EFAULT;
> > +
> > +       payload_size = max_t(ssize_t, cmd->info.size_in, size_out);
> > +       if (payload_size) {
> > +               payload =
> > +                       memdup_user(u64_to_user_ptr(u->payload), payload_size);
> 
> Me thinks this should be vmemdup_user() for payloads that exceed the
> kmalloc() max, and I think it would be worthwhile to clamp @size_out
> to some maximum and not let userspace ask for gigantic payloads.
> Return EINVAL for payloads greater than... 4MB? At least 4MB is the
> arbitrary max that libnvdimm picked.
> 

This is handled in cxl_validate_cmd_from_user() currently. The current API
allows the user to specify as large as they like with @size_out but the kernel
only writes out as much as the hardware returns.... More on this below [1].

> > +               if (IS_ERR(payload))
> > +                       return PTR_ERR(payload);
> > +       }
> > +
> > +       rc = cxl_mem_mbox_get(cxlmd->cxlm);
> > +       if (rc)
> > +               return rc;
> > +
> > +       mbox_cmd = (struct mbox_cmd){
> 
> I'd expect a space between: ){ ...looks like clang-format does not
> have that rule.
> 
> > +               .opcode = cmd->opcode,
> > +               .payload = payload,
> > +               .size_in = cmd->info.size_in,
> > +               .size_out = size_out,
> > +       };
> > +       rc = cxl_mem_mbox_send_cmd(cxlmd->cxlm, &mbox_cmd);
> > +       cxl_mem_mbox_put(cxlmd->cxlm);
> > +       if (rc)
> > +               goto out;
> > +
> > +       rc = put_user(mbox_cmd.return_code, &u->retval);
> > +       if (rc)
> > +               goto out;
> > +
> > +       rc = put_user(mbox_cmd.size_out, &u->size_out);
> > +       if (rc)
> > +               goto out;
> > +
> > +       if (mbox_cmd.size_out)
> > +               if (copy_to_user(u64_to_user_ptr(u->payload), payload,
> > +                                mbox_cmd.size_out))
> > +                       rc = -EFAULT;
> > +
> > +out:
> > +       if (payload_size)
> > +               kfree(payload);
> > +       return rc;
> > +}
> > +
> > +/**
> > + * cxl_validate_cmd_from_user() - Check fields for CXL_MEM_SEND_COMMAND.
> > + * @user_cmd: &struct cxl_send_command from userspace.
> 
> Ah cool, I did not realize kernel doc recognized type specifiers like
> that, nice.
> 
> > + * @out_cmd: Sanitized and populared &struct cxl_mem_command.
> 
> s/populared/populated/
> 
> > + *
> > + * Return:
> > + *  * %0       - Command dispatched successfully.
> > + *  * %-EFAULT - Something happened with copy_to/from_user.
> > + *  * %-EINVAL - Rerserved fields were used.
> 
> s/Rerserved/Reserved/
> 
> > + *  * %-EPERM  - Protected command used by the RAW interface.
> > + *  * %-ENOMEM - Input or output buffer wasn't large enough.
> > + *
> > + */
> > +static int cxl_validate_cmd_from_user(struct cxl_send_command __user *user_cmd,
> > +                                     struct cxl_mem_command *out_cmd)
> > +{
> > +       const struct cxl_command_info *info;
> > +       struct cxl_send_command cmd;
> > +       struct cxl_mem_command *c;
> > +
> > +       if (copy_from_user(&cmd, user_cmd, sizeof(cmd)))
> > +               return -EFAULT;
> > +
> > +       if (cmd.id == 0 || cmd.id >= CXL_MEM_COMMAND_ID_MAX)
> > +               return -EINVAL;
> 
> I wonder if the "cmd.id >= CXL_MEM_COMMAND_ID_MAX" case should return
> -ENOTTY. The command might be perfectly valid, just the kernel does
> not have that command specified which would be the case with older
> kernel with newer userspace.
> 

I think ENOTTY could be a bit confusing here as it could be confused for the
SEND ioctl not being present.

What about ENOENT, or something else?

> > +
> > +       c = &mem_commands[cmd.id];
> > +       info = &c->info;
> > +
> > +       if (cmd.flags & CXL_MEM_COMMAND_FLAG_MASK)
> > +               return -EINVAL;
> > +
> > +       if (cmd.rsvd)
> > +               return -EINVAL;
> > +
> > +       /* Check the input buffer is the expected size */
> > +       if (info->size_in >= 0 && info->size_in != cmd.size_in)
> > +               return -ENOMEM;
> > +
> > +       /* Check the output buffer is at least large enough */
> > +       if (info->size_out >= 0 && cmd.size_out < info->size_out)
> > +               return -ENOMEM;
> > +
> > +       memcpy(out_cmd, c, sizeof(*c));
> 
> Why not do the copy_from_user() directly into out_cmd to save a copy operation?
> 

[1]
cxl_validate_cmd_from_user() essentially translates a user's command into the
internal kernel representation of the command via the lookup into the array.
This will do things like adjust the output size or flags, to prevent userspace
from doing nefarious things. An actual command, especially now that you had me
remove the name[32] will be relatively small and I see the extra copy as being
well worth being able to isolate this command sanitization.

> > +
> > +       return 0;
> >  }
> >
> >  static long cxl_mem_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
> > @@ -357,6 +471,19 @@ static long cxl_mem_ioctl(struct file *file, unsigned int cmd, unsigned long arg
> >
> >                         j++;
> >                 }
> > +
> > +               return 0;
> > +       } else if (cmd == CXL_MEM_SEND_COMMAND) {
> > +               struct cxl_send_command __user *u = (void __user *)arg;
> > +               struct cxl_memdev *cxlmd = file->private_data;
> > +               struct cxl_mem_command c;
> > +               int rc;
> > +
> > +               rc = cxl_validate_cmd_from_user(u, &c);
> > +               if (rc)
> > +                       return rc;
> > +
> > +               return handle_mailbox_cmd_from_user(cxlmd, &c, u);
> >         }
> >
> >         return -ENOTTY;
> > diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
> > index 1d1e143f98ec..189d86a13637 100644
> > --- a/include/uapi/linux/cxl_mem.h
> > +++ b/include/uapi/linux/cxl_mem.h
> > @@ -19,6 +19,7 @@ extern "C" {
> >   */
> >
> >  #define CXL_MEM_QUERY_COMMANDS _IOR('C', 1, struct cxl_mem_query_commands)
> > +#define CXL_MEM_SEND_COMMAND _IOWR('C', 2, struct cxl_send_command)
> >
> >  #define CXL_MEM_COMMAND_NAME_LENGTH 32
> >
> > @@ -48,10 +49,12 @@ extern "C" {
> >  struct cxl_command_info {
> >         __u32 id;
> >  #define CXL_MEM_COMMAND_ID_INVALID 0
> > +#define CXL_MEM_COMMAND_ID_MAX (CXL_MEM_COMMAND_ID_INVALID + 1)
> >
> >         __u32 flags;
> >  #define CXL_MEM_COMMAND_FLAG_NONE 0
> >  #define CXL_MEM_COMMAND_FLAG_TAINT BIT(0)
> 
> I guess I missed this in the last patch, why would userspace specify a
> taint flag? @flags is an output parameter?
> 

The flag is from the kernel to userspace to let them know that using the command
would taint the kernel.

> > +#define CXL_MEM_COMMAND_FLAG_MASK ~BIT(0)
> >
> >         __s32 size_in;
> >         __s32 size_out;
> > @@ -95,6 +98,38 @@ struct cxl_mem_query_commands {
> >         struct cxl_command_info __user commands[]; /* out: supported commands */
> >  };
> >
> > +/**
> > + * struct cxl_send_command - Send a command to a memory device.
> > + * @id: The command to send to the memory device. This must be one of the
> > + *     commands returned by the query command.
> > + * @flags: Flags for the command
> 
> in / out field?
> 
> > + * @rsvd: Reserved for future use.
> 
> Document 'must be zero' to match the code?
> 
> > + * @retval: Return value from the memory device (output).
> 
> Is this a translated error code or the raw
> > + * @size_in: Size of the payload to provide to the device (input).
> > + * @size_out: Size of the payload received from the device (input/output). This
> > + *           field is filled in my userspace to let the driver know how much
> 
> s/my/by/
> 
> > + *           space was allocated for output. It is populated by the driver to
> > + *           let userspace know how large the output payload actually was.
> > + * @payload: Pointer to memory available for payload input/output.
> > + *
> > + * Mechanism for userspace to send a command to the hardware for processing. The
> > + * driver will do basic validation on the command sizes, but the payload input
> > + * and output are not introspected. Userspace is required to allocate large
> 
> They might be introspected, at least there were some NVDIMM ioctls
> like "clear error" where the driver in addition to telling the device
> to clear the error also cleaned up the kernel's error tracking list.
> I'd say one of the benefits of Linux-commands that get translated to
> CXL-specific payloads is that they allow for kernel translation /
> interpretation / quirking vs 'raw' where the kernel does not care.
> 
> > + * enough buffers for max(size_in, size_out).
> > + */
> > +struct cxl_send_command {
> > +       __u32 id;
> > +       __u32 flags;
> > +       __u32 rsvd;
> > +       __u32 retval;
> > +
> > +       struct {
> > +               __s32 size_in;
> > +               __s32 size_out;
> > +               __u64 payload;
> 
> per the initial comment above a separate @payload_in and a
> @payload_out pointer would seem to be more flexible and get out of the
> situation where the kernel needs to clobber inputs.

Anything I didn't specifically respond to, I'm working on fixing.

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [RFC PATCH 10/14] cxl/mem: Add send command
  2020-12-15 21:43     ` Ben Widawsky
@ 2020-12-15 22:03       ` Dan Williams
  2020-12-15 22:17         ` Ben Widawsky
  0 siblings, 1 reply; 35+ messages in thread
From: Dan Williams @ 2020-12-15 22:03 UTC (permalink / raw)
  To: Ben Widawsky
  Cc: linux-cxl, Linux Kernel Mailing List, Linux PCI, Linux ACPI,
	Ira Weiny, Vishal Verma, Kelley, Sean V, Rafael Wysocki,
	Bjorn Helgaas, Jonathan Cameron, Jon Masters, Chris Browy,
	Randy Dunlap, Christoph Hellwig

On Tue, Dec 15, 2020 at 1:44 PM Ben Widawsky <ben.widawsky@intel.com> wrote:
[..]
> > > +static int handle_mailbox_cmd_from_user(struct cxl_memdev *cxlmd,
> > > +                                       const struct cxl_mem_command *cmd,
> > > +                                       struct cxl_send_command __user *u)
> > > +{
> > > +       struct mbox_cmd mbox_cmd;
> > > +       ssize_t payload_size;
> > > +       void *payload;
> > > +       u32 size_out;
> > > +       int rc;
> > > +
> > > +       if (get_user(size_out, &u->size_out))
> > > +               return -EFAULT;
> > > +
> > > +       payload_size = max_t(ssize_t, cmd->info.size_in, size_out);
> > > +       if (payload_size) {
> > > +               payload =
> > > +                       memdup_user(u64_to_user_ptr(u->payload), payload_size);
> >
> > Me thinks this should be vmemdup_user() for payloads that exceed the
> > kmalloc() max, and I think it would be worthwhile to clamp @size_out
> > to some maximum and not let userspace ask for gigantic payloads.
> > Return EINVAL for payloads greater than... 4MB? At least 4MB is the
> > arbitrary max that libnvdimm picked.
> >
>
> This is handled in cxl_validate_cmd_from_user() currently. The current API
> allows the user to specify as large as they like with @size_out but the kernel
> only writes out as much as the hardware returns.... More on this below [1].
[..]
> > > + *  * %-EPERM  - Protected command used by the RAW interface.
> > > + *  * %-ENOMEM - Input or output buffer wasn't large enough.
> > > + *
> > > + */
> > > +static int cxl_validate_cmd_from_user(struct cxl_send_command __user *user_cmd,
> > > +                                     struct cxl_mem_command *out_cmd)
> > > +{
> > > +       const struct cxl_command_info *info;
> > > +       struct cxl_send_command cmd;
> > > +       struct cxl_mem_command *c;
> > > +
> > > +       if (copy_from_user(&cmd, user_cmd, sizeof(cmd)))
> > > +               return -EFAULT;
> > > +
> > > +       if (cmd.id == 0 || cmd.id >= CXL_MEM_COMMAND_ID_MAX)
> > > +               return -EINVAL;
> >
> > I wonder if the "cmd.id >= CXL_MEM_COMMAND_ID_MAX" case should return
> > -ENOTTY. The command might be perfectly valid, just the kernel does
> > not have that command specified which would be the case with older
> > kernel with newer userspace.
> >
>
> I think ENOTTY could be a bit confusing here as it could be confused for the
> SEND ioctl not being present.
>
> What about ENOENT, or something else?

"No such file or directory" seems more confusing to me than
"Inappropriate I/O control operation". What version of this driver,
once it's upstream, will not support the SEND ioctl? I think it is
safe to assume that the mere fact that the driver is attached means
SEND is present.

> > > +
> > > +       c = &mem_commands[cmd.id];
> > > +       info = &c->info;
> > > +
> > > +       if (cmd.flags & CXL_MEM_COMMAND_FLAG_MASK)
> > > +               return -EINVAL;
> > > +
> > > +       if (cmd.rsvd)
> > > +               return -EINVAL;
> > > +
> > > +       /* Check the input buffer is the expected size */
> > > +       if (info->size_in >= 0 && info->size_in != cmd.size_in)
> > > +               return -ENOMEM;
> > > +
> > > +       /* Check the output buffer is at least large enough */
> > > +       if (info->size_out >= 0 && cmd.size_out < info->size_out)
> > > +               return -ENOMEM;
> > > +
> > > +       memcpy(out_cmd, c, sizeof(*c));
> >
> > Why not do the copy_from_user() directly into out_cmd to save a copy operation?
> >
>
> [1]
> cxl_validate_cmd_from_user() essentially translates a user's command into the
> internal kernel representation of the command via the lookup into the array.
> This will do things like adjust the output size or flags, to prevent userspace
> from doing nefarious things. An actual command, especially now that you had me
> remove the name[32] will be relatively small and I see the extra copy as being
> well worth being able to isolate this command sanitization.

I can buy that, but it does not answer my other question above about
cases where a valid command has size_{in,out} larger than the kmalloc
max. vmemdup_user() lets you support arbitrary command payload sizes
larger than KMALLOC_MAX_SIZE. For example, I might want to overwrite
the entire label area at once in a single write.

>
> > > +
> > > +       return 0;
> > >  }
> > >
> > >  static long cxl_mem_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
> > > @@ -357,6 +471,19 @@ static long cxl_mem_ioctl(struct file *file, unsigned int cmd, unsigned long arg
> > >
> > >                         j++;
> > >                 }
> > > +
> > > +               return 0;
> > > +       } else if (cmd == CXL_MEM_SEND_COMMAND) {
> > > +               struct cxl_send_command __user *u = (void __user *)arg;
> > > +               struct cxl_memdev *cxlmd = file->private_data;
> > > +               struct cxl_mem_command c;
> > > +               int rc;
> > > +
> > > +               rc = cxl_validate_cmd_from_user(u, &c);
> > > +               if (rc)
> > > +                       return rc;
> > > +
> > > +               return handle_mailbox_cmd_from_user(cxlmd, &c, u);
> > >         }
> > >
> > >         return -ENOTTY;
> > > diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
> > > index 1d1e143f98ec..189d86a13637 100644
> > > --- a/include/uapi/linux/cxl_mem.h
> > > +++ b/include/uapi/linux/cxl_mem.h
> > > @@ -19,6 +19,7 @@ extern "C" {
> > >   */
> > >
> > >  #define CXL_MEM_QUERY_COMMANDS _IOR('C', 1, struct cxl_mem_query_commands)
> > > +#define CXL_MEM_SEND_COMMAND _IOWR('C', 2, struct cxl_send_command)
> > >
> > >  #define CXL_MEM_COMMAND_NAME_LENGTH 32
> > >
> > > @@ -48,10 +49,12 @@ extern "C" {
> > >  struct cxl_command_info {
> > >         __u32 id;
> > >  #define CXL_MEM_COMMAND_ID_INVALID 0
> > > +#define CXL_MEM_COMMAND_ID_MAX (CXL_MEM_COMMAND_ID_INVALID + 1)
> > >
> > >         __u32 flags;
> > >  #define CXL_MEM_COMMAND_FLAG_NONE 0
> > >  #define CXL_MEM_COMMAND_FLAG_TAINT BIT(0)
> >
> > I guess I missed this in the last patch, why would userspace specify a
> > taint flag? @flags is an output parameter?
> >
>
> The flag is from the kernel to userspace to let them know that using the command
> would taint the kernel.

...but they should know that by the fact that they are using the RAW
send. I.e. document it once rather than include it in the ABI. I think
a more useful flag would be to identify which commands may be blocked
because they are exclusively owned by the kernel.

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [RFC PATCH 10/14] cxl/mem: Add send command
  2020-12-15 22:03       ` Dan Williams
@ 2020-12-15 22:17         ` Ben Widawsky
  0 siblings, 0 replies; 35+ messages in thread
From: Ben Widawsky @ 2020-12-15 22:17 UTC (permalink / raw)
  To: Dan Williams
  Cc: linux-cxl, Linux Kernel Mailing List, Linux PCI, Linux ACPI,
	Ira Weiny, Vishal Verma, Kelley, Sean V, Rafael Wysocki,
	Bjorn Helgaas, Jonathan Cameron, Jon Masters, Chris Browy,
	Randy Dunlap, Christoph Hellwig

On 20-12-15 14:03:00, Dan Williams wrote:
> On Tue, Dec 15, 2020 at 1:44 PM Ben Widawsky <ben.widawsky@intel.com> wrote:
> [..]
> > > > +static int handle_mailbox_cmd_from_user(struct cxl_memdev *cxlmd,
> > > > +                                       const struct cxl_mem_command *cmd,
> > > > +                                       struct cxl_send_command __user *u)
> > > > +{
> > > > +       struct mbox_cmd mbox_cmd;
> > > > +       ssize_t payload_size;
> > > > +       void *payload;
> > > > +       u32 size_out;
> > > > +       int rc;
> > > > +
> > > > +       if (get_user(size_out, &u->size_out))
> > > > +               return -EFAULT;
> > > > +
> > > > +       payload_size = max_t(ssize_t, cmd->info.size_in, size_out);
> > > > +       if (payload_size) {
> > > > +               payload =
> > > > +                       memdup_user(u64_to_user_ptr(u->payload), payload_size);
> > >
> > > Me thinks this should be vmemdup_user() for payloads that exceed the
> > > kmalloc() max, and I think it would be worthwhile to clamp @size_out
> > > to some maximum and not let userspace ask for gigantic payloads.
> > > Return EINVAL for payloads greater than... 4MB? At least 4MB is the
> > > arbitrary max that libnvdimm picked.
> > >
> >
> > This is handled in cxl_validate_cmd_from_user() currently. The current API
> > allows the user to specify as large as they like with @size_out but the kernel
> > only writes out as much as the hardware returns.... More on this below [1].
> [..]
> > > > + *  * %-EPERM  - Protected command used by the RAW interface.
> > > > + *  * %-ENOMEM - Input or output buffer wasn't large enough.
> > > > + *
> > > > + */
> > > > +static int cxl_validate_cmd_from_user(struct cxl_send_command __user *user_cmd,
> > > > +                                     struct cxl_mem_command *out_cmd)
> > > > +{
> > > > +       const struct cxl_command_info *info;
> > > > +       struct cxl_send_command cmd;
> > > > +       struct cxl_mem_command *c;
> > > > +
> > > > +       if (copy_from_user(&cmd, user_cmd, sizeof(cmd)))
> > > > +               return -EFAULT;
> > > > +
> > > > +       if (cmd.id == 0 || cmd.id >= CXL_MEM_COMMAND_ID_MAX)
> > > > +               return -EINVAL;
> > >
> > > I wonder if the "cmd.id >= CXL_MEM_COMMAND_ID_MAX" case should return
> > > -ENOTTY. The command might be perfectly valid, just the kernel does
> > > not have that command specified which would be the case with older
> > > kernel with newer userspace.
> > >
> >
> > I think ENOTTY could be a bit confusing here as it could be confused for the
> > SEND ioctl not being present.
> >
> > What about ENOENT, or something else?
> 
> "No such file or directory" seems more confusing to me than
> "Inappropriate I/O control operation". What version of this driver,
> once it's upstream, will not support the SEND ioctl? I think it is
> safe to assume that the mere fact that the driver is attached means
> SEND is present.
> 

No version. I had been thinking it gets confusing with a new IOCTL, but you're
correct that it should be trivial to determine ENOTTY from Send is different
than ENOTTY from another IOCTL.

I'll change it.

> > > > +
> > > > +       c = &mem_commands[cmd.id];
> > > > +       info = &c->info;
> > > > +
> > > > +       if (cmd.flags & CXL_MEM_COMMAND_FLAG_MASK)
> > > > +               return -EINVAL;
> > > > +
> > > > +       if (cmd.rsvd)
> > > > +               return -EINVAL;
> > > > +
> > > > +       /* Check the input buffer is the expected size */
> > > > +       if (info->size_in >= 0 && info->size_in != cmd.size_in)
> > > > +               return -ENOMEM;
> > > > +
> > > > +       /* Check the output buffer is at least large enough */
> > > > +       if (info->size_out >= 0 && cmd.size_out < info->size_out)
> > > > +               return -ENOMEM;
> > > > +
> > > > +       memcpy(out_cmd, c, sizeof(*c));
> > >
> > > Why not do the copy_from_user() directly into out_cmd to save a copy operation?
> > >
> >
> > [1]
> > cxl_validate_cmd_from_user() essentially translates a user's command into the
> > internal kernel representation of the command via the lookup into the array.
> > This will do things like adjust the output size or flags, to prevent userspace
> > from doing nefarious things. An actual command, especially now that you had me
> > remove the name[32] will be relatively small and I see the extra copy as being
> > well worth being able to isolate this command sanitization.
> 
> I can buy that, but it does not answer my other question above about
> cases where a valid command has size_{in,out} larger than the kmalloc
> max. vmemdup_user() lets you support arbitrary command payload sizes
> larger than KMALLOC_MAX_SIZE. For example, I might want to overwrite
> the entire label area at once in a single write.
> 

It does need to switch to vmemdup_user(), I didn't mean to disagree there.

For posterity btw, I'll point out CXL 2.0 max is 2M for payload.

> >
> > > > +
> > > > +       return 0;
> > > >  }
> > > >
> > > >  static long cxl_mem_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
> > > > @@ -357,6 +471,19 @@ static long cxl_mem_ioctl(struct file *file, unsigned int cmd, unsigned long arg
> > > >
> > > >                         j++;
> > > >                 }
> > > > +
> > > > +               return 0;
> > > > +       } else if (cmd == CXL_MEM_SEND_COMMAND) {
> > > > +               struct cxl_send_command __user *u = (void __user *)arg;
> > > > +               struct cxl_memdev *cxlmd = file->private_data;
> > > > +               struct cxl_mem_command c;
> > > > +               int rc;
> > > > +
> > > > +               rc = cxl_validate_cmd_from_user(u, &c);
> > > > +               if (rc)
> > > > +                       return rc;
> > > > +
> > > > +               return handle_mailbox_cmd_from_user(cxlmd, &c, u);
> > > >         }
> > > >
> > > >         return -ENOTTY;
> > > > diff --git a/include/uapi/linux/cxl_mem.h b/include/uapi/linux/cxl_mem.h
> > > > index 1d1e143f98ec..189d86a13637 100644
> > > > --- a/include/uapi/linux/cxl_mem.h
> > > > +++ b/include/uapi/linux/cxl_mem.h
> > > > @@ -19,6 +19,7 @@ extern "C" {
> > > >   */
> > > >
> > > >  #define CXL_MEM_QUERY_COMMANDS _IOR('C', 1, struct cxl_mem_query_commands)
> > > > +#define CXL_MEM_SEND_COMMAND _IOWR('C', 2, struct cxl_send_command)
> > > >
> > > >  #define CXL_MEM_COMMAND_NAME_LENGTH 32
> > > >
> > > > @@ -48,10 +49,12 @@ extern "C" {
> > > >  struct cxl_command_info {
> > > >         __u32 id;
> > > >  #define CXL_MEM_COMMAND_ID_INVALID 0
> > > > +#define CXL_MEM_COMMAND_ID_MAX (CXL_MEM_COMMAND_ID_INVALID + 1)
> > > >
> > > >         __u32 flags;
> > > >  #define CXL_MEM_COMMAND_FLAG_NONE 0
> > > >  #define CXL_MEM_COMMAND_FLAG_TAINT BIT(0)
> > >
> > > I guess I missed this in the last patch, why would userspace specify a
> > > taint flag? @flags is an output parameter?
> > >
> >
> > The flag is from the kernel to userspace to let them know that using the command
> > would taint the kernel.
> 
> ...but they should know that by the fact that they are using the RAW
> send. I.e. document it once rather than include it in the ABI. I think
> a more useful flag would be to identify which commands may be blocked
> because they are exclusively owned by the kernel.

I think taint is something that could change between driver versions, and even
based on the underlying device. However, since Raw command will always taint, we
can save that for another day and I will drop it from this patch.

As an example, perhaps we know device X's active firmware update is flaky - we
can make that a tainted command just for that device.

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [RFC PATCH 11/14] cxl/mem: Add a "RAW" send command
  2020-12-09 22:38   ` Dan Williams
@ 2020-12-16 20:42     ` Ben Widawsky
  0 siblings, 0 replies; 35+ messages in thread
From: Ben Widawsky @ 2020-12-16 20:42 UTC (permalink / raw)
  To: Dan Williams
  Cc: linux-cxl, Linux Kernel Mailing List, Linux PCI, Linux ACPI,
	Ira Weiny, Vishal Verma, Kelley, Sean V, Rafael Wysocki,
	Bjorn Helgaas, Jonathan Cameron, Jon Masters, Chris Browy,
	Randy Dunlap, Christoph Hellwig

On 20-12-09 14:38:49, Dan Williams wrote:
> On Tue, Dec 8, 2020 at 4:24 PM Ben Widawsky <ben.widawsky@intel.com> wrote:
> >
> > The CXL memory device send interface will have a number of supported
> > commands. The raw command is not such a command. Raw commands allow
> > userspace to send a specified opcode to the underlying hardware and
> > bypass all driver checks on the command. This is useful for a couple of
> > usecases, mainly:
> > 1. Undocumented vendor specific hardware commands
> > 2. Prototyping new hardware commands not yet supported by the driver
> >
> > While this all sounds very powerful it comes with a couple of caveats:
> > 1. Bug reports using raw commands will not get the same level of
> >    attention as bug reports using supported commands (via taint).
> > 2. Supported commands will be rejected by the RAW command.
> >
> > Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
> > ---
> >  drivers/cxl/mem.c            | 32 ++++++++++++++++++++++++++++++++
> >  include/uapi/linux/cxl_mem.h | 14 ++++++++++++--
> >  2 files changed, 44 insertions(+), 2 deletions(-)
> >
> > diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
> > index 0bf03afc0c80..a2cea7ac7cc6 100644
> > --- a/drivers/cxl/mem.c
> > +++ b/drivers/cxl/mem.c
> > @@ -115,6 +115,7 @@ struct cxl_mem_command {
> >
> >  static struct cxl_mem_command mem_commands[] = {
> >         CXL_CMD(INVALID, NONE, 0, 0, "Reserved", false, 0),
> > +       CXL_CMD(RAW, TAINT, ~0, ~0, "Raw", true, 0),
> 
> Why is the taint indication in the ABI? It seems like it only needs to
> be documented.
> 

It's removed per the previous patch discussion.

> >  };
> >
> >  static int cxl_mem_wait_for_doorbell(struct cxl_mem *cxlm)
> > @@ -326,6 +327,20 @@ static int cxl_mem_count_commands(void)
> >         return n;
> >  };
> >
> > +static struct cxl_mem_command *cxl_mem_find_command(u16 opcode)
> > +{
> > +       int i;
> > +
> > +       for (i = 0; i < ARRAY_SIZE(mem_commands); i++) {
> > +               struct cxl_mem_command *c = &mem_commands[i];
> > +
> > +               if (c->opcode == opcode)
> > +                       return c;
> > +       }
> > +
> > +       return NULL;
> > +};
> > +
> >  /**
> >   * handle_mailbox_cmd_from_user() - Dispatch a mailbox command.
> >   * @cxlmd: The CXL memory device to communicate with.
> > @@ -421,6 +436,23 @@ static int cxl_validate_cmd_from_user(struct cxl_send_command __user *user_cmd,
> >         c = &mem_commands[cmd.id];
> >         info = &c->info;
> >
> > +       /* Checks are bypassed for raw commands but along comes the taint! */
> > +       if (cmd.id == CXL_MEM_COMMAND_ID_RAW) {
> > +               struct cxl_mem_command temp =
> > +                       CXL_CMD(RAW, NONE, cmd.size_in, cmd.size_out, "Raw",
> > +                               true, cmd.raw.opcode);
> 
> Oh, I thought CXL_CMD() was only used to populate the mem_commands
> array. Feels out of place to use it here when all it is doing is
> updating the size_{in,out} and opcode fields. Mainly I'm interested in
> CXL_CMD() enforcing that the command-id is the mem_commands index.
> 

Agreed and removed.

> > +
> > +               if (cmd.raw.rsvd)
> > +                       return -EINVAL;
> > +
> > +               if (cxl_mem_find_command(cmd.raw.opcode))
> > +                       return -EPERM;
> > +
> > +               add_taint(TAINT_WARN, LOCKDEP_STILL_OK);
> 
> TAINT_WARN seems the wrong value, especially since no WARN has
> occurred. I feel that this is more in the spirit of
> TAINT_PROPRIETARY_MODULE, TAINT_OVERRIDDEN_ACPI_TABLE, and
> TAINT_OOT_MODULE. How about a new TAINT_RAW_PASSTHROUGH? I could use
> this for the acpi/nfit driver as well to disclaim responsibility for
> system errors that can result from not using the nominal
> kernel-provided commands.

I like it.

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [RFC PATCH 12/14] cxl: Add basic debugging
  2020-12-09  3:06       ` Dan Williams
@ 2020-12-16 21:02         ` Ben Widawsky
  0 siblings, 0 replies; 35+ messages in thread
From: Ben Widawsky @ 2020-12-16 21:02 UTC (permalink / raw)
  To: Dan Williams
  Cc: linux-cxl, Linux Kernel Mailing List, Linux PCI, Linux ACPI,
	Ira Weiny, Vishal Verma, Kelley, Sean V, Rafael Wysocki,
	Bjorn Helgaas, Jonathan Cameron, Jon Masters, Chris Browy,
	Randy Dunlap, Christoph Hellwig

On 20-12-08 19:06:46, Dan Williams wrote:
> On Tue, Dec 8, 2020 at 6:04 PM Ben Widawsky <ben.widawsky@intel.com> wrote:
> >
> > On 20-12-08 17:17:36, Dan Williams wrote:
> > > On Tue, Dec 8, 2020 at 4:24 PM Ben Widawsky <ben.widawsky@intel.com> wrote:
> > > >
> > > > Provide a standard debug function for use throughout the driver.
> > > >
> > > > Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
> > > > ---
> > > >  drivers/cxl/cxl.h |  3 +++
> > > >  drivers/cxl/mem.c | 26 +++++++++++++++++++++++++-
> > > >  2 files changed, 28 insertions(+), 1 deletion(-)
> > > >
> > > > diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> > > > index 77c2dee6843c..e5afb89dab0b 100644
> > > > --- a/drivers/cxl/cxl.h
> > > > +++ b/drivers/cxl/cxl.h
> > > > @@ -9,6 +9,9 @@
> > > >  #include <linux/bitops.h>
> > > >  #include <linux/io.h>
> > > >
> > > > +#define cxl_debug(fmt, ...)                                                    \
> > > > +       pr_debug("CXL DEBUG: %s: " fmt, __func__, ##__VA_ARGS__)
> > > > +
> > >
> > > This should be dev_dbg(), then you don't need the CXL DEBUG prefix. In
> > > fact you don't need a cxl_debug() macro at all in that case. cxl_mem
> > > might need a ->dev attribute for this purpose.
> > >
> >
> > I really like the ability to turn specific messages on and off at will. (FWIW,
> > __func__ is also redundant because pr_debug allows you to specify a flag to
> > always print the function name). While it's not very frequent events here, in
> > the future it likely will be and I think it can be really helpful to be able to
> > have that level of control.
> >
> > If you want to avoid creating a new debug functionality, I'm okay with that, but
> > I'd really like to use pr_debug instead of dev_dbg for those messages going
> > forward. Once you take that step, it seems giving contributors a macro named
> > 'cxl_debug' so they don't have to figure out when to use what, makes sense. My
> > mental separation is, dev_* is useful primarily for errors and initialization
> > debug messaging, pr_debug/trace_printk is for runtime things.
> >
> > I probably should have put that in the commit message...
> 
> I suspect you haven't taken a look at the backed and of pr_debug() and
> dev_dbg() in a while? They both use _dynamic_func_call and enjoy all
> the benefits afforded by /sys/debug/dynamic_debug/control and the
> "dyndbg" module option for adding __func__ and enable / disable by
> line number or format message . pr_debug() and this cxl_debug() macro
> are completely superseded by dev_dbg(). Even if a driver wanted a
> common prefix on all prints there is the "#define dev_fmt ..."
> mechanism for that.
> 

In that case, I'm just going to squash in the debug messages into the various
commits that need them.

> >
> > > >  #define CXL_SET_FIELD(value, field)                                            \
> > > >         ({                                                                     \
> > > >                 WARN_ON(!FIELD_FIT(field##_MASK, value));                      \
> > > > diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
> > > > index a2cea7ac7cc6..6b2f8d3776b5 100644
> > > > --- a/drivers/cxl/mem.c
> > > > +++ b/drivers/cxl/mem.c
> > > > @@ -122,9 +122,12 @@ static int cxl_mem_wait_for_doorbell(struct cxl_mem *cxlm)
> > > >  {
> > > >         const int timeout = msecs_to_jiffies(2000);
> > > >         const unsigned long start = jiffies;
> > > > +       unsigned long end = start;
> > > >
> > > >         while (cxl_doorbell_busy(cxlm)) {
> > > > -               if (time_after(jiffies, start + timeout)) {
> > > > +               end = jiffies;
> > > > +
> > > > +               if (time_after(end, start + timeout)) {
> > > >                         /* Check again in case preempted before timeout test */
> > > >                         if (!cxl_doorbell_busy(cxlm))
> > > >                                 break;
> > > > @@ -133,6 +136,8 @@ static int cxl_mem_wait_for_doorbell(struct cxl_mem *cxlm)
> > > >                 cpu_relax();
> > > >         }
> > > >
> > > > +       cxl_debug("Doorbell wait took %dms",
> > > > +                 jiffies_to_msecs(end) - jiffies_to_msecs(start));
> > > >         return 0;
> > > >  }
> > > >
> > > > @@ -180,6 +185,8 @@ static int cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm,
> > > >         }
> > > >
> > > >         /* #4 */
> > > > +       cxl_debug("Sending command to %s\n",
> > > > +                 dev_driver_string(&cxlm->pdev->dev));
> > >
> > > dev_dbg() already includes dev_driver_string().
> > >
> > > >         cxl_write_mbox_reg32(cxlm, CXLDEV_MB_CTRL_OFFSET,
> > > >                              CXLDEV_MB_CTRL_DOORBELL);
> > > >
> > > > @@ -308,6 +315,8 @@ static int cxl_mem_open(struct inode *inode, struct file *file)
> > > >         if (!cxlmd)
> > > >                 return -ENXIO;
> > > >
> > > > +       cxl_debug("Opened %pD\n", file);
> > > > +
> > > >         file->private_data = cxlmd;
> > > >
> > > >         return 0;
> > > > @@ -383,6 +392,10 @@ static int handle_mailbox_cmd_from_user(struct cxl_memdev *cxlmd,
> > > >                 .size_in = cmd->info.size_in,
> > > >                 .size_out = size_out,
> > > >         };
> > > > +       cxl_debug("Submitting command for user\n"
> > > > +                 "\topcode: %x\n"
> > > > +                 "\tsize: %zub/%zub\n",
> > > > +                 mbox_cmd.opcode, mbox_cmd.size_in, mbox_cmd.size_out);
> > > >         rc = cxl_mem_mbox_send_cmd(cxlmd->cxlm, &mbox_cmd);
> > > >         cxl_mem_mbox_put(cxlmd->cxlm);
> > > >         if (rc)
> > > > @@ -479,6 +492,8 @@ static long cxl_mem_ioctl(struct file *file, unsigned int cmd, unsigned long arg
> > > >                 u32 n_commands;
> > > >                 int i, j;
> > > >
> > > > +               cxl_debug("Query IOCTL\n");
> > > > +
> > > >                 if (get_user(n_commands, (u32 __user *)arg))
> > > >                         return -EFAULT;
> > > >
> > > > @@ -511,6 +526,8 @@ static long cxl_mem_ioctl(struct file *file, unsigned int cmd, unsigned long arg
> > > >                 struct cxl_mem_command c;
> > > >                 int rc;
> > > >
> > > > +               cxl_debug("Send IOCTL\n");
> > > > +
> > > >                 rc = cxl_validate_cmd_from_user(u, &c);
> > > >                 if (rc)
> > > >                         return rc;
> > > > @@ -843,6 +860,13 @@ static int cxl_mem_identify(struct cxl_mem *cxlm)
> > > >
> > > >         id = (struct cxl_mbox_identify *)mbox_cmd.payload;
> > > >
> > > > +       cxl_debug("Driver identify command\n"
> > > > +                 "\tFirmware Version: %s\n"
> > > > +                 "\tTotal Capacity: %llu (%llu persistent)\n"
> > > > +                 "\tLSA size: %u\n",
> > > > +                 id->fw_revision, id->total_capacity, id->persistent_capacity,
> > > > +                 id->lsa_size);
> > > > +
> > >
> > > Seems not necessary for details that are published in sysfs?
> >
> > I was thinking for cases where driver doesn't bind, seeing the identify
> > information could be useful. This is one case where dev_dbg would also serve
> > IMO.
> 
> There's very little that can cause bind failures after
> cxl_mem_identify() succeeds. The bind failures themselves will
> dev_err() the reason.

^ permalink raw reply	[flat|nested] 35+ messages in thread

* Re: [RFC PATCH v2 07/14] cxl/mem: Implement polled mode mailbox
  2020-12-09  0:24 ` [RFC PATCH v2 07/14] cxl/mem: Implement polled mode mailbox Ben Widawsky
@ 2021-01-07 19:05   ` Ben Widawsky
  0 siblings, 0 replies; 35+ messages in thread
From: Ben Widawsky @ 2021-01-07 19:05 UTC (permalink / raw)
  To: linux-cxl
  Cc: linux-kernel, linux-pci, linux-acpi, Ira Weiny, Dan Williams,
	Vishal Verma, Kelley, Sean V, Rafael Wysocki, Bjorn Helgaas,
	Jonathan Cameron, Jon Masters, Chris Browy, Randy Dunlap,
	Christoph Hellwig

On 20-12-08 16:24:11, Ben Widawsky wrote:

[snip]

> +static int cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm,
> +				 struct mbox_cmd *mbox_cmd)
> +{
> +	u64 cmd, status;
> +	size_t out_len;
> +	int rc;
> +
> +	lockdep_assert_held(&cxlm->mbox_lock);
> +
> +	/*
> +	 * Here are the steps from 8.2.8.4 of the CXL 2.0 spec.
> +	 *   1. Caller reads MB Control Register to verify doorbell is clear
> +	 *   2. Caller writes Command Register
> +	 *   3. Caller writes Command Payload Registers if input payload is non-empty
> +	 *   4. Caller writes MB Control Register to set doorbell
> +	 *   5. Caller either polls for doorbell to be clear or waits for interrupt if configured
> +	 *   6. Caller reads MB Status Register to fetch Return code
> +	 *   7. If command successful, Caller reads Command Register to get Payload Length
> +	 *   8. If output payload is non-empty, host reads Command Payload Registers
> +	 */
> +
> +	/* #1 */
> +	WARN_ON(cxl_doorbell_busy(cxlm));
> +
> +	/* #2 */
> +	cxl_write_mbox_reg64(cxlm, CXLDEV_MB_CMD_OFFSET, mbox_cmd->opcode);
> +
> +	if (mbox_cmd->size_in) {
> +		/* #3 */
> +		CXL_SET_FIELD(mbox_cmd->size_in, CXLDEV_MB_CMD_PAYLOAD_LENGTH);
> +		cxl_mbox_payload_fill(cxlm, mbox_cmd->payload,
> +				      mbox_cmd->size_in);
> +	}

There is a bug here where the payload length isn't written on input. It was
working in v1. It will be fixed in v3.

[snip]


^ permalink raw reply	[flat|nested] 35+ messages in thread

end of thread, other threads:[~2021-01-07 19:06 UTC | newest]

Thread overview: 35+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-12-09  0:24 [RFC PATCH v2 00/14] CXL 2.0 Support Ben Widawsky
2020-12-09  0:24 ` [RFC PATCH v2 01/14] docs: cxl: Add basic documentation Ben Widawsky
2020-12-09  0:24 ` [RFC PATCH v2 02/14] cxl/acpi: Add an acpi_cxl module for the CXL interconnect Ben Widawsky
2020-12-09  0:24 ` [RFC PATCH v2 03/14] cxl/acpi: add OSC support Ben Widawsky
2020-12-09  0:24 ` [RFC PATCH v2 04/14] cxl/mem: Introduce a driver for CXL-2.0-Type-3 endpoints Ben Widawsky
2020-12-09  0:24 ` [RFC PATCH v2 05/14] cxl/mem: Map memory device registers Ben Widawsky
2020-12-09  0:24 ` [RFC PATCH v2 06/14] cxl/mem: Find device capabilities Ben Widawsky
2020-12-09  0:24 ` [RFC PATCH v2 07/14] cxl/mem: Implement polled mode mailbox Ben Widawsky
2021-01-07 19:05   ` Ben Widawsky
2020-12-09  0:24 ` [RFC PATCH v2 08/14] cxl/mem: Register CXL memX devices Ben Widawsky
2020-12-09  0:24 ` [RFC PATCH 09/14] cxl/mem: Add basic IOCTL interface Ben Widawsky
2020-12-09  1:37   ` Dan Williams
2020-12-09  2:12     ` Ben Widawsky
2020-12-09  3:33       ` Dan Williams
2020-12-09 16:23         ` Ben Widawsky
2020-12-10  3:32   ` Randy Dunlap
2020-12-14 17:29     ` Ben Widawsky
2020-12-09  0:24 ` [RFC PATCH 10/14] cxl/mem: Add send command Ben Widawsky
2020-12-09 22:06   ` Dan Williams
2020-12-15 21:43     ` Ben Widawsky
2020-12-15 22:03       ` Dan Williams
2020-12-15 22:17         ` Ben Widawsky
2020-12-09  0:24 ` [RFC PATCH 11/14] cxl/mem: Add a "RAW" " Ben Widawsky
2020-12-09 22:38   ` Dan Williams
2020-12-16 20:42     ` Ben Widawsky
2020-12-09  0:24 ` [RFC PATCH 12/14] cxl: Add basic debugging Ben Widawsky
2020-12-09  1:17   ` Dan Williams
2020-12-09  2:04     ` Ben Widawsky
2020-12-09  3:06       ` Dan Williams
2020-12-16 21:02         ` Ben Widawsky
2020-12-09  0:24 ` [RFC PATCH 13/14] MAINTAINERS: Add maintainers of the CXL driver Ben Widawsky
2020-12-09  0:24 ` [RFC PATCH 14/14] WIP/cxl/mem: Add get firmware for testing Ben Widawsky
2020-12-10  0:42   ` Dan Williams
2020-12-09  0:33 ` [RFC PATCH v2 00/14] CXL 2.0 Support Verma, Vishal L
2020-12-09  0:47 ` Ben Widawsky

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).