From: Jonathan Cameron <Jonathan.Cameron@Huawei.com>
To: Dan Williams <dan.j.williams@intel.com>
Cc: <linux-cxl@vger.kernel.org>, kernel test robot <lkp@intel.com>,
"Ben Widawsky" <ben.widawsky@intel.com>,
<linux-pci@vger.kernel.org>, <nvdimm@lists.linux.dev>
Subject: Re: [PATCH v4 24/40] cxl/port: Add a driver for 'struct cxl_port' objects
Date: Mon, 31 Jan 2022 18:11:18 +0000 [thread overview]
Message-ID: <20220131181118.00002471@Huawei.com> (raw)
In-Reply-To: <164322817812.3708001.17146719098062400994.stgit@dwillia2-desk3.amr.corp.intel.com>
On Wed, 26 Jan 2022 12:16:52 -0800
Dan Williams <dan.j.williams@intel.com> wrote:
> From: Ben Widawsky <ben.widawsky@intel.com>
>
> The need for a CXL port driver and a dedicated cxl_bus_type is driven by
> a need to simultaneously support 2 independent physical memory decode
> domains (cache coherent CXL.mem and uncached PCI.mmio) that also
> intersect at a single PCIe device node. A CXL Port is a device that
> advertises a CXL Component Register block with an "HDM Decoder
> Capability Structure".
>
> >From Documentation/driver-api/cxl/memory-devices.rst:
>
> Similar to how a RAID driver takes disk objects and assembles them into
> a new logical device, the CXL subsystem is tasked to take PCIe and ACPI
> objects and assemble them into a CXL.mem decode topology. The need for
> runtime configuration of the CXL.mem topology is also similar to RAID in
> that different environments with the same hardware configuration may
> decide to assemble the topology in contrasting ways. One may choose
> performance (RAID0) striping memory across multiple Host Bridges and
> endpoints while another may opt for fault tolerance and disable any
> striping in the CXL.mem topology.
>
> The port driver identifies whether an endpoint Memory Expander is
> connected to a CXL topology. If an active (bound to the 'cxl_port'
> driver) CXL Port is not found at every PCIe Switch Upstream port and an
> active "root" CXL Port then the device is just a plain PCIe endpoint
> only capable of participating in PCI.mmio and DMA cycles, not CXL.mem
> coherent interleave sets.
>
> The 'cxl_port' driver lets the CXL subsystem leverage driver-core
> infrastructure for setup and teardown of register resources and
> communicating device activation status to userspace. The cxl_bus_type
> can rendezvous the async arrival of platform level CXL resources (via
> the 'cxl_acpi' driver) with the asynchronous enumeration of Memory
> Expander endpoints, while also implementing a hierarchical locking model
> independent of the associated 'struct pci_dev' locking model. The
> locking for dport and decoder enumeration is now handled in the core
> rather than callers.
>
> For now the port driver only enumerates and registers CXL resources
> (downstream port metadata and decoder resources) later it will be used
> to take action on its decoders in response to CXL.mem region
> provisioning requests.
>
> Reported-by: kernel test robot <lkp@intel.com>
> Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
> [djbw: add theory of operation document, move enumeration infra to core]
> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
Nice docs. A few comments inline
All trivial though, so
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
...
> diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
> index 2b09d04d3568..682e7cdbcc9c 100644
> --- a/drivers/cxl/core/port.c
> +++ b/drivers/cxl/core/port.c
> @@ -40,6 +40,11 @@ static int cxl_device_id(struct device *dev)
...
>
> +/*
> + * Since root-level CXL dports cannot be enumerated by PCI they are not
> + * enumerated by the common port driver that acquires the port lock over
> + * dport add/remove. Instead, root dports are manually added by a
> + * platform driver and cond_port_lock() is used to take the missing port
> + * lock in that case.
> + */
> +static void cond_port_lock(struct cxl_port *port)
Could the naming here make it clear what the condition is?
cxl_port_lock_if_root(), or something like that?
> +{
> + if (is_cxl_root(port))
> + cxl_device_lock(&port->dev);
> +}
> +
> +static void cond_port_unlock(struct cxl_port *port)
> +{
> + if (is_cxl_root(port))
> + cxl_device_unlock(&port->dev);
> +}
> +
> static void cxl_dport_remove(void *data)
> {
> struct cxl_dport *dport = data;
> struct cxl_port *port = dport->port;
>
> - cxl_device_lock(&port->dev);
> + cond_port_lock(port);
> list_del_init(&dport->list);
> - cxl_device_unlock(&port->dev);
> + cond_port_unlock(port);
> put_device(dport->dport);
> }
>
> @@ -588,7 +615,9 @@ struct cxl_dport *devm_cxl_add_dport(struct device *host, struct cxl_port *port,
> dport->component_reg_phys = component_reg_phys;
> dport->port = port;
>
> + cond_port_lock(port);
> rc = add_dport(port, dport);
> + cond_port_unlock(port);
> if (rc)
> return ERR_PTR(rc);
>
> @@ -887,6 +916,7 @@ static int cxl_bus_probe(struct device *dev)
> rc = to_cxl_drv(dev->driver)->probe(dev);
> cxl_nested_unlock(dev);
>
> + dev_dbg(dev, "probe: %d\n", rc);
This feels a little bit odd to see in this patch.
I'd be tempted to drop it.
> return rc;
> }
>
>
> #define MODULE_ALIAS_CXL(type) MODULE_ALIAS("cxl:t" __stringify(type) "*")
> #define CXL_MODALIAS_FMT "cxl:t%d"
> diff --git a/drivers/cxl/cxlpci.h b/drivers/cxl/cxlpci.h
> index 103636fda198..47640f19e899 100644
> --- a/drivers/cxl/cxlpci.h
> +++ b/drivers/cxl/cxlpci.h
> @@ -2,6 +2,7 @@
> /* Copyright(c) 2020 Intel Corporation. All rights reserved. */
> #ifndef __CXL_PCI_H__
> #define __CXL_PCI_H__
> +#include <linux/pci.h>
Why in this patch?
> #include "cxl.h"
>
> #define CXL_MEMORY_PROGIF 0x10
> diff --git a/tools/testing/cxl/Kbuild b/tools/testing/cxl/Kbuild
> index 3045d7cba0db..3e2a529875ea 100644
> --- a/tools/testing/cxl/Kbuild
> +++ b/tools/testing/cxl/Kbuild
> @@ -26,6 +26,12 @@ obj-m += cxl_pmem.o
> cxl_pmem-y := $(CXL_SRC)/pmem.o
> cxl_pmem-y += config_check.o
>
> +obj-m += cxl_port.o
> +
> +cxl_port-y := $(CXL_SRC)/port.o
> +cxl_port-y += config_check.o
> +
trivial but one blank line seems like enough.
> +
> obj-m += cxl_core.o
>
> cxl_core-y := $(CXL_CORE_SRC)/port.o
next prev parent reply other threads:[~2022-01-31 18:11 UTC|newest]
Thread overview: 172+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-01-24 0:28 [PATCH v3 00/40] CXL.mem Topology Discovery and Hotplug Support Dan Williams
2022-01-24 0:28 ` [PATCH v3 01/40] cxl: Rename CXL_MEM to CXL_PCI Dan Williams
2022-01-24 0:28 ` [PATCH v3 02/40] cxl/pci: Implement Interface Ready Timeout Dan Williams
2022-01-31 22:21 ` Ben Widawsky
2022-01-31 23:11 ` Dan Williams
2022-01-31 23:25 ` Ben Widawsky
2022-01-31 23:47 ` Dan Williams
2022-01-31 23:51 ` [PATCH v4 " Dan Williams
2022-01-24 0:28 ` [PATCH v3 03/40] cxl/pci: Defer mailbox status checks to command timeouts Dan Williams
2022-01-31 22:28 ` Ben Widawsky
2022-01-24 0:29 ` [PATCH v3 04/40] cxl: Flesh out register names Dan Williams
2022-01-24 0:29 ` [PATCH v3 05/40] cxl/pci: Add new DVSEC definitions Dan Williams
2022-01-24 0:29 ` [PATCH v3 06/40] cxl/acpi: Map component registers for Root Ports Dan Williams
2022-01-24 0:29 ` [PATCH v3 07/40] cxl: Introduce module_cxl_driver Dan Williams
2022-01-24 0:29 ` [PATCH v3 08/40] cxl/core/port: Rename bus.c to port.c Dan Williams
2022-01-31 22:34 ` Ben Widawsky
2022-01-24 0:29 ` [PATCH v3 09/40] cxl/decoder: Hide physical address information from non-root Dan Williams
2022-01-31 14:14 ` Jonathan Cameron
2022-01-31 22:34 ` Ben Widawsky
2022-01-24 0:29 ` [PATCH v3 10/40] cxl/core: Convert decoder range to resource Dan Williams
2022-01-24 0:29 ` [PATCH v3 11/40] cxl/core/port: Clarify decoder creation Dan Williams
2022-01-31 14:46 ` Jonathan Cameron
2022-01-31 21:17 ` Dan Williams
2022-01-31 21:33 ` [PATCH v4 " Dan Williams
2022-02-01 10:49 ` Jonathan Cameron
2022-01-24 0:29 ` [PATCH v3 12/40] cxl/core: Fix cxl_probe_component_regs() error message Dan Williams
2022-01-31 14:53 ` Jonathan Cameron
2022-01-31 22:29 ` Dan Williams
2022-01-31 22:39 ` Ben Widawsky
2022-01-24 0:29 ` [PATCH v3 13/40] cxl/core/port: Make passthrough decoder init implicit Dan Williams
2022-01-31 14:56 ` Jonathan Cameron
2022-01-24 0:29 ` [PATCH v3 14/40] cxl/core: Track port depth Dan Williams
2022-01-31 14:57 ` Jonathan Cameron
2022-01-24 0:29 ` [PATCH v3 15/40] cxl: Prove CXL locking Dan Williams
2022-01-31 15:48 ` Jonathan Cameron
2022-01-31 19:43 ` Dan Williams
2022-01-31 19:50 ` [PATCH v4 " Dan Williams
2022-01-31 23:23 ` Ben Widawsky
2022-01-24 0:30 ` [PATCH v3 16/40] cxl/core/port: Use dedicated lock for decoder target list Dan Williams
2022-01-26 2:54 ` [PATCH v4 " Dan Williams
2022-01-31 15:59 ` Jonathan Cameron
2022-01-31 23:31 ` Dan Williams
2022-01-31 23:34 ` Ben Widawsky
2022-01-31 23:38 ` Dan Williams
2022-01-31 23:42 ` Ben Widawsky
2022-01-31 23:58 ` Dan Williams
2022-01-31 23:35 ` [PATCH v5 " Dan Williams
2022-02-01 10:52 ` Jonathan Cameron
2022-01-24 0:30 ` [PATCH v3 17/40] cxl/port: Introduce cxl_port_to_pci_bus() Dan Williams
2022-01-31 16:04 ` Jonathan Cameron
2022-01-31 16:44 ` [PATCH v4 " Dan Williams
2022-01-31 23:41 ` Ben Widawsky
2022-01-24 0:30 ` [PATCH v3 18/40] cxl/pmem: Introduce a find_cxl_root() helper Dan Williams
2022-01-26 18:55 ` [PATCH v4 " Dan Williams
2022-01-26 23:59 ` [PATCH v5 " Dan Williams
2022-01-31 16:18 ` Jonathan Cameron
2022-02-01 0:22 ` Dan Williams
2022-02-01 10:58 ` Jonathan Cameron
2022-02-01 0:34 ` [PATCH v6 " Dan Williams
2022-02-01 10:59 ` Jonathan Cameron
2022-01-24 0:30 ` [PATCH v3 19/40] cxl/port: Up-level cxl_add_dport() locking requirements to the caller Dan Williams
2022-01-31 16:20 ` Jonathan Cameron
2022-01-31 23:47 ` Ben Widawsky
2022-02-01 0:43 ` Dan Williams
2022-02-01 1:07 ` [PATCH v4 " Dan Williams
2022-02-01 11:00 ` Jonathan Cameron
2022-01-24 0:30 ` [PATCH v3 20/40] cxl/pci: Rename pci.h to cxlpci.h Dan Williams
2022-01-31 16:22 ` Jonathan Cameron
2022-02-01 0:00 ` Dan Williams
2022-01-31 23:48 ` Ben Widawsky
2022-01-24 0:30 ` [PATCH v3 21/40] cxl/core: Generalize dport enumeration in the core Dan Williams
2022-01-31 17:02 ` Jonathan Cameron
2022-02-01 1:58 ` Dan Williams
2022-02-01 2:10 ` [PATCH v4 " Dan Williams
2022-02-01 11:03 ` Jonathan Cameron
2022-01-24 0:30 ` [PATCH v3 22/40] cxl/core/hdm: Add CXL standard decoder enumeration to " Dan Williams
2022-01-26 3:09 ` [PATCH v4 " Dan Williams
2022-01-31 14:26 ` Jonathan Cameron
2022-01-31 17:51 ` Jonathan Cameron
2022-02-01 5:10 ` Dan Williams
2022-02-01 20:24 ` [PATCH v5 " Dan Williams
2022-02-02 9:31 ` Jonathan Cameron
2022-02-01 0:24 ` [PATCH v3 " Ben Widawsky
2022-02-01 4:58 ` Dan Williams
2022-01-24 0:30 ` [PATCH v3 23/40] cxl/core: Emit modalias for CXL devices Dan Williams
2022-01-31 17:57 ` Jonathan Cameron
2022-02-01 15:11 ` Ben Widawsky
2022-01-24 0:30 ` [PATCH v3 24/40] cxl/port: Add a driver for 'struct cxl_port' objects Dan Williams
2022-01-26 20:16 ` [PATCH v4 " Dan Williams
2022-01-31 18:11 ` Jonathan Cameron [this message]
2022-02-01 20:43 ` Dan Williams
2022-02-02 9:33 ` Jonathan Cameron
2022-02-01 21:07 ` [PATCH v5 " Dan Williams
2022-01-24 0:30 ` [PATCH v3 25/40] cxl/core/port: Remove @host argument for dport + decoder enumeration Dan Williams
2022-01-31 14:32 ` Jonathan Cameron
2022-01-31 18:14 ` Jonathan Cameron
2022-02-01 15:17 ` Ben Widawsky
2022-02-01 21:09 ` Dan Williams
2022-02-01 21:23 ` [PATCH v4 " Dan Williams
2022-01-24 0:30 ` [PATCH v3 26/40] cxl/pci: Store component register base in cxlds Dan Williams
2022-01-31 18:15 ` Jonathan Cameron
2022-02-01 21:28 ` [PATCH v4 " Dan Williams
2022-01-24 0:31 ` [PATCH v3 27/40] cxl/pci: Cache device DVSEC offset Dan Williams
2022-01-31 18:19 ` Jonathan Cameron
2022-02-01 15:24 ` Ben Widawsky
2022-02-01 21:41 ` Dan Williams
2022-02-01 22:11 ` Ben Widawsky
2022-02-01 22:15 ` Dan Williams
2022-02-01 22:20 ` Ben Widawsky
2022-02-01 22:24 ` Dan Williams
2022-02-02 9:36 ` Jonathan Cameron
2022-02-01 22:06 ` [PATCH v4 " Dan Williams
2022-02-02 9:36 ` Jonathan Cameron
2022-01-24 0:31 ` [PATCH v3 28/40] cxl/pci: Retrieve CXL DVSEC memory info Dan Williams
2022-01-31 18:25 ` Jonathan Cameron
2022-02-01 22:52 ` Dan Williams
2022-02-01 23:48 ` [PATCH v4 " Dan Williams
2022-02-02 9:39 ` Jonathan Cameron
2022-01-24 0:31 ` [PATCH v3 29/40] cxl/pci: Implement wait for media active Dan Williams
2022-01-31 18:29 ` Jonathan Cameron
2022-02-01 23:56 ` Dan Williams
2022-01-24 0:31 ` [PATCH v3 30/40] cxl/pci: Emit device serial number Dan Williams
2022-01-31 18:33 ` Jonathan Cameron
2022-01-31 21:43 ` Dan Williams
2022-01-31 21:56 ` [PATCH v4 " Dan Williams
2022-01-24 0:31 ` [PATCH v3 31/40] cxl/memdev: Add numa_node attribute Dan Williams
2022-01-31 18:41 ` Jonathan Cameron
2022-02-01 23:57 ` Dan Williams
2022-02-02 9:44 ` Jonathan Cameron
2022-02-02 15:44 ` Dan Williams
2022-02-03 9:41 ` Jonathan Cameron
2022-02-03 16:59 ` Dan Williams
2022-02-03 18:05 ` Jonathan Cameron
2022-02-04 4:25 ` Dan Williams
2022-02-01 15:31 ` Ben Widawsky
2022-02-01 15:49 ` Jonathan Cameron
2022-02-01 16:35 ` Ben Widawsky
2022-02-01 17:38 ` Jonathan Cameron
2022-02-01 23:59 ` Dan Williams
2022-02-02 1:18 ` Dan Williams
2022-01-24 0:31 ` [PATCH v3 32/40] cxl/core/port: Add switch port enumeration Dan Williams
2022-02-01 12:13 ` Jonathan Cameron
2022-02-02 5:26 ` Dan Williams
2022-02-01 17:37 ` Ben Widawsky
2022-02-02 6:03 ` Dan Williams
2022-02-02 17:07 ` [PATCH v4 " Dan Williams
2022-02-03 9:55 ` Jonathan Cameron
2022-02-04 15:08 ` [PATCH v5 " Dan Williams
2022-01-24 0:31 ` [PATCH v3 33/40] cxl/mem: Add the cxl_mem driver Dan Williams
2022-01-26 3:16 ` [PATCH v4 " Dan Williams
2022-02-01 12:45 ` Jonathan Cameron
2022-02-01 17:44 ` Ben Widawsky
2022-02-03 2:49 ` Dan Williams
2022-02-03 9:59 ` Jonathan Cameron
2022-02-04 14:54 ` Dan Williams
2022-02-03 3:56 ` [PATCH v5 " Dan Williams
2022-02-03 12:07 ` Jonathan Cameron
2022-02-04 15:18 ` [PATCH v6 " Dan Williams
2022-01-24 0:31 ` [PATCH v3 34/40] cxl/core: Move target_list out of base decoder attributes Dan Williams
2022-01-31 18:45 ` Jonathan Cameron
2022-02-01 17:45 ` Ben Widawsky
2022-01-24 0:31 ` [PATCH v3 35/40] cxl/core/port: Add endpoint decoders Dan Williams
2022-02-01 12:47 ` Jonathan Cameron
2022-02-03 4:02 ` [PATCH v4 " Dan Williams
2022-02-14 17:45 ` Jonathan Cameron
2022-02-14 19:14 ` Dan Williams
2022-01-24 0:31 ` [PATCH v3 36/40] tools/testing/cxl: Mock dvsec_ranges() Dan Williams
2022-01-24 0:31 ` [PATCH v3 37/40] tools/testing/cxl: Fix root port to host bridge assignment Dan Williams
2022-01-24 0:32 ` [PATCH v3 38/40] tools/testing/cxl: Mock one level of switches Dan Williams
2022-01-24 0:32 ` [PATCH v3 39/40] tools/testing/cxl: Enumerate mock decoders Dan Williams
2022-01-24 0:32 ` [PATCH v3 40/40] tools/testing/cxl: Add a physical_node link Dan Williams
2022-02-01 12:53 ` Jonathan Cameron
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220131181118.00002471@Huawei.com \
--to=jonathan.cameron@huawei.com \
--cc=ben.widawsky@intel.com \
--cc=dan.j.williams@intel.com \
--cc=linux-cxl@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=lkp@intel.com \
--cc=nvdimm@lists.linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).