From: Ben Widawsky <ben.widawsky@intel.com>
To: linux-cxl@vger.kernel.org, linux-nvdimm@lists.01.org,
linux-pci@vger.kernel.org
Cc: patches@lists.linux.dev, Bjorn Helgaas <helgaas@kernel.org>,
Alison Schofield <alison.schofield@intel.com>,
Dan Williams <dan.j.williams@intel.com>,
Ira Weiny <ira.weiny@intel.com>,
Jonathan Cameron <Jonathan.Cameron@huawei.com>,
Vishal Verma <vishal.l.verma@intel.com>
Subject: Re: [PATCH 00/13] CXL Region driver
Date: Sat, 15 Jan 2022 10:54:09 -0800 [thread overview]
Message-ID: <20220115185409.tllecpbztzblr3nc@intel.com> (raw)
In-Reply-To: <20220107003756.806582-1-ben.widawsky@intel.com>
Hi all.
I found some last minute bugs which I've fixed. Please ignore this series and
wait until I send out v3.
Thanks.
Ben
On 22-01-06 16:37:43, Ben Widawsky wrote:
> This patch series introduces the CXL region driver as well as associated APIs in
> CXL core. The region driver enables the creation of "regions" which is a concept
> defined by the CXL 2.0 specification [1]. Region verification and programming
> state are owned by the cxl_region driver (implemented in the cxl_region module).
> It relies on cxl_mem to determine if devices are CXL routed, and cxl_port to
> actually handle the programming of the HDM decoders. Much of the region driver
> is an implementation of algorithms described in the CXL Type 3 Memory Device
> Software Guide [2].
>
> The region driver will be responsible for configuring regions found on
> persistent capacities in the Label Storage Area (LSA), it will also enumerate
> regions configured by BIOS, usually volatile capacities, and will allow for
> dynamic region creation (which can then be stored in the LSA). It is the primary
> consumer of the CXL Port [3] and CXL Mem drivers introduced previously [4]. Dan
> as reworked some of this which is required for this branch. A cached copy is
> included in the gitlab for this project [5]. Those patches will be posted
> shortly.
>
> The patches for the region driver could be squashed. They're broken out to aid
> review and because that's the order they were implemented in. My preference is
> to keep those as they are.
>
> Some things are still missing and will be worked on while these are reviewed (in
> priority order):
> 1. Connection to libnvdimm labels (No plan)
> 2. Volatile regions/LSA regions (Have a plan)
> 3. Switch ports (Have a plan)
> 4. Decoder programming restrictions (No plan). The one know restriction I've
> missed is to disallow programming HDM decoders that aren't in incremental
> system physical address ranges.
> 5. Stress testing
>
> Here is an example of output when programming a x2 interleave region:
> ./cxlctl create-region -i2 -n -a -s $((512<<20)) /sys/bus/cxl/devices/decoder0.0
> [ 42.971496][ T644] cxl_core:cxl_bus_probe:1384: cxl_region region0.0:0: probe: -19
> [ 42.972400][ T644] cxl_core:cxl_add_region:478: cxl region0.0:0: Added region0.0:0 to decoder0.0
> [ 42.979388][ T644] cxl_core:cxl_commit_decoder:394: cxl_port port1: decoder1.0
> [ 42.979388][ T644] Base 0x0000004c00000000
> [ 42.979388][ T644] Size 536870912
> [ 42.979388][ T644] IG 8
> [ 42.979388][ T644] IW 2
> [ 42.979388][ T644] TargetList: 0 1 -1 -1 -1 -1 -1 -1
> [ 42.982410][ T644] cxl_core:cxl_commit_decoder:394: cxl_port endpoint3: decoder3.0
> [ 42.982410][ T644] Base 0x0000004c00000000
> [ 42.982410][ T644] Size 536870912
> [ 42.982410][ T644] IG 8
> [ 42.982410][ T644] IW 2
> [ 42.982410][ T644] TargetList: -1 -1 -1 -1 -1 -1 -1 -1
> [ 42.985427][ T644] cxl_core:cxl_commit_decoder:394: cxl_port endpoint2: decoder2.0
> [ 42.985427][ T644] Base 0x0000004c00000000
> [ 42.985427][ T644] Size 536870912
> [ 42.985427][ T644] IG 8
> [ 42.985427][ T644] IW 2
> [ 42.985427][ T644] TargetList: -1 -1 -1 -1 -1 -1 -1 -1
> [ 42.987937][ T644] cxl_core:cxl_bus_probe:1384: cxl_region region0.0:0: probe: 0
>
> If you're wondering how I tested this, I've baked it into my cxlctl app [6] and
> lib [7]. Eventually this will get absorbed by ndctl/cxl-cli/libcxl.
>
> To get the detailed errors, trace-cmd can be utilized. Until a region device
> exists, the region module will not be loaded, which means the region tracepoints
> will not exist. To get around this, modprobe cxl_region before anything.
>
> trace-cmd record -e cxl ./cxlctl create-region -n -a -s $((256<<20)) /sys/bus/cxl/devices/decoder0.0
>
> Branch can be found at gitlab [8].
>
> ---
>
> [1]: https://www.computeexpresslink.org/download-the-specification
> [2]: https://cdrdv2.intel.com/v1/dl/getContent/643805?wapkw=CXL%20memory%20device%20sw%20guide
> [3]: https://lore.kernel.org/linux-cxl/20211022183709.1199701-9-ben.widawsky@intel.com/
> [4]: https://lore.kernel.org/linux-cxl/20211022183709.1199701-17-ben.widawsky@intel.com/
> [5]: https://gitlab.com/bwidawsk/linux/-/commit/126793e22427f7975a8f2fca373764be78012e88
> [6]: https://gitlab.com/bwidawsk-cxl/cxlctl
> [7]: https://gitlab.com/bwidawsk-cxl/cxl_rs
> [8]: https://gitlab.com/bwidawsk/linux/-/tree/cxl_region
>
> ---
>
> Ben Widawsky (13):
> cxl/core: Rename find_cxl_port
> cxl/core: Track port depth
> cxl/region: Add region creation ABI
> cxl/region: Introduce concept of region configuration
> cxl/mem: Cache port created by the mem dev
> cxl/region: Introduce a cxl_region driver
> cxl/acpi: Handle address space allocation
> cxl/region: Address space allocation
> cxl/region: Implement XHB verification
> cxl/region: HB port config verification
> cxl/region: Add infrastructure for decoder programming
> cxl/region: Record host bridge target list
> cxl: Program decoders for regions
>
> .clang-format | 3 +
> Documentation/ABI/testing/sysfs-bus-cxl | 63 ++
> .../driver-api/cxl/memory-devices.rst | 14 +
> drivers/cxl/Makefile | 2 +
> drivers/cxl/acpi.c | 30 +
> drivers/cxl/core/Makefile | 1 +
> drivers/cxl/core/core.h | 4 +
> drivers/cxl/core/hdm.c | 198 +++++
> drivers/cxl/core/port.c | 132 +++-
> drivers/cxl/core/region.c | 525 +++++++++++++
> drivers/cxl/cxl.h | 32 +
> drivers/cxl/cxlmem.h | 9 +
> drivers/cxl/mem.c | 16 +-
> drivers/cxl/port.c | 42 +-
> drivers/cxl/region.c | 695 ++++++++++++++++++
> drivers/cxl/region.h | 47 ++
> drivers/cxl/trace.h | 54 ++
> tools/testing/cxl/Kbuild | 1 +
> 18 files changed, 1849 insertions(+), 19 deletions(-)
> create mode 100644 drivers/cxl/core/region.c
> create mode 100644 drivers/cxl/region.c
> create mode 100644 drivers/cxl/region.h
> create mode 100644 drivers/cxl/trace.h
>
>
> base-commit: 03716ce2db3c17ba38f26a88d75049c0472a629e
> prerequisite-patch-id: af6a0315e22bfc1099d4f58610b8b897e6e5a060
> --
> 2.34.1
>
prev parent reply other threads:[~2022-01-15 18:54 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-01-07 0:37 [PATCH 00/13] CXL Region driver Ben Widawsky
2022-01-07 0:37 ` [PATCH 01/13] cxl/core: Rename find_cxl_port Ben Widawsky
2022-01-07 0:37 ` [PATCH 02/13] cxl/core: Track port depth Ben Widawsky
2022-01-07 0:37 ` [PATCH 03/13] cxl/region: Add region creation ABI Ben Widawsky
2022-01-07 0:37 ` [PATCH 04/13] cxl/region: Introduce concept of region configuration Ben Widawsky
2022-01-07 0:37 ` [PATCH 05/13] cxl/mem: Cache port created by the mem dev Ben Widawsky
2022-01-07 0:37 ` [PATCH 06/13] cxl/region: Introduce a cxl_region driver Ben Widawsky
2022-01-07 0:37 ` [PATCH 07/13] cxl/acpi: Handle address space allocation Ben Widawsky
2022-01-07 0:37 ` [PATCH 08/13] cxl/region: Address " Ben Widawsky
2022-01-07 0:37 ` [PATCH 09/13] cxl/region: Implement XHB verification Ben Widawsky
2022-01-07 10:07 ` Jonathan Cameron
2022-01-07 11:55 ` Jonathan Cameron
2022-01-11 22:47 ` Ben Widawsky
2022-01-07 10:30 ` Jonathan Cameron
2022-01-07 10:38 ` Jonathan Cameron
2022-01-07 16:32 ` Ben Widawsky
2022-01-11 21:32 ` Ben Widawsky
2022-01-07 0:37 ` [PATCH 10/13] cxl/region: HB port config verification Ben Widawsky
2022-01-07 0:37 ` [PATCH 11/13] cxl/region: Add infrastructure for decoder programming Ben Widawsky
2022-01-07 0:37 ` [PATCH 12/13] cxl/region: Record host bridge target list Ben Widawsky
2022-01-07 18:14 ` Jonathan Cameron
2022-01-07 19:20 ` Dan Williams
2022-01-07 0:37 ` [PATCH 13/13] cxl: Program decoders for regions Ben Widawsky
2022-01-07 16:18 ` Jonathan Cameron
2022-01-07 16:33 ` Ben Widawsky
2022-01-07 17:22 ` Jonathan Cameron
2022-01-11 0:05 ` Ben Widawsky
2022-01-12 14:53 ` Jonathan Cameron
2022-01-12 16:54 ` Ben Widawsky
2022-01-15 18:54 ` Ben Widawsky [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220115185409.tllecpbztzblr3nc@intel.com \
--to=ben.widawsky@intel.com \
--cc=Jonathan.Cameron@huawei.com \
--cc=alison.schofield@intel.com \
--cc=dan.j.williams@intel.com \
--cc=helgaas@kernel.org \
--cc=ira.weiny@intel.com \
--cc=linux-cxl@vger.kernel.org \
--cc=linux-nvdimm@lists.01.org \
--cc=linux-pci@vger.kernel.org \
--cc=patches@lists.linux.dev \
--cc=vishal.l.verma@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).