From: Ben Widawsky <ben.widawsky@intel.com>
To: linux-cxl@vger.kernel.org, Chet Douglas <chet.r.douglas@intel.com>
Cc: Ben Widawsky <ben.widawsky@intel.com>,
Alison Schofield <alison.schofield@intel.com>,
Dan Williams <dan.j.williams@intel.com>,
Ira Weiny <ira.weiny@intel.com>,
Jonathan Cameron <Jonathan.Cameron@Huawei.com>,
Vishal Verma <vishal.l.verma@intel.com>
Subject: [RFC PATCH v2 00/28] CXL Region Creation / HDM decoder programming
Date: Fri, 22 Oct 2021 11:36:41 -0700 [thread overview]
Message-ID: <20211022183709.1199701-1-ben.widawsky@intel.com> (raw)
Because v1 wasn't reviewed, and I quickly asked people to ignore it in favor of
this version, I'm not going to bother listing out the changes. I've only issued minor
edits to the original cover letter. If you already read that, feel free to move
along.
I don't have plans to send further RFCs until I receive feedback, good or bad.
CXL region creation
-------------------
An interleaved set of devices is known in the Compute Express Link [1]
specification as a region. In addition to a region being comprised of devices
that can be configured in a variety of orders there are other properties that
define a region. It is expected that a region may be created as part of
provisioning by the hardware vendor, or interactively via operating system
controls. This patch series implements both the interfaces to create and
configure a region, as well as the algorithm to program the HDM decoders to
enable CXL.mem traffic while obeying those configuration parameters. The series
stops short of storing the new regions in the Label Storage Area of the CXL
devices.
Some version of this functionality has all been posted previously, with the
exception of the actual HDM decoder programming. It's probably wise to forget
those exist, and take my apology in advance for not addressing feedback you may
have already given.
There are two branches I am using as development branches. The branch for
port/mem driver [2] is fairly solid. The branch for region creation [3] is less
baked.
cxl_port
========
The cxl_port driver is implemented within the cxl_port module. While loading of
this module is optional, the other new drivers depend on it. The port driver is
responsible for all activities around HDM decoder enumeration and programming.
Introduced earlier, the concept of a port is an abstraction over CXL components
with an upstream port, every host bridge, switch, and endpoint.
cxl_mem
=======
The cxl_mem driver's main job is to walk up the hierarchy to make the
determination if it is CXL.mem routed, meaning, all components above it in the
hierarchy are participating in the CXL.mem protocol. It is implemented within
the cxl_mem module. As the host bridge ports are added by a platform specific
driver, such as cxl_acpi, the scope of the mem driver can be reduced to scan for
switches and ask cxl_core to work on enumerating them. With this done, the
determination as to whether a device is CXL.mem routed can be done simply by
checking if the struct device has a driver bound to it.
This driver is also a logical place to migrate certain functionality from
cxl_pci. That is saved for later.
cxl_region
==========
Region verification and programming state are owned by the cxl_region driver
(implemented in the cxl_region module). It relies on cxl_mem to determine if
devices are CXL routed, and cxl_port to actually handle the programming of the
HDM decoders. Much of the region driver is an implementation of algorithms
described in the CXL Type 3 Memory Device Software Guide [4].
The patches for the region driver could be squashed. They're broken out to aid
review and because that's the order they were implemented in. My preference is
to keep those as they are.
Why RFC?
--------
While I think most of the architecture is sound, I don't believe anyone but
other developers should use this branch. Where I'd really like most eyes:
- Locking and device lifetimes. I suspect I have many bugs in this area.
- Region configuration. Should values have a default, should they all be
explicit?
- What should/shouldn't be in core. I like how this ended up, but at this point
I'm fairly biased (CXL.cache pun).
- What to extend to cxl_test.
What's missing
---------------
- CXL 2.0 switch support
- A full topology lock for programming HDM decoders. I'm looking for feedback on
the best way to do this.
- Check that HDM decoder programming addresses are correct (must program higher
addresses only)
- Volatile regions (or BIOS configured persistent regions)
- Connection to libnvdimm/labels. This includes many aspects, not the least of
which is saving the region into the Label Storage Area so that it can be
reestablished on reboot.
Here is an example of output when programming a x1 interleave region:
[ 23.959814][ T645] cxl_core:cxl_add_region:406: cxl region0.0:0: Added region0.0:0 to decoder0.0
[ 23.962972][ T645] cxl_port:cxl_commit_decoder:248: cxl_port port1: decoder1.0
[ 23.962972][ T645] Base 0x0000004c00000000
[ 23.962972][ T645] Size 268435456
[ 23.962972][ T645] IG 256
[ 23.962972][ T645] IW 1
[ 23.962972][ T645] TargetList: 0 -1 -1 -1 -1 -1 -1 -1
[ 23.965529][ T645] cxl_port:cxl_commit_decoder:248: cxl_port port3: decoder3.0
[ 23.965529][ T645] Base 0x0000004c00000000
[ 23.965529][ T645] Size 268435456
[ 23.965529][ T645] IG 256
[ 23.965529][ T645] IW 1
[ 23.965529][ T645] TargetList: -1 -1 -1 -1 -1 -1 -1 -1
If you're wondering how I tested this, I've baked it into my cxlctl app [5] and
lib [6]. Eventually this will get absorbed by ndctl/cxl-cli/libcxl [7]. Region
deletion isn't implemented yet.
To get the detailed errors, trace-cmd can be utilized. Until a region device
exists, the region module will not be loaded, which means the region tracepoints
will not exist. To get around this, modprobe cxl_region before anything.
trace-cmd record -e cxl ./cxlctl create-region -n -a -s $((256<<20)) /sys/bus/cxl/devices/decoder0.0
Note: A minor bugfix is needed in QEMU if testing interleave configs. I've
pushed that to my v4 branch.
---
[1]: https://www.computeexpresslink.org/download-the-specification
[2]: https://gitlab.com/bwidawsk/linux/-/tree/cxl_port-v3
[3]: https://gitlab.com/bwidawsk/linux/-/tree/cxl_regions-v4
[4]: https://cdrdv2.intel.com/v1/dl/getContent/643805?wapkw=CXL%20memory%20device%20sw%20guide
[5]: https://gitlab.com/bwidawsk-cxl/cxlctl
[6]: https://gitlab.com/bwidawsk-cxl/cxl_rs
[7]: https://lore.kernel.org/linux-cxl/CAPcyv4joKOhTdaRBJVeoOtqhRjBvdtt9902TS=c39=zWTZXvuw@mail.gmail.com/
---
Ben Widawsky (28):
cxl: Rename CXL_MEM to CXL_PCI
cxl: Move register block enumeration to core
cxl/acpi: Map component registers for Root Ports
cxl: Add helper for new drivers
cxl/core: Convert decoder range to resource
cxl: Introduce endpoint decoders
cxl/core: Move target population locking to caller
cxl/port: Introduce a port driver
cxl/acpi: Map single port host bridge component registers
cxl/core: Store global list of root ports
cxl/acpi: Rescan bus at probe completion
cxl/core: Store component register base for memdevs
cxl: Flesh out register names
cxl: Hide devm host for ports
cxl/core: Introduce API to scan switch ports
cxl: Introduce cxl_mem driver
cxl: Disable switch hierarchies for now
cxl/region: Add region creation ABI
cxl/region: Introduce concept of region configuration
cxl/region: Introduce a cxl_region driver
cxl/acpi: Handle address space allocation
cxl/region: Address space allocation
cxl/region: Implement XHB verification
cxl/region: HB port config verification
cxl/region: Record host bridge target list
cxl/mem: Store the endpoint's uport
cxl/region: Gather HDM decoder resources
cxl: Program decoders for regions
.clang-format | 3 +
Documentation/ABI/testing/sysfs-bus-cxl | 63 ++
.../driver-api/cxl/memory-devices.rst | 28 +
drivers/cxl/Kconfig | 28 +-
drivers/cxl/Makefile | 8 +-
drivers/cxl/acpi.c | 132 +++-
drivers/cxl/core/Makefile | 2 +
drivers/cxl/core/bus.c | 437 +++++++++++-
drivers/cxl/core/core.h | 2 +
drivers/cxl/core/memdev.c | 7 +-
drivers/cxl/core/pci.c | 99 +++
drivers/cxl/core/region.c | 453 +++++++++++++
drivers/cxl/core/regs.c | 62 +-
drivers/cxl/cxl.h | 85 ++-
drivers/cxl/cxlmem.h | 8 +-
drivers/cxl/mem.c | 162 +++++
drivers/cxl/pci.c | 69 +-
drivers/cxl/pci.h | 48 +-
drivers/cxl/port.c | 491 ++++++++++++++
drivers/cxl/region.c | 629 ++++++++++++++++++
drivers/cxl/region.h | 57 ++
drivers/cxl/trace.h | 75 +++
tools/testing/cxl/Kbuild | 2 +
tools/testing/cxl/mock_acpi.c | 4 +-
tools/testing/cxl/test/mem.c | 3 +-
25 files changed, 2806 insertions(+), 151 deletions(-)
create mode 100644 drivers/cxl/core/pci.c
create mode 100644 drivers/cxl/core/region.c
create mode 100644 drivers/cxl/mem.c
create mode 100644 drivers/cxl/port.c
create mode 100644 drivers/cxl/region.c
create mode 100644 drivers/cxl/region.h
create mode 100644 drivers/cxl/trace.h
next reply other threads:[~2021-10-22 18:37 UTC|newest]
Thread overview: 112+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-10-22 18:36 Ben Widawsky [this message]
2021-10-22 18:36 ` [RFC PATCH v2 01/28] cxl: Rename CXL_MEM to CXL_PCI Ben Widawsky
2021-10-29 20:15 ` Dan Williams
2021-10-29 21:20 ` Ben Widawsky
2021-10-29 21:39 ` Dan Williams
2021-10-22 18:36 ` [RFC PATCH v2 02/28] cxl: Move register block enumeration to core Ben Widawsky
2021-10-29 20:23 ` Dan Williams
2021-10-29 21:23 ` Ben Widawsky
2021-10-22 18:36 ` [RFC PATCH v2 03/28] cxl/acpi: Map component registers for Root Ports Ben Widawsky
2021-10-29 20:28 ` Dan Williams
2021-10-22 18:36 ` [RFC PATCH v2 04/28] cxl: Add helper for new drivers Ben Widawsky
2021-10-29 20:30 ` Dan Williams
2021-10-22 18:36 ` [RFC PATCH v2 05/28] cxl/core: Convert decoder range to resource Ben Widawsky
2021-10-29 20:50 ` Dan Williams
2021-10-29 21:26 ` Ben Widawsky
2021-10-29 22:22 ` Dan Williams
2021-10-29 22:37 ` Ben Widawsky
2021-11-01 14:33 ` Ben Widawsky
2021-10-22 18:36 ` [RFC PATCH v2 06/28] cxl: Introduce endpoint decoders Ben Widawsky
2021-10-29 21:00 ` Dan Williams
2021-10-29 22:02 ` Ben Widawsky
2021-10-29 22:25 ` Dan Williams
2021-10-22 18:36 ` [RFC PATCH v2 07/28] cxl/core: Move target population locking to caller Ben Widawsky
2021-10-29 23:03 ` Dan Williams
2021-10-22 18:36 ` [RFC PATCH v2 08/28] cxl/port: Introduce a port driver Ben Widawsky
2021-10-30 1:37 ` Dan Williams
2021-10-31 17:53 ` Dan Williams
2021-10-31 18:10 ` Dan Williams
2021-11-01 17:36 ` Ben Widawsky
2021-11-01 17:53 ` Ben Widawsky
2021-11-01 17:54 ` Ben Widawsky
2021-11-02 3:31 ` Dan Williams
2021-11-02 16:27 ` Ben Widawsky
2021-11-02 17:21 ` Dan Williams
2021-11-02 16:58 ` Ben Widawsky
2021-11-04 19:10 ` Dan Williams
2021-11-04 19:49 ` Ben Widawsky
2021-11-04 20:04 ` Dan Williams
2021-11-04 21:25 ` Ben Widawsky
2021-11-04 16:37 ` Ben Widawsky
2021-11-04 19:17 ` Dan Williams
2021-11-04 19:46 ` Ben Widawsky
2021-11-04 20:00 ` Dan Williams
2021-11-04 21:26 ` Ben Widawsky
2021-11-03 15:18 ` Jonathan Cameron
2021-10-22 18:36 ` [RFC PATCH v2 09/28] cxl/acpi: Map single port host bridge component registers Ben Widawsky
2021-10-31 18:03 ` Dan Williams
2021-11-01 17:07 ` Ben Widawsky
2021-11-02 2:15 ` Dan Williams
2021-11-02 16:31 ` Ben Widawsky
2021-11-02 17:46 ` Dan Williams
2021-11-02 17:57 ` Ben Widawsky
2021-11-02 18:10 ` Dan Williams
2021-11-02 18:27 ` Ben Widawsky
2021-11-02 18:49 ` Dan Williams
2021-11-02 21:15 ` Ben Widawsky
2021-11-02 21:34 ` Dan Williams
2021-11-02 21:47 ` Ben Widawsky
2021-10-22 18:36 ` [RFC PATCH v2 10/28] cxl/core: Store global list of root ports Ben Widawsky
2021-10-31 18:32 ` Dan Williams
2021-11-01 18:43 ` Ben Widawsky
2021-11-02 2:04 ` Dan Williams
2021-10-22 18:36 ` [RFC PATCH v2 11/28] cxl/acpi: Rescan bus at probe completion Ben Widawsky
2021-10-31 19:25 ` Dan Williams
2021-11-01 18:56 ` Ben Widawsky
2021-11-01 21:45 ` Ben Widawsky
2021-11-02 1:56 ` Dan Williams
2021-10-22 18:36 ` [RFC PATCH v2 12/28] cxl/core: Store component register base for memdevs Ben Widawsky
2021-10-31 20:13 ` Dan Williams
2021-11-01 21:50 ` Ben Widawsky
2021-10-22 18:36 ` [RFC PATCH v2 13/28] cxl: Flesh out register names Ben Widawsky
2021-10-31 20:18 ` Dan Williams
2021-11-01 22:00 ` Ben Widawsky
2021-11-02 1:53 ` Dan Williams
2021-11-03 15:53 ` Jonathan Cameron
2021-11-03 16:03 ` Ben Widawsky
2021-11-03 16:42 ` Jonathan Cameron
2021-11-03 17:05 ` Ben Widawsky
2021-10-22 18:36 ` [RFC PATCH v2 14/28] cxl: Hide devm host for ports Ben Widawsky
2021-10-31 21:14 ` Dan Williams
2021-10-22 18:36 ` [RFC PATCH v2 15/28] cxl/core: Introduce API to scan switch ports Ben Widawsky
2021-11-01 5:39 ` Dan Williams
2021-11-01 22:56 ` Ben Widawsky
2021-11-02 1:45 ` Dan Williams
2021-11-02 16:39 ` Ben Widawsky
2021-11-02 20:00 ` Dan Williams
2021-11-16 16:50 ` Ben Widawsky
2021-11-16 17:51 ` Dan Williams
2021-11-16 18:02 ` Ben Widawsky
2021-11-03 16:08 ` Jonathan Cameron
2021-11-10 17:49 ` Ben Widawsky
2021-11-10 18:10 ` Jonathan Cameron
2021-11-10 21:03 ` Dan Williams
2021-10-22 18:36 ` [RFC PATCH v2 16/28] cxl: Introduce cxl_mem driver Ben Widawsky
2021-10-22 18:36 ` [RFC PATCH v2 17/28] cxl: Disable switch hierarchies for now Ben Widawsky
2021-10-22 18:36 ` [RFC PATCH v2 18/28] cxl/region: Add region creation ABI Ben Widawsky
2021-10-22 18:37 ` [RFC PATCH v2 19/28] cxl/region: Introduce concept of region configuration Ben Widawsky
2021-12-15 17:47 ` Jonathan Cameron
2021-10-22 18:37 ` [RFC PATCH v2 20/28] cxl/region: Introduce a cxl_region driver Ben Widawsky
2021-10-22 18:37 ` [RFC PATCH v2 21/28] cxl/acpi: Handle address space allocation Ben Widawsky
2021-10-22 18:37 ` [RFC PATCH v2 22/28] cxl/region: Address " Ben Widawsky
2021-10-22 18:37 ` [RFC PATCH v2 23/28] cxl/region: Implement XHB verification Ben Widawsky
2022-01-06 16:55 ` Jonathan Cameron
2022-01-06 16:58 ` Ben Widawsky
2022-01-06 17:33 ` Jonathan Cameron
2022-01-06 18:10 ` Jonathan Cameron
2022-01-06 18:34 ` Ben Widawsky
2021-10-22 18:37 ` [RFC PATCH v2 24/28] cxl/region: HB port config verification Ben Widawsky
2021-10-22 18:37 ` [RFC PATCH v2 25/28] cxl/region: Record host bridge target list Ben Widawsky
2021-10-22 18:37 ` [RFC PATCH v2 26/28] cxl/mem: Store the endpoint's uport Ben Widawsky
2021-10-22 18:37 ` [RFC PATCH v2 27/28] cxl/region: Gather HDM decoder resources Ben Widawsky
2021-10-22 18:37 ` [RFC PATCH v2 28/28] cxl: Program decoders for regions Ben Widawsky
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20211022183709.1199701-1-ben.widawsky@intel.com \
--to=ben.widawsky@intel.com \
--cc=Jonathan.Cameron@Huawei.com \
--cc=alison.schofield@intel.com \
--cc=chet.r.douglas@intel.com \
--cc=dan.j.williams@intel.com \
--cc=ira.weiny@intel.com \
--cc=linux-cxl@vger.kernel.org \
--cc=vishal.l.verma@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.