From: Dan Williams <dan.j.williams@intel.com>
To: Ben Widawsky <ben.widawsky@intel.com>
Cc: linux-cxl@vger.kernel.org, patches@lists.linux.dev,
Alison Schofield <alison.schofield@intel.com>,
Ira Weiny <ira.weiny@intel.com>,
Jonathan Cameron <Jonathan.Cameron@huawei.com>,
Vishal Verma <vishal.l.verma@intel.com>,
Bjorn Helgaas <helgaas@kernel.org>,
Linux NVDIMM <nvdimm@lists.linux.dev>,
Linux PCI <linux-pci@vger.kernel.org>
Subject: Re: [PATCH v3 05/14] cxl/acpi: Handle address space allocation
Date: Fri, 18 Feb 2022 11:17:28 -0800 [thread overview]
Message-ID: <CAPcyv4jq470iFeqXPVftSwomcf=aC_eLZUMhMK1JZkOxekkGxQ@mail.gmail.com> (raw)
In-Reply-To: <20220128002707.391076-6-ben.widawsky@intel.com>
On Thu, Jan 27, 2022 at 4:27 PM Ben Widawsky <ben.widawsky@intel.com> wrote:
>
> Regions are carved out of an addresses space which is claimed by top
> level decoders, and subsequently their children decoders. Regions are
s/children/descendant/
> created with a size and therefore must fit, with proper alignment, in
> that address space. The support for doing this fitting is handled by the
> driver automatically.
>
> As an example, a platform might configure a top level decoder to claim
> 1TB of address space @ 0x800000000 -> 0x10800000000; it would be
> possible to create M regions with appropriate alignment to occupy that
> address space. Each of those regions would have a host physical address
> somewhere in the range between 32G and 1.3TB, and the location will be
> determined by the logic added here.
>
> The request_region() usage is not strictly mandatory at this point as
> the actual handling of the address space is done with genpools. It is
> highly likely however that the resource/region APIs will become useful
> in the not too distant future.
More on this below, but I think resource APIs are critical for the
pre-existing / BIOS created region case and I have a feeling gen_pool
is not a good fit.
> All decoders manage a host physical address space while active. Only the
> root decoder has constraints on location and size. As a result, it makes
> most sense for the root decoder to be responsible for managing the
> entire address space, and mid-level decoders and endpoints can ask the
> root decoder for suballocations.
>
> Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
> ---
> drivers/cxl/acpi.c | 30 ++++++++++++++++++++++++++++++
> drivers/cxl/cxl.h | 2 ++
> 2 files changed, 32 insertions(+)
>
> diff --git a/drivers/cxl/acpi.c b/drivers/cxl/acpi.c
> index d6dcb2b6af48..74681bfbf53c 100644
> --- a/drivers/cxl/acpi.c
> +++ b/drivers/cxl/acpi.c
> @@ -1,6 +1,7 @@
> // SPDX-License-Identifier: GPL-2.0-only
> /* Copyright(c) 2021 Intel Corporation. All rights reserved. */
> #include <linux/platform_device.h>
> +#include <linux/genalloc.h>
> #include <linux/module.h>
> #include <linux/device.h>
> #include <linux/kernel.h>
> @@ -73,6 +74,27 @@ static int cxl_acpi_cfmws_verify(struct device *dev,
> return 0;
> }
>
> +/*
> + * Every decoder while active has an address space that it is decoding. However,
> + * only the root level decoders have fixed host physical address space ranges.
> + */
> +static int cxl_create_cfmws_address_space(struct cxl_decoder *cxld,
> + struct acpi_cedt_cfmws *cfmws)
> +{
> + const int order = ilog2(SZ_256M * cxld->interleave_ways);
> + struct device *dev = &cxld->dev;
> + struct gen_pool *pool;
> +
> + pool = devm_gen_pool_create(dev, order, NUMA_NO_NODE, dev_name(dev));
The cxld dev is not a suitable devm host.
Moreover, the address space is a generic property of root decoders, it
belongs in the core not in cxl_acpi.
As for the data structure / APIs to manage the address space I'm not
sure gen_pool is the right answer, because the capacity tracking will
be done in terms of __request_region() and resource trees. The
infrastructure to keep the gen_pool aligned with the resource tree
drops away if there was an interface for allocating free space out of
a resource tree to augment the base API of requesting space with known
addresses. In fact, there is already the request_free_mem_region()
helper. Did you consider that vs gen_pool? Otherwise, how to solve the
problem of pre-populating the busy areas of the gen_pool relative to
capacity that the BIOS may have consumed out of the decoder range?
That comes for free with just walking decoders at boot and doing
__request_region() against the root decoders. Then the allocation
helper can just walk that free space similar to
request_free_mem_region().
> + if (IS_ERR(pool))
> + return PTR_ERR(pool);
> +
> + cxld->address_space = pool;
> +
> + return gen_pool_add(cxld->address_space, cfmws->base_hpa,
> + cfmws->window_size, NUMA_NO_NODE);
> +}
> +
> struct cxl_cfmws_context {
> struct device *dev;
> struct cxl_port *root_port;
> @@ -113,6 +135,14 @@ static int cxl_parse_cfmws(union acpi_subtable_headers *header, void *arg,
> cxld->interleave_ways = CFMWS_INTERLEAVE_WAYS(cfmws);
> cxld->interleave_granularity = CFMWS_INTERLEAVE_GRANULARITY(cfmws);
>
> + rc = cxl_create_cfmws_address_space(cxld, cfmws);
> + if (rc) {
> + dev_err(dev,
> + "Failed to create CFMWS address space for decoder\n");
> + put_device(&cxld->dev);
> + return 0;
> + }
> +
> rc = cxl_decoder_add(cxld, target_map);
> if (rc)
> put_device(&cxld->dev);
> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> index d1a8ca19c9ea..b300673072f5 100644
> --- a/drivers/cxl/cxl.h
> +++ b/drivers/cxl/cxl.h
> @@ -251,6 +251,7 @@ enum cxl_decoder_type {
> * @flags: memory type capabilities and locking
> * @target_lock: coordinate coherent reads of the target list
> * @region_ida: allocator for region ids.
> + * @address_space: Used/free address space for regions.
> * @nr_targets: number of elements in @target
> * @target: active ordered target list in current decoder configuration
> */
> @@ -267,6 +268,7 @@ struct cxl_decoder {
> unsigned long flags;
> seqlock_t target_lock;
> struct ida region_ida;
> + struct gen_pool *address_space;
> int nr_targets;
> struct cxl_dport *target[];
> };
> --
> 2.35.0
>
next prev parent reply other threads:[~2022-02-18 19:17 UTC|newest]
Thread overview: 70+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-01-28 0:26 [PATCH v3 00/14] CXL Region driver Ben Widawsky
2022-01-28 0:26 ` [PATCH v3 01/14] cxl/region: Add region creation ABI Ben Widawsky
2022-01-28 18:14 ` Dan Williams
2022-01-28 18:59 ` Dan Williams
2022-02-02 18:26 ` Ben Widawsky
2022-02-02 18:28 ` Ben Widawsky
2022-02-02 18:48 ` Ben Widawsky
2022-02-02 19:00 ` Dan Williams
2022-02-02 19:02 ` Ben Widawsky
2022-02-02 19:15 ` Dan Williams
2022-02-01 22:42 ` Ben Widawsky
2022-02-01 15:53 ` Jonathan Cameron
2022-02-17 17:10 ` [PATCH v4 " Ben Widawsky
2022-02-17 17:19 ` [PATCH v5 01/15] " Ben Widawsky
2022-02-17 17:33 ` Ben Widawsky
2022-02-17 17:58 ` Dan Williams
2022-02-17 18:58 ` Ben Widawsky
2022-02-17 20:26 ` Dan Williams
2022-02-17 22:22 ` Ben Widawsky
2022-02-17 23:32 ` Dan Williams
2022-02-18 16:41 ` Ben Widawsky
2022-01-28 0:26 ` [PATCH v3 02/14] cxl/region: Introduce concept of region configuration Ben Widawsky
2022-01-29 0:25 ` Dan Williams
2022-02-01 14:59 ` Ben Widawsky
2022-02-03 5:06 ` Dan Williams
2022-02-01 23:11 ` Ben Widawsky
2022-02-03 17:48 ` Dan Williams
2022-02-03 22:23 ` Ben Widawsky
2022-02-03 23:27 ` Dan Williams
2022-02-04 0:19 ` Ben Widawsky
2022-02-04 2:45 ` Dan Williams
2022-02-17 18:36 ` Ben Widawsky
2022-02-17 19:57 ` Dan Williams
2022-02-17 20:20 ` Ben Widawsky
2022-02-17 21:12 ` Dan Williams
2022-02-23 21:49 ` Ben Widawsky
2022-02-23 22:24 ` Dan Williams
2022-02-23 22:31 ` Ben Widawsky
2022-02-23 22:42 ` Dan Williams
2022-01-28 0:26 ` [PATCH v3 03/14] cxl/mem: Cache port created by the mem dev Ben Widawsky
2022-02-17 1:20 ` Dan Williams
2022-01-28 0:26 ` [PATCH v3 04/14] cxl/region: Introduce a cxl_region driver Ben Widawsky
2022-02-01 16:21 ` Jonathan Cameron
2022-02-17 6:04 ` Dan Williams
2022-01-28 0:26 ` [PATCH v3 05/14] cxl/acpi: Handle address space allocation Ben Widawsky
2022-02-18 19:17 ` Dan Williams [this message]
2022-01-28 0:26 ` [PATCH v3 06/14] cxl/region: Address " Ben Widawsky
2022-02-18 19:51 ` Dan Williams
2022-01-28 0:27 ` [PATCH v3 07/14] cxl/region: Implement XHB verification Ben Widawsky
2022-02-18 20:23 ` Dan Williams
2022-01-28 0:27 ` [PATCH v3 08/14] cxl/region: HB port config verification Ben Widawsky
2022-02-14 16:20 ` Jonathan Cameron
2022-02-14 17:51 ` Ben Widawsky
2022-02-14 18:09 ` Jonathan Cameron
2022-02-15 16:35 ` Jonathan Cameron
2022-02-18 21:04 ` Dan Williams
2022-01-28 0:27 ` [PATCH v3 09/14] cxl/region: Add infrastructure for decoder programming Ben Widawsky
2022-02-01 18:16 ` Jonathan Cameron
2022-02-18 21:53 ` Dan Williams
2022-01-28 0:27 ` [PATCH v3 10/14] cxl/region: Collect host bridge decoders Ben Widawsky
2022-02-01 18:21 ` Jonathan Cameron
2022-02-18 23:42 ` Dan Williams
2022-01-28 0:27 ` [PATCH v3 11/14] cxl/region: Add support for single switch level Ben Widawsky
2022-02-01 18:26 ` Jonathan Cameron
2022-02-15 16:10 ` Jonathan Cameron
2022-02-18 18:23 ` Jonathan Cameron
2022-01-28 0:27 ` [PATCH v3 12/14] cxl: Program decoders for regions Ben Widawsky
2022-02-24 0:08 ` Dan Williams
2022-01-28 0:27 ` [PATCH v3 13/14] cxl/pmem: Convert nvdimm bridge API to use dev Ben Widawsky
2022-01-28 0:27 ` [PATCH v3 14/14] cxl/region: Create an nd_region Ben Widawsky
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAPcyv4jq470iFeqXPVftSwomcf=aC_eLZUMhMK1JZkOxekkGxQ@mail.gmail.com' \
--to=dan.j.williams@intel.com \
--cc=Jonathan.Cameron@huawei.com \
--cc=alison.schofield@intel.com \
--cc=ben.widawsky@intel.com \
--cc=helgaas@kernel.org \
--cc=ira.weiny@intel.com \
--cc=linux-cxl@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=nvdimm@lists.linux.dev \
--cc=patches@lists.linux.dev \
--cc=vishal.l.verma@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).