linux-pci.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Jonathan Cameron <Jonathan.Cameron@Huawei.com>
To: Dan Williams <dan.j.williams@intel.com>
Cc: <linux-cxl@vger.kernel.org>, <nvdimm@lists.linux.dev>,
	<linux-pci@vger.kernel.org>, <patches@lists.linux.dev>,
	<hch@lst.de>, "Ben Widawsky" <bwidawsk@kernel.org>
Subject: Re: [PATCH 36/46] cxl/region: Add interleave ways attribute
Date: Tue, 19 Jul 2022 15:47:18 +0100	[thread overview]
Message-ID: <20220719154718.000077ec@Huawei.com> (raw)
In-Reply-To: <62cb6f9a74b33_3535162944e@dwillia2-xfh.notmuch>

On Sun, 10 Jul 2022 17:32:26 -0700
Dan Williams <dan.j.williams@intel.com> wrote:

> Jonathan Cameron wrote:
> > On Thu, 23 Jun 2022 21:19:40 -0700
> > Dan Williams <dan.j.williams@intel.com> wrote:
> >   
> > > From: Ben Widawsky <bwidawsk@kernel.org>
> > > 
> > > Add an ABI to allow the number of devices that comprise a region to be
> > > set.  
> > 
> > Should at least mention interleave_granularity is being added as well!  
> 
> Added.
> 
> >   
> > > 
> > > Signed-off-by: Ben Widawsky <bwidawsk@kernel.org>
> > > [djbw: reword changelog]
> > > Signed-off-by: Dan Williams <dan.j.williams@intel.com>  
> > 
> > Random diversion inline...
> >   
> > > ---
> > >  Documentation/ABI/testing/sysfs-bus-cxl |  21 ++++
> > >  drivers/cxl/core/region.c               | 128 ++++++++++++++++++++++++
> > >  drivers/cxl/cxl.h                       |  33 ++++++
> > >  3 files changed, 182 insertions(+)  
> >   
> > > diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
> > > index f75978f846b9..78af42454760 100644
> > > --- a/drivers/cxl/core/region.c
> > > +++ b/drivers/cxl/core/region.c
> > > @@ -7,6 +7,7 @@  
> > 
> >   
> > > +static ssize_t interleave_granularity_store(struct device *dev,
> > > +					    struct device_attribute *attr,
> > > +					    const char *buf, size_t len)
> > > +{
> > > +	struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(dev->parent);
> > > +	struct cxl_decoder *cxld = &cxlrd->cxlsd.cxld;
> > > +	struct cxl_region *cxlr = to_cxl_region(dev);
> > > +	struct cxl_region_params *p = &cxlr->params;
> > > +	int rc, val;
> > > +	u16 ig;
> > > +
> > > +	rc = kstrtoint(buf, 0, &val);
> > > +	if (rc)
> > > +		return rc;
> > > +
> > > +	rc = granularity_to_cxl(val, &ig);
> > > +	if (rc)
> > > +		return rc;
> > > +
> > > +	/* region granularity must be >= root granularity */  
> > 
> > In general I think that's an implementation choice.  Sure today
> > we only support it this way, but it's perfectly possible to build
> > setups where that's not the case.  
> 
> If the region granularity is smaller than the host bridge interleave
> granularity it means that multiple devices per host bridge are needed to
> satsify a single "slot" in the interleave. Valid? Yes. Useful for Linux
> to support, not clear.

True.  Wait and see on this one makes sense to me. I only noticed because
my older test scripts (against hacks on top of Ben's code) were broken as
I did it the silly way :)

> 
> > Maybe the comment should say that this code goes with an
> > implementation choice inline with the software guide (that argues you
> > will always prefer small ig for interleaving at the host to make best
> > use of bandwidth etc).  
> 
> No, I would prefer that as far as the Linux implementation is concerned
> the software-guide does not exist. In the sense that the Linux
> implementation choices supersede and are otherwise a superset of what
> the guide recommends.

ah. I phrased that badly. I just meant lift the argument as a comment rather
than a cross reference.

> 
> Also, for the same reason that the code does not anticipate future
> implementation possibilities, neither should the comments. It is
> sufficient to just change this comment when / if the implemetation stops
> expecting region granularity >= root granularity.
> 
> > Interestingly the code I was previously testing QEMU with
> > allowed that option (might have been only option that worked).
> > That code was a mixture of Ben's earlier version and my own hacks.
> > It probably doesn't make sense to support other ways of picking
> > the interleaving granularity until / if we ever get a request
> > to do so. 
> > 
> > I think it results in a different device ordering.
> > 
> > Ordering with this
> > 
> >     Host
> >      | 4k
> >     / \
> >    /   \  
> >   HB   HB  8k
> >   |     |
> >  / \   / \
> > 0  2   1  3
> > 
> > Ordering with Larger granularity CFMWS over finer granularity HB
> > 
> >     Host
> >      | 8k
> >     / \
> >    /   \ 
> >   HB   HB 4k
> >   |     |
> >  / \   / \
> > 0  1   2  3
> > 
> > Not clear why you'd do the second one though :)  So can ignore for now.  
> 
> All I can think of is "ZOMG! My platform failed and the only one I have
> to recover my data has HB interleaves with larger granularity than my
> failed system!". Otherwise, I expect cross-platform CXL persistent
> memory recovery to be so rare as to not need to spend time too much time
> worrying about it now. It seems a straightforward constraint to lift at
> a later date without any risk to breaking the ABI.

It was cross platform that I was thinking but you make a fair point that
it is unlikely to occur that often.  + If another OS want's to do it wrong
that's their problem :)

J

  reply	other threads:[~2022-07-19 14:47 UTC|newest]

Thread overview: 157+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-06-24  2:45 [PATCH 00/46] CXL PMEM Region Provisioning Dan Williams
2022-06-24  2:45 ` [PATCH 01/46] tools/testing/cxl: Fix cxl_hdm_decode_init() calling convention Dan Williams
2022-06-28 10:37   ` Jonathan Cameron
     [not found]   ` <CGME20220629174147uscas1p211384ae262e099484440ef285be26c75@uscas1p2.samsung.com>
2022-06-29 17:41     ` Adam Manzanares
2022-07-09 20:06       ` Dan Williams
2022-07-12 22:11         ` Adam Manzanares
2022-06-24  2:45 ` [PATCH 02/46] cxl/port: Keep port->uport valid for the entire life of a port Dan Williams
2022-06-24  3:37   ` Alison Schofield
2022-06-28 11:47   ` Jonathan Cameron
2022-06-28 14:27     ` Dan Williams
     [not found]   ` <CGME20220629174622uscas1p2236a084ce25771a3ab57c6f006632f35@uscas1p2.samsung.com>
2022-06-29 17:46     ` Adam Manzanares
2022-06-24  2:45 ` [PATCH 03/46] cxl/hdm: Use local hdm variable Dan Williams
2022-06-24  3:38   ` Alison Schofield
2022-06-28 15:16   ` Jonathan Cameron
     [not found]   ` <CGME20220629200312uscas1p292303b9325dcbfe59293f002dc9e6b03@uscas1p2.samsung.com>
2022-06-29 20:03     ` Adam Manzanares
2022-06-24  2:45 ` [PATCH 04/46] cxl/core: Rename ->decoder_range ->hpa_range Dan Williams
2022-06-24  3:39   ` Alison Schofield
2022-06-28 15:17   ` Jonathan Cameron
     [not found]   ` <CGME20220629200652uscas1p2c1da644ea63a5de69e14e046379779b1@uscas1p2.samsung.com>
2022-06-29 20:06     ` Adam Manzanares
2022-06-24  2:45 ` [PATCH 05/46] cxl/core: Drop ->platform_res attribute for root decoders Dan Williams
2022-06-28 15:24   ` Jonathan Cameron
2022-07-09 23:33     ` Dan Williams
     [not found]   ` <CGME20220629202117uscas1p2892fb68ae60c4754e2f7d26882a92ae5@uscas1p2.samsung.com>
2022-06-29 20:21     ` Adam Manzanares
2022-07-09 23:38       ` Dan Williams
2022-06-24  2:45 ` [PATCH 06/46] cxl/core: Drop is_cxl_decoder() Dan Williams
2022-06-24  3:48   ` Alison Schofield
2022-06-28 15:25   ` Jonathan Cameron
     [not found]   ` <CGME20220629203448uscas1p264a7f79a1ed7f9257eefcb3064c7d943@uscas1p2.samsung.com>
2022-06-29 20:34     ` Adam Manzanares
2022-06-24  2:45 ` [PATCH 07/46] cxl: Introduce cxl_to_{ways,granularity} Dan Williams
2022-06-28 15:36   ` Jonathan Cameron
2022-07-09 23:52     ` Dan Williams
2022-06-24  2:45 ` [PATCH 08/46] cxl/core: Define a 'struct cxl_switch_decoder' Dan Williams
2022-06-28 16:12   ` Jonathan Cameron
2022-06-30 10:56     ` Jonathan Cameron
2022-07-10  0:49       ` Dan Williams
2022-07-10  0:33     ` Dan Williams
2022-06-24  2:46 ` [PATCH 09/46] cxl/acpi: Track CXL resources in iomem_resource Dan Williams
2022-06-28 16:43   ` Jonathan Cameron
2022-07-10  2:12     ` Dan Williams
2022-07-19 14:24       ` Jonathan Cameron
2022-06-24  2:46 ` [PATCH 10/46] cxl/core: Define a 'struct cxl_root_decoder' for tracking CXL window resources Dan Williams
2022-06-28 16:49   ` Jonathan Cameron
2022-07-10  2:20     ` Dan Williams
2022-06-28 16:53   ` Jonathan Cameron
2022-06-24  2:46 ` [PATCH 11/46] cxl/core: Define a 'struct cxl_endpoint_decoder' for tracking DPA resources Dan Williams
2022-06-28 16:55   ` Jonathan Cameron
2022-07-10  2:40     ` Dan Williams
2022-06-24  2:46 ` [PATCH 12/46] cxl/mem: Convert partition-info to resources Dan Williams
2022-06-28 17:02   ` Jonathan Cameron
2022-06-24  2:46 ` [PATCH 13/46] cxl/hdm: Require all decoders to be enumerated Dan Williams
2022-06-28 17:04   ` Jonathan Cameron
2022-06-24  2:46 ` [PATCH 14/46] cxl/hdm: Enumerate allocated DPA Dan Williams
2022-06-29 14:43   ` Jonathan Cameron
2022-07-10  3:03     ` Dan Williams
2022-07-19 14:25       ` Jonathan Cameron
2022-06-24  2:46 ` [PATCH 15/46] cxl/Documentation: List attribute permissions Dan Williams
2022-06-28  3:16   ` Alison Schofield
2022-06-29 14:59   ` Jonathan Cameron
2022-06-24  2:46 ` [PATCH 16/46] cxl/hdm: Add 'mode' attribute to decoder objects Dan Williams
2022-06-29 15:28   ` Jonathan Cameron
2022-07-10  3:45     ` Dan Williams
2022-06-24  2:47 ` [PATCH 17/46] cxl/hdm: Track next decoder to allocate Dan Williams
2022-06-29 15:31   ` Jonathan Cameron
2022-07-10  3:55     ` Dan Williams
2022-07-19 14:27       ` Jonathan Cameron
2022-07-10 16:34     ` Dan Williams
2022-06-24  2:47 ` [PATCH 18/46] cxl/hdm: Add support for allocating DPA to an endpoint decoder Dan Williams
2022-06-29 15:56   ` Jonathan Cameron
2022-07-10 16:53     ` Dan Williams
2022-06-24  2:47 ` [PATCH 19/46] cxl/debug: Move debugfs init to cxl_core_init() Dan Williams
2022-06-29 15:58   ` Jonathan Cameron
2022-06-24  2:47 ` [PATCH 20/46] cxl/mem: Add a debugfs version of 'iomem' for DPA, 'dpamem' Dan Williams
2022-06-29 16:08   ` Jonathan Cameron
2022-07-10 17:09     ` Dan Williams
2022-06-24  2:47 ` [PATCH 21/46] tools/testing/cxl: Move cxl_test resources to the top of memory Dan Williams
2022-06-29 16:11   ` Jonathan Cameron
2022-07-10 17:19     ` Dan Williams
2022-06-24  2:47 ` [PATCH 22/46] tools/testing/cxl: Expand CFMWS windows Dan Williams
2022-06-29 16:14   ` Jonathan Cameron
2022-06-24  2:47 ` [PATCH 23/46] tools/testing/cxl: Add partition support Dan Williams
2022-06-29 16:20   ` Jonathan Cameron
2022-06-24  2:48 ` [PATCH 24/46] tools/testing/cxl: Fix decoder default state Dan Williams
2022-06-29 16:22   ` Jonathan Cameron
2022-07-10 17:33     ` Dan Williams
2022-06-24  2:48 ` [PATCH 25/46] cxl/port: Record dport in endpoint references Dan Williams
2022-06-29 16:49   ` Jonathan Cameron
2022-07-10 18:40     ` Dan Williams
2022-06-24  4:19 ` [PATCH 26/46] cxl/port: Record parent dport when adding ports Dan Williams
2022-06-29 17:02   ` Jonathan Cameron
2022-06-24  4:19 ` [PATCH 27/46] cxl/port: Move 'cxl_ep' references to an xarray per port Dan Williams
2022-06-29 17:19   ` Jonathan Cameron
2022-06-24  4:19 ` [PATCH 28/46] cxl/port: Move dport tracking to an xarray Dan Williams
2022-06-30  9:18   ` Jonathan Cameron
2022-07-10 19:06     ` Dan Williams
2022-06-24  4:19 ` [PATCH 29/46] cxl/port: Cache CXL host bridge data Dan Williams
2022-06-30  9:21   ` Jonathan Cameron
2022-07-10 19:09     ` Dan Williams
2022-06-24  4:19 ` [PATCH 30/46] cxl/hdm: Add sysfs attributes for interleave ways + granularity Dan Williams
2022-06-30  9:26   ` Jonathan Cameron
2022-07-10 20:40     ` Dan Williams
2022-07-19 14:32       ` Jonathan Cameron
2022-06-24  4:19 ` [PATCH 31/46] cxl/hdm: Initialize decoder type for memory expander devices Dan Williams
2022-06-30  9:33   ` Jonathan Cameron
2022-06-24  4:19 ` [PATCH 32/46] cxl/mem: Enumerate port targets before adding endpoints Dan Williams
2022-06-30  9:48   ` Jonathan Cameron
2022-07-10 21:01     ` Dan Williams
2022-06-24  4:19 ` [PATCH 33/46] resource: Introduce alloc_free_mem_region() Dan Williams
2022-06-30 10:35   ` Jonathan Cameron
2022-07-10 21:58     ` Dan Williams
2022-06-24  4:19 ` [PATCH 34/46] cxl/region: Add region creation support Dan Williams
2022-06-30 13:17   ` Jonathan Cameron
2022-07-11  0:08     ` Dan Williams
2022-07-19 14:42       ` Jonathan Cameron
2022-06-24  4:19 ` [PATCH 35/46] cxl/region: Add a 'uuid' attribute Dan Williams
2022-06-28 10:29   ` Jonathan Cameron
2022-06-28 14:24     ` Dan Williams
2022-06-24  4:19 ` [PATCH 36/46] cxl/region: Add interleave ways attribute Dan Williams
2022-06-30 13:44   ` Jonathan Cameron
2022-07-11  0:32     ` Dan Williams
2022-07-19 14:47       ` Jonathan Cameron [this message]
2022-07-19 22:15         ` Dan Williams
2022-07-20  9:59           ` Jonathan Cameron
2022-06-30 13:45   ` Jonathan Cameron
2022-06-24  4:19 ` [PATCH 37/46] cxl/region: Allocate host physical address (HPA) capacity to new regions Dan Williams
2022-06-30 13:56   ` Jonathan Cameron
2022-07-11  0:47     ` Dan Williams
2022-06-24  4:19 ` [PATCH 38/46] cxl/region: Enable the assignment of endpoint decoders to regions Dan Williams
2022-06-30 14:31   ` Jonathan Cameron
2022-07-11  1:12     ` Dan Williams
2022-06-24  4:19 ` [PATCH 39/46] cxl/acpi: Add a host-bridge index lookup mechanism Dan Williams
2022-06-30 15:48   ` Jonathan Cameron
2022-06-24  4:19 ` [PATCH 40/46] cxl/region: Attach endpoint decoders Dan Williams
2022-06-24 18:25   ` Jonathan Cameron
2022-06-24 18:49     ` Dan Williams
2022-06-24 20:51     ` Dan Williams
2022-06-24 23:21       ` Dan Williams
2022-06-30 16:34   ` Jonathan Cameron
2022-07-11  2:02     ` Dan Williams
2022-06-24  4:19 ` [PATCH 41/46] cxl/region: Program target lists Dan Williams
2022-06-24  4:19 ` [PATCH 42/46] cxl/hdm: Commit decoder state to hardware Dan Williams
2022-06-30 17:05   ` Jonathan Cameron
2022-07-11  3:02     ` Dan Williams
2022-06-24  4:19 ` [PATCH 43/46] cxl/region: Add region driver boiler plate Dan Williams
2022-06-30 17:09   ` Jonathan Cameron
2022-06-24  4:19 ` [PATCH 44/46] cxl/pmem: Delete unused nvdimm attribute Dan Williams
2022-06-30 17:10   ` Jonathan Cameron
2022-06-24  4:19 ` [PATCH 45/46] cxl/pmem: Fix offline_nvdimm_bus() to offline by bridge Dan Williams
2022-06-30 17:14   ` Jonathan Cameron
2022-07-11 19:49     ` Dan Williams
2022-06-24  4:19 ` [PATCH 46/46] cxl/region: Introduce cxl_pmem_region objects Dan Williams
2022-06-30 17:34   ` Jonathan Cameron
2022-07-11 20:05     ` Dan Williams
2022-06-24 15:13 ` [PATCH 00/46] CXL PMEM Region Provisioning Jonathan Cameron
2022-06-24 15:32   ` Dan Williams
2022-06-28  3:12 ` Alison Schofield
2022-06-28  3:34   ` Dan Williams
2022-07-02  2:26 ` Alison Schofield

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20220719154718.000077ec@Huawei.com \
    --to=jonathan.cameron@huawei.com \
    --cc=bwidawsk@kernel.org \
    --cc=dan.j.williams@intel.com \
    --cc=hch@lst.de \
    --cc=linux-cxl@vger.kernel.org \
    --cc=linux-pci@vger.kernel.org \
    --cc=nvdimm@lists.linux.dev \
    --cc=patches@lists.linux.dev \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).