Re: [PATCH 18/46] cxl/hdm: Add support for allocating DPA to an endpoint decoder

[Jonathan Cameron <Jonathan.Cameron@Huawei.com>]

On Thu, 23 Jun 2022 19:47:18 -0700
Dan Williams <dan.j.williams@intel.com> wrote:

> The region provisioning flow will roughly follow a sequence of:
> 
> 1/ Allocate DPA to a set of decoders
> 
> 2/ Allocate HPA to a region
> 
> 3/ Associate decoders with a region and validate that the DPA allocations
>    and topologies match the parameters of the region.
> 
> For now, this change (step 1) arranges for DPA capacity to be allocated
> and deleted from non-committed decoders based on the decoder's mode /
> partition selection. Capacity is allocated from the lowest DPA in the
> partition and any 'pmem' allocation blocks out all remaining ram
> capacity in its 'skip' setting. DPA allocations are enforced in decoder
> instance order. I.e. decoder N + 1 always starts at a higher DPA than
> instance N, and deleting allocations must proceed from the
> highest-instance allocated decoder to the lowest.
> 
> Signed-off-by: Dan Williams <dan.j.williams@intel.com>

The error value setting in here might save a few lines, but to me it
is less readable than setting rc in each error path.

> ---
>  Documentation/ABI/testing/sysfs-bus-cxl |   37 +++++++
>  drivers/cxl/core/core.h                 |    7 +
>  drivers/cxl/core/hdm.c                  |  160 +++++++++++++++++++++++++++++++
>  drivers/cxl/core/port.c                 |   73 ++++++++++++++
>  4 files changed, 275 insertions(+), 2 deletions(-)
> 
> diff --git a/Documentation/ABI/testing/sysfs-bus-cxl b/Documentation/ABI/testing/sysfs-bus-cxl
> index 091459216e11..85844f9bc00b 100644
> --- a/Documentation/ABI/testing/sysfs-bus-cxl
> +++ b/Documentation/ABI/testing/sysfs-bus-cxl
> @@ -171,7 +171,7 @@ Date:		May, 2022
>  KernelVersion:	v5.20
>  Contact:	linux-cxl@vger.kernel.org
>  Description:
> -		(RO) When a CXL decoder is of devtype "cxl_decoder_endpoint" it
> +		(RW) When a CXL decoder is of devtype "cxl_decoder_endpoint" it
>  		translates from a host physical address range, to a device local
>  		address range. Device-local address ranges are further split
>  		into a 'ram' (volatile memory) range and 'pmem' (persistent
> @@ -180,3 +180,38 @@ Description:
>  		when a decoder straddles the volatile/persistent partition
>  		boundary, and 'none' indicates the decoder is not actively
>  		decoding, or no DPA allocation policy has been set.
> +
> +		'mode' can be written, when the decoder is in the 'disabled'
> +		state, with either 'ram' or 'pmem' to set the boundaries for the
> +		next allocation.
> +

As before, documentation above this in the file only uses single line break between
entries.

> +
> +What:		/sys/bus/cxl/devices/decoderX.Y/dpa_resource
> +Date:		May, 2022
> +KernelVersion:	v5.20
> +Contact:	linux-cxl@vger.kernel.org
> +Description:
> +		(RO) When a CXL decoder is of devtype "cxl_decoder_endpoint",
> +		and its 'dpa_size' attribute is non-zero, this attribute
> +		indicates the device physical address (DPA) base address of the
> +		allocation.

Why _resource rather than _base or _start?

> +
> +
> +What:		/sys/bus/cxl/devices/decoderX.Y/dpa_size
> +Date:		May, 2022
> +KernelVersion:	v5.20
> +Contact:	linux-cxl@vger.kernel.org
> +Description:
> +		(RW) When a CXL decoder is of devtype "cxl_decoder_endpoint" it
> +		translates from a host physical address range, to a device local
> +		address range. The range, base address plus length in bytes, of
> +		DPA allocated to this decoder is conveyed in these 2 attributes.
> +		Allocations can be mutated as long as the decoder is in the
> +		disabled state. A write to 'size' releases the previous DPA

'dpa_size' ?

> +		allocation and then attempts to allocate from the free capacity
> +		in the device partition referred to by 'decoderX.Y/mode'.
> +		Allocate and free requests can only be performed on the highest
> +		instance number disabled decoder with non-zero size. I.e.
> +		allocations are enforced to occur in increasing 'decoderX.Y/id'
> +		order and frees are enforced to occur in decreasing
> +		'decoderX.Y/id' order.
> diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h
> index 1a50c0fc399c..47cf0c286fc3 100644
> --- a/drivers/cxl/core/core.h
> +++ b/drivers/cxl/core/core.h
> @@ -17,6 +17,13 @@ int cxl_send_cmd(struct cxl_memdev *cxlmd, struct cxl_send_command __user *s);
>  void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr,
>  				   resource_size_t length);
>  
> +int cxl_dpa_set_mode(struct cxl_endpoint_decoder *cxled,
> +		     enum cxl_decoder_mode mode);
> +int cxl_dpa_alloc(struct cxl_endpoint_decoder *cxled, unsigned long long size);
> +int cxl_dpa_free(struct cxl_endpoint_decoder *cxled);
> +resource_size_t cxl_dpa_size(struct cxl_endpoint_decoder *cxled);
> +resource_size_t cxl_dpa_resource(struct cxl_endpoint_decoder *cxled);
> +
>  int cxl_memdev_init(void);
>  void cxl_memdev_exit(void);
>  void cxl_mbox_init(void);
> diff --git a/drivers/cxl/core/hdm.c b/drivers/cxl/core/hdm.c
> index 8805afe63ebf..ceb4c28abc1b 100644
> --- a/drivers/cxl/core/hdm.c
> +++ b/drivers/cxl/core/hdm.c
> @@ -248,6 +248,166 @@ static int cxl_dpa_reserve(struct cxl_endpoint_decoder *cxled,
>  	return devm_add_action_or_reset(&port->dev, cxl_dpa_release, cxled);
>  }
>  
> +resource_size_t cxl_dpa_size(struct cxl_endpoint_decoder *cxled)
> +{
> +	resource_size_t size = 0;
> +
> +	down_read(&cxl_dpa_rwsem);
> +	if (cxled->dpa_res)
> +		size = resource_size(cxled->dpa_res);
> +	up_read(&cxl_dpa_rwsem);
> +
> +	return size;
> +}
> +
> +resource_size_t cxl_dpa_resource(struct cxl_endpoint_decoder *cxled)

Instinct would be to expect this to return the resource, not the start.
Rename?


> +{
> +	resource_size_t base = -1;
> +
> +	down_read(&cxl_dpa_rwsem);
> +	if (cxled->dpa_res)
> +		base = cxled->dpa_res->start;
> +	up_read(&cxl_dpa_rwsem);
> +
> +	return base;
> +}
> +
> +int cxl_dpa_free(struct cxl_endpoint_decoder *cxled)
> +{
> +	int rc = -EBUSY;
> +	struct device *dev = &cxled->cxld.dev;
> +	struct cxl_port *port = to_cxl_port(dev->parent);
> +
> +	down_write(&cxl_dpa_rwsem);
> +	if (!cxled->dpa_res) {
> +		rc = 0;
> +		goto out;
> +	}
> +	if (cxled->cxld.flags & CXL_DECODER_F_ENABLE) {
> +		dev_dbg(dev, "decoder enabled\n");

I'd prefer explicit setting of rc = -EBUSY in the two
'error' paths to make it really clear when looking at these
that they are treated as errors.

> +		goto out;
> +	}
> +	if (cxled->cxld.id != port->dpa_end) {
> +		dev_dbg(dev, "expected decoder%d.%d\n", port->id,
> +			port->dpa_end);
> +		goto out;
> +	}
> +	__cxl_dpa_release(cxled, true);
> +	rc = 0;
> +out:
> +	up_write(&cxl_dpa_rwsem);
> +	return rc;
> +}
> +
> +int cxl_dpa_set_mode(struct cxl_endpoint_decoder *cxled,
> +		     enum cxl_decoder_mode mode)
> +{
> +	struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
> +	struct cxl_dev_state *cxlds = cxlmd->cxlds;
> +	struct device *dev = &cxled->cxld.dev;
> +	int rc = -EBUSY;

As above, I'd prefer seeing error set in each error path rther
than it being set in a few locations and having to go look
for which value it currently has.  To me having the
error code next to the condition is much easier to follow.

> +
> +	switch (mode) {
> +	case CXL_DECODER_RAM:
> +	case CXL_DECODER_PMEM:
> +		break;
> +	default:
> +		dev_dbg(dev, "unsupported mode: %d\n", mode);
> +		return -EINVAL;
> +	}
> +
> +	down_write(&cxl_dpa_rwsem);
> +	if (cxled->cxld.flags & CXL_DECODER_F_ENABLE)
> +		goto out;
> +	/*
> +	 * Only allow modes that are supported by the current partition
> +	 * configuration
> +	 */
> +	rc = -ENXIO;
> +	if (mode == CXL_DECODER_PMEM && !resource_size(&cxlds->pmem_res)) {
> +		dev_dbg(dev, "no available pmem capacity\n");
> +		goto out;
> +	}
> +	if (mode == CXL_DECODER_RAM && !resource_size(&cxlds->ram_res)) {
> +		dev_dbg(dev, "no available ram capacity\n");
> +		goto out;
> +	}
> +
> +	cxled->mode = mode;
> +	rc = 0;
> +out:
> +	up_write(&cxl_dpa_rwsem);
> +
> +	return rc;
> +}
> +
> +int cxl_dpa_alloc(struct cxl_endpoint_decoder *cxled, unsigned long long size)
> +{
> +	struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
> +	resource_size_t free_ram_start, free_pmem_start;
> +	struct cxl_port *port = cxled_to_port(cxled);
> +	struct cxl_dev_state *cxlds = cxlmd->cxlds;
> +	struct device *dev = &cxled->cxld.dev;
> +	resource_size_t start, avail, skip;
> +	struct resource *p, *last;
> +	int rc = -EBUSY;
> +
> +	down_write(&cxl_dpa_rwsem);
> +	if (cxled->cxld.flags & CXL_DECODER_F_ENABLE) {
> +		dev_dbg(dev, "decoder enabled\n");
> +		goto out;


-EBUSY only used in this path, so clearer to me to push that setting down
to in this  error path.


> +	}
> +
> +	for (p = cxlds->ram_res.child, last = NULL; p; p = p->sibling)
> +		last = p;
> +	if (last)
> +		free_ram_start = last->end + 1;
> +	else
> +		free_ram_start = cxlds->ram_res.start;
> +
> +	for (p = cxlds->pmem_res.child, last = NULL; p; p = p->sibling)
> +		last = p;
> +	if (last)
> +		free_pmem_start = last->end + 1;
> +	else
> +		free_pmem_start = cxlds->pmem_res.start;
> +
> +	if (cxled->mode == CXL_DECODER_RAM) {
> +		start = free_ram_start;
> +		avail = cxlds->ram_res.end - start + 1;
> +		skip = 0;
> +	} else if (cxled->mode == CXL_DECODER_PMEM) {
> +		resource_size_t skip_start, skip_end;
> +
> +		start = free_pmem_start;
> +		avail = cxlds->pmem_res.end - start + 1;
> +		skip_start = free_ram_start;
> +		skip_end = start - 1;
> +		skip = skip_end - skip_start + 1;
> +	} else {
> +		dev_dbg(dev, "mode not set\n");
> +		rc = -EINVAL;
> +		goto out;
> +	}
> +
> +	if (size > avail) {
> +		dev_dbg(dev, "%pa exceeds available %s capacity: %pa\n", &size,
> +			cxled->mode == CXL_DECODER_RAM ? "ram" : "pmem",
> +			&avail);
> +		rc = -ENOSPC;
> +		goto out;
> +	}
> +
> +	rc = __cxl_dpa_reserve(cxled, start, size, skip);
> +out:
> +	up_write(&cxl_dpa_rwsem);
> +
> +	if (rc)
> +		return rc;
> +
> +	return devm_add_action_or_reset(&port->dev, cxl_dpa_release, cxled);
> +}
> +
>  static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld,
>  			    int *target_map, void __iomem *hdm, int which,
>  			    u64 *dpa_base)

>  
>