Re: [RFC PATCH 7/9] cxl/mem: Implement polled mode mailbox

[Bjorn Helgaas <helgaas@kernel.org>]

On Tue, Nov 10, 2020 at 09:43:54PM -0800, Ben Widawsky wrote:
> Create a function to handle sending a command, optionally with a
> payload, to the memory device, polling on a result, and then optionally
> copying out the payload. The algorithm for doing this come straight out
> of the CXL 2.0 specification.
> 
> Primary mailboxes are capable of generating an interrupt when submitting
> a command in the background. That implementation is saved for a later
> time.
> 
> Secondary mailboxes aren't implemented at this time.
> 
> WARNING: This is untested with actual timeouts occurring.
> 
> Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
> ---
>  drivers/cxl/cxl.h |  16 +++++++
>  drivers/cxl/mem.c | 107 ++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 123 insertions(+)
> 
> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> index 482fc9cdc890..f49ab80f68bd 100644
> --- a/drivers/cxl/cxl.h
> +++ b/drivers/cxl/cxl.h
> @@ -21,8 +21,12 @@
>  #define CXLDEV_MB_CTRL 0x04
>  #define   CXLDEV_MB_CTRL_DOORBELL BIT(0)
>  #define CXLDEV_MB_CMD 0x08
> +#define   CXLDEV_MB_CMD_PAYLOAD_LENGTH_SHIFT 16
>  #define CXLDEV_MB_STATUS 0x10
> +#define   CXLDEV_MB_STATUS_RET_CODE_SHIFT 32
> +#define   CXLDEV_MB_STATUS_RET_CODE_MASK 0xffff
>  #define CXLDEV_MB_BG_CMD_STATUS 0x18
> +#define CXLDEV_MB_PAYLOAD 0x20
>  
>  /* Memory Device */
>  #define CXLMDEV_STATUS 0
> @@ -114,4 +118,16 @@ static inline u64 __cxl_raw_read_reg64(struct cxl_mem *cxlm, u32 reg)
>  
>  	return readq(reg_addr + reg);
>  }
> +
> +static inline void cxl_mbox_payload_fill(struct cxl_mem *cxlm, u8 *input,
> +					    unsigned int length)
> +{
> +	memcpy_toio(cxlm->mbox.regs + CXLDEV_MB_PAYLOAD, input, length);
> +}
> +
> +static inline void cxl_mbox_payload_drain(struct cxl_mem *cxlm,
> +					     u8 *output, unsigned int length)
> +{
> +	memcpy_fromio(output, cxlm->mbox.regs + CXLDEV_MB_PAYLOAD, length);
> +}
>  #endif /* __CXL_H__ */
> diff --git a/drivers/cxl/mem.c b/drivers/cxl/mem.c
> index 9fd2d1daa534..08913360d500 100644
> --- a/drivers/cxl/mem.c
> +++ b/drivers/cxl/mem.c
> @@ -1,5 +1,6 @@
>  // SPDX-License-Identifier: GPL-2.0-only
>  // Copyright(c) 2020 Intel Corporation. All rights reserved.

/* Copyright ... */

> +#include <linux/sched/clock.h>
>  #include <linux/module.h>
>  #include <linux/pci.h>
>  #include <linux/io.h>
> @@ -7,6 +8,112 @@
>  #include "pci.h"
>  #include "cxl.h"
>  
> +struct mbox_cmd {
> +	u16 cmd;
> +	u8 *payload;
> +	size_t payload_size;
> +	u16 return_code;
> +};
> +
> +static int cxldev_wait_for_doorbell(struct cxl_mem *cxlm)
> +{
> +	u64 start, now;
> +	int cpu, ret, timeout = 2000000000;

It'd be nice to have a hint about where this timeout comes from and
what the units are.  local_clock(), sched_clock_cpu(), etc don't have
any hints either and I got tired of following the chain.

Several callers use ns_to_ktime(local_clock()), so I guess it must be
in ns?

> +	start = local_clock();
> +	preempt_disable();
> +	cpu = smp_processor_id();
> +	for (;;) {
> +		now = local_clock();
> +		preempt_enable();
> +		if ((cxl_read_mbox_reg32(cxlm, CXLDEV_MB_CTRL) &
> +		     CXLDEV_MB_CTRL_DOORBELL) == 0) {
> +			ret = 0;
> +			break;
> +		}
> +
> +		if (now - start >= timeout) {
> +			ret = -ETIMEDOUT;
> +			break;
> +		}
> +
> +		cpu_relax();
> +		preempt_disable();
> +		if (unlikely(cpu != smp_processor_id())) {
> +			timeout -= (now - start);
> +			cpu = smp_processor_id();
> +			start = local_clock();
> +		}
> +	}