Re: [PATCH v4 2/5] PCI/DOE: Add Data Object Exchange support

From: Dan Williams <dan.j.williams@intel.com>
To: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Cc: Linux PCI <linux-pci@vger.kernel.org>,
	linux-cxl@vger.kernel.org, Bjorn Helgaas <helgaas@kernel.org>,
	"Weiny, Ira" <ira.weiny@intel.com>,
	Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>,
	Ben Widawsky <ben.widawsky@intel.com>,
	Chris Browy <cbrowy@avery-design.com>,
	Linux ACPI <linux-acpi@vger.kernel.org>,
	"Schofield, Alison" <alison.schofield@intel.com>,
	Vishal L Verma <vishal.l.verma@intel.com>,
	Fangjian <f.fangjian@huawei.com>, Linuxarm <linuxarm@huawei.com>
Subject: Re: [PATCH v4 2/5] PCI/DOE: Add Data Object Exchange support
Date: Thu, 10 Jun 2021 13:06:48 -0700	[thread overview]
Message-ID: <CAPcyv4gBORHzouArX-Fsnhew+ZYur8pp9ySJQGwrOBrGuv+-ZQ@mail.gmail.com> (raw)
In-Reply-To: <20210524133938.2815206-3-Jonathan.Cameron@huawei.com>

On Mon, May 24, 2021 at 6:41 AM Jonathan Cameron
<Jonathan.Cameron@huawei.com> wrote:
>
> Introduced in a PCI ECN [1], DOE provides a config space based mailbox with
> standard protocol discovery.  Each mailbox is accessed through a DOE
> Extended Capability.
>
> A device may have 1 or more DOE mailboxes, each of which is allowed to
> support any number of protocols (some DOE protocol specifications apply
> additional restrictions).  A given protocol may be supported on more than
> one DOE mailbox on a given function.
>
> If a driver wishes to access any number of DOE instances / protocols it
> makes a single call to pci_doe_register_all() which will find available
> DOEs, create the required infrastructure and cache the protocols they
> support.  pci_doe_find() can then retrieve a pointer to an appropriate DOE
> instance.
>
> A synchronous interface is provided in pci_doe_exchange_sync() to perform a
> single query / response exchange.
>
> Testing conducted against QEMU using:
>
> https://lore.kernel.org/qemu-devel/1619454964-10190-1-git-send-email-cbrowy@avery-design.com/

Nice.

I was hoping that by now QEMU upstream would have given us some
indication that this useful work that has a chance of being merged. I
fear it's only us CXL practitioner's that care. Perhaps the PCI IDE
support will get them to move on at least the DOE patches?

>
> [1] https://members.pcisig.com/wg/PCI-SIG/document/14143
>     Data Object Exchange (DOE) - Approved 12 March 2020
>
> Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>

The core logic of this looks good to me. The interfaces for other code
to make use of this I feel can lean heavier on existing mechanics. A
few points come to mind:

- Does this need to support anything more than queue depth 1? I know
the specification seems to allow for some overlapping and queueing,
but I don't think there are any use cases that are precluded if the
max number of tasks in flight for a given DOE is one.

- Once its queue depth 1 then the list of tasks can be replaced with a
wait_queue_head_t where submitters wait for the previous task to
finish.

- This appears to be the prototypical scenario for deploying the new
auxiliary bus facility. Rather than custom code device-like facilities
(lists and parents etc) in 'struct pci_doe' just make pci_doe a device
directly (auxiliary-device) and separate the infrastructure that
drives that device to a driver (auxiliary-driver). That makes the
lifetime management more idiomatic, allows for user space to have
typical driver-binding controls to manage kernel-user DOE conflicts,
and it allows for typical driver services like devm.