Re: [RFC PATCH v3 2/4] PCI/doe: Add Data Object Exchange support

[Jonathan Cameron <Jonathan.Cameron@Huawei.com>]

On Fri, 14 May 2021 11:37:12 -0700
Dan Williams <dan.j.williams@intel.com> wrote:

> On Fri, May 14, 2021 at 1:50 AM Jonathan Cameron
> <Jonathan.Cameron@huawei.com> wrote:
> [..]
> > > If it simplifies the kernel implementation to assume single
> > > kernel-initiator then I think that's more than enough reason to block
> > > out userspace, and/or provide userspace a method to get into the
> > > kernel's queue for service.  
> >
> > This last suggestion makes sense to me. Let's provide a 'right' way
> > to access the DOE from user space. I like the idea if it being possible
> > to run CXL compliance tests from userspace whilst the driver is loaded.  
> 
> Ah, and I like your observation that once the kernel provides a
> "right" way to access DOE then userspace direct-access of DOE is
> indeed a "you get to keep the pieces" event like any other unwanted
> userspace config-write.
> 
> > Bjorn, given this would be a generic PCI thing, any preference for what
> > this interface might look like?   /dev/pcidoe[xxxxxx].i with ioctls similar
> > to those for the BAR based CXL mailboxes?  
> 
> (warning, anti-ioctl bias incoming...)

I feel very similar about ioctls - my immediate thought was to shove this in
debugfs, but that feels the wrong choice if we are trying to persuade people
to use it instead of writing code that directly accesses the config space.

> 
> Hmm, DOE has an enumeration capability, could the DOE driver use a
> scheme to have a sysfs bin_attr per discovered object type? This would
> make it simliar to the pci-vpd sysfs interface.

We can discover the protocols, but anything beyond that is protocol
specific.  I don't think there is a enough info available by any standards
defined method. Also part of the reason to allow a safe userspace interface
would be to provide a generic interface for vendor protocols and things like
CXL compliance tests where we will almost certainly never provide a more
specific kernel interface.

Whilst sysfs would work for CDAT, some protocols are challenge response rather
than simple read back and that really doesn't fit well for sysfs model.
If we get other protocols that are simple data read back, then I would
advocate giving them a simple sysfs interface much like proposed for CDAT
as it will always be simpler to use + self describing.

On a lesser note it might be helpful to provide sysfs attrs for
what protocols are supported.  The alternative is to let userspace run
the discovery protocol. Perhaps we can do this as a later phase.

> 
> Then the kernel could cache objects like CDAT that don't change
> outside of some invalidation event.

It's been a while since I last saw any conversation on sysfs bin_attrs
but mostly I thought the feeling was pretty strongly against them for anything
but a few niche usecases.

Feels to me like it would break most of the usual rules in a way vpd does
not (IIRC VPD is supposed to be a simple in the sense that if you write a value
to a writable part, you will read back the same value).

+CC Greg who is a fount of knowledge in this area (and regularly + correctly
screams at the ways I try to abuse sysfs :)  Note I don't think Dan was
suggesting implementing response / request directly, but I think that is
all we could do given DOE protocols can be vendor specific and the standard
discovery protocol doesn't let us know the fine grained support (what commands
within a given protocol).

Jonathan