Re: [PATCH 2/6] device/pci: add cmdmem cap to pci_dev

Re: [PATCH 2/6] device/pci: add cmdmem cap to pci_dev

[Bjorn Helgaas]

On Tue, Mar 31, 2020 at 10:59:44AM -0700, Dave Jiang wrote:
> On 3/31/2020 9:03 AM, Bjorn Helgaas wrote:
> > On Mon, Mar 30, 2020 at 02:27:00PM -0700, Dave Jiang wrote:
> > > Since the current accelerator devices do not have standard PCIe capability
> > > enumeration for accepting ENQCMDS yet, for now an attribute of pdev->cmdmem has
> > > been added to struct pci_dev.  Currently a PCI quirk must be used for the
> > > devices that have such cap until the PCI cap is standardized. Add a helper
> > > function to provide the check if a device supports the cmdmem capability.
> > > 
> > > Such capability is expected to be added to PCIe device cap enumeration in
> > > the future.
> > This needs some sort of thumbnail description of what "synchronous
> > write notification" and "cmdmem" mean.
> 
> I will add more explanation.
> 
> > Do you have a pointer to a PCI-SIG ECR or similar?
> 
> Deferrable Memory Write (DMWr) ECR
> 
> https://members.pcisig.com/wg/PCI-SIG/document/13747
> 
> From what I'm told it should be available for public review by EOW.

Please use terminology from the spec instead of things like
"synchronous write notification".

AIUI, ENQCMDS is an x86 instruction.  That would have no meaning in
the PCIe domain.

I'm not committing to acking any part of this before the ECR is
accepted, but if you're adding support for the feature described by
the ECR, you might as well add support for discovering the DMWr
capability via Device Capabilities 2 as described in the ECR.

If you have hardware that supports DMWr but doesn't advertise it via
Device Capabilities 2, you can always have a quirk that works around
that lack.

Re: [PATCH 2/6] device/pci: add cmdmem cap to pci_dev

[Thomas Gleixner]

Bjorn Helgaas <helgaas@kernel.org> writes:
> On Tue, Mar 31, 2020 at 10:59:44AM -0700, Dave Jiang wrote:
>> On 3/31/2020 9:03 AM, Bjorn Helgaas wrote:
>> > On Mon, Mar 30, 2020 at 02:27:00PM -0700, Dave Jiang wrote:
>> > > Since the current accelerator devices do not have standard PCIe capability
>> > > enumeration for accepting ENQCMDS yet, for now an attribute of pdev->cmdmem has
>> > > been added to struct pci_dev.  Currently a PCI quirk must be used for the
>> > > devices that have such cap until the PCI cap is standardized. Add a helper
>> > > function to provide the check if a device supports the cmdmem capability.
>> > > 
>> > > Such capability is expected to be added to PCIe device cap enumeration in
>> > > the future.
>> > This needs some sort of thumbnail description of what "synchronous
>> > write notification" and "cmdmem" mean.
>> 
>> I will add more explanation.
>> 
>> > Do you have a pointer to a PCI-SIG ECR or similar?
>> 
>> Deferrable Memory Write (DMWr) ECR
>> 
>> https://members.pcisig.com/wg/PCI-SIG/document/13747
>> 
>> From what I'm told it should be available for public review by EOW.
>
> Please use terminology from the spec instead of things like
> "synchronous write notification".
>
> AIUI, ENQCMDS is an x86 instruction.  That would have no meaning in
> the PCIe domain.
>
> I'm not committing to acking any part of this before the ECR is
> accepted, but if you're adding support for the feature described by
> the ECR, you might as well add support for discovering the DMWr
> capability via Device Capabilities 2 as described in the ECR.

Don't worry. There is nothing to decide and ack before the basic
architecture support for ENQCMD[S] is discussed and accepted. The
patches providing this support have been posted 2 hrs before this pile
hit the mailing lists yesterday.

Thanks,

        tglx

Re: [PATCH 2/6] device/pci: add cmdmem cap to pci_dev

[Dave Jiang]

On 3/31/2020 3:00 PM, Bjorn Helgaas wrote:
> On Tue, Mar 31, 2020 at 10:59:44AM -0700, Dave Jiang wrote:
>> On 3/31/2020 9:03 AM, Bjorn Helgaas wrote:
>>> On Mon, Mar 30, 2020 at 02:27:00PM -0700, Dave Jiang wrote:
>>>> Since the current accelerator devices do not have standard PCIe capability
>>>> enumeration for accepting ENQCMDS yet, for now an attribute of pdev->cmdmem has
>>>> been added to struct pci_dev.  Currently a PCI quirk must be used for the
>>>> devices that have such cap until the PCI cap is standardized. Add a helper
>>>> function to provide the check if a device supports the cmdmem capability.
>>>>
>>>> Such capability is expected to be added to PCIe device cap enumeration in
>>>> the future.
>>> This needs some sort of thumbnail description of what "synchronous
>>> write notification" and "cmdmem" mean.
>>
>> I will add more explanation.
>>
>>> Do you have a pointer to a PCI-SIG ECR or similar?
>>
>> Deferrable Memory Write (DMWr) ECR
>>
>> https://members.pcisig.com/wg/PCI-SIG/document/13747
>>
>>  From what I'm told it should be available for public review by EOW.
> 
> Please use terminology from the spec instead of things like
> "synchronous write notification".
> 
> AIUI, ENQCMDS is an x86 instruction.  That would have no meaning in
> the PCIe domain.
> 
> I'm not committing to acking any part of this before the ECR is
> accepted, but if you're adding support for the feature described by
> the ECR, you might as well add support for discovering the DMWr
> capability via Device Capabilities 2 as described in the ECR.

I'll look into adding the support for the ECR.

> 
> If you have hardware that supports DMWr but doesn't advertise it via
> Device Capabilities 2, you can always have a quirk that works around
> that lack.
> 

Re: [PATCH 2/6] device/pci: add cmdmem cap to pci_dev

[Dave Jiang]

On 3/31/2020 9:03 AM, Bjorn Helgaas wrote:
> On Mon, Mar 30, 2020 at 02:27:00PM -0700, Dave Jiang wrote:
>> Since the current accelerator devices do not have standard PCIe capability
>> enumeration for accepting ENQCMDS yet, for now an attribute of pdev->cmdmem has
>> been added to struct pci_dev.  Currently a PCI quirk must be used for the
>> devices that have such cap until the PCI cap is standardized. Add a helper
>> function to provide the check if a device supports the cmdmem capability.
>>
>> Such capability is expected to be added to PCIe device cap enumeration in
>> the future.

Re-send. My misconfigured mail client caused mailing lists to bounce the 
send.

> 
> This needs some sort of thumbnail description of what "synchronous
> write notification" and "cmdmem" mean.

I will add more explanation.

> 
> Do you have a pointer to a PCI-SIG ECR or similar?



Deferrable Memory Write (DMWr) ECR


https://members.pcisig.com/wg/PCI-SIG/document/13747

 From what I'm told it should be available for public review by EOW.

> 
> Your window size seems to be 85 or so.  It would be easier if you used
> 80 and wrapped the commit log to fit in 75 columns so it looks decent
> when "git log" indents it by 4.
> 
Ok I will fix.

Re: [PATCH 2/6] device/pci: add cmdmem cap to pci_dev

[Dave Jiang]

On 3/31/2020 10:24 AM, Greg KH wrote:
> On Tue, Mar 31, 2020 at 10:07:07AM -0700, Dave Jiang wrote:
>> On 3/31/2020 3:04 AM, Greg KH wrote:
>>> On Mon, Mar 30, 2020 at 02:27:00PM -0700, Dave Jiang wrote:
>>>> Since the current accelerator devices do not have standard PCIe capability
>>>> enumeration for accepting ENQCMDS yet, for now an attribute of pdev->cmdmem has
>>>> been added to struct pci_dev.  Currently a PCI quirk must be used for the
>>>> devices that have such cap until the PCI cap is standardized. Add a helper
>>>> function to provide the check if a device supports the cmdmem capability.
>>>>
>>>> Such capability is expected to be added to PCIe device cap enumeration in
>>>> the future.
>>>>
>>>> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
>>>> ---
>>>>    drivers/base/core.c    |   13 +++++++++++++
>>>>    include/linux/device.h |    2 ++
>>>>    include/linux/pci.h    |    1 +
>>>>    3 files changed, 16 insertions(+)
>>>>
>>>> diff --git a/drivers/base/core.c b/drivers/base/core.c
>>>> index dbb0f9130f42..cd9f5b040ed4 100644
>>>> --- a/drivers/base/core.c
>>>> +++ b/drivers/base/core.c
>>>> @@ -27,6 +27,7 @@
>>>>    #include <linux/netdevice.h>
>>>>    #include <linux/sched/signal.h>
>>>>    #include <linux/sysfs.h>
>>>> +#include <linux/pci.h>
>>>>    #include "base.h"
>>>>    #include "power/power.h"
>>>> @@ -3790,3 +3791,15 @@ int device_match_any(struct device *dev, const void *unused)
>>>>    	return 1;
>>>>    }
>>>>    EXPORT_SYMBOL_GPL(device_match_any);
>>>> +
>>>> +bool device_supports_cmdmem(struct device *dev)
>>>> +{
>>>> +	struct pci_dev *pdev;
>>>> +
>>>> +	if (!dev_is_pci(dev))
>>>> +		return false;
>>>> +
>>>> +	pdev = to_pci_dev(dev);
>>>> +	return pdev->cmdmem;
>>>> +}
>>>> +EXPORT_SYMBOL_GPL(device_supports_cmdmem);
>>> Why would a pci-specific function like this be ok to have in the driver
>>> core?  Please keep it in the pci core code instead.
>> The original thought was to introduce a new arch level memory mapping
>> semantic.
> Please do not.  Also, that's not what you are doing here from what I can
> tell.
>
>> If you feel this should be PCI exclusive, should we make the ioremap
>> routines for this memory type pci specific as well?
> Why wouldn't it be?  Is this needed anywhere else?

Ok I'll make this pci specific.


>
> thanks,
>
> greg k-h

Re: [PATCH 2/6] device/pci: add cmdmem cap to pci_dev

[Greg KH]

On Tue, Mar 31, 2020 at 10:07:07AM -0700, Dave Jiang wrote:
> 
> On 3/31/2020 3:04 AM, Greg KH wrote:
> > On Mon, Mar 30, 2020 at 02:27:00PM -0700, Dave Jiang wrote:
> > > Since the current accelerator devices do not have standard PCIe capability
> > > enumeration for accepting ENQCMDS yet, for now an attribute of pdev->cmdmem has
> > > been added to struct pci_dev.  Currently a PCI quirk must be used for the
> > > devices that have such cap until the PCI cap is standardized. Add a helper
> > > function to provide the check if a device supports the cmdmem capability.
> > > 
> > > Such capability is expected to be added to PCIe device cap enumeration in
> > > the future.
> > > 
> > > Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> > > ---
> > >   drivers/base/core.c    |   13 +++++++++++++
> > >   include/linux/device.h |    2 ++
> > >   include/linux/pci.h    |    1 +
> > >   3 files changed, 16 insertions(+)
> > > 
> > > diff --git a/drivers/base/core.c b/drivers/base/core.c
> > > index dbb0f9130f42..cd9f5b040ed4 100644
> > > --- a/drivers/base/core.c
> > > +++ b/drivers/base/core.c
> > > @@ -27,6 +27,7 @@
> > >   #include <linux/netdevice.h>
> > >   #include <linux/sched/signal.h>
> > >   #include <linux/sysfs.h>
> > > +#include <linux/pci.h>
> > >   #include "base.h"
> > >   #include "power/power.h"
> > > @@ -3790,3 +3791,15 @@ int device_match_any(struct device *dev, const void *unused)
> > >   	return 1;
> > >   }
> > >   EXPORT_SYMBOL_GPL(device_match_any);
> > > +
> > > +bool device_supports_cmdmem(struct device *dev)
> > > +{
> > > +	struct pci_dev *pdev;
> > > +
> > > +	if (!dev_is_pci(dev))
> > > +		return false;
> > > +
> > > +	pdev = to_pci_dev(dev);
> > > +	return pdev->cmdmem;
> > > +}
> > > +EXPORT_SYMBOL_GPL(device_supports_cmdmem);
> > Why would a pci-specific function like this be ok to have in the driver
> > core?  Please keep it in the pci core code instead.
> 
> The original thought was to introduce a new arch level memory mapping
> semantic.

Please do not.  Also, that's not what you are doing here from what I can
tell.

> If you feel this should be PCI exclusive, should we make the ioremap
> routines for this memory type pci specific as well?

Why wouldn't it be?  Is this needed anywhere else?

thanks,

greg k-h

Re: [PATCH 2/6] device/pci: add cmdmem cap to pci_dev

[Dave Jiang]

On 3/31/2020 3:04 AM, Greg KH wrote:
> On Mon, Mar 30, 2020 at 02:27:00PM -0700, Dave Jiang wrote:
>> Since the current accelerator devices do not have standard PCIe capability
>> enumeration for accepting ENQCMDS yet, for now an attribute of pdev->cmdmem has
>> been added to struct pci_dev.  Currently a PCI quirk must be used for the
>> devices that have such cap until the PCI cap is standardized. Add a helper
>> function to provide the check if a device supports the cmdmem capability.
>>
>> Such capability is expected to be added to PCIe device cap enumeration in
>> the future.
>>
>> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
>> ---
>>   drivers/base/core.c    |   13 +++++++++++++
>>   include/linux/device.h |    2 ++
>>   include/linux/pci.h    |    1 +
>>   3 files changed, 16 insertions(+)
>>
>> diff --git a/drivers/base/core.c b/drivers/base/core.c
>> index dbb0f9130f42..cd9f5b040ed4 100644
>> --- a/drivers/base/core.c
>> +++ b/drivers/base/core.c
>> @@ -27,6 +27,7 @@
>>   #include <linux/netdevice.h>
>>   #include <linux/sched/signal.h>
>>   #include <linux/sysfs.h>
>> +#include <linux/pci.h>
>>   
>>   #include "base.h"
>>   #include "power/power.h"
>> @@ -3790,3 +3791,15 @@ int device_match_any(struct device *dev, const void *unused)
>>   	return 1;
>>   }
>>   EXPORT_SYMBOL_GPL(device_match_any);
>> +
>> +bool device_supports_cmdmem(struct device *dev)
>> +{
>> +	struct pci_dev *pdev;
>> +
>> +	if (!dev_is_pci(dev))
>> +		return false;
>> +
>> +	pdev = to_pci_dev(dev);
>> +	return pdev->cmdmem;
>> +}
>> +EXPORT_SYMBOL_GPL(device_supports_cmdmem);
> Why would a pci-specific function like this be ok to have in the driver
> core?  Please keep it in the pci core code instead.

The original thought was to introduce a new arch level memory mapping 
semantic. If you feel this should be PCI exclusive, should we make the 
ioremap routines for this memory type pci specific as well?


>
> thanks,
>
> greg k-h

Re: [PATCH 2/6] device/pci: add cmdmem cap to pci_dev

[Bjorn Helgaas]

On Mon, Mar 30, 2020 at 02:27:00PM -0700, Dave Jiang wrote:
> Since the current accelerator devices do not have standard PCIe capability
> enumeration for accepting ENQCMDS yet, for now an attribute of pdev->cmdmem has
> been added to struct pci_dev.  Currently a PCI quirk must be used for the
> devices that have such cap until the PCI cap is standardized. Add a helper
> function to provide the check if a device supports the cmdmem capability.
> 
> Such capability is expected to be added to PCIe device cap enumeration in
> the future.

This needs some sort of thumbnail description of what "synchronous
write notification" and "cmdmem" mean.

Do you have a pointer to a PCI-SIG ECR or similar?

Your window size seems to be 85 or so.  It would be easier if you used
80 and wrapped the commit log to fit in 75 columns so it looks decent
when "git log" indents it by 4.

Re: [PATCH 2/6] device/pci: add cmdmem cap to pci_dev

[Greg KH]

On Mon, Mar 30, 2020 at 02:27:00PM -0700, Dave Jiang wrote:
> Since the current accelerator devices do not have standard PCIe capability
> enumeration for accepting ENQCMDS yet, for now an attribute of pdev->cmdmem has
> been added to struct pci_dev.  Currently a PCI quirk must be used for the
> devices that have such cap until the PCI cap is standardized. Add a helper
> function to provide the check if a device supports the cmdmem capability.
> 
> Such capability is expected to be added to PCIe device cap enumeration in
> the future.
> 
> Signed-off-by: Dave Jiang <dave.jiang@intel.com>
> ---
>  drivers/base/core.c    |   13 +++++++++++++
>  include/linux/device.h |    2 ++
>  include/linux/pci.h    |    1 +
>  3 files changed, 16 insertions(+)
> 
> diff --git a/drivers/base/core.c b/drivers/base/core.c
> index dbb0f9130f42..cd9f5b040ed4 100644
> --- a/drivers/base/core.c
> +++ b/drivers/base/core.c
> @@ -27,6 +27,7 @@
>  #include <linux/netdevice.h>
>  #include <linux/sched/signal.h>
>  #include <linux/sysfs.h>
> +#include <linux/pci.h>
>  
>  #include "base.h"
>  #include "power/power.h"
> @@ -3790,3 +3791,15 @@ int device_match_any(struct device *dev, const void *unused)
>  	return 1;
>  }
>  EXPORT_SYMBOL_GPL(device_match_any);
> +
> +bool device_supports_cmdmem(struct device *dev)
> +{
> +	struct pci_dev *pdev;
> +
> +	if (!dev_is_pci(dev))
> +		return false;
> +
> +	pdev = to_pci_dev(dev);
> +	return pdev->cmdmem;
> +}
> +EXPORT_SYMBOL_GPL(device_supports_cmdmem);

Why would a pci-specific function like this be ok to have in the driver
core?  Please keep it in the pci core code instead.

thanks,

greg k-h

[PATCH 2/6] device/pci: add cmdmem cap to pci_dev

[Dave Jiang]

Since the current accelerator devices do not have standard PCIe capability
enumeration for accepting ENQCMDS yet, for now an attribute of pdev->cmdmem has
been added to struct pci_dev.  Currently a PCI quirk must be used for the
devices that have such cap until the PCI cap is standardized. Add a helper
function to provide the check if a device supports the cmdmem capability.

Such capability is expected to be added to PCIe device cap enumeration in
the future.

Signed-off-by: Dave Jiang <dave.jiang@intel.com>
---
 drivers/base/core.c    |   13 +++++++++++++
 include/linux/device.h |    2 ++
 include/linux/pci.h    |    1 +
 3 files changed, 16 insertions(+)

diff --git a/drivers/base/core.c b/drivers/base/core.c
index dbb0f9130f42..cd9f5b040ed4 100644
--- a/drivers/base/core.c
+++ b/drivers/base/core.c
@@ -27,6 +27,7 @@
 #include <linux/netdevice.h>
 #include <linux/sched/signal.h>
 #include <linux/sysfs.h>
+#include <linux/pci.h>
 
 #include "base.h"
 #include "power/power.h"
@@ -3790,3 +3791,15 @@ int device_match_any(struct device *dev, const void *unused)
 	return 1;
 }
 EXPORT_SYMBOL_GPL(device_match_any);
+
+bool device_supports_cmdmem(struct device *dev)
+{
+	struct pci_dev *pdev;
+
+	if (!dev_is_pci(dev))
+		return false;
+
+	pdev = to_pci_dev(dev);
+	return pdev->cmdmem;
+}
+EXPORT_SYMBOL_GPL(device_supports_cmdmem);
diff --git a/include/linux/device.h b/include/linux/device.h
index fa04dfd22bbc..3e787d3de435 100644
--- a/include/linux/device.h
+++ b/include/linux/device.h
@@ -809,6 +809,8 @@ static inline bool dev_has_sync_state(struct device *dev)
 	return false;
 }
 
+extern bool device_supports_cmdmem(struct device *dev);
+
 /*
  * High level routines for use by the bus drivers
  */
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 3840a541a9de..0bc5d581f20e 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -422,6 +422,7 @@ struct pci_dev {
 	unsigned int	is_probed:1;		/* Device probing in progress */
 	unsigned int	link_active_reporting:1;/* Device capable of reporting link active */
 	unsigned int	no_vf_scan:1;		/* Don't scan for VFs after IOV enablement */
+	unsigned int	cmdmem:1;		/* MMIO writes support command mem region with synchronous write notification */
 	pci_dev_flags_t dev_flags;
 	atomic_t	enable_cnt;	/* pci_enable_device has been called */