[virtio] [PATCH] ACCESS_PLATFORM/ACCESS_ORDER

[virtio] [PATCH] ACCESS_PLATFORM/ACCESS_ORDER

[Michael S. Tsirkin]

This is an attempt to clarify the intent behind
VIRTIO_F_IOMMU_PLATFORM and VIRTIO_F_IO_BARRIER which based on
recent discussions appear not to be well documented,
in particular not matching what drivers do or are
likely to do.

- rename VIRTIO_F_IOMMU_PLATFORM to ACCESS_PLATFORM
  It is already the fact that the DMA API that Linux
  uses does more than just IOMMUs - it includes
  cache flushing, bounce buffers for limited
  addressing, etc.
  Update spec to match this reality.

- rename VIRTIO_F_IO_BARRIER to VIRTIO_F_ORDER_PLATFORM
  this is after all what device is telling driver:
  its memory accesses are only ordered weakly,
  this is why a stronger barrier is required.

- As no one yet implemented IO_BARRIER yet, add a recommendation
  to have a software fallback so that existing drivers
  aren't broken.

Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
 content.tex | 68 ++++++++++++++++++++++++++++++++++++-----------------
 1 file changed, 47 insertions(+), 21 deletions(-)

diff --git a/content.tex b/content.tex
index c346183..4fef3ad 100644
--- a/content.tex
+++ b/content.tex
@@ -5452,26 +5452,46 @@ Descriptors} and \ref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Supp
   \item[VIRTIO_F_VERSION_1(32)] This indicates compliance with this
     specification, giving a simple way to detect legacy devices or drivers.
 
-  \item[VIRTIO_F_IOMMU_PLATFORM(33)] This feature indicates that the device is
+  \item[VIRTIO_F_ACCESS_PLATFORM(33)] This feature indicates that
+  the device can be used in a platform where device access to data
+  in memory is limited and/or translated. E.g. this is the case if the device can be located
   behind an IOMMU that translates bus addresses from the device into physical
-  addresses in memory.  If this feature bit is set to 0, then the device emits
-  physical addresses which are not translated further, even though an IOMMU
-  may be present.
+  addresses in memory, if device can be limited to only access
+  certain memory addresses or if special commands such as
+  a cache flush can be needed to synchronise data in memory with
+  the device. Whether accesses are actually limited or translated
+  is described by platform-specific means.
+  If this feature bit is set to 0, then the device
+  has same access to memory addresses supplied to it as the driver.
+  In particular it will always use physical addresses
+  matching addresses used by driver (typically the CPU)
+  and not translated further, and can access any address supplied to it by
+  driver. When clear, this overrides any platform-specific description of
+  whether device access is limited or translated in any way, e.g.
+  whether an IOMMU may be present.
   \item[VIRTIO_F_RING_PACKED(34)] This feature indicates
   support for the packed virtqueue layout as described in
   \ref{sec:Basic Facilities of a Virtio Device / Packed Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device / Packed Virtqueues}.
   \item[VIRTIO_F_IN_ORDER(35)] This feature indicates
   that all buffers are used by the device in the same
   order in which they have been made available.
-  \item[VIRTIO_F_IO_BARRIER(36)] This feature indicates
-  that the device needs the driver to use the barriers
-  suitable for hardware devices.  Some transports require
-  barriers to ensure devices have a consistent view of
-  memory.  When devices are implemented in software a
-  weaker form of barrier may be sufficient and yield
-  better performance.  This feature indicates whether
-  a stronger form of barrier suitable for hardware
-  devices is necessary.
+  \item[VIRTIO_F_ORDER_PLATFORM(36)] This feature indicates
+  that memory accesses by driver and device are ordered
+  in a way described by the platform.
+  If this feature bit is negotiated, then for any memory
+  accesses by the driver that need to be ordered
+  in a specific way with respect to access by
+  the device, the ordering in effect is the one
+  suitable for devices described by the platform,
+  and he driver needs to use memory barriers
+  suitable for devices described by the platform,
+  e.g. for the PCI transport, for hardware PCI devices.
+  If this feature bit is set to 0, then the device
+  and driver are assumed to be implemented in software, that is
+  they can be assumed to run on identical CPUs
+  in an SMP configuration.
+  Thus a weaker form of memory barriers is sufficient
+  to yield better performance.
   \item[VIRTIO_F_SR_IOV(37)] This feature indicates that
   the device supports Single Root I/O Virtualization.
   Currently only PCI devices support this feature.
@@ -5482,16 +5502,16 @@ Descriptors} and \ref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Supp
 A driver MUST accept VIRTIO_F_VERSION_1 if it is offered.  A driver
 MAY fail to operate further if VIRTIO_F_VERSION_1 is not offered.
 
-A driver SHOULD accept VIRTIO_F_IOMMU_PLATFORM if it is offered, and it MUST
+A driver SHOULD accept VIRTIO_F_ACCESS_PLATFORM if it is offered, and it MUST
 then either disable the IOMMU or configure the IOMMU to translate bus addresses
 passed to the device into physical addresses in memory.  If
-VIRTIO_F_IOMMU_PLATFORM is not offered, then a driver MUST pass only physical
+VIRTIO_F_ACCESS_PLATFORM is not offered, then a driver MUST pass only physical
 addresses to the device.
 
 A driver SHOULD accept VIRTIO_F_RING_PACKED if it is offered.
 
-A driver SHOULD accept VIRTIO_F_IO_BARRIER if it is offered.
-If VIRTIO_F_IO_BARRIER has been negotiated, a driver MUST use
+A driver SHOULD accept VIRTIO_F_ORDER_PLATFORM if it is offered.
+If VIRTIO_F_ORDER_PLATFORM has been negotiated, a driver MUST use
 the barriers suitable for hardware devices.
 
 \devicenormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
@@ -5499,15 +5519,21 @@ the barriers suitable for hardware devices.
 A device MUST offer VIRTIO_F_VERSION_1.  A device MAY fail to operate further
 if VIRTIO_F_VERSION_1 is not accepted.
 
-A device SHOULD offer VIRTIO_F_IOMMU_PLATFORM if it is behind an IOMMU that
-translates bus addresses from the device into physical addresses in memory.
-A device MAY fail to operate further if VIRTIO_F_IOMMU_PLATFORM is not
+A device SHOULD offer VIRTIO_F_ACCESS_PLATFORM if it's access to
+memory is through bus addresses distinct from and translated
+by the platform to physical addresses used by the driver, and/or
+if it can only access certain memory addresses with said access
+specified and/or granted by the platform.
+A device MAY fail to operate further if VIRTIO_F_ACCESS_PLATFORM is not
 accepted.
 
 If VIRTIO_F_IN_ORDER has been negotiated, a device MUST use
 buffers in the same order in which they have been available.
 
-A device MAY fail to operate further if VIRTIO_F_IO_BARRIER
+While a device MAY fail to operate further if VIRTIO_F_ORDER_PLATFORM
+is not accepted, it is RECOMMENDED that a device offering
+VIRTIO_F_ORDER_PLATFORM does operate possibly in a slower
+software-emulation mode if VIRTIO_F_ORDER_PLATFORM
 is not accepted.
 
 A device SHOULD offer VIRTIO_F_SR_IOV if it is a PCI device
-- 
MST

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio-dev] [PATCH] ACCESS_PLATFORM/ACCESS_ORDER

[Jason Wang]

On 2018/11/21 上午11:09, Michael S. Tsirkin wrote:
> This is an attempt to clarify the intent behind
> VIRTIO_F_IOMMU_PLATFORM and VIRTIO_F_IO_BARRIER which based on
> recent discussions appear not to be well documented,
> in particular not matching what drivers do or are
> likely to do.
>
> - rename VIRTIO_F_IOMMU_PLATFORM to ACCESS_PLATFORM
>    It is already the fact that the DMA API that Linux
>    uses does more than just IOMMUs - it includes
>    cache flushing, bounce buffers for limited
>    addressing, etc.
>    Update spec to match this reality.
>
> - rename VIRTIO_F_IO_BARRIER to VIRTIO_F_ORDER_PLATFORM
>    this is after all what device is telling driver:
>    its memory accesses are only ordered weakly,
>    this is why a stronger barrier is required.
>
> - As no one yet implemented IO_BARRIER yet, add a recommendation
>    to have a software fallback so that existing drivers
>    aren't broken.
>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>   content.tex | 68 ++++++++++++++++++++++++++++++++++++-----------------
>   1 file changed, 47 insertions(+), 21 deletions(-)
>
> diff --git a/content.tex b/content.tex
> index c346183..4fef3ad 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -5452,26 +5452,46 @@ Descriptors} and \ref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Supp
>     \item[VIRTIO_F_VERSION_1(32)] This indicates compliance with this
>       specification, giving a simple way to detect legacy devices or drivers.
>   
> -  \item[VIRTIO_F_IOMMU_PLATFORM(33)] This feature indicates that the device is
> +  \item[VIRTIO_F_ACCESS_PLATFORM(33)] This feature indicates that
> +  the device can be used in a platform where device access to data
> +  in memory is limited and/or translated. E.g. this is the case if the device can be located
>     behind an IOMMU that translates bus addresses from the device into physical
> -  addresses in memory.  If this feature bit is set to 0, then the device emits
> -  physical addresses which are not translated further, even though an IOMMU
> -  may be present.
> +  addresses in memory, if device can be limited to only access
> +  certain memory addresses or if special commands such as
> +  a cache flush can be needed to synchronise data in memory with
> +  the device. Whether accesses are actually limited or translated
> +  is described by platform-specific means.
> +  If this feature bit is set to 0, then the device
> +  has same access to memory addresses supplied to it as the driver.
> +  In particular it will always use physical addresses
> +  matching addresses used by driver (typically the CPU)
> +  and not translated further, and can access any address supplied to it by
> +  driver. When clear, this overrides any platform-specific description of
> +  whether device access is limited or translated in any way, e.g.
> +  whether an IOMMU may be present.
>     \item[VIRTIO_F_RING_PACKED(34)] This feature indicates
>     support for the packed virtqueue layout as described in
>     \ref{sec:Basic Facilities of a Virtio Device / Packed Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device / Packed Virtqueues}.
>     \item[VIRTIO_F_IN_ORDER(35)] This feature indicates
>     that all buffers are used by the device in the same
>     order in which they have been made available.
> -  \item[VIRTIO_F_IO_BARRIER(36)] This feature indicates
> -  that the device needs the driver to use the barriers
> -  suitable for hardware devices.  Some transports require
> -  barriers to ensure devices have a consistent view of
> -  memory.  When devices are implemented in software a
> -  weaker form of barrier may be sufficient and yield
> -  better performance.  This feature indicates whether
> -  a stronger form of barrier suitable for hardware
> -  devices is necessary.
> +  \item[VIRTIO_F_ORDER_PLATFORM(36)] This feature indicates
> +  that memory accesses by driver and device are ordered
> +  in a way described by the platform.
> +  If this feature bit is negotiated, then for any memory
> +  accesses by the driver that need to be ordered
> +  in a specific way with respect to access by
> +  the device, the ordering in effect is the one
> +  suitable for devices described by the platform,
> +  and he driver needs to use memory barriers
> +  suitable for devices described by the platform,
> +  e.g. for the PCI transport, for hardware PCI devices.
> +  If this feature bit is set to 0, then the device
> +  and driver are assumed to be implemented in software, that is
> +  they can be assumed to run on identical CPUs
> +  in an SMP configuration.
> +  Thus a weaker form of memory barriers is sufficient
> +  to yield better performance.


A question is. Are you suggesting to have VIRTIO_F_ORDER_PLATFORM for 
each hardware implementation? If not, can a hardware detect the platform 
by itself and advertise this bit automatically?

ACCESS_PLATFORM has similar question. The question is about the 
necessarily of duplicating platform specific feature detection into a 
device feature.

Thanks


>     \item[VIRTIO_F_SR_IOV(37)] This feature indicates that
>     the device supports Single Root I/O Virtualization.
>     Currently only PCI devices support this feature.
> @@ -5482,16 +5502,16 @@ Descriptors} and \ref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Supp
>   A driver MUST accept VIRTIO_F_VERSION_1 if it is offered.  A driver
>   MAY fail to operate further if VIRTIO_F_VERSION_1 is not offered.
>   
> -A driver SHOULD accept VIRTIO_F_IOMMU_PLATFORM if it is offered, and it MUST
> +A driver SHOULD accept VIRTIO_F_ACCESS_PLATFORM if it is offered, and it MUST
>   then either disable the IOMMU or configure the IOMMU to translate bus addresses
>   passed to the device into physical addresses in memory.  If
> -VIRTIO_F_IOMMU_PLATFORM is not offered, then a driver MUST pass only physical
> +VIRTIO_F_ACCESS_PLATFORM is not offered, then a driver MUST pass only physical
>   addresses to the device.
>   
>   A driver SHOULD accept VIRTIO_F_RING_PACKED if it is offered.
>   
> -A driver SHOULD accept VIRTIO_F_IO_BARRIER if it is offered.
> -If VIRTIO_F_IO_BARRIER has been negotiated, a driver MUST use
> +A driver SHOULD accept VIRTIO_F_ORDER_PLATFORM if it is offered.
> +If VIRTIO_F_ORDER_PLATFORM has been negotiated, a driver MUST use
>   the barriers suitable for hardware devices.
>   
>   \devicenormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
> @@ -5499,15 +5519,21 @@ the barriers suitable for hardware devices.
>   A device MUST offer VIRTIO_F_VERSION_1.  A device MAY fail to operate further
>   if VIRTIO_F_VERSION_1 is not accepted.
>   
> -A device SHOULD offer VIRTIO_F_IOMMU_PLATFORM if it is behind an IOMMU that
> -translates bus addresses from the device into physical addresses in memory.
> -A device MAY fail to operate further if VIRTIO_F_IOMMU_PLATFORM is not
> +A device SHOULD offer VIRTIO_F_ACCESS_PLATFORM if it's access to
> +memory is through bus addresses distinct from and translated
> +by the platform to physical addresses used by the driver, and/or
> +if it can only access certain memory addresses with said access
> +specified and/or granted by the platform.
> +A device MAY fail to operate further if VIRTIO_F_ACCESS_PLATFORM is not
>   accepted.
>   
>   If VIRTIO_F_IN_ORDER has been negotiated, a device MUST use
>   buffers in the same order in which they have been available.
>   
> -A device MAY fail to operate further if VIRTIO_F_IO_BARRIER
> +While a device MAY fail to operate further if VIRTIO_F_ORDER_PLATFORM
> +is not accepted, it is RECOMMENDED that a device offering
> +VIRTIO_F_ORDER_PLATFORM does operate possibly in a slower
> +software-emulation mode if VIRTIO_F_ORDER_PLATFORM
>   is not accepted.
>   
>   A device SHOULD offer VIRTIO_F_SR_IOV if it is a PCI device

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio] Re: [virtio-dev] [PATCH] ACCESS_PLATFORM/ACCESS_ORDER

[Michael S. Tsirkin]

On Wed, Nov 21, 2018 at 12:07:34PM +0800, Jason Wang wrote:
> 
> On 2018/11/21 上午11:09, Michael S. Tsirkin wrote:
> > This is an attempt to clarify the intent behind
> > VIRTIO_F_IOMMU_PLATFORM and VIRTIO_F_IO_BARRIER which based on
> > recent discussions appear not to be well documented,
> > in particular not matching what drivers do or are
> > likely to do.
> > 
> > - rename VIRTIO_F_IOMMU_PLATFORM to ACCESS_PLATFORM
> >    It is already the fact that the DMA API that Linux
> >    uses does more than just IOMMUs - it includes
> >    cache flushing, bounce buffers for limited
> >    addressing, etc.
> >    Update spec to match this reality.
> > 
> > - rename VIRTIO_F_IO_BARRIER to VIRTIO_F_ORDER_PLATFORM
> >    this is after all what device is telling driver:
> >    its memory accesses are only ordered weakly,
> >    this is why a stronger barrier is required.
> > 
> > - As no one yet implemented IO_BARRIER yet, add a recommendation
> >    to have a software fallback so that existing drivers
> >    aren't broken.
> > 
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > ---
> >   content.tex | 68 ++++++++++++++++++++++++++++++++++++-----------------
> >   1 file changed, 47 insertions(+), 21 deletions(-)
> > 
> > diff --git a/content.tex b/content.tex
> > index c346183..4fef3ad 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -5452,26 +5452,46 @@ Descriptors} and \ref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Supp
> >     \item[VIRTIO_F_VERSION_1(32)] This indicates compliance with this
> >       specification, giving a simple way to detect legacy devices or drivers.
> > -  \item[VIRTIO_F_IOMMU_PLATFORM(33)] This feature indicates that the device is
> > +  \item[VIRTIO_F_ACCESS_PLATFORM(33)] This feature indicates that
> > +  the device can be used in a platform where device access to data
> > +  in memory is limited and/or translated. E.g. this is the case if the device can be located
> >     behind an IOMMU that translates bus addresses from the device into physical
> > -  addresses in memory.  If this feature bit is set to 0, then the device emits
> > -  physical addresses which are not translated further, even though an IOMMU
> > -  may be present.
> > +  addresses in memory, if device can be limited to only access
> > +  certain memory addresses or if special commands such as
> > +  a cache flush can be needed to synchronise data in memory with
> > +  the device. Whether accesses are actually limited or translated
> > +  is described by platform-specific means.
> > +  If this feature bit is set to 0, then the device
> > +  has same access to memory addresses supplied to it as the driver.
> > +  In particular it will always use physical addresses
> > +  matching addresses used by driver (typically the CPU)
> > +  and not translated further, and can access any address supplied to it by
> > +  driver. When clear, this overrides any platform-specific description of
> > +  whether device access is limited or translated in any way, e.g.
> > +  whether an IOMMU may be present.
> >     \item[VIRTIO_F_RING_PACKED(34)] This feature indicates
> >     support for the packed virtqueue layout as described in
> >     \ref{sec:Basic Facilities of a Virtio Device / Packed Virtqueues}~\nameref{sec:Basic Facilities of a Virtio Device / Packed Virtqueues}.
> >     \item[VIRTIO_F_IN_ORDER(35)] This feature indicates
> >     that all buffers are used by the device in the same
> >     order in which they have been made available.
> > -  \item[VIRTIO_F_IO_BARRIER(36)] This feature indicates
> > -  that the device needs the driver to use the barriers
> > -  suitable for hardware devices.  Some transports require
> > -  barriers to ensure devices have a consistent view of
> > -  memory.  When devices are implemented in software a
> > -  weaker form of barrier may be sufficient and yield
> > -  better performance.  This feature indicates whether
> > -  a stronger form of barrier suitable for hardware
> > -  devices is necessary.
> > +  \item[VIRTIO_F_ORDER_PLATFORM(36)] This feature indicates
> > +  that memory accesses by driver and device are ordered
> > +  in a way described by the platform.
> > +  If this feature bit is negotiated, then for any memory
> > +  accesses by the driver that need to be ordered
> > +  in a specific way with respect to access by
> > +  the device, the ordering in effect is the one
> > +  suitable for devices described by the platform,
> > +  and he driver needs to use memory barriers
> > +  suitable for devices described by the platform,
> > +  e.g. for the PCI transport, for hardware PCI devices.
> > +  If this feature bit is set to 0, then the device
> > +  and driver are assumed to be implemented in software, that is
> > +  they can be assumed to run on identical CPUs
> > +  in an SMP configuration.
> > +  Thus a weaker form of memory barriers is sufficient
> > +  to yield better performance.
> 
> 
> A question is. Are you suggesting to have VIRTIO_F_ORDER_PLATFORM for each
> hardware implementation?

It's not a simple question. There are implementations such as
VDPA which have a hardware and a software component.
For a hardware only device that is also pluggable into an
arbitrary system, it's true.

> If not, can a hardware detect the platform by
> itself and advertise this bit automatically?

So a mixed software/hardware device might be able to,
and e.g. fallback to a software implementation
if driver assumes that.

There's actually some text that suggests this - is
that not clear then?


>
> ACCESS_PLATFORM has similar question.

That one is a bit different in that guest might rely on e.g.
vIOMMU for security.  So depending on platform device might want
to fail feature negotiation.

But it's not a problem for e.g. SEV since a legacy non encrypted
guest will just work.

Maybe a new section about security might make sense.

> The question is about the necessarily
> of duplicating platform specific feature detection into a device feature.

Historically the assumption drivers made was that
a device is exactly like driver: same physical address,
same permissions and ordering for memory accesses.

Nowdays some drivers make the assumptions about ordering
but not permissions and physical address.

We can't just ignore existing drivers, can we?
So a device needs to know what it's dealing with, to
either fail gracefully or recover by a software fallback.


> Thanks

All good questions, though I'm not sure whether they mean we
need to add some text to the spec. Suggestions about
what should be documented?

> 
> >     \item[VIRTIO_F_SR_IOV(37)] This feature indicates that
> >     the device supports Single Root I/O Virtualization.
> >     Currently only PCI devices support this feature.
> > @@ -5482,16 +5502,16 @@ Descriptors} and \ref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Supp
> >   A driver MUST accept VIRTIO_F_VERSION_1 if it is offered.  A driver
> >   MAY fail to operate further if VIRTIO_F_VERSION_1 is not offered.
> > -A driver SHOULD accept VIRTIO_F_IOMMU_PLATFORM if it is offered, and it MUST
> > +A driver SHOULD accept VIRTIO_F_ACCESS_PLATFORM if it is offered, and it MUST
> >   then either disable the IOMMU or configure the IOMMU to translate bus addresses
> >   passed to the device into physical addresses in memory.  If
> > -VIRTIO_F_IOMMU_PLATFORM is not offered, then a driver MUST pass only physical
> > +VIRTIO_F_ACCESS_PLATFORM is not offered, then a driver MUST pass only physical
> >   addresses to the device.
> >   A driver SHOULD accept VIRTIO_F_RING_PACKED if it is offered.
> > -A driver SHOULD accept VIRTIO_F_IO_BARRIER if it is offered.
> > -If VIRTIO_F_IO_BARRIER has been negotiated, a driver MUST use
> > +A driver SHOULD accept VIRTIO_F_ORDER_PLATFORM if it is offered.
> > +If VIRTIO_F_ORDER_PLATFORM has been negotiated, a driver MUST use
> >   the barriers suitable for hardware devices.
> >   \devicenormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
> > @@ -5499,15 +5519,21 @@ the barriers suitable for hardware devices.
> >   A device MUST offer VIRTIO_F_VERSION_1.  A device MAY fail to operate further
> >   if VIRTIO_F_VERSION_1 is not accepted.
> > -A device SHOULD offer VIRTIO_F_IOMMU_PLATFORM if it is behind an IOMMU that
> > -translates bus addresses from the device into physical addresses in memory.
> > -A device MAY fail to operate further if VIRTIO_F_IOMMU_PLATFORM is not
> > +A device SHOULD offer VIRTIO_F_ACCESS_PLATFORM if it's access to
> > +memory is through bus addresses distinct from and translated
> > +by the platform to physical addresses used by the driver, and/or
> > +if it can only access certain memory addresses with said access
> > +specified and/or granted by the platform.
> > +A device MAY fail to operate further if VIRTIO_F_ACCESS_PLATFORM is not
> >   accepted.
> >   If VIRTIO_F_IN_ORDER has been negotiated, a device MUST use
> >   buffers in the same order in which they have been available.
> > -A device MAY fail to operate further if VIRTIO_F_IO_BARRIER
> > +While a device MAY fail to operate further if VIRTIO_F_ORDER_PLATFORM
> > +is not accepted, it is RECOMMENDED that a device offering
> > +VIRTIO_F_ORDER_PLATFORM does operate possibly in a slower
> > +software-emulation mode if VIRTIO_F_ORDER_PLATFORM
> >   is not accepted.
> >   A device SHOULD offer VIRTIO_F_SR_IOV if it is a PCI device

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio-dev] [PATCH] ACCESS_PLATFORM/ACCESS_ORDER

[Jason Wang]

On 2018/11/21 下午1:11, Michael S. Tsirkin wrote:
>> A question is. Are you suggesting to have VIRTIO_F_ORDER_PLATFORM for each
>> hardware implementation?
> It's not a simple question. There are implementations such as
> VDPA which have a hardware and a software component.
> For a hardware only device that is also pluggable into an
> arbitrary system, it's true.
>
>> If not, can a hardware detect the platform by
>> itself and advertise this bit automatically?
> So a mixed software/hardware device might be able to,
> and e.g. fallback to a software implementation
> if driver assumes that.
>
> There's actually some text that suggests this - is
> that not clear then?


Maybe it's better to add text about the suggestion of hardware 
implementation.


>
>
>> ACCESS_PLATFORM has similar question.
> That one is a bit different in that guest might rely on e.g.
> vIOMMU for security.  So depending on platform device might want
> to fail feature negotiation.
>
> But it's not a problem for e.g. SEV since a legacy non encrypted
> guest will just work.


How about a encrypted guest that wants to use software IOTLB?


>
> Maybe a new section about security might make sense.
>
>> The question is about the necessarily
>> of duplicating platform specific feature detection into a device feature.
> Historically the assumption drivers made was that
> a device is exactly like driver: same physical address,
> same permissions and ordering for memory accesses.
>
> Nowdays some drivers make the assumptions about ordering
> but not permissions and physical address.
>
> We can't just ignore existing drivers, can we?


It can only work if both features were supported and advertised by 
devices. Consider the case that IOMMU is enable but iommu_platform is 
not, in this case:

1) we still can't recognize old guests, so it won't fail gracefully.

2) new guests can't work


It depends on how much we can gain from failing gracefully. I believe we 
don't want to force a software fallback. So I wonder how bad just let 
existing drivers fail and try to patch and backport the fixes without 
introducing new feature bits. With this we can offload those tasks to 
platform or configuration specific codes instead of our own.


Another question is whether or not we need two separate feature bits. 
E.g when should we set one bit but not for another?


> So a device needs to know what it's dealing with, to
> either fail gracefully or recover by a software fallback.
>
>
>> Thanks
> All good questions, though I'm not sure whether they mean we
> need to add some text to the spec. Suggestions about
> what should be documented?
>

Maybe it's better to clarify the suggestion for hardware or mixed 
implementations.

Thanks


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio] Re: [virtio-dev] [PATCH] ACCESS_PLATFORM/ACCESS_ORDER

[Michael S. Tsirkin]

On Wed, Nov 21, 2018 at 03:35:21PM +0800, Jason Wang wrote:
> 
> On 2018/11/21 下午1:11, Michael S. Tsirkin wrote:
> > > A question is. Are you suggesting to have VIRTIO_F_ORDER_PLATFORM for each
> > > hardware implementation?
> > It's not a simple question. There are implementations such as
> > VDPA which have a hardware and a software component.
> > For a hardware only device that is also pluggable into an
> > arbitrary system, it's true.
> > 
> > > If not, can a hardware detect the platform by
> > > itself and advertise this bit automatically?
> > So a mixed software/hardware device might be able to,
> > and e.g. fallback to a software implementation
> > if driver assumes that.
> > 
> > There's actually some text that suggests this - is
> > that not clear then?
> 
> 
> Maybe it's better to add text about the suggestion of hardware
> implementation.

As I tried to explain, what exactly is "hardware implementation"
is very much platform dependent so I don't know how to do that.
Ideas?

> 
> > 
> > 
> > > ACCESS_PLATFORM has similar question.
> > That one is a bit different in that guest might rely on e.g.
> > vIOMMU for security.  So depending on platform device might want
> > to fail feature negotiation.
> > 
> > But it's not a problem for e.g. SEV since a legacy non encrypted
> > guest will just work.
> 
> 
> How about a encrypted guest that wants to use software IOTLB?

Software iotlb by itself does not *have* to affect ACCESS_PLATFORM - it is
enough to just avoid giving device any memory outside the IOTLB.
However it is probably more robust to set ACCESS_PLATFORM in
that setup to avoid any confusion.

> 
> > 
> > Maybe a new section about security might make sense.
> > 
> > > The question is about the necessarily
> > > of duplicating platform specific feature detection into a device feature.
> > Historically the assumption drivers made was that
> > a device is exactly like driver: same physical address,
> > same permissions and ordering for memory accesses.
> > 
> > Nowdays some drivers make the assumptions about ordering
> > but not permissions and physical address.
> > 
> > We can't just ignore existing drivers, can we?
> 
> 
> It can only work if both features were supported and advertised by devices.
> Consider the case that IOMMU is enable but iommu_platform is not,

So this is a critique of existing spec text then?
Given plans for an imminent 1.1 release let's focus on
the proposed patch, ok?


> in this
> case:
> 
> 1) we still can't recognize old guests, so it won't fail gracefully.
> 
> 2) new guests can't work
> 
> 
> It depends on how much we can gain from failing gracefully. I believe we
> don't want to force a software fallback.

iommu_platform fundamentally doesn't support a software fallback,
it's a security feature. So I agree but what gave you the idea that we
force a software fallback? So ORDER_PLATFORM it seems prodent to
suggest a software fallback given not one driver in the field sets it.

Would your comment be addressed if I replace RECOMMENDED with MAY?

> So I wonder how bad just let
> existing drivers fail and try to patch and backport the fixes without
> introducing new feature bits. With this we can offload those tasks to
> platform or configuration specific codes instead of our own.
> 
> 
> Another question is whether or not we need two separate feature bits.

Well IOMMU_PLATFORM is already set in the field.
We can't make it also imply strong barriers without
breaking a bunch of existing setups.

> E.g
> when should we set one bit but not for another?

I am sorry I don't understand the question.  I did try to
document what does each bit say. Is it completely unclear then?
Could you give some hints on what is unclear? Which kind
of functionality is covered by both such that it's
an open question which one to set?

> 
> > So a device needs to know what it's dealing with, to
> > either fail gracefully or recover by a software fallback.
> > 
> > 
> > > Thanks
> > All good questions, though I'm not sure whether they mean we
> > need to add some text to the spec. Suggestions about
> > what should be documented?
> > 
> 
> Maybe it's better to clarify the suggestion for hardware or mixed
> implementations.
> 
> Thanks

Right but I don't know what to suggest. E.g. intel recently
published data about a hardware vDPA which does not set either
bits. They are working on support for a vIOMMU at which point
they will set IOMMU_PLATFORM. Once they start worrying about
portability they might add ORDER_PLATFORM. But given it's
VDPA they don't necessarily need to fail without even
on weakly ordered platforms.

-- 
MST

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

Re: [virtio-dev] [PATCH] ACCESS_PLATFORM/ACCESS_ORDER

[Jason Wang]

On 2018/11/21 下午8:10, Michael S. Tsirkin wrote:
> On Wed, Nov 21, 2018 at 03:35:21PM +0800, Jason Wang wrote:
>> On 2018/11/21 下午1:11, Michael S. Tsirkin wrote:
>>>> A question is. Are you suggesting to have VIRTIO_F_ORDER_PLATFORM for each
>>>> hardware implementation?
>>> It's not a simple question. There are implementations such as
>>> VDPA which have a hardware and a software component.
>>> For a hardware only device that is also pluggable into an
>>> arbitrary system, it's true.
>>>
>>>> If not, can a hardware detect the platform by
>>>> itself and advertise this bit automatically?
>>> So a mixed software/hardware device might be able to,
>>> and e.g. fallback to a software implementation
>>> if driver assumes that.
>>>
>>> There's actually some text that suggests this - is
>>> that not clear then?
>>
>> Maybe it's better to add text about the suggestion of hardware
>> implementation.
> As I tried to explain, what exactly is "hardware implementation"
> is very much platform dependent so I don't know how to do that.
> Ideas?
>   


Not a native speaker, but if any part of was done through emulation, it 
should be treated as a software implementation. Otherwise, it's a 
hardware emulation.

Or it's a common implication that it's impossible to implement a 
hardware without ACCESS_PLATFORM (e.g for PCI how can a device bypass 
IOMMU address translation by its own?).


>>>
>>>> ACCESS_PLATFORM has similar question.
>>> That one is a bit different in that guest might rely on e.g.
>>> vIOMMU for security.  So depending on platform device might want
>>> to fail feature negotiation.
>>>
>>> But it's not a problem for e.g. SEV since a legacy non encrypted
>>> guest will just work.
>>
>> How about a encrypted guest that wants to use software IOTLB?
> Software iotlb by itself does not *have* to affect ACCESS_PLATFORM - it is
> enough to just avoid giving device any memory outside the IOTLB.


The problem is, if e.g DMA core can recognize the case of software IOTLB 
without ACCESS_PLATFORM, it can recognize IOMMU, bounce buffer etc as 
well. If we can offload those things to platform/DMA codes, there's no 
need for ACCESS_PLATFORM at all.


> However it is probably more robust to set ACCESS_PLATFORM in
> that setup to avoid any confusion.


So it's the responsibility of management to detect the case and set the 
bit or just set it unconditionally.


>
>>> Maybe a new section about security might make sense.
>>>
>>>> The question is about the necessarily
>>>> of duplicating platform specific feature detection into a device feature.
>>> Historically the assumption drivers made was that
>>> a device is exactly like driver: same physical address,
>>> same permissions and ordering for memory accesses.
>>>
>>> Nowdays some drivers make the assumptions about ordering
>>> but not permissions and physical address.
>>>
>>> We can't just ignore existing drivers, can we?
>>
>> It can only work if both features were supported and advertised by devices.
>> Consider the case that IOMMU is enable but iommu_platform is not,
> So this is a critique of existing spec text then?


No. I meant it only work if ACCESS_PLATFORM and ACCESS_ORDER were set 
correctly.


> Given plans for an imminent 1.1 release let's focus on
> the proposed patch, ok?
>
>
>> in this
>> case:
>>
>> 1) we still can't recognize old guests, so it won't fail gracefully.
>>
>> 2) new guests can't work
>>
>>
>> It depends on how much we can gain from failing gracefully. I believe we
>> don't want to force a software fallback.
> iommu_platform fundamentally doesn't support a software fallback,
> it's a security feature. So I agree but what gave you the idea that we
> force a software fallback?


I don't suggest a software fallback. I meant without a fallback:

- when IOMMU_PLATFORM is not advertised, there are still cases that 
IOMMU_PLATFORM can't handle (the above two cases).

- when IOMMU_PLATFORM is advertised (or could not be turned of for e.g 
hardware implementation), the only difference is a graceful failure or 
silent failure for old guest, but how much important is the case?

Consider the above cases, is it better just implement the semantics of 
IOMMU_PLATFORM silently and deprecate it in the future. Then we may only 
meet one silent failure when driver doesn't correctly detect platform 
requirements.


> So ORDER_PLATFORM it seems prodent to
> suggest a software fallback given not one driver in the field sets it.
>
> Would your comment be addressed if I replace RECOMMENDED with MAY?
>
>> So I wonder how bad just let
>> existing drivers fail and try to patch and backport the fixes without
>> introducing new feature bits. With this we can offload those tasks to
>> platform or configuration specific codes instead of our own.
>>
>>
>> Another question is whether or not we need two separate feature bits.
> Well IOMMU_PLATFORM is already set in the field.
> We can't make it also imply strong barriers without
> breaking a bunch of existing setups.


Or just imply the IOMMU_PLATFORM if ACCESS_ORDER is set.


>
>> E.g
>> when should we set one bit but not for another?
> I am sorry I don't understand the question.  I did try to
> document what does each bit say. Is it completely unclear then?
> Could you give some hints on what is unclear? Which kind
> of functionality is covered by both such that it's
> an open question which one to set?


I meant, e.g spec allow the case of advertising ACCES_ORDER but not 
advertising ACCESS_PLATFORM. This sounds somehow ambiguous since both 
feature are memory related but part were accessed in a way what platform 
requires but the rest doesn't.


>
>>> So a device needs to know what it's dealing with, to
>>> either fail gracefully or recover by a software fallback.
>>>
>>>
>>>> Thanks
>>> All good questions, though I'm not sure whether they mean we
>>> need to add some text to the spec. Suggestions about
>>> what should be documented?
>>>
>> Maybe it's better to clarify the suggestion for hardware or mixed
>> implementations.
>>
>> Thanks
> Right but I don't know what to suggest. E.g. intel recently
> published data about a hardware vDPA which does not set either
> bits.


So this violates spec. Device actually will go through IOMMU but driver 
won't use DMA API at all. But since the control path is under the 
control of software, it's no hard to advertise it I believe. If not, 
it's another call for clarifying ACCESS_PLATFORM/ACCESS_ORDER is 
mandatory for hardware.


> They are working on support for a vIOMMU at which point
> they will set IOMMU_PLATFORM. Once they start worrying about
> portability they might add ORDER_PLATFORM. But given it's
> VDPA they don't necessarily need to fail without even
> on weakly ordered platforms.


If feature bit advertising could be controlled by software it probably 
won't need any suggestion. We probably need suggest only for full 
hardware implementation without any software emulation.

Thanks.


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio] Re: [virtio-dev] [PATCH] ACCESS_PLATFORM/ACCESS_ORDER

[Michael S. Tsirkin]

On Wed, Nov 21, 2018 at 09:44:02PM +0800, Jason Wang wrote:
> 
> On 2018/11/21 下午8:10, Michael S. Tsirkin wrote:
> > On Wed, Nov 21, 2018 at 03:35:21PM +0800, Jason Wang wrote:
> > > On 2018/11/21 下午1:11, Michael S. Tsirkin wrote:
> > > > > A question is. Are you suggesting to have VIRTIO_F_ORDER_PLATFORM for each
> > > > > hardware implementation?
> > > > It's not a simple question. There are implementations such as
> > > > VDPA which have a hardware and a software component.
> > > > For a hardware only device that is also pluggable into an
> > > > arbitrary system, it's true.
> > > > 
> > > > > If not, can a hardware detect the platform by
> > > > > itself and advertise this bit automatically?
> > > > So a mixed software/hardware device might be able to,
> > > > and e.g. fallback to a software implementation
> > > > if driver assumes that.
> > > > 
> > > > There's actually some text that suggests this - is
> > > > that not clear then?
> > > 
> > > Maybe it's better to add text about the suggestion of hardware
> > > implementation.
> > As I tried to explain, what exactly is "hardware implementation"
> > is very much platform dependent so I don't know how to do that.
> > Ideas?
> 
> 
> Not a native speaker, but if any part of was done through emulation, it
> should be treated as a software implementation. Otherwise, it's a hardware
> emulation.
> 
> Or it's a common implication that it's impossible to implement a hardware
> without ACCESS_PLATFORM (e.g for PCI how can a device bypass IOMMU address
> translation by its own?).

So VDPA is one example, the IOMMU is emulated but the PCI device
is passed through.

> 
> > > > 
> > > > > ACCESS_PLATFORM has similar question.
> > > > That one is a bit different in that guest might rely on e.g.
> > > > vIOMMU for security.  So depending on platform device might want
> > > > to fail feature negotiation.
> > > > 
> > > > But it's not a problem for e.g. SEV since a legacy non encrypted
> > > > guest will just work.
> > > 
> > > How about a encrypted guest that wants to use software IOTLB?
> > Software iotlb by itself does not *have* to affect ACCESS_PLATFORM - it is
> > enough to just avoid giving device any memory outside the IOTLB.
> 
> 
> The problem is, if e.g DMA core can recognize the case of software IOTLB
> without ACCESS_PLATFORM, it can recognize IOMMU, bounce buffer etc as well.
> If we can offload those things to platform/DMA codes, there's no need for
> ACCESS_PLATFORM at all.

I doubt it is possible and it did not happen in X years before
IOMMU_PLATFORM was introduced. But it seems that you really
have an issue with IOMMU_PLATFORM. Is ACCESS_PLATFORM making
things any worse in your opinion?

Going back to IOMMU_PLATFORM, there are good reasons it was
introduced.  
The assumption for years was that device is part of the
hypervisor which is exactly like driver but even more powerful.
Platforms used this fact for optimizations.
Yes they could try to come up with platform specific ways to
implement these optimizations, a virtio specific one was
easier for them though.

Now there are two ways this can be changing:
- device access to memory might be limited as compared to driver
- device access ordering might differ as compared to driver

You are asking why aren't we requiring platforms to come up with platform
specific ways to optimize the hypervisor case now?
Because
- things work for them and its our responsibility not to break
  them
- nothing changed, for some platforms it is still hard to come
  up with such ways

> 
> > However it is probably more robust to set ACCESS_PLATFORM in
> > that setup to avoid any confusion.
> 
> 
> So it's the responsibility of management to detect the case and set the bit
> or just set it unconditionally.

If device can't access all memory it's likely a good idea to tell
driver so, yes. Even though maybe it can also figure this out on
its own.

> 
> > 
> > > > Maybe a new section about security might make sense.
> > > > 
> > > > > The question is about the necessarily
> > > > > of duplicating platform specific feature detection into a device feature.
> > > > Historically the assumption drivers made was that
> > > > a device is exactly like driver: same physical address,
> > > > same permissions and ordering for memory accesses.
> > > > 
> > > > Nowdays some drivers make the assumptions about ordering
> > > > but not permissions and physical address.
> > > > 
> > > > We can't just ignore existing drivers, can we?
> > > 
> > > It can only work if both features were supported and advertised by devices.
> > > Consider the case that IOMMU is enable but iommu_platform is not,
> > So this is a critique of existing spec text then?
> 
> 
> No. I meant it only work if ACCESS_PLATFORM and ACCESS_ORDER were set
> correctly.

In fact all combinations of ACCESS_PLATFORM and ACCESS_ORDER can
exist in the field, with both software and hardware.

> 
> > Given plans for an imminent 1.1 release let's focus on
> > the proposed patch, ok?
> > 
> > 
> > > in this
> > > case:
> > > 
> > > 1) we still can't recognize old guests, so it won't fail gracefully.
> > > 
> > > 2) new guests can't work
> > > 
> > > 
> > > It depends on how much we can gain from failing gracefully. I believe we
> > > don't want to force a software fallback.
> > iommu_platform fundamentally doesn't support a software fallback,
> > it's a security feature. So I agree but what gave you the idea that we
> > force a software fallback?
> 
> 
> I don't suggest a software fallback. I meant without a fallback:
> 
> - when IOMMU_PLATFORM is not advertised, there are still cases that
> IOMMU_PLATFORM can't handle (the above two cases).
> 
> - when IOMMU_PLATFORM is advertised (or could not be turned of for e.g
> hardware implementation), the only difference is a graceful failure or
> silent failure for old guest, but how much important is the case?
> 
> Consider the above cases, is it better just implement the semantics of
> IOMMU_PLATFORM silently and deprecate it in the future. Then we may only
> meet one silent failure when driver doesn't correctly detect platform
> requirements.
> 
> 
> > So ORDER_PLATFORM it seems prodent to
> > suggest a software fallback given not one driver in the field sets it.
> > 
> > Would your comment be addressed if I replace RECOMMENDED with MAY?
> > 
> > > So I wonder how bad just let
> > > existing drivers fail and try to patch and backport the fixes without
> > > introducing new feature bits. With this we can offload those tasks to
> > > platform or configuration specific codes instead of our own.
> > > 
> > > 
> > > Another question is whether or not we need two separate feature bits.
> > Well IOMMU_PLATFORM is already set in the field.
> > We can't make it also imply strong barriers without
> > breaking a bunch of existing setups.
> 
> 
> Or just imply the IOMMU_PLATFORM if ACCESS_ORDER is set.
> 
> 
> > 
> > > E.g
> > > when should we set one bit but not for another?
> > I am sorry I don't understand the question.  I did try to
> > document what does each bit say. Is it completely unclear then?
> > Could you give some hints on what is unclear? Which kind
> > of functionality is covered by both such that it's
> > an open question which one to set?
> 
> 
> I meant, e.g spec allow the case of advertising ACCES_ORDER but not
> advertising ACCESS_PLATFORM. This sounds somehow ambiguous since both
> feature are memory related but part were accessed in a way what platform
> requires but the rest doesn't.
> 
> 
> > 
> > > > So a device needs to know what it's dealing with, to
> > > > either fail gracefully or recover by a software fallback.
> > > > 
> > > > 
> > > > > Thanks
> > > > All good questions, though I'm not sure whether they mean we
> > > > need to add some text to the spec. Suggestions about
> > > > what should be documented?
> > > > 
> > > Maybe it's better to clarify the suggestion for hardware or mixed
> > > implementations.
> > > 
> > > Thanks
> > Right but I don't know what to suggest. E.g. intel recently
> > published data about a hardware vDPA which does not set either
> > bits.
> 
> 
> So this violates spec. Device actually will go through IOMMU but driver
> won't use DMA API at all. But since the control path is under the control of
> software, it's no hard to advertise it I believe. If not, it's another call
> for clarifying ACCESS_PLATFORM/ACCESS_ORDER is mandatory for hardware.
> 
> 
> > They are working on support for a vIOMMU at which point
> > they will set IOMMU_PLATFORM. Once they start worrying about
> > portability they might add ORDER_PLATFORM. But given it's
> > VDPA they don't necessarily need to fail without even
> > on weakly ordered platforms.
> 
> 
> If feature bit advertising could be controlled by software it probably won't
> need any suggestion. We probably need suggest only for full hardware
> implementation without any software emulation.
> 
> Thanks.

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that 
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php