[virtio-dev] [RFC] VIRTIO_F_IO_BARRIER: use I/O barriers in driver

[virtio-dev] [RFC] VIRTIO_F_IO_BARRIER: use I/O barriers in driver

[Tiwei Bie]

There will be hardware virtio devices in the future, which
require drivers to use the barriers suitable for I/O device,
compared with software virtio devices which just require
drivers to use the barriers suitable for CPU core.

To fix the ordering issue for hardware virtio devices, add
a new feature: VIRTIO_F_IO_BARRIER. When negotiated, driver
will use the barriers suitable for I/O device.

Signed-off-by: Tiwei Bie <tiwei.bie@intel.com>
---
 content.tex | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/content.tex b/content.tex
index ae5fa4c..f24e320 100644
--- a/content.tex
+++ b/content.tex
@@ -5445,6 +5445,8 @@ Descriptors} and \ref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Supp
   that drivers pass extra data (besides identifying the Virtqueue)
   in their device notifications.
   See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}.
+  \item[VIRTIO_F_IO_BARRIER(37)] This feature indicates that the device
+  needs the driver to use the barriers suitable for hardware device.
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
@@ -5460,6 +5462,10 @@ addresses to the device.
 
 A driver SHOULD accept VIRTIO_F_RING_PACKED if it is offered.
 
+A driver SHOULD accept VIRTIO_F_IO_BARRIER if it is offered.
+If VIRTIO_F_IO_BARRIER has been negotiated, a driver MUST use
+the barriers suitable for hardware device.
+
 \devicenormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
 
 A device MUST offer VIRTIO_F_VERSION_1.  A device MAY fail to operate further
@@ -5473,6 +5479,9 @@ accepted.
 If VIRTIO_F_IN_ORDER has been negotiated, a device MUST use
 buffers in the same order in which they have been available.
 
+A device MAY fail to operate further if VIRTIO_F_IO_BARRIER
+is not accepted.
+
 \section{Legacy Interface: Reserved Feature Bits}\label{sec:Reserved Feature Bits / Legacy Interface: Reserved Feature Bits}
 
 Transitional devices MAY offer the following:
-- 
2.11.0


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [RFC] VIRTIO_F_IO_BARRIER: use I/O barriers in driver

[Paolo Bonzini]

On 25/04/2018 10:23, Tiwei Bie wrote:
> There will be hardware virtio devices in the future, which
> require drivers to use the barriers suitable for I/O device,
> compared with software virtio devices which just require
> drivers to use the barriers suitable for CPU core.
> 
> To fix the ordering issue for hardware virtio devices, add
> a new feature: VIRTIO_F_IO_BARRIER. When negotiated, driver
> will use the barriers suitable for I/O device.

Unfortunately this doesn't work as mentioned earlier.  Virtio live
migration assumes that features are safe if you have them on the
destination but not on the source; this feature however works the
opposite way.

Paolo


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] Re: [RFC] VIRTIO_F_IO_BARRIER: use I/O barriers in driver

[Tiwei Bie]

On Wed, Apr 25, 2018 at 03:23:39PM +0200, Paolo Bonzini wrote:
> On 25/04/2018 10:23, Tiwei Bie wrote:
> > There will be hardware virtio devices in the future, which
> > require drivers to use the barriers suitable for I/O device,
> > compared with software virtio devices which just require
> > drivers to use the barriers suitable for CPU core.
> > 
> > To fix the ordering issue for hardware virtio devices, add
> > a new feature: VIRTIO_F_IO_BARRIER. When negotiated, driver
> > will use the barriers suitable for I/O device.
> 
> Unfortunately this doesn't work as mentioned earlier.  Virtio live
> migration assumes that features are safe if you have them on the
> destination but not on the source; this feature however works the
> opposite way.

If my understanding is correct, I think Michael suggested to add
code to disallow migration between setups with and without the
feature:

http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg03094.html
"""
Right. We'll need a flag for this feature for starters. It doesn't exist
:) Paolo also points out that we should then add code to disallow
migration between setups with and without the feature.
"""

If we do it in the opposite way (i.e. use SMP barriers when feature
is negotiated), the problem is that, old kernels will go into error
silently when using hardware virtio devices.

Best regards,
Tiwei Bie


> 
> Paolo
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [RFC] VIRTIO_F_IO_BARRIER: use I/O barriers in driver

[Michael S. Tsirkin]

On Wed, Apr 25, 2018 at 03:23:39PM +0200, Paolo Bonzini wrote:
> On 25/04/2018 10:23, Tiwei Bie wrote:
> > There will be hardware virtio devices in the future, which
> > require drivers to use the barriers suitable for I/O device,
> > compared with software virtio devices which just require
> > drivers to use the barriers suitable for CPU core.
> > 
> > To fix the ordering issue for hardware virtio devices, add
> > a new feature: VIRTIO_F_IO_BARRIER. When negotiated, driver
> > will use the barriers suitable for I/O device.
> 
> Unfortunately this doesn't work as mentioned earlier.  Virtio live
> migration assumes that features are safe if you have them on the
> destination but not on the source; this feature however works the
> opposite way.
> 
> Paolo

That's a qemu code property, and it's because it does not
migrate host properties right now. When implementing this feature we
can add migration of host features.

But that's only if we want to catch mis-use. We can also
punt it up the stack and say feature flags must match
on source and destination, and if they do not the problem
is between the keyboard and the chair.


-- 
MST

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [RFC] VIRTIO_F_IO_BARRIER: use I/O barriers in driver

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2135 bytes --]

On Wed, Apr 25, 2018 at 04:23:12PM +0800, Tiwei Bie wrote:
> There will be hardware virtio devices in the future, which
> require drivers to use the barriers suitable for I/O device,
> compared with software virtio devices which just require
> drivers to use the barriers suitable for CPU core.
> 
> To fix the ordering issue for hardware virtio devices, add
> a new feature: VIRTIO_F_IO_BARRIER. When negotiated, driver
> will use the barriers suitable for I/O device.
> 
> Signed-off-by: Tiwei Bie <tiwei.bie@intel.com>
> ---
>  content.tex | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/content.tex b/content.tex
> index ae5fa4c..f24e320 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -5445,6 +5445,8 @@ Descriptors} and \ref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Supp
>    that drivers pass extra data (besides identifying the Virtqueue)
>    in their device notifications.
>    See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}.
> +  \item[VIRTIO_F_IO_BARRIER(37)] This feature indicates that the device
> +  needs the driver to use the barriers suitable for hardware device.

s/hardware device/hardware devices/ (plural)

>  \end{description}
>  
>  \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
> @@ -5460,6 +5462,10 @@ addresses to the device.
>  
>  A driver SHOULD accept VIRTIO_F_RING_PACKED if it is offered.
>  
> +A driver SHOULD accept VIRTIO_F_IO_BARRIER if it is offered.
> +If VIRTIO_F_IO_BARRIER has been negotiated, a driver MUST use
> +the barriers suitable for hardware device.

s/hardware device/hardware devices/ (plural)

Please add a little context so the purpose is clear to readers (I
wouldn't be 100% sure what the spec means without more detail):

  Some transports require barriers to ensure hardware devices have a
  consistent view of memory.  When devices are implemented in software a
  weaker form of barrier may be sufficient and yield better performance.
  This feature bit indicates whether the full hardware barrier is
  necessary.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

[virtio-dev] Re: [RFC] VIRTIO_F_IO_BARRIER: use I/O barriers in driver

[Paolo Bonzini]

On 25/04/2018 16:29, Michael S. Tsirkin wrote:
>> Unfortunately this doesn't work as mentioned earlier.  Virtio live
>> migration assumes that features are safe if you have them on the
>> destination but not on the source; this feature however works the
>> opposite way.
>
> That's a qemu code property, and it's because it does not
> migrate host properties right now. When implementing this feature we
> can add migration of host features.

You don't even need to migrate host features; it's just that guest
feature negotiation is repeated on the destination, and live migration
reports an error if the negotiation fails.  The guest feature
negotiation is complicated a little bit by this feature, but I guess it
can work...

Paolo

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [RFC] VIRTIO_F_IO_BARRIER: use I/O barriers in driver

[Tiwei Bie]

On Fri, Apr 27, 2018 at 02:02:46PM +0100, Stefan Hajnoczi wrote:
> On Wed, Apr 25, 2018 at 04:23:12PM +0800, Tiwei Bie wrote:
> > There will be hardware virtio devices in the future, which
> > require drivers to use the barriers suitable for I/O device,
> > compared with software virtio devices which just require
> > drivers to use the barriers suitable for CPU core.
> > 
> > To fix the ordering issue for hardware virtio devices, add
> > a new feature: VIRTIO_F_IO_BARRIER. When negotiated, driver
> > will use the barriers suitable for I/O device.
> > 
> > Signed-off-by: Tiwei Bie <tiwei.bie@intel.com>
> > ---
> >  content.tex | 9 +++++++++
> >  1 file changed, 9 insertions(+)
> > 
> > diff --git a/content.tex b/content.tex
> > index ae5fa4c..f24e320 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -5445,6 +5445,8 @@ Descriptors} and \ref{sec:Packed Virtqueues / Indirect Flag: Scatter-Gather Supp
> >    that drivers pass extra data (besides identifying the Virtqueue)
> >    in their device notifications.
> >    See \ref{sec:Virtqueues / Driver notifications}~\nameref{sec:Virtqueues / Driver notifications}.
> > +  \item[VIRTIO_F_IO_BARRIER(37)] This feature indicates that the device
> > +  needs the driver to use the barriers suitable for hardware device.
> 
> s/hardware device/hardware devices/ (plural)
> 
> >  \end{description}
> >  
> >  \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
> > @@ -5460,6 +5462,10 @@ addresses to the device.
> >  
> >  A driver SHOULD accept VIRTIO_F_RING_PACKED if it is offered.
> >  
> > +A driver SHOULD accept VIRTIO_F_IO_BARRIER if it is offered.
> > +If VIRTIO_F_IO_BARRIER has been negotiated, a driver MUST use
> > +the barriers suitable for hardware device.
> 
> s/hardware device/hardware devices/ (plural)
> 
> Please add a little context so the purpose is clear to readers (I
> wouldn't be 100% sure what the spec means without more detail):
> 
>   Some transports require barriers to ensure hardware devices have a
>   consistent view of memory.  When devices are implemented in software a
>   weaker form of barrier may be sufficient and yield better performance.
>   This feature bit indicates whether the full hardware barrier is
>   necessary.

Thank you very much for the review and suggestion! :)

Best regards,
Tiwei Bie

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org