From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: virtio-dev-return-5415-cohuck=redhat.com@lists.oasis-open.org Sender: List-Post: List-Help: List-Unsubscribe: List-Subscribe: Received: from lists.oasis-open.org (oasis-open.org [10.110.1.242]) by lists.oasis-open.org (Postfix) with ESMTP id D201C985D40 for ; Tue, 12 Feb 2019 19:01:35 +0000 (UTC) MIME-Version: 1.0 References: <20190204054053.GE29758@stefanha-x1.localdomain> <20190204101316.4e3e6rj32suwdmur@sirius.home.kraxel.org> <20190211092943-mutt-send-email-mst@kernel.org> <20190212112547.GC2715@work-vm> <20190212144741.60083682.cohuck@redhat.com> <20190212090121-mutt-send-email-mst@kernel.org> <20190212125701-mutt-send-email-mst@kernel.org> In-Reply-To: <20190212125701-mutt-send-email-mst@kernel.org> From: Frank Yang Date: Tue, 12 Feb 2019 11:01:21 -0800 Message-ID: Content-Type: multipart/alternative; boundary="000000000000c7f9520581b70df9" Subject: Re: [virtio-dev] Memory sharing device To: "Michael S. Tsirkin" Cc: Cornelia Huck , "Dr. David Alan Gilbert" , Roman Kiryanov , Gerd Hoffmann , Stefan Hajnoczi , virtio-dev@lists.oasis-open.org, Greg Hartman List-ID: --000000000000c7f9520581b70df9 Content-Type: text/plain; charset="UTF-8" On Tue, Feb 12, 2019 at 10:22 AM Michael S. Tsirkin wrote: > On Tue, Feb 12, 2019 at 07:56:58AM -0800, Frank Yang wrote: > > Stepping back to standardization and portability concerns, it is also not > > necessarily desirable to use general pipes to do what we want, because > even > > though that device exists and is part of the spec already, that results > in > > _de-facto_ non-portability. > > That's not different from e.g. TCP. > > > If we had some kind of spec to enumerate such > > 'user-defined' devices, at least we can have _de-jure_ non-portability; > an > > enumerated device doesn't work as advertised. > > I am not sure distinguishing between different types of non portability > will be in scope for virtio. Actually having devices that are portable > would be. > > The device itself is portable; the user-defined drivers that run on them will work or not depending on negotiating device IDs. > ... > > Note that virtio-serial/virtio-vsock is not considered because they do not > > standardize the set of devices that operate on top of them, but in > practice, > > are often used for fully general devices. Spec-wise, this is not a great > > situation because we would still have potentially non portable device > > implementations where there is no standard mechanism to determine > whether or > > not things are portable. > > Well it's easy to add an enumeration on top of sockets, and several well > known solutions exist. There's an advantage to just reusing these. Sure, but there are many unique features/desirable properties of having the virtio meta device because (as explained in the spec) there are limitations to network/socket based communication. > > virtio-user provides a device enumeration mechanism > > to better control this. > > We'll have to see what it all looks like. For virtio pci transport it's > important that you can reason about the device at a basic level based on > it's PCI ID, and that is quite fundamental. > > The spec contains more details; basically the device itself is always portable, and there is a configuration protocol to negotiate whether a particular use of the device is available. This is similar to PCI, but with more defined ways to operate the device in terms of callbacks in shared libraries on the host. > Maybe what you are looking for is a new virtio transport then? > > Perhaps, something like virtio host memory transport? But at the same time, it needs to interact with shared memory which is best set as a PCI device. Can we mix transport types? In any case, the analog of "PCI ID"'s here (the vendor/device/version numbers) are meaningful, with the contract being that the user of the device needs to match on vendor/device id and negotiate on version number. Wha are the advantages of defining a new virtio transport type? it would be something that has the IDs, and be able to handle resolving offsets to physical addresses to host memory addresses, in addition to dispatching to callbacks on the host. But it would be effectively equivalent to having a new virtio device type with device ID enumeration, right? > > In addition, for performance considerations in applications such as > graphics > > and media, virtio-serial/virtio-vsock have the overhead of sending actual > > traffic through the virtqueue, while an approach based on shared memory > can > > result in having fewer copies and virtqueue messages. virtio-serial is > also > > limited in being specialized for console forwarding and having a cap on > the > > number of clients. virtio-vsock is also not optimal in its choice of > sockets > > API for transport; shared memory cannot be used, arbitrary strings can be > > passed without an designation of the device/driver being run de-facto, > and the > > guest must have additional machinery to handle socket APIs. In > addition, on > > the host, sockets are only dependable on Linux, with less predictable > behavior > > from Windows/macOS regarding Unix sockets. Waiting for socket traffic > on the > > host also requires a poll() loop, which is suboptimal for latency. With > > virtio-user, only the bare set of standard driver calls > > (open/close/ioctl/mmap/read) is needed, and RAM is a more universal > transport > > abstraction. We also explicitly spec out callbacks on host that are > triggered > > by virtqueue messages, which results in lower latency and makes it easy > to > > dispatch to a particular device implementation without polling. > > open/close/mmap/read seem to make sense. ioctl gives one pause. > > ioctl would be to send ping messages, but I'm not fixated on that choice. write() is also a possibility to send ping messages; I preferred ioctl() because it should be clear that it's a control message not a data message. > Given open/close this begins to look a bit like virtio-fs. > Have you looked at that? > > That's an interesting possibility since virtio-fs maps host pointers as well, which fits our use cases. Another alternative is to add the features unique about virtio-user to virtio-fs: device enumeration, memory sharing operations, operation in terms of callbacks on the host. However, it doesn't seem like a good fit due to being specialized to filesystem operations. > -- > MST > --000000000000c7f9520581b70df9 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable



On Tue, Feb 12, 2019 at 10:22 AM Michael S. Tsirkin <mst@redhat.com> wrote:
On Tue, Feb 12, 2019 = at 07:56:58AM -0800, Frank Yang wrote:
> Stepping back to standardization and portability concerns, it is also = not
> necessarily desirable to use general pipes to do what we want, because= even
> though that device exists and is part of the spec already, that result= s in
> _de-facto_ non-portability.

That's not different from e.g. TCP.

> If we had some kind of spec to enumerate such
> 'user-defined' devices, at least we can have _de-jure_ non-por= tability; an
> enumerated device doesn't work as advertised.

I am not sure distinguishing between different types of non portability
will be in scope for virtio. Actually having devices that are portable
would be.

The device itself is portable; the user-defined drive= rs that run on them will work or not depending on
negotiating dev= ice IDs.
...=C2=A0
> Note that virtio-serial/virtio-vsock is not considered because they do= not
> standardize the set of devices that operate on top of them, but in pra= ctice,
> are often used for fully general devices.=C2=A0 Spec-wise, this is not= a great
> situation because we would still have potentially non portable device<= br> > implementations where there is no standard mechanism to determine whet= her or
> not things are portable.

Well it's easy to add an enumeration on top of sockets, and several wel= l
known solutions exist. There's an advantage to just reusing these.=C2= =A0=C2=A0

Sure, but there are many unique f= eatures/desirable properties of having the virtio meta device
bec= ause (as explained in the spec) there are limitations to network/socket bas= ed communication.
=C2=A0
>=C2=A0virtio-user provides a device enumeration mechanism
> to better control this.

We'll have to see what it all looks like. For virtio pci transport it&#= 39;s
important that you can reason about the device at a basic level based on it's PCI ID, and that is quite fundamental.

The spec contains more details; basically the device = itself is always portable, and there is a configuration protocol
= to negotiate whether a particular use of the device is available. This is s= imilar to PCI,
but with more defined ways to operate the device i= n terms of callbacks in shared libraries on the host.
=C2=A0
Maybe what you are looking for is a new virtio transport then?

=C2=A0
Perhaps, something like virtio host = memory transport? But
at the same time, it needs to interact with= shared memory which is best set as a PCI device.
Can we mix tran= sport types? In any case, the analog of "PCI ID"'s here (the = vendor/device/version numbers)
are meaningful, with the contract = being that the user of the device needs to match on vendor/device id and
negotiate on version number.

Wha are the a= dvantages of defining a new virtio transport type?
it would be so= mething that has the IDs, and be able to handle resolving offsets to
<= div>physical addresses to host memory addresses,
in addition to d= ispatching to callbacks on the host.
But it would be effectively = equivalent to having a new virtio device type with device ID enumeration, r= ight?


> In addition, for performance considerations in applications such as gr= aphics
> and media, virtio-serial/virtio-vsock have the overhead of sending act= ual
> traffic through the virtqueue, while an approach based on shared memor= y can
> result in having fewer copies and virtqueue messages.=C2=A0 virtio-ser= ial is also
> limited in being specialized for console forwarding and having a cap o= n the
> number of clients.=C2=A0 virtio-vsock is also not optimal in its choic= e of sockets
> API for transport; shared memory cannot be used, arbitrary strings can= be
> passed without an designation of the device/driver being run de-facto,= and the
> guest must have additional machinery to handle socket APIs.=C2=A0 In a= ddition, on
> the host, sockets are only dependable on Linux, with less predictable = behavior
> from Windows/macOS regarding Unix sockets.=C2=A0 Waiting for socket tr= affic on the
> host also requires a poll() loop, which is suboptimal for latency.=C2= =A0 With
> virtio-user, only the bare set of standard driver calls
> (open/close/ioctl/mmap/read) is needed, and RAM is a more universal tr= ansport
> abstraction.=C2=A0 We also explicitly spec out callbacks on host that = are triggered
> by virtqueue messages, which results in lower latency and makes it eas= y to
> dispatch to a particular device implementation without polling.

open/close/mmap/read seem to make sense. ioctl gives one pause.

ioctl would be to send ping messages, but I'm not= fixated on that choice. write() is also a possibility to send ping message= s; I preferred ioctl() because it should be clear that it's a control m= essage not a data message.
=C2=A0
Given open/close this begins to look a bit like virtio-fs.
Have you looked at that?

=C2=A0
That's an interesting possibilit= y since virtio-fs maps host pointers as well, which fits our use cases.
Another alternative is to add the features unique about virtio-user = to virtio-fs:
device enumeration, memory sharing operations, oper= ation in terms of callbacks on the host.
However, it doesn't = seem like a good fit due to being specialized to filesystem operations.


--
MST
--000000000000c7f9520581b70df9--