From: Stefan Hajnoczi <stefanha@redhat.com>
To: Max Gurtovoy <mgurtovoy@nvidia.com>
Cc: Jason Wang <jasowang@redhat.com>,
"Michael S. Tsirkin" <mst@redhat.com>,
Zhu Lingshan <lingshan.zhu@intel.com>,
virtio-comment@lists.oasis-open.org,
virtio-dev@lists.oasis-open.org, cohuck@redhat.com,
sgarzare@redhat.com, nrupal.jani@intel.com,
Piotr.Uminski@intel.com, hang.yuan@intel.com,
virtio@lists.oasis-open.org, pasic@linux.ibm.com,
Shahaf Shuler <shahafs@nvidia.com>,
Parav Pandit <parav@nvidia.com>
Subject: [virtio-dev] Re: [virtio-comment] Re: [virtio] Re: [PATCH v10 04/10] admin: introduce virtio admin virtqueues
Date: Wed, 8 Mar 2023 12:15:23 -0500 [thread overview]
Message-ID: <20230308171523.GA320810@fedora> (raw)
In-Reply-To: <18ddbf69-19a6-3c6b-9e42-aaae66e20bcf@nvidia.com>
[-- Attachment #1: Type: text/plain, Size: 9139 bytes --]
On Wed, Mar 08, 2023 at 06:19:41PM +0200, Max Gurtovoy wrote:
>
>
> On 08/03/2023 16:13, Stefan Hajnoczi wrote:
> > On Wed, Mar 08, 2023 at 01:17:33PM +0200, Max Gurtovoy wrote:
> > >
> > >
> > > On 06/03/2023 18:25, Stefan Hajnoczi wrote:
> > > > On Mon, Mar 06, 2023 at 05:28:03PM +0200, Max Gurtovoy wrote:
> > > > >
> > > > >
> > > > > On 06/03/2023 13:20, Stefan Hajnoczi wrote:
> > > > > > On Mon, Mar 06, 2023 at 04:00:50PM +0800, Jason Wang wrote:
> > > > > > >
> > > > > > > 在 2023/3/6 08:03, Stefan Hajnoczi 写道:
> > > > > > > > On Sun, Mar 05, 2023 at 04:38:59AM -0500, Michael S. Tsirkin wrote:
> > > > > > > > > On Fri, Mar 03, 2023 at 03:21:33PM -0500, Stefan Hajnoczi wrote:
> > > > > > > > > > What happens if a command takes 1 second to complete, is the device
> > > > > > > > > > allowed to process the next command from the virtqueue during this time,
> > > > > > > > > > possibly completing it before the first command?
> > > > > > > > > >
> > > > > > > > > > This requires additional clarification in the spec because "they are
> > > > > > > > > > processed by the device in the order in which they are queued" does not
> > > > > > > > > > explain whether commands block the virtqueue (in order completion) or
> > > > > > > > > > not (out of order completion).
> > > > > > > > > Oh I begin to see. Hmm how does e.g. virtio scsi handle this?
> > > > > > > > virtio-scsi, virtio-blk, and NVMe requests may complete out of order.
> > > > > > > > Several may be processed by the device at the same time.
> > > > > > > >
> > > > > > > > They rely on multi-queue for abort operations:
> > > > > > > >
> > > > > > > > In virtio-scsi the abort requests (VIRTIO_SCSI_T_TMF_ABORT_TASK) are
> > > > > > > > sent on the control virtqueue. The the request identifier namespace is
> > > > > > > > shared across all virtqueues so it's possible to abort a request that
> > > > > > > > was submitted to any command virtqueue.
> > > > > > > >
> > > > > > > > NVMe also follows the same design where abort commands are sent on the
> > > > > > > > Admin Submission Queue instead of an I/O Submission Queue. It's possible
> > > > > > > > to identify NVMe requests by <Submission Queue ID, Command Identifier>.
> > > > > > > >
> > > > > > > > virtio-blk doesn't support aborting requests.
> > > > > > > >
> > > > > > > > I think the logic behind this design is that if a queue gets stuck
> > > > > > > > processing long-running requests, then the device should not be forced
> > > > > > > > to perform lookahead in the queue to find abort commands. A separate
> > > > > > > > control/admin queue is used for the abort requests.
> > > > > > >
> > > > > > >
> > > > > > > Or device need mandate some kind of QOS here, e.g a request must be complete
> > > > > > > in some time. Otherwise we don't have sufficient reliability for using it as
> > > > > > > management task?
> > > > > >
> > > > > > Yes, if all commands can be executed in bounded time then a guarantee is
> > > > > > possible.
> > > > > >
> > > > > > Here is an example where that's hard: imagine a virtio-blk device backed
> > > > > > by network storage. When an admin queue command is used to delete a
> > > > > > group member, any of the group member's in-flight I/O requests need to
> > > > > > be aborted. If the network hangs while the group member is being
> > > > > > deleted, then the device can't complete an orderly shutdown of I/O
> > > > > > requests in a reasonable time.
> > > > > >
> > > > > > That example shows a basic group admin command that I think Michael is
> > > > > > about to propose. We can't avoid this problem by not making it a group
> > > > > > admin command - it needs to be a group admin command. So I think it's
> > > > > > likely that there will be admin commands that take an unbounded amount
> > > > > > of time to complete. One way to achieve what you mentioned is timeouts.
> > > > >
> > > > > I think that you're getting into device specific implementation details and
> > > > > I'm not sure it's necessary.
> > > > >
> > > > > I don't think we need to abort admin commands. Admin commands can be
> > > > > flushed/aborted during the device reset phase.
> > > > > Only IO commands should have the possibility to being aborted as you
> > > > > mentioned in NVMe and SCSI (and potentially in virtio-blk).
> > > >
> > > > It's a general design issue that should be clarified now rather than
> > > > being left unspecified.
> > > >
> > > > I'm not saying that it must be possible to abort admin commands. There
> > > > are other options, like requiring the device itself to fail a command
> > > > after a timeout.
> > >
> > > do you have an example of timeout today for control vq ?
> >
> > Do you mean the virtio-net control virtqueue? I don't think it has any
> > commands with an unbounded execution time.
>
> Correct. So why introducing it now ?
The examples I've given are the create and delete group member
operations. I think those operations will take unbounded time in some
device implementations.
>
> >
> > > >
> > > > Or we could say that admin commands must complete within bounded time,
> > > > but I'm not sure that is implementable for some device types like
> > > > virtio-blk, virtio-scsi, and virtiofs.
> > >
> > > No we can't.
> > > Some commands, for example FW upgrade can take 10 minutes and it's perfectly
> > > fine. Other commands like setting feature bit will take 1 millisec.
> > > Each device implements commands in a different internal logic so we can't
> > > expect to complete after X time.
> >
> > When I say bounded time, I mean that it finishes in a finite amount of
> > time. I'm not saying there is a specific time X that all device
> > implementations must satisfy. Unbounded means it might never finish.
>
> There might be a chance that any command for any virtio device type will
> never finish. Nothing new here in the adminq.
>
> what one can do is to set a timeout for himself and if this timeout expire -
> check the device status. If it needs_reset - do a reset. if status is ok,
> then wait some more time.
> After X retries, unmap buffers or reset the adminq.
Michael: What effect does resetting the group owner device have on group
member devices?
I'm concerned that this approach disrupts all group member devices. For
example, you try to add a new device but the command hangs. In order to
recover you now have to reset the group owner device and this breaks all
the group member devices.
> >
> > > Device can go to so FATAL state in case a command is stuck and causing
> > > internal errors in it.
> > >
> > > >
> > > > > For your example, stopping a member is possible even it there are some
> > > > > errors in the network. You can for example destroy all the connections to
> > > > > the remote target and complete all the BIOS with some error.
> > > >
> > > > Forgetting about in-flight requests doesn't necessarily make them go
> > > > away. It creates a race between forgotten requests and reconnection. In
> > > > the worst case a forgotten write request takes effect after
> > > > reconnection, causing data corruption.
> > >
> > > For making it work without data corruption we need a cooperation of the
> > > target side for sure. But this is fine since the target in that case is part
> > > of the "virtio-blk backend".
> > > One solution is that the target can decide it will flush all the requests to
> > > the storage device before accepting new connections.
> >
> > This solution shifts the unbounded time from disconnection to
> > connection. The Group Member Delete command will complete quickly but a
> > subsequent Group Member Create command for the same underlying storage
> > device would need to wait until the requests are done.
> >
> > Therefore I think the admin queue must be designed under the assumption
> > that some commands take a very long time.
>
> For sure an admin command may take long time. FW upgrade can take 10 minutes
> for example.
> But each device is free to implement internal logic as he choose.
>
> Same for live migration, when we stop/quiesce a device we must make sure it
> doesn't master any DMA operations. Thus, in some implementations we need to
> wait for all inflights to end fast. In others, we can invalidate the access
> to host/guest memory and wait for completions until the freeze state.
>
> Bottom line, this is device implementation specific consideration.
What I'm asking is that the spec clarifies the command completion order
semantics (in-order or out-of-order), whether there is a mechanism to
abort commands, etc.
Device implementers can then take advantage of those aspects to
implement devices that don't hang (e.g. health monitoring becomes
unavailable when there is a long running command).
If the spec doesn't cover this, then device implementers will not be
able to work around it when implementing standard commands like
create/delete group member.
Does that make sense?
Stefan
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
next prev parent reply other threads:[~2023-03-08 17:15 UTC|newest]
Thread overview: 169+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-03-02 13:04 [virtio-dev] [PATCH v10 00/10] Introduce device group and device management Michael S. Tsirkin
2023-03-02 13:04 ` [virtio-dev] [PATCH v10 01/10] virtio: document forward compatibility guarantees Michael S. Tsirkin
2023-03-02 18:39 ` [virtio-dev] " Parav Pandit
2023-03-02 23:43 ` [virtio-dev] " Michael S. Tsirkin
[not found] ` <m2leka0yvl.fsf@oracle.com>
2023-03-06 22:00 ` [virtio-dev] Re: [virtio] " Michael S. Tsirkin
2023-03-07 10:04 ` David Edmondson
2023-03-08 14:16 ` Cornelia Huck
2023-03-08 15:04 ` Michael S. Tsirkin
2023-03-02 13:04 ` [virtio-dev] [PATCH v10 02/10] admin: introduce device group and related concepts Michael S. Tsirkin
2023-03-02 18:39 ` [virtio-dev] " Parav Pandit
2023-03-02 23:44 ` [virtio-dev] " Michael S. Tsirkin
2023-03-02 19:40 ` [virtio-dev] Re: [virtio] " Stefan Hajnoczi
2023-03-06 17:00 ` David Edmondson
2023-03-02 13:05 ` [virtio-dev] [PATCH v10 03/10] admin: introduce group administration commands Michael S. Tsirkin
2023-03-02 18:40 ` [virtio-dev] " Parav Pandit
2023-03-02 20:19 ` [virtio-dev] " Stefan Hajnoczi
2023-03-03 0:01 ` Michael S. Tsirkin
2023-03-03 13:17 ` Stefan Hajnoczi
2023-03-03 13:19 ` Michael S. Tsirkin
[not found] ` <4f869944-4ccd-c51e-0f30-dc3ba15ffd52@nvidia.com>
2023-03-07 18:33 ` [virtio-dev] " Parav Pandit
2023-03-08 0:34 ` [virtio-dev] " Michael S. Tsirkin
2023-03-08 1:01 ` [virtio-dev] " Parav Pandit
2023-03-08 1:06 ` [virtio-dev] " Michael S. Tsirkin
2023-03-08 1:14 ` [virtio-dev] " Parav Pandit
[not found] ` <6668fd7a-3eb3-0447-9cbf-72d308b1336a@nvidia.com>
2023-03-08 12:07 ` [virtio-dev] " Michael S. Tsirkin
[not found] ` <992e9a1e-3799-842c-79f3-e66f5c823356@nvidia.com>
2023-03-08 14:43 ` Cornelia Huck
2023-03-08 14:54 ` Michael S. Tsirkin
2023-03-08 15:21 ` Michael S. Tsirkin
[not found] ` <2ef47f53-5978-4b6a-593d-4e94eba0b9ac@nvidia.com>
2023-03-09 6:38 ` [virtio-dev] Re: [virtio-comment] " Michael S. Tsirkin
2023-03-02 23:47 ` [virtio-dev] " Michael S. Tsirkin
2023-03-07 18:26 ` [virtio-dev] " Parav Pandit
2023-03-02 20:10 ` [virtio-dev] " Stefan Hajnoczi
2023-03-02 23:57 ` Michael S. Tsirkin
2023-03-03 13:13 ` Stefan Hajnoczi
2023-03-03 13:18 ` Michael S. Tsirkin
2023-03-03 20:23 ` Stefan Hajnoczi
2023-03-07 11:31 ` Jiri Pirko
2023-03-08 12:03 ` Michael S. Tsirkin
2023-03-07 10:31 ` [virtio-dev] Re: [virtio] " David Edmondson
2023-03-02 13:05 ` [virtio-dev] [PATCH v10 04/10] admin: introduce virtio admin virtqueues Michael S. Tsirkin
2023-03-02 18:40 ` [virtio-dev] " Parav Pandit
2023-03-02 23:48 ` [virtio-dev] " Michael S. Tsirkin
2023-03-02 20:40 ` Stefan Hajnoczi
2023-03-03 0:05 ` Michael S. Tsirkin
2023-03-03 13:28 ` Stefan Hajnoczi
2023-03-03 13:37 ` Michael S. Tsirkin
2023-03-03 20:21 ` [virtio-dev] Re: [virtio] " Stefan Hajnoczi
2023-03-05 9:38 ` Michael S. Tsirkin
2023-03-06 0:03 ` Stefan Hajnoczi
2023-03-06 0:18 ` Michael S. Tsirkin
[not found] ` <20230306110340.GA35392@fedora>
2023-03-06 18:37 ` Michael S. Tsirkin
2023-03-06 20:17 ` Stefan Hajnoczi
2023-03-06 21:43 ` Michael S. Tsirkin
2023-03-31 11:07 ` Michael S. Tsirkin
2023-03-07 8:03 ` [virtio-dev] Re: [virtio-comment] " Jiri Pirko
2023-03-07 14:39 ` Stefan Hajnoczi
2023-03-07 15:07 ` Jiri Pirko
2023-03-07 19:03 ` Stefan Hajnoczi
2023-03-07 19:09 ` [virtio-dev] " Parav Pandit
2023-03-08 5:17 ` [virtio-dev] Re: [virtio] " Jason Wang
2023-03-08 11:58 ` Stefan Hajnoczi
2023-03-08 11:59 ` Michael S. Tsirkin
2023-03-08 10:17 ` [virtio-dev] " Jiri Pirko
2023-03-08 12:44 ` Stefan Hajnoczi
2023-03-08 12:57 ` [virtio-dev] Re: [virtio] " Jiri Pirko
2023-03-08 17:17 ` Stefan Hajnoczi
2023-03-07 16:13 ` [virtio-dev] " Michael S. Tsirkin
2023-03-08 10:08 ` Jiri Pirko
2023-03-08 11:44 ` Michael S. Tsirkin
[not found] ` <7f63fa0a-7deb-5875-6c6b-bfc651681653@redhat.com>
[not found] ` <20230306112030.GB35392@fedora>
[not found] ` <853c78d0-f752-05e9-d79d-811e82801627@nvidia.com>
2023-03-06 16:25 ` [virtio-dev] " Stefan Hajnoczi
2023-03-07 19:04 ` [virtio-dev] " Parav Pandit
[not found] ` <e74483a4-38fa-99b5-86b8-785f0b98d029@nvidia.com>
2023-03-08 14:13 ` [virtio-dev] Re: [virtio-comment] " Stefan Hajnoczi
2023-03-08 16:21 ` [virtio-dev] " Parav Pandit
[not found] ` <18ddbf69-19a6-3c6b-9e42-aaae66e20bcf@nvidia.com>
2023-03-08 17:15 ` Stefan Hajnoczi [this message]
2023-03-08 17:21 ` [virtio-dev] " Michael S. Tsirkin
2023-03-09 12:35 ` Stefan Hajnoczi
2023-03-09 13:55 ` Michael S. Tsirkin
2023-03-09 15:56 ` Stefan Hajnoczi
[not found] ` <e8d41902-b794-66e9-8f15-e8617435047c@redhat.com>
2023-03-06 16:22 ` [virtio-dev] " Jiri Pirko
[not found] ` <027fff1b-8ed7-abc0-2331-b188b8822bf4@nvidia.com>
2023-03-06 22:49 ` [virtio-dev] Re: [virtio-comment] " Michael S. Tsirkin
[not found] ` <e12ab469-4586-f0f5-b084-8f2d7c031e11@nvidia.com>
2023-03-08 21:09 ` Michael S. Tsirkin
2023-03-08 21:17 ` [virtio-dev] " Parav Pandit
2023-03-09 7:28 ` [virtio-dev] " Jiri Pirko
2023-03-07 7:56 ` Jiri Pirko
[not found] ` <ZAXfegxCfvfLwiJT@nanopsycho>
2023-03-06 18:40 ` [virtio-dev] Re: [virtio] " Michael S. Tsirkin
2023-03-07 7:36 ` Jiri Pirko
2023-03-07 16:30 ` Michael S. Tsirkin
2023-03-08 10:05 ` [virtio-dev] Re: [virtio-comment] " Jiri Pirko
2023-03-08 11:50 ` Michael S. Tsirkin
2023-03-08 12:08 ` Jiri Pirko
[not found] ` <18d51bc0-d759-1a05-cb7c-3d46c4ed2f1a@nvidia.com>
2023-03-08 18:01 ` David Edmondson
2023-03-08 18:19 ` Jiri Pirko
2023-03-08 21:25 ` [virtio-dev] " Parav Pandit
2023-03-09 7:30 ` [virtio-dev] " Jiri Pirko
2023-03-09 13:04 ` [virtio-dev] " Parav Pandit
2023-03-09 13:57 ` [virtio-dev] " Michael S. Tsirkin
2023-03-09 14:02 ` [virtio-dev] " David Edmondson
2023-03-09 14:11 ` Parav Pandit
2023-03-09 14:22 ` [virtio-dev] " Michael S. Tsirkin
2023-03-09 14:30 ` [virtio-dev] " Parav Pandit
2023-03-09 14:43 ` [virtio-dev] " Michael S. Tsirkin
2023-03-09 16:53 ` [virtio-dev] " Parav Pandit
2023-03-09 17:14 ` [virtio-dev] " Michael S. Tsirkin
2023-03-09 17:16 ` [virtio-dev] " Parav Pandit
2023-03-08 21:45 ` Parav Pandit
2023-03-09 7:37 ` [virtio-dev] " Jiri Pirko
2023-03-09 12:36 ` [virtio-dev] " Parav Pandit
2023-03-09 16:32 ` [virtio-dev] " Jiri Pirko
[not found] ` <17471553-8981-eef9-656b-098e1acdda14@nvidia.com>
2023-03-06 18:44 ` [virtio-dev] " Michael S. Tsirkin
2023-03-07 10:41 ` David Edmondson
2023-03-02 13:05 ` [virtio-dev] [PATCH v10 05/10] pci: add admin vq registers to virtio over pci Michael S. Tsirkin
2023-03-02 20:51 ` [virtio-dev] " Stefan Hajnoczi
2023-03-02 13:05 ` [virtio-dev] [PATCH v10 06/10] mmio: document ADMIN_VQ as reserved Michael S. Tsirkin
2023-03-02 18:40 ` [virtio-dev] " Parav Pandit
2023-03-02 23:51 ` [virtio-dev] " Michael S. Tsirkin
2023-03-02 23:51 ` Michael S. Tsirkin
2023-03-03 8:34 ` Michael S. Tsirkin
[not found] ` <ZAXB44F3MS9CxIiK@nanopsycho>
2023-03-06 18:46 ` [virtio-dev] Re: [virtio] " Michael S. Tsirkin
2023-03-07 7:40 ` [virtio-dev] Re: [virtio-comment] " Jiri Pirko
2023-03-07 18:52 ` [virtio-dev] " Parav Pandit
2023-03-08 16:24 ` [virtio-dev] Re: [virtio-comment] " Cornelia Huck
2023-03-08 16:37 ` [virtio-dev] " Parav Pandit
2023-03-08 16:30 ` [virtio-dev] " Michael S. Tsirkin
2023-03-08 18:21 ` [virtio-dev] Re: [virtio] " Jiri Pirko
2023-03-02 20:52 ` [virtio-dev] " Stefan Hajnoczi
2023-03-02 13:05 ` [virtio-dev] [PATCH v10 07/10] ccw: " Michael S. Tsirkin
2023-03-02 20:53 ` [virtio-dev] " Stefan Hajnoczi
2023-03-02 13:05 ` [virtio-dev] [PATCH v10 08/10] admin: command list discovery Michael S. Tsirkin
2023-03-02 21:09 ` [virtio-dev] " Stefan Hajnoczi
2023-03-31 11:39 ` Michael S. Tsirkin
2023-03-07 10:54 ` [virtio-dev] " David Edmondson
[not found] ` <ZAXbBgN2jw8RhE/3@nanopsycho>
2023-03-08 11:54 ` Michael S. Tsirkin
2023-03-08 12:41 ` Jiri Pirko
2023-03-31 11:43 ` [virtio-dev] Re: [virtio-comment] " Michael S. Tsirkin
2023-03-08 12:38 ` Jiri Pirko
2023-03-10 8:14 ` [virtio-dev] " Zhu, Lingshan
2023-03-02 13:05 ` [virtio-dev] [PATCH v10 09/10] admin: conformance clauses Michael S. Tsirkin
2023-03-07 11:04 ` [virtio-dev] Re: [virtio-comment] " David Edmondson
2023-03-08 11:58 ` Michael S. Tsirkin
2023-03-08 12:59 ` David Edmondson
2023-03-08 13:05 ` [virtio-dev] Re: [virtio] " Jiri Pirko
2023-03-08 13:22 ` Michael S. Tsirkin
2023-03-08 13:44 ` David Edmondson
2023-03-08 14:02 ` Jiri Pirko
2023-03-08 14:12 ` Michael S. Tsirkin
2023-03-10 9:10 ` [virtio-dev] " Zhu, Lingshan
2023-03-10 9:13 ` Michael S. Tsirkin
2023-03-10 18:00 ` Zhu Lingshan
2023-03-10 9:34 ` Michael S. Tsirkin
2023-03-02 13:05 ` [virtio-dev] [PATCH v10 10/10] ccw: document more reserved features Michael S. Tsirkin
2023-03-02 21:12 ` [virtio-dev] " Stefan Hajnoczi
2023-03-02 13:38 ` [virtio-dev] RE: [PATCH v10 00/10] Introduce device group and device management Parav Pandit
2023-03-02 23:27 ` [virtio-dev] " Michael S. Tsirkin
2023-03-02 18:39 ` [virtio-dev] " Parav Pandit
2023-03-06 16:40 ` [virtio-dev] Re: [virtio-comment] " Jiri Pirko
2023-03-06 22:48 ` Michael S. Tsirkin
2023-03-07 7:17 ` Jiri Pirko
2023-03-07 17:15 ` Michael S. Tsirkin
[not found] ` <ZAXcqqdwfoLokT2l@nanopsycho>
2023-03-06 22:54 ` Michael S. Tsirkin
2023-03-07 7:21 ` Jiri Pirko
2023-03-07 17:20 ` Michael S. Tsirkin
2023-03-07 19:31 ` [virtio-dev] " Parav Pandit
2023-03-08 5:11 ` [virtio-dev] " Jason Wang
2023-03-08 12:02 ` [virtio-dev] " Parav Pandit
2023-03-10 8:32 ` [virtio-dev] " Zhu, Lingshan
2023-03-08 9:49 ` [virtio-dev] Re: [virtio] " Jiri Pirko
2023-03-08 16:30 ` [virtio-dev] " Cornelia Huck
2023-03-08 17:22 ` Michael S. Tsirkin
2023-03-08 18:15 ` Jiri Pirko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230308171523.GA320810@fedora \
--to=stefanha@redhat.com \
--cc=Piotr.Uminski@intel.com \
--cc=cohuck@redhat.com \
--cc=hang.yuan@intel.com \
--cc=jasowang@redhat.com \
--cc=lingshan.zhu@intel.com \
--cc=mgurtovoy@nvidia.com \
--cc=mst@redhat.com \
--cc=nrupal.jani@intel.com \
--cc=parav@nvidia.com \
--cc=pasic@linux.ibm.com \
--cc=sgarzare@redhat.com \
--cc=shahafs@nvidia.com \
--cc=virtio-comment@lists.oasis-open.org \
--cc=virtio-dev@lists.oasis-open.org \
--cc=virtio@lists.oasis-open.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).