From: Parav Pandit via Virtualization <virtualization@lists.linux-foundation.org> To: Gautam Dawar <gdawar@xilinx.com>, Eugenio Perez Martin <eperezma@redhat.com>, virtualization <virtualization@lists.linux-foundation.org>, qemu-level <qemu-devel@nongnu.org>, Jason Wang <jasowang@redhat.com>, Cindy Lu <lulu@redhat.com>, "virtio-networking@redhat.com" <virtio-networking@redhat.com>, Eli Cohen <elic@nvidia.com>, Laurent Vivier <lvivier@redhat.com>, Stefano Garzarella <sgarzare@redhat.com> Subject: RE: About restoring the state in vhost-vdpa device Date: Fri, 13 May 2022 18:25:45 +0000 [thread overview] Message-ID: <PH0PR12MB54811C88B389ACB495BD5AB2DCCA9@PH0PR12MB5481.namprd12.prod.outlook.com> (raw) In-Reply-To: <BY5PR02MB698052EE53B707C6C04C3C8CB1CA9@BY5PR02MB6980.namprd02.prod.outlook.com> Hi Gautam, Please fix your email client to have right response format. Otherwise, it will be confusing for the rest and us to follow the conversation. More below. > From: Gautam Dawar <gdawar@xilinx.com> > Sent: Friday, May 13, 2022 1:48 PM > > Our proposal diverge in step 7: Instead of enabling *all* the > > virtqueues, only enable the CVQ. > Just to double check, VQ 0 and 1 of the net are also not enabled, correct? > [GD>>] Yes, that's my understanding as well. > > > After that, send the DRIVER_OK and queue all the control commands to > > restore the device status (MQ, RSS, ...). Once all of them have been > > acknowledged ("device", or emulated cvq in host vdpa backend driver, > > has used all cvq buffers, enable (SET_VRING_ENABLE, set_vq_ready) all > > other queues. > > > What is special about doing DRIVER_OK and enqueuing the control > commands? > Why other configuration cannot be applied before DRIVER_OK? > [GD>>] For the device to be live (and any queue to be able to pass traffic), > DRIVER_OK is a must. This applies only to the vdpa device implemented over virtio device. For such use case/implementation anyway a proper virtio spec extension is needed for it be efficient. And what that happens this scheme will still work. Other vdpa devices doesn’t have to live with this limitation at this moment. > So, any configuration can be passed over the CVQ only > after it is started (vring configuration + DRIVER_OK). For an emulated queue, > if the order is reversed, I think the enqueued commands will remain > buffered and device should be able to service them when it goes live. I likely didn’t understand what you describe above. Vq avail index etc is programmed before doing DRIVER_OK anyway. Sequence is very straight forward at destination from user to kernel. 1. Set config space fields (such as virtio_net_config/virtio_blk_config). 2. Set other device attributes (max vqs, current num vqs) 3. Set net specific config (RSS, vlan, mac and more filters) 4. Set VQ fields (enable, msix, addr, avail indx) 5. Set DRIVER_OK, device resumes from where it left off Steps #1 to #4 can be done multiple times in pre-warm device up case in future. For now, they can be done once to get things started. _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
WARNING: multiple messages have this Message-ID (diff)
From: Parav Pandit <parav@nvidia.com> To: Gautam Dawar <gdawar@xilinx.com>, Eugenio Perez Martin <eperezma@redhat.com>, virtualization <virtualization@lists.linux-foundation.org>, qemu-level <qemu-devel@nongnu.org>, Jason Wang <jasowang@redhat.com>, Cindy Lu <lulu@redhat.com>, "virtio-networking@redhat.com" <virtio-networking@redhat.com>, Eli Cohen <elic@nvidia.com>, Laurent Vivier <lvivier@redhat.com>, Stefano Garzarella <sgarzare@redhat.com> Subject: RE: About restoring the state in vhost-vdpa device Date: Fri, 13 May 2022 18:25:45 +0000 [thread overview] Message-ID: <PH0PR12MB54811C88B389ACB495BD5AB2DCCA9@PH0PR12MB5481.namprd12.prod.outlook.com> (raw) In-Reply-To: <BY5PR02MB698052EE53B707C6C04C3C8CB1CA9@BY5PR02MB6980.namprd02.prod.outlook.com> Hi Gautam, Please fix your email client to have right response format. Otherwise, it will be confusing for the rest and us to follow the conversation. More below. > From: Gautam Dawar <gdawar@xilinx.com> > Sent: Friday, May 13, 2022 1:48 PM > > Our proposal diverge in step 7: Instead of enabling *all* the > > virtqueues, only enable the CVQ. > Just to double check, VQ 0 and 1 of the net are also not enabled, correct? > [GD>>] Yes, that's my understanding as well. > > > After that, send the DRIVER_OK and queue all the control commands to > > restore the device status (MQ, RSS, ...). Once all of them have been > > acknowledged ("device", or emulated cvq in host vdpa backend driver, > > has used all cvq buffers, enable (SET_VRING_ENABLE, set_vq_ready) all > > other queues. > > > What is special about doing DRIVER_OK and enqueuing the control > commands? > Why other configuration cannot be applied before DRIVER_OK? > [GD>>] For the device to be live (and any queue to be able to pass traffic), > DRIVER_OK is a must. This applies only to the vdpa device implemented over virtio device. For such use case/implementation anyway a proper virtio spec extension is needed for it be efficient. And what that happens this scheme will still work. Other vdpa devices doesn’t have to live with this limitation at this moment. > So, any configuration can be passed over the CVQ only > after it is started (vring configuration + DRIVER_OK). For an emulated queue, > if the order is reversed, I think the enqueued commands will remain > buffered and device should be able to service them when it goes live. I likely didn’t understand what you describe above. Vq avail index etc is programmed before doing DRIVER_OK anyway. Sequence is very straight forward at destination from user to kernel. 1. Set config space fields (such as virtio_net_config/virtio_blk_config). 2. Set other device attributes (max vqs, current num vqs) 3. Set net specific config (RSS, vlan, mac and more filters) 4. Set VQ fields (enable, msix, addr, avail indx) 5. Set DRIVER_OK, device resumes from where it left off Steps #1 to #4 can be done multiple times in pre-warm device up case in future. For now, they can be done once to get things started.
next prev parent reply other threads:[~2022-05-13 18:25 UTC|newest] Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top 2022-05-11 19:43 About restoring the state in vhost-vdpa device Eugenio Perez Martin 2022-05-12 4:00 ` Jason Wang 2022-05-12 4:00 ` Jason Wang 2022-05-13 15:08 ` Parav Pandit via Virtualization 2022-05-13 15:08 ` Parav Pandit 2022-05-13 17:48 ` Gautam Dawar 2022-05-13 18:25 ` Parav Pandit via Virtualization [this message] 2022-05-13 18:25 ` Parav Pandit 2022-05-16 8:50 ` Eugenio Perez Martin 2022-05-16 20:29 ` Parav Pandit via Virtualization 2022-05-16 20:29 ` Parav Pandit 2022-05-17 3:05 ` Jason Wang 2022-05-17 3:05 ` Jason Wang 2022-05-18 12:43 ` Parav Pandit via Virtualization 2022-05-18 12:43 ` Parav Pandit 2022-05-23 3:12 ` Jason Wang 2022-05-23 3:12 ` Jason Wang 2022-05-17 8:12 ` Eugenio Perez Martin 2022-05-18 12:51 ` Parav Pandit via Virtualization 2022-05-18 12:51 ` Parav Pandit
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=PH0PR12MB54811C88B389ACB495BD5AB2DCCA9@PH0PR12MB5481.namprd12.prod.outlook.com \ --to=virtualization@lists.linux-foundation.org \ --cc=elic@nvidia.com \ --cc=eperezma@redhat.com \ --cc=gdawar@xilinx.com \ --cc=jasowang@redhat.com \ --cc=lulu@redhat.com \ --cc=lvivier@redhat.com \ --cc=parav@nvidia.com \ --cc=qemu-devel@nongnu.org \ --cc=sgarzare@redhat.com \ --cc=virtio-networking@redhat.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.