* vDPA bus driver selection @ 2021-10-27 14:33 Stefano Garzarella 2021-10-27 15:21 ` Parav Pandit via Virtualization 2021-10-27 18:45 ` Michael S. Tsirkin 0 siblings, 2 replies; 17+ messages in thread From: Stefano Garzarella @ 2021-10-27 14:33 UTC (permalink / raw) To: Jason Wang, Michael Tsirkin; +Cc: Eli Cohen, Linux Virtualization Hi folks, I was trying to understand if we have a way to specify which vDPA bus driver (e.g. vhost-vdpa, virtio-vdpa) a device should use. IIUC we don't have it, and the first registered driver is used when a new device is registered. I was thinking if it makes sense to extend the management API to specify which bus driver to use for a device. A use case could be for example a single host handling VMs and bare-metal containers, so we would have both virtio-vdpa and vhost-vdpa loaded and we want to attach some devices to VMs through vhost-vdpa and others to containers through virtio-vdpa. What do you think? I can prepare an RFC with some code, the idea is to use the .match callback of "struct bus_type" to use a driver instead of the other, and extend netlink API to specify the vDPA bus driver name to use. Thanks, Stefano _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: vDPA bus driver selection 2021-10-27 14:33 vDPA bus driver selection Stefano Garzarella @ 2021-10-27 15:21 ` Parav Pandit via Virtualization 2021-10-27 15:47 ` Stefano Garzarella 2021-10-27 20:16 ` Michael S. Tsirkin 2021-10-27 18:45 ` Michael S. Tsirkin 1 sibling, 2 replies; 17+ messages in thread From: Parav Pandit via Virtualization @ 2021-10-27 15:21 UTC (permalink / raw) To: Stefano Garzarella, Jason Wang, Michael Tsirkin Cc: Eli Cohen, Linux Virtualization Hi Stefano, > From: Stefano Garzarella <sgarzare@redhat.com> > Sent: Wednesday, October 27, 2021 8:04 PM > > Hi folks, > I was trying to understand if we have a way to specify which vDPA bus driver > (e.g. vhost-vdpa, virtio-vdpa) a device should use. > IIUC we don't have it, and the first registered driver is used when a new device > is registered. > > I was thinking if it makes sense to extend the management API to specify which > bus driver to use for a device. A use case could be for example a single host > handling VMs and bare-metal containers, so we would have both virtio-vdpa > and vhost-vdpa loaded and we want to attach some devices to VMs through > vhost-vdpa and others to containers through virtio-vdpa. > > What do you think? > One option is, user keeps the drivers_autoprobe disabled for the vdpa bus using, $ vdpa/vdpa dev add mgmtdev vdpasim_net name vdpa0 mac 00:11:22:33:44:55 $ echo 0 > /sys/bus/vdpa/drivers_autoprobe And after vdpa device creation, it manually binds to the desired driver such as, $ echo vdpa0 > /sys/bus/vdpa/drivers/virtio_vdpa/bind Or $ echo vdpa0 > /sys/bus/vdpa/drivers/vhost_vdpa/bind In an case of VDUSE, it makes more sense to bind to the one of the above driver after user space has connected the use space backend to the kernel device. _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: vDPA bus driver selection 2021-10-27 15:21 ` Parav Pandit via Virtualization @ 2021-10-27 15:47 ` Stefano Garzarella 2021-10-27 15:56 ` Parav Pandit via Virtualization 2021-10-27 20:16 ` Michael S. Tsirkin 1 sibling, 1 reply; 17+ messages in thread From: Stefano Garzarella @ 2021-10-27 15:47 UTC (permalink / raw) To: Parav Pandit; +Cc: Linux Virtualization, Eli Cohen, Michael Tsirkin Hi Parav, On Wed, Oct 27, 2021 at 03:21:15PM +0000, Parav Pandit wrote: >Hi Stefano, > >> From: Stefano Garzarella <sgarzare@redhat.com> >> Sent: Wednesday, October 27, 2021 8:04 PM >> >> Hi folks, >> I was trying to understand if we have a way to specify which vDPA bus driver >> (e.g. vhost-vdpa, virtio-vdpa) a device should use. >> IIUC we don't have it, and the first registered driver is used when a new device >> is registered. >> >> I was thinking if it makes sense to extend the management API to specify which >> bus driver to use for a device. A use case could be for example a single host >> handling VMs and bare-metal containers, so we would have both virtio-vdpa >> and vhost-vdpa loaded and we want to attach some devices to VMs through >> vhost-vdpa and others to containers through virtio-vdpa. >> >> What do you think? >> >One option is, user keeps the drivers_autoprobe disabled for the vdpa bus using, > >$ vdpa/vdpa dev add mgmtdev vdpasim_net name vdpa0 mac 00:11:22:33:44:55 >$ echo 0 > /sys/bus/vdpa/drivers_autoprobe > >And after vdpa device creation, it manually binds to the desired driver such as, > >$ echo vdpa0 > /sys/bus/vdpa/drivers/virtio_vdpa/bind >Or >$ echo vdpa0 > /sys/bus/vdpa/drivers/vhost_vdpa/bind Cool, I didn't know that. This is very useful, but do you think it might be better to integrate it with the netlink API and specify at creation which bus driver to use? > >In an case of VDUSE, it makes more sense to bind to the one of the >above driver after user space has connected the use space backend to >the kernel device. Yep, make sense. Thanks, Stefano _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: vDPA bus driver selection 2021-10-27 15:47 ` Stefano Garzarella @ 2021-10-27 15:56 ` Parav Pandit via Virtualization 2021-10-28 9:37 ` Stefano Garzarella 0 siblings, 1 reply; 17+ messages in thread From: Parav Pandit via Virtualization @ 2021-10-27 15:56 UTC (permalink / raw) To: Stefano Garzarella; +Cc: Linux Virtualization, Eli Cohen, Michael Tsirkin Hi Stefano, > From: Stefano Garzarella <sgarzare@redhat.com> > Sent: Wednesday, October 27, 2021 9:17 PM > To: Parav Pandit <parav@nvidia.com> > Cc: Jason Wang <jasowang@redhat.com>; Michael Tsirkin <mst@redhat.com>; > Linux Virtualization <virtualization@lists.linux-foundation.org>; Eli Cohen > <elic@nvidia.com> > Subject: Re: vDPA bus driver selection > > Hi Parav, > > On Wed, Oct 27, 2021 at 03:21:15PM +0000, Parav Pandit wrote: > >Hi Stefano, > > > >> From: Stefano Garzarella <sgarzare@redhat.com> > >> Sent: Wednesday, October 27, 2021 8:04 PM > >> > >> Hi folks, > >> I was trying to understand if we have a way to specify which vDPA bus > >> driver (e.g. vhost-vdpa, virtio-vdpa) a device should use. > >> IIUC we don't have it, and the first registered driver is used when a > >> new device is registered. > >> > >> I was thinking if it makes sense to extend the management API to > >> specify which bus driver to use for a device. A use case could be for > >> example a single host handling VMs and bare-metal containers, so we > >> would have both virtio-vdpa and vhost-vdpa loaded and we want to > >> attach some devices to VMs through vhost-vdpa and others to containers > through virtio-vdpa. > >> > >> What do you think? > >> > >One option is, user keeps the drivers_autoprobe disabled for the vdpa > >bus using, > > > >$ vdpa/vdpa dev add mgmtdev vdpasim_net name vdpa0 mac > >00:11:22:33:44:55 $ echo 0 > /sys/bus/vdpa/drivers_autoprobe > > > >And after vdpa device creation, it manually binds to the desired driver > >such as, > > > >$ echo vdpa0 > /sys/bus/vdpa/drivers/virtio_vdpa/bind > >Or > >$ echo vdpa0 > /sys/bus/vdpa/drivers/vhost_vdpa/bind > > Cool, I didn't know that. This is very useful, but do you think it might be better > to integrate it with the netlink API and specify at creation which bus driver to > use? I think it is useful; for vduse case we need the ability to say "none" as well and when nothing specified it should be default driver. More than netlink, I think we need the ability in the core kernel to make this choice. I haven't explored what is already available to make that happen. If/once its available, I am sure it has more users than just vdpa. > > > > >In an case of VDUSE, it makes more sense to bind to the one of the > >above driver after user space has connected the use space backend to > >the kernel device. > > Yep, make sense. > > Thanks, > Stefano _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: vDPA bus driver selection 2021-10-27 15:56 ` Parav Pandit via Virtualization @ 2021-10-28 9:37 ` Stefano Garzarella 2021-10-28 9:48 ` Parav Pandit via Virtualization 0 siblings, 1 reply; 17+ messages in thread From: Stefano Garzarella @ 2021-10-28 9:37 UTC (permalink / raw) To: Parav Pandit; +Cc: Linux Virtualization, Eli Cohen, Michael Tsirkin On Wed, Oct 27, 2021 at 03:56:16PM +0000, Parav Pandit wrote: >Hi Stefano, > >> From: Stefano Garzarella <sgarzare@redhat.com> >> Sent: Wednesday, October 27, 2021 9:17 PM >> To: Parav Pandit <parav@nvidia.com> >> Cc: Jason Wang <jasowang@redhat.com>; Michael Tsirkin <mst@redhat.com>; >> Linux Virtualization <virtualization@lists.linux-foundation.org>; Eli Cohen >> <elic@nvidia.com> >> Subject: Re: vDPA bus driver selection >> >> Hi Parav, >> >> On Wed, Oct 27, 2021 at 03:21:15PM +0000, Parav Pandit wrote: >> >Hi Stefano, >> > >> >> From: Stefano Garzarella <sgarzare@redhat.com> >> >> Sent: Wednesday, October 27, 2021 8:04 PM >> >> >> >> Hi folks, >> >> I was trying to understand if we have a way to specify which vDPA bus >> >> driver (e.g. vhost-vdpa, virtio-vdpa) a device should use. >> >> IIUC we don't have it, and the first registered driver is used when a >> >> new device is registered. >> >> >> >> I was thinking if it makes sense to extend the management API to >> >> specify which bus driver to use for a device. A use case could be for >> >> example a single host handling VMs and bare-metal containers, so we >> >> would have both virtio-vdpa and vhost-vdpa loaded and we want to >> >> attach some devices to VMs through vhost-vdpa and others to containers >> through virtio-vdpa. >> >> >> >> What do you think? >> >> >> >One option is, user keeps the drivers_autoprobe disabled for the vdpa >> >bus using, >> > >> >$ vdpa/vdpa dev add mgmtdev vdpasim_net name vdpa0 mac >> >00:11:22:33:44:55 $ echo 0 > /sys/bus/vdpa/drivers_autoprobe >> > >> >And after vdpa device creation, it manually binds to the desired driver >> >such as, >> > >> >$ echo vdpa0 > /sys/bus/vdpa/drivers/virtio_vdpa/bind >> >Or >> >$ echo vdpa0 > /sys/bus/vdpa/drivers/vhost_vdpa/bind >> >> Cool, I didn't know that. This is very useful, but do you think it might be better >> to integrate it with the netlink API and specify at creation which bus driver to >> use? >I think it is useful; for vduse case we need the ability to say "none" >as well and when nothing specified it should be default driver. Yep, the default can be the actual behaviour. > >More than netlink, I think we need the ability in the core kernel to >make this choice. Okay, but I think we can include that in the vdpa tool. >I haven't explored what is already available to make that happen. Me too, I only saw the .match() callback of "struct bus_type" that could be used for this purpose. >If/once its available, I am sure it has more users than just vdpa. Make sense. I'll spend some time next week. Thanks, Stefano _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: vDPA bus driver selection 2021-10-28 9:37 ` Stefano Garzarella @ 2021-10-28 9:48 ` Parav Pandit via Virtualization 2021-10-29 2:34 ` Jason Wang 0 siblings, 1 reply; 17+ messages in thread From: Parav Pandit via Virtualization @ 2021-10-28 9:48 UTC (permalink / raw) To: Stefano Garzarella; +Cc: Linux Virtualization, Eli Cohen, Michael Tsirkin > From: Stefano Garzarella <sgarzare@redhat.com> > Sent: Thursday, October 28, 2021 3:08 PM > >> >$ vdpa/vdpa dev add mgmtdev vdpasim_net name vdpa0 mac > >> >00:11:22:33:44:55 $ echo 0 > /sys/bus/vdpa/drivers_autoprobe > >> > > >> >And after vdpa device creation, it manually binds to the desired > >> >driver such as, > >> > > >> >$ echo vdpa0 > /sys/bus/vdpa/drivers/virtio_vdpa/bind > >> >Or > >> >$ echo vdpa0 > /sys/bus/vdpa/drivers/vhost_vdpa/bind > >> > >> Cool, I didn't know that. This is very useful, but do you think it > >> might be better to integrate it with the netlink API and specify at > >> creation which bus driver to use? > >I think it is useful; for vduse case we need the ability to say "none" > >as well and when nothing specified it should be default driver. > > Yep, the default can be the actual behaviour. > > > > >More than netlink, I think we need the ability in the core kernel to > >make this choice. > > Okay, but I think we can include that in the vdpa tool. If functionality and interface exists in other part of the it is hard to wrap that in vdpa tool that does the duplicate work. > > >I haven't explored what is already available to make that happen. > > Me too, I only saw the .match() callback of "struct bus_type" that could be used > for this purpose. > > >If/once its available, I am sure it has more users than just vdpa. > > Make sense. I'll spend some time next week. Ok. yeah, may be a desired driver can be stored in the vdpa_device that match() routine can use. And if that driver is not available, it returns EPROBE_DEFER; so whenever such driver is loaded it can be bind. But I think before wrapping something in vdpa, need to find a reason why the current interface doesn't solve the problem and also to figure out plumbing. _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: vDPA bus driver selection 2021-10-28 9:48 ` Parav Pandit via Virtualization @ 2021-10-29 2:34 ` Jason Wang 2021-10-29 14:41 ` Stefano Garzarella 0 siblings, 1 reply; 17+ messages in thread From: Jason Wang @ 2021-10-29 2:34 UTC (permalink / raw) To: Parav Pandit; +Cc: Eli Cohen, Michael Tsirkin, Linux Virtualization On Thu, Oct 28, 2021 at 5:48 PM Parav Pandit <parav@nvidia.com> wrote: > > > > > From: Stefano Garzarella <sgarzare@redhat.com> > > Sent: Thursday, October 28, 2021 3:08 PM > > > >> >$ vdpa/vdpa dev add mgmtdev vdpasim_net name vdpa0 mac > > >> >00:11:22:33:44:55 $ echo 0 > /sys/bus/vdpa/drivers_autoprobe > > >> > > > >> >And after vdpa device creation, it manually binds to the desired > > >> >driver such as, > > >> > > > >> >$ echo vdpa0 > /sys/bus/vdpa/drivers/virtio_vdpa/bind > > >> >Or > > >> >$ echo vdpa0 > /sys/bus/vdpa/drivers/vhost_vdpa/bind > > >> > > >> Cool, I didn't know that. This is very useful, but do you think it > > >> might be better to integrate it with the netlink API and specify at > > >> creation which bus driver to use? > > >I think it is useful; for vduse case we need the ability to say "none" > > >as well and when nothing specified it should be default driver. > > > > Yep, the default can be the actual behaviour. > > > > > > > >More than netlink, I think we need the ability in the core kernel to > > >make this choice. > > > > Okay, but I think we can include that in the vdpa tool. > If functionality and interface exists in other part of the it is hard to wrap that in vdpa tool that does the duplicate work. > > > > > >I haven't explored what is already available to make that happen. > > > > Me too, I only saw the .match() callback of "struct bus_type" that could be used > > for this purpose. > > > > >If/once its available, I am sure it has more users than just vdpa. > > > > Make sense. I'll spend some time next week. > > Ok. yeah, may be a desired driver can be stored in the vdpa_device that match() routine can use. > And if that driver is not available, it returns EPROBE_DEFER; so whenever such driver is loaded it can be bind. > > But I think before wrapping something in vdpa, need to find a reason why the current interface doesn't solve the problem and also to figure out plumbing. I agree. If it's something that can easily be addressed by the management code (just a matter of the extra steps for manual setup). It's probably not worth bothering. Thanks > _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: vDPA bus driver selection 2021-10-29 2:34 ` Jason Wang @ 2021-10-29 14:41 ` Stefano Garzarella 2021-10-29 15:41 ` Parav Pandit via Virtualization 0 siblings, 1 reply; 17+ messages in thread From: Stefano Garzarella @ 2021-10-29 14:41 UTC (permalink / raw) To: Jason Wang, Parav Pandit; +Cc: Eli Cohen, Linux Virtualization, Michael Tsirkin On Fri, Oct 29, 2021 at 10:34:00AM +0800, Jason Wang wrote: >On Thu, Oct 28, 2021 at 5:48 PM Parav Pandit <parav@nvidia.com> wrote: >> >> >> >> > From: Stefano Garzarella <sgarzare@redhat.com> >> > Sent: Thursday, October 28, 2021 3:08 PM >> >> > >> >$ vdpa/vdpa dev add mgmtdev vdpasim_net name vdpa0 mac >> > >> >00:11:22:33:44:55 $ echo 0 > /sys/bus/vdpa/drivers_autoprobe >> > >> > >> > >> >And after vdpa device creation, it manually binds to the desired >> > >> >driver such as, >> > >> > >> > >> >$ echo vdpa0 > /sys/bus/vdpa/drivers/virtio_vdpa/bind >> > >> >Or >> > >> >$ echo vdpa0 > /sys/bus/vdpa/drivers/vhost_vdpa/bind >> > >> >> > >> Cool, I didn't know that. This is very useful, but do you think it >> > >> might be better to integrate it with the netlink API and specify at >> > >> creation which bus driver to use? >> > >I think it is useful; for vduse case we need the ability to say "none" >> > >as well and when nothing specified it should be default driver. >> > >> > Yep, the default can be the actual behaviour. >> > >> > > >> > >More than netlink, I think we need the ability in the core kernel to >> > >make this choice. >> > >> > Okay, but I think we can include that in the vdpa tool. >> If functionality and interface exists in other part of the it is hard to wrap that in vdpa tool that does the duplicate work. Got it. >> >> > >> > >I haven't explored what is already available to make that happen. >> > >> > Me too, I only saw the .match() callback of "struct bus_type" that could be used >> > for this purpose. >> > >> > >If/once its available, I am sure it has more users than just vdpa. >> > >> > Make sense. I'll spend some time next week. >> >> Ok. yeah, may be a desired driver can be stored in the vdpa_device that match() routine can use. >> And if that driver is not available, it returns EPROBE_DEFER; so whenever such driver is loaded it can be bind. >> >> But I think before wrapping something in vdpa, need to find a reason why the current interface doesn't solve the problem and also to figure out plumbing. > Yep, when I started this thread I wasn't aware of that APIs available through sysfs. It could be useful to start documenting vDPA (life cycle, management API, etc.). I have a plan to add some vDPA docs in linux/Documentation, maybe we can include also these things. >I agree. If it's something that can easily be addressed by the >management code (just a matter of the extra steps for manual setup). >It's probably not worth bothering. Yep, I agree too. It seems we can easily switch the vDPA bus driver at runtime. Maybe the only missing point is a way to specify the default bus driver to use for a device. Of course the workaround is to unbind it and bind to the desired one. Thank you so much for this very helpful discussion, Stefano _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: vDPA bus driver selection 2021-10-29 14:41 ` Stefano Garzarella @ 2021-10-29 15:41 ` Parav Pandit via Virtualization 0 siblings, 0 replies; 17+ messages in thread From: Parav Pandit via Virtualization @ 2021-10-29 15:41 UTC (permalink / raw) To: Stefano Garzarella, Jason Wang Cc: Eli Cohen, Linux Virtualization, Michael Tsirkin Hi Stefano, > From: Stefano Garzarella <sgarzare@redhat.com> > Sent: Friday, October 29, 2021 8:11 PM > > Maybe the only missing point is a way to specify the default bus driver to use > for a device. Of course the workaround is to unbind it and bind to the desired > one. > Unbind bind can be done but it is slower for deployment at scale. echo 0 > /sys/bus/vdpa/drivers_autoprobe above command disables binding vdpa device to the driver. So user can choose it explicitly without affecting performance. _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: vDPA bus driver selection 2021-10-27 15:21 ` Parav Pandit via Virtualization 2021-10-27 15:47 ` Stefano Garzarella @ 2021-10-27 20:16 ` Michael S. Tsirkin 2021-10-28 2:24 ` Jason Wang 2021-10-28 3:10 ` Parav Pandit via Virtualization 1 sibling, 2 replies; 17+ messages in thread From: Michael S. Tsirkin @ 2021-10-27 20:16 UTC (permalink / raw) To: Parav Pandit; +Cc: Linux Virtualization, Eli Cohen On Wed, Oct 27, 2021 at 03:21:15PM +0000, Parav Pandit wrote: > Hi Stefano, > > > From: Stefano Garzarella <sgarzare@redhat.com> > > Sent: Wednesday, October 27, 2021 8:04 PM > > > > Hi folks, > > I was trying to understand if we have a way to specify which vDPA bus driver > > (e.g. vhost-vdpa, virtio-vdpa) a device should use. > > IIUC we don't have it, and the first registered driver is used when a new device > > is registered. > > > > I was thinking if it makes sense to extend the management API to specify which > > bus driver to use for a device. A use case could be for example a single host > > handling VMs and bare-metal containers, so we would have both virtio-vdpa > > and vhost-vdpa loaded and we want to attach some devices to VMs through > > vhost-vdpa and others to containers through virtio-vdpa. > > > > What do you think? > > > One option is, user keeps the drivers_autoprobe disabled for the vdpa bus using, > > $ vdpa/vdpa dev add mgmtdev vdpasim_net name vdpa0 mac 00:11:22:33:44:55 > $ echo 0 > /sys/bus/vdpa/drivers_autoprobe > > And after vdpa device creation, it manually binds to the desired driver such as, > > $ echo vdpa0 > /sys/bus/vdpa/drivers/virtio_vdpa/bind > Or > $ echo vdpa0 > /sys/bus/vdpa/drivers/vhost_vdpa/bind > > In an case of VDUSE, it makes more sense to bind to the one of the above driver after user space has connected the use space backend to the kernel device. The only annoying thing is that manual bind is not validated. E.g. if one makes a mistake and binds an incorrect device, it just tends to crash IIRC. Another is that it all needs to be root. -- MST _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: vDPA bus driver selection 2021-10-27 20:16 ` Michael S. Tsirkin @ 2021-10-28 2:24 ` Jason Wang 2021-10-28 9:47 ` Stefano Garzarella 2021-10-28 3:10 ` Parav Pandit via Virtualization 1 sibling, 1 reply; 17+ messages in thread From: Jason Wang @ 2021-10-28 2:24 UTC (permalink / raw) To: Michael S. Tsirkin; +Cc: Eli Cohen, Linux Virtualization On Thu, Oct 28, 2021 at 4:16 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Wed, Oct 27, 2021 at 03:21:15PM +0000, Parav Pandit wrote: > > Hi Stefano, > > > > > From: Stefano Garzarella <sgarzare@redhat.com> > > > Sent: Wednesday, October 27, 2021 8:04 PM > > > > > > Hi folks, > > > I was trying to understand if we have a way to specify which vDPA bus driver > > > (e.g. vhost-vdpa, virtio-vdpa) a device should use. > > > IIUC we don't have it, and the first registered driver is used when a new device > > > is registered. > > > > > > I was thinking if it makes sense to extend the management API to specify which > > > bus driver to use for a device. Actually, we want to support this in the first version of vDPA bus. But for some reason it was dropped. The idea is to specify the device type 'virtio' or 'vhost'. But a concern is that, it may encourage vendor to implement e.g virtio specific device (without DMA isolation). >A use case could be for example a single host > > > handling VMs and bare-metal containers, so we would have both virtio-vdpa > > > and vhost-vdpa loaded and we want to attach some devices to VMs through > > > vhost-vdpa and others to containers through virtio-vdpa. > > > > > > What do you think? > > > > > One option is, user keeps the drivers_autoprobe disabled for the vdpa bus using, > > > > $ vdpa/vdpa dev add mgmtdev vdpasim_net name vdpa0 mac 00:11:22:33:44:55 > > $ echo 0 > /sys/bus/vdpa/drivers_autoprobe > > > > And after vdpa device creation, it manually binds to the desired driver such as, > > > > $ echo vdpa0 > /sys/bus/vdpa/drivers/virtio_vdpa/bind > > Or > > $ echo vdpa0 > /sys/bus/vdpa/drivers/vhost_vdpa/bind > > > > In an case of VDUSE, it makes more sense to bind to the one of the above driver after user space has connected the use space backend to the kernel device. > > The only annoying thing is that manual bind is not validated. > E.g. if one makes a mistake and binds an incorrect device, > it just tends to crash IIRC. > Another is that it all needs to be root. I'm not sure it's worth bothering with. As discussed, switching between vendor drivers and vfio requires manually bind/unbind as well. Thanks > > -- > MST > _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: vDPA bus driver selection 2021-10-28 2:24 ` Jason Wang @ 2021-10-28 9:47 ` Stefano Garzarella 2021-10-29 2:31 ` Jason Wang 0 siblings, 1 reply; 17+ messages in thread From: Stefano Garzarella @ 2021-10-28 9:47 UTC (permalink / raw) To: Jason Wang; +Cc: Eli Cohen, Linux Virtualization, Michael S. Tsirkin On Thu, Oct 28, 2021 at 10:24:47AM +0800, Jason Wang wrote: >On Thu, Oct 28, 2021 at 4:16 AM Michael S. Tsirkin <mst@redhat.com> wrote: >> >> On Wed, Oct 27, 2021 at 03:21:15PM +0000, Parav Pandit wrote: >> > Hi Stefano, >> > >> > > From: Stefano Garzarella <sgarzare@redhat.com> >> > > Sent: Wednesday, October 27, 2021 8:04 PM >> > > >> > > Hi folks, >> > > I was trying to understand if we have a way to specify which vDPA bus driver >> > > (e.g. vhost-vdpa, virtio-vdpa) a device should use. >> > > IIUC we don't have it, and the first registered driver is used when a new device >> > > is registered. >> > > >> > > I was thinking if it makes sense to extend the management API to specify which >> > > bus driver to use for a device. > >Actually, we want to support this in the first version of vDPA bus. >But for some reason it was dropped. The idea is to specify the device >type 'virtio' or 'vhost'. But a concern is that, it may encourage >vendor to implement e.g virtio specific device (without DMA >isolation). Yep, I see the issue about device type, so I think make sense to require the support of both, how it is now basically. So instead of defining the type of the device, we could provide the possibility to choose which bus to connect it to, in this way we continue to require that both are supported. As Michael suggested, instead of specify it at the creation time as was in my original idea, we can provide an API to attach/detach a device to a specific vDPA bus. Of course, providing a default behaviour like now, which connects to the first registered. Thanks, Stefano _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: vDPA bus driver selection 2021-10-28 9:47 ` Stefano Garzarella @ 2021-10-29 2:31 ` Jason Wang 2021-10-29 14:32 ` Stefano Garzarella 0 siblings, 1 reply; 17+ messages in thread From: Jason Wang @ 2021-10-29 2:31 UTC (permalink / raw) To: Stefano Garzarella; +Cc: Eli Cohen, Linux Virtualization, Michael S. Tsirkin On Thu, Oct 28, 2021 at 5:47 PM Stefano Garzarella <sgarzare@redhat.com> wrote: > > On Thu, Oct 28, 2021 at 10:24:47AM +0800, Jason Wang wrote: > >On Thu, Oct 28, 2021 at 4:16 AM Michael S. Tsirkin <mst@redhat.com> wrote: > >> > >> On Wed, Oct 27, 2021 at 03:21:15PM +0000, Parav Pandit wrote: > >> > Hi Stefano, > >> > > >> > > From: Stefano Garzarella <sgarzare@redhat.com> > >> > > Sent: Wednesday, October 27, 2021 8:04 PM > >> > > > >> > > Hi folks, > >> > > I was trying to understand if we have a way to specify which vDPA bus driver > >> > > (e.g. vhost-vdpa, virtio-vdpa) a device should use. > >> > > IIUC we don't have it, and the first registered driver is used when a new device > >> > > is registered. > >> > > > >> > > I was thinking if it makes sense to extend the management API to specify which > >> > > bus driver to use for a device. > > > >Actually, we want to support this in the first version of vDPA bus. > >But for some reason it was dropped. The idea is to specify the device > >type 'virtio' or 'vhost'. But a concern is that, it may encourage > >vendor to implement e.g virtio specific device (without DMA > >isolation). > > Yep, I see the issue about device type, so I think make sense to require > the support of both, how it is now basically. > > So instead of defining the type of the device, we could provide the > possibility to choose which bus to connect it to, I think you meant the "bus driver" here? > in this way we > continue to require that both are supported. > > As Michael suggested, instead of specify it at the creation time as was > in my original idea, we can provide an API to attach/detach a device to > a specific vDPA bus. Does such an API exist in driver core? > > Of course, providing a default behaviour like now, which connects to the > first registered. If we want to change this, we can introduce "driver_override". Thanks > > Thanks, > Stefano > _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: vDPA bus driver selection 2021-10-29 2:31 ` Jason Wang @ 2021-10-29 14:32 ` Stefano Garzarella 0 siblings, 0 replies; 17+ messages in thread From: Stefano Garzarella @ 2021-10-29 14:32 UTC (permalink / raw) To: Jason Wang; +Cc: Eli Cohen, Linux Virtualization, Michael S. Tsirkin On Fri, Oct 29, 2021 at 10:31:22AM +0800, Jason Wang wrote: >On Thu, Oct 28, 2021 at 5:47 PM Stefano Garzarella <sgarzare@redhat.com> wrote: >> >> On Thu, Oct 28, 2021 at 10:24:47AM +0800, Jason Wang wrote: >> >On Thu, Oct 28, 2021 at 4:16 AM Michael S. Tsirkin <mst@redhat.com> wrote: >> >> >> >> On Wed, Oct 27, 2021 at 03:21:15PM +0000, Parav Pandit wrote: >> >> > Hi Stefano, >> >> > >> >> > > From: Stefano Garzarella <sgarzare@redhat.com> >> >> > > Sent: Wednesday, October 27, 2021 8:04 PM >> >> > > >> >> > > Hi folks, >> >> > > I was trying to understand if we have a way to specify which vDPA bus driver >> >> > > (e.g. vhost-vdpa, virtio-vdpa) a device should use. >> >> > > IIUC we don't have it, and the first registered driver is used when a new device >> >> > > is registered. >> >> > > >> >> > > I was thinking if it makes sense to extend the management API to specify which >> >> > > bus driver to use for a device. >> > >> >Actually, we want to support this in the first version of vDPA bus. >> >But for some reason it was dropped. The idea is to specify the device >> >type 'virtio' or 'vhost'. But a concern is that, it may encourage >> >vendor to implement e.g virtio specific device (without DMA >> >isolation). >> >> Yep, I see the issue about device type, so I think make sense to require >> the support of both, how it is now basically. >> >> So instead of defining the type of the device, we could provide the >> possibility to choose which bus to connect it to, > >I think you meant the "bus driver" here? Yep, sorry! > >> in this way we >> continue to require that both are supported. >> >> As Michael suggested, instead of specify it at the creation time as >> was >> in my original idea, we can provide an API to attach/detach a device >> to >> a specific vDPA bus. > >Does such an API exist in driver core? I need to check better, Parav showed something with sysfs to bind a device to a driver, so maybe yes. I just tried the following and it worked: $ vdpa dev add mgmtdev vdpasim_net name vdpa0 $ readlink -f /sys/bus/vdpa/devices/vdpa0/driver /sys/bus/vdpa/drivers/vhost_vdpa $ echo vdpa0 > /sys/bus/vdpa/devices/vdpa0/driver/unbind $ echo vdpa0 > /sys/bus/vdpa/drivers/virtio_vdpa/bind > >> >> Of course, providing a default behaviour like now, which connects to the >> first registered. > >If we want to change this, we can introduce "driver_override". Yep, I'll take a look. Thanks, Stefano _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 17+ messages in thread
* RE: vDPA bus driver selection 2021-10-27 20:16 ` Michael S. Tsirkin 2021-10-28 2:24 ` Jason Wang @ 2021-10-28 3:10 ` Parav Pandit via Virtualization 1 sibling, 0 replies; 17+ messages in thread From: Parav Pandit via Virtualization @ 2021-10-28 3:10 UTC (permalink / raw) To: Michael S. Tsirkin; +Cc: Linux Virtualization, Eli Cohen > From: Michael S. Tsirkin <mst@redhat.com> > Sent: Thursday, October 28, 2021 1:46 AM > > On Wed, Oct 27, 2021 at 03:21:15PM +0000, Parav Pandit wrote: > > Hi Stefano, > > > > > From: Stefano Garzarella <sgarzare@redhat.com> > > > Sent: Wednesday, October 27, 2021 8:04 PM > > > > > > Hi folks, > > > I was trying to understand if we have a way to specify which vDPA > > > bus driver (e.g. vhost-vdpa, virtio-vdpa) a device should use. > > > IIUC we don't have it, and the first registered driver is used when > > > a new device is registered. > > > > > > I was thinking if it makes sense to extend the management API to > > > specify which bus driver to use for a device. A use case could be > > > for example a single host handling VMs and bare-metal containers, so > > > we would have both virtio-vdpa and vhost-vdpa loaded and we want to > > > attach some devices to VMs through vhost-vdpa and others to containers > through virtio-vdpa. > > > > > > What do you think? > > > > > One option is, user keeps the drivers_autoprobe disabled for the vdpa > > bus using, > > > > $ vdpa/vdpa dev add mgmtdev vdpasim_net name vdpa0 mac > > 00:11:22:33:44:55 $ echo 0 > /sys/bus/vdpa/drivers_autoprobe > > > > And after vdpa device creation, it manually binds to the desired > > driver such as, > > > > $ echo vdpa0 > /sys/bus/vdpa/drivers/virtio_vdpa/bind > > Or > > $ echo vdpa0 > /sys/bus/vdpa/drivers/vhost_vdpa/bind > > > > In an case of VDUSE, it makes more sense to bind to the one of the above > driver after user space has connected the use space backend to the kernel > device. > > The only annoying thing is that manual bind is not validated. > E.g. if one makes a mistake and binds an incorrect device, it just tends to crash > IIRC. Only a vdpa device can be bind/unbind to a vdpa bus driver. Such checks are done by the kernel core. I didn't follow when can it crash. Can you please share an example when can it crash? _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: vDPA bus driver selection 2021-10-27 14:33 vDPA bus driver selection Stefano Garzarella 2021-10-27 15:21 ` Parav Pandit via Virtualization @ 2021-10-27 18:45 ` Michael S. Tsirkin 2021-10-28 9:34 ` Stefano Garzarella 1 sibling, 1 reply; 17+ messages in thread From: Michael S. Tsirkin @ 2021-10-27 18:45 UTC (permalink / raw) To: Stefano Garzarella; +Cc: Eli Cohen, Linux Virtualization On Wed, Oct 27, 2021 at 04:33:50PM +0200, Stefano Garzarella wrote: > Hi folks, > I was trying to understand if we have a way to specify which vDPA bus > driver (e.g. vhost-vdpa, virtio-vdpa) a device should use. > IIUC we don't have it, and the first registered driver is used when a > new device is registered. > > I was thinking if it makes sense to extend the management API to > specify which bus driver to use for a device. A use case could be for > example a single host handling VMs and bare-metal containers, so we > would have both virtio-vdpa and vhost-vdpa loaded and we want to > attach some devices to VMs through vhost-vdpa and others to containers > through virtio-vdpa. > > What do you think? > > I can prepare an RFC with some code, the idea is to use the .match > callback of "struct bus_type" to use a driver instead of the other, > and extend netlink API to specify the vDPA bus driver name to use. > > Thanks, > Stefano So I think that doing this at create time is somewhat limited. For example a great way to do migration could be to unbind device from VM then bind virtio on the host to it, then bind macvtap to that. Ideas on how to allow that? -- MST _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: vDPA bus driver selection 2021-10-27 18:45 ` Michael S. Tsirkin @ 2021-10-28 9:34 ` Stefano Garzarella 0 siblings, 0 replies; 17+ messages in thread From: Stefano Garzarella @ 2021-10-28 9:34 UTC (permalink / raw) To: Michael S. Tsirkin; +Cc: Eli Cohen, Linux Virtualization On Wed, Oct 27, 2021 at 02:45:15PM -0400, Michael S. Tsirkin wrote: >On Wed, Oct 27, 2021 at 04:33:50PM +0200, Stefano Garzarella wrote: >> Hi folks, >> I was trying to understand if we have a way to specify which vDPA bus >> driver (e.g. vhost-vdpa, virtio-vdpa) a device should use. >> IIUC we don't have it, and the first registered driver is used when a >> new device is registered. >> >> I was thinking if it makes sense to extend the management API to >> specify which bus driver to use for a device. A use case could be for >> example a single host handling VMs and bare-metal containers, so we >> would have both virtio-vdpa and vhost-vdpa loaded and we want to >> attach some devices to VMs through vhost-vdpa and others to containers >> through virtio-vdpa. >> >> What do you think? >> >> I can prepare an RFC with some code, the idea is to use the .match >> callback of "struct bus_type" to use a driver instead of the other, >> and extend netlink API to specify the vDPA bus driver name to use. >> >> Thanks, >> Stefano > >So I think that doing this at create time is somewhat limited. >For example a great way to do migration could be to unbind >device from VM then bind virtio on the host to it, then >bind macvtap to that. Yep, make sense! > >Ideas on how to allow that? Maybe we can split the device creation and the attach (probe) in two APIs, then we can add a detach API to allow changing bus. Anyway I need to check the code better, but I think it's doable. Stefano _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2021-10-29 15:41 UTC | newest] Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-10-27 14:33 vDPA bus driver selection Stefano Garzarella 2021-10-27 15:21 ` Parav Pandit via Virtualization 2021-10-27 15:47 ` Stefano Garzarella 2021-10-27 15:56 ` Parav Pandit via Virtualization 2021-10-28 9:37 ` Stefano Garzarella 2021-10-28 9:48 ` Parav Pandit via Virtualization 2021-10-29 2:34 ` Jason Wang 2021-10-29 14:41 ` Stefano Garzarella 2021-10-29 15:41 ` Parav Pandit via Virtualization 2021-10-27 20:16 ` Michael S. Tsirkin 2021-10-28 2:24 ` Jason Wang 2021-10-28 9:47 ` Stefano Garzarella 2021-10-29 2:31 ` Jason Wang 2021-10-29 14:32 ` Stefano Garzarella 2021-10-28 3:10 ` Parav Pandit via Virtualization 2021-10-27 18:45 ` Michael S. Tsirkin 2021-10-28 9:34 ` Stefano Garzarella
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.