From: "Tian, Kevin" <kevin.tian@intel.com> To: Alex Williamson <alex.williamson@redhat.com>, Kirti Wankhede <kwankhede@nvidia.com> Cc: Paolo Bonzini <pbonzini@redhat.com>, Michal Privoznik <mprivozn@redhat.com>, "Song, Jike" <jike.song@intel.com>, "cjia@nvidia.com" <cjia@nvidia.com>, "kvm@vger.kernel.org" <kvm@vger.kernel.org>, "libvir-list@redhat.com" <libvir-list@redhat.com>, "qemu-devel@nongnu.org" <qemu-devel@nongnu.org>, "kraxel@redhat.com" <kraxel@redhat.com>, Laine Stump <laine@redhat.com>, "bjsdjshi@linux.vnet.ibm.com" <bjsdjshi@linux.vnet.ibm.com> Subject: RE: [Qemu-devel] [PATCH v7 0/4] Add Mediated device support Date: Wed, 7 Sep 2016 06:48:25 +0000 [thread overview] Message-ID: <AADFC41AFE54684AB9EE6CBC0274A5D18DECAA9A@SHSMSX101.ccr.corp.intel.com> (raw) In-Reply-To: <20160906114030.3c9a0162@t450s.home> > From: Alex Williamson [mailto:alex.williamson@redhat.com] > Sent: Wednesday, September 07, 2016 1:41 AM > > On Sat, 3 Sep 2016 22:04:56 +0530 > Kirti Wankhede <kwankhede@nvidia.com> wrote: > > > On 9/3/2016 3:18 AM, Paolo Bonzini wrote: > > > > > > > > > On 02/09/2016 20:33, Kirti Wankhede wrote: > > >> <Alex> We could even do: > > >>>> > > >>>> echo $UUID1:$GROUPA > create > > >>>> > > >>>> where $GROUPA is the group ID of a previously created mdev device into > > >>>> which $UUID1 is to be created and added to the same group. > > >> </Alex> > > > > > > From the point of view of libvirt, I think I prefer Alex's idea. > > > <group> could be an additional element in the nodedev-create XML: > > > > > > <device> > > > <name>my-vgpu</name> > > > <parent>pci_0000_86_00_0</parent> > > > <capability type='mdev'> > > > <type id='11'/> > > > <uuid>0695d332-7831-493f-9e71-1c85c8911a08</uuid> > > > <group>group1</group> > > > </capability> > > > </device> > > > > > > (should group also be a UUID?) > > > > > > > No, this should be a unique number in a system, similar to iommu_group. > > Sorry, just trying to catch up on this thread after a long weekend. > > We're talking about iommu groups here, we're not creating any sort of > parallel grouping specific to mdev devices. This is why my example > created a device and then required the user to go find the group number > given to that device in order to create another device within the same > group. iommu group numbering is not within the user's control and is > not a uuid. libvirt can refer to the group as anything it wants in the > xml, but the host group number is allocated by the host, not under user > control, is not persistent. libvirt would just be giving it a name to > know which devices are part of the same group. Perhaps the runtime xml > would fill in the group number once created. > > There were also a lot of unanswered questions in my proposal, it's not > clear that there's a standard algorithm for when mdev devices need to > be grouped together. Should we even allow groups to span multiple host > devices? Should they be allowed to span devices from different > vendors? I think we should limit the scope of iommu group for mdev here, which better only contains mdev belonging to same parent device. Spanning multiple host devices (regardless of whether from different vendors) are grouped based on physical isolation granularity. Better not to mix two levels together. I'm not sure whether NVIDIA has requirement to start all vGPUs together even when they come from different parent devices. Hope not... > > If we imagine a scenario of a group composed of a mix of Intel and > NVIDIA vGPUs, what happens when an Intel device is opened first? The > NVIDIA driver wouldn't know about this, but it would know when the > first NVIDIA device is opened and be able to establish p2p for the > NVIDIA devices at that point. Can we do what we need with that model? > What if libvirt is asked to hot-add an NVIDIA vGPU? It would need to > do a create on the NVIDIA parent device with the existing group id, at > which point the NVIDIA vendor driver could fail the device create if > the p2p setup has already been done. The Intel vendor driver might > allow it. Similar to open, the last close of the mdev device for a > given vendor (which might not be the last close of mdev devices within > the group) would need to trigger the offline process for that vendor. I assume iommu group is for minimal isolation granularity. In higher level we have VFIO container which could deliver both Intel vGPUs and NVIDIA vGPUs to the same VM. Intel vGPUs each have its own iommu group, while NVIDIA vGPUs of the same parent device may be in one group. Thanks Kevin
WARNING: multiple messages have this Message-ID (diff)
From: "Tian, Kevin" <kevin.tian@intel.com> To: Alex Williamson <alex.williamson@redhat.com>, Kirti Wankhede <kwankhede@nvidia.com> Cc: Paolo Bonzini <pbonzini@redhat.com>, Michal Privoznik <mprivozn@redhat.com>, "Song, Jike" <jike.song@intel.com>, "cjia@nvidia.com" <cjia@nvidia.com>, "kvm@vger.kernel.org" <kvm@vger.kernel.org>, "libvir-list@redhat.com" <libvir-list@redhat.com>, "qemu-devel@nongnu.org" <qemu-devel@nongnu.org>, "kraxel@redhat.com" <kraxel@redhat.com>, Laine Stump <laine@redhat.com>, "bjsdjshi@linux.vnet.ibm.com" <bjsdjshi@linux.vnet.ibm.com> Subject: Re: [Qemu-devel] [PATCH v7 0/4] Add Mediated device support Date: Wed, 7 Sep 2016 06:48:25 +0000 [thread overview] Message-ID: <AADFC41AFE54684AB9EE6CBC0274A5D18DECAA9A@SHSMSX101.ccr.corp.intel.com> (raw) In-Reply-To: <20160906114030.3c9a0162@t450s.home> > From: Alex Williamson [mailto:alex.williamson@redhat.com] > Sent: Wednesday, September 07, 2016 1:41 AM > > On Sat, 3 Sep 2016 22:04:56 +0530 > Kirti Wankhede <kwankhede@nvidia.com> wrote: > > > On 9/3/2016 3:18 AM, Paolo Bonzini wrote: > > > > > > > > > On 02/09/2016 20:33, Kirti Wankhede wrote: > > >> <Alex> We could even do: > > >>>> > > >>>> echo $UUID1:$GROUPA > create > > >>>> > > >>>> where $GROUPA is the group ID of a previously created mdev device into > > >>>> which $UUID1 is to be created and added to the same group. > > >> </Alex> > > > > > > From the point of view of libvirt, I think I prefer Alex's idea. > > > <group> could be an additional element in the nodedev-create XML: > > > > > > <device> > > > <name>my-vgpu</name> > > > <parent>pci_0000_86_00_0</parent> > > > <capability type='mdev'> > > > <type id='11'/> > > > <uuid>0695d332-7831-493f-9e71-1c85c8911a08</uuid> > > > <group>group1</group> > > > </capability> > > > </device> > > > > > > (should group also be a UUID?) > > > > > > > No, this should be a unique number in a system, similar to iommu_group. > > Sorry, just trying to catch up on this thread after a long weekend. > > We're talking about iommu groups here, we're not creating any sort of > parallel grouping specific to mdev devices. This is why my example > created a device and then required the user to go find the group number > given to that device in order to create another device within the same > group. iommu group numbering is not within the user's control and is > not a uuid. libvirt can refer to the group as anything it wants in the > xml, but the host group number is allocated by the host, not under user > control, is not persistent. libvirt would just be giving it a name to > know which devices are part of the same group. Perhaps the runtime xml > would fill in the group number once created. > > There were also a lot of unanswered questions in my proposal, it's not > clear that there's a standard algorithm for when mdev devices need to > be grouped together. Should we even allow groups to span multiple host > devices? Should they be allowed to span devices from different > vendors? I think we should limit the scope of iommu group for mdev here, which better only contains mdev belonging to same parent device. Spanning multiple host devices (regardless of whether from different vendors) are grouped based on physical isolation granularity. Better not to mix two levels together. I'm not sure whether NVIDIA has requirement to start all vGPUs together even when they come from different parent devices. Hope not... > > If we imagine a scenario of a group composed of a mix of Intel and > NVIDIA vGPUs, what happens when an Intel device is opened first? The > NVIDIA driver wouldn't know about this, but it would know when the > first NVIDIA device is opened and be able to establish p2p for the > NVIDIA devices at that point. Can we do what we need with that model? > What if libvirt is asked to hot-add an NVIDIA vGPU? It would need to > do a create on the NVIDIA parent device with the existing group id, at > which point the NVIDIA vendor driver could fail the device create if > the p2p setup has already been done. The Intel vendor driver might > allow it. Similar to open, the last close of the mdev device for a > given vendor (which might not be the last close of mdev devices within > the group) would need to trigger the offline process for that vendor. I assume iommu group is for minimal isolation granularity. In higher level we have VFIO container which could deliver both Intel vGPUs and NVIDIA vGPUs to the same VM. Intel vGPUs each have its own iommu group, while NVIDIA vGPUs of the same parent device may be in one group. Thanks Kevin
next prev parent reply other threads:[~2016-09-07 6:48 UTC|newest] Thread overview: 162+ messages / expand[flat|nested] mbox.gz Atom feed top 2016-08-25 3:53 [PATCH v7 0/4] Add Mediated device support Kirti Wankhede 2016-08-25 3:53 ` [Qemu-devel] " Kirti Wankhede 2016-08-25 3:53 ` [PATCH v7 1/4] vfio: Mediated device Core driver Kirti Wankhede 2016-08-25 3:53 ` [Qemu-devel] " Kirti Wankhede 2016-09-08 8:09 ` Jike Song 2016-09-08 8:09 ` [Qemu-devel] " Jike Song 2016-09-08 9:38 ` Neo Jia 2016-09-08 9:38 ` [Qemu-devel] " Neo Jia 2016-09-09 6:26 ` Jike Song 2016-09-09 6:26 ` [Qemu-devel] " Jike Song 2016-09-09 17:48 ` Kirti Wankhede 2016-09-09 17:48 ` [Qemu-devel] " Kirti Wankhede 2016-09-09 18:42 ` Alex Williamson 2016-09-09 18:42 ` [Qemu-devel] " Alex Williamson 2016-09-09 19:55 ` Kirti Wankhede 2016-09-09 19:55 ` [Qemu-devel] " Kirti Wankhede 2016-09-12 5:10 ` Jike Song 2016-09-12 5:10 ` [Qemu-devel] " Jike Song 2016-09-12 7:49 ` Kirti Wankhede 2016-09-12 7:49 ` [Qemu-devel] " Kirti Wankhede 2016-09-12 15:53 ` Alex Williamson 2016-09-12 15:53 ` [Qemu-devel] " Alex Williamson 2016-09-19 7:08 ` Jike Song 2016-09-19 7:08 ` [Qemu-devel] " Jike Song 2016-09-19 17:29 ` Kirti Wankhede 2016-09-19 17:29 ` [Qemu-devel] " Kirti Wankhede 2016-09-19 18:11 ` Alex Williamson 2016-09-19 18:11 ` [Qemu-devel] " Alex Williamson 2016-09-19 20:09 ` Kirti Wankhede 2016-09-19 20:09 ` [Qemu-devel] " Kirti Wankhede 2016-09-19 20:59 ` Alex Williamson 2016-09-20 12:48 ` Jike Song 2016-09-20 12:48 ` [Qemu-devel] " Jike Song 2016-08-25 3:53 ` [PATCH v7 2/4] vfio: VFIO driver for mediated devices Kirti Wankhede 2016-08-25 3:53 ` [Qemu-devel] " Kirti Wankhede 2016-08-25 9:22 ` Dong Jia 2016-08-25 9:22 ` [Qemu-devel] " Dong Jia 2016-08-26 14:13 ` Kirti Wankhede 2016-08-26 14:13 ` [Qemu-devel] " Kirti Wankhede 2016-09-08 2:38 ` Jike Song 2016-09-08 2:38 ` [Qemu-devel] " Jike Song 2016-09-19 18:22 ` Kirti Wankhede 2016-09-19 18:22 ` Kirti Wankhede 2016-09-19 18:36 ` Alex Williamson 2016-09-19 18:36 ` Alex Williamson 2016-09-19 19:13 ` Kirti Wankhede 2016-09-19 19:13 ` Kirti Wankhede 2016-09-19 20:03 ` Alex Williamson 2016-09-19 20:03 ` Alex Williamson 2016-09-20 2:50 ` Jike Song 2016-09-20 16:24 ` Alex Williamson 2016-09-21 3:19 ` Jike Song 2016-09-21 4:51 ` Alex Williamson 2016-09-21 5:02 ` Jike Song 2016-09-08 2:45 ` Jike Song 2016-09-08 2:45 ` [Qemu-devel] " Jike Song 2016-09-13 2:35 ` Jike Song 2016-09-13 2:35 ` [Qemu-devel] " Jike Song 2016-09-20 5:48 ` Dong Jia Shi 2016-09-20 5:48 ` [Qemu-devel] " Dong Jia Shi 2016-09-20 6:37 ` Jike Song 2016-09-20 6:37 ` [Qemu-devel] " Jike Song 2016-09-20 12:53 ` Jike Song 2016-09-20 12:53 ` [Qemu-devel] " Jike Song 2016-08-25 3:53 ` [PATCH v7 3/4] vfio iommu: Add support " Kirti Wankhede 2016-08-25 3:53 ` [Qemu-devel] " Kirti Wankhede 2016-08-25 7:29 ` Dong Jia 2016-08-25 7:29 ` [Qemu-devel] " Dong Jia 2016-08-26 13:50 ` Kirti Wankhede 2016-08-26 13:50 ` [Qemu-devel] " Kirti Wankhede 2016-09-29 2:17 ` Jike Song 2016-09-29 2:17 ` [Qemu-devel] " Jike Song 2016-09-29 15:06 ` Kirti Wankhede 2016-09-29 15:06 ` [Qemu-devel] " Kirti Wankhede 2016-09-30 2:58 ` Jike Song 2016-09-30 2:58 ` [Qemu-devel] " Jike Song 2016-09-30 3:10 ` Jike Song 2016-09-30 3:10 ` [Qemu-devel] " Jike Song 2016-09-30 11:44 ` Kirti Wankhede 2016-09-30 11:44 ` [Qemu-devel] " Kirti Wankhede 2016-10-08 7:09 ` Jike Song 2016-10-08 7:09 ` [Qemu-devel] " Jike Song 2016-08-25 3:53 ` [PATCH v7 4/4] docs: Add Documentation for Mediated devices Kirti Wankhede 2016-08-25 3:53 ` [Qemu-devel] " Kirti Wankhede 2016-09-03 16:40 ` Kirti Wankhede 2016-09-03 16:40 ` [Qemu-devel] " Kirti Wankhede 2016-08-30 16:16 ` [PATCH v7 0/4] Add Mediated device support Alex Williamson 2016-08-30 16:16 ` [Qemu-devel] " Alex Williamson 2016-08-31 6:12 ` Tian, Kevin 2016-08-31 6:12 ` [Qemu-devel] " Tian, Kevin 2016-08-31 7:04 ` Jike Song 2016-08-31 7:04 ` [Qemu-devel] " Jike Song 2016-08-31 15:48 ` Alex Williamson 2016-08-31 15:48 ` [Qemu-devel] " Alex Williamson 2016-09-01 4:09 ` Tian, Kevin 2016-09-01 4:09 ` [Qemu-devel] " Tian, Kevin 2016-09-01 4:10 ` Tian, Kevin 2016-09-01 4:10 ` [Qemu-devel] " Tian, Kevin 2016-09-01 18:22 ` Kirti Wankhede 2016-09-01 18:22 ` [Qemu-devel] " Kirti Wankhede 2016-09-01 20:01 ` Alex Williamson 2016-09-01 20:01 ` [Qemu-devel] " Alex Williamson 2016-09-02 6:17 ` Kirti Wankhede 2016-09-02 6:17 ` [Qemu-devel] " Kirti Wankhede 2016-09-01 16:47 ` Michal Privoznik 2016-09-01 16:59 ` Alex Williamson 2016-09-01 16:59 ` [Qemu-devel] " Alex Williamson 2016-09-02 4:48 ` Michal Privoznik 2016-09-02 5:21 ` Kirti Wankhede 2016-09-02 10:05 ` Paolo Bonzini 2016-09-02 17:15 ` Kirti Wankhede 2016-09-02 17:25 ` Paolo Bonzini 2016-09-02 18:33 ` Kirti Wankhede 2016-09-02 20:29 ` [libvirt] " John Ferlan 2016-09-02 20:29 ` [Qemu-devel] [libvirt] " John Ferlan 2016-09-03 16:31 ` Kirti Wankhede 2016-09-03 16:31 ` [Qemu-devel] " Kirti Wankhede 2016-09-06 17:54 ` [libvirt] [Qemu-devel] " Alex Williamson 2016-09-06 17:54 ` [Qemu-devel] [libvirt] " Alex Williamson 2016-09-02 21:48 ` [Qemu-devel] " Paolo Bonzini 2016-09-03 11:56 ` [libvirt] " John Ferlan 2016-09-03 11:56 ` [Qemu-devel] [libvirt] " John Ferlan 2016-09-03 13:07 ` [libvirt] [Qemu-devel] " Paolo Bonzini 2016-09-03 13:07 ` [Qemu-devel] [libvirt] " Paolo Bonzini 2016-09-03 17:47 ` Kirti Wankhede 2016-09-03 17:47 ` [Qemu-devel] " Kirti Wankhede 2016-09-03 16:34 ` [Qemu-devel] " Kirti Wankhede 2016-09-06 17:40 ` Alex Williamson 2016-09-06 19:35 ` Kirti Wankhede 2016-09-06 21:28 ` Alex Williamson 2016-09-07 8:22 ` Tian, Kevin 2016-09-07 8:22 ` Tian, Kevin 2016-09-07 16:00 ` Alex Williamson 2016-09-07 16:15 ` Kirti Wankhede 2016-09-07 16:44 ` Alex Williamson 2016-09-07 18:06 ` Kirti Wankhede 2016-09-07 22:13 ` Alex Williamson 2016-09-08 18:48 ` Kirti Wankhede 2016-09-08 20:51 ` Alex Williamson 2016-09-07 18:17 ` Neo Jia 2016-09-07 18:27 ` Daniel P. Berrange 2016-09-07 18:32 ` Neo Jia 2016-09-07 6:48 ` Tian, Kevin [this message] 2016-09-07 6:48 ` Tian, Kevin 2016-09-02 20:19 ` [libvirt] " John Ferlan 2016-09-02 20:19 ` [Qemu-devel] [libvirt] " John Ferlan 2016-09-02 21:44 ` [libvirt] [Qemu-devel] " Paolo Bonzini 2016-09-02 21:44 ` [Qemu-devel] [libvirt] " Paolo Bonzini 2016-09-02 23:57 ` [libvirt] [Qemu-devel] " Laine Stump 2016-09-02 23:57 ` [Qemu-devel] [libvirt] " Laine Stump 2016-09-03 16:49 ` [libvirt] [Qemu-devel] " Kirti Wankhede 2016-09-03 16:49 ` [Qemu-devel] [libvirt] " Kirti Wankhede 2016-09-05 7:52 ` [libvirt] [Qemu-devel] " Paolo Bonzini 2016-09-05 7:52 ` [Qemu-devel] [libvirt] " Paolo Bonzini 2016-09-03 11:57 ` [libvirt] [Qemu-devel] " John Ferlan 2016-09-03 11:57 ` [Qemu-devel] [libvirt] " John Ferlan 2016-09-05 7:54 ` [libvirt] [Qemu-devel] " Paolo Bonzini 2016-09-05 7:54 ` [Qemu-devel] [libvirt] " Paolo Bonzini 2016-09-02 17:55 ` [libvirt] [Qemu-devel] " Laine Stump 2016-09-02 17:55 ` [Qemu-devel] [libvirt] " Laine Stump 2016-09-02 19:15 ` Alex Williamson 2016-09-02 19:15 ` [Qemu-devel] " Alex Williamson
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=AADFC41AFE54684AB9EE6CBC0274A5D18DECAA9A@SHSMSX101.ccr.corp.intel.com \ --to=kevin.tian@intel.com \ --cc=alex.williamson@redhat.com \ --cc=bjsdjshi@linux.vnet.ibm.com \ --cc=cjia@nvidia.com \ --cc=jike.song@intel.com \ --cc=kraxel@redhat.com \ --cc=kvm@vger.kernel.org \ --cc=kwankhede@nvidia.com \ --cc=laine@redhat.com \ --cc=libvir-list@redhat.com \ --cc=mprivozn@redhat.com \ --cc=pbonzini@redhat.com \ --cc=qemu-devel@nongnu.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.