From: Auger Eric <eric.auger@redhat.com> To: Robin Murphy <robin.murphy@arm.com>, eric.auger.pro@gmail.com, iommu@lists.linux-foundation.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, joro@8bytes.org, alex.williamson@redhat.com, jacob.jun.pan@linux.intel.com, yi.l.liu@linux.intel.com, jean-philippe.brucker@arm.com, will.deacon@arm.com Cc: tianyu.lan@intel.com, ashok.raj@intel.com, marc.zyngier@arm.com, christoffer.dall@arm.com, peter.maydell@linaro.org Subject: Re: [RFC v2 12/20] dma-iommu: Implement NESTED_MSI cookie Date: Sat, 27 Oct 2018 11:24:52 +0200 [thread overview] Message-ID: <1d5181dd-86c6-ffe7-df27-0c8362342039@redhat.com> (raw) In-Reply-To: <70ff6bee-5775-cccb-b74a-db915c907bcd@arm.com> Hi Robin, On 10/25/18 12:05 AM, Robin Murphy wrote: > On 2018-10-24 7:44 pm, Auger Eric wrote: >> Hi Robin, >> >> On 10/24/18 8:02 PM, Robin Murphy wrote: >>> Hi Eric, >>> >>> On 2018-09-18 3:24 pm, Eric Auger wrote: >>>> Up to now, when the type was UNMANAGED, we used to >>>> allocate IOVA pages within a range provided by the user. >>>> This does not work in nested mode. >>>> >>>> If both the host and the guest are exposed with SMMUs, each >>>> would allocate an IOVA. The guest allocates an IOVA (gIOVA) >>>> to map onto the guest MSI doorbell (gDB). The Host allocates >>>> another IOVA (hIOVA) to map onto the physical doorbell (hDB). >>>> >>>> So we end up with 2 unrelated mappings, at S1 and S2: >>>> S1 S2 >>>> gIOVA -> gDB >>>> hIOVA -> hDB >>>> >>>> The PCI device would be programmed with hIOVA. >>>> >>>> iommu_dma_bind_doorbell allows to pass gIOVA/gDB to the host >>>> so that gIOVA can be used by the host instead of re-allocating >>>> a new IOVA. That way the host can create the following nested >>>> mapping: >>>> >>>> S1 S2 >>>> gIOVA -> gDB -> hDB >>>> >>>> this time, the PCI device will be programmed with the gIOVA MSI >>>> doorbell which is correctly map through the 2 stages. >>> >>> If I'm understanding things correctly, this plus a couple of the >>> preceding patches all add up to a rather involved way of coercing an >>> automatic allocator to only "allocate" predetermined addresses in an >>> entirely known-ahead-of-time manner. >> agreed >> Given that the guy calling >>> iommu_dma_bind_doorbell() could seemingly just as easily call >>> iommu_map() at that point and not bother with an allocator cookie and >>> all this machinery at all, what am I missing? >> Well iommu_dma_map_msi_msg() gets called and is part of this existing >> MSI mapping machinery. If we do not do anything this function allocates >> an hIOVA that is not involved in any nested setup. So either we coerce >> the allocator in place (which is what this series does) or we unplug the >> allocator to replace this latter with a simple S2 mapping, as you >> suggest, ie. iommu_map(gDB, hDB). Assuming we unplug the allocator, the >> guy who actually calls iommu_dma_bind_doorbell() knows gDB but does not >> know hDB. So I don't really get how we can simplify things. > > OK, there's what I was missing :D > > But that then seems to reveal a somewhat bigger problem - if the callers > are simply registering IPAs, and relying on the ITS driver to grab an > entry and fill in a PA later, then how does either one know *which* PA > is supposed to belong to a given IPA in the case where you have multiple > devices with different ITS targets assigned to the same guest? You're definitively right here. I think this can be resolved by passing the struct device handle along with the stage1 mapping and storing the info together. Then when the host MSI controller looks for a free unmapped iova, it must also check whether the device belongs to its MSI domain. (and if > it's possible to assume a guest will use per-device stage 1 mappings and > present it with a single vITS backed by multiple pITSes, I think things > start breaking even harder.) I don't really get your point here. Assigned devices on guest side should be in separate iommu domain because we want them to get isolated from each other. There is a single vITS as of now and I don't think we will change that anytime soon. The vITS driver is allocating a gIOVA for each separate domain and I currently "trap" the gIOVA/gPA mapping on irqfd routing setup. This mapping gets associated to a VFIO IOMMU, one per assigned device, so we have different vfio containers for each of them. If I then enumerate all the devices attached to the containers and pass this stage1 binding along with the device struct, I think we should be OK? Thanks Eric > > Other than allowing arbitrary disjoint IOVA pages, I'm not sure this > really works any differently from the existing MSI cookie now that I > look more closely :/ > > Robin.
WARNING: multiple messages have this Message-ID (diff)
From: Auger Eric <eric.auger@redhat.com> To: Robin Murphy <robin.murphy@arm.com>, eric.auger.pro@gmail.com, iommu@lists.linux-foundation.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, joro@8bytes.org, alex.williamson@redhat.com, jacob.jun.pan@linux.intel.com, yi.l.liu@linux.intel.com, jean-philippe.brucker@arm.com, will.deacon@arm.com Cc: tianyu.lan@intel.com, marc.zyngier@arm.com, ashok.raj@intel.com Subject: Re: [RFC v2 12/20] dma-iommu: Implement NESTED_MSI cookie Date: Sat, 27 Oct 2018 11:24:52 +0200 [thread overview] Message-ID: <1d5181dd-86c6-ffe7-df27-0c8362342039@redhat.com> (raw) In-Reply-To: <70ff6bee-5775-cccb-b74a-db915c907bcd@arm.com> Hi Robin, On 10/25/18 12:05 AM, Robin Murphy wrote: > On 2018-10-24 7:44 pm, Auger Eric wrote: >> Hi Robin, >> >> On 10/24/18 8:02 PM, Robin Murphy wrote: >>> Hi Eric, >>> >>> On 2018-09-18 3:24 pm, Eric Auger wrote: >>>> Up to now, when the type was UNMANAGED, we used to >>>> allocate IOVA pages within a range provided by the user. >>>> This does not work in nested mode. >>>> >>>> If both the host and the guest are exposed with SMMUs, each >>>> would allocate an IOVA. The guest allocates an IOVA (gIOVA) >>>> to map onto the guest MSI doorbell (gDB). The Host allocates >>>> another IOVA (hIOVA) to map onto the physical doorbell (hDB). >>>> >>>> So we end up with 2 unrelated mappings, at S1 and S2: >>>> S1 S2 >>>> gIOVA -> gDB >>>> hIOVA -> hDB >>>> >>>> The PCI device would be programmed with hIOVA. >>>> >>>> iommu_dma_bind_doorbell allows to pass gIOVA/gDB to the host >>>> so that gIOVA can be used by the host instead of re-allocating >>>> a new IOVA. That way the host can create the following nested >>>> mapping: >>>> >>>> S1 S2 >>>> gIOVA -> gDB -> hDB >>>> >>>> this time, the PCI device will be programmed with the gIOVA MSI >>>> doorbell which is correctly map through the 2 stages. >>> >>> If I'm understanding things correctly, this plus a couple of the >>> preceding patches all add up to a rather involved way of coercing an >>> automatic allocator to only "allocate" predetermined addresses in an >>> entirely known-ahead-of-time manner. >> agreed >> Given that the guy calling >>> iommu_dma_bind_doorbell() could seemingly just as easily call >>> iommu_map() at that point and not bother with an allocator cookie and >>> all this machinery at all, what am I missing? >> Well iommu_dma_map_msi_msg() gets called and is part of this existing >> MSI mapping machinery. If we do not do anything this function allocates >> an hIOVA that is not involved in any nested setup. So either we coerce >> the allocator in place (which is what this series does) or we unplug the >> allocator to replace this latter with a simple S2 mapping, as you >> suggest, ie. iommu_map(gDB, hDB). Assuming we unplug the allocator, the >> guy who actually calls iommu_dma_bind_doorbell() knows gDB but does not >> know hDB. So I don't really get how we can simplify things. > > OK, there's what I was missing :D > > But that then seems to reveal a somewhat bigger problem - if the callers > are simply registering IPAs, and relying on the ITS driver to grab an > entry and fill in a PA later, then how does either one know *which* PA > is supposed to belong to a given IPA in the case where you have multiple > devices with different ITS targets assigned to the same guest? You're definitively right here. I think this can be resolved by passing the struct device handle along with the stage1 mapping and storing the info together. Then when the host MSI controller looks for a free unmapped iova, it must also check whether the device belongs to its MSI domain. (and if > it's possible to assume a guest will use per-device stage 1 mappings and > present it with a single vITS backed by multiple pITSes, I think things > start breaking even harder.) I don't really get your point here. Assigned devices on guest side should be in separate iommu domain because we want them to get isolated from each other. There is a single vITS as of now and I don't think we will change that anytime soon. The vITS driver is allocating a gIOVA for each separate domain and I currently "trap" the gIOVA/gPA mapping on irqfd routing setup. This mapping gets associated to a VFIO IOMMU, one per assigned device, so we have different vfio containers for each of them. If I then enumerate all the devices attached to the containers and pass this stage1 binding along with the device struct, I think we should be OK? Thanks Eric > > Other than allowing arbitrary disjoint IOVA pages, I'm not sure this > really works any differently from the existing MSI cookie now that I > look more closely :/ > > Robin. _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
next prev parent reply other threads:[~2018-10-27 9:25 UTC|newest] Thread overview: 42+ messages / expand[flat|nested] mbox.gz Atom feed top 2018-09-18 14:24 [RFC v2 00/20] SMMUv3 Nested Stage Setup Eric Auger 2018-09-18 14:24 ` [RFC v2 01/20] iommu: Introduce bind_pasid_table API Eric Auger 2018-09-18 14:24 ` Eric Auger 2018-09-20 17:21 ` Jacob Pan 2018-09-20 17:21 ` Jacob Pan 2018-09-21 9:45 ` Auger Eric 2018-09-18 14:24 ` [RFC v2 02/20] iommu: Introduce cache_invalidate API Eric Auger 2018-09-18 14:24 ` [RFC v2 03/20] iommu: Introduce bind_guest_msi Eric Auger 2018-09-18 14:24 ` [RFC v2 04/20] vfio: VFIO_IOMMU_BIND_PASID_TABLE Eric Auger 2018-09-18 14:24 ` [RFC v2 05/20] vfio: VFIO_IOMMU_CACHE_INVALIDATE Eric Auger 2018-09-18 14:24 ` [RFC v2 06/20] vfio: VFIO_IOMMU_BIND_MSI Eric Auger 2018-09-18 14:24 ` [RFC v2 07/20] iommu/arm-smmu-v3: Link domains and devices Eric Auger 2018-09-18 14:24 ` [RFC v2 08/20] iommu/arm-smmu-v3: Maintain a SID->device structure Eric Auger 2018-09-18 14:24 ` [RFC v2 09/20] iommu/smmuv3: Get prepared for nested stage support Eric Auger 2018-09-18 14:24 ` Eric Auger 2018-09-18 14:24 ` [RFC v2 10/20] iommu/smmuv3: Implement bind_pasid_table Eric Auger 2018-09-18 14:24 ` [RFC v2 11/20] iommu/smmuv3: Implement cache_invalidate Eric Auger 2018-09-18 14:24 ` [RFC v2 12/20] dma-iommu: Implement NESTED_MSI cookie Eric Auger 2018-10-24 18:02 ` Robin Murphy 2018-10-24 18:02 ` Robin Murphy 2018-10-24 18:44 ` Auger Eric 2018-10-24 22:05 ` Robin Murphy 2018-10-27 9:24 ` Auger Eric [this message] 2018-10-27 9:24 ` Auger Eric 2018-09-18 14:24 ` [RFC v2 13/20] iommu/smmuv3: Implement bind_guest_msi Eric Auger 2018-09-18 14:24 ` [RFC v2 14/20] iommu: introduce device fault data Eric Auger 2018-09-20 22:06 ` Jacob Pan 2018-09-21 9:54 ` Auger Eric 2018-09-21 16:18 ` Jacob Pan 2018-12-12 8:21 ` Auger Eric 2018-12-15 0:30 ` Jacob Pan 2018-12-15 0:30 ` Jacob Pan 2018-12-17 9:04 ` Auger Eric 2018-09-18 14:24 ` [RFC v2 15/20] driver core: add per device iommu param Eric Auger 2018-09-18 14:24 ` [RFC v2 16/20] iommu: introduce device fault report API Eric Auger 2018-09-18 14:24 ` Eric Auger 2018-09-18 14:24 ` [RFC v2 17/20] vfio: VFIO_IOMMU_SET_FAULT_EVENTFD Eric Auger 2018-09-18 14:24 ` Eric Auger 2018-09-18 14:24 ` [RFC v2 18/20] vfio: VFIO_IOMMU_GET_FAULT_EVENTS Eric Auger 2018-09-18 14:24 ` Eric Auger 2018-09-18 14:24 ` [RFC v2 19/20] vfio: Document nested stage control Eric Auger 2018-09-18 14:24 ` [RFC v2 20/20] iommu/smmuv3: Report non recoverable faults Eric Auger
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=1d5181dd-86c6-ffe7-df27-0c8362342039@redhat.com \ --to=eric.auger@redhat.com \ --cc=alex.williamson@redhat.com \ --cc=ashok.raj@intel.com \ --cc=christoffer.dall@arm.com \ --cc=eric.auger.pro@gmail.com \ --cc=iommu@lists.linux-foundation.org \ --cc=jacob.jun.pan@linux.intel.com \ --cc=jean-philippe.brucker@arm.com \ --cc=joro@8bytes.org \ --cc=kvm@vger.kernel.org \ --cc=kvmarm@lists.cs.columbia.edu \ --cc=linux-kernel@vger.kernel.org \ --cc=marc.zyngier@arm.com \ --cc=peter.maydell@linaro.org \ --cc=robin.murphy@arm.com \ --cc=tianyu.lan@intel.com \ --cc=will.deacon@arm.com \ --cc=yi.l.liu@linux.intel.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.