From: Don Dutile <ddutile@redhat.com>
To: Alex Williamson <alex.williamson@redhat.com>,
Joerg Roedel <joro@8bytes.org>
Cc: Auger Eric <eric.auger@redhat.com>,
Will Deacon <will.deacon@arm.com>,
drjones@redhat.com,
Christoffer Dall <christoffer.dall@linaro.org>,
jason@lakedaemon.net, kvm@vger.kernel.org, marc.zyngier@arm.com,
benh@kernel.crashing.org, punit.agrawal@arm.com,
linux-kernel@vger.kernel.org, diana.craciun@nxp.com,
iommu@lists.linux-foundation.org, pranav.sawargaonkar@gmail.com,
arnd@arndb.de, dwmw@amazon.co.uk, jcm@redhat.com,
tglx@linutronix.de, robin.murphy@arm.com,
linux-arm-kernel@lists.infradead.org, eric.auger.pro@gmail.com
Subject: Re: Summary of LPC guest MSI discussion in Santa Fe
Date: Fri, 11 Nov 2016 11:25:25 -0500 [thread overview]
Message-ID: <5825F0F5.3070001@redhat.com> (raw)
In-Reply-To: <20161111085056.4cf8989d@t450s.home>
On 11/11/2016 10:50 AM, Alex Williamson wrote:
> On Fri, 11 Nov 2016 12:19:44 +0100
> Joerg Roedel <joro@8bytes.org> wrote:
>
>> On Thu, Nov 10, 2016 at 10:46:01AM -0700, Alex Williamson wrote:
>>> In the case of x86, we know that DMA mappings overlapping the MSI
>>> doorbells won't be translated correctly, it's not a valid mapping for
>>> that range, and therefore the iommu driver backing the IOMMU API
>>> should describe that reserved range and reject mappings to it.
>>
>> The drivers actually allow mappings to the MSI region via the IOMMU-API,
>> and I think it should stay this way also for other reserved ranges.
>> Address space management is done by the IOMMU-API user already (and has
>> to be done there nowadays), be it a DMA-API implementation which just
>> reserves these regions in its address space allocator or be it VFIO with
>> QEMU, which don't map RAM there anyway. So there is no point of checking
>> this again in the IOMMU drivers and we can keep that out of the
>> mapping/unmapping fast-path.
>
> It's really just a happenstance that we don't map RAM over the x86 MSI
> range though. That property really can't be guaranteed once we mix
> architectures, such as running an aarch64 VM on x86 host via TCG.
> AIUI, the MSI range is actually handled differently than other DMA
> ranges, so a iommu_map() overlapping a range that the iommu cannot map
> should fail just like an attempt to map beyond the address width of the
> iommu.
>
+1. As was stated at Plumbers, x86 MSI is in a fixed, hw location, so:
1) that memory space is never a valid page to the system to be used for IOVA,
therefore, nothing to micro-manage in the iommu mapping (fast) path.
2) migration btwn different systems isn't an issue b/c all x86 systems have this mapping.
3) ACS resolves DMA writes to mem going to a device(-mmio space).
For aarch64, without such a 'fixed' MSI location, whatever hole/used-space-struct
concept that is contrived for MSI (DMA) writes on aarch64 won't guarantee migration
failure across mixed aarch64 systems (migrate guest-G from sys-vendor-A to
sys-vendor-B; sys-vendor-A has MSI at addr-A; sys-vendor-B has MSI at addr-B).
Without agreement, migration only possilbe across the same systems (can even
be broken btwn two sytems from same vendor). ACS in the PCIe path handles
the iova->dev-mmio collision problem. q.e.d.
ergo, my proposal to put MSI space as the upper-most, space of any system....
FFFF.FFFF.FFFE0.0000 ... and hw drops the upper 1's/F's, and uses that for MSI.
Allows it to vary on each system based on max-memory. pseudo-fixed, but not
right smack in the middle of mem-space.
There is an inverse scenario for host phys addr's as well:
Wiring the upper-most bit of HPA to be 1==mmio, 0=mem simplifies a lot of
design issues in the cores & chipsets as well. Alpha-EV6, case in point
(18+ yr old design decision). another q.e.d.
I hate to admit it, but jcm has it right wrt 'fixed sys addr map', at least in this IO area.
>>> For PCI devices userspace can examine the topology of the iommu group
>>> and exclude MMIO ranges of peer devices based on the BARs, which are
>>> exposed in various places, pci-sysfs as well as /proc/iomem. For
>>> non-PCI or MSI controllers... ???
>>
>> Right, the hardware resources can be examined. But maybe this can be
>> extended to also cover RMRR ranges? Then we would be able to assign
>> devices with RMRR mappings to guests.
>
> RMRRs are special in a different way, the VT-d spec requires that the
> OS honor RMRRs, the user has no responsibility (and currently no
> visibility) to make that same arrangement. In order to potentially
> protect the physical host platform, the iommu drivers should prevent a
> user from remapping RMRRS. Maybe there needs to be a different
> interface used by untrusted users vs in-kernel drivers, but I think the
> kernel really needs to be defensive in the case of user mappings, which
> is where the IOMMU API is rooted. Thanks,
>
> Alex
>
next prev parent reply other threads:[~2016-11-11 16:25 UTC|newest]
Thread overview: 48+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-11-03 21:39 [RFC 0/8] KVM PCIe/MSI passthrough on ARM/ARM64 (Alt II) Eric Auger
2016-11-03 21:39 ` [RFC 1/8] vfio: fix vfio_info_cap_add/shift Eric Auger
2016-11-03 21:39 ` [RFC 2/8] iommu/iova: fix __alloc_and_insert_iova_range Eric Auger
2016-11-03 21:39 ` [RFC 3/8] iommu/dma: Allow MSI-only cookies Eric Auger
2016-11-03 21:39 ` [RFC 4/8] iommu: Add a list of iommu_reserved_region in iommu_domain Eric Auger
2016-11-03 21:39 ` [RFC 5/8] vfio/type1: Introduce RESV_IOVA_RANGE capability Eric Auger
2016-11-03 21:39 ` [RFC 6/8] iommu: Handle the list of reserved regions Eric Auger
2016-11-03 21:39 ` [RFC 7/8] iommu/vt-d: Implement add_reserved_regions callback Eric Auger
2016-11-03 21:39 ` [RFC 8/8] iommu/arm-smmu: implement " Eric Auger
2016-11-04 4:02 ` [RFC 0/8] KVM PCIe/MSI passthrough on ARM/ARM64 (Alt II) Alex Williamson
2016-11-08 2:45 ` Summary of LPC guest MSI discussion in Santa Fe (was: Re: [RFC 0/8] KVM PCIe/MSI passthrough on ARM/ARM64 (Alt II)) Will Deacon
2016-11-08 14:27 ` Summary of LPC guest MSI discussion in Santa Fe Auger Eric
2016-11-08 17:54 ` Will Deacon
2016-11-08 19:02 ` Don Dutile
2016-11-08 19:10 ` Will Deacon
2016-11-09 7:43 ` Auger Eric
2016-11-08 16:02 ` Don Dutile
2016-11-08 20:29 ` Summary of LPC guest MSI discussion in Santa Fe (was: Re: [RFC 0/8] KVM PCIe/MSI passthrough on ARM/ARM64 (Alt II)) Christoffer Dall
2016-11-08 23:35 ` Alex Williamson
2016-11-09 2:52 ` Summary of LPC guest MSI discussion in Santa Fe Don Dutile
2016-11-09 17:03 ` Will Deacon
2016-11-09 18:59 ` Don Dutile
2016-11-09 19:23 ` Christoffer Dall
2016-11-09 20:01 ` Alex Williamson
2016-11-10 14:40 ` Joerg Roedel
2016-11-10 17:07 ` Alex Williamson
2016-11-09 20:31 ` Will Deacon
2016-11-09 22:17 ` Alex Williamson
2016-11-09 22:25 ` Will Deacon
2016-11-09 23:24 ` Alex Williamson
2016-11-09 23:38 ` Will Deacon
2016-11-09 23:59 ` Alex Williamson
2016-11-10 0:14 ` Auger Eric
2016-11-10 0:55 ` Alex Williamson
2016-11-10 2:01 ` Will Deacon
2016-11-10 11:14 ` Auger Eric
2016-11-10 17:46 ` Alex Williamson
2016-11-11 11:19 ` Joerg Roedel
2016-11-11 15:50 ` Alex Williamson
2016-11-11 16:05 ` Alex Williamson
2016-11-14 15:19 ` Joerg Roedel
2016-11-11 16:25 ` Don Dutile [this message]
2016-11-11 16:00 ` Don Dutile
2016-11-10 14:52 ` Joerg Roedel
2016-11-09 20:11 ` Robin Murphy
2016-11-10 15:18 ` Joerg Roedel
2016-11-21 5:13 ` Jon Masters
2016-11-23 20:12 ` Don Dutile
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5825F0F5.3070001@redhat.com \
--to=ddutile@redhat.com \
--cc=alex.williamson@redhat.com \
--cc=arnd@arndb.de \
--cc=benh@kernel.crashing.org \
--cc=christoffer.dall@linaro.org \
--cc=diana.craciun@nxp.com \
--cc=drjones@redhat.com \
--cc=dwmw@amazon.co.uk \
--cc=eric.auger.pro@gmail.com \
--cc=eric.auger@redhat.com \
--cc=iommu@lists.linux-foundation.org \
--cc=jason@lakedaemon.net \
--cc=jcm@redhat.com \
--cc=joro@8bytes.org \
--cc=kvm@vger.kernel.org \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=marc.zyngier@arm.com \
--cc=pranav.sawargaonkar@gmail.com \
--cc=punit.agrawal@arm.com \
--cc=robin.murphy@arm.com \
--cc=tglx@linutronix.de \
--cc=will.deacon@arm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).