Re: [PATCH RFC 0/6] i386/pc: Fix creation of >= 1Tb guests on AMD systems with IOMMU

From: Alex Williamson <alex.williamson@redhat.com>
To: David Edmondson <dme@dme.org>
Cc: Eduardo Habkost <ehabkost@redhat.com>,
	"Michael S . Tsirkin" <mst@redhat.com>,
	Richard Henderson <richard.henderson@linaro.org>,
	qemu-devel@nongnu.org, Daniel Jordan <daniel.m.jordan@oracle.com>,
	Auger Eric <eric.auger@redhat.com>,
	Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>,
	Paolo Bonzini <pbonzini@redhat.com>,
	Igor Mammedov <imammedo@redhat.com>,
	Joao Martins <joao.m.martins@oracle.com>
Subject: Re: [PATCH RFC 0/6] i386/pc: Fix creation of >= 1Tb guests on AMD systems with IOMMU
Date: Wed, 23 Jun 2021 13:13:16 -0600	[thread overview]
Message-ID: <20210623131316.478ba1c8.alex.williamson@redhat.com> (raw)
In-Reply-To: <cunv965ni47.fsf@dme.org>

On Wed, 23 Jun 2021 08:40:56 +0100
David Edmondson <dme@dme.org> wrote:

> On Tuesday, 2021-06-22 at 15:16:29 -06, Alex Williamson wrote:
> 
> >> 	   Additionally, an alternative to hardcoded ranges as we do today,
> >> 	   VFIO could advertise the platform valid IOVA ranges without necessarily
> >> 	   requiring to have a PCI device added in the vfio container. That would
> >> 	   fetching the valid IOVA ranges from VFIO, rather than hardcoded IOVA
> >> 	   ranges as we do today. But sadly, wouldn't work for older hypervisors.  
> >
> >
> > $ grep -h . /sys/kernel/iommu_groups/*/reserved_regions | sort -u
> > 0x00000000fee00000 0x00000000feefffff msi
> > 0x000000fd00000000 0x000000ffffffffff reserved
> >
> > Ideally we might take that into account on all hosts, but of course
> > then we run into massive compatibility issues when we consider
> > migration.  We run into similar problems when people try to assign
> > devices to non-x86 TCG hosts, where the arch doesn't have a natural
> > memory hole overlapping the msi range.
> >
> > The issue here is similar to trying to find a set of supported CPU
> > flags across hosts, QEMU only has visibility to the host where it runs,
> > an upper level tool needs to be able to pass through information about
> > compatibility to all possible migration targets.  Towards that end, we
> > should probably have command line options that either allow to specify
> > specific usable or reserved GPA address ranges.  For example something
> > like:
> > 	--reserved-mem-ranges=host
> >
> > Or explicitly:
> >
> > 	--reserved-mem-ranges=13G@1010G,1M@4078M  
> 
> Would this not naturally be a property of a machine model?

That's an option too, maybe a better one given that we already know how
to manage different machine types.  We probably could not adopt QEMU
v6.1 versions of q35 and 440fx to include this reserved range give the
physical address width downsides Joao mentions.  The above was meant to
be more of an explicit method to match the VM address space to the host
versus the implicit mechanism of the machine type.  A new machine type
at least makes the decision a user policy rather than a magic artifact
of the processor vendor where the VM was first created, but it also
doesn't have the flexibility of the extra command line options.  Thanks,

Alex