Re: [question]: BAR allocation failing

[Ruben <rubenbryon@gmail.com>]

My lord, can't believe that was the answer, all GPUs are successfully
passed through now, thanks Alex!
In case it is of interest, here's the resulting dmesg:
https://drive.google.com/file/d/1b0f4bWPhJXifC8j9_NtB34R7GBA8Dky3/view

For the gal/chap finding this thread in 12 years trying to make an old
machine work:
In the qemu command I added "-cpu host", current kernel parameters
were "rcutree.rcu_idle_gp_delay=1 mem_encrypt=off iommu=off
amd_iommu=off pci=realloc=off,check_enable_amd_mmconf".

On to the next chapter of this machine, which is to enable MMIO
filtering by the hypervisor which is required to make VMs with less
than all resources/GPUs. Luckily NVidia provides 0 detailed
documentation on the matter.

Op vr 16 jul. 2021 om 02:08 schreef Alex Williamson
<alex.williamson@redhat.com>:
>
> On Thu, 15 Jul 2021 18:05:06 -0500
> Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> > On Thu, Jul 15, 2021 at 11:39:54PM +0300, Ruben wrote:
> > > Thanks for the response, here's a link to the entire dmesg log:
> > > https://drive.google.com/file/d/1Uau0cgd2ymYGDXNr1mA9X_UdLoMH_Azn/view
> > >
> > > Some entries that might be of interest:
> >
> > ACPI tells us the host bridge windows are:
> >
> >   acpi PNP0A08:00: host bridge window [mem 0x000a0000-0x000bffff window] (ignored)
> >   acpi PNP0A08:00: host bridge window [mem 0x80000000-0xafffffff window] (ignored)
> >   acpi PNP0A08:00: host bridge window [mem 0xc0000000-0xfebfffff window] (ignored)
> >   acpi PNP0A08:00: host bridge window [mem 0x800000000-0xfffffffff window] (ignored)
> >
> > The 0xc0000000 window is about 1GB and is below 4GB.
> > The 0x800000000 window looks like 32GB.
> >
> > But "pci=nocrs" means we ignore these windows ...
> >
> > > pci_bus 0000:00: root bus resource [mem 0x00000000-0xffffffffff]
> >
> > and instead use this 1TB of address space, from which DRAM is
> > excluded.  I think this is basically everything the CPU can address,
> > and I *think* it comes from this in setup_arch():
> >
> >   iomem_resource.end = (1ULL << boot_cpu_data.x86_phys_bits) - 1;
> >
> > But you have 8 GPUs, each of which needs 128GB + 32MB + 16MB, so you
> > need 1TB + 384MB to map them all, and the CPU can't address that much.
> >
> > Since you're running this on qemu, I assume x86_phys_bits is telling
> > us about the capabilities of the CPU qemu is emulating.  Maybe there's
> > a way to tell qemu to emulate a CPU with more address bits?
>
> "-cpu host" perhaps
>