Re: i915 dma faults on Xen

["Roger Pau Monné" <roger.pau@citrix.com>]

On Fri, Oct 16, 2020 at 12:23:22PM -0400, Jason Andryuk wrote:
> On Thu, Oct 15, 2020 at 11:16 AM Jason Andryuk <jandryuk@gmail.com> wrote:
> >
> > On Thu, Oct 15, 2020 at 7:31 AM Roger Pau Monné <roger.pau@citrix.com> wrote:
> > >
> > > On Wed, Oct 14, 2020 at 08:37:06PM +0100, Andrew Cooper wrote:
> > > > On 14/10/2020 20:28, Jason Andryuk wrote:
> > > > > Hi,
> > > > >
> > > > > Bug opened at https://gitlab.freedesktop.org/drm/intel/-/issues/2576
> > > > >
> > > > > I'm seeing DMA faults for the i915 graphics hardware on a Dell
> > > > > Latitude 5500. These were captured when I plugged into a Dell
> > > > > Thunderbolt dock with two DisplayPort monitors attached.  Xen 4.12.4
> > > > > staging and Linux 5.4.70 (and some earlier versions).
> > > > >
> > > > > Oct 14 18:41:49.056490 kernel:[   85.570347] [drm:gen8_de_irq_handler
> > > > > [i915]] *ERROR* Fault errors on pipe A: 0x00000080
> > > > > Oct 14 18:41:49.056494 kernel:[   85.570395] [drm:gen8_de_irq_handler
> > > > > [i915]] *ERROR* Fault errors on pipe A: 0x00000080
> > > > > Oct 14 18:41:49.056589 VM hypervisor: (XEN) [VT-D]DMAR:[DMA Read]
> > > > > Request device [0000:00:02.0] fault addr 39b5845000, iommu reg =
> > > > > ffff82c00021d000
> > > > > Oct 14 18:41:49.056594 VM hypervisor: (XEN) [VT-D]DMAR: reason 06 -
> > > > > PTE Read access is not set
> > > > > Oct 14 18:41:49.056784 kernel:[   85.570668] [drm:gen8_de_irq_handler
> > > > > [i915]] *ERROR* Fault errors on pipe A: 0x00000080
> > > > > Oct 14 18:41:49.056789 kernel:[   85.570687] [drm:gen8_de_irq_handler
> > > > > [i915]] *ERROR* Fault errors on pipe A: 0x00000080
> > > > > Oct 14 18:41:49.056885 VM hypervisor: (XEN) [VT-D]DMAR:[DMA Read]
> > > > > Request device [0000:00:02.0] fault addr 4238d0a000, iommu reg =
> > > > > ffff82c00021d000
> > > > > Oct 14 18:41:49.056890 VM hypervisor: (XEN) [VT-D]DMAR: reason 06 -
> > > > > PTE Read access is not set
> > > > >
> > > > > They repeat. In the log attached to
> > > > > https://gitlab.freedesktop.org/drm/intel/-/issues/2576, they start at
> > > > > "Oct 14 18:41:49.056589" and continue until I unplug the dock around
> > > > > "Oct 14 18:41:54.801802".
> > > > >
> > > > > I've also seen similar messages when attaching the laptop's HDMI port
> > > > > to a 4k monitor. The eDP display by itself seems okay.
> > > > >
> > > > > I tried Fedora 31 & 32 live images with intel_iommu=on, so no Xen, and
> > > > > didn't see any errors
> > > > >
> > > > > This is a kernel & xen log with drm.debug=0x1e. It also includes some
> > > > > application (glass) logging when it changes resolutions which seems to
> > > > > set off the DMA faults. 5500-igfx-messages-kern-xen-glass
> > > > >
> > > > > Running xen with iommu=no-igfx disables the iommu for the i915
> > > > > graphics and no faults are reported. However, that breaks some other
> > > > > devices (Dell Latitude 7200 and 5580) giving a black screen with:
> > > > >
> > > > > Oct 10 13:24:37.022117 kernel:[   14.884759] i915 0000:00:02.0: Failed
> > > > > to idle engines, declaring wedged!
> > > > > Oct 10 13:24:37.022118 kernel:[   14.964794] i915 0000:00:02.0: Failed
> > > > > to initialize GPU, declaring it wedged!
> > > > >
> > > > > Any suggestions welcome.
> > > >
> > > > Presumably this is with a PV dom0.  What are 39b5845000 and 4238d0a000
> > > > in the machine memory map?
> >
> > They are bogus?
> > End of RAM is 0x47c800000
> > Thats:
> > 0x047c800000
> > vs.
> > 0x39b5845000
> > 0x4238d0a000
> >
> > > > This smells like a missing RMRR in the ACPI tables.
> 
> The RMRRs are:
> (XEN) [VT-D]Host address width 39
> (XEN) [VT-D]found ACPI_DMAR_DRHD:
> (XEN) [VT-D]  dmaru->address = fed90000
> (XEN) [VT-D]drhd->address = fed90000 iommu->reg = ffff82c00021d000
> (XEN) [VT-D]cap = 1c0000c40660462 ecap = 19e2ff0505e
> (XEN) [VT-D] endpoint: 0000:00:02.0
> (XEN) [VT-D]found ACPI_DMAR_DRHD:
> (XEN) [VT-D]  dmaru->address = fed91000
> (XEN) [VT-D]drhd->address = fed91000 iommu->reg = ffff82c00021f000
> (XEN) [VT-D]cap = d2008c40660462 ecap = f050da
> (XEN) [VT-D] IOAPIC: 0000:00:1e.7
> (XEN) [VT-D] MSI HPET: 0000:00:1e.6
> (XEN) [VT-D]  flags: INCLUDE_ALL
> (XEN) [VT-D]found ACPI_DMAR_RMRR:
> (XEN) [VT-D] endpoint: 0000:00:14.0
> (XEN) [VT-D]dmar.c:615:   RMRR region: base_addr 78863000 end_addr 78882fff
> (XEN) [VT-D]found ACPI_DMAR_RMRR:
> (XEN) [VT-D] endpoint: 0000:00:02.0
> (XEN) [VT-D]dmar.c:615:   RMRR region: base_addr 7d000000 end_addr 7f7fffff
> (XEN) [VT-D]found ACPI_DMAR_RMRR:
> (XEN) [VT-D] endpoint: 0000:00:16.7
> (XEN) [VT-D]dmar.c:581:  Non-existent device (0000:00:16.7) is
> reported in RMRR (78907000, 78986fff)'s scope!
> (XEN) [VT-D]dmar.c:596:   Ignore the RMRR (78907000, 78986fff) due to

This is also part of a reserved region, so should be added to the
iommu page tables anyway regardless of this message.

> devices under its scope are not PCI discoverable!
> 
> > > I agree.
> > >
> > > Can you paste the memory map as printed by Xen when booting, and what
> > > command line are you using to boot Xen.
> >
> > So this is OpenXT, and it's booting EFI -> xen -> tboot -> xen
> >
> > There's the memory map
> > (XEN) TBOOT RAM map:
> > (XEN)  0000000000000000 - 0000000000060000 (usable)
> > (XEN)  0000000000060000 - 0000000000068000 (reserved)
> > (XEN)  0000000000068000 - 000000000009e000 (usable)
> > (XEN)  000000000009e000 - 000000000009f000 (reserved)
> > (XEN)  000000000009f000 - 00000000000a0000 (usable)
> > (XEN)  00000000000a0000 - 0000000000100000 (reserved)
> > (XEN)  0000000000100000 - 0000000040000000 (usable)
> > (XEN)  0000000040000000 - 0000000040400000 (reserved)
> > (XEN)  0000000040400000 - 000000007024b000 (usable)
> > (XEN)  000000007024b000 - 000000007024c000 (ACPI NVS)
> > (XEN)  000000007024c000 - 000000007024d000 (reserved)
> > (XEN)  000000007024d000 - 0000000077f19000 (usable)
> > (XEN)  0000000077f19000 - 0000000078987000 (reserved)
> > (XEN)  0000000078987000 - 0000000078a04000 (ACPI data)
> > (XEN)  0000000078a04000 - 0000000078ea3000 (ACPI NVS)
> > (XEN)  0000000078ea3000 - 000000007acff000 (reserved)
> > (XEN)  000000007acff000 - 000000007ad00000 (usable)
> > (XEN)  000000007ad00000 - 000000007f800000 (reserved)
> > (XEN)  00000000f0000000 - 00000000f8000000 (reserved)
> > (XEN)  00000000fe000000 - 00000000fe011000 (reserved)
> > (XEN)  00000000fec00000 - 00000000fec01000 (reserved)
> > (XEN)  00000000fee00000 - 00000000fee01000 (reserved)
> > (XEN)  00000000ff000000 - 0000000100000000 (reserved)
> > (XEN)  0000000100000000 - 000000047c800000 (usable)
> > (XEN) EFI memory map:
> > (XEN)  0000000000000-000000009dfff type=7 attr=000000000000000f
> > (XEN)  000000009e000-000000009efff type=0 attr=000000000000000f
> > (XEN)  000000009f000-000000009ffff type=3 attr=000000000000000f
> > (XEN)  0000000100000-000003fffffff type=7 attr=000000000000000f
> > (XEN)  0000040000000-00000403fffff type=0 attr=000000000000000f
> > (XEN)  0000040400000-000005e359fff type=7 attr=000000000000000f
> > (XEN)  000005e35a000-000005e399fff type=4 attr=000000000000000f
> > (XEN)  000005e39a000-000006a47dfff type=7 attr=000000000000000f
> > (XEN)  000006a47e000-000006c3eefff type=2 attr=000000000000000f
> > (XEN)  000006c3ef000-000006d5eefff type=1 attr=000000000000000f
> > (XEN)  000006d5ef000-000006d86cfff type=2 attr=000000000000000f
> > (XEN)  000006d86d000-000006d978fff type=1 attr=000000000000000f
> > (XEN)  000006d979000-000006dc7afff type=4 attr=000000000000000f
> > (XEN)  000006dc7b000-000006dc98fff type=3 attr=000000000000000f
> > (XEN)  000006dc99000-000006dcc7fff type=4 attr=000000000000000f
> > (XEN)  000006dcc8000-000006dccdfff type=3 attr=000000000000000f
> > (XEN)  000006dcce000-00000701a5fff type=4 attr=000000000000000f
> > (XEN)  00000701a6000-00000701c8fff type=3 attr=000000000000000f
> > (XEN)  00000701c9000-00000701edfff type=4 attr=000000000000000f
> > (XEN)  00000701ee000-0000070204fff type=3 attr=000000000000000f
> > (XEN)  0000070205000-000007022cfff type=4 attr=000000000000000f
> > (XEN)  000007022d000-000007024afff type=3 attr=000000000000000f
> > (XEN)  000007024b000-000007024bfff type=10 attr=000000000000000f
> > (XEN)  000007024c000-000007024cfff type=6 attr=800000000000000f
> > (XEN)  000007024d000-000007024dfff type=4 attr=000000000000000f
> > (XEN)  000007024e000-0000070282fff type=3 attr=000000000000000f
> > (XEN)  0000070283000-00000702c3fff type=4 attr=000000000000000f
> > (XEN)  00000702c4000-00000702c8fff type=3 attr=000000000000000f
> > (XEN)  00000702c9000-00000702defff type=4 attr=000000000000000f
> > (XEN)  00000702df000-0000070307fff type=3 attr=000000000000000f
> > (XEN)  0000070308000-0000070317fff type=4 attr=000000000000000f
> > (XEN)  0000070318000-0000070319fff type=3 attr=000000000000000f
> > (XEN)  000007031a000-0000070331fff type=4 attr=000000000000000f
> > (XEN)  0000070332000-0000070349fff type=3 attr=000000000000000f
> > (XEN)  000007034a000-0000070356fff type=2 attr=000000000000000f
> > (XEN)  0000070357000-0000070357fff type=7 attr=000000000000000f
> > (XEN)  0000070358000-0000070358fff type=2 attr=000000000000000f
> > (XEN)  0000070359000-0000076f3efff type=4 attr=000000000000000f
> > (XEN)  0000076f3f000-00000772affff type=7 attr=000000000000000f
> > (XEN)  00000772b0000-0000077f18fff type=3 attr=000000000000000f
> > (XEN)  0000077f19000-0000078986fff type=0 attr=000000000000000f
> > (XEN)  0000078987000-0000078a03fff type=9 attr=000000000000000f
> > (XEN)  0000078a04000-0000078ea2fff type=10 attr=000000000000000f
> > (XEN)  0000078ea3000-000007ab22fff type=6 attr=800000000000000f
> > (XEN)  000007ab23000-000007acfefff type=5 attr=800000000000000f
> > (XEN)  000007acff000-000007acfffff type=4 attr=000000000000000f
> > (XEN)  0000100000000-000047c7fffff type=7 attr=000000000000000f
> > (XEN)  00000000a0000-00000000fffff type=0 attr=0000000000000000
> > (XEN)  000007ad00000-000007adfffff type=0 attr=070000000000000f
> > (XEN)  000007ae00000-000007f7fffff type=0 attr=0000000000000000
> > (XEN)  00000f0000000-00000f7ffffff type=11 attr=800000000000100d
> > (XEN)  00000fe000000-00000fe010fff type=11 attr=8000000000000001
> > (XEN)  00000fec00000-00000fec00fff type=11 attr=8000000000000001
> > (XEN)  00000fee00000-00000fee00fff type=11 attr=8000000000000001
> > (XEN)  00000ff000000-00000ffffffff type=11 attr=800000000000100d
> >
> > Command line
> > console=com1 dom0_mem=min:420M,max:420M,420M efi=no-rs,attr=uc
> > com1=115200,8n1,pci mbi-video vga=current flask=enforcing loglvl=debug
> > guest_loglvl=debug smt=0 ucode=-1 bootscrub=1
> > argo=yes,mac-permissive=1 iommu=force,igfx
> >
> > iommu=force,igfx was to force igfx back on.  I added a dmi quirk to
> > set no-igfx on this platform as a temporary workaround.

I assume setting no-igfx fixed the issue and the card works fine in
that case?

> > > Have you tried adding dom0-iommu=map-inclusive to the Xen command
> > > line?
> 
> Still seeing faults with dom0-iommu=map-inclusive.  At a different
> address this time:
> Oct 16 15:58:05.110768 VM hypervisor: (XEN) [VT-D]DMAR:[DMA Read]
> Request device [0000:00:02.0] fault addr ea0c4f000, iommu reg = ffff

That's also past the end of RAM.

> 82c00021d000
> Oct 16 15:58:05.110774 VM hypervisor: (XEN) [VT-D]DMAR: reason 06 -
> PTE Read access is not set
> Oct 16 15:58:05.110777 VM hypervisor: (XEN) print_vtd_entries: iommu
> #0 dev 0000:00:02.0 gmfn ea0c4f
> Oct 16 15:58:05.110780 VM hypervisor: (XEN)     root_entry[00] = 46e129001
> Oct 16 15:58:05.110782 VM hypervisor: (XEN)     context[10] = 2_46e128001
> Oct 16 15:58:05.110785 VM hypervisor: (XEN)     l4[000] = 46e11b003
> Oct 16 15:58:05.110787 VM hypervisor: (XEN)     l3[03a] = 0
> Oct 16 15:58:05.110789 VM hypervisor: (XEN)     l3[03a] not present
> 
> The previous posting, the two faulting addresses repeated in pairs.
> Here it is only this one address repeating.
> 
> I plugged and unplugged and a different address was repeating with a
> few other random addresses with 1 or 2 faults.  Here is uniq -c output
> of the address and count pulled from the logs:
> 0x1ce9d6b000 2007
> 0x31b50d5000 1
> 0x1ce9d6b000 882
> 0x707741000 1
> 0x1ce9d6b000 1114
> 0x20d2099000 1
> 0x1ce9d6b000 3489
> 0xeb98eb000 1
> 0x1ce9d6b000 2430
> 0xeb98eb000 1
> 0x1ce9d6b000 1300
> 0x22f20bb000 1
> 0x1ce9d6b000 269
> 0x22f20bb000 1
> 0x1ce9d6b000 5091
> 0x6c99ec9000 1
> 0x1ce9d6b000 29
> 0xeb98eb000 1
> 0x1ce9d6b000 4599
> 0x6c99ec9000 1
> 0x1ce9d6b000 1989

Hm, it's hard to tell what's going on. My limited experience with
IOMMU faults on broken systems there's a small range that initially
triggers those, and then the device goes wonky and starts accessing a
whole load of invalid addresses.

You could try adding those manually using the rmrr Xen command line
option [0], maybe you can figure out which range(s) are missing?

Roger.

[0] https://xenbits.xen.org/docs/unstable/misc/xen-command-line.html#rmrr