IOMMU fault with IGD passthrough setup on XEN 4.8.0

IOMMU fault with IGD passthrough setup on XEN 4.8.0

[G.R.]

[-- Attachment #1.1: Type: text/plain, Size: 6428 bytes --]

Hi all,
I have a working IGD passthrough setup running for 4 years on XEN 4.3.2.
And it no longer works after I upgraded to XEN4.8.0 yesterday. Really need
suggestions this time.

My previous setup was built upon some local fixes in qemu-xen-traditional
(for vendor specific pci cap).
With the same set of patches, I'm seeing hanging Linux domU and XEN dmesg
flooded with IOMMU fault message in version 4.8.0.
I haven't got chance to try out with a stock build, but the chance of
relevance is pretty low to me.
It could be a security related change (like the 'rdm_policy=relaxed' config
change), but I really have no idea.
(BTW, I also tried the new qemu-upstream device model, but it doesn't work
either.)

Here are some relevant logs, please help comment what's going on here and
what's the next step of diagnose.
It appears that the fault address 0xcfxxxxxx falls within the host RMRR
region.
However, the hvmloader is setting up memory region starting from address
0xe0000000.
Is the hvmloader memory map relevant here?
Unfortunately the iommu.c does not provide detailed log on the mapping
except a simple 'd2:PCI: map 0000:00:02.0'

Thanks,
G.R.

Errors look like this:
(XEN) [VT-D]DMAR:[DMA Write] Request device [0000:00:02.0] fault addr
cfa57000, iommu reg = ffff82c000201000
(XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
(XEN) print_vtd_entries: iommu ffff8304152ec600 dev 0000:00:02.0 gmfn cfa57
(XEN)     root_entry = ffff8304152e9000
(XEN)     root_entry[0] = 251bcb001
(XEN)     context = ffff830251bcb000
(XEN)     context[10] = 1_2215f6001
(XEN)     l3 = ffff8302215f6000
(XEN)     l3_index = 3
(XEN)     l3[3] = 221476003
(XEN)     l2 = ffff830221476000
(XEN)     l2_index = 7d
(XEN)     l2[7d] = 0
(XEN)     l2[7d] not present

(XEN) [VT-D]DMAR:[DMA Write] Request device [0000:00:02.0] fault addr
cfa70000, iommu reg = ffff82c000201000
(XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
(XEN) print_vtd_entries: iommu ffff8304152ec600 dev 0000:00:02.0 gmfn cfa70
(XEN)     root_entry = ffff8304152e9000
(XEN)     root_entry[0] = 251bcb001
(XEN)     context = ffff830251bcb000
(XEN)     context[10] = 1_2215f6001
(XEN)     l3 = ffff8302215f6000
(XEN)     l3_index = 3
(XEN)     l3[3] = 221476003
(XEN)     l2 = ffff830221476000
(XEN)     l2_index = 7d
(XEN)     l2[7d] = 0
(XEN)     l2[7d] not present

>From xl dmesg:
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009d800 (usable)
(XEN)  000000000009d800 - 00000000000a0000 (reserved)
(XEN)  00000000000e0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 0000000020000000 (usable)
(XEN)  0000000020000000 - 0000000020200000 (reserved)
(XEN)  0000000020200000 - 0000000040004000 (usable)
(XEN)  0000000040004000 - 0000000040005000 (reserved)
(XEN)  0000000040005000 - 00000000cd0b9000 (usable)
(XEN)  00000000cd0b9000 - 00000000cd881000 (reserved)
(XEN)  00000000cd881000 - 00000000cd90d000 (usable)
(XEN)  00000000cd90d000 - 00000000cd9ae000 (ACPI NVS)
(XEN)  00000000cd9ae000 - 00000000ce180000 (reserved)
(XEN)  00000000ce180000 - 00000000ce181000 (usable)
(XEN)  00000000ce181000 - 00000000ce1c4000 (ACPI NVS)
(XEN)  00000000ce1c4000 - 00000000cec19000 (usable)
(XEN)  00000000cec19000 - 00000000ceff2000 (reserved)
(XEN)  00000000ceff2000 - 00000000cf000000 (usable)
(XEN)  00000000cf800000 - 00000000dfa00000 (reserved)
(XEN)  00000000f8000000 - 00000000fc000000 (reserved)
(XEN)  00000000fec00000 - 00000000fec01000 (reserved)
(XEN)  00000000fed00000 - 00000000fed04000 (reserved)
(XEN)  00000000fed1c000 - 00000000fed20000 (reserved)
(XEN)  00000000fee00000 - 00000000fee01000 (reserved)
(XEN)  00000000ff000000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 000000041f600000 (usable)

(XEN) [VT-D]Host address width 36
(XEN) [VT-D]found ACPI_DMAR_DRHD:
(XEN) [VT-D]  dmaru->address = fed90000
(XEN) [VT-D]drhd->address = fed90000 iommu->reg = ffff82c000201000
(XEN) [VT-D]cap = c0000020e60262 ecap = f0101a
(XEN) [VT-D] endpoint: 0000:00:02.0
(XEN) [VT-D]found ACPI_DMAR_DRHD:
(XEN) [VT-D]  dmaru->address = fed91000
(XEN) [VT-D]drhd->address = fed91000 iommu->reg = ffff82c000203000
(XEN) [VT-D]cap = c9008020660262 ecap = f0105a
(XEN) [VT-D] IOAPIC: 0000:f0:1f.0
(XEN) [VT-D] MSI HPET: 0000:f0:0f.0
(XEN) [VT-D]  flags: INCLUDE_ALL
(XEN) [VT-D]found ACPI_DMAR_RMRR:
(XEN) [VT-D] endpoint: 0000:00:1d.0
(XEN) [VT-D] endpoint: 0000:00:1a.0
(XEN) [VT-D] endpoint: 0000:00:14.0
(XEN) [VT-D]  RMRR region: base_addr cd7ea000 end_address cd814fff
(XEN) [VT-D]found ACPI_DMAR_RMRR:
(XEN) [VT-D] endpoint: 0000:00:02.0
(XEN) [VT-D]  RMRR region: base_addr cf800000 end_address df9fffff


For the failing domU (dom2):
(XEN) d2: bind: m_gsi=16 g_gsi=24 dev=00.00.2 intx=0
(XEN) [VT-D]d0:PCI: unmap 0000:00:02.0
(XEN) [VT-D]d2:PCI: map 0000:00:02.0
(XEN) d2: bind: m_gsi=22 g_gsi=36 dev=00.00.5 intx=0
(XEN) [VT-D]d0:PCIe: unmap 0000:00:1b.0
(XEN) [VT-D]d2:PCIe: map 0000:00:1b.0
(XEN) d2: bind: m_gsi=16 g_gsi=40 dev=00.00.6 intx=0
(XEN) [VT-D] It's risky to assign 0000:00:1a.0 with shared RMRR at cd7ea000
for Dom2.
(XEN) [VT-D]d0:PCI: unmap 0000:00:1a.0
(XEN) [VT-D]d2:PCI: map 0000:00:1a.0

(d2) Relocating guest memory for lowmem MMIO space enabled
(d2) PCI-ISA link 0 routed to IRQ5
(d2) PCI-ISA link 1 routed to IRQ10
(d2) PCI-ISA link 2 routed to IRQ11
(d2) PCI-ISA link 3 routed to IRQ5
(d2) pci dev 01:3 INTA->IRQ10
(d2) pci dev 02:0 INTA->IRQ11
(d2) pci dev 03:0 INTA->IRQ5
(d2) pci dev 04:0 INTA->IRQ5
(d2) pci dev 05:0 INTA->IRQ10
(d2) pci dev 06:0 INTA->IRQ11
(d2) RAM in high memory; setting high_mem resource base to 100000000
(d2) pci dev 02:0 bar 18 size 010000000: 0e000000c
(XEN) memory_map:add: dom2 gfn=e0000 mfn=e0000 nr=10000

(d2) BIOS map:
(d2)  f0000-fffff: Main BIOS
(d2) E820 table:
(d2)  [00]: 00000000:00000000 - 00000000:0009e000: RAM
(d2)  [01]: 00000000:0009e000 - 00000000:000a0000: RESERVED
(d2)  HOLE: 00000000:000a0000 - 00000000:000e0000
(d2)  [02]: 00000000:000e0000 - 00000000:00100000: RESERVED
(d2)  [03]: 00000000:00100000 - 00000000:2fc00000: RAM
(d2)  HOLE: 00000000:2fc00000 - 00000000:cd7ea000
(d2)  [04]: 00000000:cd7ea000 - 00000000:cd815000: RESERVED
(d2)  HOLE: 00000000:cd815000 - 00000000:fc000000
(d2)  [05]: 00000000:fc000000 - 00000000:fdffc000: RESERVED
(d2)  [06]: 00000000:fdffc000 - 00000000:fdfff000: NVS
(d2)  [07]: 00000000:fdfff000 - 00000001:00000000: RESERVED
(d2)  [08]: 00000001:00000000 - 00000001:00000000: RAM
(d2) Invoking ROMBIOS ...

[-- Attachment #1.2: Type: text/html, Size: 7277 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[Jan Beulich]

>>> On 16.01.17 at 10:25, <firemeteor@users.sourceforge.net> wrote:
> Here are some relevant logs, please help comment what's going on here and
> what's the next step of diagnose.
> It appears that the fault address 0xcfxxxxxx falls within the host RMRR
> region.

Might be a problem in the RMRR setup itself, when the guest gets
the device assigned. But I'm not sure, as you've provided only
fragments of the log, instead of the full one (allowing to see in
which order the messages got logged). In any event the addresses
are, as you say, properly within the device's RMRR range.

That RMRR setup has changed dramatically (from being basically
non-existent in the older versions), especially for USB devices (I
don't think I can conclude what type of device 0000:02:00.0 is).
There are messages logged with various failures in that process,
but some would be issued by debug hypervisors only. A good
first step (before possibly doing actual code instrumentation)
would therefore be to retry with a debug hypervisor, and post
the full log (huge amounts of trailing IOMMU fault messages may
of course be stripped as long as they're sufficiently similar, to
keep the overall log size manageable).

> However, the hvmloader is setting up memory region starting from address
> 0xe0000000.
> Is the hvmloader memory map relevant here?

No, it shouldn't be.

> Unfortunately the iommu.c does not provide detailed log on the mapping
> except a simple 'd2:PCI: map 0000:00:02.0'

If we made it so, it would become unreasonably verbose.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[G.R.]

[-- Attachment #1.1: Type: text/plain, Size: 4877 bytes --]

On Mon, Jan 16, 2017 at 8:37 PM, Jan Beulich <JBeulich@suse.com> wrote:

> >>> On 16.01.17 at 10:25, <firemeteor@users.sourceforge.net> wrote:
> > Here are some relevant logs, please help comment what's going on here and
> > what's the next step of diagnose.
> > It appears that the fault address 0xcfxxxxxx falls within the host RMRR
> > region.
>
> Might be a problem in the RMRR setup itself, when the guest gets
> the device assigned. But I'm not sure, as you've provided only
> fragments of the log, instead of the full one (allowing to see in
> which order the messages got logged). In any event the addresses
> are, as you say, properly within the device's RMRR range.
>
Thanks for your quick reply, Jan.
I meant to provide full log through third party service like pastebin but
my network at work just get it blocked.
Here it is: http://pastebin.com/RHVzhR6H
Note that the log here is before the fault issue shows up.
As I already mentioned, there are two domUs in the log and the suffering
one is dom2.

The fault log itself is really flooding. With a small 4MB ring buffer, I
wasn't able to capture how it begins.
>From what I can tell, some one is scanning through the region in a fixed
pace. (in general, with some ping-pong occasionally)
The content from print_vtd_entries if fairly stable. This is what I get
from 'sort|uniq -c' post-processing, after removing line with fault address:
   7219 (XEN)     context[10] = 1_2215f6001
   7219 (XEN)     context = ffff830251bcb000
   5259 (XEN)     l2[7d] = 0
   5259 (XEN)     l2[7d] not present
   1961 (XEN)     l2[7e] = 0
   1961 (XEN)     l2[7e] not present
   7219 (XEN)     l2 = ffff830221476000
   5258 (XEN)     l2_index = 7d
   1961 (XEN)     l2_index = 7e
   7219 (XEN)     l3[3] = 221476003
   7219 (XEN)     l3 = ffff8302215f6000
   7219 (XEN)     l3_index = 3
   7219 (XEN)     root_entry[0] = 251bcb001
   7219 (XEN)     root_entry = ffff8304152e9000
   7219 (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set

The fault address pattern could be found here: http://pastebin.com/rWWH3QUG
(Note that I dropped redundant columns to fit the size limitation...)

And here is a list of my host PCI devices:
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core
processor DRAM Controller (rev 09)
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd
Gen Core processor Graphics Controller (rev 09)
00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset
Family USB xHCI Host Controller (rev 04)
00:16.0 Communication controller: Intel Corporation 7 Series/C216 Chipset
Family MEI Controller #1 (rev 04)
00:1a.0 USB controller: Intel Corporation 7 Series/C216 Chipset Family USB
Enhanced Host Controller #2 (rev 04)
00:1b.0 Audio device: Intel Corporation 7 Series/C216 Chipset Family High
Definition Audio Controller (rev 04)
00:1c.0 PCI bridge: Intel Corporation 7 Series/C216 Chipset Family PCI
Express Root Port 1 (rev c4)
00:1c.3 PCI bridge: Intel Corporation 7 Series/C216 Chipset Family PCI
Express Root Port 4 (rev c4)
00:1d.0 USB controller: Intel Corporation 7 Series/C216 Chipset Family USB
Enhanced Host Controller #1 (rev 04)
00:1f.0 ISA bridge: Intel Corporation H77 Express Chipset LPC Controller
(rev 04)
00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset
Family 6-port SATA Controller [AHCI mode] (rev 04)
00:1f.3 SMBus: Intel Corporation 7 Series/C216 Chipset Family SMBus
Controller (rev 04)
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)

That RMRR setup has changed dramatically (from being basically
> non-existent in the older versions), especially for USB devices (I
> don't think I can conclude what type of device 0000:02:00.0 is).
> There are messages logged with various failures in that process,
> but some would be issued by debug hypervisors only. A good
> first step (before possibly doing actual code instrumentation)
> would therefore be to retry with a debug hypervisor, and post
> the full log (huge amounts of trailing IOMMU fault messages may
> of course be stripped as long as they're sufficiently similar, to
> keep the overall log size manageable).
>
I can give it a try when I get some spare time.
Could you show me the flow to build a debug hypervisor and the most
relevant debug knobs to avoid log flooding?


>
> > However, the hvmloader is setting up memory region starting from address
> > 0xe0000000.
> > Is the hvmloader memory map relevant here?
>
> No, it shouldn't be.
>
> > Unfortunately the iommu.c does not provide detailed log on the mapping
> > except a simple 'd2:PCI: map 0000:00:02.0'
>
> If we made it so, it would become unreasonably verbose.
>
> Jan
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
>

[-- Attachment #1.2: Type: text/html, Size: 6442 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[Jan Beulich]

>>> On 16.01.17 at 14:43, <firemeteor@users.sourceforge.net> wrote:
> On Mon, Jan 16, 2017 at 8:37 PM, Jan Beulich <JBeulich@suse.com> wrote:
>> >>> On 16.01.17 at 10:25, <firemeteor@users.sourceforge.net> wrote:
>> > Here are some relevant logs, please help comment what's going on here and
>> > what's the next step of diagnose.
>> > It appears that the fault address 0xcfxxxxxx falls within the host RMRR
>> > region.
>>
>> Might be a problem in the RMRR setup itself, when the guest gets
>> the device assigned. But I'm not sure, as you've provided only
>> fragments of the log, instead of the full one (allowing to see in
>> which order the messages got logged). In any event the addresses
>> are, as you say, properly within the device's RMRR range.
>>
> Thanks for your quick reply, Jan.
> I meant to provide full log through third party service like pastebin but
> my network at work just get it blocked.
> Here it is: http://pastebin.com/RHVzhR6H 
> Note that the log here is before the fault issue shows up.

Well, it stopping before the first fault occurs makes it only
marginally bette than what you've provided so far.

> The fault log itself is really flooding. With a small 4MB ring buffer, I
> wasn't able to capture how it begins.

If you can't set up a serial console, grow the ring buffer.

> That RMRR setup has changed dramatically (from being basically
>> non-existent in the older versions), especially for USB devices (I
>> don't think I can conclude what type of device 0000:02:00.0 is).
>> There are messages logged with various failures in that process,
>> but some would be issued by debug hypervisors only. A good
>> first step (before possibly doing actual code instrumentation)
>> would therefore be to retry with a debug hypervisor, and post
>> the full log (huge amounts of trailing IOMMU fault messages may
>> of course be stripped as long as they're sufficiently similar, to
>> keep the overall log size manageable).
>>
> I can give it a try when I get some spare time.
> Could you show me the flow to build a debug hypervisor and the most
> relevant debug knobs to avoid log flooding?

For building a debug hypervisor, all you need to do is set
CONFIG_DEBUG=y in xen/.config. I don't think there are any
knobs to avoid log flooding - after all you've asked for the
verbosity via "iommu=verbose,debug".

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[George Dunlap]

On Mon, Jan 16, 2017 at 1:43 PM, G.R. <firemeteor@users.sourceforge.net> wrote:
> On Mon, Jan 16, 2017 at 8:37 PM, Jan Beulich <JBeulich@suse.com> wrote:
>>
>> >>> On 16.01.17 at 10:25, <firemeteor@users.sourceforge.net> wrote:
>> > Here are some relevant logs, please help comment what's going on here
>> > and
>> > what's the next step of diagnose.
>> > It appears that the fault address 0xcfxxxxxx falls within the host RMRR
>> > region.
>>
>> Might be a problem in the RMRR setup itself, when the guest gets
>> the device assigned. But I'm not sure, as you've provided only
>> fragments of the log, instead of the full one (allowing to see in
>> which order the messages got logged). In any event the addresses
>> are, as you say, properly within the device's RMRR range.
>
> Thanks for your quick reply, Jan.
> I meant to provide full log through third party service like pastebin but my
> network at work just get it blocked.
> Here it is: http://pastebin.com/RHVzhR6H

FWIW you can also attach the log as a text file.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[G.R.]

[-- Attachment #1.1: Type: text/plain, Size: 1915 bytes --]

On Mon, Jan 16, 2017 at 9:56 PM, Jan Beulich <JBeulich@suse.com> wrote:

> >>> On 16.01.17 at 14:43, <firemeteor@users.sourceforge.net> wrote:
> > On Mon, Jan 16, 2017 at 8:37 PM, Jan Beulich <JBeulich@suse.com> wrote:
> >> >>> On 16.01.17 at 10:25, <firemeteor@users.sourceforge.net> wrote:
> > The fault log itself is really flooding. With a small 4MB ring buffer, I
> > wasn't able to capture how it begins.
>
> If you can't set up a serial console, grow the ring buffer.
>
Larger ring buffer seems to be the only option to me.
Seems that 'serial console' needs to be something physical.


> > That RMRR setup has changed dramatically (from being basically
> >> non-existent in the older versions), especially for USB devices (I
> >> don't think I can conclude what type of device 0000:02:00.0 is).
> >> There are messages logged with various failures in that process,
> >> but some would be issued by debug hypervisors only. A good
> >> first step (before possibly doing actual code instrumentation)
> >> would therefore be to retry with a debug hypervisor, and post
> >> the full log (huge amounts of trailing IOMMU fault messages may
> >> of course be stripped as long as they're sufficiently similar, to
> >> keep the overall log size manageable).
> >>
> > I can give it a try when I get some spare time.
> > Could you show me the flow to build a debug hypervisor and the most
> > relevant debug knobs to avoid log flooding?
>
> For building a debug hypervisor, all you need to do is set
> CONFIG_DEBUG=y in xen/.config. I don't think there are any
> knobs to avoid log flooding - after all you've asked for the
> verbosity via "iommu=verbose,debug".
>
I assume I do not need to redo the ./configure here.
And I assume the xen/.config here refers to the root of the repos instead
of the xen.git/xen subdirectory?
I couldn't find obvious debug knob in the gcc command-line, even though the
build is with -O1.

[-- Attachment #1.2: Type: text/html, Size: 2958 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[G.R.]

[-- Attachment #1.1: Type: text/plain, Size: 3247 bytes --]

BTW, before I generate more verbose && complete debug log, just want to
update that I also see the following in dom0 (without attempting any
pass-through to the IGD device)
But this time the log is not flooding at all. Not sure if this is relevant
to what I see from the domU with pci pass-through.

(XEN) Bogus DMIBAR 0xfed18001 on 0000:00:00.0
(XEN) [VT-D]DMAR:[DMA Write] Request device [0000:00:02.0] fault addr
7300000000, iommu reg = ffff82c000201000
(XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
(XEN) [VT-D]DMAR:[DMA Write] Request device [0000:00:02.0] fault addr
7300000000, iommu reg = ffff82c000201000
(XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
(XEN) [VT-D]DMAR:[DMA Write] Request device [0000:00:02.0] fault addr
7300000000, iommu reg = ffff82c000201000
(XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
(XEN) [VT-D]DMAR:[DMA Write] Request device [0000:00:02.0] fault addr
7300000000, iommu reg = ffff82c000201000
(XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
(XEN) [VT-D]DMAR:[DMA Write] Request device [0000:00:02.0] fault addr
7300000000, iommu reg = ffff82c000201000
(XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set


On Mon, Jan 16, 2017 at 11:15 PM, G.R. <firemeteor@users.sourceforge.net>
wrote:

>
>
> On Mon, Jan 16, 2017 at 9:56 PM, Jan Beulich <JBeulich@suse.com> wrote:
>
>> >>> On 16.01.17 at 14:43, <firemeteor@users.sourceforge.net> wrote:
>> > On Mon, Jan 16, 2017 at 8:37 PM, Jan Beulich <JBeulich@suse.com> wrote:
>> >> >>> On 16.01.17 at 10:25, <firemeteor@users.sourceforge.net> wrote:
>> > The fault log itself is really flooding. With a small 4MB ring buffer, I
>> > wasn't able to capture how it begins.
>>
>> If you can't set up a serial console, grow the ring buffer.
>>
> Larger ring buffer seems to be the only option to me.
> Seems that 'serial console' needs to be something physical.
>
>
>> > That RMRR setup has changed dramatically (from being basically
>> >> non-existent in the older versions), especially for USB devices (I
>> >> don't think I can conclude what type of device 0000:02:00.0 is).
>> >> There are messages logged with various failures in that process,
>> >> but some would be issued by debug hypervisors only. A good
>> >> first step (before possibly doing actual code instrumentation)
>> >> would therefore be to retry with a debug hypervisor, and post
>> >> the full log (huge amounts of trailing IOMMU fault messages may
>> >> of course be stripped as long as they're sufficiently similar, to
>> >> keep the overall log size manageable).
>> >>
>> > I can give it a try when I get some spare time.
>> > Could you show me the flow to build a debug hypervisor and the most
>> > relevant debug knobs to avoid log flooding?
>>
>> For building a debug hypervisor, all you need to do is set
>> CONFIG_DEBUG=y in xen/.config. I don't think there are any
>> knobs to avoid log flooding - after all you've asked for the
>> verbosity via "iommu=verbose,debug".
>>
> I assume I do not need to redo the ./configure here.
> And I assume the xen/.config here refers to the root of the repos instead
> of the xen.git/xen subdirectory?
> I couldn't find obvious debug knob in the gcc command-line, even though
> the build is with -O1.
>
>
>

[-- Attachment #1.2: Type: text/html, Size: 4726 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[Jan Beulich]

>>> On 16.01.17 at 16:15, <firemeteor@users.sourceforge.net> wrote:
> On Mon, Jan 16, 2017 at 9:56 PM, Jan Beulich <JBeulich@suse.com> wrote:
> 
>> >>> On 16.01.17 at 14:43, <firemeteor@users.sourceforge.net> wrote:
>> > On Mon, Jan 16, 2017 at 8:37 PM, Jan Beulich <JBeulich@suse.com> wrote:
>> >> >>> On 16.01.17 at 10:25, <firemeteor@users.sourceforge.net> wrote:
>> > The fault log itself is really flooding. With a small 4MB ring buffer, I
>> > wasn't able to capture how it begins.
>>
>> If you can't set up a serial console, grow the ring buffer.
>>
> Larger ring buffer seems to be the only option to me.
> Seems that 'serial console' needs to be something physical.

Indeed.

>> > That RMRR setup has changed dramatically (from being basically
>> >> non-existent in the older versions), especially for USB devices (I
>> >> don't think I can conclude what type of device 0000:02:00.0 is).
>> >> There are messages logged with various failures in that process,
>> >> but some would be issued by debug hypervisors only. A good
>> >> first step (before possibly doing actual code instrumentation)
>> >> would therefore be to retry with a debug hypervisor, and post
>> >> the full log (huge amounts of trailing IOMMU fault messages may
>> >> of course be stripped as long as they're sufficiently similar, to
>> >> keep the overall log size manageable).
>> >>
>> > I can give it a try when I get some spare time.
>> > Could you show me the flow to build a debug hypervisor and the most
>> > relevant debug knobs to avoid log flooding?
>>
>> For building a debug hypervisor, all you need to do is set
>> CONFIG_DEBUG=y in xen/.config. I don't think there are any
>> knobs to avoid log flooding - after all you've asked for the
>> verbosity via "iommu=verbose,debug".
>>
> I assume I do not need to redo the ./configure here.
> And I assume the xen/.config here refers to the root of the repos instead
> of the xen.git/xen subdirectory?

I don't understand - I'd normally assume the two to be the same
(with just different context made visible).

> I couldn't find obvious debug knob in the gcc command-line, even though the
> build is with -O1.

Nor do I understand this remark.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[Jan Beulich]

>>> On 16.01.17 at 16:21, <firemeteor@users.sourceforge.net> wrote:
> BTW, before I generate more verbose && complete debug log, just want to
> update that I also see the following in dom0 (without attempting any
> pass-through to the IGD device)
> But this time the log is not flooding at all. Not sure if this is relevant
> to what I see from the domU with pci pass-through.
> 
> (XEN) Bogus DMIBAR 0xfed18001 on 0000:00:00.0
> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:00:02.0] fault addr
> 7300000000, iommu reg = ffff82c000201000
> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:00:02.0] fault addr
> 7300000000, iommu reg = ffff82c000201000
> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:00:02.0] fault addr
> 7300000000, iommu reg = ffff82c000201000
> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:00:02.0] fault addr
> 7300000000, iommu reg = ffff82c000201000
> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:00:02.0] fault addr
> 7300000000, iommu reg = ffff82c000201000
> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set

This address is clearly neither RAM nor covered by any RMRR, so
the question is why there is I/O being generated to it. Is it perhaps
named by one of the device's BARs?

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[G.R.]

[-- Attachment #1.1: Type: text/plain, Size: 1535 bytes --]

On Tue, Jan 17, 2017 at 12:11 AM, Jan Beulich <JBeulich@suse.com> wrote:

> >>> On 16.01.17 at 16:15, <firemeteor@users.sourceforge.net> wrote:
> > On Mon, Jan 16, 2017 at 9:56 PM, Jan Beulich <JBeulich@suse.com> wrote:
> >
> >> For building a debug hypervisor, all you need to do is set
> >> CONFIG_DEBUG=y in xen/.config. I don't think there are any
> >> knobs to avoid log flooding - after all you've asked for the
> >> verbosity via "iommu=verbose,debug".
> >>
> > I assume I do not need to redo the ./configure here.
> > And I assume the xen/.config here refers to the root of the repos instead
> > of the xen.git/xen subdirectory?
>
> I don't understand - I'd normally assume the two to be the same
> (with just different context made visible).
>
I was trying to figure out if I followed your instruction properly.
My first attempt only resulted in a binary with similar size with my
previous one.
Probably something went wrong.
I put my source under /nas/src/xen, and I have a /nas/src/xen/.config file
for the
python layout knob according to the wiki.
My first attempt put th CONFIG_DEBUG=y line in the same file.
But now I suspect if I should use /nas/src/xen/xen/.config (note the double
'xen').

> I couldn't find obvious debug knob in the gcc command-line, even though
> the
> > build is with -O1.
>
> Nor do I understand this remark.
>
I checked the GCC command-lines during build process with the .config
change, with the expectation of something like -DDEBUG -g etc.
But actually I saw none of them, only saw -O1.

G.R.

[-- Attachment #1.2: Type: text/html, Size: 2532 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[Jan Beulich]

>>> On 17.01.17 at 11:49, <firemeteor@users.sourceforge.net> wrote:
> On Tue, Jan 17, 2017 at 12:11 AM, Jan Beulich <JBeulich@suse.com> wrote:
> 
>> >>> On 16.01.17 at 16:15, <firemeteor@users.sourceforge.net> wrote:
>> > On Mon, Jan 16, 2017 at 9:56 PM, Jan Beulich <JBeulich@suse.com> wrote:
>> >
>> >> For building a debug hypervisor, all you need to do is set
>> >> CONFIG_DEBUG=y in xen/.config. I don't think there are any
>> >> knobs to avoid log flooding - after all you've asked for the
>> >> verbosity via "iommu=verbose,debug".
>> >>
>> > I assume I do not need to redo the ./configure here.
>> > And I assume the xen/.config here refers to the root of the repos instead
>> > of the xen.git/xen subdirectory?
>>
>> I don't understand - I'd normally assume the two to be the same
>> (with just different context made visible).
>>
> I was trying to figure out if I followed your instruction properly.
> My first attempt only resulted in a binary with similar size with my
> previous one.
> Probably something went wrong.
> I put my source under /nas/src/xen, and I have a /nas/src/xen/.config file
> for the
> python layout knob according to the wiki.
> My first attempt put th CONFIG_DEBUG=y line in the same file.
> But now I suspect if I should use /nas/src/xen/xen/.config (note the double
> 'xen').

Yes indeed, that's the one. And you shouldn't add a new line, but
instead edit the existing one (with CONFIG_DEBUG commented out).

>> I couldn't find obvious debug knob in the gcc command-line, even though
>> the
>> > build is with -O1.
>>
>> Nor do I understand this remark.
>>
> I checked the GCC command-lines during build process with the .config
> change, with the expectation of something like -DDEBUG -g etc.
> But actually I saw none of them, only saw -O1.

The primary thing would be for -DNDEBUG to no longer be there (it
should be there for the non-debug build).

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[G.R.]

[-- Attachment #1.1: Type: text/plain, Size: 1189 bytes --]

On Tue, Jan 17, 2017 at 8:54 PM, Jan Beulich <JBeulich@suse.com> wrote:

> >>> On 17.01.17 at 11:49, <firemeteor@users.sourceforge.net> wrote:
> > I was trying to figure out if I followed your instruction properly.
> > My first attempt only resulted in a binary with similar size with my
> > previous one.
> > Probably something went wrong.
> > I put my source under /nas/src/xen, and I have a /nas/src/xen/.config
> file
> > for the
> > python layout knob according to the wiki.
> > My first attempt put th CONFIG_DEBUG=y line in the same file.
> > But now I suspect if I should use /nas/src/xen/xen/.config (note the
> double
> > 'xen').
>
> Yes indeed, that's the one. And you shouldn't add a new line, but
> instead edit the existing one (with CONFIG_DEBUG commented out).
>

Hi Jan, I think debug build works this time.
For unknown reason, debug version of hypervisor will cause domU hang if
gfx-passthrough=1 is present (traditional device model).
But fortunately commenting out that line could still reproduce the IOMMU
fault.
I was lucky to capture the full log before it fills up my 100MB ring buffer
(in less than 2 seconds).
Please find the log in the attachment.

Thanks,
G.R.

[-- Attachment #1.2: Type: text/html, Size: 1930 bytes --]

[-- Attachment #2: dmsg.log.bz2 --]
[-- Type: application/x-bzip2, Size: 29948 bytes --]

[-- Attachment #3: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[Jan Beulich]

>>> On 17.01.17 at 16:08, <firemeteor@users.sourceforge.net> wrote:
> I was lucky to capture the full log before it fills up my 100MB ring buffer
> (in less than 2 seconds).
> Please find the log in the attachment.

Sadly nothing helpful in there; I'm a little puzzled though that the
first thing we see is

(XEN) [VT-D]iommu.c:909: iommu_fault_status: Fault Overflow

which suggests there were (unlogged) faults already before.

My primary suspicion right now is that you problem is due to the
relatively large RMRR, as the first logged fault occurs on the first
2Mb boundary after the start of the RMRR. I'll therefore have to
find time to create a debugging patch for you.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[G.R.]

[-- Attachment #1.1: Type: text/plain, Size: 926 bytes --]

On Wed, Jan 18, 2017 at 12:34 AM, Jan Beulich <JBeulich@suse.com> wrote:

> >>> On 17.01.17 at 16:08, <firemeteor@users.sourceforge.net> wrote:
> > I was lucky to capture the full log before it fills up my 100MB ring
> buffer
> > (in less than 2 seconds).
> > Please find the log in the attachment.
>
> Sadly nothing helpful in there; I'm a little puzzled though that the
> first thing we see is
>
> (XEN) [VT-D]iommu.c:909: iommu_fault_status: Fault Overflow
>
> which suggests there were (unlogged) faults already before.
>
> My primary suspicion right now is that you problem is due to the
> relatively large RMRR, as the first logged fault occurs on the first
> 2Mb boundary after the start of the RMRR. I'll therefore have to
> find time to create a debugging patch for you.
>
That's unfortunate! But anyway we stepped one small step ahead.
Waiting for your patch. I'll be offline for 3 days, will check back after
that.

[-- Attachment #1.2: Type: text/html, Size: 1472 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[Jan Beulich]

[-- Attachment #1: Type: text/plain, Size: 437 bytes --]

>>> On 17.01.17 at 16:08, <firemeteor@users.sourceforge.net> wrote:
> But fortunately commenting out that line could still reproduce the IOMMU
> fault.
> I was lucky to capture the full log before it fills up my 100MB ring buffer
> (in less than 2 seconds).

So here's a first take at a debugging patch. I've tried to limit existing
output, so that you'd have better chance of again capturing all
interesting messages.

Jan


[-- Attachment #2: RMRR-mapping.patch --]
[-- Type: text/plain, Size: 5255 bytes --]

--- unstable.orig/xen/drivers/passthrough/vtd/iommu.c	2017-01-03 10:55:55.000000000 +0100
+++ unstable/xen/drivers/passthrough/vtd/iommu.c	2017-01-19 17:26:48.000000000 +0100
@@ -897,8 +897,23 @@ static int iommu_page_fault_do_one(struc
            kind, fault_reason, reason);
 
     if ( iommu_verbose && fault_type == DMA_REMAP )
+{//temp
+ static domid_t last;
+ static unsigned long cnt, thr;
+ const struct pci_dev*pdev;
+ pcidevs_lock();
+ pdev = pci_get_real_pdev(seg, PCI_BUS(source_id), PCI_DEVFN2(source_id));
+ if(pdev && pdev->domain && pdev->domain->domain_id > last) {
+  thr = cnt = 0;
+  last = pdev->domain->domain_id;
+ }
+ pcidevs_unlock();
+ if(++cnt > thr) {
+  thr |= cnt;
         print_vtd_entries(iommu, PCI_BUS(source_id), PCI_DEVFN2(source_id),
                           addr >> PAGE_SHIFT);
+ }
+}
 
     return 0;
 }
@@ -1890,6 +1905,7 @@ static void iommu_set_pgd(struct domain
 
 static int rmrr_identity_mapping(struct domain *d, bool_t map,
                                  const struct acpi_rmrr_unit *rmrr,
+u16 bdf,//temp
                                  u32 flag)
 {
     unsigned long base_pfn = rmrr->base_address >> PAGE_SHIFT_4K;
@@ -1914,6 +1930,7 @@ static int rmrr_identity_mapping(struct
             if ( map )
             {
                 ++mrmrr->count;
+printk(XENLOG_GUEST "d%d: RMRR [%lx,%lx] count %u\n", d->domain_id, base_pfn, end_pfn, mrmrr->count);//temp
                 return 0;
             }
 
@@ -1928,6 +1945,7 @@ static int rmrr_identity_mapping(struct
             }
 
             list_del(&mrmrr->list);
+printk(XENLOG_GUEST "d%d: RMRR [%lx,%lx] zapped (%d)\n", d->domain_id, mrmrr->base >> PAGE_SHIFT_4K, end_pfn, ret);//temp
             xfree(mrmrr);
             return ret;
         }
@@ -1941,11 +1959,30 @@ static int rmrr_identity_mapping(struct
         int err = set_identity_p2m_entry(d, base_pfn, p2m_access_rw, flag);
 
         if ( err )
+{//temp
+ printk(XENLOG_GUEST "d%d: RMRR [%lx,%lx] map error %d @ %lx\n", d->domain_id, rmrr->base_address >> PAGE_SHIFT_4K, end_pfn, err, base_pfn);
             return err;
+} else {
+ static domid_t last;
+ static unsigned long cnt, thr;
+ if(d->domain_id > last) {
+  thr = cnt = 0;
+  last = d->domain_id;
+ }
+ if(!(base_pfn & 0xff) && ++cnt > thr) {
+  const struct pci_dev*pdev = pci_get_pdev(rmrr->segment, PCI_BUS(bdf), PCI_DEVFN2(bdf));
+  const struct acpi_drhd_unit*drhd = pdev ? acpi_find_matched_drhd_unit(pdev) : NULL;
+  thr |= cnt;
+  printk(XENLOG_GUEST "d%d: RMRR [%lx,%lx] mapped %lx\n", d->domain_id, rmrr->base_address >> PAGE_SHIFT_4K, end_pfn, base_pfn);
+  if(drhd)
+   print_vtd_entries(drhd->iommu, PCI_BUS(bdf), PCI_DEVFN2(bdf), base_pfn);
+ }
+}
         base_pfn++;
     }
 
     mrmrr = xmalloc(struct mapped_rmrr);
+printk(XENLOG_GUEST "d%d: RMRR [%lx,%lx] alloc -> %p\n", d->domain_id, rmrr->base_address >> PAGE_SHIFT_4K, end_pfn, mrmrr);//temp
     if ( !mrmrr )
         return -ENOMEM;
     mrmrr->base = rmrr->base_address;
@@ -1987,7 +2024,7 @@ static int intel_iommu_add_device(u8 dev
              * Since RMRRs are always reserved in the e820 map for the hardware
              * domain, there shouldn't be a conflict.
              */
-            ret = rmrr_identity_mapping(pdev->domain, 1, rmrr, 0);
+            ret = rmrr_identity_mapping(pdev->domain, 1, rmrr, bdf, 0);
             if ( ret )
                 dprintk(XENLOG_ERR VTDPREFIX, "d%d: RMRR mapping failed\n",
                         pdev->domain->domain_id);
@@ -2032,7 +2069,7 @@ static int intel_iommu_remove_device(u8
          * Any flag is nothing to clear these mappings but here
          * its always safe and strict to set 0.
          */
-        rmrr_identity_mapping(pdev->domain, 0, rmrr, 0);
+        rmrr_identity_mapping(pdev->domain, 0, rmrr, bdf, 0);
     }
 
     return domain_context_unmap(pdev->domain, devfn, pdev);
@@ -2199,7 +2236,7 @@ static void __hwdom_init setup_hwdom_rmr
          * domain, there shouldn't be a conflict. So its always safe and
          * strict to set 0.
          */
-        ret = rmrr_identity_mapping(d, 1, rmrr, 0);
+        ret = rmrr_identity_mapping(d, 1, rmrr, bdf, 0);
         if ( ret )
             dprintk(XENLOG_ERR VTDPREFIX,
                      "IOMMU: mapping reserved region failed\n");
@@ -2356,7 +2393,7 @@ static int reassign_device_ownership(
                  * Any RMRR flag is always ignored when remove a device,
                  * but its always safe and strict to set 0.
                  */
-                ret = rmrr_identity_mapping(source, 0, rmrr, 0);
+                ret = rmrr_identity_mapping(source, 0, rmrr, bdf, 0);
                 if ( ret != -ENOENT )
                     return ret;
             }
@@ -2446,7 +2483,7 @@ static int intel_iommu_assign_device(
              PCI_BUS(bdf) == bus &&
              PCI_DEVFN2(bdf) == devfn )
         {
-            ret = rmrr_identity_mapping(d, 1, rmrr, flag);
+            ret = rmrr_identity_mapping(d, 1, rmrr, bdf, flag);
             if ( ret )
             {
                 reassign_device_ownership(d, hardware_domain, devfn, pdev);

[-- Attachment #3: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[G.R.]

[-- Attachment #1: Type: text/plain, Size: 2517 bytes --]

On Fri, Jan 20, 2017 at 12:30 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 17.01.17 at 16:08, <firemeteor@users.sourceforge.net> wrote:
>> But fortunately commenting out that line could still reproduce the IOMMU
>> fault.
>> I was lucky to capture the full log before it fills up my 100MB ring buffer
>> (in less than 2 seconds).
>
> So here's a first take at a debugging patch. I've tried to limit existing
> output, so that you'd have better chance of again capturing all
> interesting messages.
>

Hi Jan,
I finally get some spare time to collect the debug info.
Please find the full log in the attachment.

The mapping appears to be working:
(XEN) d8: RMRR [cf800,dfa00] mapped cf800
(XEN) d8: RMRR [cf800,dfa00] mapped cf900
(XEN) d8: RMRR [cf800,dfa00] mapped cfb00
(XEN) d8: RMRR [cf800,dfa00] mapped cff00
(XEN) d8: RMRR [cf800,dfa00] mapped d0700
(XEN) d8: RMRR [cf800,dfa00] mapped d1700
(XEN) d8: RMRR [cf800,dfa00] mapped d3700
(XEN) d8: RMRR [cf800,dfa00] mapped d7700
(XEN) d8: RMRR [cf800,dfa00] mapped df700
(XEN) d8: RMRR [cf800,dfa00] alloc -> ffff83013156ffb0

But I'm not sure if the vtd_entries look correct: (Is the 'not
present' line okay?)
(XEN) d8: RMRR [cf800,dfa00] mapped cfb00
(XEN) print_vtd_entries: iommu ffff8304152ec600 dev 0000:00:02.0 gmfn cfb00
(XEN)     root_entry = ffff820040056000
(XEN)     root_entry[0] = 201fc6001
(XEN)     context = ffff820040020000
(XEN)     context[10] = 1_13956c001
(XEN)     l3 = ffff820040022000
(XEN)     l3_index = 3
(XEN)     l3[3] = 1394ec003
(XEN)     l2 = ffff820040023000
(XEN)     l2_index = 7d
(XEN)     l2[7d] = 0
(XEN)     l2[7d] not present

Still see the 'Fault overflow' line in the very first fault.
The fault is about write-access not permitted.
Is the map read-only here? Or are we looking at the correct PTE?
(XEN) [VT-D]iommu.c:924: iommu_fault_status: Fault Overflow
(XEN) [VT-D]iommu.c:926: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]DMAR:[DMA Write] Request device [0000:00:02.0] fault addr
cfa00000, iommu reg = ffff82c000201000
(XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
(XEN) print_vtd_entries: iommu ffff8304152ec600 dev 0000:00:02.0 gmfn cfa00
(XEN)     root_entry = ffff8304152e9000
(XEN)     root_entry[0] = 201fc6001
(XEN)     context = ffff830201fc6000
(XEN)     context[10] = 1_13956c001
(XEN)     l3 = ffff83013956c000
(XEN)     l3_index = 3
(XEN)     l3[3] = 1394ec003
(XEN)     l2 = ffff8301394ec000
(XEN)     l2_index = 7d
(XEN)     l2[7d] = 0
(XEN)     l2[7d] not present

> Jan
>

[-- Attachment #2: dmesg.xz --]
[-- Type: application/x-xz, Size: 6520 bytes --]

[-- Attachment #3: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[G.R.]

[-- Attachment #1: Type: text/plain, Size: 3425 bytes --]

On Sun, Feb 5, 2017 at 1:51 PM, G.R. <firemeteor@users.sourceforge.net> wrote:
> On Fri, Jan 20, 2017 at 12:30 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>> On 17.01.17 at 16:08, <firemeteor@users.sourceforge.net> wrote:
>>> But fortunately commenting out that line could still reproduce the IOMMU
>>> fault.
>>> I was lucky to capture the full log before it fills up my 100MB ring buffer
>>> (in less than 2 seconds).
>>
>> So here's a first take at a debugging patch. I've tried to limit existing
>> output, so that you'd have better chance of again capturing all
>> interesting messages.
>>
>
> Hi Jan,
> I finally get some spare time to collect the debug info.
> Please find the full log in the attachment.
>
> The mapping appears to be working:
> (XEN) d8: RMRR [cf800,dfa00] mapped cf800
> (XEN) d8: RMRR [cf800,dfa00] mapped cf900
> (XEN) d8: RMRR [cf800,dfa00] mapped cfb00
> (XEN) d8: RMRR [cf800,dfa00] mapped cff00
> (XEN) d8: RMRR [cf800,dfa00] mapped d0700
> (XEN) d8: RMRR [cf800,dfa00] mapped d1700
> (XEN) d8: RMRR [cf800,dfa00] mapped d3700
> (XEN) d8: RMRR [cf800,dfa00] mapped d7700
> (XEN) d8: RMRR [cf800,dfa00] mapped df700
> (XEN) d8: RMRR [cf800,dfa00] alloc -> ffff83013156ffb0
>
> But I'm not sure if the vtd_entries look correct: (Is the 'not
> present' line okay?)
> (XEN) d8: RMRR [cf800,dfa00] mapped cfb00
> (XEN) print_vtd_entries: iommu ffff8304152ec600 dev 0000:00:02.0 gmfn cfb00
> (XEN)     root_entry = ffff820040056000
> (XEN)     root_entry[0] = 201fc6001
> (XEN)     context = ffff820040020000
> (XEN)     context[10] = 1_13956c001
> (XEN)     l3 = ffff820040022000
> (XEN)     l3_index = 3
> (XEN)     l3[3] = 1394ec003
> (XEN)     l2 = ffff820040023000
> (XEN)     l2_index = 7d
> (XEN)     l2[7d] = 0
> (XEN)     l2[7d] not present
>
> Still see the 'Fault overflow' line in the very first fault.
> The fault is about write-access not permitted.
> Is the map read-only here? Or are we looking at the correct PTE?
> (XEN) [VT-D]iommu.c:924: iommu_fault_status: Fault Overflow
> (XEN) [VT-D]iommu.c:926: iommu_fault_status: Primary Pending Fault
> (XEN) [VT-D]DMAR:[DMA Write] Request device [0000:00:02.0] fault addr
> cfa00000, iommu reg = ffff82c000201000
> (XEN) [VT-D]DMAR: reason 05 - PTE Write access is not set
> (XEN) print_vtd_entries: iommu ffff8304152ec600 dev 0000:00:02.0 gmfn cfa00
> (XEN)     root_entry = ffff8304152e9000
> (XEN)     root_entry[0] = 201fc6001
> (XEN)     context = ffff830201fc6000
> (XEN)     context[10] = 1_13956c001
> (XEN)     l3 = ffff83013956c000
> (XEN)     l3_index = 3
> (XEN)     l3[3] = 1394ec003
> (XEN)     l2 = ffff8301394ec000
> (XEN)     l2_index = 7d
> (XEN)     l2[7d] = 0
> (XEN)     l2[7d] not present
>

Attraching an xl dmesg log for dom0, which shows a more reasonable vtd_entry.
Does it mean that the mapping wasn't properly setup in the domU case?
But we didn't see a map error in debug log either.

(XEN) d0: RMRR [cf800,dfa00] mapped cfb00
(XEN) print_vtd_entries: iommu ffff8304152ec600 dev 0000:00:02.0 gmfn cfb00
(XEN)     root_entry = ffff8304152e9000
(XEN)     root_entry[0] = 2030ca001
(XEN)     context = ffff8302030ca000
(XEN)     context[10] = 1_2032d1001
(XEN)     l3 = ffff8302032d1000
(XEN)     l3_index = 3
(XEN)     l3[3] = 2030c7003
(XEN)     l2 = ffff8302030c7000
(XEN)     l2_index = 7d
(XEN)     l2[7d] = 2030c5003
(XEN)     l1 = ffff8302030c5000
(XEN)     l1_index = 100
(XEN)     l1[100] = cfb00003

[-- Attachment #2: dom0.xz --]
[-- Type: application/x-xz, Size: 6432 bytes --]

[-- Attachment #3: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[Jan Beulich]

>>> On 05.02.17 at 08:18, <firemeteor@users.sourceforge.net> wrote:
> On Sun, Feb 5, 2017 at 1:51 PM, G.R. <firemeteor@users.sourceforge.net> wrote:
>> On Fri, Jan 20, 2017 at 12:30 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>> On 17.01.17 at 16:08, <firemeteor@users.sourceforge.net> wrote:
>>>> But fortunately commenting out that line could still reproduce the IOMMU
>>>> fault.
>>>> I was lucky to capture the full log before it fills up my 100MB ring buffer
>>>> (in less than 2 seconds).
>>>
>>> So here's a first take at a debugging patch. I've tried to limit existing
>>> output, so that you'd have better chance of again capturing all
>>> interesting messages.
>>>
>>
>> Hi Jan,
>> I finally get some spare time to collect the debug info.
>> Please find the full log in the attachment.
>>
>> The mapping appears to be working:
>> (XEN) d8: RMRR [cf800,dfa00] mapped cf800
>> (XEN) d8: RMRR [cf800,dfa00] mapped cf900
>> (XEN) d8: RMRR [cf800,dfa00] mapped cfb00
>> (XEN) d8: RMRR [cf800,dfa00] mapped cff00
>> (XEN) d8: RMRR [cf800,dfa00] mapped d0700
>> (XEN) d8: RMRR [cf800,dfa00] mapped d1700
>> (XEN) d8: RMRR [cf800,dfa00] mapped d3700
>> (XEN) d8: RMRR [cf800,dfa00] mapped d7700
>> (XEN) d8: RMRR [cf800,dfa00] mapped df700
>> (XEN) d8: RMRR [cf800,dfa00] alloc -> ffff83013156ffb0
>>
>> But I'm not sure if the vtd_entries look correct: (Is the 'not
>> present' line okay?)

Rather not, hence ...

>> Still see the 'Fault overflow' line in the very first fault.
>> The fault is about write-access not permitted.

... this.

>> Is the map read-only here?

No, as you've noticed above, it's simply not present.

>> Or are we looking at the correct PTE?

Comparing the two dumps, I think we do.

> Attraching an xl dmesg log for dom0, which shows a more reasonable vtd_entry.
> Does it mean that the mapping wasn't properly setup in the domU case?

Looks like so.

> But we didn't see a map error in debug log either.

I'll have to look into this more closely.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[G.R.]

On Mon, Feb 6, 2017 at 3:40 PM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 05.02.17 at 08:18, <firemeteor@users.sourceforge.net> wrote:
>> But we didn't see a map error in debug log either.
>
> I'll have to look into this more closely.

Let me know when you need more info / debug log.:-)

BTW, if this helps my hardware setup is based on i7-3770 + ASRock H77M-iTX.
I'm not sure if this chip is officially VT-D supported but I've been
using this for ~3 years with SATA && IGD passthrough.
So I assume this is not an HW issue. (at least for now)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[Jan Beulich]

>>> On 05.02.17 at 06:51, <firemeteor@users.sourceforge.net> wrote:
> Please find the full log in the attachment.

Sadly that one is only a partial log again. I'd really need to see the
boot messages too, in particular to (hopefully) be able to judge
whether your system uses shared or separate EPT and VT-d tables.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[Jan Beulich]

[-- Attachment #1: Type: text/plain, Size: 449 bytes --]

>>> On 05.02.17 at 06:51, <firemeteor@users.sourceforge.net> wrote:
> I finally get some spare time to collect the debug info.

As I continue to be puzzled, best I could come up with is an
extension to the debug patch. Please use the attached one
in place of the earlier one, ideally on top of
https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00448.html
to reduce the overall amount of output (and help readability).

Jan


[-- Attachment #2: RMRR-mapping.patch --]
[-- Type: text/plain, Size: 6618 bytes --]

--- unstable.orig/xen/arch/x86/mm/p2m.c	2016-11-09 15:17:11.000000000 +0100
+++ unstable/xen/arch/x86/mm/p2m.c	2017-02-06 13:32:14.000000000 +0100
@@ -1048,10 +1048,37 @@ int set_identity_p2m_entry(struct domain
     gfn_lock(p2m, gfn, 0);
 
     mfn = p2m->get_entry(p2m, gfn, &p2mt, &a, 0, NULL, NULL);
+if(!(gfn & 0xff)) {//temp
+ static domid_t last;
+ static unsigned long cnt, thr;
+ if(d->domain_id > last) {
+  thr = cnt = 0;
+  last = d->domain_id;
+ }
+ if(++cnt > thr) {
+  thr |= cnt;
+  printk("d%d: GFN %"PRI_gfn" -> %"PRI_mfn":%d:%d\n", d->domain_id, gfn, mfn, p2mt, a);
+ }
+}
 
     if ( p2mt == p2m_invalid || p2mt == p2m_mmio_dm )
+{//temp
         ret = p2m_set_entry(p2m, gfn, _mfn(gfn), PAGE_ORDER_4K,
                             p2m_mmio_direct, p2ma);
+ if(ret || !(gfn & 0xff)) {//temp
+  static domid_t last;
+  static unsigned long cnt, thr;
+  if(d->domain_id > last) {
+   thr = cnt = 0;
+   last = d->domain_id;
+  }
+  if(ret || ++cnt > thr) {
+   thr |= cnt;
+   mfn = p2m->get_entry(p2m, gfn, &p2mt, &a, 0, NULL, NULL);
+   printk("d%d: GFN %"PRI_gfn" -> %"PRI_mfn":%d:%d (%d)\n", d->domain_id, gfn, mfn, p2mt, a, ret);
+  }
+ }
+}
     else if ( mfn_x(mfn) == gfn && p2mt == p2m_mmio_direct && a == p2ma )
     {
         ret = 0;
--- unstable.orig/xen/drivers/passthrough/vtd/iommu.c	2017-01-19 17:27:05.000000000 +0100
+++ unstable/xen/drivers/passthrough/vtd/iommu.c	2017-02-06 13:21:42.000000000 +0100
@@ -897,8 +897,23 @@ static int iommu_page_fault_do_one(struc
            kind, fault_reason, reason);
 
     if ( iommu_verbose && fault_type == DMA_REMAP )
+{//temp
+ static domid_t last;
+ static unsigned long cnt, thr;
+ const struct pci_dev*pdev;
+ pcidevs_lock();
+ pdev = pci_get_real_pdev(seg, PCI_BUS(source_id), PCI_DEVFN2(source_id));
+ if(pdev && pdev->domain && pdev->domain->domain_id > last) {
+  thr = cnt = 0;
+  last = pdev->domain->domain_id;
+ }
+ pcidevs_unlock();
+ if(++cnt > thr) {
+  thr |= cnt;
         print_vtd_entries(iommu, PCI_BUS(source_id), PCI_DEVFN2(source_id),
                           addr >> PAGE_SHIFT);
+ }
+}
 
     return 0;
 }
@@ -1890,6 +1905,7 @@ static void iommu_set_pgd(struct domain
 
 static int rmrr_identity_mapping(struct domain *d, bool_t map,
                                  const struct acpi_rmrr_unit *rmrr,
+u16 bdf,//temp
                                  u32 flag)
 {
     unsigned long base_pfn = rmrr->base_address >> PAGE_SHIFT_4K;
@@ -1914,6 +1930,7 @@ static int rmrr_identity_mapping(struct
             if ( map )
             {
                 ++mrmrr->count;
+printk(XENLOG_GUEST "d%d: RMRR [%lx,%lx] count %u\n", d->domain_id, base_pfn, end_pfn, mrmrr->count);//temp
                 return 0;
             }
 
@@ -1928,6 +1945,7 @@ static int rmrr_identity_mapping(struct
             }
 
             list_del(&mrmrr->list);
+printk(XENLOG_GUEST "d%d: RMRR [%lx,%lx] zapped (%d)\n", d->domain_id, mrmrr->base >> PAGE_SHIFT_4K, end_pfn, ret);//temp
             xfree(mrmrr);
             return ret;
         }
@@ -1941,11 +1959,32 @@ static int rmrr_identity_mapping(struct
         int err = set_identity_p2m_entry(d, base_pfn, p2m_access_rw, flag);
 
         if ( err )
+{//temp
+ printk(XENLOG_GUEST "d%d: RMRR [%lx,%lx] map error %d @ %lx\n", d->domain_id, rmrr->base_address >> PAGE_SHIFT_4K, end_pfn, err, base_pfn);
             return err;
+} else {
+ static domid_t last;
+ static unsigned long cnt, thr;
+ if(d->domain_id > last) {
+  thr = cnt = 0;
+  last = d->domain_id;
+ }
+ if(!(base_pfn & 0xff) && ++cnt > thr) {
+  const struct pci_dev*pdev = pci_get_pdev(rmrr->segment, PCI_BUS(bdf), PCI_DEVFN2(bdf));
+  const struct acpi_drhd_unit*drhd = pdev ? acpi_find_matched_drhd_unit(pdev) : NULL;
+  thr |= cnt;
+  printk(XENLOG_GUEST "d%d: RMRR [%lx,%lx] mapped %lx\n", d->domain_id, rmrr->base_address >> PAGE_SHIFT_4K, end_pfn, base_pfn);
+  if(drhd)
+   print_vtd_entries(drhd->iommu, PCI_BUS(bdf), PCI_DEVFN2(bdf), base_pfn);
+  if(hap_enabled(d))
+   ept_walk_table(d, base_pfn);
+ }
+}
         base_pfn++;
     }
 
     mrmrr = xmalloc(struct mapped_rmrr);
+printk(XENLOG_GUEST "d%d: RMRR [%lx,%lx] alloc -> %p\n", d->domain_id, rmrr->base_address >> PAGE_SHIFT_4K, end_pfn, mrmrr);//temp
     if ( !mrmrr )
         return -ENOMEM;
     mrmrr->base = rmrr->base_address;
@@ -1987,7 +2026,7 @@ static int intel_iommu_add_device(u8 dev
              * Since RMRRs are always reserved in the e820 map for the hardware
              * domain, there shouldn't be a conflict.
              */
-            ret = rmrr_identity_mapping(pdev->domain, 1, rmrr, 0);
+            ret = rmrr_identity_mapping(pdev->domain, 1, rmrr, bdf, 0);
             if ( ret )
                 dprintk(XENLOG_ERR VTDPREFIX, "d%d: RMRR mapping failed\n",
                         pdev->domain->domain_id);
@@ -2032,7 +2071,7 @@ static int intel_iommu_remove_device(u8
          * Any flag is nothing to clear these mappings but here
          * its always safe and strict to set 0.
          */
-        rmrr_identity_mapping(pdev->domain, 0, rmrr, 0);
+        rmrr_identity_mapping(pdev->domain, 0, rmrr, bdf, 0);
     }
 
     return domain_context_unmap(pdev->domain, devfn, pdev);
@@ -2199,7 +2238,7 @@ static void __hwdom_init setup_hwdom_rmr
          * domain, there shouldn't be a conflict. So its always safe and
          * strict to set 0.
          */
-        ret = rmrr_identity_mapping(d, 1, rmrr, 0);
+        ret = rmrr_identity_mapping(d, 1, rmrr, bdf, 0);
         if ( ret )
             dprintk(XENLOG_ERR VTDPREFIX,
                      "IOMMU: mapping reserved region failed\n");
@@ -2356,7 +2395,7 @@ static int reassign_device_ownership(
                  * Any RMRR flag is always ignored when remove a device,
                  * but its always safe and strict to set 0.
                  */
-                ret = rmrr_identity_mapping(source, 0, rmrr, 0);
+                ret = rmrr_identity_mapping(source, 0, rmrr, bdf, 0);
                 if ( ret != -ENOENT )
                     return ret;
             }
@@ -2446,7 +2485,7 @@ static int intel_iommu_assign_device(
              PCI_BUS(bdf) == bus &&
              PCI_DEVFN2(bdf) == devfn )
         {
-            ret = rmrr_identity_mapping(d, 1, rmrr, flag);
+            ret = rmrr_identity_mapping(d, 1, rmrr, bdf, flag);
             if ( ret )
             {
                 reassign_device_ownership(d, hardware_domain, devfn, pdev);

[-- Attachment #3: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[G.R.]

On Mon, Feb 6, 2017 at 7:43 PM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 05.02.17 at 06:51, <firemeteor@users.sourceforge.net> wrote:
>> Please find the full log in the attachment.
>
> Sadly that one is only a partial log again. I'd really need to see the
> boot messages too, in particular to (hopefully) be able to judge
> whether your system uses shared or separate EPT and VT-d tables.
>
In the dom0.xz attachment (the second one on Feb 5th), the xl dmesg
info from boot stage is retained.
If that's not good enough, please instruct on how to generate the desired log.

Quote some log snippets here, please find the full log in the old attachment:
It appears that the system uses separated EPT && VT-d tables.
Is this good or bad?

(XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
(XEN) Intel VT-d iommu 1 supported page sizes: 4kB.
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Posted Interrupt not enabled.
(XEN) Intel VT-d Shared EPT tables not enabled. <== EPT && VT-d
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed

> Jan
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[Jan Beulich]

>>> On 07.02.17 at 03:12, <firemeteor@users.sourceforge.net> wrote:
> On Mon, Feb 6, 2017 at 7:43 PM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>> On 05.02.17 at 06:51, <firemeteor@users.sourceforge.net> wrote:
>>> Please find the full log in the attachment.
>>
>> Sadly that one is only a partial log again. I'd really need to see the
>> boot messages too, in particular to (hopefully) be able to judge
>> whether your system uses shared or separate EPT and VT-d tables.
>>
> In the dom0.xz attachment (the second one on Feb 5th), the xl dmesg
> info from boot stage is retained.
> If that's not good enough, please instruct on how to generate the desired 
> log.

That's good enough; sorry for not having checked there.

> Quote some log snippets here, please find the full log in the old 
> attachment:
> It appears that the system uses separated EPT && VT-d tables.
> Is this good or bad?

Neither - it is merely relevant. In particular (and the new debug
patch may help understand this) it may mean that while EPT
tables get updated, IOMMU page tables may not be (even if the
code suggests this is being taken care of).

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[G.R.]

[-- Attachment #1: Type: text/plain, Size: 540 bytes --]

On Mon, Feb 6, 2017 at 8:40 PM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 05.02.17 at 06:51, <firemeteor@users.sourceforge.net> wrote:
>> I finally get some spare time to collect the debug info.
>
> As I continue to be puzzled, best I could come up with is an
> extension to the debug patch. Please use the attached one
> in place of the earlier one, ideally on top of
> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00448.html
> to reduce the overall amount of output (and help readability).

Please see attached...

[-- Attachment #2: dmsg2.xz --]
[-- Type: application/x-xz, Size: 49748 bytes --]

[-- Attachment #3: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[Jan Beulich]

>>> On 07.02.17 at 16:44, <firemeteor@users.sourceforge.net> wrote:
> On Mon, Feb 6, 2017 at 8:40 PM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>> On 05.02.17 at 06:51, <firemeteor@users.sourceforge.net> wrote:
>>> I finally get some spare time to collect the debug info.
>>
>> As I continue to be puzzled, best I could come up with is an
>> extension to the debug patch. Please use the attached one
>> in place of the earlier one, ideally on top of
>> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00448.html 
>> to reduce the overall amount of output (and help readability).
> 
> Please see attached...

Thanks, this confirms my suspicion that EPT tables get updated, but
IOMMU ones don't. I'll have to go over the code yet another time to
see if I can spot what's wrong, or whether I need to further extend
the debugging patch.

Are you using a PVH Dom0 here, btw? If so, is the problem
connected to that? The fact that things work there may be
attributable to the mappings already being there even before the
identity mapping calls.

I further note that there's an apparently legitimate IOMMU fault
before the first guest gets started. This may indicate an issue
with the Dom0 kernel.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[Jan Beulich]

>>> On 07.02.17 at 16:44, <firemeteor@users.sourceforge.net> wrote:
> On Mon, Feb 6, 2017 at 8:40 PM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>> On 05.02.17 at 06:51, <firemeteor@users.sourceforge.net> wrote:
>>> I finally get some spare time to collect the debug info.
>>
>> As I continue to be puzzled, best I could come up with is an
>> extension to the debug patch. Please use the attached one
>> in place of the earlier one, ideally on top of
>> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00448.html 
>> to reduce the overall amount of output (and help readability).
> 
> Please see attached...

Okay, that's because p2m_get_iommu_flags() returns zero for
p2m_mmio_direct, which iirc we already have a patch floating
around for (needed for PVHv2). Roger - what's the status of
that? Would it make sense to split this out of your series, to get
in rather sooner?

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[Jan Beulich]

>>> On 07.02.17 at 16:44, <firemeteor@users.sourceforge.net> wrote:
> On Mon, Feb 6, 2017 at 8:40 PM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>> On 05.02.17 at 06:51, <firemeteor@users.sourceforge.net> wrote:
>>> I finally get some spare time to collect the debug info.
>>
>> As I continue to be puzzled, best I could come up with is an
>> extension to the debug patch. Please use the attached one
>> in place of the earlier one, ideally on top of
>> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00448.html 
>> to reduce the overall amount of output (and help readability).
> 
> Please see attached...

So can you please give
https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00602.html
a try?

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[G.R.]

On Wed, Feb 8, 2017 at 8:44 PM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 07.02.17 at 16:44, <firemeteor@users.sourceforge.net> wrote:
>> On Mon, Feb 6, 2017 at 8:40 PM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>> On 05.02.17 at 06:51, <firemeteor@users.sourceforge.net> wrote:
>>>> I finally get some spare time to collect the debug info.
>>>
>>> As I continue to be puzzled, best I could come up with is an
>>> extension to the debug patch. Please use the attached one
>>> in place of the earlier one, ideally on top of
>>> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00448.html
>>> to reduce the overall amount of output (and help readability).
>>
>> Please see attached...
>
> So can you please give
> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00602.html
> a try?
>
Hmm, it does not help.
But I'll need to double check if I was misleading you.
I used attempt dom0pvh=1 but it was too unstable and I was only able
to disable it through hacking grub.cfg through sshfs remotely.
I forgot to touch the /etc/default/grub so the dom0pvh=1 may have come
back when I was generating the log yesterday.

Going to do it once again now.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[G.R.]

[-- Attachment #1: Type: text/plain, Size: 1609 bytes --]

On Wed, Feb 8, 2017 at 10:29 PM, G.R. <firemeteor@users.sourceforge.net> wrote:
> On Wed, Feb 8, 2017 at 8:44 PM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>> On 07.02.17 at 16:44, <firemeteor@users.sourceforge.net> wrote:
>>> On Mon, Feb 6, 2017 at 8:40 PM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>> On 05.02.17 at 06:51, <firemeteor@users.sourceforge.net> wrote:
>>>>> I finally get some spare time to collect the debug info.
>>>>
>>>> As I continue to be puzzled, best I could come up with is an
>>>> extension to the debug patch. Please use the attached one
>>>> in place of the earlier one, ideally on top of
>>>> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00448.html
>>>> to reduce the overall amount of output (and help readability).
>>>
>>> Please see attached...
>>
>> So can you please give
>> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00602.html
>> a try?
>>
> Hmm, it does not help.
> But I'll need to double check if I was misleading you.
> I used attempt dom0pvh=1 but it was too unstable and I was only able
> to disable it through hacking grub.cfg through sshfs remotely.
> I forgot to touch the /etc/default/grub so the dom0pvh=1 may have come
> back when I was generating the log yesterday.
>
> Going to do it once again now.

It appears that dom0pvh or not does not affect the debug output
without Roger's patch.
Anyway, attaching the output for you to double check.

However, dom0pvh does make different with Roger's patch.
With dom0pvh=1 + Roger's patch, the same fault message previously
observed for domU is now also showing up for dom0.

[-- Attachment #2: rmrr_dbg_dom0pvh_off.xz --]
[-- Type: application/x-xz, Size: 41020 bytes --]

[-- Attachment #3: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[Jan Beulich]

>>> On 08.02.17 at 15:56, <firemeteor@users.sourceforge.net> wrote:
> On Wed, Feb 8, 2017 at 10:29 PM, G.R. <firemeteor@users.sourceforge.net> wrote:
>> On Wed, Feb 8, 2017 at 8:44 PM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>> On 07.02.17 at 16:44, <firemeteor@users.sourceforge.net> wrote:
>>>> On Mon, Feb 6, 2017 at 8:40 PM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>>> On 05.02.17 at 06:51, <firemeteor@users.sourceforge.net> wrote:
>>>>>> I finally get some spare time to collect the debug info.
>>>>>
>>>>> As I continue to be puzzled, best I could come up with is an
>>>>> extension to the debug patch. Please use the attached one
>>>>> in place of the earlier one, ideally on top of
>>>>> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00448.html 
>>>>> to reduce the overall amount of output (and help readability).
>>>>
>>>> Please see attached...
>>>
>>> So can you please give
>>> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00602.html 
>>> a try?
>>>
>> Hmm, it does not help.
>> But I'll need to double check if I was misleading you.
>> I used attempt dom0pvh=1 but it was too unstable and I was only able
>> to disable it through hacking grub.cfg through sshfs remotely.
>> I forgot to touch the /etc/default/grub so the dom0pvh=1 may have come
>> back when I was generating the log yesterday.
>>
>> Going to do it once again now.
> 
> It appears that dom0pvh or not does not affect the debug output
> without Roger's patch.
> Anyway, attaching the output for you to double check.

Well, if this indeed was with his patch in place, then I'm puzzled.
I'd have to further extend the debugging patch then, but this may
take a few days to get to.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[G.R.]

On Wed, Feb 8, 2017 at 11:59 PM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 08.02.17 at 15:56, <firemeteor@users.sourceforge.net> wrote:
>> On Wed, Feb 8, 2017 at 10:29 PM, G.R. <firemeteor@users.sourceforge.net> wrote:
>>> On Wed, Feb 8, 2017 at 8:44 PM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>> On 07.02.17 at 16:44, <firemeteor@users.sourceforge.net> wrote:
>>>>> On Mon, Feb 6, 2017 at 8:40 PM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>>>> On 05.02.17 at 06:51, <firemeteor@users.sourceforge.net> wrote:
>>>>>>> I finally get some spare time to collect the debug info.
>>>>>>
>>>>>> As I continue to be puzzled, best I could come up with is an
>>>>>> extension to the debug patch. Please use the attached one
>>>>>> in place of the earlier one, ideally on top of
>>>>>> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00448.html
>>>>>> to reduce the overall amount of output (and help readability).
>>>>>
>>>>> Please see attached...
>>>>
>>>> So can you please give
>>>> https://lists.xenproject.org/archives/html/xen-devel/2017-02/msg00602.html
>>>> a try?
>>>>
>>> Hmm, it does not help.
>>> But I'll need to double check if I was misleading you.
>>> I used attempt dom0pvh=1 but it was too unstable and I was only able
>>> to disable it through hacking grub.cfg through sshfs remotely.
>>> I forgot to touch the /etc/default/grub so the dom0pvh=1 may have come
>>> back when I was generating the log yesterday.
>>>
>>> Going to do it once again now.
>>
>> It appears that dom0pvh or not does not affect the debug output
>> without Roger's patch.
>> Anyway, attaching the output for you to double check.
>
> Well, if this indeed was with his patch in place, then I'm puzzled.
> I'd have to further extend the debugging patch then, but this may
> take a few days to get to.

Please hold-off and let me double check for you. I'm also confused by
my current situation right now.
I think should be running without Roger's fix with dom0pvh=0.
But I happen to see a lot of fault message right now, from dom0.
Maybe I forgot to reboot last night after reverting back.
I'll build with Roger's fix again and do the experiment once more.
BTW -- I think that fix should not be conflicting with your debug change, right?

Thanks,
Rui

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[Jan Beulich]

>>> On 09.02.17 at 15:46, <firemeteor@users.sourceforge.net> wrote:
> BTW -- I think that fix should not be conflicting with your debug change, 
> right?

Yes - ideally you'd keep that one in place along with adding Roger's
patch.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[Roger Pau Monné]

On Thu, Feb 09, 2017 at 07:58:56AM -0700, Jan Beulich wrote:
> >>> On 09.02.17 at 15:46, <firemeteor@users.sourceforge.net> wrote:
> > BTW -- I think that fix should not be conflicting with your debug change, 
> > right?
> 
> Yes - ideally you'd keep that one in place along with adding Roger's
> patch.

Please use the patch below, the previous one was missing a break, which made it
completely useless, sorry.

Roger.

---8<---
From 27d234b34b95f9b5900dd0760a0f16161459bcdd Mon Sep 17 00:00:00 2001
From: Roger Pau Monne <roger.pau@citrix.com>
Date: Thu, 9 Feb 2017 17:59:32 +0000
Subject: [PATCH] x86/iommu: add IOMMU entries for p2m_mmio_direct pages
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

There's nothing wrong with allowing the domain to perform DMA transfers to
MMIO areas that it already can access from the CPU, and this allows us to
remove the hack in set_identity_p2m_entry for PVH Dom0.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
 xen/arch/x86/mm/p2m-ept.c |  5 +++--
 xen/arch/x86/mm/p2m-pt.c  | 17 ++++++++++-------
 xen/arch/x86/mm/p2m.c     |  9 ---------
 xen/include/asm-x86/p2m.h |  7 ++++++-
 4 files changed, 19 insertions(+), 19 deletions(-)

diff --git a/xen/arch/x86/mm/p2m-ept.c b/xen/arch/x86/mm/p2m-ept.c
index 04878f5..f47f323 100644
--- a/xen/arch/x86/mm/p2m-ept.c
+++ b/xen/arch/x86/mm/p2m-ept.c
@@ -672,7 +672,7 @@ ept_set_entry(struct p2m_domain *p2m, unsigned long gfn, mfn_t mfn,
     uint8_t ipat = 0;
     bool_t need_modify_vtd_table = 1;
     bool_t vtd_pte_present = 0;
-    unsigned int iommu_flags = p2m_get_iommu_flags(p2mt);
+    unsigned int iommu_flags = p2m_get_iommu_flags(p2mt, mfn);
     bool_t needs_sync = 1;
     ept_entry_t old_entry = { .epte = 0 };
     ept_entry_t new_entry = { .epte = 0 };
@@ -798,7 +798,8 @@ ept_set_entry(struct p2m_domain *p2m, unsigned long gfn, mfn_t mfn,
 
         /* Safe to read-then-write because we hold the p2m lock */
         if ( ept_entry->mfn == new_entry.mfn &&
-             p2m_get_iommu_flags(ept_entry->sa_p2mt) == iommu_flags )
+             p2m_get_iommu_flags(ept_entry->sa_p2mt, _mfn(ept_entry->mfn)) ==
+             iommu_flags )
             need_modify_vtd_table = 0;
 
         ept_p2m_type_to_flags(p2m, &new_entry, p2mt, p2ma);
diff --git a/xen/arch/x86/mm/p2m-pt.c b/xen/arch/x86/mm/p2m-pt.c
index 3b025d5..a23d0bd 100644
--- a/xen/arch/x86/mm/p2m-pt.c
+++ b/xen/arch/x86/mm/p2m-pt.c
@@ -499,7 +499,7 @@ p2m_pt_set_entry(struct p2m_domain *p2m, unsigned long gfn, mfn_t mfn,
     l2_pgentry_t l2e_content;
     l3_pgentry_t l3e_content;
     int rc;
-    unsigned int iommu_pte_flags = p2m_get_iommu_flags(p2mt);
+    unsigned int iommu_pte_flags = p2m_get_iommu_flags(p2mt, mfn);
     /*
      * old_mfn and iommu_old_flags control possible flush/update needs on the
      * IOMMU: We need to flush when MFN or flags (i.e. permissions) change.
@@ -565,9 +565,10 @@ p2m_pt_set_entry(struct p2m_domain *p2m, unsigned long gfn, mfn_t mfn,
         {
             if ( flags & _PAGE_PSE )
             {
-                iommu_old_flags =
-                    p2m_get_iommu_flags(p2m_flags_to_type(flags));
                 old_mfn = l1e_get_pfn(*p2m_entry);
+                iommu_old_flags =
+                    p2m_get_iommu_flags(p2m_flags_to_type(flags),
+                                        _mfn(old_mfn));
             }
             else
             {
@@ -609,9 +610,10 @@ p2m_pt_set_entry(struct p2m_domain *p2m, unsigned long gfn, mfn_t mfn,
         p2m_entry = p2m_find_entry(table, &gfn_remainder, gfn,
                                    0, L1_PAGETABLE_ENTRIES);
         ASSERT(p2m_entry);
-        iommu_old_flags =
-            p2m_get_iommu_flags(p2m_flags_to_type(l1e_get_flags(*p2m_entry)));
         old_mfn = l1e_get_pfn(*p2m_entry);
+        iommu_old_flags =
+            p2m_get_iommu_flags(p2m_flags_to_type(l1e_get_flags(*p2m_entry)),
+                                _mfn(old_mfn));
 
         if ( mfn_valid(mfn) || p2m_allows_invalid_mfn(p2mt) )
             entry_content = p2m_l1e_from_pfn(mfn_x(mfn),
@@ -637,9 +639,10 @@ p2m_pt_set_entry(struct p2m_domain *p2m, unsigned long gfn, mfn_t mfn,
         {
             if ( flags & _PAGE_PSE )
             {
-                iommu_old_flags =
-                    p2m_get_iommu_flags(p2m_flags_to_type(flags));
                 old_mfn = l1e_get_pfn(*p2m_entry);
+                iommu_old_flags =
+                    p2m_get_iommu_flags(p2m_flags_to_type(flags),
+                                        _mfn(old_mfn));
             }
             else
             {
diff --git a/xen/arch/x86/mm/p2m.c b/xen/arch/x86/mm/p2m.c
index 6548e9f..bd8ce35 100644
--- a/xen/arch/x86/mm/p2m.c
+++ b/xen/arch/x86/mm/p2m.c
@@ -1053,16 +1053,7 @@ int set_identity_p2m_entry(struct domain *d, unsigned long gfn,
         ret = p2m_set_entry(p2m, gfn, _mfn(gfn), PAGE_ORDER_4K,
                             p2m_mmio_direct, p2ma);
     else if ( mfn_x(mfn) == gfn && p2mt == p2m_mmio_direct && a == p2ma )
-    {
         ret = 0;
-        /*
-         * PVH fixme: during Dom0 PVH construction, p2m entries are being set
-         * but iomem regions are not mapped with IOMMU. This makes sure that
-         * RMRRs are correctly mapped with IOMMU.
-         */
-        if ( is_hardware_domain(d) && !iommu_use_hap_pt(d) )
-            ret = iommu_map_page(d, gfn, gfn, IOMMUF_readable|IOMMUF_writable);
-    }
     else
     {
         if ( flag & XEN_DOMCTL_DEV_RDM_RELAXED )
diff --git a/xen/include/asm-x86/p2m.h b/xen/include/asm-x86/p2m.h
index f720f83..c68ff58 100644
--- a/xen/include/asm-x86/p2m.h
+++ b/xen/include/asm-x86/p2m.h
@@ -799,7 +799,7 @@ void p2m_altp2m_propagate_change(struct domain *d, gfn_t gfn,
 /*
  * p2m type to IOMMU flags
  */
-static inline unsigned int p2m_get_iommu_flags(p2m_type_t p2mt)
+static inline unsigned int p2m_get_iommu_flags(p2m_type_t p2mt, mfn_t mfn)
 {
     unsigned int flags;
 
@@ -815,6 +815,11 @@ static inline unsigned int p2m_get_iommu_flags(p2m_type_t p2mt)
     case p2m_grant_map_ro:
         flags = IOMMUF_readable;
         break;
+    case p2m_mmio_direct:
+        flags = IOMMUF_readable;
+        if ( !rangeset_contains_singleton(mmio_ro_ranges, mfn_x(mfn)) )
+            flags |= IOMMUF_writable;
+        break;
     default:
         flags = 0;
         break;
-- 
2.10.1 (Apple Git-78)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[G.R.]

>Feb 10, 2017 02:00，"Roger Pau Monné" <roger.pau@citrix.com> wrote:
>
>On Thu, Feb 09, 2017 at 07:58:56AM -0700, Jan Beulich wrote:
>> >>> On 09.02.17 at 15:46, <firemeteor@users.sourceforge.net> wrote:
>> > BTW -- I think that fix should not be conflicting with your debug change,
>> > right?
>>
>> Yes - ideally you'd keep that one in place along with adding Roger's
>> patch.
>
>Please use the patch below, the previous one was missing a break, which made it
>completely useless, sorry.
>
>Roger.

Thanks for the updated patch and sorry for the delayed response.
I would like to confirm that the updated patch appears to work. The
flooding fault is gone now.
I wasn't able to get ipxe launched in domU with gfx-passthrough=1 though.
Hopefully this is not a real issue as I reported the following before:
>For unknown reason, debug version of hypervisor will cause domU hang if gfx-passthrough=1 is present (traditional device model).

Rui

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: IOMMU fault with IGD passthrough setup on XEN 4.8.0

[G.R.]

Thank you for all your help Jan && Roger.
I think we can settle on this thread now.
After disabling the debug build, the gfx-passthrough back to work now.

On Mon, Feb 20, 2017 at 12:38 PM, G.R. <firemeteor@users.sourceforge.net> wrote:
>>Feb 10, 2017 02:00，"Roger Pau Monné" <roger.pau@citrix.com> wrote:
>>
>>On Thu, Feb 09, 2017 at 07:58:56AM -0700, Jan Beulich wrote:
>>> >>> On 09.02.17 at 15:46, <firemeteor@users.sourceforge.net> wrote:
>>> > BTW -- I think that fix should not be conflicting with your debug change,
>>> > right?
>>>
>>> Yes - ideally you'd keep that one in place along with adding Roger's
>>> patch.
>>
>>Please use the patch below, the previous one was missing a break, which made it
>>completely useless, sorry.
>>
>>Roger.
>
> Thanks for the updated patch and sorry for the delayed response.
> I would like to confirm that the updated patch appears to work. The
> flooding fault is gone now.
> I wasn't able to get ipxe launched in domU with gfx-passthrough=1 though.
> Hopefully this is not a real issue as I reported the following before:
>>For unknown reason, debug version of hypervisor will cause domU hang if gfx-passthrough=1 is present (traditional device model).
>
> Rui

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel