* S3 crash with VTD Queue Invalidation enabled
@ 2013-06-03 18:29 Ben Guthro
2013-06-03 19:22 ` Andrew Cooper
0 siblings, 1 reply; 26+ messages in thread
From: Ben Guthro @ 2013-06-03 18:29 UTC (permalink / raw)
To: xen-devel
I am seeing a crash on some vPro systems in the S3 path -
specifically a Lenovo ThinkPad x220t (Sandybridge)
Once I managed to not suspend the console, I got a panic in
queue_invalidate_wait()
(I added a dump_execution_state() here, to get some more info)
(XEN) Entering ACPI S3 state.
(XEN) ----[ Xen-4.2.2 x86_64 debug=y Not tainted ]----
(XEN) CPU: 0
(XEN) RIP: e008:[<ffff82c480149091>] invalidate_sync+0x258/0x291
(XEN) RFLAGS: 0000000000010086 CONTEXT: hypervisor
(XEN) rax: 0000000000000000 rbx: ffff830137a665c0 rcx: 0000000000000000
(XEN) rdx: ffff82c48030a0a0 rsi: 000000000000000a rdi: ffff82c4802766e0
(XEN) rbp: ffff82c4802bfd30 rsp: ffff82c4802bfce0 r8: 0000000000000004
(XEN) r9: 0000000000000002 r10: 0000000000000020 r11: 0000000000000010
(XEN) r12: 0000000bf34a77bc r13: 0000000000000000 r14: ffff830137a665f8
(XEN) r15: 0000000137a5c002 cr0: 000000008005003b cr4: 00000000000426f0
(XEN) cr3: 00000000ba2cd000 cr2: ffff880024181ff0
(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008
(XEN) Xen stack trace from rsp=ffff82c4802bfce0:
(XEN) 0000000000000002 0000000000000002 0101010000000002 0000000000000082
(XEN) 00000001802bfd30 ffff830137a665c0 0000000000000000 0000000000000000
(XEN) 0000000000000000 1000000000000000 ffff82c4802bfd90 ffff82c48014919d
(XEN) ffff82c400000000 0000000000000000 ffff82c4802bfd60 0000000000000000
(XEN) ffff82c4802bfd90 ffff830137a665c0 ffff830137a66540 0000000000000000
(XEN) ffff830137a66670 ffff82c4802679e0 ffff82c4802bfde0 ffff82c480145a60
(XEN) 0000000000000000 ffff82c4802bfdc0 ffff82c480125d36 ffff82c3ffd7a00c
(XEN) 0000000000000000 0000000000000003 0000000000000003 ffff82c48030a100
(XEN) ffff82c4802bfe20 ffff82c480145b08 ffff830137a4e620 ffff82c3ffd7a00c
(XEN) 0000000000000000 0000000000000003 0000000000000003 ffff82c48030a100
(XEN) ffff82c4802bfe30 ffff82c480141e12 ffff82c4802bfe80 ffff82c48019f315
(XEN) ffff82c4802bfe60 0000000000000282 0000000000000003 ffff83010cc0a010
(XEN) ffff8300ba0fd000 0000000000000000 0000000000000003 ffff82c48030a100
(XEN) ffff82c4802bfea0 ffff82c480105ed4 ffff8300ba0fd188 ffff82c48030a170
(XEN) ffff82c4802bfec0 ffff82c480127a1e ffff82c480125b8a ffff82c48030a190
(XEN) ffff82c4802bfef0 ffff82c480127d89 ffff82c4802bff18 ffff82c4802bff18
(XEN) ffff82c4802bff18 00000000ffffffff ffff82c4802bff10 ffff82c48015a42f
(XEN) ffff8300ba59a000 ffff8300ba0fd000 ffff82c4802bfda8 0000000000001403
(XEN) 0000000000000003 0000000000003403 ffffffff81a6b278 ffff8800049f3d28
(XEN) 0000000000000000 0000000000000246 0000000000000404 0000000000000003
(XEN) Xen call trace:
(XEN) [<ffff82c480149091>] invalidate_sync+0x258/0x291
(XEN) [<ffff82c48014919d>] flush_iotlb_qi+0xd3/0xef
(XEN) [<ffff82c480145a60>] iommu_flush_all+0xb5/0xde
(XEN) [<ffff82c480145b08>] vtd_suspend+0x23/0xf1
(XEN) [<ffff82c480141e12>] iommu_suspend+0x3c/0x3e
(XEN) [<ffff82c48019f315>] enter_state_helper+0x1a0/0x3cb
(XEN) [<ffff82c480105ed4>] continue_hypercall_tasklet_handler+0x51/0xbf
(XEN) [<ffff82c480127a1e>] do_tasklet_work+0x8d/0xc7
(XEN) [<ffff82c480127d89>] do_tasklet+0x6b/0x9b
(XEN) [<ffff82c48015a42f>] idle_loop+0x67/0x6f
(XEN)
(XEN)
(XEN) DMAR_IQA_REG = 137a5c002
(XEN) DMAR_IQH_REG = 120
(XEN) DMAR_IQT_REG = 140
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) queue invalidate wait descriptor was not executed
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...
This particular dump was with Xen 4.2.2, and Linux 3.8.8
I have tested the following other combinations, with no difference in behavior:
Xen-unstable git cs da3bca931fbcf0cbdfec971aca234e7ec0f41e16, with
Linux 3.10-rc3 cs 58f8bbd2e39c3732c55698494338ee19a92c53a0
Xen-4.2.2 / linux-3.8.8
Xen-4.2.2 / linux-3.8.13
Xen-4.2.3-pre / linux-3.8.13
Booting with iommu=no-qinval or iommu=off works around the problem,
but I was wondering if there was a more elegant solution, possibly
detecting, and disabling this feature if not working properly?
Thanks in advance for any insight.
Ben
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: S3 crash with VTD Queue Invalidation enabled
2013-06-03 18:29 S3 crash with VTD Queue Invalidation enabled Ben Guthro
@ 2013-06-03 19:22 ` Andrew Cooper
2013-06-04 8:54 ` Jan Beulich
0 siblings, 1 reply; 26+ messages in thread
From: Andrew Cooper @ 2013-06-03 19:22 UTC (permalink / raw)
To: Ben Guthro; +Cc: xen-devel
On 03/06/13 19:29, Ben Guthro wrote:
> I am seeing a crash on some vPro systems in the S3 path -
> specifically a Lenovo ThinkPad x220t (Sandybridge)
>
> Once I managed to not suspend the console, I got a panic in
> queue_invalidate_wait()
> (I added a dump_execution_state() here, to get some more info)
>
> (XEN) Entering ACPI S3 state.
> (XEN) ----[ Xen-4.2.2 x86_64 debug=y Not tainted ]----
> (XEN) CPU: 0
> (XEN) RIP: e008:[<ffff82c480149091>] invalidate_sync+0x258/0x291
> (XEN) RFLAGS: 0000000000010086 CONTEXT: hypervisor
> (XEN) rax: 0000000000000000 rbx: ffff830137a665c0 rcx: 0000000000000000
> (XEN) rdx: ffff82c48030a0a0 rsi: 000000000000000a rdi: ffff82c4802766e0
> (XEN) rbp: ffff82c4802bfd30 rsp: ffff82c4802bfce0 r8: 0000000000000004
> (XEN) r9: 0000000000000002 r10: 0000000000000020 r11: 0000000000000010
> (XEN) r12: 0000000bf34a77bc r13: 0000000000000000 r14: ffff830137a665f8
> (XEN) r15: 0000000137a5c002 cr0: 000000008005003b cr4: 00000000000426f0
> (XEN) cr3: 00000000ba2cd000 cr2: ffff880024181ff0
> (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008
> (XEN) Xen stack trace from rsp=ffff82c4802bfce0:
> (XEN) 0000000000000002 0000000000000002 0101010000000002 0000000000000082
> (XEN) 00000001802bfd30 ffff830137a665c0 0000000000000000 0000000000000000
> (XEN) 0000000000000000 1000000000000000 ffff82c4802bfd90 ffff82c48014919d
> (XEN) ffff82c400000000 0000000000000000 ffff82c4802bfd60 0000000000000000
> (XEN) ffff82c4802bfd90 ffff830137a665c0 ffff830137a66540 0000000000000000
> (XEN) ffff830137a66670 ffff82c4802679e0 ffff82c4802bfde0 ffff82c480145a60
> (XEN) 0000000000000000 ffff82c4802bfdc0 ffff82c480125d36 ffff82c3ffd7a00c
> (XEN) 0000000000000000 0000000000000003 0000000000000003 ffff82c48030a100
> (XEN) ffff82c4802bfe20 ffff82c480145b08 ffff830137a4e620 ffff82c3ffd7a00c
> (XEN) 0000000000000000 0000000000000003 0000000000000003 ffff82c48030a100
> (XEN) ffff82c4802bfe30 ffff82c480141e12 ffff82c4802bfe80 ffff82c48019f315
> (XEN) ffff82c4802bfe60 0000000000000282 0000000000000003 ffff83010cc0a010
> (XEN) ffff8300ba0fd000 0000000000000000 0000000000000003 ffff82c48030a100
> (XEN) ffff82c4802bfea0 ffff82c480105ed4 ffff8300ba0fd188 ffff82c48030a170
> (XEN) ffff82c4802bfec0 ffff82c480127a1e ffff82c480125b8a ffff82c48030a190
> (XEN) ffff82c4802bfef0 ffff82c480127d89 ffff82c4802bff18 ffff82c4802bff18
> (XEN) ffff82c4802bff18 00000000ffffffff ffff82c4802bff10 ffff82c48015a42f
> (XEN) ffff8300ba59a000 ffff8300ba0fd000 ffff82c4802bfda8 0000000000001403
> (XEN) 0000000000000003 0000000000003403 ffffffff81a6b278 ffff8800049f3d28
> (XEN) 0000000000000000 0000000000000246 0000000000000404 0000000000000003
> (XEN) Xen call trace:
> (XEN) [<ffff82c480149091>] invalidate_sync+0x258/0x291
> (XEN) [<ffff82c48014919d>] flush_iotlb_qi+0xd3/0xef
> (XEN) [<ffff82c480145a60>] iommu_flush_all+0xb5/0xde
> (XEN) [<ffff82c480145b08>] vtd_suspend+0x23/0xf1
> (XEN) [<ffff82c480141e12>] iommu_suspend+0x3c/0x3e
> (XEN) [<ffff82c48019f315>] enter_state_helper+0x1a0/0x3cb
> (XEN) [<ffff82c480105ed4>] continue_hypercall_tasklet_handler+0x51/0xbf
> (XEN) [<ffff82c480127a1e>] do_tasklet_work+0x8d/0xc7
> (XEN) [<ffff82c480127d89>] do_tasklet+0x6b/0x9b
> (XEN) [<ffff82c48015a42f>] idle_loop+0x67/0x6f
> (XEN)
> (XEN)
> (XEN) DMAR_IQA_REG = 137a5c002
> (XEN) DMAR_IQH_REG = 120
> (XEN) DMAR_IQT_REG = 140
> (XEN) ****************************************
> (XEN) Panic on CPU 0:
> (XEN) queue invalidate wait descriptor was not executed
> (XEN) ****************************************
> (XEN)
> (XEN) Reboot in five seconds...
>
>
>
> This particular dump was with Xen 4.2.2, and Linux 3.8.8
> I have tested the following other combinations, with no difference in behavior:
>
> Xen-unstable git cs da3bca931fbcf0cbdfec971aca234e7ec0f41e16, with
> Linux 3.10-rc3 cs 58f8bbd2e39c3732c55698494338ee19a92c53a0
>
> Xen-4.2.2 / linux-3.8.8
> Xen-4.2.2 / linux-3.8.13
> Xen-4.2.3-pre / linux-3.8.13
>
> Booting with iommu=no-qinval or iommu=off works around the problem,
> but I was wondering if there was a more elegant solution, possibly
> detecting, and disabling this feature if not working properly?
>
>
> Thanks in advance for any insight.
>
> Ben
This was likely broken by XSA-36
My fix for the crash path is:
http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=53fd1d8458de01169dfb56feb315f02c2b521a86
You want to inspect the use of iommu_enabled and iommu_intremap.
~Andrew
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: S3 crash with VTD Queue Invalidation enabled
2013-06-03 19:22 ` Andrew Cooper
@ 2013-06-04 8:54 ` Jan Beulich
2013-06-04 12:25 ` Ben Guthro
0 siblings, 1 reply; 26+ messages in thread
From: Jan Beulich @ 2013-06-04 8:54 UTC (permalink / raw)
To: Andrew Cooper, Ben Guthro; +Cc: xen-devel
>>> On 03.06.13 at 21:22, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> On 03/06/13 19:29, Ben Guthro wrote:
>> (XEN) Xen call trace:
>> (XEN) [<ffff82c480149091>] invalidate_sync+0x258/0x291
>> (XEN) [<ffff82c48014919d>] flush_iotlb_qi+0xd3/0xef
>> (XEN) [<ffff82c480145a60>] iommu_flush_all+0xb5/0xde
>> (XEN) [<ffff82c480145b08>] vtd_suspend+0x23/0xf1
>> (XEN) [<ffff82c480141e12>] iommu_suspend+0x3c/0x3e
>> (XEN) [<ffff82c48019f315>] enter_state_helper+0x1a0/0x3cb
>> (XEN) [<ffff82c480105ed4>] continue_hypercall_tasklet_handler+0x51/0xbf
>> (XEN) [<ffff82c480127a1e>] do_tasklet_work+0x8d/0xc7
>> (XEN) [<ffff82c480127d89>] do_tasklet+0x6b/0x9b
>> (XEN) [<ffff82c48015a42f>] idle_loop+0x67/0x6f
>
> This was likely broken by XSA-36
>
> My fix for the crash path is:
> http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=53fd1d8458de01169dfb
> 56feb315f02c2b521a86
>
> You want to inspect the use of iommu_enabled and iommu_intremap.
According to the comment in vtd_suspend(),
iommu_disable_x2apic_IR() is supposed to run after
iommu_suspend() (and indeed lapic_suspend() gets called
immediately after iommu_suspend() by device_power_down()),
and hence that shouldn't be the reason here. But, Ben, to be
sure, dumping the state of the various IOMMU related enabling
variables would be a good idea.
Is this perhaps having some similarity with
http://lists.xen.org/archives/html/xen-devel/2013-04/msg00343.html?
We're clearly running single-CPU only here and there...
Jan
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: S3 crash with VTD Queue Invalidation enabled
2013-06-04 8:54 ` Jan Beulich
@ 2013-06-04 12:25 ` Ben Guthro
2013-06-04 14:01 ` Jan Beulich
0 siblings, 1 reply; 26+ messages in thread
From: Ben Guthro @ 2013-06-04 12:25 UTC (permalink / raw)
To: Jan Beulich; +Cc: Andrew Cooper, xen-devel
On Tue, Jun 4, 2013 at 4:54 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 03.06.13 at 21:22, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>> On 03/06/13 19:29, Ben Guthro wrote:
>>> (XEN) Xen call trace:
>>> (XEN) [<ffff82c480149091>] invalidate_sync+0x258/0x291
>>> (XEN) [<ffff82c48014919d>] flush_iotlb_qi+0xd3/0xef
>>> (XEN) [<ffff82c480145a60>] iommu_flush_all+0xb5/0xde
>>> (XEN) [<ffff82c480145b08>] vtd_suspend+0x23/0xf1
>>> (XEN) [<ffff82c480141e12>] iommu_suspend+0x3c/0x3e
>>> (XEN) [<ffff82c48019f315>] enter_state_helper+0x1a0/0x3cb
>>> (XEN) [<ffff82c480105ed4>] continue_hypercall_tasklet_handler+0x51/0xbf
>>> (XEN) [<ffff82c480127a1e>] do_tasklet_work+0x8d/0xc7
>>> (XEN) [<ffff82c480127d89>] do_tasklet+0x6b/0x9b
>>> (XEN) [<ffff82c48015a42f>] idle_loop+0x67/0x6f
>>
>> This was likely broken by XSA-36
>>
>> My fix for the crash path is:
>> http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=53fd1d8458de01169dfb
>> 56feb315f02c2b521a86
>>
>> You want to inspect the use of iommu_enabled and iommu_intremap.
>
> According to the comment in vtd_suspend(),
> iommu_disable_x2apic_IR() is supposed to run after
> iommu_suspend() (and indeed lapic_suspend() gets called
> immediately after iommu_suspend() by device_power_down()),
> and hence that shouldn't be the reason here. But, Ben, to be
> sure, dumping the state of the various IOMMU related enabling
> variables would be a good idea.
I assume you are referring to the variables below, defined at the top of iommu.c
At the time of the crash, they look like this:
(XEN) iommu_enabled = 1
(XEN) force_iommu; = 0
(XEN) iommu_verbose; = 0
(XEN) iommu_workaround_bios_bug; = 0
(XEN) iommu_passthrough; = 0
(XEN) iommu_snoop = 0
(XEN) iommu_qinval = 1
(XEN) iommu_intremap = 1
(XEN) iommu_hap_pt_share = 0
(XEN) iommu_debug; = 0
(XEN) amd_iommu_perdev_intremap = 1
If that gives any additional insight, please let me know.
I'm not sure I gleaned anything particularly significant from it though.
Or - perhaps you are referring to other enabling variables?
>
> Is this perhaps having some similarity with
> http://lists.xen.org/archives/html/xen-devel/2013-04/msg00343.html?
> We're clearly running single-CPU only here and there...
We certainly should be, as we have gone through the
disable_nonboot_cpus() by this point - and I can verify that from the
logs.
Ben
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: S3 crash with VTD Queue Invalidation enabled
2013-06-04 12:25 ` Ben Guthro
@ 2013-06-04 14:01 ` Jan Beulich
2013-06-04 19:20 ` Ben Guthro
0 siblings, 1 reply; 26+ messages in thread
From: Jan Beulich @ 2013-06-04 14:01 UTC (permalink / raw)
To: Ben Guthro; +Cc: Andrew Cooper, xen-devel
>>> On 04.06.13 at 14:25, Ben Guthro <ben@guthro.net> wrote:
> On Tue, Jun 4, 2013 at 4:54 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>> On 03.06.13 at 21:22, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>>> On 03/06/13 19:29, Ben Guthro wrote:
>>>> (XEN) Xen call trace:
>>>> (XEN) [<ffff82c480149091>] invalidate_sync+0x258/0x291
>>>> (XEN) [<ffff82c48014919d>] flush_iotlb_qi+0xd3/0xef
>>>> (XEN) [<ffff82c480145a60>] iommu_flush_all+0xb5/0xde
>>>> (XEN) [<ffff82c480145b08>] vtd_suspend+0x23/0xf1
>>>> (XEN) [<ffff82c480141e12>] iommu_suspend+0x3c/0x3e
>>>> (XEN) [<ffff82c48019f315>] enter_state_helper+0x1a0/0x3cb
>>>> (XEN) [<ffff82c480105ed4>] continue_hypercall_tasklet_handler+0x51/0xbf
>>>> (XEN) [<ffff82c480127a1e>] do_tasklet_work+0x8d/0xc7
>>>> (XEN) [<ffff82c480127d89>] do_tasklet+0x6b/0x9b
>>>> (XEN) [<ffff82c48015a42f>] idle_loop+0x67/0x6f
>>>
>>> This was likely broken by XSA-36
>>>
>>> My fix for the crash path is:
>>> http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=53fd1d8458de01169dfb
>>> 56feb315f02c2b521a86
>>>
>>> You want to inspect the use of iommu_enabled and iommu_intremap.
>>
>> According to the comment in vtd_suspend(),
>> iommu_disable_x2apic_IR() is supposed to run after
>> iommu_suspend() (and indeed lapic_suspend() gets called
>> immediately after iommu_suspend() by device_power_down()),
>> and hence that shouldn't be the reason here. But, Ben, to be
>> sure, dumping the state of the various IOMMU related enabling
>> variables would be a good idea.
>
> I assume you are referring to the variables below, defined at the top of
> iommu.c
> At the time of the crash, they look like this:
>
> (XEN) iommu_enabled = 1
> (XEN) force_iommu; = 0
> (XEN) iommu_verbose; = 0
> (XEN) iommu_workaround_bios_bug; = 0
> (XEN) iommu_passthrough; = 0
> (XEN) iommu_snoop = 0
> (XEN) iommu_qinval = 1
> (XEN) iommu_intremap = 1
> (XEN) iommu_hap_pt_share = 0
> (XEN) iommu_debug; = 0
> (XEN) amd_iommu_perdev_intremap = 1
>
> If that gives any additional insight, please let me know.
> I'm not sure I gleaned anything particularly significant from it though.
>
> Or - perhaps you are referring to other enabling variables?
These were exactly the ones (or really you picked a superset of
what I wanted to know the state of). To me this pretty clearly
means that Andrew's original thought here is not applicable, as
at this point we can't possibly have shut down qinval yet.
>> Is this perhaps having some similarity with
>> http://lists.xen.org/archives/html/xen-devel/2013-04/msg00343.html?
>> We're clearly running single-CPU only here and there...
>
> We certainly should be, as we have gone through the
> disable_nonboot_cpus() by this point - and I can verify that from the
> logs.
I'm much more tending towards the connection here, noting that
Andrew's original thread didn't really lead anywhere (i.e. we still
don't know what the panic he saw was actually caused by).
Jan
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: S3 crash with VTD Queue Invalidation enabled
2013-06-04 14:01 ` Jan Beulich
@ 2013-06-04 19:20 ` Ben Guthro
2013-06-04 19:49 ` Ben Guthro
0 siblings, 1 reply; 26+ messages in thread
From: Ben Guthro @ 2013-06-04 19:20 UTC (permalink / raw)
To: Jan Beulich; +Cc: Andrew Cooper, xen-devel
On Tue, Jun 4, 2013 at 10:01 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 04.06.13 at 14:25, Ben Guthro <ben@guthro.net> wrote:
>> On Tue, Jun 4, 2013 at 4:54 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>> On 03.06.13 at 21:22, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>>>> On 03/06/13 19:29, Ben Guthro wrote:
>>>>> (XEN) Xen call trace:
>>>>> (XEN) [<ffff82c480149091>] invalidate_sync+0x258/0x291
>>>>> (XEN) [<ffff82c48014919d>] flush_iotlb_qi+0xd3/0xef
>>>>> (XEN) [<ffff82c480145a60>] iommu_flush_all+0xb5/0xde
>>>>> (XEN) [<ffff82c480145b08>] vtd_suspend+0x23/0xf1
>>>>> (XEN) [<ffff82c480141e12>] iommu_suspend+0x3c/0x3e
>>>>> (XEN) [<ffff82c48019f315>] enter_state_helper+0x1a0/0x3cb
>>>>> (XEN) [<ffff82c480105ed4>] continue_hypercall_tasklet_handler+0x51/0xbf
>>>>> (XEN) [<ffff82c480127a1e>] do_tasklet_work+0x8d/0xc7
>>>>> (XEN) [<ffff82c480127d89>] do_tasklet+0x6b/0x9b
>>>>> (XEN) [<ffff82c48015a42f>] idle_loop+0x67/0x6f
>>>>
>>>> This was likely broken by XSA-36
>>>>
>>>> My fix for the crash path is:
>>>> http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=53fd1d8458de01169dfb
>>>> 56feb315f02c2b521a86
>>>>
>>>> You want to inspect the use of iommu_enabled and iommu_intremap.
>>>
>>> According to the comment in vtd_suspend(),
>>> iommu_disable_x2apic_IR() is supposed to run after
>>> iommu_suspend() (and indeed lapic_suspend() gets called
>>> immediately after iommu_suspend() by device_power_down()),
>>> and hence that shouldn't be the reason here. But, Ben, to be
>>> sure, dumping the state of the various IOMMU related enabling
>>> variables would be a good idea.
>>
>> I assume you are referring to the variables below, defined at the top of
>> iommu.c
>> At the time of the crash, they look like this:
>>
>> (XEN) iommu_enabled = 1
>> (XEN) force_iommu; = 0
>> (XEN) iommu_verbose; = 0
>> (XEN) iommu_workaround_bios_bug; = 0
>> (XEN) iommu_passthrough; = 0
>> (XEN) iommu_snoop = 0
>> (XEN) iommu_qinval = 1
>> (XEN) iommu_intremap = 1
>> (XEN) iommu_hap_pt_share = 0
>> (XEN) iommu_debug; = 0
>> (XEN) amd_iommu_perdev_intremap = 1
>>
>> If that gives any additional insight, please let me know.
>> I'm not sure I gleaned anything particularly significant from it though.
>>
>> Or - perhaps you are referring to other enabling variables?
>
> These were exactly the ones (or really you picked a superset of
> what I wanted to know the state of). To me this pretty clearly
> means that Andrew's original thought here is not applicable, as
> at this point we can't possibly have shut down qinval yet.
>
>>> Is this perhaps having some similarity with
>>> http://lists.xen.org/archives/html/xen-devel/2013-04/msg00343.html?
>>> We're clearly running single-CPU only here and there...
>>
>> We certainly should be, as we have gone through the
>> disable_nonboot_cpus() by this point - and I can verify that from the
>> logs.
>
> I'm much more tending towards the connection here, noting that
> Andrew's original thread didn't really lead anywhere (i.e. we still
> don't know what the panic he saw was actually caused by).
>
I'm starting to think you're on to something here.
I've put a bunch of trace throughout the functions in qinval.c
It seems that everything is functioning properly, up until we go
through the disable_nonboot_cpus() path.
Prior to this, I see the qinval.c functions being executed on all
cpus, and both drhd units
Afterward, it gets stuck in queue_invalidate_wait on the first drhd
unit.. and eventually panics.
I'm not exactly sure what to make of this yet.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: S3 crash with VTD Queue Invalidation enabled
2013-06-04 19:20 ` Ben Guthro
@ 2013-06-04 19:49 ` Ben Guthro
2013-06-04 21:09 ` Ben Guthro
0 siblings, 1 reply; 26+ messages in thread
From: Ben Guthro @ 2013-06-04 19:49 UTC (permalink / raw)
To: Jan Beulich; +Cc: Andrew Cooper, xen-devel
On Tue, Jun 4, 2013 at 3:20 PM, Ben Guthro <ben@guthro.net> wrote:
> On Tue, Jun 4, 2013 at 10:01 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>> On 04.06.13 at 14:25, Ben Guthro <ben@guthro.net> wrote:
>>> On Tue, Jun 4, 2013 at 4:54 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>> On 03.06.13 at 21:22, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>>>>> On 03/06/13 19:29, Ben Guthro wrote:
>>>>>> (XEN) Xen call trace:
>>>>>> (XEN) [<ffff82c480149091>] invalidate_sync+0x258/0x291
>>>>>> (XEN) [<ffff82c48014919d>] flush_iotlb_qi+0xd3/0xef
>>>>>> (XEN) [<ffff82c480145a60>] iommu_flush_all+0xb5/0xde
>>>>>> (XEN) [<ffff82c480145b08>] vtd_suspend+0x23/0xf1
>>>>>> (XEN) [<ffff82c480141e12>] iommu_suspend+0x3c/0x3e
>>>>>> (XEN) [<ffff82c48019f315>] enter_state_helper+0x1a0/0x3cb
>>>>>> (XEN) [<ffff82c480105ed4>] continue_hypercall_tasklet_handler+0x51/0xbf
>>>>>> (XEN) [<ffff82c480127a1e>] do_tasklet_work+0x8d/0xc7
>>>>>> (XEN) [<ffff82c480127d89>] do_tasklet+0x6b/0x9b
>>>>>> (XEN) [<ffff82c48015a42f>] idle_loop+0x67/0x6f
>>>>>
>>>>> This was likely broken by XSA-36
>>>>>
>>>>> My fix for the crash path is:
>>>>> http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=53fd1d8458de01169dfb
>>>>> 56feb315f02c2b521a86
>>>>>
>>>>> You want to inspect the use of iommu_enabled and iommu_intremap.
>>>>
>>>> According to the comment in vtd_suspend(),
>>>> iommu_disable_x2apic_IR() is supposed to run after
>>>> iommu_suspend() (and indeed lapic_suspend() gets called
>>>> immediately after iommu_suspend() by device_power_down()),
>>>> and hence that shouldn't be the reason here. But, Ben, to be
>>>> sure, dumping the state of the various IOMMU related enabling
>>>> variables would be a good idea.
>>>
>>> I assume you are referring to the variables below, defined at the top of
>>> iommu.c
>>> At the time of the crash, they look like this:
>>>
>>> (XEN) iommu_enabled = 1
>>> (XEN) force_iommu; = 0
>>> (XEN) iommu_verbose; = 0
>>> (XEN) iommu_workaround_bios_bug; = 0
>>> (XEN) iommu_passthrough; = 0
>>> (XEN) iommu_snoop = 0
>>> (XEN) iommu_qinval = 1
>>> (XEN) iommu_intremap = 1
>>> (XEN) iommu_hap_pt_share = 0
>>> (XEN) iommu_debug; = 0
>>> (XEN) amd_iommu_perdev_intremap = 1
>>>
>>> If that gives any additional insight, please let me know.
>>> I'm not sure I gleaned anything particularly significant from it though.
>>>
>>> Or - perhaps you are referring to other enabling variables?
>>
>> These were exactly the ones (or really you picked a superset of
>> what I wanted to know the state of). To me this pretty clearly
>> means that Andrew's original thought here is not applicable, as
>> at this point we can't possibly have shut down qinval yet.
>>
>>>> Is this perhaps having some similarity with
>>>> http://lists.xen.org/archives/html/xen-devel/2013-04/msg00343.html?
>>>> We're clearly running single-CPU only here and there...
>>>
>>> We certainly should be, as we have gone through the
>>> disable_nonboot_cpus() by this point - and I can verify that from the
>>> logs.
>>
>> I'm much more tending towards the connection here, noting that
>> Andrew's original thread didn't really lead anywhere (i.e. we still
>> don't know what the panic he saw was actually caused by).
>>
>
> I'm starting to think you're on to something here.
hmm - maybe not.
I get the same crash with "maxcpus=1"
> I've put a bunch of trace throughout the functions in qinval.c
>
> It seems that everything is functioning properly, up until we go
> through the disable_nonboot_cpus() path.
> Prior to this, I see the qinval.c functions being executed on all
> cpus, and both drhd units
> Afterward, it gets stuck in queue_invalidate_wait on the first drhd
> unit.. and eventually panics.
>
> I'm not exactly sure what to make of this yet.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: S3 crash with VTD Queue Invalidation enabled
2013-06-04 19:49 ` Ben Guthro
@ 2013-06-04 21:09 ` Ben Guthro
2013-06-05 8:24 ` Jan Beulich
0 siblings, 1 reply; 26+ messages in thread
From: Ben Guthro @ 2013-06-04 21:09 UTC (permalink / raw)
To: Jan Beulich; +Cc: Andrew Cooper, xen-devel
On Tue, Jun 4, 2013 at 3:49 PM, Ben Guthro <ben@guthro.net> wrote:
> On Tue, Jun 4, 2013 at 3:20 PM, Ben Guthro <ben@guthro.net> wrote:
>> On Tue, Jun 4, 2013 at 10:01 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>> On 04.06.13 at 14:25, Ben Guthro <ben@guthro.net> wrote:
>>>> On Tue, Jun 4, 2013 at 4:54 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>>> On 03.06.13 at 21:22, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>>>>>> On 03/06/13 19:29, Ben Guthro wrote:
>>>>>>> (XEN) Xen call trace:
>>>>>>> (XEN) [<ffff82c480149091>] invalidate_sync+0x258/0x291
>>>>>>> (XEN) [<ffff82c48014919d>] flush_iotlb_qi+0xd3/0xef
>>>>>>> (XEN) [<ffff82c480145a60>] iommu_flush_all+0xb5/0xde
>>>>>>> (XEN) [<ffff82c480145b08>] vtd_suspend+0x23/0xf1
>>>>>>> (XEN) [<ffff82c480141e12>] iommu_suspend+0x3c/0x3e
>>>>>>> (XEN) [<ffff82c48019f315>] enter_state_helper+0x1a0/0x3cb
>>>>>>> (XEN) [<ffff82c480105ed4>] continue_hypercall_tasklet_handler+0x51/0xbf
>>>>>>> (XEN) [<ffff82c480127a1e>] do_tasklet_work+0x8d/0xc7
>>>>>>> (XEN) [<ffff82c480127d89>] do_tasklet+0x6b/0x9b
>>>>>>> (XEN) [<ffff82c48015a42f>] idle_loop+0x67/0x6f
>>>>>>
>>>>>> This was likely broken by XSA-36
>>>>>>
>>>>>> My fix for the crash path is:
>>>>>> http://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=53fd1d8458de01169dfb
>>>>>> 56feb315f02c2b521a86
>>>>>>
>>>>>> You want to inspect the use of iommu_enabled and iommu_intremap.
>>>>>
>>>>> According to the comment in vtd_suspend(),
>>>>> iommu_disable_x2apic_IR() is supposed to run after
>>>>> iommu_suspend() (and indeed lapic_suspend() gets called
>>>>> immediately after iommu_suspend() by device_power_down()),
>>>>> and hence that shouldn't be the reason here. But, Ben, to be
>>>>> sure, dumping the state of the various IOMMU related enabling
>>>>> variables would be a good idea.
>>>>
>>>> I assume you are referring to the variables below, defined at the top of
>>>> iommu.c
>>>> At the time of the crash, they look like this:
>>>>
>>>> (XEN) iommu_enabled = 1
>>>> (XEN) force_iommu; = 0
>>>> (XEN) iommu_verbose; = 0
>>>> (XEN) iommu_workaround_bios_bug; = 0
>>>> (XEN) iommu_passthrough; = 0
>>>> (XEN) iommu_snoop = 0
>>>> (XEN) iommu_qinval = 1
>>>> (XEN) iommu_intremap = 1
>>>> (XEN) iommu_hap_pt_share = 0
>>>> (XEN) iommu_debug; = 0
>>>> (XEN) amd_iommu_perdev_intremap = 1
>>>>
>>>> If that gives any additional insight, please let me know.
>>>> I'm not sure I gleaned anything particularly significant from it though.
>>>>
>>>> Or - perhaps you are referring to other enabling variables?
>>>
>>> These were exactly the ones (or really you picked a superset of
>>> what I wanted to know the state of). To me this pretty clearly
>>> means that Andrew's original thought here is not applicable, as
>>> at this point we can't possibly have shut down qinval yet.
>>>
>>>>> Is this perhaps having some similarity with
>>>>> http://lists.xen.org/archives/html/xen-devel/2013-04/msg00343.html?
>>>>> We're clearly running single-CPU only here and there...
>>>>
>>>> We certainly should be, as we have gone through the
>>>> disable_nonboot_cpus() by this point - and I can verify that from the
>>>> logs.
>>>
>>> I'm much more tending towards the connection here, noting that
>>> Andrew's original thread didn't really lead anywhere (i.e. we still
>>> don't know what the panic he saw was actually caused by).
>>>
>>
>> I'm starting to think you're on to something here.
>
> hmm - maybe not.
> I get the same crash with "maxcpus=1"
>
>
>
>> I've put a bunch of trace throughout the functions in qinval.c
>>
>> It seems that everything is functioning properly, up until we go
>> through the disable_nonboot_cpus() path.
>> Prior to this, I see the qinval.c functions being executed on all
>> cpus, and both drhd units
>> Afterward, it gets stuck in queue_invalidate_wait on the first drhd
>> unit.. and eventually panics.
>>
>> I'm not exactly sure what to make of this yet.
querying status of the hardware all seems to be working correctly...
it just doesn't work with querying the QINVAL_STAT_DONE state, as far
as I can tell.
Other register state is:
(XEN) VER = 10
(XEN) CAP = c0000020e60262
(XEN) n_fault_reg = 1
(XEN) fault_recording_offset = 200
(XEN) fault_recording_reg_l = 0
(XEN) fault_recording_reg_h = 0
(XEN) ECAP = f0101a
(XEN) GCMD = 0
(XEN) GSTS = c7000000
(XEN) RTADDR = 137a31000
(XEN) CCMD = 800000000000000
(XEN) FSTS = 0
(XEN) FECTL = 0
(XEN) FEDATA = 4128
(XEN) FEADDR = fee0000c
(XEN) FEUADDR = 0
(with code lifted from print_iommu_regs() )
None of this looks suspicious to my untrained eye - but I'm including
it here in case someone else sees something I don't.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: S3 crash with VTD Queue Invalidation enabled
2013-06-04 21:09 ` Ben Guthro
@ 2013-06-05 8:24 ` Jan Beulich
2013-06-05 13:54 ` Ben Guthro
0 siblings, 1 reply; 26+ messages in thread
From: Jan Beulich @ 2013-06-05 8:24 UTC (permalink / raw)
To: Ben Guthro, xiantao.zhang; +Cc: Andrew Cooper, xen-devel
>>> On 04.06.13 at 23:09, Ben Guthro <ben@guthro.net> wrote:
> On Tue, Jun 4, 2013 at 3:49 PM, Ben Guthro <ben@guthro.net> wrote:
>> On Tue, Jun 4, 2013 at 3:20 PM, Ben Guthro <ben@guthro.net> wrote:
>>> On Tue, Jun 4, 2013 at 10:01 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>> On 04.06.13 at 14:25, Ben Guthro <ben@guthro.net> wrote:
>>>>> On Tue, Jun 4, 2013 at 4:54 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>> Is this perhaps having some similarity with
>>>>>> http://lists.xen.org/archives/html/xen-devel/2013-04/msg00343.html?
>>>>>> We're clearly running single-CPU only here and there...
>>>>>
>>>>> We certainly should be, as we have gone through the
>>>>> disable_nonboot_cpus() by this point - and I can verify that from the
>>>>> logs.
>>>>
>>>> I'm much more tending towards the connection here, noting that
>>>> Andrew's original thread didn't really lead anywhere (i.e. we still
>>>> don't know what the panic he saw was actually caused by).
>>>>
>>>
>>> I'm starting to think you're on to something here.
>>
>> hmm - maybe not.
>> I get the same crash with "maxcpus=1"
>>
>>
>>
>>> I've put a bunch of trace throughout the functions in qinval.c
>>>
>>> It seems that everything is functioning properly, up until we go
>>> through the disable_nonboot_cpus() path.
>>> Prior to this, I see the qinval.c functions being executed on all
>>> cpus, and both drhd units
>>> Afterward, it gets stuck in queue_invalidate_wait on the first drhd
>>> unit.. and eventually panics.
>>>
>>> I'm not exactly sure what to make of this yet.
>
> querying status of the hardware all seems to be working correctly...
> it just doesn't work with querying the QINVAL_STAT_DONE state, as far
> as I can tell.
>
> Other register state is:
>
> (XEN) VER = 10
> (XEN) CAP = c0000020e60262
> (XEN) n_fault_reg = 1
> (XEN) fault_recording_offset = 200
> (XEN) fault_recording_reg_l = 0
> (XEN) fault_recording_reg_h = 0
> (XEN) ECAP = f0101a
> (XEN) GCMD = 0
> (XEN) GSTS = c7000000
> (XEN) RTADDR = 137a31000
> (XEN) CCMD = 800000000000000
> (XEN) FSTS = 0
> (XEN) FECTL = 0
> (XEN) FEDATA = 4128
> (XEN) FEADDR = fee0000c
> (XEN) FEUADDR = 0
>
> (with code lifted from print_iommu_regs() )
>
>
> None of this looks suspicious to my untrained eye - but I'm including
> it here in case someone else sees something I don't.
Xiantao, you certainly will want to give some advice here. I won't
be able to look into this more deeply right away.
Jan
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: S3 crash with VTD Queue Invalidation enabled
2013-06-05 8:24 ` Jan Beulich
@ 2013-06-05 13:54 ` Ben Guthro
2013-06-05 15:14 ` Jan Beulich
0 siblings, 1 reply; 26+ messages in thread
From: Ben Guthro @ 2013-06-05 13:54 UTC (permalink / raw)
To: Jan Beulich; +Cc: Andrew Cooper, xiantao.zhang, xen-devel
On Wed, Jun 5, 2013 at 4:24 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 04.06.13 at 23:09, Ben Guthro <ben@guthro.net> wrote:
>> On Tue, Jun 4, 2013 at 3:49 PM, Ben Guthro <ben@guthro.net> wrote:
>>> On Tue, Jun 4, 2013 at 3:20 PM, Ben Guthro <ben@guthro.net> wrote:
>>>> On Tue, Jun 4, 2013 at 10:01 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>>> On 04.06.13 at 14:25, Ben Guthro <ben@guthro.net> wrote:
>>>>>> On Tue, Jun 4, 2013 at 4:54 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>> Is this perhaps having some similarity with
>>>>>>> http://lists.xen.org/archives/html/xen-devel/2013-04/msg00343.html?
>>>>>>> We're clearly running single-CPU only here and there...
>>>>>>
>>>>>> We certainly should be, as we have gone through the
>>>>>> disable_nonboot_cpus() by this point - and I can verify that from the
>>>>>> logs.
>>>>>
>>>>> I'm much more tending towards the connection here, noting that
>>>>> Andrew's original thread didn't really lead anywhere (i.e. we still
>>>>> don't know what the panic he saw was actually caused by).
>>>>>
>>>>
>>>> I'm starting to think you're on to something here.
>>>
>>> hmm - maybe not.
>>> I get the same crash with "maxcpus=1"
>>>
>>>
>>>
>>>> I've put a bunch of trace throughout the functions in qinval.c
>>>>
>>>> It seems that everything is functioning properly, up until we go
>>>> through the disable_nonboot_cpus() path.
>>>> Prior to this, I see the qinval.c functions being executed on all
>>>> cpus, and both drhd units
>>>> Afterward, it gets stuck in queue_invalidate_wait on the first drhd
>>>> unit.. and eventually panics.
>>>>
>>>> I'm not exactly sure what to make of this yet.
>>
>> querying status of the hardware all seems to be working correctly...
>> it just doesn't work with querying the QINVAL_STAT_DONE state, as far
>> as I can tell.
>>
>> Other register state is:
>>
>> (XEN) VER = 10
>> (XEN) CAP = c0000020e60262
>> (XEN) n_fault_reg = 1
>> (XEN) fault_recording_offset = 200
>> (XEN) fault_recording_reg_l = 0
>> (XEN) fault_recording_reg_h = 0
>> (XEN) ECAP = f0101a
>> (XEN) GCMD = 0
>> (XEN) GSTS = c7000000
>> (XEN) RTADDR = 137a31000
>> (XEN) CCMD = 800000000000000
>> (XEN) FSTS = 0
>> (XEN) FECTL = 0
>> (XEN) FEDATA = 4128
>> (XEN) FEADDR = fee0000c
>> (XEN) FEUADDR = 0
>>
>> (with code lifted from print_iommu_regs() )
>>
>>
>> None of this looks suspicious to my untrained eye - but I'm including
>> it here in case someone else sees something I don't.
>
> Xiantao, you certainly will want to give some advice here. I won't
> be able to look into this more deeply right away.
Thanks Jan. Xiantao - I'd appreciate any insight you may have.
One curious thing I have found, that seems buggy to me, is that
{dis,en}able_qinval() is being called prior to the platform quirks
being executed.
It appears they are being called through iommu_{en,dis}able_x2apic_IR()
However, when I try to put a BUG(), or dump_execution_state in that
code, it would not dump a stack.
I was going to put a platform quirk in, to detect, and disable qinval
on this platform, but it seems that may be too late in the process.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: S3 crash with VTD Queue Invalidation enabled
2013-06-05 13:54 ` Ben Guthro
@ 2013-06-05 15:14 ` Jan Beulich
2013-06-05 15:25 ` Ben Guthro
0 siblings, 1 reply; 26+ messages in thread
From: Jan Beulich @ 2013-06-05 15:14 UTC (permalink / raw)
To: Ben Guthro; +Cc: Andrew Cooper, xiantao.zhang, xen-devel
>>> On 05.06.13 at 15:54, Ben Guthro <ben@guthro.net> wrote:
> On Wed, Jun 5, 2013 at 4:24 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>> On 04.06.13 at 23:09, Ben Guthro <ben@guthro.net> wrote:
>>> On Tue, Jun 4, 2013 at 3:49 PM, Ben Guthro <ben@guthro.net> wrote:
>>>> On Tue, Jun 4, 2013 at 3:20 PM, Ben Guthro <ben@guthro.net> wrote:
>>>>> On Tue, Jun 4, 2013 at 10:01 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>>>> On 04.06.13 at 14:25, Ben Guthro <ben@guthro.net> wrote:
>>>>>>> On Tue, Jun 4, 2013 at 4:54 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>>> Is this perhaps having some similarity with
>>>>>>>> http://lists.xen.org/archives/html/xen-devel/2013-04/msg00343.html?
>>>>>>>> We're clearly running single-CPU only here and there...
>>>>>>>
>>>>>>> We certainly should be, as we have gone through the
>>>>>>> disable_nonboot_cpus() by this point - and I can verify that from the
>>>>>>> logs.
>>>>>>
>>>>>> I'm much more tending towards the connection here, noting that
>>>>>> Andrew's original thread didn't really lead anywhere (i.e. we still
>>>>>> don't know what the panic he saw was actually caused by).
>>>>>>
>>>>>
>>>>> I'm starting to think you're on to something here.
>>>>
>>>> hmm - maybe not.
>>>> I get the same crash with "maxcpus=1"
>>>>
>>>>
>>>>
>>>>> I've put a bunch of trace throughout the functions in qinval.c
>>>>>
>>>>> It seems that everything is functioning properly, up until we go
>>>>> through the disable_nonboot_cpus() path.
>>>>> Prior to this, I see the qinval.c functions being executed on all
>>>>> cpus, and both drhd units
>>>>> Afterward, it gets stuck in queue_invalidate_wait on the first drhd
>>>>> unit.. and eventually panics.
>>>>>
>>>>> I'm not exactly sure what to make of this yet.
>>>
>>> querying status of the hardware all seems to be working correctly...
>>> it just doesn't work with querying the QINVAL_STAT_DONE state, as far
>>> as I can tell.
>>>
>>> Other register state is:
>>>
>>> (XEN) VER = 10
>>> (XEN) CAP = c0000020e60262
>>> (XEN) n_fault_reg = 1
>>> (XEN) fault_recording_offset = 200
>>> (XEN) fault_recording_reg_l = 0
>>> (XEN) fault_recording_reg_h = 0
>>> (XEN) ECAP = f0101a
>>> (XEN) GCMD = 0
>>> (XEN) GSTS = c7000000
With
#define DMA_GCMD_QIE (((u64)1) << 26)
and
#define DMA_GSTS_QIES (((u64)1) <<26)
this means qinval is still enabled at this point.
>>> (XEN) RTADDR = 137a31000
>>> (XEN) CCMD = 800000000000000
>>> (XEN) FSTS = 0
>>> (XEN) FECTL = 0
>>> (XEN) FEDATA = 4128
>>> (XEN) FEADDR = fee0000c
>>> (XEN) FEUADDR = 0
>>>
>>> (with code lifted from print_iommu_regs() )
>>>
>>>
>>> None of this looks suspicious to my untrained eye - but I'm including
>>> it here in case someone else sees something I don't.
>>
>> Xiantao, you certainly will want to give some advice here. I won't
>> be able to look into this more deeply right away.
>
> Thanks Jan. Xiantao - I'd appreciate any insight you may have.
>
> One curious thing I have found, that seems buggy to me, is that
> {dis,en}able_qinval() is being called prior to the platform quirks
> being executed.
> It appears they are being called through iommu_{en,dis}able_x2apic_IR()
That's because this setup needs to happen when interrupt (i.e.
APIC) initialization is happening, not when the IOMMUs get set
up (which is a process that assumes interrupts can already be
requested).
In effect we have
lapic_suspend() -> iommu_disable_x2apic_IR() ->
disable_intremap()/disable_qinval()
after
iommu_suspend() -> vtd_suspend() -> disable_qinval()
but the latter tail call only when !iommu_intremap, and you
have been running with interrupt remapping enabled, so only
the former code path would result in qinval getting disabled,
which is after the point of the hang.
Depending on whether ATS is in use, more than one invalidation
can be done in the processing here - could you therefore check
whether there's any sign of ATS use ("iommu=verbose" should
make you see respective messages), and if so see whether
disabling it ("ats=off") makes a difference?
Jan
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: S3 crash with VTD Queue Invalidation enabled
2013-06-05 15:14 ` Jan Beulich
@ 2013-06-05 15:25 ` Ben Guthro
2013-06-05 15:38 ` Jan Beulich
0 siblings, 1 reply; 26+ messages in thread
From: Ben Guthro @ 2013-06-05 15:25 UTC (permalink / raw)
To: Jan Beulich; +Cc: Andrew Cooper, xiantao.zhang, xen-devel
On Wed, Jun 5, 2013 at 11:14 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 05.06.13 at 15:54, Ben Guthro <ben@guthro.net> wrote:
>> On Wed, Jun 5, 2013 at 4:24 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>> On 04.06.13 at 23:09, Ben Guthro <ben@guthro.net> wrote:
>>>> On Tue, Jun 4, 2013 at 3:49 PM, Ben Guthro <ben@guthro.net> wrote:
>>>>> On Tue, Jun 4, 2013 at 3:20 PM, Ben Guthro <ben@guthro.net> wrote:
>>>>>> On Tue, Jun 4, 2013 at 10:01 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>>>>> On 04.06.13 at 14:25, Ben Guthro <ben@guthro.net> wrote:
>>>>>>>> On Tue, Jun 4, 2013 at 4:54 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>>>> Is this perhaps having some similarity with
>>>>>>>>> http://lists.xen.org/archives/html/xen-devel/2013-04/msg00343.html?
>>>>>>>>> We're clearly running single-CPU only here and there...
>>>>>>>>
>>>>>>>> We certainly should be, as we have gone through the
>>>>>>>> disable_nonboot_cpus() by this point - and I can verify that from the
>>>>>>>> logs.
>>>>>>>
>>>>>>> I'm much more tending towards the connection here, noting that
>>>>>>> Andrew's original thread didn't really lead anywhere (i.e. we still
>>>>>>> don't know what the panic he saw was actually caused by).
>>>>>>>
>>>>>>
>>>>>> I'm starting to think you're on to something here.
>>>>>
>>>>> hmm - maybe not.
>>>>> I get the same crash with "maxcpus=1"
>>>>>
>>>>>
>>>>>
>>>>>> I've put a bunch of trace throughout the functions in qinval.c
>>>>>>
>>>>>> It seems that everything is functioning properly, up until we go
>>>>>> through the disable_nonboot_cpus() path.
>>>>>> Prior to this, I see the qinval.c functions being executed on all
>>>>>> cpus, and both drhd units
>>>>>> Afterward, it gets stuck in queue_invalidate_wait on the first drhd
>>>>>> unit.. and eventually panics.
>>>>>>
>>>>>> I'm not exactly sure what to make of this yet.
>>>>
>>>> querying status of the hardware all seems to be working correctly...
>>>> it just doesn't work with querying the QINVAL_STAT_DONE state, as far
>>>> as I can tell.
>>>>
>>>> Other register state is:
>>>>
>>>> (XEN) VER = 10
>>>> (XEN) CAP = c0000020e60262
>>>> (XEN) n_fault_reg = 1
>>>> (XEN) fault_recording_offset = 200
>>>> (XEN) fault_recording_reg_l = 0
>>>> (XEN) fault_recording_reg_h = 0
>>>> (XEN) ECAP = f0101a
>>>> (XEN) GCMD = 0
>>>> (XEN) GSTS = c7000000
>
> With
>
> #define DMA_GCMD_QIE (((u64)1) << 26)
>
> and
>
> #define DMA_GSTS_QIES (((u64)1) <<26)
>
> this means qinval is still enabled at this point.
>
>>>> (XEN) RTADDR = 137a31000
>>>> (XEN) CCMD = 800000000000000
>>>> (XEN) FSTS = 0
>>>> (XEN) FECTL = 0
>>>> (XEN) FEDATA = 4128
>>>> (XEN) FEADDR = fee0000c
>>>> (XEN) FEUADDR = 0
>>>>
>>>> (with code lifted from print_iommu_regs() )
>>>>
>>>>
>>>> None of this looks suspicious to my untrained eye - but I'm including
>>>> it here in case someone else sees something I don't.
>>>
>>> Xiantao, you certainly will want to give some advice here. I won't
>>> be able to look into this more deeply right away.
>>
>> Thanks Jan. Xiantao - I'd appreciate any insight you may have.
>>
>> One curious thing I have found, that seems buggy to me, is that
>> {dis,en}able_qinval() is being called prior to the platform quirks
>> being executed.
>> It appears they are being called through iommu_{en,dis}able_x2apic_IR()
>
> That's because this setup needs to happen when interrupt (i.e.
> APIC) initialization is happening, not when the IOMMUs get set
> up (which is a process that assumes interrupts can already be
> requested).
>
> In effect we have
>
> lapic_suspend() -> iommu_disable_x2apic_IR() ->
> disable_intremap()/disable_qinval()
>
> after
>
> iommu_suspend() -> vtd_suspend() -> disable_qinval()
>
> but the latter tail call only when !iommu_intremap, and you
> have been running with interrupt remapping enabled, so only
> the former code path would result in qinval getting disabled,
> which is after the point of the hang.
>
> Depending on whether ATS is in use, more than one invalidation
> can be done in the processing here - could you therefore check
> whether there's any sign of ATS use ("iommu=verbose" should
> make you see respective messages), and if so see whether
> disabling it ("ats=off") makes a difference?
ATS does not appear to be running:
(XEN) [VT-D]dmar.c:737: Host address width 36
(XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
(XEN) [VT-D]dmar.c:412: dmaru->address = fed90000
(XEN) [VT-D]iommu.c:1197: drhd->address = fed90000 iommu->reg = ffff82c3ffd57000
(XEN) [VT-D]iommu.c:1199: cap = c0000020e60262 ecap = f0101a
(XEN) [VT-D]dmar.c:338: endpoint: 0000:00:02.0
(XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
(XEN) [VT-D]dmar.c:412: dmaru->address = fed91000
(XEN) [VT-D]iommu.c:1197: drhd->address = fed91000 iommu->reg = ffff82c3ffd56000
(XEN) [VT-D]iommu.c:1199: cap = c9008020660262 ecap = f0105a
(XEN) [VT-D]dmar.c:354: IOAPIC: 0000:f0:1f.0
(XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.0
(XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.1
(XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.2
(XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.3
(XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.4
(XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.5
(XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.6
(XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.7
(XEN) [VT-D]dmar.c:426: flags: INCLUDE_ALL
(XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:338: endpoint: 0000:00:1d.0
(XEN) [VT-D]dmar.c:338: endpoint: 0000:00:1a.0
(XEN) [VT-D]dmar.c:625: RMRR region: base_addr ba8d5000 end_address ba8ebfff
(XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
(XEN) [VT-D]dmar.c:338: endpoint: 0000:00:02.0
(XEN) [VT-D]dmar.c:625: RMRR region: base_addr bb800000 end_address bf9fffff
I would expect a line with "found ACPI_DMAR_ATSR" to be printed, if it
was found.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: S3 crash with VTD Queue Invalidation enabled
2013-06-05 15:25 ` Ben Guthro
@ 2013-06-05 15:38 ` Jan Beulich
2013-06-05 20:27 ` Ben Guthro
0 siblings, 1 reply; 26+ messages in thread
From: Jan Beulich @ 2013-06-05 15:38 UTC (permalink / raw)
To: Ben Guthro; +Cc: Andrew Cooper, xiantao.zhang, xen-devel
>>> On 05.06.13 at 17:25, Ben Guthro <ben@guthro.net> wrote:
> On Wed, Jun 5, 2013 at 11:14 AM, Jan Beulich <JBeulich@suse.com> wrote:
>> Depending on whether ATS is in use, more than one invalidation
>> can be done in the processing here - could you therefore check
>> whether there's any sign of ATS use ("iommu=verbose" should
>> make you see respective messages), and if so see whether
>> disabling it ("ats=off") makes a difference?
>
> ATS does not appear to be running:
>
> (XEN) [VT-D]dmar.c:737: Host address width 36
> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
> (XEN) [VT-D]dmar.c:412: dmaru->address = fed90000
> (XEN) [VT-D]iommu.c:1197: drhd->address = fed90000 iommu->reg = ffff82c3ffd57000
> (XEN) [VT-D]iommu.c:1199: cap = c0000020e60262 ecap = f0101a
> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:02.0
> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
> (XEN) [VT-D]dmar.c:412: dmaru->address = fed91000
> (XEN) [VT-D]iommu.c:1197: drhd->address = fed91000 iommu->reg = ffff82c3ffd56000
> (XEN) [VT-D]iommu.c:1199: cap = c9008020660262 ecap = f0105a
> (XEN) [VT-D]dmar.c:354: IOAPIC: 0000:f0:1f.0
> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.0
> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.1
> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.2
> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.3
> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.4
> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.5
> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.6
> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.7
> (XEN) [VT-D]dmar.c:426: flags: INCLUDE_ALL
> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:1d.0
> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:1a.0
> (XEN) [VT-D]dmar.c:625: RMRR region: base_addr ba8d5000 end_address
> ba8ebfff
> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:02.0
> (XEN) [VT-D]dmar.c:625: RMRR region: base_addr bb800000 end_address
> bf9fffff
>
> I would expect a line with "found ACPI_DMAR_ATSR" to be printed, if it
> was found.
Right. So one less variable.
Jan
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: S3 crash with VTD Queue Invalidation enabled
2013-06-05 15:38 ` Jan Beulich
@ 2013-06-05 20:27 ` Ben Guthro
2013-06-05 23:53 ` Ben Guthro
0 siblings, 1 reply; 26+ messages in thread
From: Ben Guthro @ 2013-06-05 20:27 UTC (permalink / raw)
To: Jan Beulich; +Cc: Andrew Cooper, xiantao.zhang, xen-devel
On Wed, Jun 5, 2013 at 11:38 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 05.06.13 at 17:25, Ben Guthro <ben@guthro.net> wrote:
>> On Wed, Jun 5, 2013 at 11:14 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>> Depending on whether ATS is in use, more than one invalidation
>>> can be done in the processing here - could you therefore check
>>> whether there's any sign of ATS use ("iommu=verbose" should
>>> make you see respective messages), and if so see whether
>>> disabling it ("ats=off") makes a difference?
>>
>> ATS does not appear to be running:
>>
>> (XEN) [VT-D]dmar.c:737: Host address width 36
>> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
>> (XEN) [VT-D]dmar.c:412: dmaru->address = fed90000
>> (XEN) [VT-D]iommu.c:1197: drhd->address = fed90000 iommu->reg = ffff82c3ffd57000
>> (XEN) [VT-D]iommu.c:1199: cap = c0000020e60262 ecap = f0101a
>> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:02.0
>> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
>> (XEN) [VT-D]dmar.c:412: dmaru->address = fed91000
>> (XEN) [VT-D]iommu.c:1197: drhd->address = fed91000 iommu->reg = ffff82c3ffd56000
>> (XEN) [VT-D]iommu.c:1199: cap = c9008020660262 ecap = f0105a
>> (XEN) [VT-D]dmar.c:354: IOAPIC: 0000:f0:1f.0
>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.0
>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.1
>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.2
>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.3
>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.4
>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.5
>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.6
>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.7
>> (XEN) [VT-D]dmar.c:426: flags: INCLUDE_ALL
>> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
>> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:1d.0
>> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:1a.0
>> (XEN) [VT-D]dmar.c:625: RMRR region: base_addr ba8d5000 end_address
>> ba8ebfff
>> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
>> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:02.0
>> (XEN) [VT-D]dmar.c:625: RMRR region: base_addr bb800000 end_address
>> bf9fffff
>>
>> I would expect a line with "found ACPI_DMAR_ATSR" to be printed, if it
>> was found.
>
> Right. So one less variable.
Some more info.
Ross Philipson provided me with a handy utility to dump a bunch more
info about the DMAR tables, and with some more trace, this appears to
be tied to the IGD.
Early in the boot process, I see queue_invalidate_wait() called for
DRHD unit 0, and 1
(unit 0 is wired up to the IGD, unit 1 is everything else)
Up until i915 does the following, I see that unit being flushed with
queue_invalidate_wait() :
[ 0.704537] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
[ 0.704537] ENERGY_PERF_BIAS: View and update with x86_energy_p
(XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
(XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
[ 1.983028] [drm] GMBUS [i915 gmbus dpb] timed out, falling back to
bit banging on pin 5
[ 2.253551] fbcon: inteldrmfb (fb0) is primary device
[ 3.111838] Console: switching to colour frame buffer device 170x48
[ 3.171631] i915 0000:00:02.0: fb0: inteldrmfb frame buffer device
[ 3.171634] i915 0000:00:02.0: registered panic notifier
[ 3.173339] acpi device:00: registered as cooling_device1
[ 3.173401] ACPI: Video Device [VID] (multi-head: yes rom: no post: no)
[ 3.173962] input: Video Bus as
/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input4
[ 3.174232] [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on minor 0
[ 3.174258] ahci 0000:00:1f.2: version 3.0
[ 3.174270] xen: registering gsi 19 triggering 0 polarity 1
[ 3.174274] Already setup the GSI :19
After that - the unit never seems to be flushed.
...until we enter into the S3 hypercall, which loops over all DRHD
units, and explicitly flushes all of them via iommu_flush_all()
It is at that point that it hangs up when talking to the device that
the IGD is plumbed up to.
Does this point to something in the i915 driver doing something that
is incompatible with Xen?
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: S3 crash with VTD Queue Invalidation enabled
2013-06-05 20:27 ` Ben Guthro
@ 2013-06-05 23:53 ` Ben Guthro
2013-06-06 6:58 ` Jan Beulich
2013-06-14 8:38 ` Jan Beulich
0 siblings, 2 replies; 26+ messages in thread
From: Ben Guthro @ 2013-06-05 23:53 UTC (permalink / raw)
To: Jan Beulich; +Cc: Andrew Cooper, xiantao.zhang, xen-devel
On Wed, Jun 5, 2013 at 4:27 PM, Ben Guthro <ben@guthro.net> wrote:
> On Wed, Jun 5, 2013 at 11:38 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>> On 05.06.13 at 17:25, Ben Guthro <ben@guthro.net> wrote:
>>> On Wed, Jun 5, 2013 at 11:14 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> Depending on whether ATS is in use, more than one invalidation
>>>> can be done in the processing here - could you therefore check
>>>> whether there's any sign of ATS use ("iommu=verbose" should
>>>> make you see respective messages), and if so see whether
>>>> disabling it ("ats=off") makes a difference?
>>>
>>> ATS does not appear to be running:
>>>
>>> (XEN) [VT-D]dmar.c:737: Host address width 36
>>> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
>>> (XEN) [VT-D]dmar.c:412: dmaru->address = fed90000
>>> (XEN) [VT-D]iommu.c:1197: drhd->address = fed90000 iommu->reg = ffff82c3ffd57000
>>> (XEN) [VT-D]iommu.c:1199: cap = c0000020e60262 ecap = f0101a
>>> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:02.0
>>> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
>>> (XEN) [VT-D]dmar.c:412: dmaru->address = fed91000
>>> (XEN) [VT-D]iommu.c:1197: drhd->address = fed91000 iommu->reg = ffff82c3ffd56000
>>> (XEN) [VT-D]iommu.c:1199: cap = c9008020660262 ecap = f0105a
>>> (XEN) [VT-D]dmar.c:354: IOAPIC: 0000:f0:1f.0
>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.0
>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.1
>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.2
>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.3
>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.4
>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.5
>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.6
>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.7
>>> (XEN) [VT-D]dmar.c:426: flags: INCLUDE_ALL
>>> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
>>> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:1d.0
>>> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:1a.0
>>> (XEN) [VT-D]dmar.c:625: RMRR region: base_addr ba8d5000 end_address
>>> ba8ebfff
>>> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
>>> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:02.0
>>> (XEN) [VT-D]dmar.c:625: RMRR region: base_addr bb800000 end_address
>>> bf9fffff
>>>
>>> I would expect a line with "found ACPI_DMAR_ATSR" to be printed, if it
>>> was found.
>>
>> Right. So one less variable.
>
> Some more info.
> Ross Philipson provided me with a handy utility to dump a bunch more
> info about the DMAR tables, and with some more trace, this appears to
> be tied to the IGD.
>
> Early in the boot process, I see queue_invalidate_wait() called for
> DRHD unit 0, and 1
> (unit 0 is wired up to the IGD, unit 1 is everything else)
>
> Up until i915 does the following, I see that unit being flushed with
> queue_invalidate_wait() :
>
> [ 0.704537] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
> [ 0.704537] ENERGY_PERF_BIAS: View and update with x86_energy_p
> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
> [ 1.983028] [drm] GMBUS [i915 gmbus dpb] timed out, falling back to
> bit banging on pin 5
> [ 2.253551] fbcon: inteldrmfb (fb0) is primary device
> [ 3.111838] Console: switching to colour frame buffer device 170x48
> [ 3.171631] i915 0000:00:02.0: fb0: inteldrmfb frame buffer device
> [ 3.171634] i915 0000:00:02.0: registered panic notifier
> [ 3.173339] acpi device:00: registered as cooling_device1
> [ 3.173401] ACPI: Video Device [VID] (multi-head: yes rom: no post: no)
> [ 3.173962] input: Video Bus as
> /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input4
> [ 3.174232] [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on minor 0
> [ 3.174258] ahci 0000:00:1f.2: version 3.0
> [ 3.174270] xen: registering gsi 19 triggering 0 polarity 1
> [ 3.174274] Already setup the GSI :19
>
>
> After that - the unit never seems to be flushed.
>
> ...until we enter into the S3 hypercall, which loops over all DRHD
> units, and explicitly flushes all of them via iommu_flush_all()
>
> It is at that point that it hangs up when talking to the device that
> the IGD is plumbed up to.
>
>
> Does this point to something in the i915 driver doing something that
> is incompatible with Xen?
I actually separated it from the S3 hypercall, adding a new debug key
'F' - to just call iommu_flush_all()
I can crash it on demand with this.
Booting with "i915.modeset=0 single" (to prevent both KMS, and Xorg) -
it does not occur.
So, that pretty much narrows it down to the IGD, in my mind.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: S3 crash with VTD Queue Invalidation enabled
2013-06-05 23:53 ` Ben Guthro
@ 2013-06-06 6:58 ` Jan Beulich
2013-06-06 15:06 ` Zhang, Xiantao
2013-06-14 8:38 ` Jan Beulich
1 sibling, 1 reply; 26+ messages in thread
From: Jan Beulich @ 2013-06-06 6:58 UTC (permalink / raw)
To: Ben Guthro; +Cc: Andrew Cooper, xiantao.zhang, xen-devel
>>> On 06.06.13 at 01:53, Ben Guthro <ben@guthro.net> wrote:
> On Wed, Jun 5, 2013 at 4:27 PM, Ben Guthro <ben@guthro.net> wrote:
>> On Wed, Jun 5, 2013 at 11:38 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>> On 05.06.13 at 17:25, Ben Guthro <ben@guthro.net> wrote:
>>>> On Wed, Jun 5, 2013 at 11:14 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>> Depending on whether ATS is in use, more than one invalidation
>>>>> can be done in the processing here - could you therefore check
>>>>> whether there's any sign of ATS use ("iommu=verbose" should
>>>>> make you see respective messages), and if so see whether
>>>>> disabling it ("ats=off") makes a difference?
>>>>
>>>> ATS does not appear to be running:
>>>>
>>>> (XEN) [VT-D]dmar.c:737: Host address width 36
>>>> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
>>>> (XEN) [VT-D]dmar.c:412: dmaru->address = fed90000
>>>> (XEN) [VT-D]iommu.c:1197: drhd->address = fed90000 iommu->reg = ffff82c3ffd57000
>>>> (XEN) [VT-D]iommu.c:1199: cap = c0000020e60262 ecap = f0101a
>>>> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:02.0
>>>> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
>>>> (XEN) [VT-D]dmar.c:412: dmaru->address = fed91000
>>>> (XEN) [VT-D]iommu.c:1197: drhd->address = fed91000 iommu->reg = ffff82c3ffd56000
>>>> (XEN) [VT-D]iommu.c:1199: cap = c9008020660262 ecap = f0105a
>>>> (XEN) [VT-D]dmar.c:354: IOAPIC: 0000:f0:1f.0
>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.0
>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.1
>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.2
>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.3
>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.4
>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.5
>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.6
>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.7
>>>> (XEN) [VT-D]dmar.c:426: flags: INCLUDE_ALL
>>>> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
>>>> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:1d.0
>>>> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:1a.0
>>>> (XEN) [VT-D]dmar.c:625: RMRR region: base_addr ba8d5000 end_address
>>>> ba8ebfff
>>>> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
>>>> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:02.0
>>>> (XEN) [VT-D]dmar.c:625: RMRR region: base_addr bb800000 end_address
>>>> bf9fffff
>>>>
>>>> I would expect a line with "found ACPI_DMAR_ATSR" to be printed, if it
>>>> was found.
>>>
>>> Right. So one less variable.
>>
>> Some more info.
>> Ross Philipson provided me with a handy utility to dump a bunch more
>> info about the DMAR tables, and with some more trace, this appears to
>> be tied to the IGD.
>>
>> Early in the boot process, I see queue_invalidate_wait() called for
>> DRHD unit 0, and 1
>> (unit 0 is wired up to the IGD, unit 1 is everything else)
>>
>> Up until i915 does the following, I see that unit being flushed with
>> queue_invalidate_wait() :
>>
>> [ 0.704537] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
>> [ 0.704537] ENERGY_PERF_BIAS: View and update with x86_energy_p
>> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
>> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
>> [ 1.983028] [drm] GMBUS [i915 gmbus dpb] timed out, falling back to
>> bit banging on pin 5
>> [ 2.253551] fbcon: inteldrmfb (fb0) is primary device
>> [ 3.111838] Console: switching to colour frame buffer device 170x48
>> [ 3.171631] i915 0000:00:02.0: fb0: inteldrmfb frame buffer device
>> [ 3.171634] i915 0000:00:02.0: registered panic notifier
>> [ 3.173339] acpi device:00: registered as cooling_device1
>> [ 3.173401] ACPI: Video Device [VID] (multi-head: yes rom: no post: no)
>> [ 3.173962] input: Video Bus as
>> /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input4
>> [ 3.174232] [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on
> minor 0
>> [ 3.174258] ahci 0000:00:1f.2: version 3.0
>> [ 3.174270] xen: registering gsi 19 triggering 0 polarity 1
>> [ 3.174274] Already setup the GSI :19
>>
>>
>> After that - the unit never seems to be flushed.
>>
>> ...until we enter into the S3 hypercall, which loops over all DRHD
>> units, and explicitly flushes all of them via iommu_flush_all()
>>
>> It is at that point that it hangs up when talking to the device that
>> the IGD is plumbed up to.
>>
>>
>> Does this point to something in the i915 driver doing something that
>> is incompatible with Xen?
>
> I actually separated it from the S3 hypercall, adding a new debug key
> 'F' - to just call iommu_flush_all()
> I can crash it on demand with this.
>
> Booting with "i915.modeset=0 single" (to prevent both KMS, and Xorg) -
> it does not occur.
> So, that pretty much narrows it down to the IGD, in my mind.
Indeed, I agree. Yet I can't in any way comment on what or why.
Xiantao (perhaps some graphics person would good to be Cc-ed
here too)?
Jan
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: S3 crash with VTD Queue Invalidation enabled
2013-06-06 6:58 ` Jan Beulich
@ 2013-06-06 15:06 ` Zhang, Xiantao
2013-06-06 15:07 ` Ben Guthro
0 siblings, 1 reply; 26+ messages in thread
From: Zhang, Xiantao @ 2013-06-06 15:06 UTC (permalink / raw)
To: Jan Beulich, Ben Guthro; +Cc: Andrew Cooper, Zhang, Xiantao, xen-devel
> -----Original Message-----
> From: Jan Beulich [mailto:JBeulich@suse.com]
> Sent: Thursday, June 06, 2013 2:59 PM
> To: Ben Guthro
> Cc: Andrew Cooper; Zhang, Xiantao; xen-devel
> Subject: Re: [Xen-devel] S3 crash with VTD Queue Invalidation enabled
>
> >>> On 06.06.13 at 01:53, Ben Guthro <ben@guthro.net> wrote:
> > On Wed, Jun 5, 2013 at 4:27 PM, Ben Guthro <ben@guthro.net> wrote:
> >> On Wed, Jun 5, 2013 at 11:38 AM, Jan Beulich <JBeulich@suse.com> wrote:
> >>>>>> On 05.06.13 at 17:25, Ben Guthro <ben@guthro.net> wrote:
> >>>> On Wed, Jun 5, 2013 at 11:14 AM, Jan Beulich <JBeulich@suse.com>
> wrote:
> >>>>> Depending on whether ATS is in use, more than one invalidation
> >>>>> can be done in the processing here - could you therefore check
> >>>>> whether there's any sign of ATS use ("iommu=verbose" should
> >>>>> make you see respective messages), and if so see whether
> >>>>> disabling it ("ats=off") makes a difference?
> >>>>
> >>>> ATS does not appear to be running:
> >>>>
> >>>> (XEN) [VT-D]dmar.c:737: Host address width 36
> >>>> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
> >>>> (XEN) [VT-D]dmar.c:412: dmaru->address = fed90000
> >>>> (XEN) [VT-D]iommu.c:1197: drhd->address = fed90000 iommu->reg =
> ffff82c3ffd57000
> >>>> (XEN) [VT-D]iommu.c:1199: cap = c0000020e60262 ecap = f0101a
> >>>> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:02.0
> >>>> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
> >>>> (XEN) [VT-D]dmar.c:412: dmaru->address = fed91000
> >>>> (XEN) [VT-D]iommu.c:1197: drhd->address = fed91000 iommu->reg =
> ffff82c3ffd56000
> >>>> (XEN) [VT-D]iommu.c:1199: cap = c9008020660262 ecap = f0105a
> >>>> (XEN) [VT-D]dmar.c:354: IOAPIC: 0000:f0:1f.0
> >>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.0
> >>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.1
> >>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.2
> >>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.3
> >>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.4
> >>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.5
> >>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.6
> >>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.7
> >>>> (XEN) [VT-D]dmar.c:426: flags: INCLUDE_ALL
> >>>> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
> >>>> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:1d.0
> >>>> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:1a.0
> >>>> (XEN) [VT-D]dmar.c:625: RMRR region: base_addr ba8d5000
> end_address
> >>>> ba8ebfff
> >>>> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
> >>>> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:02.0
> >>>> (XEN) [VT-D]dmar.c:625: RMRR region: base_addr bb800000
> end_address
> >>>> bf9fffff
> >>>>
> >>>> I would expect a line with "found ACPI_DMAR_ATSR" to be printed, if it
> >>>> was found.
> >>>
> >>> Right. So one less variable.
> >>
> >> Some more info.
> >> Ross Philipson provided me with a handy utility to dump a bunch more
> >> info about the DMAR tables, and with some more trace, this appears to
> >> be tied to the IGD.
> >>
> >> Early in the boot process, I see queue_invalidate_wait() called for
> >> DRHD unit 0, and 1
> >> (unit 0 is wired up to the IGD, unit 1 is everything else)
> >>
> >> Up until i915 does the following, I see that unit being flushed with
> >> queue_invalidate_wait() :
> >>
> >> [ 0.704537] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
> >> [ 0.704537] ENERGY_PERF_BIAS: View and update with x86_energy_p
> >> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
> >> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
> >> [ 1.983028] [drm] GMBUS [i915 gmbus dpb] timed out, falling back to
> >> bit banging on pin 5
> >> [ 2.253551] fbcon: inteldrmfb (fb0) is primary device
> >> [ 3.111838] Console: switching to colour frame buffer device 170x48
> >> [ 3.171631] i915 0000:00:02.0: fb0: inteldrmfb frame buffer device
> >> [ 3.171634] i915 0000:00:02.0: registered panic notifier
> >> [ 3.173339] acpi device:00: registered as cooling_device1
> >> [ 3.173401] ACPI: Video Device [VID] (multi-head: yes rom: no post: no)
> >> [ 3.173962] input: Video Bus as
> >>
> /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input4
> >> [ 3.174232] [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on
> > minor 0
> >> [ 3.174258] ahci 0000:00:1f.2: version 3.0
> >> [ 3.174270] xen: registering gsi 19 triggering 0 polarity 1
> >> [ 3.174274] Already setup the GSI :19
> >>
> >>
> >> After that - the unit never seems to be flushed.
> >>
> >> ...until we enter into the S3 hypercall, which loops over all DRHD
> >> units, and explicitly flushes all of them via iommu_flush_all()
> >>
> >> It is at that point that it hangs up when talking to the device that
> >> the IGD is plumbed up to.
> >>
> >>
> >> Does this point to something in the i915 driver doing something that
> >> is incompatible with Xen?
> >
> > I actually separated it from the S3 hypercall, adding a new debug key
> > 'F' - to just call iommu_flush_all()
> > I can crash it on demand with this.
> >
> > Booting with "i915.modeset=0 single" (to prevent both KMS, and Xorg) -
> > it does not occur.
> > So, that pretty much narrows it down to the IGD, in my mind.
>
> Indeed, I agree. Yet I can't in any way comment on what or why.
> Xiantao (perhaps some graphics person would good to be Cc-ed
> here too)?
Hi, Jan/Ben
Thanks for your analysis! Could you try to enable "snb_igd_quirk" to have a try ? thanks!
Xiantao
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: S3 crash with VTD Queue Invalidation enabled
2013-06-06 15:06 ` Zhang, Xiantao
@ 2013-06-06 15:07 ` Ben Guthro
2013-06-06 15:13 ` Zhang, Xiantao
0 siblings, 1 reply; 26+ messages in thread
From: Ben Guthro @ 2013-06-06 15:07 UTC (permalink / raw)
To: Zhang, Xiantao; +Cc: Andrew Cooper, Ben Guthro, Jan Beulich, xen-devel
On Jun 6, 2013, at 11:06 AM, "Zhang, Xiantao" <xiantao.zhang@intel.com> wrote:
>
>
>> -----Original Message-----
>> From: Jan Beulich [mailto:JBeulich@suse.com]
>> Sent: Thursday, June 06, 2013 2:59 PM
>> To: Ben Guthro
>> Cc: Andrew Cooper; Zhang, Xiantao; xen-devel
>> Subject: Re: [Xen-devel] S3 crash with VTD Queue Invalidation enabled
>>
>>>>> On 06.06.13 at 01:53, Ben Guthro <ben@guthro.net> wrote:
>>> On Wed, Jun 5, 2013 at 4:27 PM, Ben Guthro <ben@guthro.net> wrote:
>>>> On Wed, Jun 5, 2013 at 11:38 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>>>>> On 05.06.13 at 17:25, Ben Guthro <ben@guthro.net> wrote:
>>>>>> On Wed, Jun 5, 2013 at 11:14 AM, Jan Beulich <JBeulich@suse.com>
>> wrote:
>>>>>>> Depending on whether ATS is in use, more than one invalidation
>>>>>>> can be done in the processing here - could you therefore check
>>>>>>> whether there's any sign of ATS use ("iommu=verbose" should
>>>>>>> make you see respective messages), and if so see whether
>>>>>>> disabling it ("ats=off") makes a difference?
>>>>>>
>>>>>> ATS does not appear to be running:
>>>>>>
>>>>>> (XEN) [VT-D]dmar.c:737: Host address width 36
>>>>>> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
>>>>>> (XEN) [VT-D]dmar.c:412: dmaru->address = fed90000
>>>>>> (XEN) [VT-D]iommu.c:1197: drhd->address = fed90000 iommu->reg =
>> ffff82c3ffd57000
>>>>>> (XEN) [VT-D]iommu.c:1199: cap = c0000020e60262 ecap = f0101a
>>>>>> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:02.0
>>>>>> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
>>>>>> (XEN) [VT-D]dmar.c:412: dmaru->address = fed91000
>>>>>> (XEN) [VT-D]iommu.c:1197: drhd->address = fed91000 iommu->reg =
>> ffff82c3ffd56000
>>>>>> (XEN) [VT-D]iommu.c:1199: cap = c9008020660262 ecap = f0105a
>>>>>> (XEN) [VT-D]dmar.c:354: IOAPIC: 0000:f0:1f.0
>>>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.0
>>>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.1
>>>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.2
>>>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.3
>>>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.4
>>>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.5
>>>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.6
>>>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.7
>>>>>> (XEN) [VT-D]dmar.c:426: flags: INCLUDE_ALL
>>>>>> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
>>>>>> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:1d.0
>>>>>> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:1a.0
>>>>>> (XEN) [VT-D]dmar.c:625: RMRR region: base_addr ba8d5000
>> end_address
>>>>>> ba8ebfff
>>>>>> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
>>>>>> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:02.0
>>>>>> (XEN) [VT-D]dmar.c:625: RMRR region: base_addr bb800000
>> end_address
>>>>>> bf9fffff
>>>>>>
>>>>>> I would expect a line with "found ACPI_DMAR_ATSR" to be printed, if it
>>>>>> was found.
>>>>>
>>>>> Right. So one less variable.
>>>>
>>>> Some more info.
>>>> Ross Philipson provided me with a handy utility to dump a bunch more
>>>> info about the DMAR tables, and with some more trace, this appears to
>>>> be tied to the IGD.
>>>>
>>>> Early in the boot process, I see queue_invalidate_wait() called for
>>>> DRHD unit 0, and 1
>>>> (unit 0 is wired up to the IGD, unit 1 is everything else)
>>>>
>>>> Up until i915 does the following, I see that unit being flushed with
>>>> queue_invalidate_wait() :
>>>>
>>>> [ 0.704537] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
>>>> [ 0.704537] ENERGY_PERF_BIAS: View and update with x86_energy_p
>>>> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
>>>> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
>>>> [ 1.983028] [drm] GMBUS [i915 gmbus dpb] timed out, falling back to
>>>> bit banging on pin 5
>>>> [ 2.253551] fbcon: inteldrmfb (fb0) is primary device
>>>> [ 3.111838] Console: switching to colour frame buffer device 170x48
>>>> [ 3.171631] i915 0000:00:02.0: fb0: inteldrmfb frame buffer device
>>>> [ 3.171634] i915 0000:00:02.0: registered panic notifier
>>>> [ 3.173339] acpi device:00: registered as cooling_device1
>>>> [ 3.173401] ACPI: Video Device [VID] (multi-head: yes rom: no post: no)
>>>> [ 3.173962] input: Video Bus as
>> /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input4
>>>> [ 3.174232] [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on
>>> minor 0
>>>> [ 3.174258] ahci 0000:00:1f.2: version 3.0
>>>> [ 3.174270] xen: registering gsi 19 triggering 0 polarity 1
>>>> [ 3.174274] Already setup the GSI :19
>>>>
>>>>
>>>> After that - the unit never seems to be flushed.
>>>>
>>>> ...until we enter into the S3 hypercall, which loops over all DRHD
>>>> units, and explicitly flushes all of them via iommu_flush_all()
>>>>
>>>> It is at that point that it hangs up when talking to the device that
>>>> the IGD is plumbed up to.
>>>>
>>>>
>>>> Does this point to something in the i915 driver doing something that
>>>> is incompatible with Xen?
>>>
>>> I actually separated it from the S3 hypercall, adding a new debug key
>>> 'F' - to just call iommu_flush_all()
>>> I can crash it on demand with this.
>>>
>>> Booting with "i915.modeset=0 single" (to prevent both KMS, and Xorg) -
>>> it does not occur.
>>> So, that pretty much narrows it down to the IGD, in my mind.
>>
>> Indeed, I agree. Yet I can't in any way comment on what or why.
>> Xiantao (perhaps some graphics person would good to be Cc-ed
>> here too)?
> Hi, Jan/Ben
> Thanks for your analysis! Could you try to enable "snb_igd_quirk" to have a try ? thanks!
> Xiantao
>
Thanks for your reply. I tried this param yesterday, but it did not
change the behavior.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: S3 crash with VTD Queue Invalidation enabled
2013-06-06 15:07 ` Ben Guthro
@ 2013-06-06 15:13 ` Zhang, Xiantao
2013-06-06 15:17 ` Ben Guthro
0 siblings, 1 reply; 26+ messages in thread
From: Zhang, Xiantao @ 2013-06-06 15:13 UTC (permalink / raw)
To: Ben Guthro
Cc: Andrew Cooper, Zhang, Xiantao, Ben Guthro, Jan Beulich, xen-devel
> -----Original Message-----
> From: Ben Guthro [mailto:ben.guthro@gmail.com]
> Sent: Thursday, June 06, 2013 11:08 PM
> To: Zhang, Xiantao
> Cc: Jan Beulich; Ben Guthro; Andrew Cooper; xen-devel
> Subject: Re: [Xen-devel] S3 crash with VTD Queue Invalidation enabled
>
> On Jun 6, 2013, at 11:06 AM, "Zhang, Xiantao" <xiantao.zhang@intel.com>
> wrote:
>
> >
> >
> >> -----Original Message-----
> >> From: Jan Beulich [mailto:JBeulich@suse.com]
> >> Sent: Thursday, June 06, 2013 2:59 PM
> >> To: Ben Guthro
> >> Cc: Andrew Cooper; Zhang, Xiantao; xen-devel
> >> Subject: Re: [Xen-devel] S3 crash with VTD Queue Invalidation enabled
> >>
> >>>>> On 06.06.13 at 01:53, Ben Guthro <ben@guthro.net> wrote:
> >>> On Wed, Jun 5, 2013 at 4:27 PM, Ben Guthro <ben@guthro.net> wrote:
> >>>> On Wed, Jun 5, 2013 at 11:38 AM, Jan Beulich <JBeulich@suse.com>
> wrote:
> >>>>>>>> On 05.06.13 at 17:25, Ben Guthro <ben@guthro.net> wrote:
> >>>>>> On Wed, Jun 5, 2013 at 11:14 AM, Jan Beulich <JBeulich@suse.com>
> >> wrote:
> >>>>>>> Depending on whether ATS is in use, more than one invalidation
> >>>>>>> can be done in the processing here - could you therefore check
> >>>>>>> whether there's any sign of ATS use ("iommu=verbose" should
> >>>>>>> make you see respective messages), and if so see whether
> >>>>>>> disabling it ("ats=off") makes a difference?
> >>>>>>
> >>>>>> ATS does not appear to be running:
> >>>>>>
> >>>>>> (XEN) [VT-D]dmar.c:737: Host address width 36
> >>>>>> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
> >>>>>> (XEN) [VT-D]dmar.c:412: dmaru->address = fed90000
> >>>>>> (XEN) [VT-D]iommu.c:1197: drhd->address = fed90000 iommu->reg =
> >> ffff82c3ffd57000
> >>>>>> (XEN) [VT-D]iommu.c:1199: cap = c0000020e60262 ecap = f0101a
> >>>>>> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:02.0
> >>>>>> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
> >>>>>> (XEN) [VT-D]dmar.c:412: dmaru->address = fed91000
> >>>>>> (XEN) [VT-D]iommu.c:1197: drhd->address = fed91000 iommu->reg =
> >> ffff82c3ffd56000
> >>>>>> (XEN) [VT-D]iommu.c:1199: cap = c9008020660262 ecap = f0105a
> >>>>>> (XEN) [VT-D]dmar.c:354: IOAPIC: 0000:f0:1f.0
> >>>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.0
> >>>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.1
> >>>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.2
> >>>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.3
> >>>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.4
> >>>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.5
> >>>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.6
> >>>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.7
> >>>>>> (XEN) [VT-D]dmar.c:426: flags: INCLUDE_ALL
> >>>>>> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
> >>>>>> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:1d.0
> >>>>>> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:1a.0
> >>>>>> (XEN) [VT-D]dmar.c:625: RMRR region: base_addr ba8d5000
> >> end_address
> >>>>>> ba8ebfff
> >>>>>> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
> >>>>>> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:02.0
> >>>>>> (XEN) [VT-D]dmar.c:625: RMRR region: base_addr bb800000
> >> end_address
> >>>>>> bf9fffff
> >>>>>>
> >>>>>> I would expect a line with "found ACPI_DMAR_ATSR" to be printed, if it
> >>>>>> was found.
> >>>>>
> >>>>> Right. So one less variable.
> >>>>
> >>>> Some more info.
> >>>> Ross Philipson provided me with a handy utility to dump a bunch more
> >>>> info about the DMAR tables, and with some more trace, this appears to
> >>>> be tied to the IGD.
> >>>>
> >>>> Early in the boot process, I see queue_invalidate_wait() called for
> >>>> DRHD unit 0, and 1
> >>>> (unit 0 is wired up to the IGD, unit 1 is everything else)
> >>>>
> >>>> Up until i915 does the following, I see that unit being flushed with
> >>>> queue_invalidate_wait() :
> >>>>
> >>>> [ 0.704537] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
> >>>> [ 0.704537] ENERGY_PERF_BIAS: View and update with x86_energy_p
> >>>> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
> >>>> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
> >>>> [ 1.983028] [drm] GMBUS [i915 gmbus dpb] timed out, falling back to
> >>>> bit banging on pin 5
> >>>> [ 2.253551] fbcon: inteldrmfb (fb0) is primary device
> >>>> [ 3.111838] Console: switching to colour frame buffer device 170x48
> >>>> [ 3.171631] i915 0000:00:02.0: fb0: inteldrmfb frame buffer device
> >>>> [ 3.171634] i915 0000:00:02.0: registered panic notifier
> >>>> [ 3.173339] acpi device:00: registered as cooling_device1
> >>>> [ 3.173401] ACPI: Video Device [VID] (multi-head: yes rom: no post: no)
> >>>> [ 3.173962] input: Video Bus as
> >>
> /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input4
> >>>> [ 3.174232] [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on
> >>> minor 0
> >>>> [ 3.174258] ahci 0000:00:1f.2: version 3.0
> >>>> [ 3.174270] xen: registering gsi 19 triggering 0 polarity 1
> >>>> [ 3.174274] Already setup the GSI :19
> >>>>
> >>>>
> >>>> After that - the unit never seems to be flushed.
> >>>>
> >>>> ...until we enter into the S3 hypercall, which loops over all DRHD
> >>>> units, and explicitly flushes all of them via iommu_flush_all()
> >>>>
> >>>> It is at that point that it hangs up when talking to the device that
> >>>> the IGD is plumbed up to.
> >>>>
> >>>>
> >>>> Does this point to something in the i915 driver doing something that
> >>>> is incompatible with Xen?
> >>>
> >>> I actually separated it from the S3 hypercall, adding a new debug key
> >>> 'F' - to just call iommu_flush_all()
> >>> I can crash it on demand with this.
> >>>
> >>> Booting with "i915.modeset=0 single" (to prevent both KMS, and Xorg) -
> >>> it does not occur.
> >>> So, that pretty much narrows it down to the IGD, in my mind.
> >>
> >> Indeed, I agree. Yet I can't in any way comment on what or why.
> >> Xiantao (perhaps some graphics person would good to be Cc-ed
> >> here too)?
> > Hi, Jan/Ben
> > Thanks for your analysis! Could you try to enable "snb_igd_quirk" to have a
> try ? thanks!
> > Xiantao
> >
>
>
> Thanks for your reply. I tried this param yesterday, but it did not
> change the behavior.
Okay, I recalled one bug in IGD i915 driver is found recently, and it may bring some errors to VT-d, and should be fixed in latest kernel. Could you try latest kernel 3.9.4 or 3.10-rcx ?
Xiantao
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: S3 crash with VTD Queue Invalidation enabled
2013-06-06 15:13 ` Zhang, Xiantao
@ 2013-06-06 15:17 ` Ben Guthro
2013-06-07 1:33 ` Zhang, Xiantao
0 siblings, 1 reply; 26+ messages in thread
From: Ben Guthro @ 2013-06-06 15:17 UTC (permalink / raw)
To: Zhang, Xiantao; +Cc: Andrew Cooper, Ben Guthro, Jan Beulich, xen-devel
On Jun 6, 2013, at 11:13 AM, "Zhang, Xiantao" <xiantao.zhang@intel.com> wrote:
>
>
>> -----Original Message-----
>> From: Ben Guthro [mailto:ben.guthro@gmail.com]
>> Sent: Thursday, June 06, 2013 11:08 PM
>> To: Zhang, Xiantao
>> Cc: Jan Beulich; Ben Guthro; Andrew Cooper; xen-devel
>> Subject: Re: [Xen-devel] S3 crash with VTD Queue Invalidation enabled
>>
>> On Jun 6, 2013, at 11:06 AM, "Zhang, Xiantao" <xiantao.zhang@intel.com>
>> wrote:
>>
>>>
>>>
>>>> -----Original Message-----
>>>> From: Jan Beulich [mailto:JBeulich@suse.com]
>>>> Sent: Thursday, June 06, 2013 2:59 PM
>>>> To: Ben Guthro
>>>> Cc: Andrew Cooper; Zhang, Xiantao; xen-devel
>>>> Subject: Re: [Xen-devel] S3 crash with VTD Queue Invalidation enabled
>>>>
>>>>>>> On 06.06.13 at 01:53, Ben Guthro <ben@guthro.net> wrote:
>>>>> On Wed, Jun 5, 2013 at 4:27 PM, Ben Guthro <ben@guthro.net> wrote:
>>>>>> On Wed, Jun 5, 2013 at 11:38 AM, Jan Beulich <JBeulich@suse.com>
>> wrote:
>>>>>>>>>> On 05.06.13 at 17:25, Ben Guthro <ben@guthro.net> wrote:
>>>>>>>> On Wed, Jun 5, 2013 at 11:14 AM, Jan Beulich <JBeulich@suse.com>
>>>> wrote:
>>>>>>>>> Depending on whether ATS is in use, more than one invalidation
>>>>>>>>> can be done in the processing here - could you therefore check
>>>>>>>>> whether there's any sign of ATS use ("iommu=verbose" should
>>>>>>>>> make you see respective messages), and if so see whether
>>>>>>>>> disabling it ("ats=off") makes a difference?
>>>>>>>>
>>>>>>>> ATS does not appear to be running:
>>>>>>>>
>>>>>>>> (XEN) [VT-D]dmar.c:737: Host address width 36
>>>>>>>> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
>>>>>>>> (XEN) [VT-D]dmar.c:412: dmaru->address = fed90000
>>>>>>>> (XEN) [VT-D]iommu.c:1197: drhd->address = fed90000 iommu->reg =
>>>> ffff82c3ffd57000
>>>>>>>> (XEN) [VT-D]iommu.c:1199: cap = c0000020e60262 ecap = f0101a
>>>>>>>> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:02.0
>>>>>>>> (XEN) [VT-D]dmar.c:751: found ACPI_DMAR_DRHD:
>>>>>>>> (XEN) [VT-D]dmar.c:412: dmaru->address = fed91000
>>>>>>>> (XEN) [VT-D]iommu.c:1197: drhd->address = fed91000 iommu->reg =
>>>> ffff82c3ffd56000
>>>>>>>> (XEN) [VT-D]iommu.c:1199: cap = c9008020660262 ecap = f0105a
>>>>>>>> (XEN) [VT-D]dmar.c:354: IOAPIC: 0000:f0:1f.0
>>>>>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.0
>>>>>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.1
>>>>>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.2
>>>>>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.3
>>>>>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.4
>>>>>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.5
>>>>>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.6
>>>>>>>> (XEN) [VT-D]dmar.c:332: MSI HPET: 0000:00:0f.7
>>>>>>>> (XEN) [VT-D]dmar.c:426: flags: INCLUDE_ALL
>>>>>>>> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
>>>>>>>> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:1d.0
>>>>>>>> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:1a.0
>>>>>>>> (XEN) [VT-D]dmar.c:625: RMRR region: base_addr ba8d5000
>>>> end_address
>>>>>>>> ba8ebfff
>>>>>>>> (XEN) [VT-D]dmar.c:756: found ACPI_DMAR_RMRR:
>>>>>>>> (XEN) [VT-D]dmar.c:338: endpoint: 0000:00:02.0
>>>>>>>> (XEN) [VT-D]dmar.c:625: RMRR region: base_addr bb800000
>>>> end_address
>>>>>>>> bf9fffff
>>>>>>>>
>>>>>>>> I would expect a line with "found ACPI_DMAR_ATSR" to be printed, if it
>>>>>>>> was found.
>>>>>>>
>>>>>>> Right. So one less variable.
>>>>>>
>>>>>> Some more info.
>>>>>> Ross Philipson provided me with a handy utility to dump a bunch more
>>>>>> info about the DMAR tables, and with some more trace, this appears to
>>>>>> be tied to the IGD.
>>>>>>
>>>>>> Early in the boot process, I see queue_invalidate_wait() called for
>>>>>> DRHD unit 0, and 1
>>>>>> (unit 0 is wired up to the IGD, unit 1 is everything else)
>>>>>>
>>>>>> Up until i915 does the following, I see that unit being flushed with
>>>>>> queue_invalidate_wait() :
>>>>>>
>>>>>> [ 0.704537] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
>>>>>> [ 0.704537] ENERGY_PERF_BIAS: View and update with x86_energy_p
>>>>>> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
>>>>>> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
>>>>>> [ 1.983028] [drm] GMBUS [i915 gmbus dpb] timed out, falling back to
>>>>>> bit banging on pin 5
>>>>>> [ 2.253551] fbcon: inteldrmfb (fb0) is primary device
>>>>>> [ 3.111838] Console: switching to colour frame buffer device 170x48
>>>>>> [ 3.171631] i915 0000:00:02.0: fb0: inteldrmfb frame buffer device
>>>>>> [ 3.171634] i915 0000:00:02.0: registered panic notifier
>>>>>> [ 3.173339] acpi device:00: registered as cooling_device1
>>>>>> [ 3.173401] ACPI: Video Device [VID] (multi-head: yes rom: no post: no)
>>>>>> [ 3.173962] input: Video Bus as
>> /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input4
>>>>>> [ 3.174232] [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on
>>>>> minor 0
>>>>>> [ 3.174258] ahci 0000:00:1f.2: version 3.0
>>>>>> [ 3.174270] xen: registering gsi 19 triggering 0 polarity 1
>>>>>> [ 3.174274] Already setup the GSI :19
>>>>>>
>>>>>>
>>>>>> After that - the unit never seems to be flushed.
>>>>>>
>>>>>> ...until we enter into the S3 hypercall, which loops over all DRHD
>>>>>> units, and explicitly flushes all of them via iommu_flush_all()
>>>>>>
>>>>>> It is at that point that it hangs up when talking to the device that
>>>>>> the IGD is plumbed up to.
>>>>>>
>>>>>>
>>>>>> Does this point to something in the i915 driver doing something that
>>>>>> is incompatible with Xen?
>>>>>
>>>>> I actually separated it from the S3 hypercall, adding a new debug key
>>>>> 'F' - to just call iommu_flush_all()
>>>>> I can crash it on demand with this.
>>>>>
>>>>> Booting with "i915.modeset=0 single" (to prevent both KMS, and Xorg) -
>>>>> it does not occur.
>>>>> So, that pretty much narrows it down to the IGD, in my mind.
>>>>
>>>> Indeed, I agree. Yet I can't in any way comment on what or why.
>>>> Xiantao (perhaps some graphics person would good to be Cc-ed
>>>> here too)?
>>> Hi, Jan/Ben
>>> Thanks for your analysis! Could you try to enable "snb_igd_quirk" to have a
>> try ? thanks!
>>> Xiantao
>>
>>
>> Thanks for your reply. I tried this param yesterday, but it did not
>> change the behavior.
> Okay, I recalled one bug in IGD i915 driver is found recently, and it may bring some errors to VT-d, and should be fixed in latest kernel. Could you try latest kernel 3.9.4 or 3.10-rcx ?
> Xiantao
It may have been dropped off of the top of this thread, but i sent out
what i have tested with, and this was one of them.
Testing 3.10 did not change this behavior.
Did you have a particular changeset in mind?
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: S3 crash with VTD Queue Invalidation enabled
2013-06-06 15:17 ` Ben Guthro
@ 2013-06-07 1:33 ` Zhang, Xiantao
2013-06-07 15:52 ` Ben Guthro
0 siblings, 1 reply; 26+ messages in thread
From: Zhang, Xiantao @ 2013-06-07 1:33 UTC (permalink / raw)
To: Ben Guthro
Cc: Andrew Cooper, Zhang, Xiantao, Ben Guthro, Jan Beulich, xen-devel
> >>>>> Booting with "i915.modeset=0 single" (to prevent both KMS, and Xorg) -
> >>>>> it does not occur.
> >>>>> So, that pretty much narrows it down to the IGD, in my mind.
> >>>>
> >>>> Indeed, I agree. Yet I can't in any way comment on what or why.
> >>>> Xiantao (perhaps some graphics person would good to be Cc-ed
> >>>> here too)?
> >>> Hi, Jan/Ben
> >>> Thanks for your analysis! Could you try to enable "snb_igd_quirk" to have
> a
> >> try ? thanks!
> >>> Xiantao
> >>
> >>
> >> Thanks for your reply. I tried this param yesterday, but it did not
> >> change the behavior.
> > Okay, I recalled one bug in IGD i915 driver is found recently, and it may bring
> some errors to VT-d, and should be fixed in latest kernel. Could you try latest
> kernel 3.9.4 or 3.10-rcx ?
> > Xiantao
>
> It may have been dropped off of the top of this thread, but i sent out
> what i have tested with, and this was one of them.
>
> Testing 3.10 did not change this behavior.
>
> Did you have a particular changeset in mind?
I will try to find it out, and I am wondering whether your code base includes this quirk ? See: http://www.gossamer-threads.com/lists/xen/devel/275623
Xiantao
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: S3 crash with VTD Queue Invalidation enabled
2013-06-07 1:33 ` Zhang, Xiantao
@ 2013-06-07 15:52 ` Ben Guthro
0 siblings, 0 replies; 26+ messages in thread
From: Ben Guthro @ 2013-06-07 15:52 UTC (permalink / raw)
To: Zhang, Xiantao; +Cc: Andrew Cooper, Jan Beulich, xen-devel
On Thu, Jun 6, 2013 at 9:33 PM, Zhang, Xiantao <xiantao.zhang@intel.com> wrote:
>> >>>>> Booting with "i915.modeset=0 single" (to prevent both KMS, and Xorg) -
>> >>>>> it does not occur.
>> >>>>> So, that pretty much narrows it down to the IGD, in my mind.
>> >>>>
>> >>>> Indeed, I agree. Yet I can't in any way comment on what or why.
>> >>>> Xiantao (perhaps some graphics person would good to be Cc-ed
>> >>>> here too)?
>> >>> Hi, Jan/Ben
>> >>> Thanks for your analysis! Could you try to enable "snb_igd_quirk" to have
>> a
>> >> try ? thanks!
>> >>> Xiantao
>> >>
>> >>
>> >> Thanks for your reply. I tried this param yesterday, but it did not
>> >> change the behavior.
>> > Okay, I recalled one bug in IGD i915 driver is found recently, and it may bring
>> some errors to VT-d, and should be fixed in latest kernel. Could you try latest
>> kernel 3.9.4 or 3.10-rcx ?
>> > Xiantao
>>
>> It may have been dropped off of the top of this thread, but i sent out
>> what i have tested with, and this was one of them.
>>
>> Testing 3.10 did not change this behavior.
>>
>> Did you have a particular changeset in mind?
> I will try to find it out, and I am wondering whether your code base includes this quirk ? See: http://www.gossamer-threads.com/lists/xen/devel/275623
> Xiantao
Yes, it does.
I've worked around the issue, for now by disabling queue invalidation
for SNB systems.
Not ideal...but better than crashing
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: S3 crash with VTD Queue Invalidation enabled
2013-06-05 23:53 ` Ben Guthro
2013-06-06 6:58 ` Jan Beulich
@ 2013-06-14 8:38 ` Jan Beulich
2013-06-14 17:01 ` Ben Guthro
1 sibling, 1 reply; 26+ messages in thread
From: Jan Beulich @ 2013-06-14 8:38 UTC (permalink / raw)
To: Ben Guthro; +Cc: Andrew Cooper, xiantao.zhang, xen-devel
>>> On 06.06.13 at 01:53, Ben Guthro <ben@guthro.net> wrote:
>> Early in the boot process, I see queue_invalidate_wait() called for
>> DRHD unit 0, and 1
>> (unit 0 is wired up to the IGD, unit 1 is everything else)
>>
>> Up until i915 does the following, I see that unit being flushed with
>> queue_invalidate_wait() :
>>
>> [ 0.704537] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
>> [ 0.704537] ENERGY_PERF_BIAS: View and update with x86_energy_p
>> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
>> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
>> [ 1.983028] [drm] GMBUS [i915 gmbus dpb] timed out, falling back to
>> bit banging on pin 5
>> [ 2.253551] fbcon: inteldrmfb (fb0) is primary device
>> [ 3.111838] Console: switching to colour frame buffer device 170x48
>> [ 3.171631] i915 0000:00:02.0: fb0: inteldrmfb frame buffer device
>> [ 3.171634] i915 0000:00:02.0: registered panic notifier
>> [ 3.173339] acpi device:00: registered as cooling_device1
>> [ 3.173401] ACPI: Video Device [VID] (multi-head: yes rom: no post: no)
>> [ 3.173962] input: Video Bus as
>> /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input4
>> [ 3.174232] [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on minor 0
>> [ 3.174258] ahci 0000:00:1f.2: version 3.0
>> [ 3.174270] xen: registering gsi 19 triggering 0 polarity 1
>> [ 3.174274] Already setup the GSI :19
>>
>>
>> After that - the unit never seems to be flushed.
With queue_invalidate_wait() having a single caller -
invalidate_sync() -, and with invalidate_sync() being called from
all interrupt setup (IO-APIC as well as MSI), that's quite odd to be
the case. At least upon network driver load or interface-up, this
should be getting called.
>> ...until we enter into the S3 hypercall, which loops over all DRHD
>> units, and explicitly flushes all of them via iommu_flush_all()
>>
>> It is at that point that it hangs up when talking to the device that
>> the IGD is plumbed up to.
>>
>>
>> Does this point to something in the i915 driver doing something that
>> is incompatible with Xen?
>
> I actually separated it from the S3 hypercall, adding a new debug key
> 'F' - to just call iommu_flush_all()
> I can crash it on demand with this.
>
> Booting with "i915.modeset=0 single" (to prevent both KMS, and Xorg) -
> it does not occur.
> So, that pretty much narrows it down to the IGD, in my mind.
Which reminds me of a change I did several weeks back to our kernel,
but which isn't as easily done with pv-ops: There are a number of
cases in the AGP and DRM code that qualify upon CONFIG_INTEL_IOMMU
and use intel_iommu_gfx_mapped. As you certainly know, Linux when
running on Xen doesn't see any IOMMU, and hence the config option
being enabled or disabled is completely unrelated to whether the
driver actually runs on top of an enabled IOMMU. Similarly the setting
of intel_iommu_gfx_mapped cannot possibly happen when running on
top of Xen, as it sits in code that never gets used in this case.
A possibly simple, but rather hacky solution might be to always set
that variable when running on Xen. But that wouldn't cover the case
of a kernel being built without CONFIG_INTEL_IOMMU, yet in that
case the driver might still run with an IOMMU enabled underneath.
(In our case I can simply always #define intel_iommu_gfx_mapped
to 1, with the INTEL_IOMMU option getting forcibly disabled for the
Xen kernel flavors anyway. Whether that's entirely correct when
not running on an enabled IOMMU I can't tell yet, and don't know
whom to ask.)
And that wouldn't cover the IGD getting passed through to a DomU
at all - obviously Xen's ability to properly drive all IOMMU operations
(including qinval) must not depend on the owning guest's driver code.
I have to admit though that it entirely escapes me why a graphics
driver needs to peek into IOMMU code/state in the first place. This
very much smells of bad design.
Jan
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: S3 crash with VTD Queue Invalidation enabled
2013-06-14 8:38 ` Jan Beulich
@ 2013-06-14 17:01 ` Ben Guthro
2013-06-14 18:27 ` Ben Guthro
0 siblings, 1 reply; 26+ messages in thread
From: Ben Guthro @ 2013-06-14 17:01 UTC (permalink / raw)
To: Jan Beulich; +Cc: Andrew Cooper, xiantao.zhang, xen-devel
On Fri, Jun 14, 2013 at 4:38 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>> On 06.06.13 at 01:53, Ben Guthro <ben@guthro.net> wrote:
>>> Early in the boot process, I see queue_invalidate_wait() called for
>>> DRHD unit 0, and 1
>>> (unit 0 is wired up to the IGD, unit 1 is everything else)
>>>
>>> Up until i915 does the following, I see that unit being flushed with
>>> queue_invalidate_wait() :
>>>
>>> [ 0.704537] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
>>> [ 0.704537] ENERGY_PERF_BIAS: View and update with x86_energy_p
>>> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
>>> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
>>> [ 1.983028] [drm] GMBUS [i915 gmbus dpb] timed out, falling back to
>>> bit banging on pin 5
>>> [ 2.253551] fbcon: inteldrmfb (fb0) is primary device
>>> [ 3.111838] Console: switching to colour frame buffer device 170x48
>>> [ 3.171631] i915 0000:00:02.0: fb0: inteldrmfb frame buffer device
>>> [ 3.171634] i915 0000:00:02.0: registered panic notifier
>>> [ 3.173339] acpi device:00: registered as cooling_device1
>>> [ 3.173401] ACPI: Video Device [VID] (multi-head: yes rom: no post: no)
>>> [ 3.173962] input: Video Bus as
>>> /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input4
>>> [ 3.174232] [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on minor 0
>>> [ 3.174258] ahci 0000:00:1f.2: version 3.0
>>> [ 3.174270] xen: registering gsi 19 triggering 0 polarity 1
>>> [ 3.174274] Already setup the GSI :19
>>>
>>>
>>> After that - the unit never seems to be flushed.
>
> With queue_invalidate_wait() having a single caller -
> invalidate_sync() -, and with invalidate_sync() being called from
> all interrupt setup (IO-APIC as well as MSI), that's quite odd to be
> the case. At least upon network driver load or interface-up, this
> should be getting called.
>
>>> ...until we enter into the S3 hypercall, which loops over all DRHD
>>> units, and explicitly flushes all of them via iommu_flush_all()
>>>
>>> It is at that point that it hangs up when talking to the device that
>>> the IGD is plumbed up to.
>>>
>>>
>>> Does this point to something in the i915 driver doing something that
>>> is incompatible with Xen?
>>
>> I actually separated it from the S3 hypercall, adding a new debug key
>> 'F' - to just call iommu_flush_all()
>> I can crash it on demand with this.
>>
>> Booting with "i915.modeset=0 single" (to prevent both KMS, and Xorg) -
>> it does not occur.
>> So, that pretty much narrows it down to the IGD, in my mind.
>
> Which reminds me of a change I did several weeks back to our kernel,
> but which isn't as easily done with pv-ops: There are a number of
> cases in the AGP and DRM code that qualify upon CONFIG_INTEL_IOMMU
> and use intel_iommu_gfx_mapped. As you certainly know, Linux when
> running on Xen doesn't see any IOMMU, and hence the config option
> being enabled or disabled is completely unrelated to whether the
> driver actually runs on top of an enabled IOMMU. Similarly the setting
> of intel_iommu_gfx_mapped cannot possibly happen when running on
> top of Xen, as it sits in code that never gets used in this case.
>
> A possibly simple, but rather hacky solution might be to always set
> that variable when running on Xen. But that wouldn't cover the case
> of a kernel being built without CONFIG_INTEL_IOMMU, yet in that
> case the driver might still run with an IOMMU enabled underneath.
> (In our case I can simply always #define intel_iommu_gfx_mapped
> to 1, with the INTEL_IOMMU option getting forcibly disabled for the
> Xen kernel flavors anyway. Whether that's entirely correct when
> not running on an enabled IOMMU I can't tell yet, and don't know
> whom to ask.)
>
> And that wouldn't cover the IGD getting passed through to a DomU
> at all - obviously Xen's ability to properly drive all IOMMU operations
> (including qinval) must not depend on the owning guest's driver code.
>
> I have to admit though that it entirely escapes me why a graphics
> driver needs to peek into IOMMU code/state in the first place. This
> very much smells of bad design.
This all makes sense, and I agree with your assessment.
Unfortunately, I went and got the machine back from our QA department,
to do some tests on this, and now I am unable to reproduce the issue,
to prove your analysis is correct.
It was 100% reproducible a week ago, and now I can't make it happen,
using the same code base & build.
It is all very strange, and smells of a race condition, or
uninitialized variable.
I blame Alpha particles.
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: S3 crash with VTD Queue Invalidation enabled
2013-06-14 17:01 ` Ben Guthro
@ 2013-06-14 18:27 ` Ben Guthro
2013-06-17 7:23 ` Jan Beulich
0 siblings, 1 reply; 26+ messages in thread
From: Ben Guthro @ 2013-06-14 18:27 UTC (permalink / raw)
To: Jan Beulich
Cc: Andrew Cooper, Konrad Rzeszutek Wilk, xiantao.zhang, xen-devel
On Fri, Jun 14, 2013 at 1:01 PM, Ben Guthro <ben@guthro.net> wrote:
> On Fri, Jun 14, 2013 at 4:38 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>>> On 06.06.13 at 01:53, Ben Guthro <ben@guthro.net> wrote:
>>>> Early in the boot process, I see queue_invalidate_wait() called for
>>>> DRHD unit 0, and 1
>>>> (unit 0 is wired up to the IGD, unit 1 is everything else)
>>>>
>>>> Up until i915 does the following, I see that unit being flushed with
>>>> queue_invalidate_wait() :
>>>>
>>>> [ 0.704537] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
>>>> [ 0.704537] ENERGY_PERF_BIAS: View and update with x86_energy_p
>>>> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
>>>> (XEN) XXX queue_invalidate_wait:282 CPU0 DRHD0 ret=0
>>>> [ 1.983028] [drm] GMBUS [i915 gmbus dpb] timed out, falling back to
>>>> bit banging on pin 5
>>>> [ 2.253551] fbcon: inteldrmfb (fb0) is primary device
>>>> [ 3.111838] Console: switching to colour frame buffer device 170x48
>>>> [ 3.171631] i915 0000:00:02.0: fb0: inteldrmfb frame buffer device
>>>> [ 3.171634] i915 0000:00:02.0: registered panic notifier
>>>> [ 3.173339] acpi device:00: registered as cooling_device1
>>>> [ 3.173401] ACPI: Video Device [VID] (multi-head: yes rom: no post: no)
>>>> [ 3.173962] input: Video Bus as
>>>> /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input4
>>>> [ 3.174232] [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on minor 0
>>>> [ 3.174258] ahci 0000:00:1f.2: version 3.0
>>>> [ 3.174270] xen: registering gsi 19 triggering 0 polarity 1
>>>> [ 3.174274] Already setup the GSI :19
>>>>
>>>>
>>>> After that - the unit never seems to be flushed.
>>
>> With queue_invalidate_wait() having a single caller -
>> invalidate_sync() -, and with invalidate_sync() being called from
>> all interrupt setup (IO-APIC as well as MSI), that's quite odd to be
>> the case. At least upon network driver load or interface-up, this
>> should be getting called.
>>
>>>> ...until we enter into the S3 hypercall, which loops over all DRHD
>>>> units, and explicitly flushes all of them via iommu_flush_all()
>>>>
>>>> It is at that point that it hangs up when talking to the device that
>>>> the IGD is plumbed up to.
>>>>
>>>>
>>>> Does this point to something in the i915 driver doing something that
>>>> is incompatible with Xen?
>>>
>>> I actually separated it from the S3 hypercall, adding a new debug key
>>> 'F' - to just call iommu_flush_all()
>>> I can crash it on demand with this.
>>>
>>> Booting with "i915.modeset=0 single" (to prevent both KMS, and Xorg) -
>>> it does not occur.
>>> So, that pretty much narrows it down to the IGD, in my mind.
>>
>> Which reminds me of a change I did several weeks back to our kernel,
>> but which isn't as easily done with pv-ops: There are a number of
>> cases in the AGP and DRM code that qualify upon CONFIG_INTEL_IOMMU
>> and use intel_iommu_gfx_mapped. As you certainly know, Linux when
>> running on Xen doesn't see any IOMMU, and hence the config option
>> being enabled or disabled is completely unrelated to whether the
>> driver actually runs on top of an enabled IOMMU. Similarly the setting
>> of intel_iommu_gfx_mapped cannot possibly happen when running on
>> top of Xen, as it sits in code that never gets used in this case.
>>
>> A possibly simple, but rather hacky solution might be to always set
>> that variable when running on Xen. But that wouldn't cover the case
>> of a kernel being built without CONFIG_INTEL_IOMMU, yet in that
>> case the driver might still run with an IOMMU enabled underneath.
>> (In our case I can simply always #define intel_iommu_gfx_mapped
>> to 1, with the INTEL_IOMMU option getting forcibly disabled for the
>> Xen kernel flavors anyway. Whether that's entirely correct when
>> not running on an enabled IOMMU I can't tell yet, and don't know
>> whom to ask.)
>>
>> And that wouldn't cover the IGD getting passed through to a DomU
>> at all - obviously Xen's ability to properly drive all IOMMU operations
>> (including qinval) must not depend on the owning guest's driver code.
>>
>> I have to admit though that it entirely escapes me why a graphics
>> driver needs to peek into IOMMU code/state in the first place. This
>> very much smells of bad design.
>
>
> This all makes sense, and I agree with your assessment.
>
> Unfortunately, I went and got the machine back from our QA department,
> to do some tests on this, and now I am unable to reproduce the issue,
> to prove your analysis is correct.
> It was 100% reproducible a week ago, and now I can't make it happen,
> using the same code base & build.
>
> It is all very strange, and smells of a race condition, or
> uninitialized variable.
> I blame Alpha particles.
I did a little more bisecting of our builds, and it appears I was not
actually testing the version that I thought I was here, and once I did
some bisection, I found it got inadvertently fixed by another change
someone else committed to solve an unrelated problem.
The following changes
Revert:
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=c79c49826270b8b0061b2fca840fc3f013c8a78a
Apply:
https://lkml.org/lkml/2012/2/10/229
I don't have a good explanation as to why re-enabling PAT would change
the behavior of this IOMMU feature...but I have a very reproducible
test case showing that it, in fact does.
Konrad - do you have any theories that would explain this one?
Or, would we like to leave this one as "Here be Dragons" and look the other way?
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: S3 crash with VTD Queue Invalidation enabled
2013-06-14 18:27 ` Ben Guthro
@ 2013-06-17 7:23 ` Jan Beulich
0 siblings, 0 replies; 26+ messages in thread
From: Jan Beulich @ 2013-06-17 7:23 UTC (permalink / raw)
To: Ben Guthro, xiantao.zhang; +Cc: Andrew Cooper, Konrad Rzeszutek Wilk, xen-devel
>>> On 14.06.13 at 20:27, Ben Guthro <ben@guthro.net> wrote:
> I did a little more bisecting of our builds, and it appears I was not
> actually testing the version that I thought I was here, and once I did
> some bisection, I found it got inadvertently fixed by another change
> someone else committed to solve an unrelated problem.
>
> The following changes
>
> Revert:
> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=c7
> 9c49826270b8b0061b2fca840fc3f013c8a78a
>
> Apply:
> https://lkml.org/lkml/2012/2/10/229
>
> I don't have a good explanation as to why re-enabling PAT would change
> the behavior of this IOMMU feature...but I have a very reproducible
> test case showing that it, in fact does.
Now, while this is good news in terms of knowing at least something
that addresses (or more likely works around) the issue, this still
leaves Xen at the mercy of the kernel running in the domain owning
the IGD. I.e. still a latent security issue. We really need to find a
solution that's independent of the guest kernel.
Xiantao - we certainly will need your (Intel's) help with this, and a
first step might be understanding how the above mentioned kernel
side changes can result in masking the observed problem.
Jan
^ permalink raw reply [flat|nested] 26+ messages in thread
end of thread, other threads:[~2013-06-17 7:23 UTC | newest]
Thread overview: 26+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-06-03 18:29 S3 crash with VTD Queue Invalidation enabled Ben Guthro
2013-06-03 19:22 ` Andrew Cooper
2013-06-04 8:54 ` Jan Beulich
2013-06-04 12:25 ` Ben Guthro
2013-06-04 14:01 ` Jan Beulich
2013-06-04 19:20 ` Ben Guthro
2013-06-04 19:49 ` Ben Guthro
2013-06-04 21:09 ` Ben Guthro
2013-06-05 8:24 ` Jan Beulich
2013-06-05 13:54 ` Ben Guthro
2013-06-05 15:14 ` Jan Beulich
2013-06-05 15:25 ` Ben Guthro
2013-06-05 15:38 ` Jan Beulich
2013-06-05 20:27 ` Ben Guthro
2013-06-05 23:53 ` Ben Guthro
2013-06-06 6:58 ` Jan Beulich
2013-06-06 15:06 ` Zhang, Xiantao
2013-06-06 15:07 ` Ben Guthro
2013-06-06 15:13 ` Zhang, Xiantao
2013-06-06 15:17 ` Ben Guthro
2013-06-07 1:33 ` Zhang, Xiantao
2013-06-07 15:52 ` Ben Guthro
2013-06-14 8:38 ` Jan Beulich
2013-06-14 17:01 ` Ben Guthro
2013-06-14 18:27 ` Ben Guthro
2013-06-17 7:23 ` Jan Beulich
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.