Re: About Xen bridged pci devices and suspend/resume for the X10SAE motherboard

Re: About Xen bridged pci devices and suspend/resume for the X10SAE motherboard

[Konrad Rzeszutek Wilk]

On Mon, Aug 10, 2015 at 02:11:38AM +0300, M. Ivanov wrote:
> Hello,
> 
> excuse me for bothering you, but I've read an old thread on a mailing
> list about X10SAE compatibility. 
> http://lists.xen.org/archives/html/xen-devel/2014-02/msg02111.html

CC-ing Xen devel.
> 
> Currently I own this board and am trying to use it with Xen and be able
> to suspend and resume.
> 
> But I am getting errors from the USB 3 Renesas controller about parity
> in my bios event log, and my system hangs on resume,
> so I was wondering if that is connected to the bridge(tundra) you've
> mentioned.

Did you update the BIOS to the latest version?
> 
> I will be very glad if you could share any information regarding this
> matter. 
> 
> Best regards,
> M. Ivanov

Re: About Xen bridged pci devices and suspend/resume for the X10SAE motherboard

[M. Ivanov]

[-- Attachment #1.1: Type: text/plain, Size: 985 bytes --]

On Mon, 2015-08-10 at 09:58 -0400, Konrad Rzeszutek Wilk wrote:
> On Mon, Aug 10, 2015 at 02:11:38AM +0300, M. Ivanov wrote:
> > Hello,
> > 
> > excuse me for bothering you, but I've read an old thread on a mailing
> > list about X10SAE compatibility. 
> > http://lists.xen.org/archives/html/xen-devel/2014-02/msg02111.html
> 
> CC-ing Xen devel.
> > 
> > Currently I own this board and am trying to use it with Xen and be able
> > to suspend and resume.
> > 
> > But I am getting errors from the USB 3 Renesas controller about parity
> > in my bios event log, and my system hangs on resume,
> > so I was wondering if that is connected to the bridge(tundra) you've
> > mentioned.
> 
> Did you update the BIOS to the latest version?
Will updating to version 3 solve my issue?
Can you do a suspend/resume on your X10SAE?
> > 
> > I will be very glad if you could share any information regarding this
> > matter. 
> > 
> > Best regards,
> > M. Ivanov
> 
> 


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: About Xen bridged pci devices and suspend/resume for the X10SAE motherboard

[Konrad Rzeszutek Wilk]

On Mon, Aug 10, 2015 at 05:14:28PM +0300, M. Ivanov wrote:
> On Mon, 2015-08-10 at 09:58 -0400, Konrad Rzeszutek Wilk wrote:
> > On Mon, Aug 10, 2015 at 02:11:38AM +0300, M. Ivanov wrote:
> > > Hello,
> > > 
> > > excuse me for bothering you, but I've read an old thread on a mailing
> > > list about X10SAE compatibility. 
> > > http://lists.xen.org/archives/html/xen-devel/2014-02/msg02111.html
> > 
> > CC-ing Xen devel.
> > > 
> > > Currently I own this board and am trying to use it with Xen and be able
> > > to suspend and resume.
> > > 
> > > But I am getting errors from the USB 3 Renesas controller about parity
> > > in my bios event log, and my system hangs on resume,
> > > so I was wondering if that is connected to the bridge(tundra) you've
> > > mentioned.
> > 
> > Did you update the BIOS to the latest version?
> Will updating to version 3 solve my issue?
> Can you do a suspend/resume on your X10SAE?

It did work at some point. I will find out when I am at home later today.

> > > 
> > > I will be very glad if you could share any information regarding this
> > > matter. 
> > > 
> > > Best regards,
> > > M. Ivanov
> > 
> > 
> 

Re: About Xen bridged pci devices and suspend/resume for the X10SAE motherboard

[M. Ivanov]

[-- Attachment #1.1: Type: text/plain, Size: 1846 bytes --]

On Mon, 2015-08-10 at 10:47 -0400, Konrad Rzeszutek Wilk wrote:
> On Mon, Aug 10, 2015 at 05:14:28PM +0300, M. Ivanov wrote:
> > On Mon, 2015-08-10 at 09:58 -0400, Konrad Rzeszutek Wilk wrote:
> > > On Mon, Aug 10, 2015 at 02:11:38AM +0300, M. Ivanov wrote:
> > > > Hello,
> > > > 
> > > > excuse me for bothering you, but I've read an old thread on a mailing
> > > > list about X10SAE compatibility. 
> > > > http://lists.xen.org/archives/html/xen-devel/2014-02/msg02111.html
> > > 
> > > CC-ing Xen devel.
> > > > 
> > > > Currently I own this board and am trying to use it with Xen and be able
> > > > to suspend and resume.
> > > > 
> > > > But I am getting errors from the USB 3 Renesas controller about parity
> > > > in my bios event log, and my system hangs on resume,
> > > > so I was wondering if that is connected to the bridge(tundra) you've
> > > > mentioned.
> > > 
> > > Did you update the BIOS to the latest version?
> > Will updating to version 3 solve my issue?
> > Can you do a suspend/resume on your X10SAE?
> 
> It did work at some point. I will find out when I am at home later today.
> 
Looking forward to your reply and am really thankful for your time,
so far I've tried changing many of the settings in the bios,
fiddling with Xen's kernel params,
blacklisting the xhci driver, doing a xl detach.

The only thing I haven't done yet is updating the bios,
but Supermicro's support couldn't give me a changelog:

"The primary objective for ver3.0 BIOS release is to support Intel
Broadwell CPUs"
"We do not know if BIOS update will fix the issue you are seeing as we
never tested it with Xen."
> > > > 
> > > > I will be very glad if you could share any information regarding this
> > > > matter. 
> > > > 
> > > > Best regards,
> > > > M. Ivanov
> > > 
> > > 
> > 
> 
> 


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: About Xen bridged pci devices and suspend/resume for the X10SAE motherboard (SuperMicro)

[Konrad Rzeszutek Wilk]

On Mon, Aug 10, 2015 at 09:21:35PM +0300, M. Ivanov wrote:
> On Mon, 2015-08-10 at 10:47 -0400, Konrad Rzeszutek Wilk wrote:
> > On Mon, Aug 10, 2015 at 05:14:28PM +0300, M. Ivanov wrote:
> > > On Mon, 2015-08-10 at 09:58 -0400, Konrad Rzeszutek Wilk wrote:
> > > > On Mon, Aug 10, 2015 at 02:11:38AM +0300, M. Ivanov wrote:
> > > > > Hello,
> > > > > 
> > > > > excuse me for bothering you, but I've read an old thread on a mailing
> > > > > list about X10SAE compatibility. 
> > > > > http://lists.xen.org/archives/html/xen-devel/2014-02/msg02111.html
> > > > 
> > > > CC-ing Xen devel.
> > > > > 
> > > > > Currently I own this board and am trying to use it with Xen and be able
> > > > > to suspend and resume.
> > > > > 
> > > > > But I am getting errors from the USB 3 Renesas controller about parity
> > > > > in my bios event log, and my system hangs on resume,
> > > > > so I was wondering if that is connected to the bridge(tundra) you've
> > > > > mentioned.
> > > > 
> > > > Did you update the BIOS to the latest version?
> > > Will updating to version 3 solve my issue?
> > > Can you do a suspend/resume on your X10SAE?
> > 
> > It did work at some point. I will find out when I am at home later today.
> > 
> Looking forward to your reply and am really thankful for your time,
> so far I've tried changing many of the settings in the bios,
> fiddling with Xen's kernel params,
> blacklisting the xhci driver, doing a xl detach.
> 
> The only thing I haven't done yet is updating the bios,
> but Supermicro's support couldn't give me a changelog:
> 
> "The primary objective for ver3.0 BIOS release is to support Intel
> Broadwell CPUs"
> "We do not know if BIOS update will fix the issue you are seeing as we
> never tested it with Xen."

I did test it remotely and it did something very odd. It suspended and then
immediately resumed with tons of VT-d errors!?

I will try again but be actually right by it.
> > > > > 
> > > > > I will be very glad if you could share any information regarding this
> > > > > matter. 
> > > > > 
> > > > > Best regards,
> > > > > M. Ivanov
> > > > 
> > > > 
> > > 
> > 
> > 
> 

Re: About Xen bridged pci devices and suspend/resume for the X10SAE motherboard (SuperMicro)

[M. Ivanov]

[-- Attachment #1.1: Type: text/plain, Size: 3608 bytes --]

On Thu, 2015-08-13 at 15:49 -0400, Konrad Rzeszutek Wilk wrote:
> On Mon, Aug 10, 2015 at 09:21:35PM +0300, M. Ivanov wrote:
> > On Mon, 2015-08-10 at 10:47 -0400, Konrad Rzeszutek Wilk wrote:
> > > On Mon, Aug 10, 2015 at 05:14:28PM +0300, M. Ivanov wrote:
> > > > On Mon, 2015-08-10 at 09:58 -0400, Konrad Rzeszutek Wilk wrote:
> > > > > On Mon, Aug 10, 2015 at 02:11:38AM +0300, M. Ivanov wrote:
> > > > > > Hello,
> > > > > > 
> > > > > > excuse me for bothering you, but I've read an old thread on a mailing
> > > > > > list about X10SAE compatibility. 
> > > > > > http://lists.xen.org/archives/html/xen-devel/2014-02/msg02111.html
> > > > > 
> > > > > CC-ing Xen devel.
> > > > > > 
> > > > > > Currently I own this board and am trying to use it with Xen and be able
> > > > > > to suspend and resume.
> > > > > > 
> > > > > > But I am getting errors from the USB 3 Renesas controller about parity
> > > > > > in my bios event log, and my system hangs on resume,
> > > > > > so I was wondering if that is connected to the bridge(tundra) you've
> > > > > > mentioned.
> > > > > 
> > > > > Did you update the BIOS to the latest version?
> > > > Will updating to version 3 solve my issue?
> > > > Can you do a suspend/resume on your X10SAE?
> > > 
> > > It did work at some point. I will find out when I am at home later today.
> > > 
> > Looking forward to your reply and am really thankful for your time,
> > so far I've tried changing many of the settings in the bios,
> > fiddling with Xen's kernel params,
> > blacklisting the xhci driver, doing a xl detach.
> > 
> > The only thing I haven't done yet is updating the bios,
> > but Supermicro's support couldn't give me a changelog:
> > 
> > "The primary objective for ver3.0 BIOS release is to support Intel
> > Broadwell CPUs"
> > "We do not know if BIOS update will fix the issue you are seeing as we
> > never tested it with Xen."
> 
> I did test it remotely and it did something very odd. It suspended and then
> immediately resumed with tons of VT-d errors!?
> 
> I will try again but be actually right by it.
Thanks for your effort,
Can you suggest a way for me to log what is happening?

Since my machine just hangs up and I don't get a picture on the screen.
Or when I do(sometimes) - it's just a cursor,(on a black screen with
nothing else,no errors shown) and the machine doesn't react to any key
combinations.

On a side note:

I did try updating the bios,
but got some really strange result.
The first time I got checksums about everything OK,
(erase,flash,verify),
but then it said - FDT is locked!
And I am at least 90% sure I've enabled reflash in 
the BIOS setup prior to flashing.
So I've restarted but didn't clear the CMOS(through the jumper on the
board). And it said Bios v 3.0 in the setup, then I also did the
Reset to optmized defaults.

Tried suspending and couldn't resume like always.
But perhaps currently my BIOS is in a broken/corrupted state.

After that I've tried reflashing the bios again. But this time - 
after the messages about Erasing,Flashing,Verifying:
it just froze. I didn't get any message about FDT, restarting or
whatsoever.

So I rebooted and it seems to work like before,
says version 3.0 in the BIOS setup.
I wonder if I should try clearing the CMOS and
flashing again.

> > > > > > 
> > > > > > I will be very glad if you could share any information regarding this
> > > > > > matter. 
> > > > > > 
> > > > > > Best regards,
> > > > > > M. Ivanov
> > > > > 
> > > > > 
> > > > 
> > > 
> > > 
> > 
> 
> 


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: About Xen bridged pci devices and suspend/resume for the X10SAE motherboard (SuperMicro)

[Konrad Rzeszutek Wilk]

On Fri, Aug 14, 2015 at 05:08:32PM +0300, M. Ivanov wrote:
> On Thu, 2015-08-13 at 15:49 -0400, Konrad Rzeszutek Wilk wrote:
> > On Mon, Aug 10, 2015 at 09:21:35PM +0300, M. Ivanov wrote:
> > > On Mon, 2015-08-10 at 10:47 -0400, Konrad Rzeszutek Wilk wrote:
> > > > On Mon, Aug 10, 2015 at 05:14:28PM +0300, M. Ivanov wrote:
> > > > > On Mon, 2015-08-10 at 09:58 -0400, Konrad Rzeszutek Wilk wrote:
> > > > > > On Mon, Aug 10, 2015 at 02:11:38AM +0300, M. Ivanov wrote:
> > > > > > > Hello,
> > > > > > > 
> > > > > > > excuse me for bothering you, but I've read an old thread on a mailing
> > > > > > > list about X10SAE compatibility. 
> > > > > > > http://lists.xen.org/archives/html/xen-devel/2014-02/msg02111.html
> > > > > > 
> > > > > > CC-ing Xen devel.
> > > > > > > 
> > > > > > > Currently I own this board and am trying to use it with Xen and be able
> > > > > > > to suspend and resume.
> > > > > > > 
> > > > > > > But I am getting errors from the USB 3 Renesas controller about parity
> > > > > > > in my bios event log, and my system hangs on resume,
> > > > > > > so I was wondering if that is connected to the bridge(tundra) you've
> > > > > > > mentioned.
> > > > > > 
> > > > > > Did you update the BIOS to the latest version?
> > > > > Will updating to version 3 solve my issue?
> > > > > Can you do a suspend/resume on your X10SAE?
> > > > 
> > > > It did work at some point. I will find out when I am at home later today.
> > > > 
> > > Looking forward to your reply and am really thankful for your time,
> > > so far I've tried changing many of the settings in the bios,
> > > fiddling with Xen's kernel params,
> > > blacklisting the xhci driver, doing a xl detach.
> > > 
> > > The only thing I haven't done yet is updating the bios,
> > > but Supermicro's support couldn't give me a changelog:
> > > 
> > > "The primary objective for ver3.0 BIOS release is to support Intel
> > > Broadwell CPUs"
> > > "We do not know if BIOS update will fix the issue you are seeing as we
> > > never tested it with Xen."
> > 
> > I did test it remotely and it did something very odd. It suspended and then
> > immediately resumed with tons of VT-d errors!?
> > 
> > I will try again but be actually right by it.
> Thanks for your effort,
> Can you suggest a way for me to log what is happening?

I usually have an serial cable attached to it and log that.
Cranking up the debug on everything gives me some idea.
> 
> Since my machine just hangs up and I don't get a picture on the screen.
> Or when I do(sometimes) - it's just a cursor,(on a black screen with
> nothing else,no errors shown) and the machine doesn't react to any key
> combinations.

Yeah. That is frustrating.
> 
> On a side note:
> 
> I did try updating the bios,
> but got some really strange result.
> The first time I got checksums about everything OK,
> (erase,flash,verify),
> but then it said - FDT is locked!
> And I am at least 90% sure I've enabled reflash in 
> the BIOS setup prior to flashing.
> So I've restarted but didn't clear the CMOS(through the jumper on the
> board). And it said Bios v 3.0 in the setup, then I also did the
> Reset to optmized defaults.
> 
> Tried suspending and couldn't resume like always.
> But perhaps currently my BIOS is in a broken/corrupted state.
> 
> After that I've tried reflashing the bios again. But this time - 
> after the messages about Erasing,Flashing,Verifying:
> it just froze. I didn't get any message about FDT, restarting or
> whatsoever.
> 
> So I rebooted and it seems to work like before,
> says version 3.0 in the BIOS setup.
> I wonder if I should try clearing the CMOS and
> flashing again.

One test I hadn't done is to try to suspend/resume
under baremental Linux and see how that works. Does it work for you?

The DMI tells me:
[    7.276963] Hardware name: Supermicro X10SAE/X10SAE, BIOS 2.00 04/21/2014

So a bit older BIOS.
The last thing I see:


# dmesg | grep -i Super
[    0.000000] DMI: Supermicro X10SAE/X10SAE, BIOS 2.00 04/21/2014
[    0.000000] ACPI: RSDP 0x00000000000F0490 000024 (v02-MB 01072009 AMI  00010013)
[    0.000000] ACPI: DSDT 0x000000009B9DA1E8 00CDDA (v02 SUPERM SMCI--MB 00000000 INTL 20120711)
[    0.000000] ACPI: APIC 0x000000009B9E70D8 000092 (v03 SUPERM SMCI--MB 01072009 AMI  00010013)
[    0.000000] ACPI: FPDT 0x000000009B9E7170 000044 (v01 SUPERM SMCI--MB 01072009 AMI  00010013)
[    0.000000] ACPI: MCFG 0x000000009B9E9488 00003C (v01 SUPERM SMCI--MB 01072009 MSFT 00000097)
[    0.000000] ACPI: HPET 0x000000009B9E94C8 000038 (v01 SUPERM SMCI--MB 01072009 AMI. 00000005)
[    7.623166] Hardware name: Supermicro X10SAE/X10SAE, BIOS 2.00 04/21/2014
# \a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\apm-sus\apoe\a\b \b\b \b# ^[[Jcd /sys/power
# ls
disk	    pm_freeze_timeout  pm_trace		   resume
image_size  pm_print_times     pm_trace_dev_match  state
pm_async    pm_test	       reserved_size	   wakeup_count
# c# ^[[Jcat state
freeze standby disk
# echo standby > state
[  204.755164] PM: Syncing filesystems ... done.
[  204.755223] PM: Preparing system for standby sleep
[  204.755606] Freezing user space processes ... (elapsed 0.000 seconds) done.
[  204.756506] Freezing reý+Ë+Ë•ÿezable tasks ... (elapsed 0.001 seconds) done.
[  204.757607] PM: Entering standby sleep
[  204.757762] Suspending console(s affinity for irq 40
(XEN) Broke affinity for irq 41
(XEN) Broke affinity for irq 23
(XEN) Broke affinity for irq 16
(XEN) Broke affinity fouppress EOI broadcast on CPU#3
(XEN) masked ExtINT on CPU#3
(XEN) Suppress EOI broadcast on CPU#4
(XEN) masked ExtINT on CPU#4
(XEN) Suppress EOI broadcast on CPU#5
(XEN) masked ExtINT on CPU#5
(XEN) Suppress EOI broadcast on CPU#6
(XEN) masked ExtINT on CPU#6
(XEN) Suppress EOI broadcast on CPU#7
(XEN) masked ExtINT on CPU#7
(XEN) traps.c:2653:d0v1 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) traps.c:2653:d0v1 Domain attempted WRMSR 00000000c0000082 from 0xffff83082727b000 to 0xffffffff8172e910.
(XEN) traps.c:2653:d0v1 Domain attempted WRMSR 00000000c0000083 from 0xffff83082727b080 to 0xffffffff81730c40.
(XEN) traps.c:2653:d0v1 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) traps.c:2653:d0v1 Domain attempted WRMSR 0000000000000175 from 0xffff83082727ffc0 to 0x0000000000000000.
(XEN) traps.c:2653:d0v1 Domain attempted WRMSR 0000000000000176 from 0xffff82d08022c7d0 to 0xffffffff81730c70.
(XEN) traps.c:2653:d0v1 Domain attempted WRMSR 00000000c0000083 from 0xffff83082727b080 to 0xffffffff81730eb0.
(XEN) traps.c:2653:d0v1 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) d0 attempted to change d0v1's CR4 flags 00002660 -> 00042760
(XEN) traps.c:2653:d0v2 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) traps.c:2653:d0v2 Domain attempted WRMSR 00000000c0000082 from 0xffff82d0803db000 to 0xffffffff8172e910.
(XEN) traps.c:2653:d0v2 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0803db080 to 0xffffffff81730c40.
(XEN) traps.c:2653:d0v2 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) traps.c:2653:d0v2 Domain attempted WRMSR 0000000000000175 from 0xffff82d0803dffc0 to 0x0000000000000000.
(XEN) traps.c:2653:d0v2 Domain attempted WRMSR 0000000000000176 from 0xffff82d08022c7d0 to 0xffffffff81730c70.
(XEN) traps.c:2653:d0v2 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0803db080 to 0xffffffff81730eb0.
(XEN) traps.c:2653:d0v2 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) d0 attempted to change d0v2's CR4 flags 00002660 -> 00042760
(XEN) traps.c:2653:d0v3 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) traps.c:2653:d0v3 Domain attempted WRMSR 00000000c0000082 from 0xffff82d0803db000 to 0xffffffff8172e910.
(XEN) traps.c:2653:d0v3 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0803db080 to 0xffffffff81730c40.
(XEN) traps.c:2653:d0v3 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) traps.c:2653:d0v3 Domain attempted WRMSR 0000000000000175 from 0xffff82d0803dffc0 to 0x0000000000000000.
(XEN) traps.c:2653:d0v3 Domain attempted WRMSR 0000000000000176 from 0xffff82d08022c7d0 to 0xffffffff81730c70.
(XEN) traps.c:2653:d0v3 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0803db080 to 0xffffffff81730eb0.
(XEN) traps.c:2653:d0v3 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) d0 attempted to change d0v3's CR4 flags 00002660 -> 00042760
(XEN) traps.c:2653:d0v4 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) traps.c:2653:d0v4 Domain attempted WRMSR 00000000c0000082 from 0xffff82d0803db000 to 0xffffffff8172e910.
(XEN) traps.c:2653:d0v4 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0803db080 to 0xffffffff81730c40.
(XEN) traps.c:2653:d0v4 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) traps.c:2653:d0v4 Domain attempted WRMSR 0000000000000175 from 0xffff82d0803dffc0 to 0x0000000000000000.
(XEN) traps.c:2653:d0v4 Domain attempted WRMSR 0000000000000176 from 0xffff82d08022c7d0 to 0xffffffff81730c70.
(XEN) traps.c:2653:d0v4 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0803db080 to 0xffffffff81730eb0.
(XEN) traps.c:2653:d0v4 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) d0 attempted to change d0v4's CR4 flags 00002660 -> 00042760
(XEN) traps.c:2653:d0v5 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) traps.c:2653:d0v5 Domain attempted WRMSR 00000000c0000082 from 0xffff8308272f3000 to 0xffffffff8172e910.
(XEN) traps.c:2653:d0v5 Domain attempted WRMSR 00000000c0000083 from 0xffff8308272f3080 to 0xffffffff81730c40.
(XEN) traps.c:2653:d0v5 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) traps.c:2653:d0v5 Domain attempted WRMSR 0000000000000175 from 0xffff8308272f7fc0 to 0x0000000000000000.
(XEN) traps.c:2653:d0v5 Domain attempted WRMSR 0000000000000176 from 0xffff82d08022c7d0 to 0xffffffff81730c70.
(XEN) traps.c:2653:d0v5 Domain attempted WRMSR 00000000c0000083 from 0xffff8308272f3080 to 0xffffffff81730eb0.
(XEN) traps.c:2653:d0v5 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) d0 attempted to change d0v5's CR4 flags 00002660 -> 00042760
(XEN) traps.c:2653:d0v6 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) traps.c:2653:d0v6 Domain attempted WRMSR 00000000c0000082 from 0xffff82d0803db000 to 0xffffffff8172e910.
(XEN) traps.c:2653:d0v6 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0803db080 to 0xffffffff81730c40.
(XEN) traps.c:2653:d0v6 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) traps.c:2653:d0v6 Domain attempted WRMSR 0000000000000175 from 0xffff82d0803dffc0 to 0x0000000000000000.
(XEN) traps.c:2653:d0v6 Domain attempted WRMSR 0000000000000176 from 0xffff82d08022c7d0 to 0xffffffff81730c70.
(XEN) traps.c:2653:d0v6 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0803db080 to 0xffffffff81730eb0.
(XEN) traps.c:2653:d0v6 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) d0 attempted to change d0v6's CR4 flags 00002660 -> 00042760
(XEN) event_fifo.c:50:d0v6 domain 0, port 50 already on a queue
(XEN) traps.c:2653:d0v7 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
(XEN) traps.c:2653:d0v7 Domain attempted WRMSR 00000000c0000082 from 0xffff83082727b000 to 0xffffffff8172e910.
(XEN) traps.c:2653:d0v7 Domain attempted WRMSR 00000000c0000083 from 0xffff83082727b080 to 0xffffffff81730c40.
(XEN) traps.c:2653:d0v7 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
(XEN) traps.c:2653:d0v7 Domain attempR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
(XEN) d0 attempted to change d0v7's CR4 flags 00002660 -> 00042760
^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G(XEN) Domain 0 crashed: rebooting machine in 5 seconds.

telnet> Connection closed.
[Connecting to system 50 ]

Great, so 'Domain 0 crashed'!

Pff - and oddly enough the same kernel suspend/resumes fine on my desktop and laptop.
> 
> > > > > > > 
> > > > > > > I will be very glad if you could share any information regarding this
> > > > > > > matter. 
> > > > > > > 
> > > > > > > Best regards,
> > > > > > > M. Ivanov
> > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > > 
> > > 
> > 
> > 
> 

Re: About Xen bridged pci devices and suspend/resume for the X10SAE motherboard (SuperMicro)

[M. Ivanov]

[-- Attachment #1.1: Type: text/plain, Size: 14130 bytes --]

On Fri, 2015-08-14 at 12:15 -0400, Konrad Rzeszutek Wilk wrote:
> On Fri, Aug 14, 2015 at 05:08:32PM +0300, M. Ivanov wrote:
> > On Thu, 2015-08-13 at 15:49 -0400, Konrad Rzeszutek Wilk wrote:
> > > On Mon, Aug 10, 2015 at 09:21:35PM +0300, M. Ivanov wrote:
> > > > On Mon, 2015-08-10 at 10:47 -0400, Konrad Rzeszutek Wilk wrote:
> > > > > On Mon, Aug 10, 2015 at 05:14:28PM +0300, M. Ivanov wrote:
> > > > > > On Mon, 2015-08-10 at 09:58 -0400, Konrad Rzeszutek Wilk wrote:
> > > > > > > On Mon, Aug 10, 2015 at 02:11:38AM +0300, M. Ivanov wrote:
> > > > > > > > Hello,
> > > > > > > > 
> > > > > > > > excuse me for bothering you, but I've read an old thread on a mailing
> > > > > > > > list about X10SAE compatibility. 
> > > > > > > > http://lists.xen.org/archives/html/xen-devel/2014-02/msg02111.html
> > > > > > > 
> > > > > > > CC-ing Xen devel.
> > > > > > > > 
> > > > > > > > Currently I own this board and am trying to use it with Xen and be able
> > > > > > > > to suspend and resume.
> > > > > > > > 
> > > > > > > > But I am getting errors from the USB 3 Renesas controller about parity
> > > > > > > > in my bios event log, and my system hangs on resume,
> > > > > > > > so I was wondering if that is connected to the bridge(tundra) you've
> > > > > > > > mentioned.
> > > > > > > 
> > > > > > > Did you update the BIOS to the latest version?
> > > > > > Will updating to version 3 solve my issue?
> > > > > > Can you do a suspend/resume on your X10SAE?
> > > > > 
> > > > > It did work at some point. I will find out when I am at home later today.
> > > > > 
> > > > Looking forward to your reply and am really thankful for your time,
> > > > so far I've tried changing many of the settings in the bios,
> > > > fiddling with Xen's kernel params,
> > > > blacklisting the xhci driver, doing a xl detach.
> > > > 
> > > > The only thing I haven't done yet is updating the bios,
> > > > but Supermicro's support couldn't give me a changelog:
> > > > 
> > > > "The primary objective for ver3.0 BIOS release is to support Intel
> > > > Broadwell CPUs"
> > > > "We do not know if BIOS update will fix the issue you are seeing as we
> > > > never tested it with Xen."
> > > 
> > > I did test it remotely and it did something very odd. It suspended and then
> > > immediately resumed with tons of VT-d errors!?
> > > 
> > > I will try again but be actually right by it.
> > Thanks for your effort,
> > Can you suggest a way for me to log what is happening?
> 
> I usually have an serial cable attached to it and log that.
> Cranking up the debug on everything gives me some idea.
Reasonable enough, will do.

> > 
> > Since my machine just hangs up and I don't get a picture on the screen.
> > Or when I do(sometimes) - it's just a cursor,(on a black screen with
> > nothing else,no errors shown) and the machine doesn't react to any key
> > combinations.
> 
> Yeah. That is frustrating.
> > 
> > On a side note:
> > 
> > I did try updating the bios,
> > but got some really strange result.
> > The first time I got checksums about everything OK,
> > (erase,flash,verify),
> > but then it said - FDT is locked!
> > And I am at least 90% sure I've enabled reflash in 
> > the BIOS setup prior to flashing.
> > So I've restarted but didn't clear the CMOS(through the jumper on the
> > board). And it said Bios v 3.0 in the setup, then I also did the
> > Reset to optmized defaults.
> > 
> > Tried suspending and couldn't resume like always.
> > But perhaps currently my BIOS is in a broken/corrupted state.
> > 
> > After that I've tried reflashing the bios again. But this time - 
> > after the messages about Erasing,Flashing,Verifying:
> > it just froze. I didn't get any message about FDT, restarting or
> > whatsoever.
> > 
> > So I rebooted and it seems to work like before,
> > says version 3.0 in the BIOS setup.
> > I wonder if I should try clearing the CMOS and
> > flashing again.
> 
> One test I hadn't done is to try to suspend/resume
> under baremental Linux and see how that works. Does it work for you?
> 
Yep, without Xen I can suspend fine.

> The DMI tells me:
> [    7.276963] Hardware name: Supermicro X10SAE/X10SAE, BIOS 2.00 04/21/2014
> 
> So a bit older BIOS.
So we've had the same bios versions. But now I have a broken BIOS 3.0..
(probably).
Maybe I will try reflashing to 2.0 if you have hints about getting
suspend/resume on it.

> The last thing I see:
> 
> 
> # dmesg | grep -i Super
> [    0.000000] DMI: Supermicro X10SAE/X10SAE, BIOS 2.00 04/21/2014
> [    0.000000] ACPI: RSDP 0x00000000000F0490 000024 (v02-MB 01072009 AMI  00010013)
> [    0.000000] ACPI: DSDT 0x000000009B9DA1E8 00CDDA (v02 SUPERM SMCI--MB 00000000 INTL 20120711)
> [    0.000000] ACPI: APIC 0x000000009B9E70D8 000092 (v03 SUPERM SMCI--MB 01072009 AMI  00010013)
> [    0.000000] ACPI: FPDT 0x000000009B9E7170 000044 (v01 SUPERM SMCI--MB 01072009 AMI  00010013)
> [    0.000000] ACPI: MCFG 0x000000009B9E9488 00003C (v01 SUPERM SMCI--MB 01072009 MSFT 00000097)
> [    0.000000] ACPI: HPET 0x000000009B9E94C8 000038 (v01 SUPERM SMCI--MB 01072009 AMI. 00000005)
> [    7.623166] Hardware name: Supermicro X10SAE/X10SAE, BIOS 2.00 04/21/2014
> # \a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\a\apm-sus\apoe\a\b \b\b \b# ^[[Jcd /sys/power
> # ls
> disk	    pm_freeze_timeout  pm_trace		   resume
> image_size  pm_print_times     pm_trace_dev_match  state
> pm_async    pm_test	       reserved_size	   wakeup_count
> # c# ^[[Jcat state
> freeze standby disk
> # echo standby > state
Since I am using CentOS in my experiments(version 6.7 has no systemd
support and 7 has no Xen support...) 
I do a echo mem > /sys/power/state.
> [  204.755164] PM: Syncing filesystems ... done.
> [  204.755223] PM: Preparing system for standby sleep
> [  204.755606] Freezing user space processes ... (elapsed 0.000 seconds) done.
> [  204.756506] Freezing reý+Ë+Ë•ÿezable tasks ... (elapsed 0.001 seconds) done.
> [  204.757607] PM: Entering standby sleep
> [  204.757762] Suspending console(s affinity for irq 40
> (XEN) Broke affinity for irq 41
> (XEN) Broke affinity for irq 23
> (XEN) Broke affinity for irq 16
> (XEN) Broke affinity fouppress EOI broadcast on CPU#3
> (XEN) masked ExtINT on CPU#3
> (XEN) Suppress EOI broadcast on CPU#4
> (XEN) masked ExtINT on CPU#4
> (XEN) Suppress EOI broadcast on CPU#5
> (XEN) masked ExtINT on CPU#5
> (XEN) Suppress EOI broadcast on CPU#6
> (XEN) masked ExtINT on CPU#6
> (XEN) Suppress EOI broadcast on CPU#7
> (XEN) masked ExtINT on CPU#7
> (XEN) traps.c:2653:d0v1 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
> (XEN) traps.c:2653:d0v1 Domain attempted WRMSR 00000000c0000082 from 0xffff83082727b000 to 0xffffffff8172e910.
> (XEN) traps.c:2653:d0v1 Domain attempted WRMSR 00000000c0000083 from 0xffff83082727b080 to 0xffffffff81730c40.
> (XEN) traps.c:2653:d0v1 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
> (XEN) traps.c:2653:d0v1 Domain attempted WRMSR 0000000000000175 from 0xffff83082727ffc0 to 0x0000000000000000.
> (XEN) traps.c:2653:d0v1 Domain attempted WRMSR 0000000000000176 from 0xffff82d08022c7d0 to 0xffffffff81730c70.
> (XEN) traps.c:2653:d0v1 Domain attempted WRMSR 00000000c0000083 from 0xffff83082727b080 to 0xffffffff81730eb0.
> (XEN) traps.c:2653:d0v1 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
> (XEN) d0 attempted to change d0v1's CR4 flags 00002660 -> 00042760
> (XEN) traps.c:2653:d0v2 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
> (XEN) traps.c:2653:d0v2 Domain attempted WRMSR 00000000c0000082 from 0xffff82d0803db000 to 0xffffffff8172e910.
> (XEN) traps.c:2653:d0v2 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0803db080 to 0xffffffff81730c40.
> (XEN) traps.c:2653:d0v2 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
> (XEN) traps.c:2653:d0v2 Domain attempted WRMSR 0000000000000175 from 0xffff82d0803dffc0 to 0x0000000000000000.
> (XEN) traps.c:2653:d0v2 Domain attempted WRMSR 0000000000000176 from 0xffff82d08022c7d0 to 0xffffffff81730c70.
> (XEN) traps.c:2653:d0v2 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0803db080 to 0xffffffff81730eb0.
> (XEN) traps.c:2653:d0v2 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
> (XEN) d0 attempted to change d0v2's CR4 flags 00002660 -> 00042760
> (XEN) traps.c:2653:d0v3 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
> (XEN) traps.c:2653:d0v3 Domain attempted WRMSR 00000000c0000082 from 0xffff82d0803db000 to 0xffffffff8172e910.
> (XEN) traps.c:2653:d0v3 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0803db080 to 0xffffffff81730c40.
> (XEN) traps.c:2653:d0v3 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
> (XEN) traps.c:2653:d0v3 Domain attempted WRMSR 0000000000000175 from 0xffff82d0803dffc0 to 0x0000000000000000.
> (XEN) traps.c:2653:d0v3 Domain attempted WRMSR 0000000000000176 from 0xffff82d08022c7d0 to 0xffffffff81730c70.
> (XEN) traps.c:2653:d0v3 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0803db080 to 0xffffffff81730eb0.
> (XEN) traps.c:2653:d0v3 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
> (XEN) d0 attempted to change d0v3's CR4 flags 00002660 -> 00042760
> (XEN) traps.c:2653:d0v4 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
> (XEN) traps.c:2653:d0v4 Domain attempted WRMSR 00000000c0000082 from 0xffff82d0803db000 to 0xffffffff8172e910.
> (XEN) traps.c:2653:d0v4 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0803db080 to 0xffffffff81730c40.
> (XEN) traps.c:2653:d0v4 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
> (XEN) traps.c:2653:d0v4 Domain attempted WRMSR 0000000000000175 from 0xffff82d0803dffc0 to 0x0000000000000000.
> (XEN) traps.c:2653:d0v4 Domain attempted WRMSR 0000000000000176 from 0xffff82d08022c7d0 to 0xffffffff81730c70.
> (XEN) traps.c:2653:d0v4 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0803db080 to 0xffffffff81730eb0.
> (XEN) traps.c:2653:d0v4 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
> (XEN) d0 attempted to change d0v4's CR4 flags 00002660 -> 00042760
> (XEN) traps.c:2653:d0v5 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
> (XEN) traps.c:2653:d0v5 Domain attempted WRMSR 00000000c0000082 from 0xffff8308272f3000 to 0xffffffff8172e910.
> (XEN) traps.c:2653:d0v5 Domain attempted WRMSR 00000000c0000083 from 0xffff8308272f3080 to 0xffffffff81730c40.
> (XEN) traps.c:2653:d0v5 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
> (XEN) traps.c:2653:d0v5 Domain attempted WRMSR 0000000000000175 from 0xffff8308272f7fc0 to 0x0000000000000000.
> (XEN) traps.c:2653:d0v5 Domain attempted WRMSR 0000000000000176 from 0xffff82d08022c7d0 to 0xffffffff81730c70.
> (XEN) traps.c:2653:d0v5 Domain attempted WRMSR 00000000c0000083 from 0xffff8308272f3080 to 0xffffffff81730eb0.
> (XEN) traps.c:2653:d0v5 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
> (XEN) d0 attempted to change d0v5's CR4 flags 00002660 -> 00042760
> (XEN) traps.c:2653:d0v6 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
> (XEN) traps.c:2653:d0v6 Domain attempted WRMSR 00000000c0000082 from 0xffff82d0803db000 to 0xffffffff8172e910.
> (XEN) traps.c:2653:d0v6 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0803db080 to 0xffffffff81730c40.
> (XEN) traps.c:2653:d0v6 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
> (XEN) traps.c:2653:d0v6 Domain attempted WRMSR 0000000000000175 from 0xffff82d0803dffc0 to 0x0000000000000000.
> (XEN) traps.c:2653:d0v6 Domain attempted WRMSR 0000000000000176 from 0xffff82d08022c7d0 to 0xffffffff81730c70.
> (XEN) traps.c:2653:d0v6 Domain attempted WRMSR 00000000c0000083 from 0xffff82d0803db080 to 0xffffffff81730eb0.
> (XEN) traps.c:2653:d0v6 Domain attempted WRMSR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
> (XEN) d0 attempted to change d0v6's CR4 flags 00002660 -> 00042760
> (XEN) event_fifo.c:50:d0v6 domain 0, port 50 already on a queue
> (XEN) traps.c:2653:d0v7 Domain attempted WRMSR 00000000c0000081 from 0xe023e00800000000 to 0x0023001000000000.
> (XEN) traps.c:2653:d0v7 Domain attempted WRMSR 00000000c0000082 from 0xffff83082727b000 to 0xffffffff8172e910.
> (XEN) traps.c:2653:d0v7 Domain attempted WRMSR 00000000c0000083 from 0xffff83082727b080 to 0xffffffff81730c40.
> (XEN) traps.c:2653:d0v7 Domain attempted WRMSR 0000000000000174 from 0x000000000000e008 to 0x0000000000000010.
> (XEN) traps.c:2653:d0v7 Domain attempR 00000000c0000084 from 0x0000000000074700 to 0x0000000000047700.
> (XEN) d0 attempted to change d0v7's CR4 flags 00002660 -> 00042760
> ^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G^G(XEN) Domain 0 crashed: rebooting machine in 5 seconds.
> 
> telnet> Connection closed.
> [Connecting to system 50 ]
> 
> Great, so 'Domain 0 crashed'!
> 
> Pff - and oddly enough the same kernel suspend/resumes fine on my desktop and laptop.
And yes, it's not about the kernel, it's about Xen.
I've tried like at least 7 Linux kernels from 3.11 to 3.19.

Also did you get the parity error in your bios event log?
About the Bus06 or Bus05(if you have nothing in your PCI-E slots).
Which I think is the Renesas USB 3 controller.
> > 
> > > > > > > > 
> > > > > > > > I will be very glad if you could share any information regarding this
> > > > > > > > matter. 
> > > > > > > > 
> > > > > > > > Best regards,
> > > > > > > > M. Ivanov
> > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > > 
> > > > 
> > > 
> > > 
> > 
> 
> 


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: About Xen bridged pci devices and suspend/resume for the X10SAE motherboard (SuperMicro)

[M. Ivanov]

[-- Attachment #1.1: Type: text/plain, Size: 4190 bytes --]

Hello,

I've experimented with my X10SAE and I think the problem with being
unable to resume after suspending to RAM has something to do with the
PCI Bridge violating the spec by trying to DMA from another address,
since I got a DMAR error about DMA access on address 05:00.0(but in the
bios event log it says Bus06), also the Tundra PCI Bridge is on address
04:00.0. (so like your case with address 7 and 8, but mine's 4 and 5),
btw adding a PCI-E vga seems to change the addresses.

When I disable IOMMU+Xen it works fine, so I mostly sure it's that.
Though I've tried running just Linux with the iommu param on and I
didn't get an error when sleeping/resuming. But I haven't tried doing a
pass-through with it

I've read in a previous thread about a patch of yours for the X10SAE
problem. Which version of Xen can I use it on?(I am currently tinkering
with 4.4.3-RELEASE).

Also I take it - I need to use hack.c to tell xen to create the fake
device,(in my case 05:00.0) and to link it with 04:00.0? But how do I
get that file to compile? Since I don't have a makefile/etc. for it.

Also, can't I just disable the PCI Tundra bridge somehow? And what about
phantom pci and the pciback-hide? Can they help?

I've read about problems regarding the Asmedia controller, so I've
disabled it from the bios, but that didn't help at all.


As for my Xen crash log

DMAR:[DMA Write] Request device [05:00.0] fault addr Can't remember it
DMAR:[fault reason 02] Present bit in context entry is clear

Here is my tree:

00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM
Controller (rev 06)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core
Processor PCI Express x16 Controller (rev 06)
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3
Processor Integrated Graphics Controller (rev 06)
00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core
Processor HD Audio Controller (rev 06)
00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset
Family USB xHCI (rev 05)
00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series
Chipset Family MEI Controller #1 (rev 04)
00:16.3 Serial controller: Intel Corporation 8 Series/C220 Series
Chipset Family KT Controller (rev 04)
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection
I217-LM (rev 05)
00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset
Family USB EHCI #2 (rev 05)
00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset
High Definition Audio Controller (rev 05)
00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset
Family PCI Express Root Port #1 (rev d5)
00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset
Family PCI Express Root Port #4 (rev d5)
00:1c.5 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset
Family PCI Express Root Port #6 (rev d5)
00:1c.6 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset
Family PCI Express Root Port #7 (rev d5)
00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset
Family USB EHCI #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation C226 Series Chipset Family Server
Advanced SKU LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset
Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family
SMBus Controller (rev 05)
00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset
Family Thermal Management Controller (rev 05)
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc.
[AMD/ATI] Cayman PRO [Radeon HD 6950]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI]
Cayman/Antilles HDMI Audio [Radeon HD 6900 Series]
03:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network
Connection (rev 03)
04:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01)
05:03.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22A
IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
06:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host
Controller (rev 02)



[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: About Xen bridged pci devices and suspend/resume for the X10SAE motherboard (SuperMicro)

[Konrad Rzeszutek Wilk]

On Wed, Nov 04, 2015 at 02:49:11AM +0200, M. Ivanov wrote:
> Hello,
> 
> I've experimented with my X10SAE and I think the problem with being
> unable to resume after suspending to RAM has something to do with the
> PCI Bridge violating the spec by trying to DMA from another address,
> since I got a DMAR error about DMA access on address 05:00.0(but in the
> bios event log it says Bus06), also the Tundra PCI Bridge is on address
> 04:00.0. (so like your case with address 7 and 8, but mine's 4 and 5),
> btw adding a PCI-E vga seems to change the addresses.

<nods>
> 
> When I disable IOMMU+Xen it works fine, so I mostly sure it's that.
> Though I've tried running just Linux with the iommu param on and I
> didn't get an error when sleeping/resuming. But I haven't tried doing a
> pass-through with it

<nods>
> 
> I've read in a previous thread about a patch of yours for the X10SAE
> problem. Which version of Xen can I use it on?(I am currently tinkering
> with 4.4.3-RELEASE).

Oh my I can't remember.
> 
> Also I take it - I need to use hack.c to tell xen to create the fake
> device,(in my case 05:00.0) and to link it with 04:00.0? But how do I
> get that file to compile? Since I don't have a makefile/etc. for it.

So.. that motherboard is a pain to work with. I found after numerous
emails to their technical support that the PCI chipset is not capable
of dealing with VT-d. That is PCI passhtrough of any PCI devices - nada.
> 
> Also, can't I just disable the PCI Tundra bridge somehow? And what about
> phantom pci and the pciback-hide? Can they help?

That would be nice.
> 
> I've read about problems regarding the Asmedia controller, so I've
> disabled it from the bios, but that didn't help at all.

Lets take one problem at a time. The current issue you are seeing
is suspend/resume right? That is you just booted Xen + Linux
and ran 'pm-suspend'. And the motherboard did not resume from there?

But it works OK with Linux?

Is AMT enabled on your setup?
> 
> 
> As for my Xen crash log
> 
> DMAR:[DMA Write] Request device [05:00.0] fault addr Can't remember it
> DMAR:[fault reason 02] Present bit in context entry is clear
> 
> Here is my tree:
> 
> 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM
> Controller (rev 06)
> 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core
> Processor PCI Express x16 Controller (rev 06)
> 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3
> Processor Integrated Graphics Controller (rev 06)
> 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core
> Processor HD Audio Controller (rev 06)
> 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset
> Family USB xHCI (rev 05)
> 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series
> Chipset Family MEI Controller #1 (rev 04)
> 00:16.3 Serial controller: Intel Corporation 8 Series/C220 Series
> Chipset Family KT Controller (rev 04)
> 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection
> I217-LM (rev 05)
> 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset
> Family USB EHCI #2 (rev 05)
> 00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset
> High Definition Audio Controller (rev 05)
> 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset
> Family PCI Express Root Port #1 (rev d5)
> 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset
> Family PCI Express Root Port #4 (rev d5)
> 00:1c.5 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset
> Family PCI Express Root Port #6 (rev d5)
> 00:1c.6 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset
> Family PCI Express Root Port #7 (rev d5)
> 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset
> Family USB EHCI #1 (rev 05)
> 00:1f.0 ISA bridge: Intel Corporation C226 Series Chipset Family Server
> Advanced SKU LPC Controller (rev 05)
> 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset
> Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
> 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family
> SMBus Controller (rev 05)
> 00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset
> Family Thermal Management Controller (rev 05)
> 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc.
> [AMD/ATI] Cayman PRO [Radeon HD 6950]
> 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI]
> Cayman/Antilles HDMI Audio [Radeon HD 6900 Series]
> 03:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network
> Connection (rev 03)
> 04:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01)
> 05:03.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22A
> IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
> 06:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host
> Controller (rev 02)
> 
> 

Re: About Xen bridged pci devices and suspend/resume for the X10SAE motherboard (SuperMicro)

[M. Ivanov]

[-- Attachment #1.1: Type: text/plain, Size: 6583 bytes --]

On Wed, 2015-11-04 at 16:06 -0500, Konrad Rzeszutek Wilk wrote:
> On Wed, Nov 04, 2015 at 02:49:11AM +0200, M. Ivanov wrote:
> > Hello,
> > 
> > I've experimented with my X10SAE and I think the problem with being
> > unable to resume after suspending to RAM has something to do with the
> > PCI Bridge violating the spec by trying to DMA from another address,
> > since I got a DMAR error about DMA access on address 05:00.0(but in the
> > bios event log it says Bus06), also the Tundra PCI Bridge is on address
> > 04:00.0. (so like your case with address 7 and 8, but mine's 4 and 5),
> > btw adding a PCI-E vga seems to change the addresses.
> 
> <nods>
> > 
> > When I disable IOMMU+Xen it works fine, so I mostly sure it's that.
> > Though I've tried running just Linux with the iommu param on and I
> > didn't get an error when sleeping/resuming. But I haven't tried doing a
> > pass-through with it
> 
> <nods>
> > 
> > I've read in a previous thread about a patch of yours for the X10SAE
> > problem. Which version of Xen can I use it on?(I am currently tinkering
> > with 4.4.3-RELEASE).
> 
> Oh my I can't remember.
http://lists.xenproject.org/archives/html/xen-devel/2014-02/msg02037.html
I've succeeded in compiling Xen 4.4.3-RELEASE with that patch and i've
also loaded the kernel module you've provided(hack.c) but I've tried
changing the address to 5 instead of 7, since my Tundra is on 4. But
maybe I made some mistake since it didn't make a difference, still
couldn't resume from sleep.
> > 
> > Also I take it - I need to use hack.c to tell xen to create the fake
> > device,(in my case 05:00.0) and to link it with 04:00.0? But how do I
> > get that file to compile? Since I don't have a makefile/etc. for it.
> 
> So.. that motherboard is a pain to work with. I found after numerous
> emails to their technical support that the PCI chipset is not capable
> of dealing with VT-d. That is PCI passhtrough of any PCI devices - nada.
Well, as of my new bios 3.0 I haven't tried passing through devices. But
it probably works, since I could do so in 2.0a(ethernet controllers and
usb controller). The only problem I've experienced is not being able to
resume after suspend.
> > 
> > Also, can't I just disable the PCI Tundra bridge somehow? And what about
> > phantom pci and the pciback-hide? Can they help?
> 
> That would be nice.
If all else fails, maybe I will try modifying the BIOS with a hex
editor.
> > 
> > I've read about problems regarding the Asmedia controller, so I've
> > disabled it from the bios, but that didn't help at all.
> 
> Lets take one problem at a time. The current issue you are seeing
> is suspend/resume right? That is you just booted Xen + Linux
> and ran 'pm-suspend'. And the motherboard did not resume from there?
> 
> But it works OK with Linux?
> 
> Is AMT enabled on your setup?
Hm, I will try disabling it now(since it says Enabled in the bios), but
AMT doesn't work anyway since I've updated my BIOS to 3.0 and it says in
the Supermicro support docs that 3.0 doesn't support AMT even though the
option is in the BIOS. Will buy a Supermicro serial cable these days.
Annoying that I can't use a regular one.

In the meantime, I don't get it why the Xen log isn't working. I'm using
the log_lvl=all and guest_loglvl=all, yet I ain't getting anything
in /var/log/xen. It did work once on Fedora 22 + Xen 4.5, atm on Fedora
23 + Xen 4.5 - doesn't work, nor on Fedora 21 + Xen 4.4.
> > 
> > 
> > As for my Xen crash log
> > 
> > DMAR:[DMA Write] Request device [05:00.0] fault addr Can't remember it
> > DMAR:[fault reason 02] Present bit in context entry is clear
> > 
> > Here is my tree:
> > 
> > 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM
> > Controller (rev 06)
> > 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core
> > Processor PCI Express x16 Controller (rev 06)
> > 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3
> > Processor Integrated Graphics Controller (rev 06)
> > 00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core
> > Processor HD Audio Controller (rev 06)
> > 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset
> > Family USB xHCI (rev 05)
> > 00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series
> > Chipset Family MEI Controller #1 (rev 04)
> > 00:16.3 Serial controller: Intel Corporation 8 Series/C220 Series
> > Chipset Family KT Controller (rev 04)
> > 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection
> > I217-LM (rev 05)
> > 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset
> > Family USB EHCI #2 (rev 05)
> > 00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset
> > High Definition Audio Controller (rev 05)
> > 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset
> > Family PCI Express Root Port #1 (rev d5)
> > 00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset
> > Family PCI Express Root Port #4 (rev d5)
> > 00:1c.5 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset
> > Family PCI Express Root Port #6 (rev d5)
> > 00:1c.6 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset
> > Family PCI Express Root Port #7 (rev d5)
> > 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset
> > Family USB EHCI #1 (rev 05)
> > 00:1f.0 ISA bridge: Intel Corporation C226 Series Chipset Family Server
> > Advanced SKU LPC Controller (rev 05)
> > 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset
> > Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
> > 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family
> > SMBus Controller (rev 05)
> > 00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset
> > Family Thermal Management Controller (rev 05)
> > 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc.
> > [AMD/ATI] Cayman PRO [Radeon HD 6950]
> > 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI]
> > Cayman/Antilles HDMI Audio [Radeon HD 6900 Series]
> > 03:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network
> > Connection (rev 03)
> > 04:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01)
> > 05:03.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22A
> > IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
> > 06:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host
> > Controller (rev 02)
> > 
> > 
> 
> 


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel