According to Jans suggestion, I did a second build with xen up to the changeset he suggested and attached another round of logs.
Also, to answer Yangs questions, the devices 00:12.0 and 00:13.0 are regular noboard USB controller from the mainboard:
00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
But I did do a second run without them passed through the domU and added the log (xldmesg_secondrun.log). Since I'm seeing IO_PAGE_FAULTs there (05:00.0 is the vga i'm passing), I would assume that basically any device I pass through causes these page faults and it has nothing to do with specific faulty devices.