Hi Konrad and all,

Patch was applied against kernel 4.4.12-9.pvops.qubes.x86_64 and tested
against Xen 4.6.1-17.fc23.

Quick result:
user@user-ThinkPad-X200:~$ grep "DMA ops" XEN\{iommu\=*
XEN{iommu=0} KERNEL{rhgb drm.debug=255}:intel_gtt_init: Using DMA ops
XEN{iommu=1} KERNEL{drm.debug=255}:intel_gtt_init: Using DMA ops
XEN{iommu=1} KERNEL{rhgb drm.debug=255
intel_iommu=igfx_off}:intel_gtt_init: Using DMA ops
XEN{iommu=no-igfx} KERNEL{rhgb drm.debug=255}:intel_gtt_init: Using DMA ops
XEN{iommu=no-igfx} KERNEL{rhgb drm.debug=255
intel_iommu=igfx_off}:intel_gtt_init: Using DMA ops

With iommu=1:
Unfortunately, the behavior haven't changed: it's impossible to boot with
default xen cmd line (iommu=1). System hangs at "clearing unused GTT
space".

The following message seems problematic:
"XEN{iommu=1} KERNEL{drm.debug=255}:(XEN) [VT-D]DMAR:[DMA Write] Request
device [0000:00:02.0] fault addr 200000000, iommu reg = ffff82c000203000"
seems to be problematic.

With iommu=no-igfx, it hangs after vt-d remappings:
XEN{iommu=no-igfx} KERNEL{rhgb drm.debug=255 intel_iommu=igfx_off}:(XEN)
[VT-D]d0:PCI: unmap 0000:00:19.0
XEN{iommu=no-igfx} KERNEL{rhgb drm.debug=255 intel_iommu=igfx_off}:(XEN)
[VT-D]d1:PCI: map 0000:00:19.0
XEN{iommu=no-igfx} KERNEL{rhgb drm.debug=255 intel_iommu=igfx_off}:(XEN)
[VT-D]d0:PCIe: unmap 0000:03:00.0
XEN{iommu=no-igfx} KERNEL{rhgb drm.debug=255 intel_iommu=igfx_off}:(XEN)
[VT-D]d1:PCIe: map 0000:03:00.0
Those devices are network devices, which netvm attempts to use
exclusively.

With iommu=0, it boots.

See attached logs collected with AMT SOL and previous post for lspci -v
output. Thanks for your time.

Regards,
Thierry

Le mar. 5 juil. 2016 à 22:07, Thierry Laurion <thierry.laurion@gmail.com> a
écrit :

> I Konrad, first, thanks for your input and your time, it is much
> appreciated.
>
> I understand that those changes are torward the linux kernel, which is
> used by xen compilation. I applied the changes and i'm rebuilding Qubes
> with xen 4.6.1 based on a kernel-4.1.24. Will test the build in the next
> days and post back the results.
>
> output of sudo lspci -v from dom0:
> 00:00.0 Host bridge: Intel Corporation Mobile 4 Series Chipset Memory
> Controller Hub (rev 07)
>     Subsystem: Lenovo Device 20e0
>     Flags: bus master, fast devsel, latency 0
>     Capabilities: [e0] Vendor Specific Information: Len=0a <?>
>     Kernel driver in use: agpgart-intel
>
> 00:02.0 VGA compatible controller: Intel Corporation Mobile 4 Series
> Chipset Integrated Graphics Controller (rev 07) (prog-if 00 [VGA
> controller])
>     Subsystem: Lenovo Device 20e4
>     Flags: bus master, fast devsel, latency 0, IRQ 47
>     Memory at e1000000 (64-bit, non-prefetchable) [size=4M]
>     Memory at d0000000 (64-bit, prefetchable) [size=256M]
>     I/O ports at 3400 [size=8]
>     Expansion ROM at <unassigned> [disabled]
>     Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
>     Capabilities: [d0] Power Management version 3
>     Kernel driver in use: i915
>     Kernel modules: i915
>
> 00:02.1 Display controller: Intel Corporation Mobile 4 Series Chipset
> Integrated Graphics Controller (rev 07)
>     Subsystem: Lenovo Device 20e4
>     Flags: fast devsel
>     Memory at e1400000 (64-bit, non-prefetchable) [size=1M]
>     Capabilities: [d0] Power Management version 3
>
> 00:19.0 Ethernet controller: Intel Corporation 82567LF Gigabit Network
> Connection (rev 03)
>     Subsystem: Lenovo Device 20ee
>     Flags: bus master, fast devsel, latency 0, IRQ 60
>     Memory at e1600000 (32-bit, non-prefetchable) [size=128K]
>     Memory at e1624000 (32-bit, non-prefetchable) [size=4K]
>     I/O ports at 3000 [size=32]
>     Capabilities: [c8] Power Management version 2
>     Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
>     Capabilities: [e0] PCI Advanced Features
>     Kernel driver in use: pciback
>     Kernel modules: e1000e
>
> 00:1a.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
> Controller #4 (rev 03) (prog-if 00 [UHCI])
>     Subsystem: Lenovo Device 20f0
>     Flags: bus master, medium devsel, latency 0, IRQ 16
>     I/O ports at 3020 [size=32]
>     Capabilities: [50] PCI Advanced Features
>     Kernel driver in use: pciback
>     Kernel modules: uhci_hcd
>
> 00:1a.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
> Controller #5 (rev 03) (prog-if 00 [UHCI])
>     Subsystem: Lenovo Device 20f0
>     Flags: bus master, medium devsel, latency 0, IRQ 17
>     I/O ports at 3040 [size=32]
>     Capabilities: [50] PCI Advanced Features
>     Kernel driver in use: pciback
>     Kernel modules: uhci_hcd
>
> 00:1a.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
> Controller #6 (rev 03) (prog-if 00 [UHCI])
>     Subsystem: Lenovo Device 20f0
>     Flags: bus master, medium devsel, latency 0, IRQ 18
>     I/O ports at 3060 [size=32]
>     Capabilities: [50] PCI Advanced Features
>     Kernel driver in use: pciback
>     Kernel modules: uhci_hcd
>
> 00:1a.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI
> Controller #2 (rev 03) (prog-if 20 [EHCI])
>     Subsystem: Lenovo Device 20f1
>     Flags: bus master, medium devsel, latency 0, IRQ 18
>     Memory at e1626000 (32-bit, non-prefetchable) [size=1K]
>     Capabilities: [50] Power Management version 2
>     Capabilities: [58] Debug port: BAR=1 offset=00a0
>     Capabilities: [98] PCI Advanced Features
>     Kernel driver in use: pciback
>     Kernel modules: ehci_pci
>
> 00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio
> Controller (rev 03)
>     Subsystem: Lenovo Device 20f2
>     Flags: bus master, fast devsel, latency 0, IRQ 48
>     Memory at e1620000 (64-bit, non-prefetchable) [size=16K]
>     Capabilities: [50] Power Management version 2
>     Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit+
>     Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
>     Capabilities: [100] Virtual Channel
>     Capabilities: [130] Root Complex Link
>     Kernel driver in use: snd_hda_intel
>     Kernel modules: snd_hda_intel
>
> 00:1c.0 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express
> Port 1 (rev 03) (prog-if 00 [Normal decode])
>     Flags: bus master, fast devsel, latency 0, IRQ 40
>     Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
>     Capabilities: [40] Express Root Port (Slot-), MSI 00
>     Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
>     Capabilities: [90] Subsystem: Lenovo Device 20f3
>     Capabilities: [a0] Power Management version 2
>     Capabilities: [100] Virtual Channel
>     Capabilities: [180] Root Complex Link
>     Kernel driver in use: pcieport
>     Kernel modules: shpchp
>
> 00:1c.1 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express
> Port 2 (rev 03) (prog-if 00 [Normal decode])
>     Flags: bus master, fast devsel, latency 0, IRQ 41
>     Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
>     Memory behind bridge: e1500000-e15fffff
>     Capabilities: [40] Express Root Port (Slot-), MSI 00
>     Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
>     Capabilities: [90] Subsystem: Lenovo Device 20f3
>     Capabilities: [a0] Power Management version 2
>     Capabilities: [100] Virtual Channel
>     Capabilities: [180] Root Complex Link
>     Kernel driver in use: pcieport
>     Kernel modules: shpchp
>
> 00:1c.2 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express
> Port 3 (rev 03) (prog-if 00 [Normal decode])
>     Flags: bus master, fast devsel, latency 0, IRQ 42
>     Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
>     Capabilities: [40] Express Root Port (Slot-), MSI 00
>     Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
>     Capabilities: [90] Subsystem: Lenovo Device 20f3
>     Capabilities: [a0] Power Management version 2
>     Capabilities: [100] Virtual Channel
>     Capabilities: [180] Root Complex Link
>     Kernel driver in use: pcieport
>     Kernel modules: shpchp
>
> 00:1c.3 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express
> Port 4 (rev 03) (prog-if 00 [Normal decode])
>     Flags: bus master, fast devsel, latency 0, IRQ 43
>     Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
>     I/O behind bridge: 00002000-00002fff
>     Memory behind bridge: e0800000-e0ffffff
>     Prefetchable memory behind bridge: 00000000e0000000-00000000e07fffff
>     Capabilities: [40] Express Root Port (Slot-), MSI 00
>     Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
>     Capabilities: [90] Subsystem: Lenovo Device 20f3
>     Capabilities: [a0] Power Management version 2
>     Capabilities: [100] Virtual Channel
>     Capabilities: [180] Root Complex Link
>     Kernel driver in use: pcieport
>     Kernel modules: shpchp
>
> 00:1d.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
> Controller #1 (rev 03) (prog-if 00 [UHCI])
>     Subsystem: Lenovo Device 20f0
>     Flags: bus master, medium devsel, latency 0, IRQ 16
>     I/O ports at 3080 [size=32]
>     Capabilities: [50] PCI Advanced Features
>     Kernel driver in use: pciback
>     Kernel modules: uhci_hcd
>
> 00:1d.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
> Controller #2 (rev 03) (prog-if 00 [UHCI])
>     Subsystem: Lenovo Device 20f0
>     Flags: bus master, medium devsel, latency 0, IRQ 17
>     I/O ports at 30a0 [size=32]
>     Capabilities: [50] PCI Advanced Features
>     Kernel driver in use: pciback
>     Kernel modules: uhci_hcd
>
> 00:1d.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
> Controller #3 (rev 03) (prog-if 00 [UHCI])
>     Subsystem: Lenovo Device 20f0
>     Flags: bus master, medium devsel, latency 0, IRQ 18
>     I/O ports at 30c0 [size=32]
>     Capabilities: [50] PCI Advanced Features
>     Kernel driver in use: pciback
>     Kernel modules: uhci_hcd
>
> 00:1d.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI
> Controller #1 (rev 03) (prog-if 20 [EHCI])
>     Subsystem: Lenovo Device 20f1
>     Flags: bus master, medium devsel, latency 0, IRQ 16
>     Memory at e1627000 (32-bit, non-prefetchable) [size=1K]
>     Capabilities: [50] Power Management version 2
>     Capabilities: [58] Debug port: BAR=1 offset=00a0
>     Capabilities: [98] PCI Advanced Features
>     Kernel driver in use: pciback
>     Kernel modules: ehci_pci
>
> 00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev 93)
> (prog-if 01 [Subtractive decode])
>     Flags: fast devsel
>     Bus: primary=00, secondary=05, subordinate=05, sec-latency=32
>     Capabilities: [50] Subsystem: Lenovo Device 20f4
>
> 00:1f.0 ISA bridge: Intel Corporation ICH9M LPC Interface Controller (rev
> 03)
>     Subsystem: Lenovo Device 20f5
>     Flags: bus master, medium devsel, latency 0
>     Capabilities: [e0] Vendor Specific Information: Len=0c <?>
>     Kernel driver in use: lpc_ich
>     Kernel modules: lpc_ich
>
> 00:1f.2 SATA controller: Intel Corporation 82801IBM/IEM (ICH9M/ICH9M-E) 4
> port SATA Controller [AHCI mode] (rev 03) (prog-if 01 [AHCI 1.0])
>     Subsystem: Lenovo Device 20f8
>     Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 46
>     I/O ports at 3408 [size=8]
>     I/O ports at 3418 [size=4]
>     I/O ports at 3410 [size=8]
>     I/O ports at 341c [size=4]
>     I/O ports at 30e0 [size=32]
>     Memory at e1625000 (32-bit, non-prefetchable) [size=2K]
>     Capabilities: [80] MSI: Enable+ Count=1/16 Maskable- 64bit-
>     Capabilities: [70] Power Management version 3
>     Capabilities: [a8] SATA HBA v1.0
>     Capabilities: [b0] PCI Advanced Features
>     Kernel driver in use: ahci
>
> 00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller
> (rev 03)
>     Subsystem: Lenovo Device 20f9
>     Flags: medium devsel, IRQ 18
>     Memory at e1628000 (64-bit, non-prefetchable) [size=256]
>     I/O ports at 0400 [size=32]
>     Kernel modules: i2c_i801
>
> 02:00.0 Network controller: Qualcomm Atheros AR9285 Wireless Network
> Adapter (PCI-Express) (rev 01)
>     Subsystem: Foxconn International, Inc. T77H126.00 802.11bgn Wireless
> Half-size Mini PCIe Card
>     Flags: bus master, fast devsel, latency 0, IRQ 17
>     Memory at e1500000 (64-bit, non-prefetchable) [size=64K]
>     Capabilities: [40] Power Management version 3
>     Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit-
>     Capabilities: [60] Express Legacy Endpoint, MSI 00
>     Capabilities: [100] Advanced Error Reporting
>     Capabilities: [140] Virtual Channel
>     Capabilities: [160] Device Serial Number 00-15-17-ff-ff-24-14-12
>     Capabilities: [170] Power Budgeting <?>
>     Kernel driver in use: pciback
>     Kernel modules: ath9k
>

>
>
> Le jeu. 30 juin 2016 à 09:37, Konrad Rzeszutek Wilk <
> konrad.wilk@oracle.com> a écrit :
>
>> On Sun, Jun 26, 2016 at 11:48:44PM +0000, Thierry Laurion wrote:
>> > Sorry for the precedent post that was written a bit too fast. Libreboot
>> was
>> > flashed when I wrote it, which is the equivalent of a having vt-d
>> > deactivated (iommu=0). Thanks to a user that read this post and wrote
>> to me
>> > personally so I could do my mea culpa. Sorry for the precedent
>> misleading
>> > post.
>> >
>> > Xen on a GM45 chipset and with IGD i915 driver is still getting the
>> system
>> > hanged when vt-d is activated. I'm willing to borrow a machine to the
>> Xen
>> > developer that could fix the iommu=no-igfx code for gm45 chipset to
>> > actually work.
>>
>> This sounds like http://wiki.xenproject.org/wiki/Paravirtualized_DRM
>> issues.
>>
>> Can you try and also attach lspci -v ?
>>
>>
>> diff --git a/drivers/char/agp/intel-gtt.c b/drivers/char/agp/intel-gtt.c
>> index aef87fd..cf31aad 100644
>> --- a/drivers/char/agp/intel-gtt.c
>> +++ b/drivers/char/agp/intel-gtt.c
>> @@ -35,7 +35,7 @@
>>  #ifdef CONFIG_INTEL_IOMMU
>>  #define USE_PCI_DMA_API 1
>>  #else
>> -#define USE_PCI_DMA_API 0
>> +#define USE_PCI_DMA_API 1
>>  #endif
>>
>>  struct intel_gtt_driver {
>> @@ -654,6 +654,7 @@ static int intel_gtt_init(void)
>>
>>         intel_private.needs_dmar = USE_PCI_DMA_API && INTEL_GTT_GEN > 2;
>>
>> +       printk("%s: %s DMA ops\n", __func__,intel_private.needs_dmar ?
>> "Using" : "Not using");
>>         ret = intel_gtt_setup_scratch_page();
>>         if (ret != 0) {
>>                 intel_gtt_cleanup();
>> >
>> > A ticket is opened here with current states of thing:
>> >
>> https://github.com/QubesOS/qubes-issues/issues/1594#issuecomment-209213917
>> >
>> > Sorry about that (and repost since I wrote the same misleading post to
>> two
>> > places)
>> > Thierry
>> >
>> > Le dim. 28 févr. 2016 à 14:03, Thierry Laurion <
>> thierry.laurion@gmail.com>
>> > a écrit :
>> >
>> > > The problem wasn't with xen iommu support but kms/drm and i915 driver.
>> > >
>> > > Passing to the kernel i915.preliminary_hw_support=1 fixes it all :)
>> > >
>> > > Thanks
>> > >
>> > > Le mer. 6 janv. 2016 à 22:11, Thierry Laurion <
>> thierry.laurion@gmail.com>
>> > > a écrit :
>> > >
>> > >> Nope. That commit is present in 4.6 and results in x200 being able to
>> > >> boot xen.
>> > >>
>> > >> Not having that option makes xen hang at boot.
>> > >>
>> > >> If present, it works until other vm access pass-through devices,
>> which
>> > >> I'm not able to troubleshoot even through amt SOL.
>> > >>
>> > >> See here for debug logs:
>> > >> https://groups.google.com/forum/m/#!topic/qubes-users/bHQHjXqinaU
>> > >>
>> > >> Le mer. 6 janv. 2016 09:35, Jan Beulich <JBeulich@suse.com> a écrit
>> :
>> > >>
>> > >>> >>> On 22.12.15 at 19:04, <thierry.laurion@gmail.com> wrote:
>> > >>> > iommu=no-igfx is a gamechanger for Qubes support through 3.1 RC1
>> > >>> release,
>> > >>> > thanks to Xen 4.6 :)
>> > >>> >
>> > >>> > The Lenovo X200 supports vt-x, vt-d and TPM as reported and
>> required by
>> > >>> > Qubes in the HCL attached to this e-mail. The problem is that when
>> > >>> Qubes
>> > >>> > launches it's netvm which uses IOMMU to talk to it's network
>> card, it
>> > >>> > freezes the whole system up. Even when specifying sync_console, I
>> > >>> don't get
>> > >>> > much more verbosity. I ordered a PCMCIA to serial adapter which
>> will be
>> > >>> > shipped to my door late January... Meanwhile, booting with iommu=0
>> > >>> makes
>> > >>> > things work, but a potential hardware component being compromised
>> has
>> > >>> > chances to compromise the whole system since compartmentalization
>> is
>> > >>> not
>> > >>> > guaranteed without IOMMU (vt-d).
>> > >>> >
>> > >>> > A little more love is needed from xen to make that laptop line
>> > >>> supported by
>> > >>> > Qubes and a nice alternative to the costy Librem currently
>> promoted by
>> > >>> > Qubes-Purism
>> > >>> > partnership
>> > >>>
>> > >>> Is all of the above and below a quite complicated way of expressing
>> > >>> that you'd like to see commit 146341187a backported to 4.6.x?
>> > >>>
>> > >>> Jan
>> > >>>
>> > >>> > <
>> > >>>
>> http://arstechnica.com/gadgets/2015/12/qubes-os-will-ship-pre-installed-on-p
>> > >>> > urisms-security-focused-librem-13-laptop/>which
>> > >>> > suggest that the laptop will be Respect Your Freedom compliant in
>> the
>> > >>> > future with Intel participation in removing ME and AMT
>> > >>> > <http://libreboot.org/faq/#intelme>, which is not guaranteed at
>> all.
>> > >>> > <
>> > >>>
>> http://www.phoronix.com/scan.php?page=news_item&px=Purism-Librem-Still-Blobbe
>> > >>> > d>
>> > >>> > If Xen 4.6 can cooperate with Penryn GM45 chipset, it's all
>> MiniFree
>> > >>> laptops
>> > >>> > <http://minifree.org/product-category/laptops/> (and Libreboot
>> > >>> support of
>> > >>> > those <http://libreboot.org/docs/hcl/x200.html>) that will be
>> > >>> potential
>> > >>> > candidates!
>> > >>> > Please share the love so that the community has a cheap
>> alternative.
>> > >>> >
>> > >>> > Requirements to replicate bug:
>> > >>> > Model: X200 745434U with p8700 CPU running 1067a
>> microcode(important),
>> > >>> > upgrable to 8go
>> > >>> > BIOS: Lenovo 3.22/1.07 (latest from 2013
>> > >>> > <http://support.lenovo.com/ca/en/downloads/ds015007>)
>> > >>> > Network card supports FLReset+ as requested here
>> > >>> > <http://wiki.xen.org/wiki/VTd_HowTo>.
>> > >>> > Bios settings: vt-d and vt-x needs to be enforced.
>> > >>> > Xen command line option required
>> > >>> > <http://www.gossamer-threads.com/lists/xen/devel/393647> to boot:
>> > >>> > iommu=no-igfx
>> > >>> >
>> > >>> > Here is the current debug trace/status on Qubes side of things
>> > >>> > <https://groups.google.com/forum/#!topic/qubes-users/bHQHjXqinaU
>> >.
>> > >>> > If you have any hint, please contribute :)
>> > >>> >
>> > >>> > Help me say happy new years to all security conscious people out
>> there
>> > >>> :)
>> > >>> >
>> > >>> > Merry Christmas all,
>> > >>> > Thierry Laurion
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > --
>> > >>> > Thierry Laurion
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>>
>> > _______________________________________________
>> > Xen-devel mailing list
>> > Xen-devel@lists.xen.org
>> > http://lists.xen.org/xen-devel
>>
>>