From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45563) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aTNpN-0002Wf-5X for qemu-devel@nongnu.org; Wed, 10 Feb 2016 01:00:39 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aTNpK-00066a-7W for qemu-devel@nongnu.org; Wed, 10 Feb 2016 01:00:37 -0500 Received: from mx1.redhat.com ([209.132.183.28]:33406) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aTNpJ-00066R-S7 for qemu-devel@nongnu.org; Wed, 10 Feb 2016 01:00:34 -0500 Date: Tue, 9 Feb 2016 23:00:30 -0700 From: Alex Williamson Message-ID: <20160209230030.7fa2e029@t450s.home> In-Reply-To: <003AAFE53969E14CB1F09B6FD68C3CD47BB75E56@ORSMSX106.amr.corp.intel.com> References: <1436938670-7677-1-git-send-email-tiejun.chen@intel.com> <1436938670-7677-4-git-send-email-tiejun.chen@intel.com> <20160208203235.69088310@t450s.home> <20160209104332.775b88f8@t450s.home> <003AAFE53969E14CB1F09B6FD68C3CD47BB7580F@ORSMSX106.amr.corp.intel.com> <20160209143253.0d73e002@t450s.home> <003AAFE53969E14CB1F09B6FD68C3CD47BB75E56@ORSMSX106.amr.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [v10][PATCH 03/10] piix: create host bridge to passthrough List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Kay, Allen M" Cc: "Ruan, Shuai" , "ehabkost@redhat.com" , "stefano.stabellini@eu.citrix.com" , "mst@redhat.com" , "qemu-devel@nongnu.org" , "pbonzini@redhat.com" , "rth@twiddle.net" Hi Allen, On Wed, 10 Feb 2016 03:40:54 +0000 "Kay, Allen M" wrote: > > -----Original Message----- > > From: Alex Williamson [mailto:alex.williamson@redhat.com] > > Sent: Tuesday, February 09, 2016 1:33 PM > > To: Kay, Allen M > > Cc: ehabkost@redhat.com; mst@redhat.com; > > stefano.stabellini@eu.citrix.com; qemu-devel@nongnu.org; > > pbonzini@redhat.com; rth@twiddle.net; Ruan, Shuai > > > > Subject: Re: [Qemu-devel] [v10][PATCH 03/10] piix: create host bridge to > > passthrough > > > > Hi Allen, > > > Hi Alex, > > > Thanks for the reply. I'm totally onboard with you for BDW/SKL for > > supported platforms, but I'd like to understand what the incremental > > investment is for each back step within reason for older GPUs, at least for > > best-effort, community support. If we want to assume BDW/SKL and > > Universal Passthrough Mode, then we could abandon the host bridge and > > ISA bridge modifications altogether, right? > > Right. > > > On the other hand, it's not clear > > to me that UPT drivers are that pervasive and if we need to enable some > > degree of host bridge/ISA bridge poke thrus, why not enable a fair chunk of > > users, including me. > > > > Sounds good. > > > My IVB desktop seems to be working well (win10 + linux) with only poking > > through the standard host bridge and ISA bridge identifications. Yes, I need > > to know about the different GMCH layout of IVB vs BDW in the IGD device, > > but I've already taken care of that. > > Other than VendorID/DeviceID, which other PCI config fields in host bridge and ISA bridge fields did you have to passthrough to get your IVB to work. Are you passing IGD through as primary or secondary? The host bridge I stripped down to: {PCI_REVISION_ID, 2}, {PCI_SUBSYSTEM_VENDOR_ID, 2}, {PCI_SUBSYSTEM_ID, 2}, And used this for the ISA bridge: {PCI_VENDOR_ID, 2}, {PCI_DEVICE_ID, 2}, {PCI_REVISION_ID, 2}, {PCI_SUBSYSTEM_VENDOR_ID, 2}, {PCI_SUBSYSTEM_ID, 2}, So nothing to do with GMCH/BDSM seems to be necessary for the IVB desktop system. I didn't try to trim further. I can run that system as either primary or secondary. On both systems I'm using pci-stub to keep i915 from interfering on the host, on the SNB laptop I also use video=efifb:off since it's configured for UEFI boot, I'm not sure if that play any role in why it's not working. I've tried adding a bunch more registers on the SNB system to see if it helps, for the host bridge: {0x50, 2}, {0x52, 2}, {0xa4, 4}, {0xa8, 4}, (ie. the registers Tiejun added with this patch) And on the ISA bridge: {0x40, 4}, {0x44, 4}, {0x48, 4}, {0x4c, 4}, {0xf0, 4}, These were registers I saw accessed with tracing enabled, but nothing seemed to change by adding them. I don't see any accesses to the BDSM register on the host bridge, but I do see GMCH accesses. Sort of interesting is that the guest reads 201h from that register and tries to write 203h. In the read case we have a 2MB GGMS and GGCLCK set, the guest tries to set IVD (IGD VGA Disable). I'm not sure why it bothers to try to do this with GGCLCK indicated since the register is locked, but it is slightly troubling that the spec indicates that the BIOS must *not* set IVD to zero (VGA enabled) if GMS pre-allocates no memory, which is exactly what we're doing to try to indicate no stolen memory. If we could actually disable VGA on IGD, that might be a nice option, but I know there are issues with that (iirc, there's no disable for one of either the VGA MMIO or I/O port space, so the only option is to disable that address space at the PCI command register, which has obvious implications - I think it was probably I/O port space). I'm currently trying a Linux live CD on the guest for the SNB, mostly because I can see driver error messages and look at the driver source, neither of which are available to me for the Windows driver. Both of the driver warnings I get are from the driver thinking a certain feature should be disabled but finds it enabled. They're also sort of on a strange drm release path, maybe due to being in secondary mode and Xorg not being configured to use the IGD on this image. In case it looks familiar: [ 12.397908] ------------[ cut here ]------------ [ 12.399175] WARNING: CPU: 0 PID: 1135 at drivers/gpu/drm/i915/intel_display.c:1259 assert_fdi_rx_pll+0x78/0xb0 [i915]() [ 12.401720] FDI RX PLL assertion failure (expected off, current on) [ 12.403239] Modules linked in: ebtable_broute bridge stp llc ebtable_filter ebtable_nat ebtables ip6table_security ip6table_mangle ip6table_raw ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_filter ip6_tables iptable_security iptable_mangle iptable_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iTCO_wdt gpio_ich iTCO_vendor_support ppdev joydev parport_pc lpc_ich i2c_piix4 parport acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace isofs squashfs i915 bochs_drm video ttm i2c_algo_bit drm_kms_helper drm ata_generic serio_raw pata_acpi scsi_dh_rdac scsi_dh_emc scsi_dh_alua sunrpc loop [ 12.417994] CPU: 0 PID: 1135 Comm: Xorg Not tainted 4.2.3-300.fc23.x86_64 #1 [ 12.419733] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.0-80-g2390eb5 04/01/2014 [ 12.422100] 0000000000000000 00000000d92c128c ffff880078f53ab8 ffffffff81771fca [ 12.424023] 0000000000000000 ffff880078f53b10 ffff880078f53af8 ffffffff8109e4a6 [ 12.425949] ffff880078f53ad8 0000000000000000 ffff88007f680000 ffff88007fb72000 [ 12.427719] Call Trace: [ 12.428354] [] dump_stack+0x45/0x57 [ 12.429580] [] warn_slowpath_common+0x86/0xc0 [ 12.431037] [] warn_slowpath_fmt+0x55/0x70 [ 12.432497] [] assert_fdi_rx_pll+0x78/0xb0 [i915] [ 12.434033] [] intel_pre_enable_lvds+0x53/0x180 [i915] [ 12.435760] [] ironlake_crtc_enable+0x199/0xd30 [i915] [ 12.437433] [] ? intel_runtime_pm_put+0x50/0x60 [i915] [ 12.439083] [] ? intel_display_power_put+0x108/0x160 [i915] [ 12.440914] [] __intel_set_mode+0x916/0xb60 [i915] [ 12.442458] [] intel_crtc_set_config+0x2b6/0x580 [i915] [ 12.444180] [] drm_mode_set_config_internal+0x66/0x100 [drm] [ 12.445917] [] restore_fbdev_mode+0xc2/0xf0 [drm_kms_helper] [ 12.447695] [] drm_fb_helper_restore_fbdev_mode_unlocked+0x29/0x70 [drm_kms_helper] [ 12.450116] [] intel_fbdev_restore_mode+0x1e/0x50 [i915] [ 12.451794] [] i915_driver_lastclose+0xe/0x20 [i915] [ 12.453360] [] drm_lastclose+0x2e/0x140 [drm] [ 12.454755] [] drm_release+0x2af/0x490 [drm] [ 12.456324] [] __fput+0xdc/0x1e0 [ 12.457525] [] ____fput+0xe/0x10 [ 12.458736] [] task_work_run+0x85/0xb0 [ 12.460028] [] do_notify_resume+0x8d/0x90 [ 12.461424] [] int_signal+0x12/0x17 [ 12.462757] ---[ end trace 3278f15671a6fa94 ]--- [ 12.546650] ------------[ cut here ]------------ [ 12.547765] WARNING: CPU: 0 PID: 1135 at drivers/gpu/drm/i915/intel_display.c:1466 assert_pch_transcoder_disabled+0x67/0x70 [i915]() [ 12.550847] transcoder assertion failed, should be off on pipe A but is still active [ 12.552858] Modules linked in: ebtable_broute bridge stp llc ebtable_filter ebtable_nat ebtables ip6table_security ip6table_mangle ip6table_raw ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_filter ip6_tables iptable_security iptable_mangle iptable_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iTCO_wdt gpio_ich iTCO_vendor_support ppdev joydev parport_pc lpc_ich i2c_piix4 parport acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace isofs squashfs i915 bochs_drm video ttm i2c_algo_bit drm_kms_helper drm ata_generic serio_raw pata_acpi scsi_dh_rdac scsi_dh_emc scsi_dh_alua sunrpc loop [ 12.568444] CPU: 0 PID: 1135 Comm: Xorg Tainted: G W 4.2.3-300.fc23.x86_64 #1 [ 12.570599] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.0-80-g2390eb5 04/01/2014 [ 12.572988] 0000000000000000 00000000d92c128c ffff880078f53b08 ffffffff81771fca [ 12.574973] 0000000000000000 ffff880078f53b60 ffff880078f53b48 ffffffff8109e4a6 [ 12.576826] ffff880078f53b28 0000000000000000 ffff88007f680000 ffff88007fb72000 [ 12.578804] Call Trace: [ 12.579454] [] dump_stack+0x45/0x57 [ 12.580775] [] warn_slowpath_common+0x86/0xc0 [ 12.582419] [] warn_slowpath_fmt+0x55/0x70 [ 12.583924] [] assert_pch_transcoder_disabled+0x67/0x70 [i915] [ 12.585834] [] ironlake_crtc_enable+0x5ec/0xd30 [i915] [ 12.587634] [] ? intel_runtime_pm_put+0x50/0x60 [i915] [ 12.589402] [] ? intel_display_power_put+0x108/0x160 [i915] [ 12.591294] [] __intel_set_mode+0x916/0xb60 [i915] [ 12.593019] [] intel_crtc_set_config+0x2b6/0x580 [i915] [ 12.594797] [] drm_mode_set_config_internal+0x66/0x100 [drm] [ 12.596651] [] restore_fbdev_mode+0xc2/0xf0 [drm_kms_helper] [ 12.598574] [] drm_fb_helper_restore_fbdev_mode_unlocked+0x29/0x70 [drm_kms_helper] [ 12.601022] [] intel_fbdev_restore_mode+0x1e/0x50 [i915] [ 12.602888] [] i915_driver_lastclose+0xe/0x20 [i915] [ 12.604576] [] drm_lastclose+0x2e/0x140 [drm] [ 12.606098] [] drm_release+0x2af/0x490 [drm] [ 12.607662] [] __fput+0xdc/0x1e0 [ 12.609019] [] ____fput+0xe/0x10 [ 12.610283] [] task_work_run+0x85/0xb0 [ 12.611718] [] do_notify_resume+0x8d/0x90 [ 12.613225] [] int_signal+0x12/0x17 [ 12.614600] ---[ end trace 3278f15671a6fa95 ]--- [ 12.635700] [drm:intel_set_pch_fifo_underrun_reporting [i915]] *ERROR* uncleared pch fifo underrun on pch transcoder A [ 12.636007] [drm:intel_pch_fifo_underrun_irq_handler [i915]] *ERROR* PCH transcoder A FIFO underrun [ 12.642883] [drm:intel_cpu_fifo_underrun_irq_handler [i915]] *ERROR* CPU pipe A FIFO underrun Otherwise i915 seems fairly happy: [ 4.940225] i915: No ACPI video bus found [ 4.943223] [drm] Initialized i915 1.6.0 20150522 for 0000:00:03.0 on minor 1 [ 5.000053] [drm] GMBUS [i915 gmbus vga] timed out, falling back to bit banging on pin 2 [ 5.038055] i915 0000:00:03.0: fb1: inteldrmfb frame buffer device The bit banging messages is apparently normal on this system (Lenovo W520, Nvidia disabled in BIOS - could optimus still be getting in the way?). I don't know that any of the above is actually fatal though, I just know that I don't get a signal out of the LVDS panel or any of the other outputs. However, the Xorg log file looks normal and if I disable mmap support in vfio and move the mouse cursor around on the VNC window, the trace log looks a lot like graphics is running ok, the panel is just turned off. I'm thinking that makes the above warnings more relevant since they're in the space of FDI and PCH transcoders, which I think is how we get to a display output. > > It seems like SNB is pretty similar to IVB (they run on the same chipsets), > > Agree. > > > but I haven't yet fgured out the magic to make an SNB laptop light up with IGD > > assignment, which would be useful to show that OpRegion passthrough is > > actually doing something. > > > > There might be a difference in BDSM definition. You might want to add 0xb0 in host bridge passthrough and try SNB again. I'm not seeing any accesses there in the tracing, so I don't think the driver cares about it. > It is also possible the difference might be caused by difference in driver version. If I google "sandybridge windows graphics driver" and "ivybridge windows graphics driver" I get the following: > > SNB driver: https://downloadcenter.intel.com/product/81502/Intel-HD-Graphics-2000-for-2nd-Generation-Intel-Core-Processors > IVB driver: https://downloadcenter.intel.com/product/81501/Intel-HD-Graphics-2500-for-3rd-Generation-Intel-Core-Processors > > SNB driver points to version 15.28 driver binary. This driver supports SNB and ILK graphics. > IVB driver points to version 15.33 driver binary. This driver supports IVB and SNB graphics. > > One experiment you can try is to install same IVB 15.33 binary on both IVB and SNB systems. To prevent Windows update messing with driver version, you should do the following: > > 1) disable MSFT driver update: Computer->"Advance system settings"->Hardware->"Device Installation Settings"->"No, let me choose what to do"->"Never install driver software from Windows Update". > 2) Disable Windows update to never check or download updates. > 3) remove "c:\windows\system32\DriverStore\FileRepository\igd*" > 4) re-install IGD driver to desired binary. > 5) Double check driver versions of both IVB and SNB are the same after reboot guest. Unfortunately, it won't said 15.33 but have a version number starts with 10.*. There are certainly a lot of subtle differences between IVB and SNB in the Linux driver, if I have time I'll see about setting up a Windows VM on the SNB system. Linux feels pretty close though if I could just get the panel turned on. > > If we ignore IronLake and older GPUs, how many VM chipset hacks do we > > need? What combinations would require GMCH mirrored in the host bridge, > > Base on original code in Xen, you can try the following host bridge offset passthrough. I have also attached pt-graphics.c from Xen/QEMU for your reference. > > case 0x00: /* vendor id */ > case 0x02: /* device id */ > case 0x08: /* revision id */ > case 0x2c: /* sybsystem vendor id */ > case 0x2e: /* sybsystem id */ > case 0x50: /* SNB: processor graphics control register */ > case 0x52: /* processor graphics control register */ > case 0xa0: /* top of memory */ > case 0xb0: /* ILK: BSM: should read from dev 2 offset 0x5c */ > case 0x58: /* SNB: PAVPC Offset */ > case 0xa4: /* SNB: graphics base of stolen memory */ > case 0xa8: /* SNB: base of GTT stolen memory */ > > > and couldn't we mirror it from the IGD device itself since it's present in the > > same location all the way back through SNB. > > > > Yes, you can copy content from MCCG in IGD device to same 0x50 offset in host bridge. > > > We certainly don't want to get into the nastiness of RMRRs in a VM, but do > > the stolen memory use cases you've outlined explain the DMAR faults to that > > region that I was seeing just booting a VM with a Linux live CD? > > What I described about stolen memory was based on Windows driver usage. I will need to try Linux live CD to see DMAR faults you encountered. I will try that after I finish setting up my environment. I have been following your blog. Great instructions! Good, glad you found it. I'm just running a simple VM like this: /home/alwillia/local/bin/qemu-system-x86_64 -m 2G -bios /home/alwillia/Work/seabios.git/out/bios.bin -net none -cdrom /net/store/exports/ISOs/Fedora-Live-Cinnamon-x86_64-23-10.iso -boot d -monitor stdio -device vfio-pci,host=00:02.0,rombar=0,x-no-mmap=on -vnc :1 -vga std -enable-kvm -trace events=events.txt events.txt contains: vfio_region_read vfio_region_write vfio_pci_read_config vfio_pci_write_config vfio_pci_igd* pci_cfg_* If you base on the last patch series I posted on top of Gerd's changes, you'll need -M pc,igd_passthru=on I believe too, the code I'm working on passes the host and ISA bridge config through vfio and initiates the ISA bridge automatically. I'll try to post it this week. > > Again, it seems just as easy, if not easier to copy GMCH from the IGD itself into the > > host bridge. > > > Not sure if I follow ... how does this would solve stolen memory/RMRR issue...? In the vfio code I have now, the kernel virtualizes GMCH and BDSM on the device to provide a pre-sanitized version, so if we need those on the host bridge, it's easier to copy what's on the IGD instead of adding yet more code that knows about the different GMCH layouts. The solution is the same, clear BDSM and GMCH.GMS to make the guest not try to access the host stolen memory. > > Ok, unless anyone shouts really loudly and it doesn't affect anything newer > > than IronLake, I'm happy to let those fall off the plate. I don't have any older > > systems that I care to make work with this. > > > Sounds good. > > > > > Ok, so the code comment is pretty misleading for these. Would anything > > since SNB need these? > > > We did not do much pruning pre-SNB. Tiejun and I did checked HSW last year, most (maybe all) of these register are not needed. > > > What about the BDSM register in the host bridge? > > Is the Xen experience of not needing to copy it relevant since stolen memory is > > identity mapped into the VM anyway? Maybe Xen achieves the same effect > > by not copying it and exposing it as zero. > > In SNB, BDSM is at offset 0xb0 in host bridge. Xen/QEMU passes thru 0xb0 in host bridge. Although the comment says it is for ILK. It is also true for SNB. In current BDW/SKL driver, BDSM is located at 0x5c in IGD device. If I could spot any accesses to BDSM on the host bridge, I'd jump on this, but my guest driver doesn't seem to care about it according to the tracing. Lighting up an output seems to be the issue. Thanks, Alex