From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jan Kiszka Subject: Re: Graphics pass-through Date: Mon, 09 May 2011 18:00:42 +0200 Message-ID: <4DC80FAA.2030500@siemens.com> References:

<4D415D70.8070105@web.de> <1296175554.2891.29.camel@x201> <4DC264CD.9080700@siemens.com> <1304608642.3081.35.camel@x201> <4DC7CCB3.8030607@siemens.com> <4DC807E3.4080706@siemens.com> <1304956091.26106.39.camel@x201> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: Prasad Joshi , =?UTF-8?B?QW5kcsOpIFdlaWRlbWE=?= =?UTF-8?B?bm4=?= , "kvm@vger.kernel.org" , Oswaldo Cadenas , "Nikolaev, Maxim" To: Alex Williamson Return-path: Received: from thoth.sbs.de ([192.35.17.2]:24328 "EHLO thoth.sbs.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753048Ab1EIQAu (ORCPT ); Mon, 9 May 2011 12:00:50 -0400 In-Reply-To: <1304956091.26106.39.camel@x201> Sender: kvm-owner@vger.kernel.org List-ID: On 2011-05-09 17:48, Alex Williamson wrote: > On Mon, 2011-05-09 at 17:27 +0200, Jan Kiszka wrote: >> On 2011-05-09 16:55, Prasad Joshi wrote: >>> On Mon, May 9, 2011 at 12:14 PM, Jan Kiszka wrote: >>>> On 2011-05-05 17:17, Alex Williamson wrote: >>>>>> And what about the host? When does Linux release the legacy range? >>>>>> Always or only when a specific (!=vga/vesa) framebuffer driver is loaded? >>>>> >>>>> Well, that's where it'd be nice if the vga arbiter was actually in more >>>>> widespread use. It currently seems to be nothing more than a shared >>>>> mutex, but it would actually be useful if it included backends to do the >>>>> chipset vga routing changes. I think when I was testing this, I was >>>>> externally poking PCI bridge chipset to toggle the VGA_EN bit. >>>> >>>> Right, we had to drop the approach to pass through the secondary card >>>> for now, the arbiter was not switching properly. Haven't checked yet if >>>> VGA_EN was properly set, though the kernel code looks like it should >>>> take care of this. >>>> >>>> Even with handing out the primary adapter, we had only mixed success so >>>> far. The onboard adapter worked well (in VESA mode), but the NVIDIA is >>>> not displaying early boot messages at all. Maybe a vgabios issue. >>>> Windows was booting nevertheless - until we installed the NVIDIA >>>> drivers. Than it ran into a blue screen. >>>> >>>> BTW, what ATI adapter did you use precisely, and what did work, what not? >>> >>> Not hijacking the mail thread. Just wanted to provide some inputs. >> >> Much appreciated in fact! >> >>> >>> Few days back I had tried passing through the secondary graphics card. >>> I could pass-through two graphics cards to virtual machine. >>> >>> 02:00.0 VGA compatible controller: ATI Technologies Inc Redwood >>> [Radeon HD 5670] (prog-if 00 [VGA controller]) >>> Subsystem: PC Partner Limited Device e151 >>> Flags: bus master, fast devsel, latency 0, IRQ 87 >>> Memory at d0000000 (64-bit, prefetchable) [size=256M] >>> Memory at fe6e0000 (64-bit, non-prefetchable) [size=128K] >>> I/O ports at b000 [size=256] >>> Expansion ROM at fe6c0000 [disabled] [size=128K] >>> Capabilities: >>> Kernel driver in use: radeon >>> Kernel modules: radeon >>> >>> 07:00.0 VGA compatible controller: nVidia Corporation G86 [Quadro NVS >>> 290] (rev a1) (prog-if 00 [VGA controller]) >>> Subsystem: nVidia Corporation Device 0492 >>> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- >>> ParErr-Stepping- SERR+ FastB2B- DisINTx- >>> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >>>> TAbort-SERR- >> Latency: 0, Cache Line Size: 64 bytes >>> Interrupt: pin A routed to IRQ 24 >>> Region 0: Memory at fd000000 (32-bit, non-prefetchable) [size=16M] >>> Region 1: Memory at d0000000 (64-bit, prefetchable) [size=256M] >>> Region 3: Memory at fa000000 (64-bit, non-prefetchable) [size=32M] >>> Region 5: I/O ports at ec00 [size=128] >>> Expansion ROM at fe9e0000 [disabled] [size=128K] >>> Capabilities: >>> Kernel driver in use: nouveau >>> Kernel modules: nouveau, nvidiafb >>> >>> Both of them are PCIe cards. I have one more ATI card and another >>> NVIDIA card which does not work. >> >> Interesting. That may rule out missing PCIe capabilities as source for >> the NVIDIA driver indisposition. >> >> Did you passed those cards each as primary to the guest, or was the >> guest seeing multiple adapters? I presume you only got output after >> early boot was completed, right? >> >> To avoid having to deal with legacy I/O forwarding, we started with a >> dual adapter setup in the hope to leave the primary guest adapter at >> know-to-work cirrus-vga. But already in a native setup with on-board >> primary + NVIDIA secondary, the NVIDIA Windows drivers refused to talk >> to its hardware in this constellation. >> >>> >>> One of the reason the pass-through did not work is because of the >>> limit on amount of pci configuration memory by SeaBIOS. SeaBIOS places >>> a hard limit of 256MB or so on the amount of PCI memory space. Thus, >>> for some of the VGA device that need more memory never worked for me. >>> >>> SeaBIOS allows this memory region to be extended to some value near >>> 512MB, but even then the range is not enough. >>> >>> Another problem with SeaBIOS which limits the amount of memory space >>> is: SeaBIOS allocates the BAR regions as they are encountered. As far >>> as I know, the BAR regions should be naturally aligned. Thus the >>> simple strategy of the SeaBIOS results in large fragmentation. >>> Therefore, even after increasing the PCI memory space to 512MB the BAR >>> regions were unallocated. >> >> That's an interesting trace! We'll check this here, but I bet it >> contributes to the problems. Our FX 3800 has 1G memory... > > Yes, qemu leaves far too little MMIO space to think about assigning > graphics cards. Both of my cards have 512MB and I hacked qemu to leave > a bigger gap via something like: > > diff --git a/hw/pc.c b/hw/pc.c > index 0ea6d10..a6376f8 100644 > --- a/hw/pc.c > +++ b/hw/pc.c > @@ -879,6 +879,8 @@ void pc_cpus_init(const char *cpu_model) > } > } > > +#define PC_MAX_LOW_RAM 0xc0000000 > + > void pc_memory_init(ram_addr_t ram_size, > const char *kernel_filename, > const char *kernel_cmdline, > @@ -893,9 +895,9 @@ void pc_memory_init(ram_addr_t ram_size, > int bios_size, isa_bios_size; > void *fw_cfg; > > - if (ram_size >= 0xe0000000 ) { > - above_4g_mem_size = ram_size - 0xe0000000; > - below_4g_mem_size = 0xe0000000; > + if (ram_size >= PC_MAX_LOW_RAM ) { > + above_4g_mem_size = ram_size - PC_MAX_LOW_RAM; > + below_4g_mem_size = PC_MAX_LOW_RAM; > } else { > below_4g_mem_size = ram_size; > } > > There's also a #define that needs to be changed in seabios config.h and > and acpi dsdt update, but I can't seem to find patches for those. Hmm, as this does not scale with the constantly growing memory sizes of GPUs, I guess this should would for us as well, even with 1G. The adapters likely only map a window to their on-board RAM. > Also > pay attention to the cpu_register_physical_memory calls in > i440fx_update_memory_mappings(), those can steal the legacy VGA MMIO > range from you. I just commented them out: > > diff --git a/hw/piix_pci.c b/hw/piix_pci.c > index b5589b9..1327563 100644 > --- a/hw/piix_pci.c > +++ b/hw/piix_pci.c > @@ -106,11 +106,11 @@ static void i440fx_update_memory_mappings(PCII440FXState *d) > } > smram = d->dev.config[I440FX_SMRAM]; > if ((d->smm_enabled && (smram & 0x08)) || (smram & 0x40)) { > - cpu_register_physical_memory(0xa0000, 0x20000, 0xa0000); > + //cpu_register_physical_memory(0xa0000, 0x20000, 0xa0000); > } else { > for(addr = 0xa0000; addr < 0xc0000; addr += 4096) { > - cpu_register_physical_memory(addr, 4096, > - d->isa_page_descs[(addr - 0xa0000) >> 12]); > + //cpu_register_physical_memory(addr, 4096, > + // d->isa_page_descs[(addr - 0xa0000) >> 12]); > } > } > } > > That's all the tricks I remember. Thanks, Yeah, we are already carrying half of the above in our tree (only the second disabling is actually needed, KVM does not support SMM). I started looking into fixing PAM/SMRAM mess, but it's not yet beautiful - partly because we urgently need slot management at core level. Jan -- Siemens AG, Corporate Technology, CT T DE IT 1 Corporate Competence Center Embedded Linux