From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alex Williamson Subject: Re: Graphics pass-through Date: Mon, 09 May 2011 09:48:11 -0600 Message-ID: <1304956091.26106.39.camel@x201> References:

<4D415D70.8070105@web.de> <1296175554.2891.29.camel@x201> <4DC264CD.9080700@siemens.com> <1304608642.3081.35.camel@x201> <4DC7CCB3.8030607@siemens.com> <4DC807E3.4080706@siemens.com> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Cc: Prasad Joshi , =?ISO-8859-1?Q?Andr=E9?= Weidemann , "kvm@vger.kernel.org" , Oswaldo Cadenas , Maxim Nikolaev To: Jan Kiszka Return-path: Received: from mx1.redhat.com ([209.132.183.28]:62970 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751758Ab1EIPsQ (ORCPT ); Mon, 9 May 2011 11:48:16 -0400 In-Reply-To: <4DC807E3.4080706@siemens.com> Sender: kvm-owner@vger.kernel.org List-ID: On Mon, 2011-05-09 at 17:27 +0200, Jan Kiszka wrote: > On 2011-05-09 16:55, Prasad Joshi wrote: > > On Mon, May 9, 2011 at 12:14 PM, Jan Kiszka wrote: > >> On 2011-05-05 17:17, Alex Williamson wrote: > >>>> And what about the host? When does Linux release the legacy range? > >>>> Always or only when a specific (!=vga/vesa) framebuffer driver is loaded? > >>> > >>> Well, that's where it'd be nice if the vga arbiter was actually in more > >>> widespread use. It currently seems to be nothing more than a shared > >>> mutex, but it would actually be useful if it included backends to do the > >>> chipset vga routing changes. I think when I was testing this, I was > >>> externally poking PCI bridge chipset to toggle the VGA_EN bit. > >> > >> Right, we had to drop the approach to pass through the secondary card > >> for now, the arbiter was not switching properly. Haven't checked yet if > >> VGA_EN was properly set, though the kernel code looks like it should > >> take care of this. > >> > >> Even with handing out the primary adapter, we had only mixed success so > >> far. The onboard adapter worked well (in VESA mode), but the NVIDIA is > >> not displaying early boot messages at all. Maybe a vgabios issue. > >> Windows was booting nevertheless - until we installed the NVIDIA > >> drivers. Than it ran into a blue screen. > >> > >> BTW, what ATI adapter did you use precisely, and what did work, what not? > > > > Not hijacking the mail thread. Just wanted to provide some inputs. > > Much appreciated in fact! > > > > > Few days back I had tried passing through the secondary graphics card. > > I could pass-through two graphics cards to virtual machine. > > > > 02:00.0 VGA compatible controller: ATI Technologies Inc Redwood > > [Radeon HD 5670] (prog-if 00 [VGA controller]) > > Subsystem: PC Partner Limited Device e151 > > Flags: bus master, fast devsel, latency 0, IRQ 87 > > Memory at d0000000 (64-bit, prefetchable) [size=256M] > > Memory at fe6e0000 (64-bit, non-prefetchable) [size=128K] > > I/O ports at b000 [size=256] > > Expansion ROM at fe6c0000 [disabled] [size=128K] > > Capabilities: > > Kernel driver in use: radeon > > Kernel modules: radeon > > > > 07:00.0 VGA compatible controller: nVidia Corporation G86 [Quadro NVS > > 290] (rev a1) (prog-if 00 [VGA controller]) > > Subsystem: nVidia Corporation Device 0492 > > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- > > ParErr-Stepping- SERR+ FastB2B- DisINTx- > > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast > >> TAbort-SERR- > Latency: 0, Cache Line Size: 64 bytes > > Interrupt: pin A routed to IRQ 24 > > Region 0: Memory at fd000000 (32-bit, non-prefetchable) [size=16M] > > Region 1: Memory at d0000000 (64-bit, prefetchable) [size=256M] > > Region 3: Memory at fa000000 (64-bit, non-prefetchable) [size=32M] > > Region 5: I/O ports at ec00 [size=128] > > Expansion ROM at fe9e0000 [disabled] [size=128K] > > Capabilities: > > Kernel driver in use: nouveau > > Kernel modules: nouveau, nvidiafb > > > > Both of them are PCIe cards. I have one more ATI card and another > > NVIDIA card which does not work. > > Interesting. That may rule out missing PCIe capabilities as source for > the NVIDIA driver indisposition. > > Did you passed those cards each as primary to the guest, or was the > guest seeing multiple adapters? I presume you only got output after > early boot was completed, right? > > To avoid having to deal with legacy I/O forwarding, we started with a > dual adapter setup in the hope to leave the primary guest adapter at > know-to-work cirrus-vga. But already in a native setup with on-board > primary + NVIDIA secondary, the NVIDIA Windows drivers refused to talk > to its hardware in this constellation. > > > > > One of the reason the pass-through did not work is because of the > > limit on amount of pci configuration memory by SeaBIOS. SeaBIOS places > > a hard limit of 256MB or so on the amount of PCI memory space. Thus, > > for some of the VGA device that need more memory never worked for me. > > > > SeaBIOS allows this memory region to be extended to some value near > > 512MB, but even then the range is not enough. > > > > Another problem with SeaBIOS which limits the amount of memory space > > is: SeaBIOS allocates the BAR regions as they are encountered. As far > > as I know, the BAR regions should be naturally aligned. Thus the > > simple strategy of the SeaBIOS results in large fragmentation. > > Therefore, even after increasing the PCI memory space to 512MB the BAR > > regions were unallocated. > > That's an interesting trace! We'll check this here, but I bet it > contributes to the problems. Our FX 3800 has 1G memory... Yes, qemu leaves far too little MMIO space to think about assigning graphics cards. Both of my cards have 512MB and I hacked qemu to leave a bigger gap via something like: diff --git a/hw/pc.c b/hw/pc.c index 0ea6d10..a6376f8 100644 --- a/hw/pc.c +++ b/hw/pc.c @@ -879,6 +879,8 @@ void pc_cpus_init(const char *cpu_model) } } +#define PC_MAX_LOW_RAM 0xc0000000 + void pc_memory_init(ram_addr_t ram_size, const char *kernel_filename, const char *kernel_cmdline, @@ -893,9 +895,9 @@ void pc_memory_init(ram_addr_t ram_size, int bios_size, isa_bios_size; void *fw_cfg; - if (ram_size >= 0xe0000000 ) { - above_4g_mem_size = ram_size - 0xe0000000; - below_4g_mem_size = 0xe0000000; + if (ram_size >= PC_MAX_LOW_RAM ) { + above_4g_mem_size = ram_size - PC_MAX_LOW_RAM; + below_4g_mem_size = PC_MAX_LOW_RAM; } else { below_4g_mem_size = ram_size; } There's also a #define that needs to be changed in seabios config.h and and acpi dsdt update, but I can't seem to find patches for those. Also pay attention to the cpu_register_physical_memory calls in i440fx_update_memory_mappings(), those can steal the legacy VGA MMIO range from you. I just commented them out: diff --git a/hw/piix_pci.c b/hw/piix_pci.c index b5589b9..1327563 100644 --- a/hw/piix_pci.c +++ b/hw/piix_pci.c @@ -106,11 +106,11 @@ static void i440fx_update_memory_mappings(PCII440FXState *d) } smram = d->dev.config[I440FX_SMRAM]; if ((d->smm_enabled && (smram & 0x08)) || (smram & 0x40)) { - cpu_register_physical_memory(0xa0000, 0x20000, 0xa0000); + //cpu_register_physical_memory(0xa0000, 0x20000, 0xa0000); } else { for(addr = 0xa0000; addr < 0xc0000; addr += 4096) { - cpu_register_physical_memory(addr, 4096, - d->isa_page_descs[(addr - 0xa0000) >> 12]); + //cpu_register_physical_memory(addr, 4096, + // d->isa_page_descs[(addr - 0xa0000) >> 12]); } } } That's all the tricks I remember. Thanks, Alex