From mboxrd@z Thu Jan 1 00:00:00 1970 From: Gordon Messmer Subject: Device pass-through Date: Mon, 02 Jan 2012 11:41:30 -0800 Message-ID: <4F02086A.6050607@eburg.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit To: kvm@vger.kernel.org Return-path: Received: from home.dragonsdawn.net ([173.160.248.33]:34769 "EHLO ascension.private.dragonsdawn.net" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751251Ab2ABTko (ORCPT ); Mon, 2 Jan 2012 14:40:44 -0500 Sender: kvm-owner@vger.kernel.org List-ID: I'm going to summarize my questions first and write the long explanation afterward... 1: What happens when Linux rejects the aperture allocated by the BIOS for an AMD IOMMU? 2: What is unsafe about the allow_unsafe_assigned_interrupts option? What are the potential consequences of its use? 3: How can I get the ROM from a graphics card? 4: What do these errors mean, and is there any way I can correct the problem? > create_userspace_phys_mem: File exists > assigned_dev_iomem_map: Error: create new mapping failed My goal is to replace two machines that I previously used with one machine and a KVM guest. I previously had one system that was always on and ran a stable OS, and a second system for my desktop. I could afford to experiment or format the desktop system without interrupting access to my data, all of which is on the stable system. I'd like to use KVM to accomplish something more or less equivalent by creating a guest and passing the keyboard, mouse, audio device, and a video card to it. To this end, I have a new AMD system with a Gigabyte GA-970A-D3 motherboard and two Radeon video cards. Based on my reading, I should probably be able to pass one of those to the guest as a PCI device. It isn't in use by the host, and I'm not trying to make it the guest's primary display, so I don't have to worry about early boot output. The guest will still have a VNC console as its primary display. I'm mostly there now. I can pass through the host's keyboard, mouse, and audio device. Naturally, there are a couple of challenges. First, the IOMMU on this board is a little wonky. The kernel currently reports: > kernel: Checking aperture... > kernel: No AGP bridge found > kernel: Node 0: aperture @ 20000000 size 32 MB > kernel: Aperture pointing to e820 RAM. Ignoring. > kernel: Your BIOS doesn't leave a aperture memory hole > kernel: Please enable the IOMMU option in the BIOS setup > kernel: This costs you 64 MB of RAM > kernel: Mapping aperture over 65536 KB of RAM @ 20000000 > kernel: PM: Registered nosave memory: 0000000020000000 - > 0000000024000000 > kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB) > kernel: Placing 64MB software IO TLB between ffff880037ff0000 - > ffff88003bff0000 > kernel: software IO TLB at phys 0x37ff0000 - 0x3bff0000 additionally: > kernel: BIOS-provided physical RAM map: > kernel: BIOS-e820: 0000000000000000 - 0000000000098800 (usable) > kernel: BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved) > kernel: BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved) > kernel: BIOS-e820: 0000000000100000 - 00000000bfda0000 (usable) > kernel: BIOS-e820: 00000000bfda0000 - 00000000bfdd1000 (ACPI NVS) > kernel: BIOS-e820: 00000000bfdd1000 - 00000000bfe00000 (ACPI data) > kernel: BIOS-e820: 00000000bfe00000 - 00000000bff00000 (reserved) > kernel: BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved) > kernel: BIOS-e820: 00000000fec00000 - 0000000100000000 (reserved) > kernel: BIOS-e820: 0000000100000000 - 0000000440000000 (usable) > ... > kernel: ACPI: IVRS 00000000bfddff00 000D0 > (v01 AMD RD890S 00202031 AMD 00000000) Initially the kernel reported "AMD-Vi disabled by default: pass amd_iommu=on to enable" After booting with that option, I seem to be able to boot guests with devices passed through. That leads me to the first question above. How bad is the state of the IOMMU on this board? Where should the IVRS be mapped instead? In addition to the amd_iommu boot option, I had to add: options kvm allow_unsafe_assigned_interrupts=1 in order to pass through the audio device. That leads me to the second question. What's unsafe about this? When I try to boot the guest and pass in the video card, I get a warning about the ROM: > pci-assign: Cannot read from host /sys/bus/pci/devices/0000:05:00.0/rom > Device option ROM contents are probably invalid (check dmesg). > Skip option ROM probe with rombar=0, or load from file with > romfile= > create_userspace_phys_mem: File exists > assigned_dev_iomem_map: Error: create new mapping failed Third question, when I try to read that path, I get an IO error and the kernel says "pci 0000:05:00.0: Invalid ROM contents". Is there any other way to get the ROM from the card? I've tried the other option. I added to the libvirt definition for this guest. When I try to boot the guest with that option, the kernel doesn't add anything to dmesg that looks relevant, but the libvirt log says: > LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -S -M rhel6.2.0 -enable-kvm -m 4096 -smp 4,sockets=4,cores=1,threads=1 -name herald -uuid 1929f6a4-b2dd-a4fd-fe07-747b7fdf9fae -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/herald.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -drive file=/dev/mapper/VolGroup-lv_vm_herald,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native -device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=26 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:86:e0:ea,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev= charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vnc 127.0.0.1:0 -vga cirrus -device pci-assign,host=00:14.2,id=hostdev0,configfd=27,bus=pci.0,addr=0x5 -device pci-assign,host=05:00 .0,id=hostdev1,configfd=28,bus=pci.0,addr=0x7,rombar=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 > char device redirected to /dev/pts/1 > create_userspace_phys_mem: File exists > assigned_dev_iomem_map: Error: create new mapping failed So, my last question is what do the last two lines mean?