From: Maik Broemme <mbroemme@parallels.com>
To: Alex Williamson <alex.williamson@redhat.com>
Cc: Knut Omang <knuto@ifi.uio.no>, kvm <kvm@vger.kernel.org>,
qemu-devel <qemu-devel@nongnu.org>
Subject: Re: VFIO VGA test branches
Date: Wed, 29 May 2013 18:16:56 +0200 [thread overview]
Message-ID: <20130529161656.GM28436@parallels.com> (raw)
In-Reply-To: <20130529152720.GL28436@parallels.com>
[-- Attachment #1: Type: text/plain, Size: 19584 bytes --]
Hi,
Maik Broemme <mbroemme@parallels.com> wrote:
> > >
> > > Here is the DEBUG_VFIO output:
> > >
> > > vfio: vfio_initfn(0000:04:00.0) group 14
> > > vfio: region_add 0 - afffffff [0x7f8698000000]
> > > vfio: SKIPPING region_add fec00000 - fec00fff
> > > vfio: SKIPPING region_add fed00000 - fed003ff
> > > vfio: SKIPPING region_add fee00000 - feefffff
> > > vfio: region_add fffe0000 - ffffffff [0x7f88aa400000]
> > > vfio: region_add 100000000 - 24fffffff [0x7f8748000000]
> > > vfio: Device 0000:04:00.0 flags: 3, regions: 9, irgs: 4
> > > vfio: Device 0000:04:00.0 region 0:
> > > vfio: size: 0x10000000, offset: 0x0, flags: 0x7
> > > vfio: Device 0000:04:00.0 region 1:
> > > vfio: size: 0x0, offset: 0x10000000000, flags: 0x0
> > > vfio: Device 0000:04:00.0 region 2:
> > > vfio: size: 0x40000, offset: 0x20000000000, flags: 0x7
> > > vfio: Device 0000:04:00.0 region 3:
> > > vfio: size: 0x0, offset: 0x30000000000, flags: 0x0
> > > vfio: Device 0000:04:00.0 region 4:
> > > vfio: size: 0x100, offset: 0x40000000000, flags: 0x3
> > > vfio: Device 0000:04:00.0 region 5:
> > > vfio: size: 0x0, offset: 0x50000000000, flags: 0x0
> > > vfio: Device 0000:04:00.0 ROM:
> > > vfio: size: 0x20000, offset: 0x60000000000, flags: 0x1
> > > vfio: Device 0000:04:00.0 config:
> > > vfio: size: 0x1000, offset: 0x70000000000, flags: 0x3
> > > vfio: vfio_load_rom(0000:04:00.0)
> > > vfio: Enabled ATI/AMD BAR2 0x4000 quirk for device 0000:04:00.0
> > > vfio: Enabled ATI/AMD BAR4 window quirk for device 0000:04:00.0
> > > vfio: Enabled ATI/AMD quirk 0x3c3 BAR4 for device 0000:04:00.0
> > > vfio: 0000:04:00.0 PCI MSI CAP @0xa0
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> > > vfio: vfio_enable_intx_kvm(0000:04:00.0) KVM INTx accel enabled
> > > vfio: vfio_enable_intx(0000:04:00.0)
> > > vfio: vfio_initfn(0000:04:00.1) group 14
> > > vfio: Device 0000:04:00.1 flags: 3, regions: 9, irgs: 4
> > > vfio: Device 0000:04:00.1 region 0:
> > > vfio: size: 0x4000, offset: 0x0, flags: 0x7
> > > vfio: Device 0000:04:00.1 region 1:
> > > vfio: size: 0x0, offset: 0x10000000000, flags: 0x0
> > > vfio: Device 0000:04:00.1 region 2:
> > > vfio: size: 0x0, offset: 0x20000000000, flags: 0x0
> > > vfio: Device 0000:04:00.1 region 3:
> > > vfio: size: 0x0, offset: 0x30000000000, flags: 0x0
> > > vfio: Device 0000:04:00.1 region 4:
> > > vfio: size: 0x0, offset: 0x40000000000, flags: 0x0
> > > vfio: Device 0000:04:00.1 region 5:
> > > vfio: size: 0x0, offset: 0x50000000000, flags: 0x0
> > > vfio: Device 0000:04:00.1 ROM:
> > > vfio: size: 0x0, offset: 0x60000000000, flags: 0x0
> > > vfio: Device 0000:04:00.1 config:
> > > vfio: size: 0x1000, offset: 0x70000000000, flags: 0x3
> > > vfio: 0000:04:00.1 PCI MSI CAP @0xa0
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> > > vfio: vfio_enable_intx_kvm(0000:04:00.1) KVM INTx accel enabled
> > > vfio: vfio_enable_intx(0000:04:00.1)
> > > vfio: region_del 0 - afffffff
> > > vfio: region_add 0 - bffff [0x7f8698000000]
> > > vfio: region_add c0000 - dffff [0x7f88aa200000]
> > > vfio: region_add e0000 - fffff [0x7f88aa400000]
> > > vfio: region_add 100000 - afffffff [0x7f8698100000]
> > > vfio: vfio_pci_reset(0000:04:00.0)
> > > vfio: vfio_disable_intx_kvm(0000:04:00.0) KVM INTx accel disabled
> > > vfio: vfio_disable_intx(0000:04:00.0)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x54, len=0x2) 0
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 3
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x4, 0x0, len=0x2)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> > > vfio: vfio_enable_intx_kvm(0000:04:00.0) KVM INTx accel enabled
> > > vfio: vfio_enable_intx(0000:04:00.0)
> > > vfio: vfio_pci_reset(0000:04:00.1)
> > > vfio: vfio_disable_intx_kvm(0000:04:00.1) KVM INTx accel disabled
> > > vfio: vfio_disable_intx(0000:04:00.1)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x54, len=0x2) 0
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x4, len=0x2) 6
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x4, 0x0, len=0x2)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> > > vfio: vfio_enable_intx_kvm(0000:04:00.1) KVM INTx accel enabled
> > > vfio: vfio_enable_intx(0000:04:00.1)
> > > vfio: region_del 0 - bffff
> > > vfio: region_del c0000 - dffff
> > > vfio: region_add 0 - c7fff [0x7f8698000000]
> > > vfio: region_add c8000 - dffff [0x7f88aa208000]
> > > vfio: region_del 0 - c7fff
> > > vfio: region_del c8000 - dffff
> > > vfio: region_add 0 - cffff [0x7f8698000000]
> > > vfio: region_add d0000 - dffff [0x7f88aa210000]
> > > vfio: region_del 0 - cffff
> > > vfio: region_del d0000 - dffff
> > > vfio: region_add 0 - d7fff [0x7f8698000000]
> > > vfio: region_add d8000 - dffff [0x7f88aa218000]
> > > vfio: region_del 0 - d7fff
> > > vfio: region_del d8000 - dffff
> > > vfio: region_add 0 - dffff [0x7f8698000000]
> > > vfio: region_del 0 - dffff
> > > vfio: region_del e0000 - fffff
> > > vfio: region_add 0 - e7fff [0x7f8698000000]
> > > vfio: region_add e8000 - fffff [0x7f88aa408000]
> > > vfio: region_del 0 - e7fff
> > > vfio: region_del e8000 - fffff
> > > vfio: region_add 0 - effff [0x7f8698000000]
> > > vfio: region_add f0000 - fffff [0x7f88aa410000]
> > > vfio: region_del 0 - effff
> > > vfio: region_del f0000 - fffff
> > > vfio: region_del 100000 - afffffff
> > > vfio: region_add 0 - afffffff [0x7f8698000000]
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x2) 1002
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0xa, len=0x2) 300
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x2) 1002
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0xa, len=0x2) 403
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x2) 1002
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0xa, len=0x2) 300
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x2) 1002
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0xa, len=0x2) 403
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x2) 1002
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x4) 68181002
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x8, len=0x4) 3000000
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x2) 1002
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x4) aab01002
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x8, len=0x4) 4030000
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0xe, len=0x1) 80
> > > vfio: SKIPPING region_add b0000000 - bfffffff
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x10, len=0x4) c000000c
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x10, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x10, len=0x4) f000000c
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x10, 0xc000000c, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x14, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x14, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x14, len=0x4) ffffffff
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x14, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x18, len=0x4) fde80004
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x18, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x18, len=0x4) fffc0004
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x18, 0xfde80004, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x1c, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x1c, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x1c, len=0x4) ffffffff
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x1c, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x20, len=0x4) ce01
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x20, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x20, len=0x4) ffffff01
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x20, 0xce01, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x24, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x24, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x24, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x24, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfffff800, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) fffe0000
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x10, len=0x4) fdefc004
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x10, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x10, len=0x4) ffffc004
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x10, 0xfdefc004, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x14, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x14, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x14, len=0x4) ffffffff
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x14, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x18, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x18, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x18, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x18, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x1c, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x1c, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x1c, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x1c, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x20, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x20, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x20, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x20, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x24, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x24, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x24, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x24, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x30, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x30, 0xfffff800, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x30, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x30, 0x0, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x20, 0xc000, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x18, 0xfea00000, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x1c, 0x0, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfea40000, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x10, 0xfea60000, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x14, 0x0, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x10, 0xe0000000, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x14, 0x0, len=0x4)
> > > vfio: SKIPPING region_add feb40000 - feb4002f
> > > vfio: SKIPPING region_add feb40800 - feb40807
> > > vfio: SKIPPING region_add feb41000 - feb4101f
> > > vfio: SKIPPING region_add feb41800 - feb41807
> > > vfio: vfio_update_irq(0000:04:00.1) IRQ moved 20 -> 10
> > > vfio: vfio_disable_intx_kvm(0000:04:00.1) KVM INTx accel disabled
> > > vfio: vfio_enable_intx_kvm(0000:04:00.1) KVM INTx accel enabled
> > > vfio: vfio_update_irq(0000:04:00.0) IRQ moved 23 -> 11
> > > vfio: vfio_disable_intx_kvm(0000:04:00.0) KVM INTx accel disabled
> > > vfio: vfio_enable_intx_kvm(0000:04:00.0) KVM INTx accel enabled
> > > vfio: SKIPPING region_add feb42000 - feb42fff
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x3c, 0xb, len=0x1)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 0
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x4, 0x103, len=0x2)
> > > vfio: region_add e0000000 - efffffff [0x7f8688000000]
> > > vfio: region_add fea00000 - fea03fff [0x7f88aa7b8000]
> > > vfio: SKIPPING region_add fea04000 - fea04fff
> > > vfio: region_add fea05000 - fea3ffff [0x7f88aa7bd000]
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x3c, 0xa, len=0x1)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x4, len=0x2) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x4, 0x103, len=0x2)
> > > vfio: region_add fea60000 - fea63fff [0x7f88bc710000]
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 103
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 103
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x4, 0x103, len=0x2)
> > > vfio: region_del 0 - afffffff
> > > vfio: region_add 0 - 9ffff [0x7f8698000000]
> > > vfio: SKIPPING region_add a0000 - bffff
> > > vfio: region_add c0000 - afffffff [0x7f86980c0000]
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 103
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) fea40000
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfffffffe, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) fffe0000
> >
> > Here the option ROM was sized
> >
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfea40001, len=0x4)
> >
> > Then enabled
> >
> > > vfio: region_add fea40000 - fea5ffff [0x7f88a9e00000]
> >
> > Adding this memory region
> >
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfea40000, len=0x4)
> > > vfio: region_del fea40000 - fea5ffff
> >
> > Then disabled, removing the memory region. Presumably between the
> > enable and disable the contents were read and copied to 0xc0000, which
> > is where the VGA BIOS is shadowed.
> >
> > > Here is the strace output from this failure:
> > >
> > > 1110 ioctl(14, KVM_RUN, 0) = 0
> > > 1110 pread(20, <unfinished ...>
> > > 1099 <... poll resumed> ) = 1 ([{fd=0, revents=POLLIN}])
> > > 1099 futex(0x7ff73ca62fa0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
> > > 1109 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out)
> > > 1109 madvise(0x7ff72fe17000, 8368128, MADV_DONTNEED) = 0
> > > 1109 _exit(0) = ?
> > > 1109 +++ exited with 0 +++
> > >
> > > From reading the source 'hw/misc/vfio.c' it looks like the following
> > > in 'vfio_vga_read' never finished:
> > >
> > > if (pread(vga->fd, &buf, size, offset) != size) {
> > > error_report("%s(,0x%"HWADDR_PRIx", %d) failed: %m",
> > > __func__, region->offset + addr, size);
> > > return (uint64_t)-1;
> > > }
> >
> > I agree. Every VGA access requires us to lock the VGA resources on the
> > device, so if we can't get the lock, we stop making progress. I took a
> > look at Xorg last night and it seems like it should be taking and
> > releasing the VGA arbiter lock in a way that would be compatible with
> > our use. That's in the xserver, not the actual display hardware driver,
> > and it wraps access functions in the arbiter support, so should be
> > transparent to the drivers. So for nouveau, it seems like it should
> > work. For nvidia, we don't really know, it could be locking the device
> > from the kernel module.
> >
> > You could instrument vga_get, vga_tryget, and vga_put to figure out
> > what's happening. It might be enough to look at /dev/vga_arbiter at
> > each step in the sequence to reproduce (sudo head
> > --lines=1 /dev/vga_arbiter). Thanks,
> >
>
> I've played a bit more with it and there are some differences in
> behavior with 'nouveau' and 'nvidia' driver. As soon as I load the
> binary driver I see the following:
>
> [ 18.628676] [drm] Initialized drm 1.1.0 20060810
> [ 18.668038] nvidia: module license 'NVIDIA' taints kernel.
> [ 18.668107] Disabling lock debugging due to kernel taint
> [ 18.676638] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=io+mem
> [ 18.676722] vgaarb: transferring owner from PCI:0000:01:00.0 to PCI:0000:04:00.0
> [ 18.677007] [drm] Initialized nvidia-drm 0.0.0 20130102 for 0000:01:00.0 on minor 0
> [ 18.677090] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 319.23 Thu May 16 19:36:02 PDT 2013
>
> root@homer:~# head --lines=1 /dev/vga_arbiter
> count:2,PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none(0:0)
> root@homer:~# modprobe nvidia
> root@homer:~# head --lines=1 /dev/vga_arbiter
> count:1,PCI:0000:01:00.0,decodes=none,owns=none,locks=io+mem(1:1)
> root@homer:~# /usr/local/bin/qemu-system-x86_64 \
> -L /usr/local/share/qemu \
> -L /usr/local/share/qemu \
> -M q35 -enable-kvm -cpu host -smp cores=4,threads=1,sockets=1 \
> -m 8192 -rtc base=localtime -k de -nodefaults -vga none \
> -drive file=/home/mbroemme/.kvm/maggie.img,id=drive0,if=none,cache=none,aio=threads \
> -device virtio-blk-pci,drive=drive0,ioeventfd=on \
> -device ioh3420,id=pcie0,multifunction=on \
> -device vfio-pci,host=04:00.0,addr=0.0,bus=pcie0,multifunction=on,x-vga=on \
> -device vfio-pci,host=04:00.1,addr=0.1,bus=pcie0 -monitor stdio -nographic
> root@homer:~# head --lines=1 /dev/vga_arbiter
> count:1,PCI:0000:01:00.0,decodes=none,owns=none,locks=io+mem(1:1)
>
> It looks like nvidia binary driver locks some resources. :( Whereas
> with nouveau driver the VGA arbitration starts first if I start the
> VM with VGA passthrough:
>
> [ 178.187706] vfio-pci 0000:04:00.0: enabling device (0000 -> 0003)
> [ 178.209599] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x19@0x270
> [ 178.209631] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x1b@0x2d0
> [ 181.198191] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=io+mem
> [ 181.198208] vgaarb: transferring owner from PCI:0000:01:00.0 to PCI:0000:04:00.0
>
> root@homer:~# head --lines=1 /dev/vga_arbiter
> count:2,PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none(0:0)
> root@homer:~# /usr/local/bin/qemu-system-x86_64 \
> -L /usr/local/share/qemu \
> -L /usr/local/share/qemu \
> -M q35 -enable-kvm -cpu host -smp cores=4,threads=1,sockets=1 \
> -m 8192 -rtc base=localtime -k de -nodefaults -vga none \
> -drive file=/home/mbroemme/.kvm/maggie.img,id=drive0,if=none,cache=none,aio=threads \
> -device virtio-blk-pci,drive=drive0,ioeventfd=on \
> -device ioh3420,id=pcie0,multifunction=on \
> -device vfio-pci,host=04:00.0,addr=0.0,bus=pcie0,multifunction=on,x-vga=on \
> -device vfio-pci,host=04:00.1,addr=0.1,bus=pcie0 -monitor stdio -nographic
> root@homer:~# head --lines=1 /dev/vga_arbiter
> count:1,PCI:0000:01:00.0,decodes=none,owns=none,locks=none(0:0)
>
Alex you pointed me in the right direction, many thanks! I got it
working with 'nvidia' binary driver now, but I had to patch it. I've
attached it here just for reference if others want to try the same
and I don't know if it is the proper way. After some testing (Xorg,
DRI, VDPAU, switching between X and text-console) it looks like a stable
workaround for me now until NVIDIA will fix their driver.
--Maik
[-- Attachment #2: NVIDIA-Linux-x86_64-319.23-vfio-vgaarb-fix.patch --]
[-- Type: text/plain, Size: 1355 bytes --]
diff -Nur NVIDIA-Linux-x86_64-319.23/kernel/nv-linux.h NVIDIA-Linux-x86_64-319.23-vfio-vgaarb-fix/kernel/nv-linux.h
--- NVIDIA-Linux-x86_64-319.23/kernel/nv-linux.h 2013-05-17 04:00:02.000000000 +0200
+++ NVIDIA-Linux-x86_64-319.23-vfio-vgaarb-fix/kernel/nv-linux.h 2013-05-29 18:09:42.382925622 +0200
@@ -151,9 +151,9 @@
#error "struct file_operations compile test likely failed!"
#endif
-#if defined(CONFIG_VGA_ARB)
-#include <linux/vgaarb.h>
-#endif
+//#if defined(CONFIG_VGA_ARB)
+//#include <linux/vgaarb.h>
+//#endif
#if defined(NV_VM_INSERT_PAGE_PRESENT)
#include <linux/pagemap.h>
diff -Nur NVIDIA-Linux-x86_64-319.23/kernel/nv.c NVIDIA-Linux-x86_64-319.23-vfio-vgaarb-fix/kernel/nv.c
--- NVIDIA-Linux-x86_64-319.23/kernel/nv.c 2013-05-17 04:00:02.000000000 +0200
+++ NVIDIA-Linux-x86_64-319.23-vfio-vgaarb-fix/kernel/nv.c 2013-05-29 18:10:01.494277314 +0200
@@ -2914,12 +2914,12 @@
pci_set_master(dev);
-#if defined(CONFIG_VGA_ARB)
-#if defined(VGA_DEFAULT_DEVICE)
- vga_tryget(VGA_DEFAULT_DEVICE, VGA_RSRC_LEGACY_MASK);
-#endif
- vga_set_legacy_decoding(dev, VGA_RSRC_NONE);
-#endif
+//#if defined(CONFIG_VGA_ARB)
+//#if defined(VGA_DEFAULT_DEVICE)
+// vga_tryget(VGA_DEFAULT_DEVICE, VGA_RSRC_LEGACY_MASK);
+//#endif
+// vga_set_legacy_decoding(dev, VGA_RSRC_NONE);
+//#endif
if (NV_IS_GVI_DEVICE(nv))
{
WARNING: multiple messages have this Message-ID (diff)
From: Maik Broemme <mbroemme@parallels.com>
To: Alex Williamson <alex.williamson@redhat.com>
Cc: Knut Omang <knuto@ifi.uio.no>, kvm <kvm@vger.kernel.org>,
qemu-devel <qemu-devel@nongnu.org>
Subject: Re: [Qemu-devel] VFIO VGA test branches
Date: Wed, 29 May 2013 18:16:56 +0200 [thread overview]
Message-ID: <20130529161656.GM28436@parallels.com> (raw)
In-Reply-To: <20130529152720.GL28436@parallels.com>
[-- Attachment #1: Type: text/plain, Size: 19584 bytes --]
Hi,
Maik Broemme <mbroemme@parallels.com> wrote:
> > >
> > > Here is the DEBUG_VFIO output:
> > >
> > > vfio: vfio_initfn(0000:04:00.0) group 14
> > > vfio: region_add 0 - afffffff [0x7f8698000000]
> > > vfio: SKIPPING region_add fec00000 - fec00fff
> > > vfio: SKIPPING region_add fed00000 - fed003ff
> > > vfio: SKIPPING region_add fee00000 - feefffff
> > > vfio: region_add fffe0000 - ffffffff [0x7f88aa400000]
> > > vfio: region_add 100000000 - 24fffffff [0x7f8748000000]
> > > vfio: Device 0000:04:00.0 flags: 3, regions: 9, irgs: 4
> > > vfio: Device 0000:04:00.0 region 0:
> > > vfio: size: 0x10000000, offset: 0x0, flags: 0x7
> > > vfio: Device 0000:04:00.0 region 1:
> > > vfio: size: 0x0, offset: 0x10000000000, flags: 0x0
> > > vfio: Device 0000:04:00.0 region 2:
> > > vfio: size: 0x40000, offset: 0x20000000000, flags: 0x7
> > > vfio: Device 0000:04:00.0 region 3:
> > > vfio: size: 0x0, offset: 0x30000000000, flags: 0x0
> > > vfio: Device 0000:04:00.0 region 4:
> > > vfio: size: 0x100, offset: 0x40000000000, flags: 0x3
> > > vfio: Device 0000:04:00.0 region 5:
> > > vfio: size: 0x0, offset: 0x50000000000, flags: 0x0
> > > vfio: Device 0000:04:00.0 ROM:
> > > vfio: size: 0x20000, offset: 0x60000000000, flags: 0x1
> > > vfio: Device 0000:04:00.0 config:
> > > vfio: size: 0x1000, offset: 0x70000000000, flags: 0x3
> > > vfio: vfio_load_rom(0000:04:00.0)
> > > vfio: Enabled ATI/AMD BAR2 0x4000 quirk for device 0000:04:00.0
> > > vfio: Enabled ATI/AMD BAR4 window quirk for device 0000:04:00.0
> > > vfio: Enabled ATI/AMD quirk 0x3c3 BAR4 for device 0000:04:00.0
> > > vfio: 0000:04:00.0 PCI MSI CAP @0xa0
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> > > vfio: vfio_enable_intx_kvm(0000:04:00.0) KVM INTx accel enabled
> > > vfio: vfio_enable_intx(0000:04:00.0)
> > > vfio: vfio_initfn(0000:04:00.1) group 14
> > > vfio: Device 0000:04:00.1 flags: 3, regions: 9, irgs: 4
> > > vfio: Device 0000:04:00.1 region 0:
> > > vfio: size: 0x4000, offset: 0x0, flags: 0x7
> > > vfio: Device 0000:04:00.1 region 1:
> > > vfio: size: 0x0, offset: 0x10000000000, flags: 0x0
> > > vfio: Device 0000:04:00.1 region 2:
> > > vfio: size: 0x0, offset: 0x20000000000, flags: 0x0
> > > vfio: Device 0000:04:00.1 region 3:
> > > vfio: size: 0x0, offset: 0x30000000000, flags: 0x0
> > > vfio: Device 0000:04:00.1 region 4:
> > > vfio: size: 0x0, offset: 0x40000000000, flags: 0x0
> > > vfio: Device 0000:04:00.1 region 5:
> > > vfio: size: 0x0, offset: 0x50000000000, flags: 0x0
> > > vfio: Device 0000:04:00.1 ROM:
> > > vfio: size: 0x0, offset: 0x60000000000, flags: 0x0
> > > vfio: Device 0000:04:00.1 config:
> > > vfio: size: 0x1000, offset: 0x70000000000, flags: 0x3
> > > vfio: 0000:04:00.1 PCI MSI CAP @0xa0
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> > > vfio: vfio_enable_intx_kvm(0000:04:00.1) KVM INTx accel enabled
> > > vfio: vfio_enable_intx(0000:04:00.1)
> > > vfio: region_del 0 - afffffff
> > > vfio: region_add 0 - bffff [0x7f8698000000]
> > > vfio: region_add c0000 - dffff [0x7f88aa200000]
> > > vfio: region_add e0000 - fffff [0x7f88aa400000]
> > > vfio: region_add 100000 - afffffff [0x7f8698100000]
> > > vfio: vfio_pci_reset(0000:04:00.0)
> > > vfio: vfio_disable_intx_kvm(0000:04:00.0) KVM INTx accel disabled
> > > vfio: vfio_disable_intx(0000:04:00.0)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x54, len=0x2) 0
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 3
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x4, 0x0, len=0x2)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> > > vfio: vfio_enable_intx_kvm(0000:04:00.0) KVM INTx accel enabled
> > > vfio: vfio_enable_intx(0000:04:00.0)
> > > vfio: vfio_pci_reset(0000:04:00.1)
> > > vfio: vfio_disable_intx_kvm(0000:04:00.1) KVM INTx accel disabled
> > > vfio: vfio_disable_intx(0000:04:00.1)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x54, len=0x2) 0
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x4, len=0x2) 6
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x4, 0x0, len=0x2)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> > > vfio: vfio_enable_intx_kvm(0000:04:00.1) KVM INTx accel enabled
> > > vfio: vfio_enable_intx(0000:04:00.1)
> > > vfio: region_del 0 - bffff
> > > vfio: region_del c0000 - dffff
> > > vfio: region_add 0 - c7fff [0x7f8698000000]
> > > vfio: region_add c8000 - dffff [0x7f88aa208000]
> > > vfio: region_del 0 - c7fff
> > > vfio: region_del c8000 - dffff
> > > vfio: region_add 0 - cffff [0x7f8698000000]
> > > vfio: region_add d0000 - dffff [0x7f88aa210000]
> > > vfio: region_del 0 - cffff
> > > vfio: region_del d0000 - dffff
> > > vfio: region_add 0 - d7fff [0x7f8698000000]
> > > vfio: region_add d8000 - dffff [0x7f88aa218000]
> > > vfio: region_del 0 - d7fff
> > > vfio: region_del d8000 - dffff
> > > vfio: region_add 0 - dffff [0x7f8698000000]
> > > vfio: region_del 0 - dffff
> > > vfio: region_del e0000 - fffff
> > > vfio: region_add 0 - e7fff [0x7f8698000000]
> > > vfio: region_add e8000 - fffff [0x7f88aa408000]
> > > vfio: region_del 0 - e7fff
> > > vfio: region_del e8000 - fffff
> > > vfio: region_add 0 - effff [0x7f8698000000]
> > > vfio: region_add f0000 - fffff [0x7f88aa410000]
> > > vfio: region_del 0 - effff
> > > vfio: region_del f0000 - fffff
> > > vfio: region_del 100000 - afffffff
> > > vfio: region_add 0 - afffffff [0x7f8698000000]
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x2) 1002
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0xa, len=0x2) 300
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x2) 1002
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0xa, len=0x2) 403
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x2) 1002
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0xa, len=0x2) 300
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x2) 1002
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0xa, len=0x2) 403
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x2) 1002
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x0, len=0x4) 68181002
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x8, len=0x4) 3000000
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0xe, len=0x1) 80
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x2) 1002
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x0, len=0x4) aab01002
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x8, len=0x4) 4030000
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0xe, len=0x1) 80
> > > vfio: SKIPPING region_add b0000000 - bfffffff
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x10, len=0x4) c000000c
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x10, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x10, len=0x4) f000000c
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x10, 0xc000000c, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x14, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x14, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x14, len=0x4) ffffffff
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x14, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x18, len=0x4) fde80004
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x18, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x18, len=0x4) fffc0004
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x18, 0xfde80004, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x1c, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x1c, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x1c, len=0x4) ffffffff
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x1c, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x20, len=0x4) ce01
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x20, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x20, len=0x4) ffffff01
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x20, 0xce01, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x24, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x24, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x24, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x24, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfffff800, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) fffe0000
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x10, len=0x4) fdefc004
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x10, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x10, len=0x4) ffffc004
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x10, 0xfdefc004, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x14, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x14, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x14, len=0x4) ffffffff
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x14, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x18, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x18, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x18, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x18, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x1c, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x1c, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x1c, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x1c, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x20, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x20, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x20, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x20, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x24, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x24, 0xffffffff, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x24, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x24, 0x0, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x30, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x30, 0xfffff800, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x30, len=0x4) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x30, 0x0, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x20, 0xc000, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x18, 0xfea00000, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x1c, 0x0, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfea40000, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x10, 0xfea60000, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x14, 0x0, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x10, 0xe0000000, len=0x4)
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x14, 0x0, len=0x4)
> > > vfio: SKIPPING region_add feb40000 - feb4002f
> > > vfio: SKIPPING region_add feb40800 - feb40807
> > > vfio: SKIPPING region_add feb41000 - feb4101f
> > > vfio: SKIPPING region_add feb41800 - feb41807
> > > vfio: vfio_update_irq(0000:04:00.1) IRQ moved 20 -> 10
> > > vfio: vfio_disable_intx_kvm(0000:04:00.1) KVM INTx accel disabled
> > > vfio: vfio_enable_intx_kvm(0000:04:00.1) KVM INTx accel enabled
> > > vfio: vfio_update_irq(0000:04:00.0) IRQ moved 23 -> 11
> > > vfio: vfio_disable_intx_kvm(0000:04:00.0) KVM INTx accel disabled
> > > vfio: vfio_enable_intx_kvm(0000:04:00.0) KVM INTx accel enabled
> > > vfio: SKIPPING region_add feb42000 - feb42fff
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x3d, len=0x1) 1
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x3c, 0xb, len=0x1)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 0
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x4, 0x103, len=0x2)
> > > vfio: region_add e0000000 - efffffff [0x7f8688000000]
> > > vfio: region_add fea00000 - fea03fff [0x7f88aa7b8000]
> > > vfio: SKIPPING region_add fea04000 - fea04fff
> > > vfio: region_add fea05000 - fea3ffff [0x7f88aa7bd000]
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x3d, len=0x1) 2
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x3c, 0xa, len=0x1)
> > > vfio: vfio_pci_read_config(0000:04:00.1, @0x4, len=0x2) 0
> > > vfio: vfio_pci_write_config(0000:04:00.1, @0x4, 0x103, len=0x2)
> > > vfio: region_add fea60000 - fea63fff [0x7f88bc710000]
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 103
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 103
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x4, 0x103, len=0x2)
> > > vfio: region_del 0 - afffffff
> > > vfio: region_add 0 - 9ffff [0x7f8698000000]
> > > vfio: SKIPPING region_add a0000 - bffff
> > > vfio: region_add c0000 - afffffff [0x7f86980c0000]
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x4, len=0x2) 103
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) fea40000
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfffffffe, len=0x4)
> > > vfio: vfio_pci_read_config(0000:04:00.0, @0x30, len=0x4) fffe0000
> >
> > Here the option ROM was sized
> >
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfea40001, len=0x4)
> >
> > Then enabled
> >
> > > vfio: region_add fea40000 - fea5ffff [0x7f88a9e00000]
> >
> > Adding this memory region
> >
> > > vfio: vfio_pci_write_config(0000:04:00.0, @0x30, 0xfea40000, len=0x4)
> > > vfio: region_del fea40000 - fea5ffff
> >
> > Then disabled, removing the memory region. Presumably between the
> > enable and disable the contents were read and copied to 0xc0000, which
> > is where the VGA BIOS is shadowed.
> >
> > > Here is the strace output from this failure:
> > >
> > > 1110 ioctl(14, KVM_RUN, 0) = 0
> > > 1110 pread(20, <unfinished ...>
> > > 1099 <... poll resumed> ) = 1 ([{fd=0, revents=POLLIN}])
> > > 1099 futex(0x7ff73ca62fa0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
> > > 1109 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out)
> > > 1109 madvise(0x7ff72fe17000, 8368128, MADV_DONTNEED) = 0
> > > 1109 _exit(0) = ?
> > > 1109 +++ exited with 0 +++
> > >
> > > From reading the source 'hw/misc/vfio.c' it looks like the following
> > > in 'vfio_vga_read' never finished:
> > >
> > > if (pread(vga->fd, &buf, size, offset) != size) {
> > > error_report("%s(,0x%"HWADDR_PRIx", %d) failed: %m",
> > > __func__, region->offset + addr, size);
> > > return (uint64_t)-1;
> > > }
> >
> > I agree. Every VGA access requires us to lock the VGA resources on the
> > device, so if we can't get the lock, we stop making progress. I took a
> > look at Xorg last night and it seems like it should be taking and
> > releasing the VGA arbiter lock in a way that would be compatible with
> > our use. That's in the xserver, not the actual display hardware driver,
> > and it wraps access functions in the arbiter support, so should be
> > transparent to the drivers. So for nouveau, it seems like it should
> > work. For nvidia, we don't really know, it could be locking the device
> > from the kernel module.
> >
> > You could instrument vga_get, vga_tryget, and vga_put to figure out
> > what's happening. It might be enough to look at /dev/vga_arbiter at
> > each step in the sequence to reproduce (sudo head
> > --lines=1 /dev/vga_arbiter). Thanks,
> >
>
> I've played a bit more with it and there are some differences in
> behavior with 'nouveau' and 'nvidia' driver. As soon as I load the
> binary driver I see the following:
>
> [ 18.628676] [drm] Initialized drm 1.1.0 20060810
> [ 18.668038] nvidia: module license 'NVIDIA' taints kernel.
> [ 18.668107] Disabling lock debugging due to kernel taint
> [ 18.676638] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=io+mem
> [ 18.676722] vgaarb: transferring owner from PCI:0000:01:00.0 to PCI:0000:04:00.0
> [ 18.677007] [drm] Initialized nvidia-drm 0.0.0 20130102 for 0000:01:00.0 on minor 0
> [ 18.677090] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 319.23 Thu May 16 19:36:02 PDT 2013
>
> root@homer:~# head --lines=1 /dev/vga_arbiter
> count:2,PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none(0:0)
> root@homer:~# modprobe nvidia
> root@homer:~# head --lines=1 /dev/vga_arbiter
> count:1,PCI:0000:01:00.0,decodes=none,owns=none,locks=io+mem(1:1)
> root@homer:~# /usr/local/bin/qemu-system-x86_64 \
> -L /usr/local/share/qemu \
> -L /usr/local/share/qemu \
> -M q35 -enable-kvm -cpu host -smp cores=4,threads=1,sockets=1 \
> -m 8192 -rtc base=localtime -k de -nodefaults -vga none \
> -drive file=/home/mbroemme/.kvm/maggie.img,id=drive0,if=none,cache=none,aio=threads \
> -device virtio-blk-pci,drive=drive0,ioeventfd=on \
> -device ioh3420,id=pcie0,multifunction=on \
> -device vfio-pci,host=04:00.0,addr=0.0,bus=pcie0,multifunction=on,x-vga=on \
> -device vfio-pci,host=04:00.1,addr=0.1,bus=pcie0 -monitor stdio -nographic
> root@homer:~# head --lines=1 /dev/vga_arbiter
> count:1,PCI:0000:01:00.0,decodes=none,owns=none,locks=io+mem(1:1)
>
> It looks like nvidia binary driver locks some resources. :( Whereas
> with nouveau driver the VGA arbitration starts first if I start the
> VM with VGA passthrough:
>
> [ 178.187706] vfio-pci 0000:04:00.0: enabling device (0000 -> 0003)
> [ 178.209599] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x19@0x270
> [ 178.209631] vfio_ecap_init: 0000:04:00.0 hiding ecap 0x1b@0x2d0
> [ 181.198191] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=io+mem
> [ 181.198208] vgaarb: transferring owner from PCI:0000:01:00.0 to PCI:0000:04:00.0
>
> root@homer:~# head --lines=1 /dev/vga_arbiter
> count:2,PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none(0:0)
> root@homer:~# /usr/local/bin/qemu-system-x86_64 \
> -L /usr/local/share/qemu \
> -L /usr/local/share/qemu \
> -M q35 -enable-kvm -cpu host -smp cores=4,threads=1,sockets=1 \
> -m 8192 -rtc base=localtime -k de -nodefaults -vga none \
> -drive file=/home/mbroemme/.kvm/maggie.img,id=drive0,if=none,cache=none,aio=threads \
> -device virtio-blk-pci,drive=drive0,ioeventfd=on \
> -device ioh3420,id=pcie0,multifunction=on \
> -device vfio-pci,host=04:00.0,addr=0.0,bus=pcie0,multifunction=on,x-vga=on \
> -device vfio-pci,host=04:00.1,addr=0.1,bus=pcie0 -monitor stdio -nographic
> root@homer:~# head --lines=1 /dev/vga_arbiter
> count:1,PCI:0000:01:00.0,decodes=none,owns=none,locks=none(0:0)
>
Alex you pointed me in the right direction, many thanks! I got it
working with 'nvidia' binary driver now, but I had to patch it. I've
attached it here just for reference if others want to try the same
and I don't know if it is the proper way. After some testing (Xorg,
DRI, VDPAU, switching between X and text-console) it looks like a stable
workaround for me now until NVIDIA will fix their driver.
--Maik
[-- Attachment #2: NVIDIA-Linux-x86_64-319.23-vfio-vgaarb-fix.patch --]
[-- Type: text/plain, Size: 1355 bytes --]
diff -Nur NVIDIA-Linux-x86_64-319.23/kernel/nv-linux.h NVIDIA-Linux-x86_64-319.23-vfio-vgaarb-fix/kernel/nv-linux.h
--- NVIDIA-Linux-x86_64-319.23/kernel/nv-linux.h 2013-05-17 04:00:02.000000000 +0200
+++ NVIDIA-Linux-x86_64-319.23-vfio-vgaarb-fix/kernel/nv-linux.h 2013-05-29 18:09:42.382925622 +0200
@@ -151,9 +151,9 @@
#error "struct file_operations compile test likely failed!"
#endif
-#if defined(CONFIG_VGA_ARB)
-#include <linux/vgaarb.h>
-#endif
+//#if defined(CONFIG_VGA_ARB)
+//#include <linux/vgaarb.h>
+//#endif
#if defined(NV_VM_INSERT_PAGE_PRESENT)
#include <linux/pagemap.h>
diff -Nur NVIDIA-Linux-x86_64-319.23/kernel/nv.c NVIDIA-Linux-x86_64-319.23-vfio-vgaarb-fix/kernel/nv.c
--- NVIDIA-Linux-x86_64-319.23/kernel/nv.c 2013-05-17 04:00:02.000000000 +0200
+++ NVIDIA-Linux-x86_64-319.23-vfio-vgaarb-fix/kernel/nv.c 2013-05-29 18:10:01.494277314 +0200
@@ -2914,12 +2914,12 @@
pci_set_master(dev);
-#if defined(CONFIG_VGA_ARB)
-#if defined(VGA_DEFAULT_DEVICE)
- vga_tryget(VGA_DEFAULT_DEVICE, VGA_RSRC_LEGACY_MASK);
-#endif
- vga_set_legacy_decoding(dev, VGA_RSRC_NONE);
-#endif
+//#if defined(CONFIG_VGA_ARB)
+//#if defined(VGA_DEFAULT_DEVICE)
+// vga_tryget(VGA_DEFAULT_DEVICE, VGA_RSRC_LEGACY_MASK);
+//#endif
+// vga_set_legacy_decoding(dev, VGA_RSRC_NONE);
+//#endif
if (NV_IS_GVI_DEVICE(nv))
{
next prev parent reply other threads:[~2013-05-29 16:16 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-05-03 22:56 VFIO VGA test branches Alex Williamson
2013-05-03 22:56 ` [Qemu-devel] " Alex Williamson
2013-05-08 16:05 ` Alex Williamson
2013-05-08 16:05 ` [Qemu-devel] " Alex Williamson
2013-05-13 20:55 ` Knut Omang
2013-05-13 20:55 ` [Qemu-devel] " Knut Omang
2013-05-13 22:23 ` Alex Williamson
2013-05-13 22:23 ` [Qemu-devel] " Alex Williamson
2013-05-14 6:42 ` Knut Omang
2013-05-19 15:35 ` Knut Omang
2013-05-19 15:35 ` [Qemu-devel] " Knut Omang
2013-05-19 19:26 ` Maik Broemme
2013-05-19 19:26 ` [Qemu-devel] " Maik Broemme
2013-05-20 3:17 ` Alex Williamson
2013-05-20 3:17 ` Alex Williamson
2013-05-20 11:05 ` Maik Broemme
2013-05-20 11:05 ` Maik Broemme
2013-05-28 1:40 ` Maik Broemme
2013-05-28 3:21 ` Alex Williamson
2013-05-28 18:45 ` Maik Broemme
2013-05-28 22:28 ` Alex Williamson
2013-05-29 15:27 ` Maik Broemme
2013-05-29 16:16 ` Maik Broemme [this message]
2013-05-29 16:16 ` Maik Broemme
2013-05-29 17:07 ` Alex Williamson
2013-05-20 21:08 ` Knut Omang
2013-05-20 4:15 ` Alex Williamson
2013-05-20 4:15 ` [Qemu-devel] " Alex Williamson
2013-05-20 21:11 ` Knut Omang
2013-05-20 21:11 ` [Qemu-devel] " Knut Omang
2013-05-28 5:33 ` Knut Omang
2013-05-28 13:53 ` Alex Williamson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20130529161656.GM28436@parallels.com \
--to=mbroemme@parallels.com \
--cc=alex.williamson@redhat.com \
--cc=knuto@ifi.uio.no \
--cc=kvm@vger.kernel.org \
--cc=qemu-devel@nongnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.