qemu-xen-dir + PCI passthrough = BOOM

qemu-xen-dir + PCI passthrough = BOOM

[Konrad Rzeszutek Wilk]

Hey,

I just started noticing it today - with qemu-xen (tip is
commit b97307ecaad98360f41ea36cd9674ef810c4f8cf
    xen_disk: mark ioreq as mapped before unmapping in error case)

when I try to pass in a PCI device at bootup it blows up with:

char device redirected to /dev/pts/2 (label serial0)
qemu: hardware error: xen: failed to populate ram at 40050000
CPU #0:
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000633
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 ffff0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     00000000 0000ffff
IDT=     00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=00000000 DR1=00000000 DR2=00000000 DR3=00000000 
DR6=ffff0ff0 DR7=00000400
EFER=0000000000000000
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=00000000000000000000000000000000
XMM01=00000000000000000000000000000000
XMM02=00000000000000000000000000000000
XMM03=00000000000000000000000000000000
XMM04=00000000000000000000000000000000
XMM05=00000000000000000000000000000000
XMM06=00000000000000000000000000000000
XMM07=00000000000000000000000000000000
CPU #1:
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000633
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=1
ES =0000 00000000 0000ffff 00009300
CS =f000 ffff0000 0000ffff 00009b00


...
-bash-4.1# xl dmesg | tail
[ 2788.038463] xen_pciback: vpci(d3) [2013-12-04 19:43:40] System
requested SeaBIOS
: 0000:01:00.1: (d3) [2013-12-04 19:43:40] CPU speed is 3093 MHz
assign to virtua(d3) [2013-12-04 19:43:40] Relocating guest memory for
lowmem MMIO space disabled
l slot 0
[ 2788.076396] device vif3.0 entered promiscuous mode

If I don't have 'pci=' stanze in my guest config it boots just fine.

-bash-4.1# more /vm.cfg 
builder='hvm'
disk = [ 'file:/mnt/lab/latest/root_image.iso,hdc:cdrom,r']
memory=1024
boot="d"
vcpus=2
serial="pty"
vnclisten="0.0.0.0"
name="latest"
vif = [ 'mac=00:0F:4B:00:00:68, bridge=switch' ]
on_crash="preserve"
pci=["01:00.0"]

Any ideas?

This is with today's Xen and 3.13-rc2. The device in question is
-bash-4.1# lspci -s 01:00.0 -v
01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network
Connection (rev 01)
        Subsystem: Intel Corporation Gigabit ET Dual Port Server Adapter
        Flags: fast devsel, IRQ 16
        Memory at fbc20000 (32-bit, non-prefetchable) [disabled]
[size=128K]
        Memory at fb800000 (32-bit, non-prefetchable) [disabled]
[size=4M]
        I/O ports at e020 [disabled] [size=32]
        Memory at fbc44000 (32-bit, non-prefetchable) [disabled]
[size=16K]
        Expansion ROM at fb400000 [disabled] [size=4M]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [70] MSI-X: Enable- Count=10 Masked-
        Capabilities: [a0] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number 00-1b-21-ff-ff-45-d9-ac
        Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
        Kernel driver in use: pciback
        Kernel modules: igb


Oh and of course it boots with the traditional QEMU.

Re: qemu-xen-dir + PCI passthrough = BOOM

[Anthony PERARD]

On Wed, Dec 04, 2013 at 02:51:48PM -0500, Konrad Rzeszutek Wilk wrote:
> Hey,
> 
> I just started noticing it today - with qemu-xen (tip is
> commit b97307ecaad98360f41ea36cd9674ef810c4f8cf
>     xen_disk: mark ioreq as mapped before unmapping in error case)
> 
> when I try to pass in a PCI device at bootup it blows up with:
> 
> char device redirected to /dev/pts/2 (label serial0)
> qemu: hardware error: xen: failed to populate ram at 40050000

There is a good chance that the issue is in QEMU.

By default, qemu-xen is not very verbose, so a
#define XEN_PT_LOGGING_ENABLED
can help identifying what is missing.

The error is reported by xen_ram_alloc(), so it might help to enable
trace for it. It's easy:
sh$ cat > /tmp/trace-to-enable << EOF
xen_ram_alloc
EOF

then
device_model_args_hvm = [ '-trace', 'events=/tmp/trace-to-enable' ]

There is more of those trace events in qemu.git/trace-events

Does it works if you `xl pci-attach` after the guest has booted ?

> ...
> -bash-4.1# xl dmesg | tail
> [ 2788.038463] xen_pciback: vpci(d3) [2013-12-04 19:43:40] System
> requested SeaBIOS
> : 0000:01:00.1: (d3) [2013-12-04 19:43:40] CPU speed is 3093 MHz
> assign to virtua(d3) [2013-12-04 19:43:40] Relocating guest memory for
> lowmem MMIO space disabled
> l slot 0
> [ 2788.076396] device vif3.0 entered promiscuous mode
> 
> If I don't have 'pci=' stanze in my guest config it boots just fine.
> 
> -bash-4.1# more /vm.cfg 
> builder='hvm'
> disk = [ 'file:/mnt/lab/latest/root_image.iso,hdc:cdrom,r']
> memory=1024
> boot="d"
> vcpus=2
> serial="pty"
> vnclisten="0.0.0.0"
> name="latest"
> vif = [ 'mac=00:0F:4B:00:00:68, bridge=switch' ]
> on_crash="preserve"
> pci=["01:00.0"]
> 
> Any ideas?
> 
> This is with today's Xen and 3.13-rc2. The device in question is
> -bash-4.1# lspci -s 01:00.0 -v
> 01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network
> Connection (rev 01)
>         Subsystem: Intel Corporation Gigabit ET Dual Port Server Adapter
>         Flags: fast devsel, IRQ 16
>         Memory at fbc20000 (32-bit, non-prefetchable) [disabled]
> [size=128K]
>         Memory at fb800000 (32-bit, non-prefetchable) [disabled]
> [size=4M]
>         I/O ports at e020 [disabled] [size=32]
>         Memory at fbc44000 (32-bit, non-prefetchable) [disabled]
> [size=16K]
>         Expansion ROM at fb400000 [disabled] [size=4M]
>         Capabilities: [40] Power Management version 3
>         Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
>         Capabilities: [70] MSI-X: Enable- Count=10 Masked-
>         Capabilities: [a0] Express Endpoint, MSI 00
>         Capabilities: [100] Advanced Error Reporting
>         Capabilities: [140] Device Serial Number 00-1b-21-ff-ff-45-d9-ac
>         Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
>         Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
>         Kernel driver in use: pciback
>         Kernel modules: igb
> 
> 
> Oh and of course it boots with the traditional QEMU.

-- 
Anthony PERARD

Re: qemu-xen-dir + PCI passthrough = BOOM

[Konrad Rzeszutek Wilk]

On Thu, Dec 05, 2013 at 12:16:32PM +0000, Anthony PERARD wrote:
> On Wed, Dec 04, 2013 at 02:51:48PM -0500, Konrad Rzeszutek Wilk wrote:
> > Hey,
> > 
> > I just started noticing it today - with qemu-xen (tip is
> > commit b97307ecaad98360f41ea36cd9674ef810c4f8cf
> >     xen_disk: mark ioreq as mapped before unmapping in error case)
> > 
> > when I try to pass in a PCI device at bootup it blows up with:
> > 
> > char device redirected to /dev/pts/2 (label serial0)
> > qemu: hardware error: xen: failed to populate ram at 40050000
> 
> There is a good chance that the issue is in QEMU.
> 
> By default, qemu-xen is not very verbose, so a
> #define XEN_PT_LOGGING_ENABLED
> can help identifying what is missing.
> 
> The error is reported by xen_ram_alloc(), so it might help to enable
> trace for it. It's easy:
> sh$ cat > /tmp/trace-to-enable << EOF
> xen_ram_alloc
> EOF
> 
> then
> device_model_args_hvm = [ '-trace', 'events=/tmp/trace-to-enable' ]
> 
> There is more of those trace events in qemu.git/trace-events

char device redirected to /dev/pts/2 (label serial0)
xen_ram_alloc requested: 0x3f800000, size 0x800000
xen_ram_alloc requested: 0x40000000, size 0x10000
xen_ram_alloc requested: 0x40010000, size 0x40000
[00:05.0] xen_pt_initfn: Assigning real physical device 01:00.0 to devfn
0x28
[00:05.0] xen_pt_register_regions: IO region 0 registered
(size=0x20000lx base_addr=0xfbc20000lx type: 0)
[00:05.0] xen_pt_register_regions: IO region 1 registered
(size=0x400000lx base_addr=0xfb800000lx type: 0)
[00:05.0] xen_pt_register_regions: IO region 2 registered (size=0x20lx
base_addr=0xe020lx type: 0x1)
[00:05.0] xen_pt_register_regions: IO region 3 registered (size=0x4000lx
base_addr=0xfbc44000lx type: 0)
xen_ram_alloc requested: 0x40050000, size 0x400000
qemu: hardware error: xen: failed to populate ram at 40050000

> 
> Does it works if you `xl pci-attach` after the guest has booted ?

-bash-4.1# xl pci-attach latest 01:00.0
libxl: error: libxl_qmp.c:448:qmp_next: Socket read error: Connection
reset by peer
-bash-4.1# head -16 /var/log/xen/qemu-dm-latest.log 
char device redirected to /dev/pts/3 (label serial0)
xen_ram_alloc requested: 0x3f800000, size 0x800000
xen_ram_alloc requested: 0x40000000, size 0x10000
xen_ram_alloc requested: 0x40010000, size 0x40000
xen be: vkbd-0: initialise() failed
xen be: vkbd-0: initialise() failed
xen be: vkbd-0: initialise() failed
[00:04.0] xen_pt_initfn: Assigning real physical device 01:00.0 to devfn
0x20
[00:04.0] xen_pt_register_regions: IO region 0 registered
(size=0x20000lx base_addr=0xfbc20000lx type: 0)
[00:04.0] xen_pt_register_regions: IO region 1 registered
(size=0x400000lx base_addr=0xfb800000lx type: 0)
[00:04.0] xen_pt_register_regions: IO region 2 registered (size=0x20lx
base_addr=0xe020lx type: 0x1)
[00:04.0] xen_pt_register_regions: IO region 3 registered (size=0x4000lx
base_addr=0xfbc44000lx type: 0)
xen_ram_alloc requested: 0x40010000, size 0x400000
qemu: hardware error: xen: failed to populate ram at 40010000
CPU #0:
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000633

Re: qemu-xen-dir + PCI passthrough = BOOM

[Anthony PERARD]

On Fri, Dec 06, 2013 at 09:49:36AM -0500, Konrad Rzeszutek Wilk wrote:
> On Thu, Dec 05, 2013 at 12:16:32PM +0000, Anthony PERARD wrote:
> > On Wed, Dec 04, 2013 at 02:51:48PM -0500, Konrad Rzeszutek Wilk wrote:
> > > Hey,
> > > 
> > > I just started noticing it today - with qemu-xen (tip is
> > > commit b97307ecaad98360f41ea36cd9674ef810c4f8cf
> > >     xen_disk: mark ioreq as mapped before unmapping in error case)
> > > 
> > > when I try to pass in a PCI device at bootup it blows up with:
> > > 
> > > char device redirected to /dev/pts/2 (label serial0)
> > > qemu: hardware error: xen: failed to populate ram at 40050000
> > 
> > There is a good chance that the issue is in QEMU.
> > 
> > By default, qemu-xen is not very verbose, so a
> > #define XEN_PT_LOGGING_ENABLED
> > can help identifying what is missing.
> > 
> > The error is reported by xen_ram_alloc(), so it might help to enable
> > trace for it. It's easy:
> > sh$ cat > /tmp/trace-to-enable << EOF
> > xen_ram_alloc
> > EOF
> > 
> > then
> > device_model_args_hvm = [ '-trace', 'events=/tmp/trace-to-enable' ]
> > 
> > There is more of those trace events in qemu.git/trace-events
> 
> char device redirected to /dev/pts/2 (label serial0)
> xen_ram_alloc requested: 0x3f800000, size 0x800000
> xen_ram_alloc requested: 0x40000000, size 0x10000
> xen_ram_alloc requested: 0x40010000, size 0x40000
> [00:05.0] xen_pt_initfn: Assigning real physical device 01:00.0 to devfn
> 0x28
> [00:05.0] xen_pt_register_regions: IO region 0 registered
> (size=0x20000lx base_addr=0xfbc20000lx type: 0)
> [00:05.0] xen_pt_register_regions: IO region 1 registered
> (size=0x400000lx base_addr=0xfb800000lx type: 0)
> [00:05.0] xen_pt_register_regions: IO region 2 registered (size=0x20lx
> base_addr=0xe020lx type: 0x1)
> [00:05.0] xen_pt_register_regions: IO region 3 registered (size=0x4000lx
> base_addr=0xfbc44000lx type: 0)
> xen_ram_alloc requested: 0x40050000, size 0x400000
> qemu: hardware error: xen: failed to populate ram at 40050000

Those Xen report something like:
(XEN) page_alloc.c:1460:d0 Over-allocation for domain 46: 131329 >
131328
(XEN) memory.c:132:d0 Could not allocate order=0 extent: id=46
memflags=0 (62 of 64)

?

(I tryied to reproduce the issue by simply add many emulated e1000 in
QEMU :) )

-- 
Anthony PERARD

Re: qemu-xen-dir + PCI passthrough = BOOM

[Wei Liu]

On Fri, Dec 06, 2013 at 03:35:04PM +0000, Anthony PERARD wrote:
> On Fri, Dec 06, 2013 at 09:49:36AM -0500, Konrad Rzeszutek Wilk wrote:
> > On Thu, Dec 05, 2013 at 12:16:32PM +0000, Anthony PERARD wrote:
> > > On Wed, Dec 04, 2013 at 02:51:48PM -0500, Konrad Rzeszutek Wilk wrote:
> > > > Hey,
> > > > 
> > > > I just started noticing it today - with qemu-xen (tip is
> > > > commit b97307ecaad98360f41ea36cd9674ef810c4f8cf
> > > >     xen_disk: mark ioreq as mapped before unmapping in error case)
> > > > 
> > > > when I try to pass in a PCI device at bootup it blows up with:
> > > > 
> > > > char device redirected to /dev/pts/2 (label serial0)
> > > > qemu: hardware error: xen: failed to populate ram at 40050000
> > > 
> > > There is a good chance that the issue is in QEMU.
> > > 
> > > By default, qemu-xen is not very verbose, so a
> > > #define XEN_PT_LOGGING_ENABLED
> > > can help identifying what is missing.
> > > 
> > > The error is reported by xen_ram_alloc(), so it might help to enable
> > > trace for it. It's easy:
> > > sh$ cat > /tmp/trace-to-enable << EOF
> > > xen_ram_alloc
> > > EOF
> > > 
> > > then
> > > device_model_args_hvm = [ '-trace', 'events=/tmp/trace-to-enable' ]
> > > 
> > > There is more of those trace events in qemu.git/trace-events
> > 
> > char device redirected to /dev/pts/2 (label serial0)
> > xen_ram_alloc requested: 0x3f800000, size 0x800000
> > xen_ram_alloc requested: 0x40000000, size 0x10000
> > xen_ram_alloc requested: 0x40010000, size 0x40000
> > [00:05.0] xen_pt_initfn: Assigning real physical device 01:00.0 to devfn
> > 0x28
> > [00:05.0] xen_pt_register_regions: IO region 0 registered
> > (size=0x20000lx base_addr=0xfbc20000lx type: 0)
> > [00:05.0] xen_pt_register_regions: IO region 1 registered
> > (size=0x400000lx base_addr=0xfb800000lx type: 0)
> > [00:05.0] xen_pt_register_regions: IO region 2 registered (size=0x20lx
> > base_addr=0xe020lx type: 0x1)
> > [00:05.0] xen_pt_register_regions: IO region 3 registered (size=0x4000lx
> > base_addr=0xfbc44000lx type: 0)
> > xen_ram_alloc requested: 0x40050000, size 0x400000
> > qemu: hardware error: xen: failed to populate ram at 40050000
> 
> Those Xen report something like:
> (XEN) page_alloc.c:1460:d0 Over-allocation for domain 46: 131329 >
> 131328
> (XEN) memory.c:132:d0 Could not allocate order=0 extent: id=46
> memflags=0 (62 of 64)
> 
> ?
> 
> (I tryied to reproduce the issue by simply add many emulated e1000 in
> QEMU :) )
> 

FWIW this probably comes from XENMEM_populate_physmap. Simply speaking
your domain is exceeding memory limit...

Wei.

> -- 
> Anthony PERARD
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: qemu-xen-dir + PCI passthrough = BOOM

[Wei Liu]

On Fri, Dec 06, 2013 at 04:00:18PM +0000, Wei Liu wrote:
[...]
> > Those Xen report something like:
> > (XEN) page_alloc.c:1460:d0 Over-allocation for domain 46: 131329 >
> > 131328
> > (XEN) memory.c:132:d0 Could not allocate order=0 extent: id=46
> > memflags=0 (62 of 64)
> > 
> > ?
> > 
> > (I tryied to reproduce the issue by simply add many emulated e1000 in
> > QEMU :) )
> > 
> 
> FWIW this probably comes from XENMEM_populate_physmap. Simply speaking
> your domain is exceeding memory limit...
> 

And, if this comes from hvmloader it is not fatal. It will use lower
memory to fill in the holes.

Wei.

Re: qemu-xen-dir + PCI passthrough = BOOM

[Konrad Rzeszutek Wilk]

On Fri, Dec 06, 2013 at 04:03:10PM +0000, Wei Liu wrote:
> On Fri, Dec 06, 2013 at 04:00:18PM +0000, Wei Liu wrote:
> [...]
> > > Those Xen report something like:
> > > (XEN) page_alloc.c:1460:d0 Over-allocation for domain 46: 131329 >
> > > 131328
> > > (XEN) memory.c:132:d0 Could not allocate order=0 extent: id=46
> > > memflags=0 (62 of 64)
> > > 
> > > ?
> > > 
> > > (I tryied to reproduce the issue by simply add many emulated e1000 in
> > > QEMU :) )
> > > 
> > 
> > FWIW this probably comes from XENMEM_populate_physmap. Simply speaking
> > your domain is exceeding memory limit...
> > 
> 
> And, if this comes from hvmloader it is not fatal. It will use lower
> memory to fill in the holes.

It says:

[  626.260261] d(d4) [2013-12-16 15:06:53] HVM Loader
evice vif4.0 ent(d4) [2013-12-16 15:06:53] Detected Xen v4.4-unstable
ered promiscuous(d4) [2013-12-16 15:06:53] Xenbus rings @0xfeffc000, event channel 3
 mode
(d4) [2013-12-16 15:06:53] System requested SeaBIOS
[  626.266706] x(d4) [2013-12-16 15:06:53] CPU speed is 3093 MHz
en_pciback: vpci(d4) [2013-12-16 15:06:53] Relocating guest memory for lowmem MMIO space disabled
: 0000:01:00.0: assign to virtual slot 0

And the device I am trying to pass in is:
-bash-4.1# lspci -s 01:00.0 -v 
01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
        Subsystem: Intel Corporation Gigabit ET Dual Port Server Adapter
        Flags: fast devsel, IRQ 16
        Memory at fbc20000 (32-bit, non-prefetchable) [disabled] [size=128K]
        Memory at fb800000 (32-bit, non-prefetchable) [disabled] [size=4M]
        I/O ports at e020 [disabled] [size=32]
        Memory at fbc44000 (32-bit, non-prefetchable) [disabled] [size=16K]
        Expansion ROM at fb400000 [disabled] [size=4M]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
        Capabilities: [70] MSI-X: Enable- Count=10 Masked-
        Capabilities: [a0] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Device Serial Number 00-1b-21-ff-ff-45-d9-ac
        Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
        Kernel driver in use: pciback
        Kernel modules: igb



> 
> Wei.

Re: qemu-xen-dir + PCI passthrough = BOOM

[Anthony PERARD]

On Mon, Dec 16, 2013 at 10:08:16AM -0500, Konrad Rzeszutek Wilk wrote:
> On Fri, Dec 06, 2013 at 04:03:10PM +0000, Wei Liu wrote:
> > On Fri, Dec 06, 2013 at 04:00:18PM +0000, Wei Liu wrote:
> > [...]
> > > > Those Xen report something like:
> > > > (XEN) page_alloc.c:1460:d0 Over-allocation for domain 46: 131329 >
> > > > 131328
> > > > (XEN) memory.c:132:d0 Could not allocate order=0 extent: id=46
> > > > memflags=0 (62 of 64)
> > > > 
> > > > ?
> > > > 
> > > > (I tryied to reproduce the issue by simply add many emulated e1000 in
> > > > QEMU :) )
> > > > 

> -bash-4.1# lspci -s 01:00.0 -v 
> 01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
>         Subsystem: Intel Corporation Gigabit ET Dual Port Server Adapter
>         Flags: fast devsel, IRQ 16
>         Memory at fbc20000 (32-bit, non-prefetchable) [disabled] [size=128K]
>         Memory at fb800000 (32-bit, non-prefetchable) [disabled] [size=4M]
>         I/O ports at e020 [disabled] [size=32]
>         Memory at fbc44000 (32-bit, non-prefetchable) [disabled] [size=16K]
>         Expansion ROM at fb400000 [disabled] [size=4M]

BTW, I think this is the issue, the Expansion ROM. qemu-xen will
allocate memory for it. Will have maybe have to find another way.
qemu-trad those not seems to allocate memory, but I haven't been very
far in trying to check that.

-- 
Anthony PERARD

Re: qemu-xen-dir + PCI passthrough = BOOM

[Ian Campbell]

On Wed, 2013-12-18 at 14:48 +0000, Anthony PERARD wrote:
> On Mon, Dec 16, 2013 at 10:08:16AM -0500, Konrad Rzeszutek Wilk wrote:
> > On Fri, Dec 06, 2013 at 04:03:10PM +0000, Wei Liu wrote:
> > > On Fri, Dec 06, 2013 at 04:00:18PM +0000, Wei Liu wrote:
> > > [...]
> > > > > Those Xen report something like:
> > > > > (XEN) page_alloc.c:1460:d0 Over-allocation for domain 46: 131329 >
> > > > > 131328
> > > > > (XEN) memory.c:132:d0 Could not allocate order=0 extent: id=46
> > > > > memflags=0 (62 of 64)
> > > > > 
> > > > > ?
> > > > > 
> > > > > (I tryied to reproduce the issue by simply add many emulated e1000 in
> > > > > QEMU :) )
> > > > > 
> 
> > -bash-4.1# lspci -s 01:00.0 -v 
> > 01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
> >         Subsystem: Intel Corporation Gigabit ET Dual Port Server Adapter
> >         Flags: fast devsel, IRQ 16
> >         Memory at fbc20000 (32-bit, non-prefetchable) [disabled] [size=128K]
> >         Memory at fb800000 (32-bit, non-prefetchable) [disabled] [size=4M]
> >         I/O ports at e020 [disabled] [size=32]
> >         Memory at fbc44000 (32-bit, non-prefetchable) [disabled] [size=16K]
> >         Expansion ROM at fb400000 [disabled] [size=4M]
> 
> BTW, I think this is the issue, the Expansion ROM. qemu-xen will
> allocate memory for it. Will have maybe have to find another way.
> qemu-trad those not seems to allocate memory, but I haven't been very
> far in trying to check that.

This might SeaBIOS, since it is the BIOS which runs the expansion roms.

ROMBIOS didn't run expansion ROMS, instead IIRC you have to extract the
ones you care about from the device(s) and build them into hvmloader
(somehow) to be predeployed.

Could try building a seabios without CONFIG_OPTIONROMS (edit
tools/firmware/seabios-config) and see if that helps -- although not if
there is some other essential option rom of course.

That's not the solution but it would help diagnose the issue.

Ian.

Re: qemu-xen-dir + PCI passthrough = BOOM

[Konrad Rzeszutek Wilk]

On Wed, Dec 18, 2013 at 02:57:26PM +0000, Ian Campbell wrote:
> On Wed, 2013-12-18 at 14:48 +0000, Anthony PERARD wrote:
> > On Mon, Dec 16, 2013 at 10:08:16AM -0500, Konrad Rzeszutek Wilk wrote:
> > > On Fri, Dec 06, 2013 at 04:03:10PM +0000, Wei Liu wrote:
> > > > On Fri, Dec 06, 2013 at 04:00:18PM +0000, Wei Liu wrote:
> > > > [...]
> > > > > > Those Xen report something like:
> > > > > > (XEN) page_alloc.c:1460:d0 Over-allocation for domain 46: 131329 >
> > > > > > 131328
> > > > > > (XEN) memory.c:132:d0 Could not allocate order=0 extent: id=46
> > > > > > memflags=0 (62 of 64)
> > > > > > 
> > > > > > ?
> > > > > > 
> > > > > > (I tryied to reproduce the issue by simply add many emulated e1000 in
> > > > > > QEMU :) )
> > > > > > 
> > 
> > > -bash-4.1# lspci -s 01:00.0 -v 
> > > 01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
> > >         Subsystem: Intel Corporation Gigabit ET Dual Port Server Adapter
> > >         Flags: fast devsel, IRQ 16
> > >         Memory at fbc20000 (32-bit, non-prefetchable) [disabled] [size=128K]
> > >         Memory at fb800000 (32-bit, non-prefetchable) [disabled] [size=4M]
> > >         I/O ports at e020 [disabled] [size=32]
> > >         Memory at fbc44000 (32-bit, non-prefetchable) [disabled] [size=16K]
> > >         Expansion ROM at fb400000 [disabled] [size=4M]
> > 
> > BTW, I think this is the issue, the Expansion ROM. qemu-xen will
> > allocate memory for it. Will have maybe have to find another way.
> > qemu-trad those not seems to allocate memory, but I haven't been very
> > far in trying to check that.
> 
> This might SeaBIOS, since it is the BIOS which runs the expansion roms.
> 
> ROMBIOS didn't run expansion ROMS, instead IIRC you have to extract the
> ones you care about from the device(s) and build them into hvmloader
> (somehow) to be predeployed.

In 'rombios_load_roms' I see:

106     option_rom_phys_addr = etherboot_phys_addr + etherboot_sz;                  
107     option_rom_sz = pci_load_option_roms(OPTIONROM_PHYSICAL_END,                
108                                          option_rom_phys_addr);                 
109                                                                                 
110     printf("Option ROMs:\n");                                                   
111     if ( vgabios_sz )                                                           
112         printf(" %05x-%05x: VGA BIOS\n",                                        
113                VGABIOS_PHYSICAL_ADDRESS,                                        
114                VGABIOS_PHYSICAL_ADDRESS + vgabios_sz - 1);                      
115     if ( etherboot_sz )                                                         
116         printf(" %05x-%05x: Etherboot ROM\n",                                   
117                etherboot_phys_addr,                                             
118                etherboot_phys_addr + etherboot_sz - 1);                         
119     if ( option_rom_sz )                                                        
120         printf(" %05x-%05x: PCI Option ROMs\n",                                 
121                option_rom_phys_addr,                                            
122                option_rom_phys_addr + option_rom_sz - 1);                   


seems to imply that it sets it up. Doing a git log on rombios
shows this:


commit c2901a5cc44c85a9ceeebd47ff21946bb35ba930
Author: Keir Fraser <keir.fraser@citrix.com>
Date:   Tue Mar 31 14:04:50 2009 +0100

    hvmloader: ROMBIOS must POST in big real mode, since it supports PMM,
    and hence Option ROMs can expect noi segment limit checks.

which would imply that the Option ROMs would be loaded and run.

> 
> Could try building a seabios without CONFIG_OPTIONROMS (edit
> tools/firmware/seabios-config) and see if that helps -- although not if
> there is some other essential option rom of course.
> 
> That's not the solution but it would help diagnose the issue.

Looking at hvmloader in 'pci_load_option_roms' I see:

                                                                               
165         /*                                                                      
166          * Currently only scan options from mass storage devices and serial     
167          * bus controller (Fibre Channel included).                             
168          */                                                                     
169         if ( (class != 0x1) && (class != 0xc) )                                 
170             continue;        

Which would imply that the option rom for the NIC is not loaded.


But more interestingly, the hvmloader has this:

136 struct bios_config seabios_config = {                                           
137     .name = "SeaBIOS",                                                          
138                                                                                 
139     .image = seabios,                                                           
140     .image_size = sizeof(seabios),                                              
141                                                                                 
142     .bios_address = 0x100000 - sizeof(seabios),                                 
143                                                                                 
144     .load_roms = NULL,                                                          
145                                                                                 
146     .bios_load = NULL,                                                          
147                                                                                 

load_roms and bios_load are not set - so it wouldn't even do it.
It only does it for Bochs BIOS.
> 
> Ian.
> 

Re: qemu-xen-dir + PCI passthrough = BOOM

[Ian Campbell]

On Wed, 2013-12-18 at 10:22 -0500, Konrad Rzeszutek Wilk wrote:
> load_roms and bios_load are not set - so it wouldn't even do it.
> It only does it for Bochs BIOS.

Right, this is deliberate.

For ROMBIOS (AKA BOchs BIOS) hvmloader loads the options roms. and I
think ROMBIOS subsequently loads them.

For SeaBIOS it is the BIOS itself which both loads and executes the
ROMS, which is why it is NULL in hvmloader.

The SeaBIOS way is far more like how systems normally work and because
the BIOS is in charge it can do a better job than splitting it between
two entities.

Seems like perhaps I was mistaken about hvmloader+rombios not supporting
loading from the ROM BAR of the device, it does seem to have some
limited support for that (serial controllers and disk controllers only)

hvmloader+rombios does hardcode the VGA bios (rather than getting it
from the emulated devices ROM bar), which is probably what I was
thinking of, that's why people doing VGA passthrough with rombios need
to grab the bios image and build it into hvmloader.

hvmloader+rombios also hardcodes the etherboot rom.

Ian.

Re: qemu-xen-dir + PCI passthrough = BOOM

[Konrad Rzeszutek Wilk]

On Wed, Dec 18, 2013 at 04:13:05PM +0000, Ian Campbell wrote:
> On Wed, 2013-12-18 at 10:22 -0500, Konrad Rzeszutek Wilk wrote:
> > load_roms and bios_load are not set - so it wouldn't even do it.
> > It only does it for Bochs BIOS.
> 
> Right, this is deliberate.
> 
> For ROMBIOS (AKA BOchs BIOS) hvmloader loads the options roms. and I
> think ROMBIOS subsequently loads them.
> 
> For SeaBIOS it is the BIOS itself which both loads and executes the
> ROMS, which is why it is NULL in hvmloader.

Aha!
> 
> The SeaBIOS way is far more like how systems normally work and because
> the BIOS is in charge it can do a better job than splitting it between
> two entities.
> 
> Seems like perhaps I was mistaken about hvmloader+rombios not supporting
> loading from the ROM BAR of the device, it does seem to have some
> limited support for that (serial controllers and disk controllers only)
> 
> hvmloader+rombios does hardcode the VGA bios (rather than getting it
> from the emulated devices ROM bar), which is probably what I was
> thinking of, that's why people doing VGA passthrough with rombios need
> to grab the bios image and build it into hvmloader.

Right, which I hope to fix to make this work without this madness.
> 
> hvmloader+rombios also hardcodes the etherboot rom.
> 
> Ian.
> 

Re: qemu-xen-dir + PCI passthrough = BOOM

[Wu, Feng]

> -----Original Message-----
> From: xen-devel-bounces@lists.xen.org
> [mailto:xen-devel-bounces@lists.xen.org] On Behalf Of Ian Campbell
> Sent: Thursday, December 19, 2013 12:13 AM
> To: Konrad Rzeszutek Wilk
> Cc: Anthony PERARD; xen-devel@lists.xenproject.org;
> stefano.stabellini@citrix.com
> Subject: Re: [Xen-devel] qemu-xen-dir + PCI passthrough = BOOM
> 
> On Wed, 2013-12-18 at 10:22 -0500, Konrad Rzeszutek Wilk wrote:
> > load_roms and bios_load are not set - so it wouldn't even do it.
> > It only does it for Bochs BIOS.
> 
> Right, this is deliberate.
> 
> For ROMBIOS (AKA BOchs BIOS) hvmloader loads the options roms. and I
> think ROMBIOS subsequently loads them.
> 
> For SeaBIOS it is the BIOS itself which both loads and executes the
> ROMS, which is why it is NULL in hvmloader.
> 
> The SeaBIOS way is far more like how systems normally work and because
> the BIOS is in charge it can do a better job than splitting it between
> two entities.
> 

I am also interested in this thread. Do you know why ROMBIOS doesn't handle
option ROM the same way as seaBIOS? Is there any special case that ROMBIOS
cannot handle? Thanks!

> Seems like perhaps I was mistaken about hvmloader+rombios not supporting
> loading from the ROM BAR of the device, it does seem to have some
> limited support for that (serial controllers and disk controllers only)
> 
> hvmloader+rombios does hardcode the VGA bios (rather than getting it
> from the emulated devices ROM bar), which is probably what I was
> thinking of, that's why people doing VGA passthrough with rombios need
> to grab the bios image and build it into hvmloader.
> 
> hvmloader+rombios also hardcodes the etherboot rom.
> 
> Ian.
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Thanks,
Feng

Re: qemu-xen-dir + PCI passthrough = BOOM

[Konrad Rzeszutek Wilk]

On Sun, Dec 22, 2013 at 11:25:20AM +0000, Wu, Feng wrote:
> 
> 
> > -----Original Message-----
> > From: xen-devel-bounces@lists.xen.org
> > [mailto:xen-devel-bounces@lists.xen.org] On Behalf Of Ian Campbell
> > Sent: Thursday, December 19, 2013 12:13 AM
> > To: Konrad Rzeszutek Wilk
> > Cc: Anthony PERARD; xen-devel@lists.xenproject.org;
> > stefano.stabellini@citrix.com
> > Subject: Re: [Xen-devel] qemu-xen-dir + PCI passthrough = BOOM
> > 
> > On Wed, 2013-12-18 at 10:22 -0500, Konrad Rzeszutek Wilk wrote:
> > > load_roms and bios_load are not set - so it wouldn't even do it.
> > > It only does it for Bochs BIOS.
> > 
> > Right, this is deliberate.
> > 
> > For ROMBIOS (AKA BOchs BIOS) hvmloader loads the options roms. and I
> > think ROMBIOS subsequently loads them.
> > 
> > For SeaBIOS it is the BIOS itself which both loads and executes the
> > ROMS, which is why it is NULL in hvmloader.
> > 
> > The SeaBIOS way is far more like how systems normally work and because
> > the BIOS is in charge it can do a better job than splitting it between
> > two entities.
> > 
> 
> I am also interested in this thread. Do you know why ROMBIOS doesn't handle
> option ROM the same way as seaBIOS? Is there any special case that ROMBIOS
> cannot handle? Thanks!

When I find out I will make sure to notify you as well.

Thanks!

Re: qemu-xen-dir + PCI passthrough = BOOM

[Ian Campbell]

On Sun, 2013-12-22 at 11:25 +0000, Wu, Feng wrote:
> 
> > -----Original Message-----
> > From: xen-devel-bounces@lists.xen.org
> > [mailto:xen-devel-bounces@lists.xen.org] On Behalf Of Ian Campbell
> > Sent: Thursday, December 19, 2013 12:13 AM
> > To: Konrad Rzeszutek Wilk
> > Cc: Anthony PERARD; xen-devel@lists.xenproject.org;
> > stefano.stabellini@citrix.com
> > Subject: Re: [Xen-devel] qemu-xen-dir + PCI passthrough = BOOM
> > 
> > On Wed, 2013-12-18 at 10:22 -0500, Konrad Rzeszutek Wilk wrote:
> > > load_roms and bios_load are not set - so it wouldn't even do it.
> > > It only does it for Bochs BIOS.
> > 
> > Right, this is deliberate.
> > 
> > For ROMBIOS (AKA BOchs BIOS) hvmloader loads the options roms. and I
> > think ROMBIOS subsequently loads them.
> > 
> > For SeaBIOS it is the BIOS itself which both loads and executes the
> > ROMS, which is why it is NULL in hvmloader.
> > 
> > The SeaBIOS way is far more like how systems normally work and because
> > the BIOS is in charge it can do a better job than splitting it between
> > two entities.
> > 
> 
> I am also interested in this thread. Do you know why ROMBIOS doesn't handle
> option ROM the same way as seaBIOS?

ROMBIOS was always this way and nobody has updated it to do things
properly. It's unclear whether it is worth the effort.

IMHO any effort expended would be better spent bringing qemu-upstream
+seabios up to scratch rather than spending time fixing qemu-traditional
+ROMBIOS for new usecases. Although qemu-traditional+ROMBIOS is not
going to be removed (in order to retain compatibility with existing
configurations) it is also not taking new features etc.

Ian.

Re: qemu-xen-dir + PCI passthrough = BOOM

[Wu, Feng]

> -----Original Message-----
> From: Ian Campbell [mailto:Ian.Campbell@citrix.com]
> Sent: Monday, January 06, 2014 5:54 PM
> To: Wu, Feng
> Cc: Konrad Rzeszutek Wilk; Anthony PERARD; xen-devel@lists.xenproject.org;
> stefano.stabellini@citrix.com
> Subject: Re: [Xen-devel] qemu-xen-dir + PCI passthrough = BOOM
> 
> On Sun, 2013-12-22 at 11:25 +0000, Wu, Feng wrote:
> >
> > > -----Original Message-----
> > > From: xen-devel-bounces@lists.xen.org
> > > [mailto:xen-devel-bounces@lists.xen.org] On Behalf Of Ian Campbell
> > > Sent: Thursday, December 19, 2013 12:13 AM
> > > To: Konrad Rzeszutek Wilk
> > > Cc: Anthony PERARD; xen-devel@lists.xenproject.org;
> > > stefano.stabellini@citrix.com
> > > Subject: Re: [Xen-devel] qemu-xen-dir + PCI passthrough = BOOM
> > >
> > > On Wed, 2013-12-18 at 10:22 -0500, Konrad Rzeszutek Wilk wrote:
> > > > load_roms and bios_load are not set - so it wouldn't even do it.
> > > > It only does it for Bochs BIOS.
> > >
> > > Right, this is deliberate.
> > >
> > > For ROMBIOS (AKA BOchs BIOS) hvmloader loads the options roms. and I
> > > think ROMBIOS subsequently loads them.
> > >
> > > For SeaBIOS it is the BIOS itself which both loads and executes the
> > > ROMS, which is why it is NULL in hvmloader.
> > >
> > > The SeaBIOS way is far more like how systems normally work and because
> > > the BIOS is in charge it can do a better job than splitting it between
> > > two entities.
> > >
> >
> > I am also interested in this thread. Do you know why ROMBIOS doesn't handle
> > option ROM the same way as seaBIOS?
> 
> ROMBIOS was always this way and nobody has updated it to do things
> properly. It's unclear whether it is worth the effort.
> 
> IMHO any effort expended would be better spent bringing qemu-upstream
> +seabios up to scratch rather than spending time fixing qemu-traditional
> +ROMBIOS for new usecases. Although qemu-traditional+ROMBIOS is not
> going to be removed (in order to retain compatibility with existing
> configurations) it is also not taking new features etc.
> 

Thanks a lot for your explanation!

> Ian.

Re: qemu-xen-dir + PCI passthrough = BOOM

[Konrad Rzeszutek Wilk]

On Wed, Dec 18, 2013 at 02:48:24PM +0000, Anthony PERARD wrote:
> On Mon, Dec 16, 2013 at 10:08:16AM -0500, Konrad Rzeszutek Wilk wrote:
> > On Fri, Dec 06, 2013 at 04:03:10PM +0000, Wei Liu wrote:
> > > On Fri, Dec 06, 2013 at 04:00:18PM +0000, Wei Liu wrote:
> > > [...]
> > > > > Those Xen report something like:
> > > > > (XEN) page_alloc.c:1460:d0 Over-allocation for domain 46: 131329 >
> > > > > 131328
> > > > > (XEN) memory.c:132:d0 Could not allocate order=0 extent: id=46
> > > > > memflags=0 (62 of 64)
> > > > > 
> > > > > ?
> > > > > 
> > > > > (I tryied to reproduce the issue by simply add many emulated e1000 in
> > > > > QEMU :) )
> > > > > 
> 
> > -bash-4.1# lspci -s 01:00.0 -v 
> > 01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
> >         Subsystem: Intel Corporation Gigabit ET Dual Port Server Adapter
> >         Flags: fast devsel, IRQ 16
> >         Memory at fbc20000 (32-bit, non-prefetchable) [disabled] [size=128K]
> >         Memory at fb800000 (32-bit, non-prefetchable) [disabled] [size=4M]
> >         I/O ports at e020 [disabled] [size=32]
> >         Memory at fbc44000 (32-bit, non-prefetchable) [disabled] [size=16K]
> >         Expansion ROM at fb400000 [disabled] [size=4M]
> 
> BTW, I think this is the issue, the Expansion ROM. qemu-xen will
> allocate memory for it. Will have maybe have to find another way.
> qemu-trad those not seems to allocate memory, but I haven't been very
> far in trying to check that.

And indeed that is the case. The "Fix" below fixes it.


Based on that and this guest config:
disk = [ 'file:/mnt/lab/latest/root_image.iso,hdc:cdrom,r']
memory = 2048
boot="d"
maxvcpus=32
vcpus=1
serial='pty'
vnclisten="0.0.0.0"
name="latest"
vif = [ 'mac=00:0F:4B:00:00:68, bridge=switch' ]
pci = ["01:00.0"]

I can boot the guest.

diff --git a/hw/xen/xen_pt.c b/hw/xen/xen_pt.c
index ca2d460..82b3890 100644
--- a/hw/xen/xen_pt.c
+++ b/hw/xen/xen_pt.c
@@ -425,6 +425,7 @@ static int xen_pt_register_regions(XenPCIPassthroughState *s)
                    i, r->size, r->base_addr, type);
     }
 
+#if 0
     /* Register expansion ROM address */
     if (d->rom.base_addr && d->rom.size) {
         uint32_t bar_data = 0;
@@ -449,7 +450,7 @@ static int xen_pt_register_regions(XenPCIPassthroughState *s)
                    " base_addr=0x%08"PRIx64")\n",
                    d->rom.size, d->rom.base_addr);
     }
-
+#endif
     return 0;
 }

Re: qemu-xen-dir + PCI passthrough = BOOM

[Anthony PERARD]

On Wed, Jan 08, 2014 at 02:44:51PM -0500, Konrad Rzeszutek Wilk wrote:
> On Wed, Dec 18, 2013 at 02:48:24PM +0000, Anthony PERARD wrote:
> > On Mon, Dec 16, 2013 at 10:08:16AM -0500, Konrad Rzeszutek Wilk wrote:
> > > On Fri, Dec 06, 2013 at 04:03:10PM +0000, Wei Liu wrote:
> > > > On Fri, Dec 06, 2013 at 04:00:18PM +0000, Wei Liu wrote:
> > > > [...]
> > > > > > Those Xen report something like:
> > > > > > (XEN) page_alloc.c:1460:d0 Over-allocation for domain 46: 131329 >
> > > > > > 131328
> > > > > > (XEN) memory.c:132:d0 Could not allocate order=0 extent: id=46
> > > > > > memflags=0 (62 of 64)
> > > > > > 
> > > > > > ?
> > > > > > 
> > > > > > (I tryied to reproduce the issue by simply add many emulated e1000 in
> > > > > > QEMU :) )
> > > > > > 
> > 
> > > -bash-4.1# lspci -s 01:00.0 -v 
> > > 01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
> > >         Subsystem: Intel Corporation Gigabit ET Dual Port Server Adapter
> > >         Flags: fast devsel, IRQ 16
> > >         Memory at fbc20000 (32-bit, non-prefetchable) [disabled] [size=128K]
> > >         Memory at fb800000 (32-bit, non-prefetchable) [disabled] [size=4M]
> > >         I/O ports at e020 [disabled] [size=32]
> > >         Memory at fbc44000 (32-bit, non-prefetchable) [disabled] [size=16K]
> > >         Expansion ROM at fb400000 [disabled] [size=4M]
> > 
> > BTW, I think this is the issue, the Expansion ROM. qemu-xen will
> > allocate memory for it. Will have maybe have to find another way.
> > qemu-trad those not seems to allocate memory, but I haven't been very
> > far in trying to check that.
> 
> And indeed that is the case. The "Fix" below fixes it.
> 
> 
> Based on that and this guest config:
> disk = [ 'file:/mnt/lab/latest/root_image.iso,hdc:cdrom,r']
> memory = 2048
> boot="d"
> maxvcpus=32
> vcpus=1
> serial='pty'
> vnclisten="0.0.0.0"
> name="latest"
> vif = [ 'mac=00:0F:4B:00:00:68, bridge=switch' ]
> pci = ["01:00.0"]
> 
> I can boot the guest.

And can you access the ROM from the guest ?


Also, I have another patch, it will initialize the PCI ROM BAR like any
other BAR. In this case, if qemu is envolved in the access to ROM, it
will print an error, like it the case for other BAR. 

I tried to test it, but it was with an embedded VGA card. When I dump
the ROM, I got the same one as the emulated card instead of the ROM from
the device.


diff --git a/hw/xen/xen_pt.c b/hw/xen/xen_pt.c
index 6dd7a68..2bbdb6d 100644
--- a/hw/xen/xen_pt.c
+++ b/hw/xen/xen_pt.c
@@ -440,8 +440,8 @@ static int xen_pt_register_regions(XenPCIPassthroughState *s)
 
         s->bases[PCI_ROM_SLOT].access.maddr = d->rom.base_addr;
 
-        memory_region_init_rom_device(&s->rom, OBJECT(s), NULL, NULL,
-                                      "xen-pci-pt-rom", d->rom.size);
+        memory_region_init_io(&s->rom, OBJECT(s), &ops, &s->dev,
+                              "xen-pci-pt-rom", d->rom.size);
         pci_register_bar(&s->dev, PCI_ROM_SLOT, PCI_BASE_ADDRESS_MEM_PREFETCH,
                          &s->rom);
 

-- 
Anthony PERARD

Re: qemu-xen-dir + PCI passthrough = BOOM

[Pasi Kärkkäinen]

On Thu, Jan 09, 2014 at 02:56:24PM +0000, Anthony PERARD wrote:
> On Wed, Jan 08, 2014 at 02:44:51PM -0500, Konrad Rzeszutek Wilk wrote:
> > On Wed, Dec 18, 2013 at 02:48:24PM +0000, Anthony PERARD wrote:
> > > On Mon, Dec 16, 2013 at 10:08:16AM -0500, Konrad Rzeszutek Wilk wrote:
> > > > On Fri, Dec 06, 2013 at 04:03:10PM +0000, Wei Liu wrote:
> > > > > On Fri, Dec 06, 2013 at 04:00:18PM +0000, Wei Liu wrote:
> > > > > [...]
> > > > > > > Those Xen report something like:
> > > > > > > (XEN) page_alloc.c:1460:d0 Over-allocation for domain 46: 131329 >
> > > > > > > 131328
> > > > > > > (XEN) memory.c:132:d0 Could not allocate order=0 extent: id=46
> > > > > > > memflags=0 (62 of 64)
> > > > > > > 
> > > > > > > ?
> > > > > > > 
> > > > > > > (I tryied to reproduce the issue by simply add many emulated e1000 in
> > > > > > > QEMU :) )
> > > > > > > 
> > > 
> > > > -bash-4.1# lspci -s 01:00.0 -v 
> > > > 01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
> > > >         Subsystem: Intel Corporation Gigabit ET Dual Port Server Adapter
> > > >         Flags: fast devsel, IRQ 16
> > > >         Memory at fbc20000 (32-bit, non-prefetchable) [disabled] [size=128K]
> > > >         Memory at fb800000 (32-bit, non-prefetchable) [disabled] [size=4M]
> > > >         I/O ports at e020 [disabled] [size=32]
> > > >         Memory at fbc44000 (32-bit, non-prefetchable) [disabled] [size=16K]
> > > >         Expansion ROM at fb400000 [disabled] [size=4M]
> > > 
> > > BTW, I think this is the issue, the Expansion ROM. qemu-xen will
> > > allocate memory for it. Will have maybe have to find another way.
> > > qemu-trad those not seems to allocate memory, but I haven't been very
> > > far in trying to check that.
> > 
> > And indeed that is the case. The "Fix" below fixes it.
> > 
> > 
> > Based on that and this guest config:
> > disk = [ 'file:/mnt/lab/latest/root_image.iso,hdc:cdrom,r']
> > memory = 2048
> > boot="d"
> > maxvcpus=32
> > vcpus=1
> > serial='pty'
> > vnclisten="0.0.0.0"
> > name="latest"
> > vif = [ 'mac=00:0F:4B:00:00:68, bridge=switch' ]
> > pci = ["01:00.0"]
> > 
> > I can boot the guest.
> 
> And can you access the ROM from the guest ?
> 
> 
> Also, I have another patch, it will initialize the PCI ROM BAR like any
> other BAR. In this case, if qemu is envolved in the access to ROM, it
> will print an error, like it the case for other BAR. 
> 
> I tried to test it, but it was with an embedded VGA card. When I dump
> the ROM, I got the same one as the emulated card instead of the ROM from
> the device.
> 

This issue has been reported multiple times on the list, and discussed during the last couple of months on another threads,
mostly related to GPU passthru.

I think some clues were found recently about why you get the emulated rom instead of the actual device rom.
Sorry that I don't have the link available right now..

-- Pasi

> 
> diff --git a/hw/xen/xen_pt.c b/hw/xen/xen_pt.c
> index 6dd7a68..2bbdb6d 100644
> --- a/hw/xen/xen_pt.c
> +++ b/hw/xen/xen_pt.c
> @@ -440,8 +440,8 @@ static int xen_pt_register_regions(XenPCIPassthroughState *s)
>  
>          s->bases[PCI_ROM_SLOT].access.maddr = d->rom.base_addr;
>  
> -        memory_region_init_rom_device(&s->rom, OBJECT(s), NULL, NULL,
> -                                      "xen-pci-pt-rom", d->rom.size);
> +        memory_region_init_io(&s->rom, OBJECT(s), &ops, &s->dev,
> +                              "xen-pci-pt-rom", d->rom.size);
>          pci_register_bar(&s->dev, PCI_ROM_SLOT, PCI_BASE_ADDRESS_MEM_PREFETCH,
>                           &s->rom);
>  
> 
> -- 
> Anthony PERARD
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: qemu-xen-dir + PCI passthrough = BOOM

[Pasi Kärkkäinen]

On Thu, Jan 09, 2014 at 05:36:40PM +0200, Pasi Kärkkäinen wrote:
> > > 
> > > I can boot the guest.
> > 
> > And can you access the ROM from the guest ?
> > 
> > 
> > Also, I have another patch, it will initialize the PCI ROM BAR like any
> > other BAR. In this case, if qemu is envolved in the access to ROM, it
> > will print an error, like it the case for other BAR. 
> > 
> > I tried to test it, but it was with an embedded VGA card. When I dump
> > the ROM, I got the same one as the emulated card instead of the ROM from
> > the device.
> > 
> 
> This issue has been reported multiple times on the list, and discussed during the last couple of months on another threads,
> mostly related to GPU passthru.
> 
> I think some clues were found recently about why you get the emulated rom instead of the actual device rom.
> Sorry that I don't have the link available right now..
> 

Heh.. it seems it was Konrad in this very same thread :)

http://lists.xen.org/archives/html/xen-devel/2013-12/msg02837.html


-- Pasi 

Re: qemu-xen-dir + PCI passthrough = BOOM

[Sander Eikelenboom]

Thursday, January 9, 2014, 3:56:24 PM, you wrote:

> On Wed, Jan 08, 2014 at 02:44:51PM -0500, Konrad Rzeszutek Wilk wrote:
>> On Wed, Dec 18, 2013 at 02:48:24PM +0000, Anthony PERARD wrote:
>> > On Mon, Dec 16, 2013 at 10:08:16AM -0500, Konrad Rzeszutek Wilk wrote:
>> > > On Fri, Dec 06, 2013 at 04:03:10PM +0000, Wei Liu wrote:
>> > > > On Fri, Dec 06, 2013 at 04:00:18PM +0000, Wei Liu wrote:
>> > > > [...]
>> > > > > > Those Xen report something like:
>> > > > > > (XEN) page_alloc.c:1460:d0 Over-allocation for domain 46: 131329 >
>> > > > > > 131328
>> > > > > > (XEN) memory.c:132:d0 Could not allocate order=0 extent: id=46
>> > > > > > memflags=0 (62 of 64)
>> > > > > > 
>> > > > > > ?
>> > > > > > 
>> > > > > > (I tryied to reproduce the issue by simply add many emulated e1000 in
>> > > > > > QEMU :) )
>> > > > > > 
>> > 
>> > > -bash-4.1# lspci -s 01:00.0 -v 
>> > > 01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
>> > >         Subsystem: Intel Corporation Gigabit ET Dual Port Server Adapter
>> > >         Flags: fast devsel, IRQ 16
>> > >         Memory at fbc20000 (32-bit, non-prefetchable) [disabled] [size=128K]
>> > >         Memory at fb800000 (32-bit, non-prefetchable) [disabled] [size=4M]
>> > >         I/O ports at e020 [disabled] [size=32]
>> > >         Memory at fbc44000 (32-bit, non-prefetchable) [disabled] [size=16K]
>> > >         Expansion ROM at fb400000 [disabled] [size=4M]
>> > 
>> > BTW, I think this is the issue, the Expansion ROM. qemu-xen will
>> > allocate memory for it. Will have maybe have to find another way.
>> > qemu-trad those not seems to allocate memory, but I haven't been very
>> > far in trying to check that.
>> 
>> And indeed that is the case. The "Fix" below fixes it.
>> 
>> 
>> Based on that and this guest config:
>> disk = [ 'file:/mnt/lab/latest/root_image.iso,hdc:cdrom,r']
>> memory = 2048
>> boot="d"
>> maxvcpus=32
>> vcpus=1
>> serial='pty'
>> vnclisten="0.0.0.0"
>> name="latest"
>> vif = [ 'mac=00:0F:4B:00:00:68, bridge=switch' ]
>> pci = ["01:00.0"]
>> 
>> I can boot the guest.

> And can you access the ROM from the guest ?


> Also, I have another patch, it will initialize the PCI ROM BAR like any
> other BAR. In this case, if qemu is envolved in the access to ROM, it
> will print an error, like it the case for other BAR. 

> I tried to test it, but it was with an embedded VGA card. When I dump
> the ROM, I got the same one as the emulated card instead of the ROM from
> the device.

Ah this is what i reported earlier ..

If you would like it more funky .. use Dario's patches to be able to use vga="none" and use:
vga="none"
nographic=1
xen_platform_pci=1

And when you dump the rombar of the passedthrough vga card, you will end up with the iPXE rom of the emulated NIC.

So it is pointing at the first / last / a random rom ... but at least it doesn't seem directly tied to pointing at
another VGA rom (which was my first assumption some time ago).

When i go one step further .. by also disabling the xen platform pci device .. it doesn't boot any more.


Strange thing is that all the addresses in debug messages (host kernel .. hvmloader seabios qemu and guest kernel) for the rom bar seem to correspond when the translation is taken into account ..
so nothing obvious there ...

--
Sander

> diff --git a/hw/xen/xen_pt.c b/hw/xen/xen_pt.c
> index 6dd7a68..2bbdb6d 100644
> --- a/hw/xen/xen_pt.c
> +++ b/hw/xen/xen_pt.c
> @@ -440,8 +440,8 @@ static int xen_pt_register_regions(XenPCIPassthroughState *s)
>  
>          s->bases[PCI_ROM_SLOT].access.maddr = d->rom.base_addr;
>  
> -        memory_region_init_rom_device(&s->rom, OBJECT(s), NULL, NULL,
> -                                      "xen-pci-pt-rom", d->rom.size);
> +        memory_region_init_io(&s->rom, OBJECT(s), &ops, &s->dev,
> +                              "xen-pci-pt-rom", d->rom.size);
>          pci_register_bar(&s->dev, PCI_ROM_SLOT, PCI_BASE_ADDRESS_MEM_PREFETCH,
>                           &s->rom);
>  

Re: qemu-xen-dir + PCI passthrough = BOOM

[Sander Eikelenboom]

Friday, January 10, 2014, 12:54:43 AM, you wrote:


> Thursday, January 9, 2014, 3:56:24 PM, you wrote:

>> On Wed, Jan 08, 2014 at 02:44:51PM -0500, Konrad Rzeszutek Wilk wrote:
>>> On Wed, Dec 18, 2013 at 02:48:24PM +0000, Anthony PERARD wrote:
>>> > On Mon, Dec 16, 2013 at 10:08:16AM -0500, Konrad Rzeszutek Wilk wrote:
>>> > > On Fri, Dec 06, 2013 at 04:03:10PM +0000, Wei Liu wrote:
>>> > > > On Fri, Dec 06, 2013 at 04:00:18PM +0000, Wei Liu wrote:
>>> > > > [...]
>>> > > > > > Those Xen report something like:
>>> > > > > > (XEN) page_alloc.c:1460:d0 Over-allocation for domain 46: 131329 >
>>> > > > > > 131328
>>> > > > > > (XEN) memory.c:132:d0 Could not allocate order=0 extent: id=46
>>> > > > > > memflags=0 (62 of 64)
>>> > > > > > 
>>> > > > > > ?
>>> > > > > > 
>>> > > > > > (I tryied to reproduce the issue by simply add many emulated e1000 in
>>> > > > > > QEMU :) )
>>> > > > > > 
>>> > 
>>> > > -bash-4.1# lspci -s 01:00.0 -v 
>>> > > 01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
>>> > >         Subsystem: Intel Corporation Gigabit ET Dual Port Server Adapter
>>> > >         Flags: fast devsel, IRQ 16
>>> > >         Memory at fbc20000 (32-bit, non-prefetchable) [disabled] [size=128K]
>>> > >         Memory at fb800000 (32-bit, non-prefetchable) [disabled] [size=4M]
>>> > >         I/O ports at e020 [disabled] [size=32]
>>> > >         Memory at fbc44000 (32-bit, non-prefetchable) [disabled] [size=16K]
>>> > >         Expansion ROM at fb400000 [disabled] [size=4M]
>>> > 
>>> > BTW, I think this is the issue, the Expansion ROM. qemu-xen will
>>> > allocate memory for it. Will have maybe have to find another way.
>>> > qemu-trad those not seems to allocate memory, but I haven't been very
>>> > far in trying to check that.
>>> 
>>> And indeed that is the case. The "Fix" below fixes it.
>>> 
>>> 
>>> Based on that and this guest config:
>>> disk = [ 'file:/mnt/lab/latest/root_image.iso,hdc:cdrom,r']
>>> memory = 2048
>>> boot="d"
>>> maxvcpus=32
>>> vcpus=1
>>> serial='pty'
>>> vnclisten="0.0.0.0"
>>> name="latest"
>>> vif = [ 'mac=00:0F:4B:00:00:68, bridge=switch' ]
>>> pci = ["01:00.0"]
>>> 
>>> I can boot the guest.

>> And can you access the ROM from the guest ?


>> Also, I have another patch, it will initialize the PCI ROM BAR like any
>> other BAR. In this case, if qemu is envolved in the access to ROM, it
>> will print an error, like it the case for other BAR. 

>> I tried to test it, but it was with an embedded VGA card. When I dump
>> the ROM, I got the same one as the emulated card instead of the ROM from
>> the device.

> Ah this is what i reported earlier ..

> If you would like it more funky .. use Dario's patches to be able to use vga="none" and use:
> vga="none"
> nographic=1
> xen_platform_pci=1

> And when you dump the rombar of the passedthrough vga card, you will end up with the iPXE rom of the emulated NIC.

> So it is pointing at the first / last / a random rom ... but at least it doesn't seem directly tied to pointing at
> another VGA rom (which was my first assumption some time ago).

> When i go one step further .. by also disabling the xen platform pci device .. it doesn't boot any more.


> Strange thing is that all the addresses in debug messages (host kernel .. hvmloader seabios qemu and guest kernel) for the rom bar seem to correspond when the translation is taken into account ..
> so nothing obvious there ...

Perhaps i have to correct myself here ... with:
 vga="none"
 nographic=1
 xen_platform_pci=1

There seems to be a discrepancy between what the guest kernel reports at boot and the guest lspci output:
[    0.000000] e820: [mem 0x40000000-0xfbffffff] available for PCI devices
<snip>
[    1.453530] PCI host bridge to bus 0000:00
[    1.460018] pci_bus 0000:00: root bus resource [bus 00-ff]
[    1.466690] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7]
[    1.473355] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff]
[    1.480024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff]
[    1.490023] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xfbffffff]
[    1.500013] pci_bus 0000:00: scanning bus
[    1.500504] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000
[    1.500600] pci 0000:00:00.0: calling quirk_mmio_always_on+0x0/0x10
[    1.508515] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100
[    1.515216] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180
[    1.531714] pci 0000:00:01.1: reg 0x20: [io  0xc240-0xc24f]
[    1.540859] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000
[    1.540859] pci 0000:00:01.3: calling acpi_pm_check_blacklist+0x0/0x40
[    1.545776] pci 0000:00:01.3: calling quirk_piix4_acpi+0x0/0x140
[    1.545883] pci 0000:00:01.3: quirk: [io  0xb000-0xb03f] claimed by PIIX4 ACPI
[    1.546916] pci 0000:00:01.3: quirk: [io  0xb100-0xb10f] claimed by PIIX4 SMB
[    1.551173] pci 0000:00:01.3: calling pci_fixup_piix4_acpi+0x0/0x10
[    1.554237] pci 0000:00:02.0: [5853:0001] type 00 class 0xff8000
[    1.556666] pci 0000:00:02.0: reg 0x10: [io  0xc000-0xc0ff]
[    1.566373] pci 0000:00:02.0: reg 0x14: [mem 0xf0000000-0xf0ffffff pref]
[    1.596780] pci 0000:00:04.0: [1002:6759] type 00 class 0x030000
[    1.656719] pci 0000:00:04.0: reg 0x10: [mem 0xe0000000-0xefffffff 64bit pref]
[    1.693355] pci 0000:00:04.0: reg 0x18: [mem 0xf1060000-0xf107ffff 64bit]
[    1.743361] pci 0000:00:04.0: reg 0x20: [io  0xc100-0xc1ff]
[    1.826692] pci 0000:00:04.0: reg 0x30: [mem 0x00000000-0x0001ffff pref]
[    1.828722] pci 0000:00:04.0: supports D1 D2

00:04.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Turks [Radeon HD 6570] (prog-if 00 [VGA controller])
        Subsystem: PC Partner Limited Device e193
        Physical Slot: 4
        Flags: fast devsel, IRQ 32
        Memory at e0000000 (64-bit, prefetchable) [size=256M]
        Memory at f1060000 (64-bit, non-prefetchable) [size=128K]
        I/O ports at c100 [size=256]
        [virtual] Expansion ROM at f1000000 [disabled] [size=128K]
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [100] #1002


At least these 2 don't seem to match up:
[    1.826692] pci 0000:00:04.0: reg 0x30: [mem 0x00000000-0x0001ffff pref]

[virtual] Expansion ROM at f1000000 [disabled] [size=128K]

> --
> Sander

>> diff --git a/hw/xen/xen_pt.c b/hw/xen/xen_pt.c
>> index 6dd7a68..2bbdb6d 100644
>> --- a/hw/xen/xen_pt.c
>> +++ b/hw/xen/xen_pt.c
>> @@ -440,8 +440,8 @@ static int xen_pt_register_regions(XenPCIPassthroughState *s)
>>  
>>          s->bases[PCI_ROM_SLOT].access.maddr = d->rom.base_addr;
>>  
>> -        memory_region_init_rom_device(&s->rom, OBJECT(s), NULL, NULL,
>> -                                      "xen-pci-pt-rom", d->rom.size);
>> +        memory_region_init_io(&s->rom, OBJECT(s), &ops, &s->dev,
>> +                              "xen-pci-pt-rom", d->rom.size);
>>          pci_register_bar(&s->dev, PCI_ROM_SLOT, PCI_BASE_ADDRESS_MEM_PREFETCH,
>>                           &s->rom);
>>  

Re: qemu-xen-dir + PCI passthrough = BOOM

[Sander Eikelenboom]

Friday, January 10, 2014, 1:17:08 AM, you wrote:


> Friday, January 10, 2014, 12:54:43 AM, you wrote:


>> Thursday, January 9, 2014, 3:56:24 PM, you wrote:

>>> On Wed, Jan 08, 2014 at 02:44:51PM -0500, Konrad Rzeszutek Wilk wrote:
>>>> On Wed, Dec 18, 2013 at 02:48:24PM +0000, Anthony PERARD wrote:
>>>> > On Mon, Dec 16, 2013 at 10:08:16AM -0500, Konrad Rzeszutek Wilk wrote:
>>>> > > On Fri, Dec 06, 2013 at 04:03:10PM +0000, Wei Liu wrote:
>>>> > > > On Fri, Dec 06, 2013 at 04:00:18PM +0000, Wei Liu wrote:
>>>> > > > [...]
>>>> > > > > > Those Xen report something like:
>>>> > > > > > (XEN) page_alloc.c:1460:d0 Over-allocation for domain 46: 131329 >
>>>> > > > > > 131328
>>>> > > > > > (XEN) memory.c:132:d0 Could not allocate order=0 extent: id=46
>>>> > > > > > memflags=0 (62 of 64)
>>>> > > > > > 
>>>> > > > > > ?
>>>> > > > > > 
>>>> > > > > > (I tryied to reproduce the issue by simply add many emulated e1000 in
>>>> > > > > > QEMU :) )
>>>> > > > > > 
>>>> > 
>>>> > > -bash-4.1# lspci -s 01:00.0 -v 
>>>> > > 01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
>>>> > >         Subsystem: Intel Corporation Gigabit ET Dual Port Server Adapter
>>>> > >         Flags: fast devsel, IRQ 16
>>>> > >         Memory at fbc20000 (32-bit, non-prefetchable) [disabled] [size=128K]
>>>> > >         Memory at fb800000 (32-bit, non-prefetchable) [disabled] [size=4M]
>>>> > >         I/O ports at e020 [disabled] [size=32]
>>>> > >         Memory at fbc44000 (32-bit, non-prefetchable) [disabled] [size=16K]
>>>> > >         Expansion ROM at fb400000 [disabled] [size=4M]
>>>> > 
>>>> > BTW, I think this is the issue, the Expansion ROM. qemu-xen will
>>>> > allocate memory for it. Will have maybe have to find another way.
>>>> > qemu-trad those not seems to allocate memory, but I haven't been very
>>>> > far in trying to check that.
>>>> 
>>>> And indeed that is the case. The "Fix" below fixes it.
>>>> 
>>>> 
>>>> Based on that and this guest config:
>>>> disk = [ 'file:/mnt/lab/latest/root_image.iso,hdc:cdrom,r']
>>>> memory = 2048
>>>> boot="d"
>>>> maxvcpus=32
>>>> vcpus=1
>>>> serial='pty'
>>>> vnclisten="0.0.0.0"
>>>> name="latest"
>>>> vif = [ 'mac=00:0F:4B:00:00:68, bridge=switch' ]
>>>> pci = ["01:00.0"]
>>>> 
>>>> I can boot the guest.

>>> And can you access the ROM from the guest ?


>>> Also, I have another patch, it will initialize the PCI ROM BAR like any
>>> other BAR. In this case, if qemu is envolved in the access to ROM, it
>>> will print an error, like it the case for other BAR. 

>>> I tried to test it, but it was with an embedded VGA card. When I dump
>>> the ROM, I got the same one as the emulated card instead of the ROM from
>>> the device.

>> Ah this is what i reported earlier ..

>> If you would like it more funky .. use Dario's patches to be able to use vga="none" and use:
>> vga="none"
>> nographic=1
>> xen_platform_pci=1

>> And when you dump the rombar of the passedthrough vga card, you will end up with the iPXE rom of the emulated NIC.

>> So it is pointing at the first / last / a random rom ... but at least it doesn't seem directly tied to pointing at
>> another VGA rom (which was my first assumption some time ago).

>> When i go one step further .. by also disabling the xen platform pci device .. it doesn't boot any more.


>> Strange thing is that all the addresses in debug messages (host kernel .. hvmloader seabios qemu and guest kernel) for the rom bar seem to correspond when the translation is taken into account ..
>> so nothing obvious there ...

> Perhaps i have to correct myself here ... with:
>  vga="none"
>  nographic=1
>  xen_platform_pci=1

> There seems to be a discrepancy between what the guest kernel reports at boot and the guest lspci output:
> [    0.000000] e820: [mem 0x40000000-0xfbffffff] available for PCI devices
> <snip>
> [    1.453530] PCI host bridge to bus 0000:00
> [    1.460018] pci_bus 0000:00: root bus resource [bus 00-ff]
> [    1.466690] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7]
> [    1.473355] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff]
> [    1.480024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff]
> [    1.490023] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xfbffffff]
> [    1.500013] pci_bus 0000:00: scanning bus
> [    1.500504] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000
> [    1.500600] pci 0000:00:00.0: calling quirk_mmio_always_on+0x0/0x10
> [    1.508515] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100
> [    1.515216] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180
> [    1.531714] pci 0000:00:01.1: reg 0x20: [io  0xc240-0xc24f]
> [    1.540859] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000
> [    1.540859] pci 0000:00:01.3: calling acpi_pm_check_blacklist+0x0/0x40
> [    1.545776] pci 0000:00:01.3: calling quirk_piix4_acpi+0x0/0x140
> [    1.545883] pci 0000:00:01.3: quirk: [io  0xb000-0xb03f] claimed by PIIX4 ACPI
> [    1.546916] pci 0000:00:01.3: quirk: [io  0xb100-0xb10f] claimed by PIIX4 SMB
> [    1.551173] pci 0000:00:01.3: calling pci_fixup_piix4_acpi+0x0/0x10
> [    1.554237] pci 0000:00:02.0: [5853:0001] type 00 class 0xff8000
> [    1.556666] pci 0000:00:02.0: reg 0x10: [io  0xc000-0xc0ff]
> [    1.566373] pci 0000:00:02.0: reg 0x14: [mem 0xf0000000-0xf0ffffff pref]
> [    1.596780] pci 0000:00:04.0: [1002:6759] type 00 class 0x030000
> [    1.656719] pci 0000:00:04.0: reg 0x10: [mem 0xe0000000-0xefffffff 64bit pref]
> [    1.693355] pci 0000:00:04.0: reg 0x18: [mem 0xf1060000-0xf107ffff 64bit]
> [    1.743361] pci 0000:00:04.0: reg 0x20: [io  0xc100-0xc1ff]
> [    1.826692] pci 0000:00:04.0: reg 0x30: [mem 0x00000000-0x0001ffff pref]
> [    1.828722] pci 0000:00:04.0: supports D1 D2

> 00:04.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Turks [Radeon HD 6570] (prog-if 00 [VGA controller])
>         Subsystem: PC Partner Limited Device e193
>         Physical Slot: 4
>         Flags: fast devsel, IRQ 32
>         Memory at e0000000 (64-bit, prefetchable) [size=256M]
>         Memory at f1060000 (64-bit, non-prefetchable) [size=128K]
>         I/O ports at c100 [size=256]
>         [virtual] Expansion ROM at f1000000 [disabled] [size=128K]
>         Capabilities: [50] Power Management version 3
>         Capabilities: [58] Express Legacy Endpoint, MSI 00
>         Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
>         Capabilities: [100] #1002


> At least these 2 don't seem to match up:
> [    1.826692] pci 0000:00:04.0: reg 0x30: [mem 0x00000000-0x0001ffff pref]

> [virtual] Expansion ROM at f1000000 [disabled] [size=128K]

And from xl dmesg:

(d23) [2014-01-10 00:07:47] pci dev 03:0 bar 30 size 000040000: 0f1000000

which is .. probably the NIC ... but which is mysteriously never shown in lspci (thought i had reported that earlier as well):

lspci -v
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
        Subsystem: Red Hat, Inc Qemu virtual machine
        Flags: bus master, fast devsel, latency 0

00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
        Subsystem: Red Hat, Inc Qemu virtual machine
        Physical Slot: 1
        Flags: bus master, medium devsel, latency 0

00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] (prog-if 80 [Master])
        Subsystem: Red Hat, Inc Qemu virtual machine
        Physical Slot: 1
        Flags: bus master, medium devsel, latency 0
        [virtual] Memory at 000001f0 (32-bit, non-prefetchable) [size=8]
        [virtual] Memory at 000003f0 (type 3, non-prefetchable) [size=1]
        [virtual] Memory at 00000170 (32-bit, non-prefetchable) [size=8]
        [virtual] Memory at 00000370 (type 3, non-prefetchable) [size=1]
        I/O ports at c240 [size=16]
        Kernel driver in use: PIIX_IDE

00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
        Subsystem: Red Hat, Inc Qemu virtual machine
        Physical Slot: 1
        Flags: bus master, medium devsel, latency 0, IRQ 9

00:02.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01)
        Subsystem: XenSource, Inc. Xen Platform Device
        Physical Slot: 2
        Flags: bus master, fast devsel, latency 0, IRQ 24
        I/O ports at c000 [size=256]
        Memory at f0000000 (32-bit, prefetchable) [size=16M]
        Kernel driver in use: xen-platform-pci

00:04.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Turks [Radeon HD 6570] (prog-if 00 [VGA controller])
        Subsystem: PC Partner Limited Device e193
        Physical Slot: 4
        Flags: fast devsel, IRQ 32
        Memory at e0000000 (64-bit, prefetchable) [size=256M]
        Memory at f1060000 (64-bit, non-prefetchable) [size=128K]
        I/O ports at c100 [size=256]
        [virtual] Expansion ROM at f1000000 [disabled] [size=128K]
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [100] #1002


which is defined in the guest config as:
vif = [ 'bridge=xen_bridge,ip=192.168.1.44,mac=00:16:3A:C6:76:65, model=e1000'  ]


>> --
>> Sander

>>> diff --git a/hw/xen/xen_pt.c b/hw/xen/xen_pt.c
>>> index 6dd7a68..2bbdb6d 100644
>>> --- a/hw/xen/xen_pt.c
>>> +++ b/hw/xen/xen_pt.c
>>> @@ -440,8 +440,8 @@ static int xen_pt_register_regions(XenPCIPassthroughState *s)
>>>  
>>>          s->bases[PCI_ROM_SLOT].access.maddr = d->rom.base_addr;
>>>  
>>> -        memory_region_init_rom_device(&s->rom, OBJECT(s), NULL, NULL,
>>> -                                      "xen-pci-pt-rom", d->rom.size);
>>> +        memory_region_init_io(&s->rom, OBJECT(s), &ops, &s->dev,
>>> +                              "xen-pci-pt-rom", d->rom.size);
>>>          pci_register_bar(&s->dev, PCI_ROM_SLOT, PCI_BASE_ADDRESS_MEM_PREFETCH,
>>>                           &s->rom);
>>>  

Re: qemu-xen-dir + PCI passthrough = BOOM

[Konrad Rzeszutek Wilk]

On Thu, Jan 09, 2014 at 02:56:24PM +0000, Anthony PERARD wrote:
> On Wed, Jan 08, 2014 at 02:44:51PM -0500, Konrad Rzeszutek Wilk wrote:
> > On Wed, Dec 18, 2013 at 02:48:24PM +0000, Anthony PERARD wrote:
> > > On Mon, Dec 16, 2013 at 10:08:16AM -0500, Konrad Rzeszutek Wilk wrote:
> > > > On Fri, Dec 06, 2013 at 04:03:10PM +0000, Wei Liu wrote:
> > > > > On Fri, Dec 06, 2013 at 04:00:18PM +0000, Wei Liu wrote:
> > > > > [...]
> > > > > > > Those Xen report something like:
> > > > > > > (XEN) page_alloc.c:1460:d0 Over-allocation for domain 46: 131329 >
> > > > > > > 131328
> > > > > > > (XEN) memory.c:132:d0 Could not allocate order=0 extent: id=46
> > > > > > > memflags=0 (62 of 64)
> > > > > > > 
> > > > > > > ?
> > > > > > > 
> > > > > > > (I tryied to reproduce the issue by simply add many emulated e1000 in
> > > > > > > QEMU :) )
> > > > > > > 
> > > 
> > > > -bash-4.1# lspci -s 01:00.0 -v 
> > > > 01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
> > > >         Subsystem: Intel Corporation Gigabit ET Dual Port Server Adapter
> > > >         Flags: fast devsel, IRQ 16
> > > >         Memory at fbc20000 (32-bit, non-prefetchable) [disabled] [size=128K]
> > > >         Memory at fb800000 (32-bit, non-prefetchable) [disabled] [size=4M]
> > > >         I/O ports at e020 [disabled] [size=32]
> > > >         Memory at fbc44000 (32-bit, non-prefetchable) [disabled] [size=16K]
> > > >         Expansion ROM at fb400000 [disabled] [size=4M]
> > > 
> > > BTW, I think this is the issue, the Expansion ROM. qemu-xen will
> > > allocate memory for it. Will have maybe have to find another way.
> > > qemu-trad those not seems to allocate memory, but I haven't been very
> > > far in trying to check that.
> > 
> > And indeed that is the case. The "Fix" below fixes it.
> > 
> > 
> > Based on that and this guest config:
> > disk = [ 'file:/mnt/lab/latest/root_image.iso,hdc:cdrom,r']
> > memory = 2048
> > boot="d"
> > maxvcpus=32
> > vcpus=1
> > serial='pty'
> > vnclisten="0.0.0.0"
> > name="latest"
> > vif = [ 'mac=00:0F:4B:00:00:68, bridge=switch' ]
> > pci = ["01:00.0"]
> > 
> > I can boot the guest.
> 
> And can you access the ROM from the guest ?

I hadn't tried it. This is with a NIC and I just wanted to see if it
could do PCI passthrough without using the Option ROM.
> 
> 
> Also, I have another patch, it will initialize the PCI ROM BAR like any
> other BAR. In this case, if qemu is envolved in the access to ROM, it
> will print an error, like it the case for other BAR. 
> 
> I tried to test it, but it was with an embedded VGA card. When I dump
> the ROM, I got the same one as the emulated card instead of the ROM from
> the device.

Oddly enough for me with your patch the NIC's BIOS was invoked and
it tried to PXE boot:

(d1) [2014-01-10 03:20:29] Running option rom at ca00:0003

(d1) [2014-01-10 03:20:47] Booting from DVD/CD...
(d1) [2014-01-10 03:20:47] Booting from 0000:7c00
..
and I did see the PXE boot menu in my guest - so even
better!

I have not yet done the GPU - this issue was preventing me from using
qemu-xen as it would always blow up before SeaBIOS was in the picture.

If you would like to put 'Reported-and-Tested-by: Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com>' please do.

Thank you!
> 
> 
> diff --git a/hw/xen/xen_pt.c b/hw/xen/xen_pt.c
> index 6dd7a68..2bbdb6d 100644
> --- a/hw/xen/xen_pt.c
> +++ b/hw/xen/xen_pt.c
> @@ -440,8 +440,8 @@ static int xen_pt_register_regions(XenPCIPassthroughState *s)
>  
>          s->bases[PCI_ROM_SLOT].access.maddr = d->rom.base_addr;
>  
> -        memory_region_init_rom_device(&s->rom, OBJECT(s), NULL, NULL,
> -                                      "xen-pci-pt-rom", d->rom.size);
> +        memory_region_init_io(&s->rom, OBJECT(s), &ops, &s->dev,
> +                              "xen-pci-pt-rom", d->rom.size);
>          pci_register_bar(&s->dev, PCI_ROM_SLOT, PCI_BASE_ADDRESS_MEM_PREFETCH,
>                           &s->rom);
>  
> 
> -- 
> Anthony PERARD

Re: qemu-xen-dir + PCI passthrough = BOOM

[Anthony PERARD]

On Thu, Jan 09, 2014 at 10:28:47PM -0500, Konrad Rzeszutek Wilk wrote:
> On Thu, Jan 09, 2014 at 02:56:24PM +0000, Anthony PERARD wrote:
> > On Wed, Jan 08, 2014 at 02:44:51PM -0500, Konrad Rzeszutek Wilk wrote:
> > > On Wed, Dec 18, 2013 at 02:48:24PM +0000, Anthony PERARD wrote:
> > > > On Mon, Dec 16, 2013 at 10:08:16AM -0500, Konrad Rzeszutek Wilk wrote:
> > > > > On Fri, Dec 06, 2013 at 04:03:10PM +0000, Wei Liu wrote:
> > > > > > On Fri, Dec 06, 2013 at 04:00:18PM +0000, Wei Liu wrote:
> > > > > > [...]
> > > > > > > > Those Xen report something like:
> > > > > > > > (XEN) page_alloc.c:1460:d0 Over-allocation for domain 46: 131329 >
> > > > > > > > 131328
> > > > > > > > (XEN) memory.c:132:d0 Could not allocate order=0 extent: id=46
> > > > > > > > memflags=0 (62 of 64)
> > > > > > > > 
> > > > > > > > ?
> > > > > > > > 
> > > > > > > > (I tryied to reproduce the issue by simply add many emulated e1000 in
> > > > > > > > QEMU :) )
> > > > > > > > 
> > > > 
> > > > > -bash-4.1# lspci -s 01:00.0 -v 
> > > > > 01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
> > > > >         Subsystem: Intel Corporation Gigabit ET Dual Port Server Adapter
> > > > >         Flags: fast devsel, IRQ 16
> > > > >         Memory at fbc20000 (32-bit, non-prefetchable) [disabled] [size=128K]
> > > > >         Memory at fb800000 (32-bit, non-prefetchable) [disabled] [size=4M]
> > > > >         I/O ports at e020 [disabled] [size=32]
> > > > >         Memory at fbc44000 (32-bit, non-prefetchable) [disabled] [size=16K]
> > > > >         Expansion ROM at fb400000 [disabled] [size=4M]
> > > > 
> > > > BTW, I think this is the issue, the Expansion ROM. qemu-xen will
> > > > allocate memory for it. Will have maybe have to find another way.
> > > > qemu-trad those not seems to allocate memory, but I haven't been very
> > > > far in trying to check that.
> > > 
> > > And indeed that is the case. The "Fix" below fixes it.
> > > 
> > > 
> > > Based on that and this guest config:
> > > disk = [ 'file:/mnt/lab/latest/root_image.iso,hdc:cdrom,r']
> > > memory = 2048
> > > boot="d"
> > > maxvcpus=32
> > > vcpus=1
> > > serial='pty'
> > > vnclisten="0.0.0.0"
> > > name="latest"
> > > vif = [ 'mac=00:0F:4B:00:00:68, bridge=switch' ]
> > > pci = ["01:00.0"]
> > > 
> > > I can boot the guest.
> > 
> > And can you access the ROM from the guest ?
> 
> I hadn't tried it. This is with a NIC and I just wanted to see if it
> could do PCI passthrough without using the Option ROM.
> > 
> > 
> > Also, I have another patch, it will initialize the PCI ROM BAR like any
> > other BAR. In this case, if qemu is envolved in the access to ROM, it
> > will print an error, like it the case for other BAR. 
> > 
> > I tried to test it, but it was with an embedded VGA card. When I dump
> > the ROM, I got the same one as the emulated card instead of the ROM from
> > the device.
> 
> Oddly enough for me with your patch the NIC's BIOS was invoked and
> it tried to PXE boot:
> 
> (d1) [2014-01-10 03:20:29] Running option rom at ca00:0003
> 
> (d1) [2014-01-10 03:20:47] Booting from DVD/CD...
> (d1) [2014-01-10 03:20:47] Booting from 0000:7c00
> ..
> and I did see the PXE boot menu in my guest - so even
> better!

Perfect, look like it is the fix for PCI passthrough.

> I have not yet done the GPU - this issue was preventing me from using
> qemu-xen as it would always blow up before SeaBIOS was in the picture.
> 
> If you would like to put 'Reported-and-Tested-by: Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com>' please do.

Will do.

> Thank you!
> > 
> > 
> > diff --git a/hw/xen/xen_pt.c b/hw/xen/xen_pt.c
> > index 6dd7a68..2bbdb6d 100644
> > --- a/hw/xen/xen_pt.c
> > +++ b/hw/xen/xen_pt.c
> > @@ -440,8 +440,8 @@ static int xen_pt_register_regions(XenPCIPassthroughState *s)
> >  
> >          s->bases[PCI_ROM_SLOT].access.maddr = d->rom.base_addr;
> >  
> > -        memory_region_init_rom_device(&s->rom, OBJECT(s), NULL, NULL,
> > -                                      "xen-pci-pt-rom", d->rom.size);
> > +        memory_region_init_io(&s->rom, OBJECT(s), &ops, &s->dev,
> > +                              "xen-pci-pt-rom", d->rom.size);
> >          pci_register_bar(&s->dev, PCI_ROM_SLOT, PCI_BASE_ADDRESS_MEM_PREFETCH,
> >                           &s->rom);
> >  
> > 
> > -- 
> > Anthony PERARD

-- 
Anthony PERARD

Re: qemu-xen-dir + PCI passthrough = BOOM

[Sander Eikelenboom]

Friday, January 10, 2014, 4:19:15 PM, you wrote:

> On Thu, Jan 09, 2014 at 10:28:47PM -0500, Konrad Rzeszutek Wilk wrote:
>> On Thu, Jan 09, 2014 at 02:56:24PM +0000, Anthony PERARD wrote:
>> > On Wed, Jan 08, 2014 at 02:44:51PM -0500, Konrad Rzeszutek Wilk wrote:
>> > > On Wed, Dec 18, 2013 at 02:48:24PM +0000, Anthony PERARD wrote:
>> > > > On Mon, Dec 16, 2013 at 10:08:16AM -0500, Konrad Rzeszutek Wilk wrote:
>> > > > > On Fri, Dec 06, 2013 at 04:03:10PM +0000, Wei Liu wrote:
>> > > > > > On Fri, Dec 06, 2013 at 04:00:18PM +0000, Wei Liu wrote:
>> > > > > > [...]
>> > > > > > > > Those Xen report something like:
>> > > > > > > > (XEN) page_alloc.c:1460:d0 Over-allocation for domain 46: 131329 >
>> > > > > > > > 131328
>> > > > > > > > (XEN) memory.c:132:d0 Could not allocate order=0 extent: id=46
>> > > > > > > > memflags=0 (62 of 64)
>> > > > > > > > 
>> > > > > > > > ?
>> > > > > > > > 
>> > > > > > > > (I tryied to reproduce the issue by simply add many emulated e1000 in
>> > > > > > > > QEMU :) )
>> > > > > > > > 
>> > > > 
>> > > > > -bash-4.1# lspci -s 01:00.0 -v 
>> > > > > 01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
>> > > > >         Subsystem: Intel Corporation Gigabit ET Dual Port Server Adapter
>> > > > >         Flags: fast devsel, IRQ 16
>> > > > >         Memory at fbc20000 (32-bit, non-prefetchable) [disabled] [size=128K]
>> > > > >         Memory at fb800000 (32-bit, non-prefetchable) [disabled] [size=4M]
>> > > > >         I/O ports at e020 [disabled] [size=32]
>> > > > >         Memory at fbc44000 (32-bit, non-prefetchable) [disabled] [size=16K]
>> > > > >         Expansion ROM at fb400000 [disabled] [size=4M]
>> > > > 
>> > > > BTW, I think this is the issue, the Expansion ROM. qemu-xen will
>> > > > allocate memory for it. Will have maybe have to find another way.
>> > > > qemu-trad those not seems to allocate memory, but I haven't been very
>> > > > far in trying to check that.
>> > > 
>> > > And indeed that is the case. The "Fix" below fixes it.
>> > > 
>> > > 
>> > > Based on that and this guest config:
>> > > disk = [ 'file:/mnt/lab/latest/root_image.iso,hdc:cdrom,r']
>> > > memory = 2048
>> > > boot="d"
>> > > maxvcpus=32
>> > > vcpus=1
>> > > serial='pty'
>> > > vnclisten="0.0.0.0"
>> > > name="latest"
>> > > vif = [ 'mac=00:0F:4B:00:00:68, bridge=switch' ]
>> > > pci = ["01:00.0"]
>> > > 
>> > > I can boot the guest.
>> > 
>> > And can you access the ROM from the guest ?
>> 
>> I hadn't tried it. This is with a NIC and I just wanted to see if it
>> could do PCI passthrough without using the Option ROM.
>> > 
>> > 
>> > Also, I have another patch, it will initialize the PCI ROM BAR like any
>> > other BAR. In this case, if qemu is envolved in the access to ROM, it
>> > will print an error, like it the case for other BAR. 
>> > 
>> > I tried to test it, but it was with an embedded VGA card. When I dump
>> > the ROM, I got the same one as the emulated card instead of the ROM from
>> > the device.
>> 
>> Oddly enough for me with your patch the NIC's BIOS was invoked and
>> it tried to PXE boot:
>> 
>> (d1) [2014-01-10 03:20:29] Running option rom at ca00:0003
>> 
>> (d1) [2014-01-10 03:20:47] Booting from DVD/CD...
>> (d1) [2014-01-10 03:20:47] Booting from 0000:7c00
>> ..
>> and I did see the PXE boot menu in my guest - so even
>> better!

> Perfect, look like it is the fix for PCI passthrough.

Hi Konrad,

Are you sure it's the rom of the NIC, and not the iPXE rom from the emulated device that
gets run ?

With this patch and VGA devices it still points to another rom for me.
(it looks like it is pointing to the rom of the device with a BDF just one lower than the passed through one)

Do you by any chance know if there is a difference in how lspci and the linux kernel scan / list pci devices ?
(for example one by reading acpi tables .. the other one by real probing .. ?)

--
Sander


>> I have not yet done the GPU - this issue was preventing me from using
>> qemu-xen as it would always blow up before SeaBIOS was in the picture.
>> 
>> If you would like to put 'Reported-and-Tested-by: Konrad Rzeszutek Wilk
>> <konrad.wilk@oracle.com>' please do.

> Will do.

>> Thank you!
>> > 
>> > 
>> > diff --git a/hw/xen/xen_pt.c b/hw/xen/xen_pt.c
>> > index 6dd7a68..2bbdb6d 100644
>> > --- a/hw/xen/xen_pt.c
>> > +++ b/hw/xen/xen_pt.c
>> > @@ -440,8 +440,8 @@ static int xen_pt_register_regions(XenPCIPassthroughState *s)
>> >  
>> >          s->bases[PCI_ROM_SLOT].access.maddr = d->rom.base_addr;
>> >  
>> > -        memory_region_init_rom_device(&s->rom, OBJECT(s), NULL, NULL,
>> > -                                      "xen-pci-pt-rom", d->rom.size);
>> > +        memory_region_init_io(&s->rom, OBJECT(s), &ops, &s->dev,
>> > +                              "xen-pci-pt-rom", d->rom.size);
>> >          pci_register_bar(&s->dev, PCI_ROM_SLOT, PCI_BASE_ADDRESS_MEM_PREFETCH,
>> >                           &s->rom);
>> >  
>> > 
>> > -- 
>> > Anthony PERARD

Re: qemu-xen-dir + PCI passthrough = BOOM

[Anthony PERARD]

On Fri, Jan 10, 2014 at 05:05:44PM +0100, Sander Eikelenboom wrote:
> With this patch and VGA devices it still points to another rom for me.
> (it looks like it is pointing to the rom of the device with a BDF just one lower than the passed through one)

I do think VGA devices are a particular case. hvmloader or SeaBIOS might
"replace" the ROM of the card.

-- 
Anthony PERARD

Re: qemu-xen-dir + PCI passthrough = BOOM

[Konrad Rzeszutek Wilk]

> >> (d1) [2014-01-10 03:20:29] Running option rom at ca00:0003
> >> 
> >> (d1) [2014-01-10 03:20:47] Booting from DVD/CD...
> >> (d1) [2014-01-10 03:20:47] Booting from 0000:7c00
> >> ..
> >> and I did see the PXE boot menu in my guest - so even
> >> better!
> 
> > Perfect, look like it is the fix for PCI passthrough.
> 
> Hi Konrad,
> 
> Are you sure it's the rom of the NIC, and not the iPXE rom from the emulated device that
> gets run ?

Yes. I double checked that the MAC address that was given an DHCP
address was indeed for the physical hardware. And it was.

> 
> With this patch and VGA devices it still points to another rom for me.
> (it looks like it is pointing to the rom of the device with a BDF just one lower than the passed through one)

That all sounds to me like a bug in QEMU which constructs
the 'world'. Then 'hvmloader' and the kernel ingest this to
create their view of what the PCI configuration/slots/etc should look like.
They use 'inb' and 'outb' instructions on the 0xcf8/0xcfc port.

Perhaps the 'vga=none' is having an hard time dealing with 'no-VGA-but-wait-there
is-and-PT-VGA'?

Just to make sure I am not forgetting a crucial fact - if you don't
have vga=none, does the lspci output look sane?

> 
> Do you by any chance know if there is a difference in how lspci and the linux kernel scan / list pci devices ?
> (for example one by reading acpi tables .. the other one by real probing .. ?)

The kernel and hvmloader all use the 'inb' and 'outb' to figure out
what the PCI space looks like.

'lspci' uses /sysfs. Thought if you use '-xxxx' I think it also does
'inb' and 'outb'. 

So it all seems to point to QEMU. 
> 
> --
> Sander
> 
> 
> >> I have not yet done the GPU - this issue was preventing me from using
> >> qemu-xen as it would always blow up before SeaBIOS was in the picture.
> >> 
> >> If you would like to put 'Reported-and-Tested-by: Konrad Rzeszutek Wilk
> >> <konrad.wilk@oracle.com>' please do.
> 
> > Will do.
> 
> >> Thank you!
> >> > 
> >> > 
> >> > diff --git a/hw/xen/xen_pt.c b/hw/xen/xen_pt.c
> >> > index 6dd7a68..2bbdb6d 100644
> >> > --- a/hw/xen/xen_pt.c
> >> > +++ b/hw/xen/xen_pt.c
> >> > @@ -440,8 +440,8 @@ static int xen_pt_register_regions(XenPCIPassthroughState *s)
> >> >  
> >> >          s->bases[PCI_ROM_SLOT].access.maddr = d->rom.base_addr;
> >> >  
> >> > -        memory_region_init_rom_device(&s->rom, OBJECT(s), NULL, NULL,
> >> > -                                      "xen-pci-pt-rom", d->rom.size);
> >> > +        memory_region_init_io(&s->rom, OBJECT(s), &ops, &s->dev,
> >> > +                              "xen-pci-pt-rom", d->rom.size);
> >> >          pci_register_bar(&s->dev, PCI_ROM_SLOT, PCI_BASE_ADDRESS_MEM_PREFETCH,
> >> >                           &s->rom);
> >> >  
> >> > 
> >> > -- 
> >> > Anthony PERARD
> 
> 

Re: qemu-xen-dir + PCI passthrough = BOOM

[Sander Eikelenboom]

Friday, January 10, 2014, 5:25:35 PM, you wrote:

> On Fri, Jan 10, 2014 at 05:05:44PM +0100, Sander Eikelenboom wrote:
>> With this patch and VGA devices it still points to another rom for me.
>> (it looks like it is pointing to the rom of the device with a BDF just one lower than the passed through one)

> I do think VGA devices are a particular case. hvmloader or SeaBIOS might
> "replace" the ROM of the card.

Any way / patch to find out ?

It would be quite silly if it replaces the vga rom with a nic rom.

If that only happens for vga cards .. it would need to check for device class,
would seem very strange to replace it with another rom and not check for class there.

Re: qemu-xen-dir + PCI passthrough = BOOM

[Sander Eikelenboom]

Friday, January 10, 2014, 5:35:44 PM, you wrote:

>> >> (d1) [2014-01-10 03:20:29] Running option rom at ca00:0003
>> >> 
>> >> (d1) [2014-01-10 03:20:47] Booting from DVD/CD...
>> >> (d1) [2014-01-10 03:20:47] Booting from 0000:7c00
>> >> ..
>> >> and I did see the PXE boot menu in my guest - so even
>> >> better!
>> 
>> > Perfect, look like it is the fix for PCI passthrough.
>> 
>> Hi Konrad,
>> 
>> Are you sure it's the rom of the NIC, and not the iPXE rom from the emulated device that
>> gets run ?

> Yes. I double checked that the MAC address that was given an DHCP
> address was indeed for the physical hardware. And it was.

OK

>> 
>> With this patch and VGA devices it still points to another rom for me.
>> (it looks like it is pointing to the rom of the device with a BDF just one lower than the passed through one)

> That all sounds to me like a bug in QEMU which constructs
> the 'world'. Then 'hvmloader' and the kernel ingest this to
> create their view of what the PCI configuration/slots/etc should look like.
> They use 'inb' and 'outb' instructions on the 0xcf8/0xcfc port.

> Perhaps the 'vga=none' is having an hard time dealing with 'no-VGA-but-wait-there
> is-and-PT-VGA'?

> Just to make sure I am not forgetting a crucial fact - if you don't
> have vga=none, does the lspci output look sane?

It does .. except when using an PV NIC, that one doesn't show in lspci, but it does occupy a slot (the numbers are not
consecutive anymore, one is "hidden"), but the PV nic itself is working ok.
This doesn't seem to happen for disk so that seems a bit strange to me ..

Apart from that it looks sane.

But I'm rebuilding everything now from scratch and will try again .. just too damned many parameters :-)

Will see if i can make a complete post again with all data, although the previous time i tried that,
it was a little bit to intimidating i guess.

>> 
>> Do you by any chance know if there is a difference in how lspci and the linux kernel scan / list pci devices ?
>> (for example one by reading acpi tables .. the other one by real probing .. ?)

> The kernel and hvmloader all use the 'inb' and 'outb' to figure out
> what the PCI space looks like.

> 'lspci' uses /sysfs. Thought if you use '-xxxx' I think it also does
> 'inb' and 'outb'.

Hmm ok .. so i would expect that dumping with the:
echo 1 > rom; cat rom > rom.bin; echo 0 > rom;
sequence in /sys/bus/pci/devices/<BDF> would do everything according to the addresses that lspci gives for the rombar ..
and when i do that the resulting rom.bin differs according to the emulated devices i put it.

The devices that have a rom in the guest are always in the order (with there BDF):
NIC
Emulated VGA
Passthroughed VGA

When all enabled, i end up with the rom of the emulated VGA
When i disable that with vga="none", i end up with the rom of the NIC
When i also disable that .. by not specifying any vif and using xen_platform_pci=0 ..
     the kernel complains when trying to dump that it's not a valid rom (and when forced too it's all zero's so it's
     right because it can't find the start signature of a rom)

Is there an easy tool to dump the content of arbitrary mem addresses ?
(except enabling the /dev/kmem and trying to do some voodoo calculations and use "dd" ?)

Hmm when i make a core dump using "xl dump-core" would/should that also contain the content of the passed throughed rom ?
Then at least i could search that dump for the strings of the real vga rom of the passedtroughed card  and see if it is there .. perhaps on some other
address as expected .. or at least know it isn't.

Because at the moment i'm not very well in progressing with ruling things out (which seems to be the only method),
i only have some symptoms that differ. Wish i had another NIC device with a rom to see if that does work.


> So it all seems to point to QEMU.
>> 
>> --
>> Sander
>> 
>> 
>> >> I have not yet done the GPU - this issue was preventing me from using
>> >> qemu-xen as it would always blow up before SeaBIOS was in the picture.
>> >> 
>> >> If you would like to put 'Reported-and-Tested-by: Konrad Rzeszutek Wilk
>> >> <konrad.wilk@oracle.com>' please do.
>> 
>> > Will do.
>> 
>> >> Thank you!
>> >> > 
>> >> > 
>> >> > diff --git a/hw/xen/xen_pt.c b/hw/xen/xen_pt.c
>> >> > index 6dd7a68..2bbdb6d 100644
>> >> > --- a/hw/xen/xen_pt.c
>> >> > +++ b/hw/xen/xen_pt.c
>> >> > @@ -440,8 +440,8 @@ static int xen_pt_register_regions(XenPCIPassthroughState *s)
>> >> >  
>> >> >          s->bases[PCI_ROM_SLOT].access.maddr = d->rom.base_addr;
>> >> >  
>> >> > -        memory_region_init_rom_device(&s->rom, OBJECT(s), NULL, NULL,
>> >> > -                                      "xen-pci-pt-rom", d->rom.size);
>> >> > +        memory_region_init_io(&s->rom, OBJECT(s), &ops, &s->dev,
>> >> > +                              "xen-pci-pt-rom", d->rom.size);
>> >> >          pci_register_bar(&s->dev, PCI_ROM_SLOT, PCI_BASE_ADDRESS_MEM_PREFETCH,
>> >> >                           &s->rom);
>> >> >  
>> >> > 
>> >> > -- 
>> >> > Anthony PERARD
>> 
>> 

Re: qemu-xen-dir + PCI passthrough = BOOM

[Konrad Rzeszutek Wilk]

On Fri, Jan 10, 2014 at 06:38:18PM +0100, Sander Eikelenboom wrote:
> 
> Friday, January 10, 2014, 5:35:44 PM, you wrote:
> 
> >> >> (d1) [2014-01-10 03:20:29] Running option rom at ca00:0003
> >> >> 
> >> >> (d1) [2014-01-10 03:20:47] Booting from DVD/CD...
> >> >> (d1) [2014-01-10 03:20:47] Booting from 0000:7c00
> >> >> ..
> >> >> and I did see the PXE boot menu in my guest - so even
> >> >> better!
> >> 
> >> > Perfect, look like it is the fix for PCI passthrough.
> >> 
> >> Hi Konrad,
> >> 
> >> Are you sure it's the rom of the NIC, and not the iPXE rom from the emulated device that
> >> gets run ?
> 
> > Yes. I double checked that the MAC address that was given an DHCP
> > address was indeed for the physical hardware. And it was.
> 
> OK
> 
> >> 
> >> With this patch and VGA devices it still points to another rom for me.
> >> (it looks like it is pointing to the rom of the device with a BDF just one lower than the passed through one)
> 
> > That all sounds to me like a bug in QEMU which constructs
> > the 'world'. Then 'hvmloader' and the kernel ingest this to
> > create their view of what the PCI configuration/slots/etc should look like.
> > They use 'inb' and 'outb' instructions on the 0xcf8/0xcfc port.
> 
> > Perhaps the 'vga=none' is having an hard time dealing with 'no-VGA-but-wait-there
> > is-and-PT-VGA'?
> 
> > Just to make sure I am not forgetting a crucial fact - if you don't
> > have vga=none, does the lspci output look sane?
> 
> It does .. except when using an PV NIC, that one doesn't show in lspci, but it does occupy a slot (the numbers are not
> consecutive anymore, one is "hidden"), but the PV nic itself is working ok.
Right, because your Intel emulated gets 'unplugged' (disappears)
when the Xen PV one kicks in.

> This doesn't seem to happen for disk so that seems a bit strange to me ..

That might be due to the CD-ROM option. The disk should
nonetless disappear (/dev/hda).

> 
> Apart from that it looks sane.
> 
> But I'm rebuilding everything now from scratch and will try again .. just too damned many parameters :-)
> 
> Will see if i can make a complete post again with all data, although the previous time i tried that,
> it was a little bit to intimidating i guess.
> 
> >> 
> >> Do you by any chance know if there is a difference in how lspci and the linux kernel scan / list pci devices ?
> >> (for example one by reading acpi tables .. the other one by real probing .. ?)
> 
> > The kernel and hvmloader all use the 'inb' and 'outb' to figure out
> > what the PCI space looks like.
> 
> > 'lspci' uses /sysfs. Thought if you use '-xxxx' I think it also does
> > 'inb' and 'outb'.
> 
> Hmm ok .. so i would expect that dumping with the:
> echo 1 > rom; cat rom > rom.bin; echo 0 > rom;
> sequence in /sys/bus/pci/devices/<BDF> would do everything according to the addresses that lspci gives for the rombar ..
> and when i do that the resulting rom.bin differs according to the emulated devices i put it.
> 
> The devices that have a rom in the guest are always in the order (with there BDF):
> NIC
> Emulated VGA
> Passthroughed VGA
> 
> When all enabled, i end up with the rom of the emulated VGA
> When i disable that with vga="none", i end up with the rom of the NIC
> When i also disable that .. by not specifying any vif and using xen_platform_pci=0 ..
>      the kernel complains when trying to dump that it's not a valid rom (and when forced too it's all zero's so it's
>      right because it can't find the start signature of a rom)
> 

Ok,so the passthrough ROM is definitly not showing up. Which is
odd, b/c it does show up for the physical NIC.

If you do the 'echo 1' .. rune in dom0, can you extract your
VGA (Radeon) BIOS? Perhaps the ROM extraction part for video
cards is different?

> Is there an easy tool to dump the content of arbitrary mem addresses ?
> (except enabling the /dev/kmem and trying to do some voodoo calculations and use "dd" ?)

That is the easiest.
> 
> Hmm when i make a core dump using "xl dump-core" would/should that also contain the content of the passed throughed rom ?

I think not. As the ROM is not RAM it should not include that.

> Then at least i could search that dump for the strings of the real vga rom of the passedtroughed card  and see if it is there .. perhaps on some other
> address as expected .. or at least know it isn't.
> 
> Because at the moment i'm not very well in progressing with ruling things out (which seems to be the only method),
> i only have some symptoms that differ. Wish i had another NIC device with a rom to see if that does work.
> 
> 
> > So it all seems to point to QEMU.
> >> 
> >> --
> >> Sander
> >> 
> >> 
> >> >> I have not yet done the GPU - this issue was preventing me from using
> >> >> qemu-xen as it would always blow up before SeaBIOS was in the picture.
> >> >> 
> >> >> If you would like to put 'Reported-and-Tested-by: Konrad Rzeszutek Wilk
> >> >> <konrad.wilk@oracle.com>' please do.
> >> 
> >> > Will do.
> >> 
> >> >> Thank you!
> >> >> > 
> >> >> > 
> >> >> > diff --git a/hw/xen/xen_pt.c b/hw/xen/xen_pt.c
> >> >> > index 6dd7a68..2bbdb6d 100644
> >> >> > --- a/hw/xen/xen_pt.c
> >> >> > +++ b/hw/xen/xen_pt.c
> >> >> > @@ -440,8 +440,8 @@ static int xen_pt_register_regions(XenPCIPassthroughState *s)
> >> >> >  
> >> >> >          s->bases[PCI_ROM_SLOT].access.maddr = d->rom.base_addr;
> >> >> >  
> >> >> > -        memory_region_init_rom_device(&s->rom, OBJECT(s), NULL, NULL,
> >> >> > -                                      "xen-pci-pt-rom", d->rom.size);
> >> >> > +        memory_region_init_io(&s->rom, OBJECT(s), &ops, &s->dev,
> >> >> > +                              "xen-pci-pt-rom", d->rom.size);
> >> >> >          pci_register_bar(&s->dev, PCI_ROM_SLOT, PCI_BASE_ADDRESS_MEM_PREFETCH,
> >> >> >                           &s->rom);
> >> >> >  
> >> >> > 
> >> >> > -- 
> >> >> > Anthony PERARD
> >> 
> >> 
> 
> 

Re: qemu-xen-dir + PCI passthrough = BOOM

[Zhang, Yang Z]

Konrad Rzeszutek Wilk wrote on 2013-12-05:
> Hey,
> 
> I just started noticing it today - with qemu-xen (tip is commit
> b97307ecaad98360f41ea36cd9674ef810c4f8cf
>     xen_disk: mark ioreq as mapped before unmapping in error case)
> when I try to pass in a PCI device at bootup it blows up with:
> 
Hi, Konrad,

I cannot reproduce this issue with the same qemu-xen. Is there any special step needed?

Here is the NIC info:
08:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
  Subsystem: Intel Corporation PRO/1000 PT Desktop Adapter
  Flags: bus master, fast devsel, latency 0, IRQ 34
  Memory at d2440000 (32-bit, non-prefetchable) [size=128K]
  Memory at d2420000 (32-bit, non-prefetchable) [size=128K]
  I/O ports at 5000 [size=32]
  Expansion ROM at d2400000 [disabled] [size=128K]
  Capabilities: [c8] Power Management version 2
  Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
  Capabilities: [e0] Express Endpoint, MSI 00
  Capabilities: [100] Advanced Error Reporting
  Capabilities: [140] Device Serial Number 00-1b-21-ff-ff-1e-8e-97
  Kernel driver in use: pciback
  Kernel modules: e1000e


BTW, is the fixing already in Qemu upstream?

> char device redirected to /dev/pts/2 (label serial0) qemu: hardware
> error: xen: failed to populate ram at 40050000 CPU #0: EAX=00000000
> EBX=00000000 ECX=00000000 EDX=00000633 ESI=00000000 EDI=00000000
> EBP=00000000 ESP=00000000 EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0
> A20=1 SMM=0 HLT=0 ES =0000 00000000 0000ffff 00009300 CS =f000 ffff0000
> 0000ffff 00009b00 SS =0000 00000000 0000ffff 00009300 DS =0000 00000000
> 0000ffff 00009300 FS =0000 00000000 0000ffff 00009300 GS =0000 00000000
> 0000ffff 00009300 LDT=0000 00000000 0000ffff 00008200 TR =0000 00000000
> 0000ffff 00008b00 GDT=     00000000 0000ffff IDT=     00000000 0000ffff
> CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000 DR0=00000000
> DR1=00000000 DR2=00000000 DR3=00000000 DR6=ffff0ff0 DR7=00000400
> EFER=0000000000000000 FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
> FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
> FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
> FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
> FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
> XMM00=00000000000000000000000000000000
> XMM01=00000000000000000000000000000000
> XMM02=00000000000000000000000000000000
> XMM03=00000000000000000000000000000000
> XMM04=00000000000000000000000000000000
> XMM05=00000000000000000000000000000000
> XMM06=00000000000000000000000000000000
> XMM07=00000000000000000000000000000000 CPU #1: EAX=00000000 EBX=00000000
> ECX=00000000 EDX=00000633 ESI=00000000 EDI=00000000 EBP=00000000
> ESP=00000000 EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0
> HLT=1 ES =0000 00000000 0000ffff 00009300 CS =f000 ffff0000 0000ffff
> 00009b00
> 
> 
> ...
> -bash-4.1# xl dmesg | tail
> [ 2788.038463] xen_pciback: vpci(d3) [2013-12-04 19:43:40] System
> requested SeaBIOS
> : 0000:01:00.1: (d3) [2013-12-04 19:43:40] CPU speed is 3093 MHz
> assign to virtua(d3) [2013-12-04 19:43:40] Relocating guest memory for
> lowmem MMIO space disabled l slot 0 [ 2788.076396] device vif3.0
> entered promiscuous mode
> 
> If I don't have 'pci=' stanze in my guest config it boots just fine.
> 
> -bash-4.1# more /vm.cfg builder='hvm' disk = [
> 'file:/mnt/lab/latest/root_image.iso,hdc:cdrom,r'] memory=1024 boot="d"
> vcpus=2 serial="pty" vnclisten="0.0.0.0" name="latest" vif = [
> 'mac=00:0F:4B:00:00:68, bridge=switch' ] on_crash="preserve"
> pci=["01:00.0"]
> 
> Any ideas?
> 
> This is with today's Xen and 3.13-rc2. The device in question is
> -bash-4.1# lspci -s 01:00.0 -v
> 01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network
> Connection (rev 01)
>         Subsystem: Intel Corporation Gigabit ET Dual Port Server Adapter
>         Flags: fast devsel, IRQ 16 Memory at fbc20000 (32-bit,
>         non-prefetchable) [disabled] [size=128K] Memory at fb800000
>         (32-bit, non-prefetchable) [disabled] [size=4M] I/O ports at
>         e020 [disabled] [size=32] Memory at fbc44000 (32-bit,
>         non-prefetchable) [disabled] [size=16K] Expansion ROM at
>         fb400000 [disabled] [size=4M] Capabilities: [40] Power
>         Management version 3 Capabilities: [50] MSI: Enable- Count=1/1
>         Maskable+ 64bit+ Capabilities: [70] MSI-X: Enable- Count=10
>         Masked- Capabilities: [a0] Express Endpoint, MSI 00
>         Capabilities: [100] Advanced Error Reporting Capabilities: [140]
>         Device Serial Number 00-1b-21-ff-ff-45-d9-ac Capabilities: [150]
>         Alternative Routing-ID Interpretation (ARI) Capabilities: [160]
>         Single Root I/O Virtualization (SR-IOV) Kernel driver in use:
>         pciback Kernel modules: igb
> 
> Oh and of course it boots with the traditional QEMU.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel


Best regards,
Yang

Re: qemu-xen-dir + PCI passthrough = BOOM

[Konrad Rzeszutek Wilk]

"Zhang, Yang Z" <yang.z.zhang@intel.com> wrote:
>Konrad Rzeszutek Wilk wrote on 2013-12-05:
>> Hey,
>> 
>> I just started noticing it today - with qemu-xen (tip is commit
>> b97307ecaad98360f41ea36cd9674ef810c4f8cf
>>     xen_disk: mark ioreq as mapped before unmapping in error case)
>> when I try to pass in a PCI device at bootup it blows up with:
>> 
>Hi, Konrad,

Hey!
>
>I cannot reproduce this issue with the same qemu-xen. Is there any
>special step needed?
>
>Here is the NIC info:
>08:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet
>Controller (Copper) (rev 06)
>  Subsystem: Intel Corporation PRO/1000 PT Desktop Adapter
>  Flags: bus master, fast devsel, latency 0, IRQ 34
>  Memory at d2440000 (32-bit, non-prefetchable) [size=128K]
>  Memory at d2420000 (32-bit, non-prefetchable) [size=128K]
>  I/O ports at 5000 [size=32]
>  Expansion ROM at d2400000 [disabled] [size=128K]

Your ROM is 128k while mine is 4MB. You need that (a big ROM image) to trigger this.

>  Capabilities: [c8] Power Management version 2
>  Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
>  Capabilities: [e0] Express Endpoint, MSI 00
>  Capabilities: [100] Advanced Error Reporting
>  Capabilities: [140] Device Serial Number 00-1b-21-ff-ff-1e-8e-97
>  Kernel driver in use: pciback
>  Kernel modules: e1000e
>
>
>BTW, is the fixing already in Qemu upstream?

I don't know. Anthony would know since his patch fixed it.
>
>> char device redirected to /dev/pts/2 (label serial0) qemu: hardware
>> error: xen: failed to populate ram at 40050000 CPU #0: EAX=00000000
>> EBX=00000000 ECX=00000000 EDX=00000633 ESI=00000000 EDI=00000000
>> EBP=00000000 ESP=00000000 EIP=0000fff0 EFL=00000002 [-------] CPL=0
>II=0
>> A20=1 SMM=0 HLT=0 ES =0000 00000000 0000ffff 00009300 CS =f000
>ffff0000
>> 0000ffff 00009b00 SS =0000 00000000 0000ffff 00009300 DS =0000
>00000000
>> 0000ffff 00009300 FS =0000 00000000 0000ffff 00009300 GS =0000
>00000000
>> 0000ffff 00009300 LDT=0000 00000000 0000ffff 00008200 TR =0000
>00000000
>> 0000ffff 00008b00 GDT=     00000000 0000ffff IDT=     00000000
>0000ffff
>> CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000 DR0=00000000
>> DR1=00000000 DR2=00000000 DR3=00000000 DR6=ffff0ff0 DR7=00000400
>> EFER=0000000000000000 FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
>> FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
>> FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
>> FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
>> FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
>> XMM00=00000000000000000000000000000000
>> XMM01=00000000000000000000000000000000
>> XMM02=00000000000000000000000000000000
>> XMM03=00000000000000000000000000000000
>> XMM04=00000000000000000000000000000000
>> XMM05=00000000000000000000000000000000
>> XMM06=00000000000000000000000000000000
>> XMM07=00000000000000000000000000000000 CPU #1: EAX=00000000
>EBX=00000000
>> ECX=00000000 EDX=00000633 ESI=00000000 EDI=00000000 EBP=00000000
>> ESP=00000000 EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1
>SMM=0
>> HLT=1 ES =0000 00000000 0000ffff 00009300 CS =f000 ffff0000 0000ffff
>> 00009b00
>> 
>> 
>> ...
>> -bash-4.1# xl dmesg | tail
>> [ 2788.038463] xen_pciback: vpci(d3) [2013-12-04 19:43:40] System
>> requested SeaBIOS
>> : 0000:01:00.1: (d3) [2013-12-04 19:43:40] CPU speed is 3093 MHz
>> assign to virtua(d3) [2013-12-04 19:43:40] Relocating guest memory
>for
>> lowmem MMIO space disabled l slot 0 [ 2788.076396] device vif3.0
>> entered promiscuous mode
>> 
>> If I don't have 'pci=' stanze in my guest config it boots just fine.
>> 
>> -bash-4.1# more /vm.cfg builder='hvm' disk = [
>> 'file:/mnt/lab/latest/root_image.iso,hdc:cdrom,r'] memory=1024
>boot="d"
>> vcpus=2 serial="pty" vnclisten="0.0.0.0" name="latest" vif = [
>> 'mac=00:0F:4B:00:00:68, bridge=switch' ] on_crash="preserve"
>> pci=["01:00.0"]
>> 
>> Any ideas?
>> 
>> This is with today's Xen and 3.13-rc2. The device in question is
>> -bash-4.1# lspci -s 01:00.0 -v
>> 01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network
>> Connection (rev 01)
>>         Subsystem: Intel Corporation Gigabit ET Dual Port Server
>Adapter
>>         Flags: fast devsel, IRQ 16 Memory at fbc20000 (32-bit,
>>         non-prefetchable) [disabled] [size=128K] Memory at fb800000
>>         (32-bit, non-prefetchable) [disabled] [size=4M] I/O ports at
>>         e020 [disabled] [size=32] Memory at fbc44000 (32-bit,
>>         non-prefetchable) [disabled] [size=16K] Expansion ROM at
>>         fb400000 [disabled] [size=4M] Capabilities: [40] Power
>>         Management version 3 Capabilities: [50] MSI: Enable-
>Count=1/1
>>         Maskable+ 64bit+ Capabilities: [70] MSI-X: Enable- Count=10
>>         Masked- Capabilities: [a0] Express Endpoint, MSI 00
>>         Capabilities: [100] Advanced Error Reporting Capabilities:
>[140]
>>         Device Serial Number 00-1b-21-ff-ff-45-d9-ac Capabilities:
>[150]
>>         Alternative Routing-ID Interpretation (ARI) Capabilities:
>[160]
>>         Single Root I/O Virtualization (SR-IOV) Kernel driver in use:
>>         pciback Kernel modules: igb
>> 
>> Oh and of course it boots with the traditional QEMU.
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel
>
>
>Best regards,
>Yang

Re: qemu-xen-dir + PCI passthrough = BOOM

[Zhang, Yang Z]

Anthony PERARD wrote on 2014-01-09:
> On Wed, Jan 08, 2014 at 02:44:51PM -0500, Konrad Rzeszutek Wilk wrote:
>> On Wed, Dec 18, 2013 at 02:48:24PM +0000, Anthony PERARD wrote:
>>> On Mon, Dec 16, 2013 at 10:08:16AM -0500, Konrad Rzeszutek Wilk wrote:
>>>> On Fri, Dec 06, 2013 at 04:03:10PM +0000, Wei Liu wrote:
>>>>> On Fri, Dec 06, 2013 at 04:00:18PM +0000, Wei Liu wrote:
>>>>> [...]
>>>>>>> Those Xen report something like:
>>>>>>> (XEN) page_alloc.c:1460:d0 Over-allocation for domain 46:
>>>>>>> 131329 >
>>>>>>> 131328
>>>>>>> (XEN) memory.c:132:d0 Could not allocate order=0 extent:
>>>>>>> id=46
>>>>>>> memflags=0 (62 of 64)
>>>>>>> 
>>>>>>> ?
>>>>>>> 
>>>>>>> (I tryied to reproduce the issue by simply add many
>>>>>>> emulated
>>>>>>> e1000 in QEMU :) )
>>>>>>> 
>>> 
>>>> -bash-4.1# lspci -s 01:00.0 -v
>>>> 01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit
>>>> Network
> Connection (rev 01)
>>>>         Subsystem: Intel Corporation Gigabit ET Dual Port Server
>>>>         Adapter Flags: fast devsel, IRQ 16 Memory at fbc20000
>>>>         (32-bit, non-prefetchable) [disabled] [size=128K] Memory at
>>>>         fb800000 (32-bit, non-prefetchable) [disabled] [size=4M] I/O
>>>>         ports at e020 [disabled] [size=32] Memory at fbc44000
>>>>         (32-bit, non-prefetchable) [disabled] [size=16K] Expansion
>>>>         ROM at fb400000 [disabled] [size=4M]
>>> 
>>> BTW, I think this is the issue, the Expansion ROM. qemu-xen will
>>> allocate memory for it. Will have maybe have to find another way.
>>> qemu-trad those not seems to allocate memory, but I haven't been
>>> very far in trying to check that.
>> 
>> And indeed that is the case. The "Fix" below fixes it.
>> 
>> 
>> Based on that and this guest config:
>> disk = [ 'file:/mnt/lab/latest/root_image.iso,hdc:cdrom,r']
>> memory = 2048
>> boot="d"
>> maxvcpus=32
>> vcpus=1
>> serial='pty'
>> vnclisten="0.0.0.0"
>> name="latest"
>> vif = [ 'mac=00:0F:4B:00:00:68, bridge=switch' ] pci = ["01:00.0"]
>> 
>> I can boot the guest.
> 
> And can you access the ROM from the guest ?
> 
> 
> Also, I have another patch, it will initialize the PCI ROM BAR like any other BAR.
> In this case, if qemu is envolved in the access to ROM, it will print
> an error, like it the case for other BAR.
> 
> I tried to test it, but it was with an embedded VGA card. When I dump
> the ROM, I got the same one as the emulated card instead of the ROM from the device.
> 
> 
> diff --git a/hw/xen/xen_pt.c b/hw/xen/xen_pt.c index 6dd7a68..2bbdb6d
> 100644 --- a/hw/xen/xen_pt.c +++ b/hw/xen/xen_pt.c @@ -440,8 +440,8 @@
> static int xen_pt_register_regions(XenPCIPassthroughState *s)
> 
>          s->bases[PCI_ROM_SLOT].access.maddr = d->rom.base_addr;
> -        memory_region_init_rom_device(&s->rom, OBJECT(s), NULL, NULL,
> -                                      "xen-pci-pt-rom", d->rom.size);
> +        memory_region_init_io(&s->rom, OBJECT(s), &ops, &s->dev,
> +                              "xen-pci-pt-rom", d->rom.size);
>          pci_register_bar(&s->dev, PCI_ROM_SLOT,
> PCI_BASE_ADDRESS_MEM_PREFETCH,
>                           &s->rom);
>

Hi, Anthony,

Does your fixing is the final solution for this issue? If yes, will you push it before Xen 4.4 release?

Best regards,
Yang

Re: qemu-xen-dir + PCI passthrough = BOOM

[Zhang, Yang Z]

Zhang, Yang Z wrote on 2014-01-26:
> Anthony PERARD wrote on 2014-01-09:
>> On Wed, Jan 08, 2014 at 02:44:51PM -0500, Konrad Rzeszutek Wilk wrote:
>>> On Wed, Dec 18, 2013 at 02:48:24PM +0000, Anthony PERARD wrote:
>>>> On Mon, Dec 16, 2013 at 10:08:16AM -0500, Konrad Rzeszutek Wilk wrote:
>>>>> On Fri, Dec 06, 2013 at 04:03:10PM +0000, Wei Liu wrote:
>>>>>> On Fri, Dec 06, 2013 at 04:00:18PM +0000, Wei Liu wrote:
>>>>>> [...]
>>>>>>>> Those Xen report something like:
>>>>>>>> (XEN) page_alloc.c:1460:d0 Over-allocation for domain 46:
>>>>>>>> 131329 >
>>>>>>>> 131328
>>>>>>>> (XEN) memory.c:132:d0 Could not allocate order=0 extent:
>>>>>>>> id=46
>>>>>>>> memflags=0 (62 of 64)
>>>>>>>> 
>>>>>>>> ?
>>>>>>>> 
>>>>>>>> (I tryied to reproduce the issue by simply add many emulated
>>>>>>>> e1000 in QEMU :) )
>>>>>>>> 
>>>> 
>>>>> -bash-4.1# lspci -s 01:00.0 -v
>>>>> 01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit
>>>>> Network
>> Connection (rev 01)
>>>>>         Subsystem: Intel Corporation Gigabit ET Dual Port Server
>>>>>         Adapter Flags: fast devsel, IRQ 16 Memory at fbc20000
>>>>>         (32-bit, non-prefetchable) [disabled] [size=128K] Memory at
>>>>>         fb800000 (32-bit, non-prefetchable) [disabled] [size=4M] I/O
>>>>>         ports at e020 [disabled] [size=32] Memory at fbc44000
>>>>>         (32-bit, non-prefetchable) [disabled] [size=16K] Expansion
>>>>>         ROM at fb400000 [disabled] [size=4M]
>>>> 
>>>> BTW, I think this is the issue, the Expansion ROM. qemu-xen will
>>>> allocate memory for it. Will have maybe have to find another way.
>>>> qemu-trad those not seems to allocate memory, but I haven't been
>>>> very far in trying to check that.
>>> 
>>> And indeed that is the case. The "Fix" below fixes it.
>>> 
>>> 
>>> Based on that and this guest config:
>>> disk = [ 'file:/mnt/lab/latest/root_image.iso,hdc:cdrom,r']
>>> memory = 2048
>>> boot="d"
>>> maxvcpus=32
>>> vcpus=1
>>> serial='pty'
>>> vnclisten="0.0.0.0"
>>> name="latest"
>>> vif = [ 'mac=00:0F:4B:00:00:68, bridge=switch' ] pci = ["01:00.0"]
>>> 
>>> I can boot the guest.
>> 
>> And can you access the ROM from the guest ?
>> 
>> 
>> Also, I have another patch, it will initialize the PCI ROM BAR like any
>> other BAR. In this case, if qemu is envolved in the access to ROM, it
>> will print an error, like it the case for other BAR.
>> 
>> I tried to test it, but it was with an embedded VGA card. When I dump
>> the ROM, I got the same one as the emulated card instead of the ROM
>> from the device.
>> 
>> 
>> diff --git a/hw/xen/xen_pt.c b/hw/xen/xen_pt.c index 6dd7a68..2bbdb6d
>> 100644 --- a/hw/xen/xen_pt.c +++ b/hw/xen/xen_pt.c @@ -440,8 +440,8 @@
>> static int xen_pt_register_regions(XenPCIPassthroughState *s)
>> 
>>          s->bases[PCI_ROM_SLOT].access.maddr = d->rom.base_addr;
>> -        memory_region_init_rom_device(&s->rom, OBJECT(s), NULL, NULL,
>> -                                      "xen-pci-pt-rom", d->rom.size);
>> +        memory_region_init_io(&s->rom, OBJECT(s), &ops, &s->dev, +    
>>                          "xen-pci-pt-rom", d->rom.size);
>>          pci_register_bar(&s->dev, PCI_ROM_SLOT,
>> PCI_BASE_ADDRESS_MEM_PREFETCH,
>>                           &s->rom);
> 
> Hi, Anthony,
> 
> Does your fixing is the final solution for this issue? If yes, will
> you push it before Xen 4.4 release?
> 
> Best regards,
> Yang
>

CC songtao who still saw this issue with qemu-xen. If the patch is in, please let us know. We hope it will be included in Xen 4.4 release.

Best regards,
Yang

Re: qemu-xen-dir + PCI passthrough = BOOM

[Stefano Stabellini]

On Sun, 26 Jan 2014, Zhang, Yang Z wrote:
> Anthony PERARD wrote on 2014-01-09:
> > On Wed, Jan 08, 2014 at 02:44:51PM -0500, Konrad Rzeszutek Wilk wrote:
> >> On Wed, Dec 18, 2013 at 02:48:24PM +0000, Anthony PERARD wrote:
> >>> On Mon, Dec 16, 2013 at 10:08:16AM -0500, Konrad Rzeszutek Wilk wrote:
> >>>> On Fri, Dec 06, 2013 at 04:03:10PM +0000, Wei Liu wrote:
> >>>>> On Fri, Dec 06, 2013 at 04:00:18PM +0000, Wei Liu wrote:
> >>>>> [...]
> >>>>>>> Those Xen report something like:
> >>>>>>> (XEN) page_alloc.c:1460:d0 Over-allocation for domain 46:
> >>>>>>> 131329 >
> >>>>>>> 131328
> >>>>>>> (XEN) memory.c:132:d0 Could not allocate order=0 extent:
> >>>>>>> id=46
> >>>>>>> memflags=0 (62 of 64)
> >>>>>>> 
> >>>>>>> ?
> >>>>>>> 
> >>>>>>> (I tryied to reproduce the issue by simply add many
> >>>>>>> emulated
> >>>>>>> e1000 in QEMU :) )
> >>>>>>> 
> >>> 
> >>>> -bash-4.1# lspci -s 01:00.0 -v
> >>>> 01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit
> >>>> Network
> > Connection (rev 01)
> >>>>         Subsystem: Intel Corporation Gigabit ET Dual Port Server
> >>>>         Adapter Flags: fast devsel, IRQ 16 Memory at fbc20000
> >>>>         (32-bit, non-prefetchable) [disabled] [size=128K] Memory at
> >>>>         fb800000 (32-bit, non-prefetchable) [disabled] [size=4M] I/O
> >>>>         ports at e020 [disabled] [size=32] Memory at fbc44000
> >>>>         (32-bit, non-prefetchable) [disabled] [size=16K] Expansion
> >>>>         ROM at fb400000 [disabled] [size=4M]
> >>> 
> >>> BTW, I think this is the issue, the Expansion ROM. qemu-xen will
> >>> allocate memory for it. Will have maybe have to find another way.
> >>> qemu-trad those not seems to allocate memory, but I haven't been
> >>> very far in trying to check that.
> >> 
> >> And indeed that is the case. The "Fix" below fixes it.
> >> 
> >> 
> >> Based on that and this guest config:
> >> disk = [ 'file:/mnt/lab/latest/root_image.iso,hdc:cdrom,r']
> >> memory = 2048
> >> boot="d"
> >> maxvcpus=32
> >> vcpus=1
> >> serial='pty'
> >> vnclisten="0.0.0.0"
> >> name="latest"
> >> vif = [ 'mac=00:0F:4B:00:00:68, bridge=switch' ] pci = ["01:00.0"]
> >> 
> >> I can boot the guest.
> > 
> > And can you access the ROM from the guest ?
> > 
> > 
> > Also, I have another patch, it will initialize the PCI ROM BAR like any other BAR.
> > In this case, if qemu is envolved in the access to ROM, it will print
> > an error, like it the case for other BAR.
> > 
> > I tried to test it, but it was with an embedded VGA card. When I dump
> > the ROM, I got the same one as the emulated card instead of the ROM from the device.
> > 
> > 
> > diff --git a/hw/xen/xen_pt.c b/hw/xen/xen_pt.c index 6dd7a68..2bbdb6d
> > 100644 --- a/hw/xen/xen_pt.c +++ b/hw/xen/xen_pt.c @@ -440,8 +440,8 @@
> > static int xen_pt_register_regions(XenPCIPassthroughState *s)
> > 
> >          s->bases[PCI_ROM_SLOT].access.maddr = d->rom.base_addr;
> > -        memory_region_init_rom_device(&s->rom, OBJECT(s), NULL, NULL,
> > -                                      "xen-pci-pt-rom", d->rom.size);
> > +        memory_region_init_io(&s->rom, OBJECT(s), &ops, &s->dev,
> > +                              "xen-pci-pt-rom", d->rom.size);
> >          pci_register_bar(&s->dev, PCI_ROM_SLOT,
> > PCI_BASE_ADDRESS_MEM_PREFETCH,
> >                           &s->rom);
> >
> 
> Hi, Anthony,
> 
> Does your fixing is the final solution for this issue? If yes, will you push it before Xen 4.4 release?

I included this patch in the last pull request I sent to Anthony
Liguori:

http://marc.info/?l=qemu-devel&m=138997319906095

It hasn't been pulled yet, but I would expect that it is going to be
upstream soon.
Regarding the 4.4 release, we are trying to fix a couple of other
serious bugs in the qemu-xen tree right now, but it is still conceivable
to have this fix backported to the tree in time for the release.

Re: qemu-xen-dir + PCI passthrough = BOOM

[Zhang, Yang Z]

Stefano Stabellini wrote on 2014-01-27:
> On Sun, 26 Jan 2014, Zhang, Yang Z wrote:
>> Anthony PERARD wrote on 2014-01-09:
>>> On Wed, Jan 08, 2014 at 02:44:51PM -0500, Konrad Rzeszutek Wilk wrote:
>>>> On Wed, Dec 18, 2013 at 02:48:24PM +0000, Anthony PERARD wrote:
>>>>> On Mon, Dec 16, 2013 at 10:08:16AM -0500, Konrad Rzeszutek Wilk
> wrote:
>>>>>> On Fri, Dec 06, 2013 at 04:03:10PM +0000, Wei Liu wrote:
>>>>>>> On Fri, Dec 06, 2013 at 04:00:18PM +0000, Wei Liu wrote:
>>>>>>> [...]
>>>>>>>>> Those Xen report something like:
>>>>>>>>> (XEN) page_alloc.c:1460:d0 Over-allocation for domain 46:
>>>>>>>>> 131329 >
>>>>>>>>> 131328
>>>>>>>>> (XEN) memory.c:132:d0 Could not allocate order=0 extent:
>>>>>>>>> id=46
>>>>>>>>> memflags=0 (62 of 64)
>>>>>>>>> 
>>>>>>>>> ?
>>>>>>>>> 
>>>>>>>>> (I tryied to reproduce the issue by simply add many emulated
>>>>>>>>> e1000 in QEMU :) )
>>>>>>>>> 
>>>>> 
>>>>>> -bash-4.1# lspci -s 01:00.0 -v
>>>>>> 01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit
>>>>>> Network
>>> Connection (rev 01)
>>>>>>         Subsystem: Intel Corporation Gigabit ET Dual Port Server
>>>>>>         Adapter Flags: fast devsel, IRQ 16 Memory at fbc20000
>>>>>>         (32-bit, non-prefetchable) [disabled] [size=128K] Memory at
>>>>>>         fb800000 (32-bit, non-prefetchable) [disabled] [size=4M] I/O
>>>>>>         ports at e020 [disabled] [size=32] Memory at fbc44000
>>>>>>         (32-bit, non-prefetchable) [disabled] [size=16K] Expansion
>>>>>>         ROM at fb400000 [disabled] [size=4M]
>>>>> 
>>>>> BTW, I think this is the issue, the Expansion ROM. qemu-xen will
>>>>> allocate memory for it. Will have maybe have to find another way.
>>>>> qemu-trad those not seems to allocate memory, but I haven't been
>>>>> very far in trying to check that.
>>>> 
>>>> And indeed that is the case. The "Fix" below fixes it.
>>>> 
>>>> 
>>>> Based on that and this guest config:
>>>> disk = [ 'file:/mnt/lab/latest/root_image.iso,hdc:cdrom,r']
>>>> memory = 2048
>>>> boot="d"
>>>> maxvcpus=32
>>>> vcpus=1
>>>> serial='pty'
>>>> vnclisten="0.0.0.0"
>>>> name="latest"
>>>> vif = [ 'mac=00:0F:4B:00:00:68, bridge=switch' ] pci =
>>>> ["01:00.0"]
>>>> 
>>>> I can boot the guest.
>>> 
>>> And can you access the ROM from the guest ?
>>> 
>>> 
>>> Also, I have another patch, it will initialize the PCI ROM BAR like
>>> any other BAR. In this case, if qemu is envolved in the access to ROM,
>>> it will print an error, like it the case for other BAR.
>>> 
>>> I tried to test it, but it was with an embedded VGA card. When I dump
>>> the ROM, I got the same one as the emulated card instead of the ROM
>>> from the device.
>>> 
>>> 
>>> diff --git a/hw/xen/xen_pt.c b/hw/xen/xen_pt.c index
>>> 6dd7a68..2bbdb6d
>>> 100644 --- a/hw/xen/xen_pt.c +++ b/hw/xen/xen_pt.c @@ -440,8
>>> +440,8 @@ static int
>>> xen_pt_register_regions(XenPCIPassthroughState *s)
>>> 
>>>          s->bases[PCI_ROM_SLOT].access.maddr = d->rom.base_addr;
>>> -        memory_region_init_rom_device(&s->rom, OBJECT(s), NULL, NULL,
>>> -                                      "xen-pci-pt-rom", d->rom.size);
>>> +        memory_region_init_io(&s->rom, OBJECT(s), &ops, &s->dev, +   
>>>                           "xen-pci-pt-rom", d->rom.size);
>>>          pci_register_bar(&s->dev, PCI_ROM_SLOT,
>>> PCI_BASE_ADDRESS_MEM_PREFETCH,
>>>                           &s->rom);
>> 
>> Hi, Anthony,
>> 
>> Does your fixing is the final solution for this issue? If yes, will
>> you push it
> before Xen 4.4 release?
> 
> I included this patch in the last pull request I sent to Anthony Liguori:
> 
> http://marc.info/?l=qemu-devel&m=138997319906095
> 
> It hasn't been pulled yet, but I would expect that it is going to be
> upstream soon.
> Regarding the 4.4 release, we are trying to fix a couple of other
> serious bugs in the qemu-xen tree right now, but it is still
> conceivable to have this fix backported to the tree in time for the release.

Hope it can catch the 4.4 release. 

BTW: Do you know when Xen 4.4 will be released? I saw the wiki documented that it should be Jan 21. But it seems there still will have RC3 come out on Feb 9.

Best regards,
Yang

Re: qemu-xen-dir + PCI passthrough = BOOM

[Stefano Stabellini]

On Mon, 27 Jan 2014, Zhang, Yang Z wrote:
> Stefano Stabellini wrote on 2014-01-27:
> > On Sun, 26 Jan 2014, Zhang, Yang Z wrote:
> >> Anthony PERARD wrote on 2014-01-09:
> >>> On Wed, Jan 08, 2014 at 02:44:51PM -0500, Konrad Rzeszutek Wilk wrote:
> >>>> On Wed, Dec 18, 2013 at 02:48:24PM +0000, Anthony PERARD wrote:
> >>>>> On Mon, Dec 16, 2013 at 10:08:16AM -0500, Konrad Rzeszutek Wilk
> > wrote:
> >>>>>> On Fri, Dec 06, 2013 at 04:03:10PM +0000, Wei Liu wrote:
> >>>>>>> On Fri, Dec 06, 2013 at 04:00:18PM +0000, Wei Liu wrote:
> >>>>>>> [...]
> >>>>>>>>> Those Xen report something like:
> >>>>>>>>> (XEN) page_alloc.c:1460:d0 Over-allocation for domain 46:
> >>>>>>>>> 131329 >
> >>>>>>>>> 131328
> >>>>>>>>> (XEN) memory.c:132:d0 Could not allocate order=0 extent:
> >>>>>>>>> id=46
> >>>>>>>>> memflags=0 (62 of 64)
> >>>>>>>>> 
> >>>>>>>>> ?
> >>>>>>>>> 
> >>>>>>>>> (I tryied to reproduce the issue by simply add many emulated
> >>>>>>>>> e1000 in QEMU :) )
> >>>>>>>>> 
> >>>>> 
> >>>>>> -bash-4.1# lspci -s 01:00.0 -v
> >>>>>> 01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit
> >>>>>> Network
> >>> Connection (rev 01)
> >>>>>>         Subsystem: Intel Corporation Gigabit ET Dual Port Server
> >>>>>>         Adapter Flags: fast devsel, IRQ 16 Memory at fbc20000
> >>>>>>         (32-bit, non-prefetchable) [disabled] [size=128K] Memory at
> >>>>>>         fb800000 (32-bit, non-prefetchable) [disabled] [size=4M] I/O
> >>>>>>         ports at e020 [disabled] [size=32] Memory at fbc44000
> >>>>>>         (32-bit, non-prefetchable) [disabled] [size=16K] Expansion
> >>>>>>         ROM at fb400000 [disabled] [size=4M]
> >>>>> 
> >>>>> BTW, I think this is the issue, the Expansion ROM. qemu-xen will
> >>>>> allocate memory for it. Will have maybe have to find another way.
> >>>>> qemu-trad those not seems to allocate memory, but I haven't been
> >>>>> very far in trying to check that.
> >>>> 
> >>>> And indeed that is the case. The "Fix" below fixes it.
> >>>> 
> >>>> 
> >>>> Based on that and this guest config:
> >>>> disk = [ 'file:/mnt/lab/latest/root_image.iso,hdc:cdrom,r']
> >>>> memory = 2048
> >>>> boot="d"
> >>>> maxvcpus=32
> >>>> vcpus=1
> >>>> serial='pty'
> >>>> vnclisten="0.0.0.0"
> >>>> name="latest"
> >>>> vif = [ 'mac=00:0F:4B:00:00:68, bridge=switch' ] pci =
> >>>> ["01:00.0"]
> >>>> 
> >>>> I can boot the guest.
> >>> 
> >>> And can you access the ROM from the guest ?
> >>> 
> >>> 
> >>> Also, I have another patch, it will initialize the PCI ROM BAR like
> >>> any other BAR. In this case, if qemu is envolved in the access to ROM,
> >>> it will print an error, like it the case for other BAR.
> >>> 
> >>> I tried to test it, but it was with an embedded VGA card. When I dump
> >>> the ROM, I got the same one as the emulated card instead of the ROM
> >>> from the device.
> >>> 
> >>> 
> >>> diff --git a/hw/xen/xen_pt.c b/hw/xen/xen_pt.c index
> >>> 6dd7a68..2bbdb6d
> >>> 100644 --- a/hw/xen/xen_pt.c +++ b/hw/xen/xen_pt.c @@ -440,8
> >>> +440,8 @@ static int
> >>> xen_pt_register_regions(XenPCIPassthroughState *s)
> >>> 
> >>>          s->bases[PCI_ROM_SLOT].access.maddr = d->rom.base_addr;
> >>> -        memory_region_init_rom_device(&s->rom, OBJECT(s), NULL, NULL,
> >>> -                                      "xen-pci-pt-rom", d->rom.size);
> >>> +        memory_region_init_io(&s->rom, OBJECT(s), &ops, &s->dev, +   
> >>>                           "xen-pci-pt-rom", d->rom.size);
> >>>          pci_register_bar(&s->dev, PCI_ROM_SLOT,
> >>> PCI_BASE_ADDRESS_MEM_PREFETCH,
> >>>                           &s->rom);
> >> 
> >> Hi, Anthony,
> >> 
> >> Does your fixing is the final solution for this issue? If yes, will
> >> you push it
> > before Xen 4.4 release?
> > 
> > I included this patch in the last pull request I sent to Anthony Liguori:
> > 
> > http://marc.info/?l=qemu-devel&m=138997319906095
> > 
> > It hasn't been pulled yet, but I would expect that it is going to be
> > upstream soon.
> > Regarding the 4.4 release, we are trying to fix a couple of other
> > serious bugs in the qemu-xen tree right now, but it is still
> > conceivable to have this fix backported to the tree in time for the release.
> 
> Hope it can catch the 4.4 release. 
> 
> BTW: Do you know when Xen 4.4 will be released? I saw the wiki documented that it should be Jan 21. But it seems there still will have RC3 come out on Feb 9.

When it is ready ;-)
It is probably going to be "soon".

Re: qemu-xen-dir + PCI passthrough = BOOM

[Stefano Stabellini]

On Mon, 27 Jan 2014, Zhang, Yang Z wrote:
> Stefano Stabellini wrote on 2014-01-27:
> > On Sun, 26 Jan 2014, Zhang, Yang Z wrote:
> >> Anthony PERARD wrote on 2014-01-09:
> >>> On Wed, Jan 08, 2014 at 02:44:51PM -0500, Konrad Rzeszutek Wilk wrote:
> >>>> On Wed, Dec 18, 2013 at 02:48:24PM +0000, Anthony PERARD wrote:
> >>>>> On Mon, Dec 16, 2013 at 10:08:16AM -0500, Konrad Rzeszutek Wilk
> > wrote:
> >>>>>> On Fri, Dec 06, 2013 at 04:03:10PM +0000, Wei Liu wrote:
> >>>>>>> On Fri, Dec 06, 2013 at 04:00:18PM +0000, Wei Liu wrote:
> >>>>>>> [...]
> >>>>>>>>> Those Xen report something like:
> >>>>>>>>> (XEN) page_alloc.c:1460:d0 Over-allocation for domain 46:
> >>>>>>>>> 131329 >
> >>>>>>>>> 131328
> >>>>>>>>> (XEN) memory.c:132:d0 Could not allocate order=0 extent:
> >>>>>>>>> id=46
> >>>>>>>>> memflags=0 (62 of 64)
> >>>>>>>>> 
> >>>>>>>>> ?
> >>>>>>>>> 
> >>>>>>>>> (I tryied to reproduce the issue by simply add many emulated
> >>>>>>>>> e1000 in QEMU :) )
> >>>>>>>>> 
> >>>>> 
> >>>>>> -bash-4.1# lspci -s 01:00.0 -v
> >>>>>> 01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit
> >>>>>> Network
> >>> Connection (rev 01)
> >>>>>>         Subsystem: Intel Corporation Gigabit ET Dual Port Server
> >>>>>>         Adapter Flags: fast devsel, IRQ 16 Memory at fbc20000
> >>>>>>         (32-bit, non-prefetchable) [disabled] [size=128K] Memory at
> >>>>>>         fb800000 (32-bit, non-prefetchable) [disabled] [size=4M] I/O
> >>>>>>         ports at e020 [disabled] [size=32] Memory at fbc44000
> >>>>>>         (32-bit, non-prefetchable) [disabled] [size=16K] Expansion
> >>>>>>         ROM at fb400000 [disabled] [size=4M]
> >>>>> 
> >>>>> BTW, I think this is the issue, the Expansion ROM. qemu-xen will
> >>>>> allocate memory for it. Will have maybe have to find another way.
> >>>>> qemu-trad those not seems to allocate memory, but I haven't been
> >>>>> very far in trying to check that.
> >>>> 
> >>>> And indeed that is the case. The "Fix" below fixes it.
> >>>> 
> >>>> 
> >>>> Based on that and this guest config:
> >>>> disk = [ 'file:/mnt/lab/latest/root_image.iso,hdc:cdrom,r']
> >>>> memory = 2048
> >>>> boot="d"
> >>>> maxvcpus=32
> >>>> vcpus=1
> >>>> serial='pty'
> >>>> vnclisten="0.0.0.0"
> >>>> name="latest"
> >>>> vif = [ 'mac=00:0F:4B:00:00:68, bridge=switch' ] pci =
> >>>> ["01:00.0"]
> >>>> 
> >>>> I can boot the guest.
> >>> 
> >>> And can you access the ROM from the guest ?
> >>> 
> >>> 
> >>> Also, I have another patch, it will initialize the PCI ROM BAR like
> >>> any other BAR. In this case, if qemu is envolved in the access to ROM,
> >>> it will print an error, like it the case for other BAR.
> >>> 
> >>> I tried to test it, but it was with an embedded VGA card. When I dump
> >>> the ROM, I got the same one as the emulated card instead of the ROM
> >>> from the device.
> >>> 
> >>> 
> >>> diff --git a/hw/xen/xen_pt.c b/hw/xen/xen_pt.c index
> >>> 6dd7a68..2bbdb6d
> >>> 100644 --- a/hw/xen/xen_pt.c +++ b/hw/xen/xen_pt.c @@ -440,8
> >>> +440,8 @@ static int
> >>> xen_pt_register_regions(XenPCIPassthroughState *s)
> >>> 
> >>>          s->bases[PCI_ROM_SLOT].access.maddr = d->rom.base_addr;
> >>> -        memory_region_init_rom_device(&s->rom, OBJECT(s), NULL, NULL,
> >>> -                                      "xen-pci-pt-rom", d->rom.size);
> >>> +        memory_region_init_io(&s->rom, OBJECT(s), &ops, &s->dev, +   
> >>>                           "xen-pci-pt-rom", d->rom.size);
> >>>          pci_register_bar(&s->dev, PCI_ROM_SLOT,
> >>> PCI_BASE_ADDRESS_MEM_PREFETCH,
> >>>                           &s->rom);
> >> 
> >> Hi, Anthony,
> >> 
> >> Does your fixing is the final solution for this issue? If yes, will
> >> you push it
> > before Xen 4.4 release?
> > 
> > I included this patch in the last pull request I sent to Anthony Liguori:
> > 
> > http://marc.info/?l=qemu-devel&m=138997319906095
> > 
> > It hasn't been pulled yet, but I would expect that it is going to be
> > upstream soon.
> > Regarding the 4.4 release, we are trying to fix a couple of other
> > serious bugs in the qemu-xen tree right now, but it is still
> > conceivable to have this fix backported to the tree in time for the release.
> 
> Hope it can catch the 4.4 release. 

The fix is in qemu-xen now.