[Qemu-devel] Help: Does Qemu support virtio-pci for net-device and disk device?

[Qemu-devel] Help: Does Qemu support virtio-pci for net-device and disk device?

[Kevin Zhao]

Hi all,
     Now I'm investigating net device hot plug and disk hotplug for
AArch64. For virtio , the default address is virtio-mmio. After Libvirt
1.3.5, user can explicitly specify the address-type to pci and so libvirt
will pass the virtio-pci parameters to the Qemu.
     Both my host and guest OS is Debian8, and Qemu version is 2.6.0.
Libvirt version is 1.3.5.
     For net-device, I change the address-type to pci, and libvirt pass the
command below:
     -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:0d:25:25,bus=pci.2,addr=0x1

     After booting, the eth0 device disappear(eth0 occur when the address
is virtio-mmio),
but I can find another net-device enp2s1, also it can't work for dhcp:
Running lspci: 02:01.0 Ethernet controller: Red Hat, Inc Virtio network
device
I'm not sure whether it worked.

     For disk device,* when I change the address-type to pci, the whole
qemu command is :*
https://paste.fedoraproject.org/409553/,  but the VM can not boot
successfully. Does Qemu not support device disk of virtio-pci in AArch64
just as it in X86_64?
     Thanks~Since I am not very familiar with Qemu, really looking forward
to your response.

Best Regards,
Kevin Zhao

Re: [Qemu-devel] Help: Does Qemu support virtio-pci for net-device and disk device?

[Andrew Jones]

On Wed, Aug 17, 2016 at 08:08:11PM +0800, Kevin Zhao wrote:
> Hi all,
>      Now I'm investigating net device hot plug and disk hotplug for
> AArch64. For virtio , the default address is virtio-mmio. After Libvirt
> 1.3.5, user can explicitly specify the address-type to pci and so libvirt
> will pass the virtio-pci parameters to the Qemu.
>      Both my host and guest OS is Debian8, and Qemu version is 2.6.0.
> Libvirt version is 1.3.5.
>      For net-device, I change the address-type to pci, and libvirt pass the
> command below:
>      -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:0d:25:25,bus=pci.2,addr=0x1
> 
>      After booting, the eth0 device disappear(eth0 occur when the address
> is virtio-mmio),
> but I can find another net-device enp2s1, also it can't work for dhcp:
> Running lspci: 02:01.0 Ethernet controller: Red Hat, Inc Virtio network
> device
> I'm not sure whether it worked.
> 
>      For disk device,* when I change the address-type to pci, the whole
> qemu command is :*
> https://paste.fedoraproject.org/409553/,  but the VM can not boot
> successfully. Does Qemu not support device disk of virtio-pci in AArch64
> just as it in X86_64?
>      Thanks~Since I am not very familiar with Qemu, really looking forward
> to your response.
> 
> Best Regards,
> Kevin Zhao

libvirt 1.3.5 is a bit old. Later versions no longer unconditionally add
the i82801b11 bridge, which was necessary to use PCI devices with the PCIe
host bridge mach-virt has. IMO, libvirt and qemu still have a long way to
go in order to configure a base/standard mach-virt PCIe machine.

1) If we want to support both PCIe devices and PCI, then things are messy.
   Currently we propose dropping PCI support. mach-virt pretty much
   exclusively uses virtio, which can be set to PCIe mode (virtio-1.0)
2) root complex ports, switches (upstream/downstream ports) are currently
   based on Intel parts. Marcel is thinking about creating generic models.
3) libvirt needs to learn how to plug everything together, in proper PCIe
   fashion, leaving holes for hotplug.
4) Probably more... I forget all the different issues we discovered when
   we started playing with this a few months ago.

The good news is that x86 folk want all the same things for the q35 model.
mach-virt enthusiasts like us get to ride along pretty much for free.

So, using virtio-pci with mach-virt and libvirt isn't possible right now,
not without manual changes to the XML. It might be nice to document how to
manually convert a guest, so developers who want to use virtio-pci don't
have to abandon libvirt. I'd have to look into that, or ask one of our
libvirt friends to help. Certainly the instructions would be for latest
libvirt though.

Finally, FWIW, with a guest kernel of 4.6.4-301.fc24.aarch64. The
following qemu command line works for me.
(notice the use of PCIe), and my network interface gets labeled enp0s1.

$QEMU -machine virt-2.6,accel=kvm -cpu host \
 -m 1024 -smp 1 -nographic \
 -bios /usr/share/AAVMF/AAVMF_CODE.fd \
 -device ioh3420,bus=pcie.0,id=pcie.1,port=1,chassis=1 \
 -device ioh3420,bus=pcie.0,id=pcie.2,port=2,chassis=2 \
 -device virtio-scsi-pci,disable-modern=off,disable-legacy=on,bus=pcie.1,addr=00.0,id=scsi0 \
 -drive file=/home/drjones/.local/libvirt/images/fedora.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0 \
 -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 \
 -netdev user,id=hostnet0 \
 -device virtio-net-pci,disable-modern=off,disable-legacy=on,bus=pcie.2,addr=00.0,netdev=hostnet0,id=net0

I prefer always using virtio-scsi for the disk, but a similar command
line can be used for a virtio-blk-pci disk.

Thanks,
drew

Re: [Qemu-devel] Help: Does Qemu support virtio-pci for net-device and disk device?

[Andrea Bolognani]

On Wed, 2016-08-17 at 18:13 +0200, Andrew Jones wrote:
> On Wed, Aug 17, 2016 at 08:08:11PM +0800, Kevin Zhao wrote:
> > 
> > Hi all,
> >      Now I'm investigating net device hot plug and disk hotplug for
> > AArch64. For virtio , the default address is virtio-mmio. After Libvirt
> > 1.3.5, user can explicitly specify the address-type to pci and so libvirt
> > will pass the virtio-pci parameters to the Qemu.
> >      Both my host and guest OS is Debian8, and Qemu version is 2.6.0.
> > Libvirt version is 1.3.5.
> >      For net-device, I change the address-type to pci, and libvirt pass the
> > command below:
> >      -device
> > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:0d:25:25,bus=pci.2,addr=0x1
> > 
> >      After booting, the eth0 device disappear(eth0 occur when the address
> > is virtio-mmio),
> > but I can find another net-device enp2s1, also it can't work for dhcp:
> > Running lspci: 02:01.0 Ethernet controller: Red Hat, Inc Virtio network
> > device
> > I'm not sure whether it worked.
> > 
> >      For disk device,* when I change the address-type to pci, the whole
> > qemu command is :*
> > https://paste.fedoraproject.org/409553/,  but the VM can not boot
> > successfully. Does Qemu not support device disk of virtio-pci in AArch64
> > just as it in X86_64?
> >      Thanks~Since I am not very familiar with Qemu, really looking forward
> > to your response.
> > 
> > Best Regards,
> > Kevin Zhao
> 
> libvirt 1.3.5 is a bit old. Later versions no longer unconditionally add
> the i82801b11 bridge, which was necessary to use PCI devices with the PCIe
> host bridge mach-virt has. IMO, libvirt and qemu still have a long way to
> go in order to configure a base/standard mach-virt PCIe machine.

Debian 8, the guest OS Kevin is trying to boot, is even older,
and in particular it doesn't have any virtio-pci support.

By the way, the same issue was raised on the libvirt list as
well

  https://www.redhat.com/archives/libvir-list/2016-August/msg00854.html

and there's some more information there.

> 1) If we want to support both PCIe devices and PCI, then things are messy.
>    Currently we propose dropping PCI support. mach-virt pretty much
>    exclusively uses virtio, which can be set to PCIe mode (virtio-1.0)
> 2) root complex ports, switches (upstream/downstream ports) are currently
>    based on Intel parts. Marcel is thinking about creating generic models.

Huge +1 from me! Way to go, Marcel! :)

> 3) libvirt needs to learn how to plug everything together, in proper PCIe
>    fashion, leaving holes for hotplug.

Work on this front is ongoing on the libvirt front as we speak.

> 4) Probably more... I forget all the different issues we discovered when
>    we started playing with this a few months ago.
> 
> The good news is that x86 folk want all the same things for the q35 model.
> mach-virt enthusiasts like us get to ride along pretty much for free.
> 
> So, using virtio-pci with mach-virt and libvirt isn't possible right now,
> not without manual changes to the XML. It might be nice to document how to
> manually convert a guest, so developers who want to use virtio-pci don't
> have to abandon libvirt. I'd have to look into that, or ask one of our
> libvirt friends to help. Certainly the instructions would be for latest
> libvirt though.

Things are very much in flux, though, so I'm not entirely sure
putting out any sort of official document would be a good idea
right now. We'll definitely help eg. through the mailing lists
and similar channels, but committing any configuration to a
more static media seems premature.

> Finally, FWIW, with a guest kernel of 4.6.4-301.fc24.aarch64. The
> following qemu command line works for me.
> (notice the use of PCIe), and my network interface gets labeled enp0s1.
> 
> $QEMU -machine virt-2.6,accel=kvm -cpu host \
>  -m 1024 -smp 1 -nographic \
>  -bios /usr/share/AAVMF/AAVMF_CODE.fd \
>  -device ioh3420,bus=pcie.0,id=pcie.1,port=1,chassis=1 \
>  -device ioh3420,bus=pcie.0,id=pcie.2,port=2,chassis=2 \
>  -device virtio-scsi-pci,disable-modern=off,disable-legacy=on,bus=pcie.1,addr=00.0,id=scsi0 \
>  -drive file=/home/drjones/.local/libvirt/images/fedora.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0 \
>  -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 \
>  -netdev user,id=hostnet0 \
>  -device virtio-net-pci,disable-modern=off,disable-legacy=on,bus=pcie.2,addr=00.0,netdev=hostnet0,id=net0
> 
> I prefer always using virtio-scsi for the disk, but a similar command
> line can be used for a virtio-blk-pci disk.

Does the same command line work if you don't specify any of
the disable-* options?

I'm asking because I tried running a Fedora 24 guest through
libvirt, which doesn't support those options yet, and I get

  virtio_blk virtio2: virtio: device uses modern interface but
                              does not have VIRTIO_F_VERSION_1
  virtio_blk: probe of virtio2 failed with error -22

Isn't the default for 2.6 disable-modern=off,
disable-legacy=off? Or was that 2.7? I tried both anyway ;)

-- 
Andrea Bolognani / Red Hat / Virtualization

Re: [Qemu-devel] Help: Does Qemu support virtio-pci for net-device and disk device?

[Laine Stump]

On 08/17/2016 12:13 PM, Andrew Jones wrote:
> On Wed, Aug 17, 2016 at 08:08:11PM +0800, Kevin Zhao wrote:
>> Hi all,
>>       Now I'm investigating net device hot plug and disk hotplug for
>> AArch64. For virtio , the default address is virtio-mmio. After Libvirt
>> 1.3.5, user can explicitly specify the address-type to pci and so libvirt
>> will pass the virtio-pci parameters to the Qemu.
>>       Both my host and guest OS is Debian8, and Qemu version is 2.6.0.
>> Libvirt version is 1.3.5.
>>       For net-device, I change the address-type to pci, and libvirt pass the
>> command below:
>>       -device
>> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:0d:25:25,bus=pci.2,addr=0x1
>>
>>       After booting, the eth0 device disappear(eth0 occur when the address
>> is virtio-mmio),
>> but I can find another net-device enp2s1, also it can't work for dhcp:
>> Running lspci: 02:01.0 Ethernet controller: Red Hat, Inc Virtio network
>> device
>> I'm not sure whether it worked.
>>
>>       For disk device,* when I change the address-type to pci, the whole
>> qemu command is :*
>> https://paste.fedoraproject.org/409553/,  but the VM can not boot
>> successfully. Does Qemu not support device disk of virtio-pci in AArch64
>> just as it in X86_64?
>>       Thanks~Since I am not very familiar with Qemu, really looking forward
>> to your response.
>>
>> Best Regards,
>> Kevin Zhao
> libvirt 1.3.5 is a bit old. Later versions no longer unconditionally add
> the i82801b11 bridge, which was necessary to use PCI devices with the PCIe
> host bridge mach-virt has. IMO, libvirt and qemu still have a long way to
> go in order to configure a base/standard mach-virt PCIe machine.

Well, you can do it now, but you have to manually assign the PCI 
addresses of devices (and if you want hotplug you need to live with 
Intel/TI-specific PCIe controllers).


>
> 1) If we want to support both PCIe devices and PCI, then things are messy.
>     Currently we propose dropping PCI support. mach-virt pretty much
>     exclusively uses virtio, which can be set to PCIe mode (virtio-1.0)

I have a libvirt patch just about ACKed for pushing upstream that will 
automatically assign virtio-pci devices to a PCIe slot (if the qemu 
binary supports virtio-1.0):

https://www.redhat.com/archives/libvir-list/2016-August/msg00852.html

Separate patches do the same for the e1000e emulated network device 
(which you probably don't care about) and the nec-usb-xhci (USB3) 
controller (more useful):

https://www.redhat.com/archives/libvir-list/2016-August/msg00732.html

Once these are in place, the only type of device of any consequence that 
I can see still having no PCIe alternative is audio (even though only 
the virgl video device is PCIe, libvirt has always assigned the primary 
video to slot 1 on pcie-root anyway (although you shouldn't put a legacy 
PCI device on a pcie-root-port or pcie-switch-downstream-port, it is 
acceptable to plug it directly into pcie-root (as long as you know you 
won't need to hotplug it).

> 2) root complex ports, switches (upstream/downstream ports) are currently
>     based on Intel parts. Marcel is thinking about creating generic models.

I say this every time it comes up, so just to be consistent: +1 :-)

> 3) libvirt needs to learn how to plug everything together, in proper PCIe
>     fashion, leaving holes for hotplug.

See above about virtio, although that doesn't cover the whole story. The 
other part (which I'm working on right now) is that libvirt needs to 
automatically add pcie-root-port, pcie-switch-upstream-port, and 
pcie-switch-downstream-port devices as necessary. With the patches I 
mentioned above, you still have to manually add enough pcie-*-port 
controllers to the config, and then libvirt will plug the PCIe devices 
into the right place. This is simple enough to do, but it does require 
intervention.

As far as leaving holes for hotplug, there's actually still a bit of an 
open question there - with machinetypes that use only legacy-PCI, *all* 
slots are hotpluggable, and they're added 31 at a time, so there was 
never any question about which slots were hotpluggable, and it would be 
very rare to end up with a configuration that had 0 free slots available 
for hotplug (actually libvirt would always make sure there was at least 
one, but in practice there would be many more open slots). With 
PCIe-capable machinetypes that is changed, since the root complex 
(pcie-root) doesn't support hotplug, and new slots are added 1 at a time 
(pcie-*-port) rather than 31 at a time. This means you have to really go 
out of your way if you want open slots for hotplug (and even if you want 
devices in the machine at boot time to be hot-unpluggable).

I'm still not sure just how far we need to go in this regard.  We've 
already decided that, unless manually set to an address on pcie-root by 
the user/management application, all PCI devices will be auto-assigned 
to a slot that supports hotplug. What I'm not sure about is whether we 
should always auto-add an extra pcie-*-root to be sure a device can be 
hotplugged, or if we should admit that 1 available slot isn't good 
enough for all situations, so we should instead just leave it up to the 
user/management to manually add extra ports if they think they'll want 
to hotplug something later.

> 4) Probably more... I forget all the different issues we discovered when
>     we started playing with this a few months ago.
>
> The good news is that x86 folk want all the same things for the q35 model.
> mach-virt enthusiasts like us get to ride along pretty much for free.
>
> So, using virtio-pci with mach-virt and libvirt isn't possible right now,
> not without manual changes to the XML. It might be nice to document how to
> manually convert a guest, so developers who want to use virtio-pci don't
> have to abandon libvirt. I'd have to look into that, or ask one of our
> libvirt friends to help. Certainly the instructions would be for latest
> libvirt though.
>
> Finally, FWIW, with a guest kernel of 4.6.4-301.fc24.aarch64. The
> following qemu command line works for me.
> (notice the use of PCIe), and my network interface gets labeled enp0s1.
>
> $QEMU -machine virt-2.6,accel=kvm -cpu host \
>   -m 1024 -smp 1 -nographic \
>   -bios /usr/share/AAVMF/AAVMF_CODE.fd \
>   -device ioh3420,bus=pcie.0,id=pcie.1,port=1,chassis=1 \
>   -device ioh3420,bus=pcie.0,id=pcie.2,port=2,chassis=2 \
>   -device virtio-scsi-pci,disable-modern=off,disable-legacy=on,bus=pcie.1,addr=00.0,id=scsi0 \
>   -drive file=/home/drjones/.local/libvirt/images/fedora.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0 \
>   -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 \
>   -netdev user,id=hostnet0 \
>   -device virtio-net-pci,disable-modern=off,disable-legacy=on,bus=pcie.2,addr=00.0,netdev=hostnet0,id=net0
>
> I prefer always using virtio-scsi for the disk, but a similar command
> line can be used for a virtio-blk-pci disk.
>
> Thanks,
> drew

Re: [Qemu-devel] Help: Does Qemu support virtio-pci for net-device and disk device?

[Andrew Jones]

On Wed, Aug 17, 2016 at 06:41:33PM +0200, Andrea Bolognani wrote:
> On Wed, 2016-08-17 at 18:13 +0200, Andrew Jones wrote:
> > On Wed, Aug 17, 2016 at 08:08:11PM +0800, Kevin Zhao wrote:
> > > 
> > > Hi all,
> > >      Now I'm investigating net device hot plug and disk hotplug for
> > > AArch64. For virtio , the default address is virtio-mmio. After Libvirt
> > > 1.3.5, user can explicitly specify the address-type to pci and so libvirt
> > > will pass the virtio-pci parameters to the Qemu.
> > >      Both my host and guest OS is Debian8, and Qemu version is 2.6.0.
> > > Libvirt version is 1.3.5.
> > >      For net-device, I change the address-type to pci, and libvirt pass the
> > > command below:
> > >      -device
> > > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:0d:25:25,bus=pci.2,addr=0x1
> > > 
> > >      After booting, the eth0 device disappear(eth0 occur when the address
> > > is virtio-mmio),
> > > but I can find another net-device enp2s1, also it can't work for dhcp:
> > > Running lspci: 02:01.0 Ethernet controller: Red Hat, Inc Virtio network
> > > device
> > > I'm not sure whether it worked.
> > > 
> > >      For disk device,* when I change the address-type to pci, the whole
> > > qemu command is :*
> > > https://paste.fedoraproject.org/409553/,  but the VM can not boot
> > > successfully. Does Qemu not support device disk of virtio-pci in AArch64
> > > just as it in X86_64?
> > >      Thanks~Since I am not very familiar with Qemu, really looking forward
> > > to your response.
> > > 
> > > Best Regards,
> > > Kevin Zhao
> > 
> > libvirt 1.3.5 is a bit old. Later versions no longer unconditionally add
> > the i82801b11 bridge, which was necessary to use PCI devices with the PCIe
> > host bridge mach-virt has. IMO, libvirt and qemu still have a long way to
> > go in order to configure a base/standard mach-virt PCIe machine.
> 
> Debian 8, the guest OS Kevin is trying to boot, is even older,
> and in particular it doesn't have any virtio-pci support.
> 
> By the way, the same issue was raised on the libvirt list as
> well
> 
>   https://www.redhat.com/archives/libvir-list/2016-August/msg00854.html
> 
> and there's some more information there.
> 
> > 1) If we want to support both PCIe devices and PCI, then things are messy.
> >    Currently we propose dropping PCI support. mach-virt pretty much
> >    exclusively uses virtio, which can be set to PCIe mode (virtio-1.0)
> > 2) root complex ports, switches (upstream/downstream ports) are currently
> >    based on Intel parts. Marcel is thinking about creating generic models.
> 
> Huge +1 from me! Way to go, Marcel! :)
> 
> > 3) libvirt needs to learn how to plug everything together, in proper PCIe
> >    fashion, leaving holes for hotplug.
> 
> Work on this front is ongoing on the libvirt front as we speak.
> 
> > 4) Probably more... I forget all the different issues we discovered when
> >    we started playing with this a few months ago.
> > 
> > The good news is that x86 folk want all the same things for the q35 model.
> > mach-virt enthusiasts like us get to ride along pretty much for free.
> > 
> > So, using virtio-pci with mach-virt and libvirt isn't possible right now,
> > not without manual changes to the XML. It might be nice to document how to
> > manually convert a guest, so developers who want to use virtio-pci don't
> > have to abandon libvirt. I'd have to look into that, or ask one of our
> > libvirt friends to help. Certainly the instructions would be for latest
> > libvirt though.
> 
> Things are very much in flux, though, so I'm not entirely sure
> putting out any sort of official document would be a good idea
> right now. We'll definitely help eg. through the mailing lists
> and similar channels, but committing any configuration to a
> more static media seems premature.

Understood, and based on Laine's response, things are looking just
around the corner now anyway, so forget I asked :-)

> 
> > Finally, FWIW, with a guest kernel of 4.6.4-301.fc24.aarch64. The
> > following qemu command line works for me.
> > (notice the use of PCIe), and my network interface gets labeled enp0s1.
> > 
> > $QEMU -machine virt-2.6,accel=kvm -cpu host \
> >  -m 1024 -smp 1 -nographic \
> >  -bios /usr/share/AAVMF/AAVMF_CODE.fd \
> >  -device ioh3420,bus=pcie.0,id=pcie.1,port=1,chassis=1 \
> >  -device ioh3420,bus=pcie.0,id=pcie.2,port=2,chassis=2 \
> >  -device virtio-scsi-pci,disable-modern=off,disable-legacy=on,bus=pcie.1,addr=00.0,id=scsi0 \
> >  -drive file=/home/drjones/.local/libvirt/images/fedora.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0 \
> >  -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 \
> >  -netdev user,id=hostnet0 \
> >  -device virtio-net-pci,disable-modern=off,disable-legacy=on,bus=pcie.2,addr=00.0,netdev=hostnet0,id=net0
> > 
> > I prefer always using virtio-scsi for the disk, but a similar command
> > line can be used for a virtio-blk-pci disk.
> 
> Does the same command line work if you don't specify any of
> the disable-* options?
> 
> I'm asking because I tried running a Fedora 24 guest through
> libvirt, which doesn't support those options yet, and I get
> 
>   virtio_blk virtio2: virtio: device uses modern interface but
>                               does not have VIRTIO_F_VERSION_1
>   virtio_blk: probe of virtio2 failed with error -22

Doesn't work for me either. I can only boot with disable-modern=off,
disable-legacy=on (at least when building my config the way I try to
build it...) I presume that's a guest kernel issue.

> 
> Isn't the default for 2.6 disable-modern=off,
> disable-legacy=off? Or was that 2.7? I tried both anyway ;)

Dunno. With the command line getting longer all the time, I just
have a script that generates one that works for me, and haven't
worried much about the defaults...

Thanks,
drew

Re: [Qemu-devel] Help: Does Qemu support virtio-pci for net-device and disk device?

[Andrew Jones]

On Wed, Aug 17, 2016 at 01:00:05PM -0400, Laine Stump wrote:
> On 08/17/2016 12:13 PM, Andrew Jones wrote:
> > On Wed, Aug 17, 2016 at 08:08:11PM +0800, Kevin Zhao wrote:
> > > Hi all,
> > >       Now I'm investigating net device hot plug and disk hotplug for
> > > AArch64. For virtio , the default address is virtio-mmio. After Libvirt
> > > 1.3.5, user can explicitly specify the address-type to pci and so libvirt
> > > will pass the virtio-pci parameters to the Qemu.
> > >       Both my host and guest OS is Debian8, and Qemu version is 2.6.0.
> > > Libvirt version is 1.3.5.
> > >       For net-device, I change the address-type to pci, and libvirt pass the
> > > command below:
> > >       -device
> > > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:0d:25:25,bus=pci.2,addr=0x1
> > > 
> > >       After booting, the eth0 device disappear(eth0 occur when the address
> > > is virtio-mmio),
> > > but I can find another net-device enp2s1, also it can't work for dhcp:
> > > Running lspci: 02:01.0 Ethernet controller: Red Hat, Inc Virtio network
> > > device
> > > I'm not sure whether it worked.
> > > 
> > >       For disk device,* when I change the address-type to pci, the whole
> > > qemu command is :*
> > > https://paste.fedoraproject.org/409553/,  but the VM can not boot
> > > successfully. Does Qemu not support device disk of virtio-pci in AArch64
> > > just as it in X86_64?
> > >       Thanks~Since I am not very familiar with Qemu, really looking forward
> > > to your response.
> > > 
> > > Best Regards,
> > > Kevin Zhao
> > libvirt 1.3.5 is a bit old. Later versions no longer unconditionally add
> > the i82801b11 bridge, which was necessary to use PCI devices with the PCIe
> > host bridge mach-virt has. IMO, libvirt and qemu still have a long way to
> > go in order to configure a base/standard mach-virt PCIe machine.
> 
> Well, you can do it now, but you have to manually assign the PCI addresses
> of devices (and if you want hotplug you need to live with Intel/TI-specific
> PCIe controllers).
> 
> 
> > 
> > 1) If we want to support both PCIe devices and PCI, then things are messy.
> >     Currently we propose dropping PCI support. mach-virt pretty much
> >     exclusively uses virtio, which can be set to PCIe mode (virtio-1.0)
> 
> I have a libvirt patch just about ACKed for pushing upstream that will
> automatically assign virtio-pci devices to a PCIe slot (if the qemu binary
> supports virtio-1.0):
> 
> https://www.redhat.com/archives/libvir-list/2016-August/msg00852.html
> 
> Separate patches do the same for the e1000e emulated network device (which
> you probably don't care about) and the nec-usb-xhci (USB3) controller (more
> useful):
> 
> https://www.redhat.com/archives/libvir-list/2016-August/msg00732.html
> 

Thanks for the update Laine. This sounds great to me. With those patches
we can switch from virtio-mmio to virtio-pci easily, even if we're still
missing hotplug a bit longer. What limit do we have for the number of
devices, when we don't have any switches? I think I experimented once and
found it to be 7.

> Once these are in place, the only type of device of any consequence that I
> can see still having no PCIe alternative is audio (even though only the
> virgl video device is PCIe, libvirt has always assigned the primary video to
> slot 1 on pcie-root anyway (although you shouldn't put a legacy PCI device
> on a pcie-root-port or pcie-switch-downstream-port, it is acceptable to plug
> it directly into pcie-root (as long as you know you won't need to hotplug
> it).
> 
> > 2) root complex ports, switches (upstream/downstream ports) are currently
> >     based on Intel parts. Marcel is thinking about creating generic models.
> 
> I say this every time it comes up, so just to be consistent: +1 :-)
> 
> > 3) libvirt needs to learn how to plug everything together, in proper PCIe
> >     fashion, leaving holes for hotplug.
> 
> See above about virtio, although that doesn't cover the whole story. The
> other part (which I'm working on right now) is that libvirt needs to
> automatically add pcie-root-port, pcie-switch-upstream-port, and
> pcie-switch-downstream-port devices as necessary. With the patches I
> mentioned above, you still have to manually add enough pcie-*-port
> controllers to the config, and then libvirt will plug the PCIe devices into
> the right place. This is simple enough to do, but it does require
> intervention.

OK, so we want this to support hotplug and eventually chain switches,
bumping our device limit up higher and higher. To what? I'm not sure,
I guess we're still limited by address space.

> 
> As far as leaving holes for hotplug, there's actually still a bit of an open
> question there - with machinetypes that use only legacy-PCI, *all* slots are
> hotpluggable, and they're added 31 at a time, so there was never any
> question about which slots were hotpluggable, and it would be very rare to
> end up with a configuration that had 0 free slots available for hotplug
> (actually libvirt would always make sure there was at least one, but in
> practice there would be many more open slots). With PCIe-capable
> machinetypes that is changed, since the root complex (pcie-root) doesn't
> support hotplug, and new slots are added 1 at a time (pcie-*-port) rather
> than 31 at a time. This means you have to really go out of your way if you
> want open slots for hotplug (and even if you want devices in the machine at
> boot time to be hot-unpluggable).
> 
> I'm still not sure just how far we need to go in this regard.  We've already
> decided that, unless manually set to an address on pcie-root by the
> user/management application, all PCI devices will be auto-assigned to a slot
> that supports hotplug. What I'm not sure about is whether we should always
> auto-add an extra pcie-*-root to be sure a device can be hotplugged, or if
> we should admit that 1 available slot isn't good enough for all situations,
> so we should instead just leave it up to the user/management to manually add
> extra ports if they think they'll want to hotplug something later.

Hmm... Maybe the tools can make this easier by offering an option to
provide N extra ports.

Hmm2... I think I agree that we don't need to worry too much about
providing free ports for hotplug (maybe just one for fun). With
virtio-scsi we can plug disks already. If we want to provide multiple
virtio-net devices for the price of one port, we can enable mutlifunction.
And, iirc, there's work to get ARI functioning, allowing mutlifunction to
go nuts (assuming I understand its purpose correctly)

So maybe the default config just needs 3 ports?
 1 virtio-scsi with as many disks as requested
 1 virtio-net with as many functions as nics are requested
 1 extra port

Thanks,
drew

Re: [Qemu-devel] Help: Does Qemu support virtio-pci for net-device and disk device?

[Marcel Apfelbaum]

On 08/17/2016 08:00 PM, Laine Stump wrote:
> On 08/17/2016 12:13 PM, Andrew Jones wrote:
>> On Wed, Aug 17, 2016 at 08:08:11PM +0800, Kevin Zhao wrote:
>>> Hi all,
[...]

Hi,

>>
>> 1) If we want to support both PCIe devices and PCI, then things are messy.
>>     Currently we propose dropping PCI support. mach-virt pretty much
>>     exclusively uses virtio, which can be set to PCIe mode (virtio-1.0)
>
> I have a libvirt patch just about ACKed for pushing upstream that will automatically assign virtio-pci devices to a PCIe slot (if the qemu binary supports virtio-1.0):
>
> https://www.redhat.com/archives/libvir-list/2016-August/msg00852.html
>
> Separate patches do the same for the e1000e emulated network device (which you probably don't care about) and the nec-usb-xhci (USB3) controller (more useful):
>
> https://www.redhat.com/archives/libvir-list/2016-August/msg00732.html
>
> Once these are in place, the only type of device of any consequence that I can see still having no PCIe alternative is audio (even though only the virgl video device is PCIe, libvirt has always
> assigned the primary video to slot 1 on pcie-root anyway (although you shouldn't put a legacy PCI device on a pcie-root-port or pcie-switch-downstream-port, it is acceptable to plug it directly into
> pcie-root (as long as you know you won't need to hotplug it).
>

I agree, please don't allow plugging PCI devices into PCIe ports (root ports/downstream ports). Use pcie.0 bus as you mentioned or start a pci "hierarchy" with a dmi-pci and pci-pci bridge.

>> 2) root complex ports, switches (upstream/downstream ports) are currently
>>     based on Intel parts. Marcel is thinking about creating generic models.
>
> I say this every time it comes up, so just to be consistent: +1 :-)
>

I plan do that for my "own selfish" reasons, one of them is to be able to specify
the IO/MEM range the guest firmware should assign to a PCIe port. But, of course,
I'll keep them neutral enough; I might need help with testing the mach-virt machines.

>> 3) libvirt needs to learn how to plug everything together, in proper PCIe
>>     fashion, leaving holes for hotplug.
>
> See above about virtio, although that doesn't cover the whole story. The other part (which I'm working on right now) is that libvirt needs to automatically add pcie-root-port,
> pcie-switch-upstream-port, and pcie-switch-downstream-port devices as necessary. With the patches I mentioned above, you still have to manually add enough pcie-*-port controllers to the config, and
> then libvirt will plug the PCIe devices into the right place. This is simple enough to do, but it does require intervention.
>
> As far as leaving holes for hotplug, there's actually still a bit of an open question there - with machinetypes that use only legacy-PCI, *all* slots are hotpluggable, and they're added 31 at a time,
> so there was never any question about which slots were hotpluggable, and it would be very rare to end up with a configuration that had 0 free slots available for hotplug (actually libvirt would always
> make sure there was at least one, but in practice there would be many more open slots). With PCIe-capable machinetypes that is changed, since the root complex (pcie-root) doesn't support hotplug, and
> new slots are added 1 at a time (pcie-*-port) rather than 31 at a time. This means you have to really go out of your way if you want open slots for hotplug (and even if you want devices in the machine
> at boot time to be hot-unpluggable).
>

I agree you need to take into account hotplug for Q35, by leaving some PCIe root ports available, but it shouldn't be too much trouble.

> I'm still not sure just how far we need to go in this regard.  We've already decided that, unless manually set to an address on pcie-root by the user/management application, all PCI devices will be
> auto-assigned to a slot that supports hotplug.

Good idea.

  What I'm not sure about is whether we should always auto-add an extra pcie-*-root to be sure a device can be hotplugged, or if we should admit that 1
> available slot isn't good enough for all situations, so we should instead just leave it up to the user/management to manually add extra ports if they think they'll want to hotplug something later.

Why not? Leaving 1 or 2 PCIe ports should be enough. On each port you can hotplug a switch with several downstream ports. You can continue nesting the switches up to depth of 6-7 I think.

Thanks,
Marcel

>
[...]
>>
>> Thanks,
>> drew
>
>

Re: [Qemu-devel] Help: Does Qemu support virtio-pci for net-device and disk device?

[Kevin Zhao]

Hi Jones:
   Thanks~It is great that Qemu has been working on that :-)

On 18 August 2016 at 00:13, Andrew Jones <drjones@redhat.com> wrote:

> On Wed, Aug 17, 2016 at 08:08:11PM +0800, Kevin Zhao wrote:
> > Hi all,
> >      Now I'm investigating net device hot plug and disk hotplug for
> > AArch64. For virtio , the default address is virtio-mmio. After Libvirt
> > 1.3.5, user can explicitly specify the address-type to pci and so libvirt
> > will pass the virtio-pci parameters to the Qemu.
> >      Both my host and guest OS is Debian8, and Qemu version is 2.6.0.
> > Libvirt version is 1.3.5.
> >      For net-device, I change the address-type to pci, and libvirt pass
> the
> > command below:
> >      -device
> > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:
> 0d:25:25,bus=pci.2,addr=0x1
> >
> >      After booting, the eth0 device disappear(eth0 occur when the address
> > is virtio-mmio),
> > but I can find another net-device enp2s1, also it can't work for dhcp:
> > Running lspci: 02:01.0 Ethernet controller: Red Hat, Inc Virtio network
> > device
> > I'm not sure whether it worked.
> >
> >      For disk device,* when I change the address-type to pci, the whole
> > qemu command is :*
> > https://paste.fedoraproject.org/409553/,  but the VM can not boot
> > successfully. Does Qemu not support device disk of virtio-pci in AArch64
> > just as it in X86_64?
> >      Thanks~Since I am not very familiar with Qemu, really looking
> forward
> > to your response.
> >
> > Best Regards,
> > Kevin Zhao
>
> libvirt 1.3.5 is a bit old. Later versions no longer unconditionally add
> the i82801b11 bridge, which was necessary to use PCI devices with the PCIe
> host bridge mach-virt has. IMO, libvirt and qemu still have a long way to
> go in order to configure a base/standard mach-virt PCIe machine.
>
>
Yeah, I am changing to libvirt 2.1.0, I find that I should use PCI by
manually add
the slots and bus to it.

1) If we want to support both PCIe devices and PCI, then things are messy.
>    Currently we propose dropping PCI support. mach-virt pretty much
>    exclusively uses virtio, which can be set to PCIe mode (virtio-1.0)
> 2) root complex ports, switches (upstream/downstream ports) are currently
>    based on Intel parts. Marcel is thinking about creating generic models.
> 3) libvirt needs to learn how to plug everything together, in proper PCIe
>    fashion, leaving holes for hotplug.
> 4) Probably more... I forget all the different issues we discovered when
>    we started playing with this a few months ago.
>
> The good news is that x86 folk want all the same things for the q35 model.
> mach-virt enthusiasts like us get to ride along pretty much for free.
>
> So, using virtio-pci with mach-virt and libvirt isn't possible right now,
> not without manual changes to the XML. It might be nice to document how to
> manually convert a guest, so developers who want to use virtio-pci don't
> have to abandon libvirt. I'd have to look into that, or ask one of our
> libvirt friends to help. Certainly the instructions would be for latest
> libvirt though.
>
>
As you said,  that means that I can use PCIe as the bus for disk and
net-device.
I will try using pcie in libvirt. I will try the newest version of libvirt.
Do I need to change <address type = 'pcie'> to enable it in AArch64 ?
The pcie bus will  be automatically assigned in libvirt ?


> Finally, FWIW, with a guest kernel of 4.6.4-301.fc24.aarch64. The
> following qemu command line works for me.
> (notice the use of PCIe), and my network interface gets labeled enp0s1.
>
> $QEMU -machine virt-2.6,accel=kvm -cpu host \
>  -m 1024 -smp 1 -nographic \
>  -bios /usr/share/AAVMF/AAVMF_CODE.fd \
>  -device ioh3420,bus=pcie.0,id=pcie.1,port=1,chassis=1 \
>  -device ioh3420,bus=pcie.0,id=pcie.2,port=2,chassis=2 \
>  -device virtio-scsi-pci,disable-modern=off,disable-legacy=on,bus=pcie.1,addr=00.0,id=scsi0
> \
>  -drive file=/home/drjones/.local/libvirt/images/fedora.qcow2,
> format=qcow2,if=none,id=drive-scsi0-0-0-0 \
>  -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-
> scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 \
>  -netdev user,id=hostnet0 \
>  -device virtio-net-pci,disable-modern=off,disable-legacy=on,bus=
> pcie.2,addr=00.0,netdev=hostnet0,id=net0
>
> I prefer always using virtio-scsi for the disk, but a similar command
> line can be used for a virtio-blk-pci disk.
>
> OK great! Because in Openstack Nova ,AArch64 need to realize the hotplug
only with
the virtio bus, so investigate the virtio-pci. :-)
Thanks again!


> Thanks,
> drew
>

Re: [Qemu-devel] Help: Does Qemu support virtio-pci for net-device and disk device?

[Kevin Zhao]

Hi Laine,
    Thanks :-) I also has a little questions below.

On 18 August 2016 at 01:00, Laine Stump <laine@redhat.com> wrote:

> On 08/17/2016 12:13 PM, Andrew Jones wrote:
>
>> On Wed, Aug 17, 2016 at 08:08:11PM +0800, Kevin Zhao wrote:
>>
>>> Hi all,
>>>       Now I'm investigating net device hot plug and disk hotplug for
>>> AArch64. For virtio , the default address is virtio-mmio. After Libvirt
>>> 1.3.5, user can explicitly specify the address-type to pci and so libvirt
>>> will pass the virtio-pci parameters to the Qemu.
>>>       Both my host and guest OS is Debian8, and Qemu version is 2.6.0.
>>> Libvirt version is 1.3.5.
>>>       For net-device, I change the address-type to pci, and libvirt pass
>>> the
>>> command below:
>>>       -device
>>> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:0d:25:
>>> 25,bus=pci.2,addr=0x1
>>>
>>>       After booting, the eth0 device disappear(eth0 occur when the
>>> address
>>> is virtio-mmio),
>>> but I can find another net-device enp2s1, also it can't work for dhcp:
>>> Running lspci: 02:01.0 Ethernet controller: Red Hat, Inc Virtio network
>>> device
>>> I'm not sure whether it worked.
>>>
>>>       For disk device,* when I change the address-type to pci, the whole
>>> qemu command is :*
>>> https://paste.fedoraproject.org/409553/,  but the VM can not boot
>>> successfully. Does Qemu not support device disk of virtio-pci in AArch64
>>> just as it in X86_64?
>>>       Thanks~Since I am not very familiar with Qemu, really looking
>>> forward
>>> to your response.
>>>
>>> Best Regards,
>>> Kevin Zhao
>>>
>> libvirt 1.3.5 is a bit old. Later versions no longer unconditionally add
>> the i82801b11 bridge, which was necessary to use PCI devices with the PCIe
>> host bridge mach-virt has. IMO, libvirt and qemu still have a long way to
>> go in order to configure a base/standard mach-virt PCIe machine.
>>
>
> Well, you can do it now, but you have to manually assign the PCI addresses
> of devices (and if you want hotplug you need to live with Intel/TI-specific
> PCIe controllers).

OK. It seems that Qemu will drop PCI for machine-virt and turning to PCIE
in the future.
Do I need to do more for  Intel/TI-specific PCIe controllers?  what do I
need to add in the guest XML or more?

>
>
>> 1) If we want to support both PCIe devices and PCI, then things are messy.
>>     Currently we propose dropping PCI support. mach-virt pretty much
>>     exclusively uses virtio, which can be set to PCIe mode (virtio-1.0)
>>
>
> I have a libvirt patch just about ACKed for pushing upstream that will
> automatically assign virtio-pci devices to a PCIe slot (if the qemu binary
> supports virtio-1.0):
>
> https://www.redhat.com/archives/libvir-list/2016-August/msg00852.html
>

What's the minimum version of  Qemu that support virito-1.0? Does Qemu 2.6
works?
Also as  I see your patch for automatically assign virtio-pci to PCIE
slot,after it merged  I think thing will go much more easier.
Now I will manually add the slots and bus to pcie. Because I am not
familiar with it,  if it convenient, could you give me an available xml
file which PCIE disk and PCIE
net device can work for machine virt ?

Thanks~

Separate patches do the same for the e1000e emulated network device (which
> you probably don't care about) and the nec-usb-xhci (USB3) controller (more
> useful):
>
> https://www.redhat.com/archives/libvir-list/2016-August/msg00732.html
>
> Once these are in place, the only type of device of any consequence that I
> can see still having no PCIe alternative is audio (even though only the
> virgl video device is PCIe, libvirt has always assigned the primary video
> to slot 1 on pcie-root anyway (although you shouldn't put a legacy PCI
> device on a pcie-root-port or pcie-switch-downstream-port, it is acceptable
> to plug it directly into pcie-root (as long as you know you won't need to
> hotplug it).
>
> 2) root complex ports, switches (upstream/downstream ports) are currently
>>     based on Intel parts. Marcel is thinking about creating generic
>> models.
>>
>
> I say this every time it comes up, so just to be consistent: +1 :-)
>
> 3) libvirt needs to learn how to plug everything together, in proper PCIe
>>     fashion, leaving holes for hotplug.
>>
>
> See above about virtio, although that doesn't cover the whole story. The
> other part (which I'm working on right now) is that libvirt needs to
> automatically add pcie-root-port, pcie-switch-upstream-port, and
> pcie-switch-downstream-port devices as necessary. With the patches I
> mentioned above, you still have to manually add enough pcie-*-port
> controllers to the config, and then libvirt will plug the PCIe devices into
> the right place. This is simple enough to do, but it does require
> intervention.
>
> As far as leaving holes for hotplug, there's actually still a bit of an
> open question there - with machinetypes that use only legacy-PCI, *all*
> slots are hotpluggable, and they're added 31 at a time, so there was never
> any question about which slots were hotpluggable, and it would be very rare
> to end up with a configuration that had 0 free slots available for hotplug
> (actually libvirt would always make sure there was at least one, but in
> practice there would be many more open slots). With PCIe-capable
> machinetypes that is changed, since the root complex (pcie-root) doesn't
> support hotplug, and new slots are added 1 at a time (pcie-*-port) rather
> than 31 at a time. This means you have to really go out of your way if you
> want open slots for hotplug (and even if you want devices in the machine at
> boot time to be hot-unpluggable).
>
> I'm still not sure just how far we need to go in this regard.  We've
> already decided that, unless manually set to an address on pcie-root by the
> user/management application, all PCI devices will be auto-assigned to a
> slot that supports hotplug. What I'm not sure about is whether we should
> always auto-add an extra pcie-*-root to be sure a device can be hotplugged,
> or if we should admit that 1 available slot isn't good enough for all
> situations, so we should instead just leave it up to the user/management to
> manually add extra ports if they think they'll want to hotplug something
> later.
>
>
> 4) Probably more... I forget all the different issues we discovered when
>>     we started playing with this a few months ago.
>>
>> The good news is that x86 folk want all the same things for the q35 model.
>> mach-virt enthusiasts like us get to ride along pretty much for free.
>>
>> So, using virtio-pci with mach-virt and libvirt isn't possible right now,
>> not without manual changes to the XML. It might be nice to document how to
>> manually convert a guest, so developers who want to use virtio-pci don't
>> have to abandon libvirt. I'd have to look into that, or ask one of our
>> libvirt friends to help. Certainly the instructions would be for latest
>> libvirt though.
>>
>> Finally, FWIW, with a guest kernel of 4.6.4-301.fc24.aarch64. The
>> following qemu command line works for me.
>> (notice the use of PCIe), and my network interface gets labeled enp0s1.
>>
>> $QEMU -machine virt-2.6,accel=kvm -cpu host \
>>   -m 1024 -smp 1 -nographic \
>>   -bios /usr/share/AAVMF/AAVMF_CODE.fd \
>>   -device ioh3420,bus=pcie.0,id=pcie.1,port=1,chassis=1 \
>>   -device ioh3420,bus=pcie.0,id=pcie.2,port=2,chassis=2 \
>>   -device virtio-scsi-pci,disable-modern=off,disable-legacy=on,bus=pcie.1,addr=00.0,id=scsi0
>> \
>>   -drive file=/home/drjones/.local/libvirt/images/fedora.qcow2,format
>> =qcow2,if=none,id=drive-scsi0-0-0-0 \
>>   -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-sc
>> si0-0-0-0,id=scsi0-0-0-0,bootindex=1 \
>>   -netdev user,id=hostnet0 \
>>   -device virtio-net-pci,disable-modern=off,disable-legacy=on,bus=pcie
>> .2,addr=00.0,netdev=hostnet0,id=net0
>>
>> I prefer always using virtio-scsi for the disk, but a similar command
>> line can be used for a virtio-blk-pci disk.
>>
>> Thanks,
>> drew
>>
>
>
>

Re: [Qemu-devel] Help: Does Qemu support virtio-pci for net-device and disk device?

[Kevin Zhao]

Hi All,
    Thanks for your all kindly response. Really Great and helpful :-)

On 18 August 2016 at 20:30, Kevin Zhao <kevin.zhao@linaro.org> wrote:

> Hi Jones:
>    Thanks~It is great that Qemu has been working on that :-)
>
> On 18 August 2016 at 00:13, Andrew Jones <drjones@redhat.com> wrote:
>
>> On Wed, Aug 17, 2016 at 08:08:11PM +0800, Kevin Zhao wrote:
>> > Hi all,
>> >      Now I'm investigating net device hot plug and disk hotplug for
>> > AArch64. For virtio , the default address is virtio-mmio. After Libvirt
>> > 1.3.5, user can explicitly specify the address-type to pci and so
>> libvirt
>> > will pass the virtio-pci parameters to the Qemu.
>> >      Both my host and guest OS is Debian8, and Qemu version is 2.6.0.
>> > Libvirt version is 1.3.5.
>> >      For net-device, I change the address-type to pci, and libvirt pass
>> the
>> > command below:
>> >      -device
>> > virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:0d:25:
>> 25,bus=pci.2,addr=0x1
>> >
>> >      After booting, the eth0 device disappear(eth0 occur when the
>> address
>> > is virtio-mmio),
>> > but I can find another net-device enp2s1, also it can't work for dhcp:
>> > Running lspci: 02:01.0 Ethernet controller: Red Hat, Inc Virtio network
>> > device
>> > I'm not sure whether it worked.
>> >
>> >      For disk device,* when I change the address-type to pci, the whole
>> > qemu command is :*
>> > https://paste.fedoraproject.org/409553/,  but the VM can not boot
>> > successfully. Does Qemu not support device disk of virtio-pci in AArch64
>> > just as it in X86_64?
>> >      Thanks~Since I am not very familiar with Qemu, really looking
>> forward
>> > to your response.
>> >
>> > Best Regards,
>> > Kevin Zhao
>>
>> libvirt 1.3.5 is a bit old. Later versions no longer unconditionally add
>> the i82801b11 bridge, which was necessary to use PCI devices with the PCIe
>> host bridge mach-virt has. IMO, libvirt and qemu still have a long way to
>> go in order to configure a base/standard mach-virt PCIe machine.
>>
>>
> Yeah, I am changing to libvirt 2.1.0, I find that I should use PCI by
> manually add
> the slots and bus to it.
>
> 1) If we want to support both PCIe devices and PCI, then things are messy.
>>    Currently we propose dropping PCI support. mach-virt pretty much
>>    exclusively uses virtio, which can be set to PCIe mode (virtio-1.0)
>> 2) root complex ports, switches (upstream/downstream ports) are currently
>>    based on Intel parts. Marcel is thinking about creating generic models.
>> 3) libvirt needs to learn how to plug everything together, in proper PCIe
>>    fashion, leaving holes for hotplug.
>> 4) Probably more... I forget all the different issues we discovered when
>>    we started playing with this a few months ago.
>>
>> The good news is that x86 folk want all the same things for the q35 model.
>> mach-virt enthusiasts like us get to ride along pretty much for free.
>>
>> So, using virtio-pci with mach-virt and libvirt isn't possible right now,
>> not without manual changes to the XML. It might be nice to document how to
>> manually convert a guest, so developers who want to use virtio-pci don't
>> have to abandon libvirt. I'd have to look into that, or ask one of our
>> libvirt friends to help. Certainly the instructions would be for latest
>> libvirt though.
>>
>>
> As you said,  that means that I can use PCIe as the bus for disk and
> net-device.
> I will try using pcie in libvirt. I will try the newest version of
> libvirt.
> Do I need to change <address type = 'pcie'> to enable it in AArch64 ?
> The pcie bus will  be automatically assigned in libvirt ?
>

Just to remind that I made a mistake here, as Laine said now just manually
adding pcie
slots can work. He is going on push a patch to libvirt about automatically
assigment.

>
>
>> Finally, FWIW, with a guest kernel of 4.6.4-301.fc24.aarch64. The
>> following qemu command line works for me.
>> (notice the use of PCIe), and my network interface gets labeled enp0s1.
>>
>> $QEMU -machine virt-2.6,accel=kvm -cpu host \
>>  -m 1024 -smp 1 -nographic \
>>  -bios /usr/share/AAVMF/AAVMF_CODE.fd \
>>  -device ioh3420,bus=pcie.0,id=pcie.1,port=1,chassis=1 \
>>  -device ioh3420,bus=pcie.0,id=pcie.2,port=2,chassis=2 \
>>  -device virtio-scsi-pci,disable-modern=off,disable-legacy=on,bus=pcie.1,addr=00.0,id=scsi0
>> \
>>  -drive file=/home/drjones/.local/libvirt/images/fedora.qcow2,format
>> =qcow2,if=none,id=drive-scsi0-0-0-0 \
>>  -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-sc
>> si0-0-0-0,id=scsi0-0-0-0,bootindex=1 \
>>  -netdev user,id=hostnet0 \
>>  -device virtio-net-pci,disable-modern=off,disable-legacy=on,bus=pcie
>> .2,addr=00.0,netdev=hostnet0,id=net0
>>
>> I prefer always using virtio-scsi for the disk, but a similar command
>> line can be used for a virtio-blk-pci disk.
>>
>> OK great! Because in Openstack Nova ,AArch64 need to realize the hotplug
> only with
> the virtio bus, so investigate the virtio-pci. :-)
> Thanks again!
>
>
>> Thanks,
>> drew
>>
>
>

Re: [Qemu-devel] Help: Does Qemu support virtio-pci for net-device and disk device?

[Andrea Bolognani]

On Thu, 2016-08-18 at 20:43 +0800, Kevin Zhao wrote:
> What's the minimum version of  Qemu that support virito-1.0?
> Does Qemu 2.6 works? 

2.6 definitely has virtio 1.0 support, however libvirt does
not yet allow you to control whether a device uses 0.9, 1.0
or both. The default for 2.6 should be both IIRC.

> Now I will manually add the slots and bus to pcie. Because
> I am not familiar with it,  if it convenient, could you give
> me an available xml file which PCIE disk and PCIE
> net device can work for machine virt ?

The XML you're looking for is at the end of this message.

Note that a Fedora 24 guest configured this way will not
boot at all if the machine type is virt-2.6; on the other
hand, an identically-configured RHEL 7.3 guest will boot
even with virt-2.6, but both the disk and the network
adapter will be legacy PCI instead of PCIe.


<domain type='kvm'>
  <name>abologna-f24</name>
  <uuid>f6d0428b-a034-4c4e-8ef2-f12f6aa9cab0</uuid>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='aarch64' machine='virt-2.7'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/AAVMF/AAVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/abologna-f24_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <gic version='2'/>
  </features>
  <cpu mode='host-passthrough'/>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/abologna-qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/abologna-f24.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </disk>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='ioh3420'/>
      <target chassis='1' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='ioh3420'/>
      <target chassis='2' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='virtio-mmio'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:10:07:41'/>
      <source network='default'/>
      <model type='virtio'/>
      <rom bar='off'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <source mode='bind'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
  </devices>
</domain>

-- 
Andrea Bolognani / Red Hat / Virtualization

Re: [Qemu-devel] Help: Does Qemu support virtio-pci for net-device and disk device?

[Laine Stump]

On 08/18/2016 03:41 AM, Andrew Jones wrote:
> On Wed, Aug 17, 2016 at 01:00:05PM -0400, Laine Stump wrote:
>> On 08/17/2016 12:13 PM, Andrew Jones wrote:
>>> On Wed, Aug 17, 2016 at 08:08:11PM +0800, Kevin Zhao wrote:
>>>> Hi all,
>>>>       Now I'm investigating net device hot plug and disk hotplug for
>>>> AArch64. For virtio , the default address is virtio-mmio. After Libvirt
>>>> 1.3.5, user can explicitly specify the address-type to pci and so libvirt
>>>> will pass the virtio-pci parameters to the Qemu.
>>>>       Both my host and guest OS is Debian8, and Qemu version is 2.6.0.
>>>> Libvirt version is 1.3.5.
>>>>       For net-device, I change the address-type to pci, and libvirt pass the
>>>> command below:
>>>>       -device
>>>> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:0d:25:25,bus=pci.2,addr=0x1
>>>>
>>>>       After booting, the eth0 device disappear(eth0 occur when the address
>>>> is virtio-mmio),
>>>> but I can find another net-device enp2s1, also it can't work for dhcp:
>>>> Running lspci: 02:01.0 Ethernet controller: Red Hat, Inc Virtio network
>>>> device
>>>> I'm not sure whether it worked.
>>>>
>>>>       For disk device,* when I change the address-type to pci, the whole
>>>> qemu command is :*
>>>> https://paste.fedoraproject.org/409553/,  but the VM can not boot
>>>> successfully. Does Qemu not support device disk of virtio-pci in AArch64
>>>> just as it in X86_64?
>>>>       Thanks~Since I am not very familiar with Qemu, really looking forward
>>>> to your response.
>>>>
>>>> Best Regards,
>>>> Kevin Zhao
>>> libvirt 1.3.5 is a bit old. Later versions no longer unconditionally add
>>> the i82801b11 bridge, which was necessary to use PCI devices with the PCIe
>>> host bridge mach-virt has. IMO, libvirt and qemu still have a long way to
>>> go in order to configure a base/standard mach-virt PCIe machine.
>>
>> Well, you can do it now, but you have to manually assign the PCI addresses
>> of devices (and if you want hotplug you need to live with Intel/TI-specific
>> PCIe controllers).
>>
>>
>>>
>>> 1) If we want to support both PCIe devices and PCI, then things are messy.
>>>     Currently we propose dropping PCI support. mach-virt pretty much
>>>     exclusively uses virtio, which can be set to PCIe mode (virtio-1.0)
>>
>> I have a libvirt patch just about ACKed for pushing upstream that will
>> automatically assign virtio-pci devices to a PCIe slot (if the qemu binary
>> supports virtio-1.0):
>>
>> https://www.redhat.com/archives/libvir-list/2016-August/msg00852.html
>>
>> Separate patches do the same for the e1000e emulated network device (which
>> you probably don't care about) and the nec-usb-xhci (USB3) controller (more
>> useful):
>>
>> https://www.redhat.com/archives/libvir-list/2016-August/msg00732.html
>>
>
> Thanks for the update Laine. This sounds great to me. With those patches
> we can switch from virtio-mmio to virtio-pci easily, even if we're still
> missing hotplug a bit longer. What limit do we have for the number of
> devices, when we don't have any switches? I think I experimented once and
> found it to be 7.

Theoretically you should be able to put something in each slot of 
pcie-root, and there are 31 slots (but slot 0x1f is always used by 
integrated devices, slot 1 is usually used by video, and slot 0x1D is 
usually used by a USB controller). Anyway, you should be able to get a 
lot more than 7 devices (one potential problem is that if you add a PCI 
controller that has a device plugged into it which requests IO port 
space, that will get chewed up very quickly. That's not an issue if you 
are connecting endpoints directly to pcie-root though.


>
>> Once these are in place, the only type of device of any consequence that I
>> can see still having no PCIe alternative is audio (even though only the
>> virgl video device is PCIe, libvirt has always assigned the primary video to
>> slot 1 on pcie-root anyway (although you shouldn't put a legacy PCI device
>> on a pcie-root-port or pcie-switch-downstream-port, it is acceptable to plug
>> it directly into pcie-root (as long as you know you won't need to hotplug
>> it).
>>
>>> 2) root complex ports, switches (upstream/downstream ports) are currently
>>>     based on Intel parts. Marcel is thinking about creating generic models.
>>
>> I say this every time it comes up, so just to be consistent: +1 :-)
>>
>>> 3) libvirt needs to learn how to plug everything together, in proper PCIe
>>>     fashion, leaving holes for hotplug.
>>
>> See above about virtio, although that doesn't cover the whole story. The
>> other part (which I'm working on right now) is that libvirt needs to
>> automatically add pcie-root-port, pcie-switch-upstream-port, and
>> pcie-switch-downstream-port devices as necessary. With the patches I
>> mentioned above, you still have to manually add enough pcie-*-port
>> controllers to the config, and then libvirt will plug the PCIe devices into
>> the right place. This is simple enough to do, but it does require
>> intervention.
>
> OK, so we want this to support hotplug and eventually chain switches,
> bumping our device limit up higher and higher. To what? I'm not sure,
> I guess we're still limited by address space.

As long as the endpoint devices don't require IO port space, the 
BIOS/firmware shouldn't try to reserve it, and the limit will be "more 
than you would ever need".

>
>>
>> As far as leaving holes for hotplug, there's actually still a bit of an open
>> question there - with machinetypes that use only legacy-PCI, *all* slots are
>> hotpluggable, and they're added 31 at a time, so there was never any
>> question about which slots were hotpluggable, and it would be very rare to
>> end up with a configuration that had 0 free slots available for hotplug
>> (actually libvirt would always make sure there was at least one, but in
>> practice there would be many more open slots). With PCIe-capable
>> machinetypes that is changed, since the root complex (pcie-root) doesn't
>> support hotplug, and new slots are added 1 at a time (pcie-*-port) rather
>> than 31 at a time. This means you have to really go out of your way if you
>> want open slots for hotplug (and even if you want devices in the machine at
>> boot time to be hot-unpluggable).
>>
>> I'm still not sure just how far we need to go in this regard.  We've already
>> decided that, unless manually set to an address on pcie-root by the
>> user/management application, all PCI devices will be auto-assigned to a slot
>> that supports hotplug. What I'm not sure about is whether we should always
>> auto-add an extra pcie-*-root to be sure a device can be hotplugged, or if
>> we should admit that 1 available slot isn't good enough for all situations,
>> so we should instead just leave it up to the user/management to manually add
>> extra ports if they think they'll want to hotplug something later.
>
> Hmm... Maybe the tools can make this easier by offering an option to
> provide N extra ports.

I've thought about this, and there have even been emails about it, but I 
don't know if it could ever be accepted into libvirt. My idea was to 
have an attribute that says "always maintain X hotpluggable slots when 
coldplugging new devices". I have a feeling that will get as warm of a 
reception as my proposed "hotpluggable='no'" attribute for devices (i.e. 
not warm at all :-))

The problem is that if you just tell the user to put in extra 
pcie-*-port controllers to allow for hotplug, those will eventually be 
eaten up, and then the next time you start the guest you'll again have 
no open hotpluggable slots again (even though restarting would have 
given you the opportunity to automatically add new *-port controllers).


>
> Hmm2... I think I agree that we don't need to worry too much about
> providing free ports for hotplug (maybe just one for fun). With
> virtio-scsi we can plug disks already.


until you run out of open units/buses/whatever they're called in 
virtio-scsi (I'm too lazy to look it up right now). And then you have 
the same problem.

We really can never solve the problem, we can only *delay* it, and the 
more we try to delay it the more convoluted the code gets. And of course 
the same number that is "not enough" for some users is "too much" for 
others. That's why I've been wondering if maybe we should just throw up 
our hands and punt (i.e. unabashedly document "if you want slots 
available for hotplug, you'll need to add them. Of course *even that* 
doesn't work unless you add the hotpluggable slots in a separate change 
from any endpoint devices; otherwise the new endpoint devices would just 
be immediately assigned to the new slots you added "for future hotplug" 
:-/).


> If we want to provide multiple
> virtio-net devices for the price of one port, we can enable mutlifunction.


Except that you can't hotplug/unplug the functions independent of each 
other - all the functions in one slot must be plugged/unplugged together.

> And, iirc, there's work to get ARI functioning, allowing mutlifunction to
> go nuts (assuming I understand its purpose correctly)

I keep forgetting all the acronyms. Is ARI the one that allows you (on 
controllers that only have a single slot, e.g. pcie-root-port) to 
interpret the byte that's normally a 5 bit slot# + 3 bit function# as a 
single 8 bit function#? (The idea being that you still have only a 
single slot, but that slot has 256 functions). There again, you still 
have the "hotplug as a single unit" problem, only the effects are 
multiplied because there are so many devices.


>
> So maybe the default config just needs 3 ports?
>  1 virtio-scsi with as many disks as requested
>  1 virtio-net with as many functions as nics are requested
>  1 extra port

None of those are in any default config created by libvirt. In a libvirt 
config with no specified devices (and no specified "anti-devices" such 
as "<controller type='usb' model='none'/>"), there is just the 
integrated ACHI (sata) controller, a USB controller (usually), and a 
<balloon> device (not sure why that's considered so important that it's 
in there by default, but it is).

Re: [Qemu-devel] Help: Does Qemu support virtio-pci for net-device and disk device?

[Laine Stump]

On 08/18/2016 08:10 AM, Marcel Apfelbaum wrote:
> On 08/17/2016 08:00 PM, Laine Stump wrote:

>
>  What I'm not sure about is whether we should always auto-add an extra
> pcie-*-root to be sure a device can be hotplugged, or if we should admit
> that 1
>> available slot isn't good enough for all situations, so we should
>> instead just leave it up to the user/management to manually add extra
>> ports if they think they'll want to hotplug something later.
>
> Why not? Leaving 1 or 2 PCIe ports should be enough. On each port you
> can hotplug a switch with several downstream ports. You can continue
> nesting the switches up to depth of 6-7 I think.

When did qemu start supporting hotplug of pcie switch ports? My 
understanding is that in real hardware the only way this is supported is 
to plug in the entire "upstream+downstream+downstream+..." set as a 
single unit, since there is no mechanism for guest kernel to notify the 
upstream port that a new downstream port has been attached to it (or 
something like that; I'm vague on the details). From the other end, qemu 
can only hotplug a single PCI device at a time, so by the time we get to 
the point of plugging in the downstream ports, the upstream port is 
already cemented in place by the guest kernel.

I think that would be a really desirable feature though. Maybe qemu 
could queue up any downstream-ports which are pointing to an 
as-yet-nonexistent upstream-port id, then when the upstream-port with 
the proper id is finally attached, it could send the right magic to the 
guest (similar to the way it allows hotplugging all non-0 functions 
first, then takes action when function 0 is hotplugged).

If that was available, then yes, it would make perfect sense for libvirt 
to simply always make sure at least one empty pcie-*-port was available. 
If you have plans of doing something to support hotplugging a 
pcie-switch-* collection, then that's what I'll do.

Re: [Qemu-devel] Help: Does Qemu support virtio-pci for net-device and disk device?

[Laine Stump]

On 08/18/2016 08:43 AM, Kevin Zhao wrote:
> Hi Laine,
>     Thanks :-) I also has a little questions below.
>
> On 18 August 2016 at 01:00, Laine Stump <laine@redhat.com> wrote:
>
>> On 08/17/2016 12:13 PM, Andrew Jones wrote:
>>
>>> On Wed, Aug 17, 2016 at 08:08:11PM +0800, Kevin Zhao wrote:
>>>
>>>> Hi all,
>>>>       Now I'm investigating net device hot plug and disk hotplug for
>>>> AArch64. For virtio , the default address is virtio-mmio. After Libvirt
>>>> 1.3.5, user can explicitly specify the address-type to pci and so libvirt
>>>> will pass the virtio-pci parameters to the Qemu.
>>>>       Both my host and guest OS is Debian8, and Qemu version is 2.6.0.
>>>> Libvirt version is 1.3.5.
>>>>       For net-device, I change the address-type to pci, and libvirt pass
>>>> the
>>>> command below:
>>>>       -device
>>>> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:0d:25:
>>>> 25,bus=pci.2,addr=0x1
>>>>
>>>>       After booting, the eth0 device disappear(eth0 occur when the
>>>> address
>>>> is virtio-mmio),
>>>> but I can find another net-device enp2s1, also it can't work for dhcp:
>>>> Running lspci: 02:01.0 Ethernet controller: Red Hat, Inc Virtio network
>>>> device
>>>> I'm not sure whether it worked.
>>>>
>>>>       For disk device,* when I change the address-type to pci, the whole
>>>> qemu command is :*
>>>> https://paste.fedoraproject.org/409553/,  but the VM can not boot
>>>> successfully. Does Qemu not support device disk of virtio-pci in AArch64
>>>> just as it in X86_64?
>>>>       Thanks~Since I am not very familiar with Qemu, really looking
>>>> forward
>>>> to your response.
>>>>
>>>> Best Regards,
>>>> Kevin Zhao
>>>>
>>> libvirt 1.3.5 is a bit old. Later versions no longer unconditionally add
>>> the i82801b11 bridge, which was necessary to use PCI devices with the PCIe
>>> host bridge mach-virt has. IMO, libvirt and qemu still have a long way to
>>> go in order to configure a base/standard mach-virt PCIe machine.
>>>
>>
>> Well, you can do it now, but you have to manually assign the PCI addresses
>> of devices (and if you want hotplug you need to live with Intel/TI-specific
>> PCIe controllers).
>
> OK. It seems that Qemu will drop PCI for machine-virt and turning to PCIE
> in the future.

This isn't a qemu issue, but a libvirt issue.

> Do I need to do more for  Intel/TI-specific PCIe controllers?


>  what do I
> need to add in the guest XML or more?

As soon as my first set of patches is pushed, what you'll need to do is 
to add:

     <controller type='pci' model='pcie-root-port'/>

for each virtio device.

As soon as I've figured out a useful algorithm for adding the pcie 
controllers automatically, you won't need to do anything at all. (Keep 
in mind that any existing configurations will maintain their original 
config. If you want an existing configuration to be switched to using 
the new recommended (pcie) bus topology you will need to replace every 
instance of:

     <address type='pci' domain='0x0000' ......../>

with a plain:

     <address type='pci'/>

This will force libvirt to reassign PCI addresses for the devices.

>
>>
>>
>>> 1) If we want to support both PCIe devices and PCI, then things are messy.
>>>     Currently we propose dropping PCI support. mach-virt pretty much
>>>     exclusively uses virtio, which can be set to PCIe mode (virtio-1.0)
>>>
>>
>> I have a libvirt patch just about ACKed for pushing upstream that will
>> automatically assign virtio-pci devices to a PCIe slot (if the qemu binary
>> supports virtio-1.0):
>>
>> https://www.redhat.com/archives/libvir-list/2016-August/msg00852.html
>>
>
> What's the minimum version of  Qemu that support virito-1.0? Does Qemu 2.6
> works?

qemu as old as 2.4.0 has the "--disable-legacy" option for virtio 
devices. I don't know if it was actually working properly back that far.


> Also as  I see your patch for automatically assign virtio-pci to PCIE
> slot,after it merged  I think thing will go much more easier.
> Now I will manually add the slots and bus to pcie. Because I am not
> familiar with it,  if it convenient, could you give me an available xml
> file which PCIE disk and PCIE
> net device can work for machine virt ?

I think Andrea sent an example.

Re: [Qemu-devel] Help: Does Qemu support virtio-pci for net-device and disk device?

[Andrea Bolognani]

On Thu, 2016-08-18 at 08:38 +0200, Andrew Jones wrote:
> > > Finally, FWIW, with a guest kernel of 4.6.4-301.fc24.aarch64. The
> > > following qemu command line works for me.
> > > (notice the use of PCIe), and my network interface gets labeled enp0s1.
> > >  
> > > $QEMU -machine virt-2.6,accel=kvm -cpu host \
> > >   -m 1024 -smp 1 -nographic \
> > >   -bios /usr/share/AAVMF/AAVMF_CODE.fd \
> > >   -device ioh3420,bus=pcie.0,id=pcie.1,port=1,chassis=1 \
> > >   -device ioh3420,bus=pcie.0,id=pcie.2,port=2,chassis=2 \
> > >   -device virtio-scsi-pci,disable-modern=off,disable-legacy=on,bus=pcie.1,addr=00.0,id=scsi0 \
> > >   -drive file=/home/drjones/.local/libvirt/images/fedora.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0 \
> > >   -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 \
> > >   -netdev user,id=hostnet0 \
> > >   -device virtio-net-pci,disable-modern=off,disable-legacy=on,bus=pcie.2,addr=00.0,netdev=hostnet0,id=net0
> > >  
> > > I prefer always using virtio-scsi for the disk, but a similar command
> > > line can be used for a virtio-blk-pci disk.
> > 
> > Does the same command line work if you don't specify any of
> > the disable-* options?
> > 
> > I'm asking because I tried running a Fedora 24 guest through
> > libvirt, which doesn't support those options yet, and I get
> > 
> >   virtio_blk virtio2: virtio: device uses modern interface but
> >                               does not have VIRTIO_F_VERSION_1
> >   virtio_blk: probe of virtio2 failed with error -22
> 
> Doesn't work for me either. I can only boot with disable-modern=off,
> disable-legacy=on (at least when building my config the way I try to
> build it...) I presume that's a guest kernel issue.

I tried Fedora 24 and Debian testing, and for both of them
the result is the same: I can only boot the guest if I'm
setting up a legacy-free PCIe topology and use virt-2.7 to
obtain virtio-1.0 devices (see below); for every other
permutation of

  { PCI topology, PCIe topology } x { virt-2.6, virt-2.7 }

the guest doesn't boot at all.

On the other hand, a RHEL 7.3 guest was able to boot *every
single time*, even though the result was in some cases
quite questionable (eg. legacy PCI devices plugged into
ioh3420 ports).

> > Isn't the default for 2.6 disable-modern=off,
> > disable-legacy=off? Or was that 2.7? I tried both anyway ;)
> 
> Dunno. With the command line getting longer all the time, I just
> have a script that generates one that works for me, and haven't
> worried much about the defaults...

So I thought the default for 2.6 was supposed to be

  disable-modern=off,disable-legacy=off  [0.9+1.0]

but it turns out it's actually

  disable-modern=on,disable-legacy=off       [0.9]

whereas the default for 2.7 is

  disable-modern=off,disable-legacy=on       [1.0]

Is the idea that there would be a QEMU release with both
0.9 and 1.0 enabled by default something that I just
imagined? Or did the plan just change at some point?

-- 
Andrea Bolognani / Red Hat / Virtualization

Re: [Qemu-devel] Help: Does Qemu support virtio-pci for net-device and disk device?

[Laine Stump]

On 08/19/2016 11:43 AM, Andrea Bolognani wrote:
> On Thu, 2016-08-18 at 08:38 +0200, Andrew Jones wrote:
>>>> Finally, FWIW, with a guest kernel of 4.6.4-301.fc24.aarch64. The
>>>> following qemu command line works for me.
>>>> (notice the use of PCIe), and my network interface gets labeled enp0s1.
>>>>    
>>>> $QEMU -machine virt-2.6,accel=kvm -cpu host \
>>>>     -m 1024 -smp 1 -nographic \
>>>>     -bios /usr/share/AAVMF/AAVMF_CODE.fd \
>>>>     -device ioh3420,bus=pcie.0,id=pcie.1,port=1,chassis=1 \
>>>>     -device ioh3420,bus=pcie.0,id=pcie.2,port=2,chassis=2 \
>>>>     -device virtio-scsi-pci,disable-modern=off,disable-legacy=on,bus=pcie.1,addr=00.0,id=scsi0 \
>>>>     -drive file=/home/drjones/.local/libvirt/images/fedora.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0 \
>>>>     -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 \
>>>>     -netdev user,id=hostnet0 \
>>>>     -device virtio-net-pci,disable-modern=off,disable-legacy=on,bus=pcie.2,addr=00.0,netdev=hostnet0,id=net0
>>>>    
>>>> I prefer always using virtio-scsi for the disk, but a similar command
>>>> line can be used for a virtio-blk-pci disk.
>>>   
>>> Does the same command line work if you don't specify any of
>>> the disable-* options?
>>>   
>>> I'm asking because I tried running a Fedora 24 guest through
>>> libvirt, which doesn't support those options yet, and I get
>>>   
>>>     virtio_blk virtio2: virtio: device uses modern interface but
>>>                                 does not have VIRTIO_F_VERSION_1
>>>     virtio_blk: probe of virtio2 failed with error -22
>>   
>> Doesn't work for me either. I can only boot with disable-modern=off,
>> disable-legacy=on (at least when building my config the way I try to
>> build it...) I presume that's a guest kernel issue.
> I tried Fedora 24 and Debian testing, and for both of them
> the result is the same: I can only boot the guest if I'm
> setting up a legacy-free PCIe topology and use virt-2.7 to
> obtain virtio-1.0 devices (see below); for every other
> permutation of
>
>    { PCI topology, PCIe topology } x { virt-2.6, virt-2.7 }
>
> the guest doesn't boot at all.
>
> On the other hand, a RHEL 7.3 guest was able to boot *every
> single time*, even though the result was in some cases
> quite questionable (eg. legacy PCI devices plugged into
> ioh3420 ports).
>
>>> Isn't the default for 2.6 disable-modern=off,
>>> disable-legacy=off? Or was that 2.7? I tried both anyway ;)
>>   
>> Dunno. With the command line getting longer all the time, I just
>> have a script that generates one that works for me, and haven't
>> worried much about the defaults...
> So I thought the default for 2.6 was supposed to be
>
>    disable-modern=off,disable-legacy=off  [0.9+1.0]
>
> but it turns out it's actually
>
>    disable-modern=on,disable-legacy=off       [0.9]

Huh. I had thought it switched to disable-modern=off with 2.6 as well.

>
> whereas the default for 2.7 is
>
>    disable-modern=off,disable-legacy=on       [1.0]
>
> Is the idea that there would be a QEMU release with both
> 0.9 and 1.0 enabled by default something that I just
> imagined? Or did the plan just change at some point?

If I understand the patches correctly, the default for 2.7 will be 
different depending on the type of slot:

pcie-root -> disable-modern=off,disable-legacy=off
legacy PCI -> disable-modern=off,disable-legacy=off
pcie-*-port -> disable-modern=off,disable-legacy=on

The idea is to eliminate the need for the pcie-*-port to reserve IO port 
space (this isn't an issue on pcie-root, so the default is different).

Re: [Qemu-devel] Help: Does Qemu support virtio-pci for net-device and disk device?

[Kevin Zhao]

Great ~ Thanks for your valuable information~
I will try with the xml and any update I will post here.

On 18 August 2016 at 21:51, Andrea Bolognani <abologna@redhat.com> wrote:

> On Thu, 2016-08-18 at 20:43 +0800, Kevin Zhao wrote:
> > What's the minimum version of  Qemu that support virito-1.0?
> > Does Qemu 2.6 works?
>
> 2.6 definitely has virtio 1.0 support, however libvirt does
> not yet allow you to control whether a device uses 0.9, 1.0
> or both. The default for 2.6 should be both IIRC.
>
> > Now I will manually add the slots and bus to pcie. Because
> > I am not familiar with it,  if it convenient, could you give
> > me an available xml file which PCIE disk and PCIE
> > net device can work for machine virt ?
>
> The XML you're looking for is at the end of this message.
>
> Note that a Fedora 24 guest configured this way will not
> boot at all if the machine type is virt-2.6; on the other
> hand, an identically-configured RHEL 7.3 guest will boot
> even with virt-2.6, but both the disk and the network
> adapter will be legacy PCI instead of PCIe.
>
>
> <domain type='kvm'>
>   <name>abologna-f24</name>
>   <uuid>f6d0428b-a034-4c4e-8ef2-f12f6aa9cab0</uuid>
>   <memory unit='KiB'>2097152</memory>
>   <currentMemory unit='KiB'>2097152</currentMemory>
>   <vcpu placement='static'>4</vcpu>
>   <os>
>     <type arch='aarch64' machine='virt-2.7'>hvm</type>
>     <loader readonly='yes' type='pflash'>/usr/share/
> AAVMF/AAVMF_CODE.fd</loader>
>     <nvram>/var/lib/libvirt/qemu/nvram/abologna-f24_VARS.fd</nvram>
>     <boot dev='hd'/>
>   </os>
>   <features>
>     <gic version='2'/>
>   </features>
>   <cpu mode='host-passthrough'/>
>   <clock offset='utc'/>
>   <on_poweroff>destroy</on_poweroff>
>   <on_reboot>restart</on_reboot>
>   <on_crash>restart</on_crash>
>   <devices>
>     <emulator>/usr/libexec/abologna-qemu-kvm</emulator>
>     <disk type='file' device='disk'>
>       <driver name='qemu' type='qcow2'/>
>       <source file='/var/lib/libvirt/images/abologna-f24.qcow2'/>
>       <target dev='vda' bus='virtio'/>
>       <address type='pci' domain='0x0000' bus='0x01' slot='0x00'
> function='0x0'/>
>     </disk>
>     <controller type='pci' index='0' model='pcie-root'/>
>     <controller type='pci' index='1' model='pcie-root-port'>
>       <model name='ioh3420'/>
>       <target chassis='1' port='0x8'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> function='0x0'/>
>     </controller>
>     <controller type='pci' index='2' model='pcie-root-port'>
>       <model name='ioh3420'/>
>       <target chassis='2' port='0x10'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> function='0x0'/>
>     </controller>
>     <controller type='virtio-serial' index='0'>
>       <address type='virtio-mmio'/>
>     </controller>
>     <interface type='network'>
>       <mac address='52:54:00:10:07:41'/>
>       <source network='default'/>
>       <model type='virtio'/>
>       <rom bar='off'/>
>       <address type='pci' domain='0x0000' bus='0x02' slot='0x00'
> function='0x0'/>
>     </interface>
>     <serial type='pty'>
>       <target port='0'/>
>     </serial>
>     <console type='pty'>
>       <target type='serial' port='0'/>
>     </console>
>     <channel type='unix'>
>       <source mode='bind'/>
>       <target type='virtio' name='org.qemu.guest_agent.0'/>
>       <address type='virtio-serial' controller='0' bus='0' port='1'/>
>     </channel>
>   </devices>
> </domain>
>
> --
> Andrea Bolognani / Red Hat / Virtualization
>

Re: [Qemu-devel] Help: Does Qemu support virtio-pci for net-device and disk device?

[Kevin Zhao]

Hi  All,
   As discussed before, upstream are working about PCIE instead of PCI in
AArch64.
   Thanks for your efforts about this on AArch64 :-)
   If it convenient, could you tell me when we plan  to finish this task ?
and which qemu version will support this functions in the future?
   Big Thanks~

Best Regards,
Kevin

On 24 August 2016 at 09:52, Kevin Zhao <kevin.zhao@linaro.org> wrote:

> Great ~ Thanks for your valuable information~
> I will try with the xml and any update I will post here.
>
> On 18 August 2016 at 21:51, Andrea Bolognani <abologna@redhat.com> wrote:
>
>> On Thu, 2016-08-18 at 20:43 +0800, Kevin Zhao wrote:
>> > What's the minimum version of  Qemu that support virito-1.0?
>> > Does Qemu 2.6 works?
>>
>> 2.6 definitely has virtio 1.0 support, however libvirt does
>> not yet allow you to control whether a device uses 0.9, 1.0
>> or both. The default for 2.6 should be both IIRC.
>>
>> > Now I will manually add the slots and bus to pcie. Because
>> > I am not familiar with it,  if it convenient, could you give
>> > me an available xml file which PCIE disk and PCIE
>> > net device can work for machine virt ?
>>
>> The XML you're looking for is at the end of this message.
>>
>> Note that a Fedora 24 guest configured this way will not
>> boot at all if the machine type is virt-2.6; on the other
>> hand, an identically-configured RHEL 7.3 guest will boot
>> even with virt-2.6, but both the disk and the network
>> adapter will be legacy PCI instead of PCIe.
>>
>>
>> <domain type='kvm'>
>>   <name>abologna-f24</name>
>>   <uuid>f6d0428b-a034-4c4e-8ef2-f12f6aa9cab0</uuid>
>>   <memory unit='KiB'>2097152</memory>
>>   <currentMemory unit='KiB'>2097152</currentMemory>
>>   <vcpu placement='static'>4</vcpu>
>>   <os>
>>     <type arch='aarch64' machine='virt-2.7'>hvm</type>
>>     <loader readonly='yes' type='pflash'>/usr/share/AAVMF
>> /AAVMF_CODE.fd</loader>
>>     <nvram>/var/lib/libvirt/qemu/nvram/abologna-f24_VARS.fd</nvram>
>>     <boot dev='hd'/>
>>   </os>
>>   <features>
>>     <gic version='2'/>
>>   </features>
>>   <cpu mode='host-passthrough'/>
>>   <clock offset='utc'/>
>>   <on_poweroff>destroy</on_poweroff>
>>   <on_reboot>restart</on_reboot>
>>   <on_crash>restart</on_crash>
>>   <devices>
>>     <emulator>/usr/libexec/abologna-qemu-kvm</emulator>
>>     <disk type='file' device='disk'>
>>       <driver name='qemu' type='qcow2'/>
>>       <source file='/var/lib/libvirt/images/abologna-f24.qcow2'/>
>>       <target dev='vda' bus='virtio'/>
>>       <address type='pci' domain='0x0000' bus='0x01' slot='0x00'
>> function='0x0'/>
>>     </disk>
>>     <controller type='pci' index='0' model='pcie-root'/>
>>     <controller type='pci' index='1' model='pcie-root-port'>
>>       <model name='ioh3420'/>
>>       <target chassis='1' port='0x8'/>
>>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
>> function='0x0'/>
>>     </controller>
>>     <controller type='pci' index='2' model='pcie-root-port'>
>>       <model name='ioh3420'/>
>>       <target chassis='2' port='0x10'/>
>>       <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
>> function='0x0'/>
>>     </controller>
>>     <controller type='virtio-serial' index='0'>
>>       <address type='virtio-mmio'/>
>>     </controller>
>>     <interface type='network'>
>>       <mac address='52:54:00:10:07:41'/>
>>       <source network='default'/>
>>       <model type='virtio'/>
>>       <rom bar='off'/>
>>       <address type='pci' domain='0x0000' bus='0x02' slot='0x00'
>> function='0x0'/>
>>     </interface>
>>     <serial type='pty'>
>>       <target port='0'/>
>>     </serial>
>>     <console type='pty'>
>>       <target type='serial' port='0'/>
>>     </console>
>>     <channel type='unix'>
>>       <source mode='bind'/>
>>       <target type='virtio' name='org.qemu.guest_agent.0'/>
>>       <address type='virtio-serial' controller='0' bus='0' port='1'/>
>>     </channel>
>>   </devices>
>> </domain>
>>
>> --
>> Andrea Bolognani / Red Hat / Virtualization
>>
>
>