* Re: [Qemu-devel] Hitting 29 NIC limit
@ 2010-10-14 12:07 ` Avi Kivity
0 siblings, 0 replies; 72+ messages in thread
From: Avi Kivity @ 2010-10-14 12:07 UTC (permalink / raw)
To: Anthony Liguori; +Cc: Anjali Kulkarni, qemu-devel, kvm
On 10/14/2010 12:54 AM, Anthony Liguori wrote:
> On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
>>
>> Hi,
>>
>> Using the legacy way of starting up NICs, I am hitting a limitation
>> after 29
>> NICs ie no more than 29 are detected (that's because of the 32 PCI slot
>> limit on a single bus- 3 are already taken up)
>> I had initially increased the MAX_NICS to 48, just on my tree, to get to
>> more, but ofcource that wont work.
>> Is there any way to go beyond 29 NICs the legacy way? What is the
>> maximum
>> that can be supported by the qdev mothod?
>
> I got up to 104 without trying very hard using the following script:
>
> args=""
> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
> for fn in 0 1 2 3 4 5 6 7; do
> args="$args -netdev user,id=eth${slot}_${fn}"
> args="$args -device
> virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
> done
> done
>
> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
> -enable-kvm
>
> The key is to make the virtio-net devices multifunction and to fill
> out all 8 functions for each slot.
This is unlikely to work right wrt pci hotplug. If we want to support a
large number of interfaces, we need true multiport cards.
What's the motivation for such a huge number of interfaces?
--
error compiling committee.c: too many arguments to function
^ permalink raw reply [flat|nested] 72+ messages in thread* Re: [Qemu-devel] Hitting 29 NIC limit
2010-10-14 12:07 ` Avi Kivity
@ 2010-10-14 12:10 ` Daniel P. Berrange
-1 siblings, 0 replies; 72+ messages in thread
From: Daniel P. Berrange @ 2010-10-14 12:10 UTC (permalink / raw)
To: Avi Kivity; +Cc: Anthony Liguori, Anjali Kulkarni, kvm, qemu-devel
On Thu, Oct 14, 2010 at 02:07:17PM +0200, Avi Kivity wrote:
> On 10/14/2010 12:54 AM, Anthony Liguori wrote:
> >On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
> >>
> >>Hi,
> >>
> >>Using the legacy way of starting up NICs, I am hitting a limitation
> >>after 29
> >>NICs ie no more than 29 are detected (that's because of the 32 PCI slot
> >>limit on a single bus- 3 are already taken up)
> >>I had initially increased the MAX_NICS to 48, just on my tree, to get to
> >>more, but ofcource that wont work.
> >>Is there any way to go beyond 29 NICs the legacy way? What is the
> >>maximum
> >>that can be supported by the qdev mothod?
> >
> >I got up to 104 without trying very hard using the following script:
> >
> >args=""
> >for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
> >for fn in 0 1 2 3 4 5 6 7; do
> > args="$args -netdev user,id=eth${slot}_${fn}"
> > args="$args -device
> >virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
> >done
> >done
> >
> >x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
> >-enable-kvm
> >
> >The key is to make the virtio-net devices multifunction and to fill
> >out all 8 functions for each slot.
>
> This is unlikely to work right wrt pci hotplug. If we want to support a
> large number of interfaces, we need true multiport cards.
Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
any type of device we emulate.
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
^ permalink raw reply [flat|nested] 72+ messages in thread* Re: [Qemu-devel] Hitting 29 NIC limit
@ 2010-10-14 12:10 ` Daniel P. Berrange
0 siblings, 0 replies; 72+ messages in thread
From: Daniel P. Berrange @ 2010-10-14 12:10 UTC (permalink / raw)
To: Avi Kivity; +Cc: Anjali Kulkarni, kvm, qemu-devel
On Thu, Oct 14, 2010 at 02:07:17PM +0200, Avi Kivity wrote:
> On 10/14/2010 12:54 AM, Anthony Liguori wrote:
> >On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
> >>
> >>Hi,
> >>
> >>Using the legacy way of starting up NICs, I am hitting a limitation
> >>after 29
> >>NICs ie no more than 29 are detected (that's because of the 32 PCI slot
> >>limit on a single bus- 3 are already taken up)
> >>I had initially increased the MAX_NICS to 48, just on my tree, to get to
> >>more, but ofcource that wont work.
> >>Is there any way to go beyond 29 NICs the legacy way? What is the
> >>maximum
> >>that can be supported by the qdev mothod?
> >
> >I got up to 104 without trying very hard using the following script:
> >
> >args=""
> >for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
> >for fn in 0 1 2 3 4 5 6 7; do
> > args="$args -netdev user,id=eth${slot}_${fn}"
> > args="$args -device
> >virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
> >done
> >done
> >
> >x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
> >-enable-kvm
> >
> >The key is to make the virtio-net devices multifunction and to fill
> >out all 8 functions for each slot.
>
> This is unlikely to work right wrt pci hotplug. If we want to support a
> large number of interfaces, we need true multiport cards.
Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
any type of device we emulate.
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
^ permalink raw reply [flat|nested] 72+ messages in thread* Re: [Qemu-devel] Hitting 29 NIC limit
2010-10-14 12:10 ` Daniel P. Berrange
@ 2010-10-14 12:57 ` Anthony Liguori
-1 siblings, 0 replies; 72+ messages in thread
From: Anthony Liguori @ 2010-10-14 12:57 UTC (permalink / raw)
To: Daniel P. Berrange; +Cc: Avi Kivity, Anjali Kulkarni, kvm, qemu-devel
On 10/14/2010 07:10 AM, Daniel P. Berrange wrote:
> On Thu, Oct 14, 2010 at 02:07:17PM +0200, Avi Kivity wrote:
>
>> On 10/14/2010 12:54 AM, Anthony Liguori wrote:
>>
>>> On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
>>>
>>>> Hi,
>>>>
>>>> Using the legacy way of starting up NICs, I am hitting a limitation
>>>> after 29
>>>> NICs ie no more than 29 are detected (that's because of the 32 PCI slot
>>>> limit on a single bus- 3 are already taken up)
>>>> I had initially increased the MAX_NICS to 48, just on my tree, to get to
>>>> more, but ofcource that wont work.
>>>> Is there any way to go beyond 29 NICs the legacy way? What is the
>>>> maximum
>>>> that can be supported by the qdev mothod?
>>>>
>>> I got up to 104 without trying very hard using the following script:
>>>
>>> args=""
>>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>>> for fn in 0 1 2 3 4 5 6 7; do
>>> args="$args -netdev user,id=eth${slot}_${fn}"
>>> args="$args -device
>>> virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
>>> done
>>> done
>>>
>>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>>> -enable-kvm
>>>
>>> The key is to make the virtio-net devices multifunction and to fill
>>> out all 8 functions for each slot.
>>>
>> This is unlikely to work right wrt pci hotplug. If we want to support a
>> large number of interfaces, we need true multiport cards.
>>
> Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
> any type of device we emulate.
>
I've always been sceptical of this. When physical systems have a large
number of NICs, it's via multiple functions, not a bunch of PCI bridges.
With just a handful of 8-port NICs, you can exceed the current
slot-based limit on physical hardware. It's not an extremely common
configuration but it does exist.
BTW, I don't think it's possible to hot-add physical functions. I
believe I know of a card that supports dynamic add of physical functions
(pre-dating SR-IOV).
Regards,
Anthony Liguori
> Daniel
>
^ permalink raw reply [flat|nested] 72+ messages in thread* Re: [Qemu-devel] Hitting 29 NIC limit
@ 2010-10-14 12:57 ` Anthony Liguori
0 siblings, 0 replies; 72+ messages in thread
From: Anthony Liguori @ 2010-10-14 12:57 UTC (permalink / raw)
To: Daniel P. Berrange; +Cc: Anjali Kulkarni, Avi Kivity, kvm, qemu-devel
On 10/14/2010 07:10 AM, Daniel P. Berrange wrote:
> On Thu, Oct 14, 2010 at 02:07:17PM +0200, Avi Kivity wrote:
>
>> On 10/14/2010 12:54 AM, Anthony Liguori wrote:
>>
>>> On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
>>>
>>>> Hi,
>>>>
>>>> Using the legacy way of starting up NICs, I am hitting a limitation
>>>> after 29
>>>> NICs ie no more than 29 are detected (that's because of the 32 PCI slot
>>>> limit on a single bus- 3 are already taken up)
>>>> I had initially increased the MAX_NICS to 48, just on my tree, to get to
>>>> more, but ofcource that wont work.
>>>> Is there any way to go beyond 29 NICs the legacy way? What is the
>>>> maximum
>>>> that can be supported by the qdev mothod?
>>>>
>>> I got up to 104 without trying very hard using the following script:
>>>
>>> args=""
>>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>>> for fn in 0 1 2 3 4 5 6 7; do
>>> args="$args -netdev user,id=eth${slot}_${fn}"
>>> args="$args -device
>>> virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
>>> done
>>> done
>>>
>>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>>> -enable-kvm
>>>
>>> The key is to make the virtio-net devices multifunction and to fill
>>> out all 8 functions for each slot.
>>>
>> This is unlikely to work right wrt pci hotplug. If we want to support a
>> large number of interfaces, we need true multiport cards.
>>
> Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
> any type of device we emulate.
>
I've always been sceptical of this. When physical systems have a large
number of NICs, it's via multiple functions, not a bunch of PCI bridges.
With just a handful of 8-port NICs, you can exceed the current
slot-based limit on physical hardware. It's not an extremely common
configuration but it does exist.
BTW, I don't think it's possible to hot-add physical functions. I
believe I know of a card that supports dynamic add of physical functions
(pre-dating SR-IOV).
Regards,
Anthony Liguori
> Daniel
>
^ permalink raw reply [flat|nested] 72+ messages in thread* Re: [Qemu-devel] Hitting 29 NIC limit
2010-10-14 12:57 ` Anthony Liguori
@ 2010-10-18 20:07 ` H. Peter Anvin
-1 siblings, 0 replies; 72+ messages in thread
From: H. Peter Anvin @ 2010-10-18 20:07 UTC (permalink / raw)
To: Anthony Liguori
Cc: Daniel P. Berrange, Anjali Kulkarni, Avi Kivity, kvm, qemu-devel
On 10/14/2010 05:57 AM, Anthony Liguori wrote:
>
> I've always been sceptical of this. When physical systems have a large
> number of NICs, it's via multiple functions, not a bunch of PCI bridges.
>
Actually a lot of multiport PCI cards are in fact single or dual NICs
behind PCI bridges.
-hpa
^ permalink raw reply [flat|nested] 72+ messages in thread
* Re: [Qemu-devel] Hitting 29 NIC limit
@ 2010-10-18 20:07 ` H. Peter Anvin
0 siblings, 0 replies; 72+ messages in thread
From: H. Peter Anvin @ 2010-10-18 20:07 UTC (permalink / raw)
To: Anthony Liguori; +Cc: Anjali Kulkarni, Avi Kivity, kvm, qemu-devel
On 10/14/2010 05:57 AM, Anthony Liguori wrote:
>
> I've always been sceptical of this. When physical systems have a large
> number of NICs, it's via multiple functions, not a bunch of PCI bridges.
>
Actually a lot of multiport PCI cards are in fact single or dual NICs
behind PCI bridges.
-hpa
^ permalink raw reply [flat|nested] 72+ messages in thread
* Re: [Qemu-devel] Hitting 29 NIC limit
2010-10-14 12:10 ` Daniel P. Berrange
@ 2010-10-14 21:42 ` Richard W.M. Jones
-1 siblings, 0 replies; 72+ messages in thread
From: Richard W.M. Jones @ 2010-10-14 21:42 UTC (permalink / raw)
To: Daniel P. Berrange; +Cc: Avi Kivity, Anjali Kulkarni, kvm, qemu-devel
On Thu, Oct 14, 2010 at 01:10:47PM +0100, Daniel P. Berrange wrote:
> Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
> any type of device we emulate.
Break the 29/30/31 virtio-blk limit ... please!
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://et.redhat.com/~rjones/virt-df/
^ permalink raw reply [flat|nested] 72+ messages in thread
* Re: [Qemu-devel] Hitting 29 NIC limit
@ 2010-10-14 21:42 ` Richard W.M. Jones
0 siblings, 0 replies; 72+ messages in thread
From: Richard W.M. Jones @ 2010-10-14 21:42 UTC (permalink / raw)
To: Daniel P. Berrange; +Cc: Anjali Kulkarni, Avi Kivity, kvm, qemu-devel
On Thu, Oct 14, 2010 at 01:10:47PM +0100, Daniel P. Berrange wrote:
> Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
> any type of device we emulate.
Break the 29/30/31 virtio-blk limit ... please!
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine. Supports Linux and Windows.
http://et.redhat.com/~rjones/virt-df/
^ permalink raw reply [flat|nested] 72+ messages in thread
* Re: [Qemu-devel] Hitting 29 NIC limit
2010-10-14 21:42 ` Richard W.M. Jones
@ 2010-10-14 21:57 ` Anthony Liguori
-1 siblings, 0 replies; 72+ messages in thread
From: Anthony Liguori @ 2010-10-14 21:57 UTC (permalink / raw)
To: Richard W.M. Jones
Cc: Daniel P. Berrange, Avi Kivity, Anjali Kulkarni, kvm, qemu-devel
On 10/14/2010 04:42 PM, Richard W.M. Jones wrote:
> On Thu, Oct 14, 2010 at 01:10:47PM +0100, Daniel P. Berrange wrote:
>
>> Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
>> any type of device we emulate.
>>
> Break the 29/30/31 virtio-blk limit ... please!
>
It was broken ages ago:
anthony@howler:~$ wc -l /proc/partitions; tail /proc/partitions
422 /proc/partitions
251 1618 1 vdcx2
251 1621 489951 vdcx5
251 1632 10485760 vdcy
251 1633 9992398 vdcy1
251 1634 1 vdcy2
251 1637 489951 vdcy5
251 1648 10485760 vdcz
251 1649 9992398 vdcz1
251 1650 1 vdcz2
251 1653 489951 vdcz5
This is what makes qdev so useful.
args=""
for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
for fn in 0 1 2 3 4 5 6 7; do
args="$args -drive
file=/home/anthony/images/linux.img,if=none,snapshot=on,id=disk${slot}_${fn}"
args="$args -device
virtio-blk-pci,addr=${slot}.${fn},drive=disk${slot}_${fn},multifunction=on"
done
done
x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
-enable-kvm -serial stdio
Regards,
Anthony Liguori
> Rich.
>
>
^ permalink raw reply [flat|nested] 72+ messages in thread* Re: [Qemu-devel] Hitting 29 NIC limit
@ 2010-10-14 21:57 ` Anthony Liguori
0 siblings, 0 replies; 72+ messages in thread
From: Anthony Liguori @ 2010-10-14 21:57 UTC (permalink / raw)
To: Richard W.M. Jones; +Cc: Anjali Kulkarni, Avi Kivity, kvm, qemu-devel
On 10/14/2010 04:42 PM, Richard W.M. Jones wrote:
> On Thu, Oct 14, 2010 at 01:10:47PM +0100, Daniel P. Berrange wrote:
>
>> Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
>> any type of device we emulate.
>>
> Break the 29/30/31 virtio-blk limit ... please!
>
It was broken ages ago:
anthony@howler:~$ wc -l /proc/partitions; tail /proc/partitions
422 /proc/partitions
251 1618 1 vdcx2
251 1621 489951 vdcx5
251 1632 10485760 vdcy
251 1633 9992398 vdcy1
251 1634 1 vdcy2
251 1637 489951 vdcy5
251 1648 10485760 vdcz
251 1649 9992398 vdcz1
251 1650 1 vdcz2
251 1653 489951 vdcz5
This is what makes qdev so useful.
args=""
for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
for fn in 0 1 2 3 4 5 6 7; do
args="$args -drive
file=/home/anthony/images/linux.img,if=none,snapshot=on,id=disk${slot}_${fn}"
args="$args -device
virtio-blk-pci,addr=${slot}.${fn},drive=disk${slot}_${fn},multifunction=on"
done
done
x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
-enable-kvm -serial stdio
Regards,
Anthony Liguori
> Rich.
>
>
^ permalink raw reply [flat|nested] 72+ messages in thread* Re: [Qemu-devel] Hitting 29 NIC limit
2010-10-14 21:57 ` Anthony Liguori
@ 2010-10-14 22:00 ` Anjali Kulkarni
-1 siblings, 0 replies; 72+ messages in thread
From: Anjali Kulkarni @ 2010-10-14 22:00 UTC (permalink / raw)
To: Anthony Liguori, Richard W.M. Jones
Cc: Daniel P. Berrange, Avi Kivity, kvm, qemu-devel
Can you send me pointers to the qdev documentation? How can I use it? Will
it allow us to scale above the 32 PCI limit?
Anjali
On 10/14/10 2:57 PM, "Anthony Liguori" <anthony@codemonkey.ws> wrote:
> On 10/14/2010 04:42 PM, Richard W.M. Jones wrote:
>> On Thu, Oct 14, 2010 at 01:10:47PM +0100, Daniel P. Berrange wrote:
>>
>>> Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
>>> any type of device we emulate.
>>>
>> Break the 29/30/31 virtio-blk limit ... please!
>>
>
> It was broken ages ago:
>
> anthony@howler:~$ wc -l /proc/partitions; tail /proc/partitions
> 422 /proc/partitions
> 251 1618 1 vdcx2
> 251 1621 489951 vdcx5
> 251 1632 10485760 vdcy
> 251 1633 9992398 vdcy1
> 251 1634 1 vdcy2
> 251 1637 489951 vdcy5
> 251 1648 10485760 vdcz
> 251 1649 9992398 vdcz1
> 251 1650 1 vdcz2
> 251 1653 489951 vdcz5
>
> This is what makes qdev so useful.
>
> args=""
> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
> for fn in 0 1 2 3 4 5 6 7; do
> args="$args -drive
> file=/home/anthony/images/linux.img,if=none,snapshot=on,id=disk${slot}_${fn}"
> args="$args -device
> virtio-blk-pci,addr=${slot}.${fn},drive=disk${slot}_${fn},multifunction=on"
> done
> done
>
> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
> -enable-kvm -serial stdio
>
> Regards,
>
> Anthony Liguori
>
>> Rich.
>>
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 72+ messages in thread* Re: [Qemu-devel] Hitting 29 NIC limit
@ 2010-10-14 22:00 ` Anjali Kulkarni
0 siblings, 0 replies; 72+ messages in thread
From: Anjali Kulkarni @ 2010-10-14 22:00 UTC (permalink / raw)
To: Anthony Liguori, Richard W.M. Jones; +Cc: Avi Kivity, kvm, qemu-devel
Can you send me pointers to the qdev documentation? How can I use it? Will
it allow us to scale above the 32 PCI limit?
Anjali
On 10/14/10 2:57 PM, "Anthony Liguori" <anthony@codemonkey.ws> wrote:
> On 10/14/2010 04:42 PM, Richard W.M. Jones wrote:
>> On Thu, Oct 14, 2010 at 01:10:47PM +0100, Daniel P. Berrange wrote:
>>
>>> Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
>>> any type of device we emulate.
>>>
>> Break the 29/30/31 virtio-blk limit ... please!
>>
>
> It was broken ages ago:
>
> anthony@howler:~$ wc -l /proc/partitions; tail /proc/partitions
> 422 /proc/partitions
> 251 1618 1 vdcx2
> 251 1621 489951 vdcx5
> 251 1632 10485760 vdcy
> 251 1633 9992398 vdcy1
> 251 1634 1 vdcy2
> 251 1637 489951 vdcy5
> 251 1648 10485760 vdcz
> 251 1649 9992398 vdcz1
> 251 1650 1 vdcz2
> 251 1653 489951 vdcz5
>
> This is what makes qdev so useful.
>
> args=""
> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
> for fn in 0 1 2 3 4 5 6 7; do
> args="$args -drive
> file=/home/anthony/images/linux.img,if=none,snapshot=on,id=disk${slot}_${fn}"
> args="$args -device
> virtio-blk-pci,addr=${slot}.${fn},drive=disk${slot}_${fn},multifunction=on"
> done
> done
>
> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
> -enable-kvm -serial stdio
>
> Regards,
>
> Anthony Liguori
>
>> Rich.
>>
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 72+ messages in thread* Re: [Qemu-devel] Hitting 29 NIC limit
2010-10-14 22:00 ` Anjali Kulkarni
@ 2010-10-14 22:09 ` Anthony Liguori
-1 siblings, 0 replies; 72+ messages in thread
From: Anthony Liguori @ 2010-10-14 22:09 UTC (permalink / raw)
To: Anjali Kulkarni
Cc: Richard W.M. Jones, Daniel P. Berrange, Avi Kivity, kvm, qemu-devel
On 10/14/2010 05:00 PM, Anjali Kulkarni wrote:
> Can you send me pointers to the qdev documentation? How can I use it? Will
> it allow us to scale above the 32 PCI limit?
>
It's all below. You just have to create a PCI device and mark the
multifunction flag to on and then assign it a PCI address that includes
a function number. Then you can pack 8 virtio PCI devices into a single
slot.
Regards,
Anthony Liguori
> Anjali
>
>
> On 10/14/10 2:57 PM, "Anthony Liguori"<anthony@codemonkey.ws> wrote:
>
>
>> On 10/14/2010 04:42 PM, Richard W.M. Jones wrote:
>>
>>> On Thu, Oct 14, 2010 at 01:10:47PM +0100, Daniel P. Berrange wrote:
>>>
>>>
>>>> Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
>>>> any type of device we emulate.
>>>>
>>>>
>>> Break the 29/30/31 virtio-blk limit ... please!
>>>
>>>
>> It was broken ages ago:
>>
>> anthony@howler:~$ wc -l /proc/partitions; tail /proc/partitions
>> 422 /proc/partitions
>> 251 1618 1 vdcx2
>> 251 1621 489951 vdcx5
>> 251 1632 10485760 vdcy
>> 251 1633 9992398 vdcy1
>> 251 1634 1 vdcy2
>> 251 1637 489951 vdcy5
>> 251 1648 10485760 vdcz
>> 251 1649 9992398 vdcz1
>> 251 1650 1 vdcz2
>> 251 1653 489951 vdcz5
>>
>> This is what makes qdev so useful.
>>
>> args=""
>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>> for fn in 0 1 2 3 4 5 6 7; do
>> args="$args -drive
>> file=/home/anthony/images/linux.img,if=none,snapshot=on,id=disk${slot}_${fn}"
>> args="$args -device
>> virtio-blk-pci,addr=${slot}.${fn},drive=disk${slot}_${fn},multifunction=on"
>> done
>> done
>>
>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>> -enable-kvm -serial stdio
>>
>> Regards,
>>
>> Anthony Liguori
>>
>>
>>> Rich.
>>>
>>>
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>
^ permalink raw reply [flat|nested] 72+ messages in thread* Re: [Qemu-devel] Hitting 29 NIC limit
@ 2010-10-14 22:09 ` Anthony Liguori
0 siblings, 0 replies; 72+ messages in thread
From: Anthony Liguori @ 2010-10-14 22:09 UTC (permalink / raw)
To: Anjali Kulkarni; +Cc: qemu-devel, Richard W.M. Jones, kvm, Avi Kivity
On 10/14/2010 05:00 PM, Anjali Kulkarni wrote:
> Can you send me pointers to the qdev documentation? How can I use it? Will
> it allow us to scale above the 32 PCI limit?
>
It's all below. You just have to create a PCI device and mark the
multifunction flag to on and then assign it a PCI address that includes
a function number. Then you can pack 8 virtio PCI devices into a single
slot.
Regards,
Anthony Liguori
> Anjali
>
>
> On 10/14/10 2:57 PM, "Anthony Liguori"<anthony@codemonkey.ws> wrote:
>
>
>> On 10/14/2010 04:42 PM, Richard W.M. Jones wrote:
>>
>>> On Thu, Oct 14, 2010 at 01:10:47PM +0100, Daniel P. Berrange wrote:
>>>
>>>
>>>> Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
>>>> any type of device we emulate.
>>>>
>>>>
>>> Break the 29/30/31 virtio-blk limit ... please!
>>>
>>>
>> It was broken ages ago:
>>
>> anthony@howler:~$ wc -l /proc/partitions; tail /proc/partitions
>> 422 /proc/partitions
>> 251 1618 1 vdcx2
>> 251 1621 489951 vdcx5
>> 251 1632 10485760 vdcy
>> 251 1633 9992398 vdcy1
>> 251 1634 1 vdcy2
>> 251 1637 489951 vdcy5
>> 251 1648 10485760 vdcz
>> 251 1649 9992398 vdcz1
>> 251 1650 1 vdcz2
>> 251 1653 489951 vdcz5
>>
>> This is what makes qdev so useful.
>>
>> args=""
>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>> for fn in 0 1 2 3 4 5 6 7; do
>> args="$args -drive
>> file=/home/anthony/images/linux.img,if=none,snapshot=on,id=disk${slot}_${fn}"
>> args="$args -device
>> virtio-blk-pci,addr=${slot}.${fn},drive=disk${slot}_${fn},multifunction=on"
>> done
>> done
>>
>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>> -enable-kvm -serial stdio
>>
>> Regards,
>>
>> Anthony Liguori
>>
>>
>>> Rich.
>>>
>>>
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>
^ permalink raw reply [flat|nested] 72+ messages in thread* Re: [Qemu-devel] Hitting 29 NIC limit
2010-10-14 22:09 ` Anthony Liguori
@ 2010-10-14 22:12 ` Anjali Kulkarni
-1 siblings, 0 replies; 72+ messages in thread
From: Anjali Kulkarni @ 2010-10-14 22:12 UTC (permalink / raw)
To: Anthony Liguori
Cc: Richard W.M. Jones, Daniel P. Berrange, Avi Kivity, kvm, qemu-devel
Thanks. Does this work for e1000 as well?
Also, does it support pci hotplug?
Anjali
On 10/14/10 3:09 PM, "Anthony Liguori" <anthony@codemonkey.ws> wrote:
> On 10/14/2010 05:00 PM, Anjali Kulkarni wrote:
>> Can you send me pointers to the qdev documentation? How can I use it? Will
>> it allow us to scale above the 32 PCI limit?
>>
>
> It's all below. You just have to create a PCI device and mark the
> multifunction flag to on and then assign it a PCI address that includes
> a function number. Then you can pack 8 virtio PCI devices into a single
> slot.
>
> Regards,
>
> Anthony Liguori
>
>> Anjali
>>
>>
>> On 10/14/10 2:57 PM, "Anthony Liguori"<anthony@codemonkey.ws> wrote:
>>
>>
>>> On 10/14/2010 04:42 PM, Richard W.M. Jones wrote:
>>>
>>>> On Thu, Oct 14, 2010 at 01:10:47PM +0100, Daniel P. Berrange wrote:
>>>>
>>>>
>>>>> Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
>>>>> any type of device we emulate.
>>>>>
>>>>>
>>>> Break the 29/30/31 virtio-blk limit ... please!
>>>>
>>>>
>>> It was broken ages ago:
>>>
>>> anthony@howler:~$ wc -l /proc/partitions; tail /proc/partitions
>>> 422 /proc/partitions
>>> 251 1618 1 vdcx2
>>> 251 1621 489951 vdcx5
>>> 251 1632 10485760 vdcy
>>> 251 1633 9992398 vdcy1
>>> 251 1634 1 vdcy2
>>> 251 1637 489951 vdcy5
>>> 251 1648 10485760 vdcz
>>> 251 1649 9992398 vdcz1
>>> 251 1650 1 vdcz2
>>> 251 1653 489951 vdcz5
>>>
>>> This is what makes qdev so useful.
>>>
>>> args=""
>>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>>> for fn in 0 1 2 3 4 5 6 7; do
>>> args="$args -drive
>>>
file=/home/anthony/images/linux.img,if=none,snapshot=on,id=disk${slot}_${fn}>>>
"
>>> args="$args -device
>>> virtio-blk-pci,addr=${slot}.${fn},drive=disk${slot}_${fn},multifunction=on"
>>> done
>>> done
>>>
>>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>>> -enable-kvm -serial stdio
>>>
>>> Regards,
>>>
>>> Anthony Liguori
>>>
>>>
>>>> Rich.
>>>>
>>>>
>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>
>
^ permalink raw reply [flat|nested] 72+ messages in thread* Re: [Qemu-devel] Hitting 29 NIC limit
@ 2010-10-14 22:12 ` Anjali Kulkarni
0 siblings, 0 replies; 72+ messages in thread
From: Anjali Kulkarni @ 2010-10-14 22:12 UTC (permalink / raw)
To: Anthony Liguori; +Cc: qemu-devel, Richard W.M. Jones, kvm, Avi Kivity
Thanks. Does this work for e1000 as well?
Also, does it support pci hotplug?
Anjali
On 10/14/10 3:09 PM, "Anthony Liguori" <anthony@codemonkey.ws> wrote:
> On 10/14/2010 05:00 PM, Anjali Kulkarni wrote:
>> Can you send me pointers to the qdev documentation? How can I use it? Will
>> it allow us to scale above the 32 PCI limit?
>>
>
> It's all below. You just have to create a PCI device and mark the
> multifunction flag to on and then assign it a PCI address that includes
> a function number. Then you can pack 8 virtio PCI devices into a single
> slot.
>
> Regards,
>
> Anthony Liguori
>
>> Anjali
>>
>>
>> On 10/14/10 2:57 PM, "Anthony Liguori"<anthony@codemonkey.ws> wrote:
>>
>>
>>> On 10/14/2010 04:42 PM, Richard W.M. Jones wrote:
>>>
>>>> On Thu, Oct 14, 2010 at 01:10:47PM +0100, Daniel P. Berrange wrote:
>>>>
>>>>
>>>>> Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
>>>>> any type of device we emulate.
>>>>>
>>>>>
>>>> Break the 29/30/31 virtio-blk limit ... please!
>>>>
>>>>
>>> It was broken ages ago:
>>>
>>> anthony@howler:~$ wc -l /proc/partitions; tail /proc/partitions
>>> 422 /proc/partitions
>>> 251 1618 1 vdcx2
>>> 251 1621 489951 vdcx5
>>> 251 1632 10485760 vdcy
>>> 251 1633 9992398 vdcy1
>>> 251 1634 1 vdcy2
>>> 251 1637 489951 vdcy5
>>> 251 1648 10485760 vdcz
>>> 251 1649 9992398 vdcz1
>>> 251 1650 1 vdcz2
>>> 251 1653 489951 vdcz5
>>>
>>> This is what makes qdev so useful.
>>>
>>> args=""
>>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>>> for fn in 0 1 2 3 4 5 6 7; do
>>> args="$args -drive
>>>
file=/home/anthony/images/linux.img,if=none,snapshot=on,id=disk${slot}_${fn}>>>
"
>>> args="$args -device
>>> virtio-blk-pci,addr=${slot}.${fn},drive=disk${slot}_${fn},multifunction=on"
>>> done
>>> done
>>>
>>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>>> -enable-kvm -serial stdio
>>>
>>> Regards,
>>>
>>> Anthony Liguori
>>>
>>>
>>>> Rich.
>>>>
>>>>
>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>
>
^ permalink raw reply [flat|nested] 72+ messages in thread* Re: [Qemu-devel] Hitting 29 NIC limit
2010-10-14 22:12 ` Anjali Kulkarni
@ 2010-10-14 22:17 ` Anthony Liguori
-1 siblings, 0 replies; 72+ messages in thread
From: Anthony Liguori @ 2010-10-14 22:17 UTC (permalink / raw)
To: Anjali Kulkarni
Cc: Richard W.M. Jones, Daniel P. Berrange, Avi Kivity, kvm, qemu-devel
On 10/14/2010 05:12 PM, Anjali Kulkarni wrote:
> Thanks. Does this work for e1000 as well?
>
Haven't tried. I don't know how various e1000 drivers would react.
> Also, does it support pci hotplug?
>
No, but that's fixable down the road.
Regards,
Anthony Liguori
> Anjali
>
> On 10/14/10 3:09 PM, "Anthony Liguori"<anthony@codemonkey.ws> wrote:
>
>
>> On 10/14/2010 05:00 PM, Anjali Kulkarni wrote:
>>
>>> Can you send me pointers to the qdev documentation? How can I use it? Will
>>> it allow us to scale above the 32 PCI limit?
>>>
>>>
>> It's all below. You just have to create a PCI device and mark the
>> multifunction flag to on and then assign it a PCI address that includes
>> a function number. Then you can pack 8 virtio PCI devices into a single
>> slot.
>>
>> Regards,
>>
>> Anthony Liguori
>>
>>
>>> Anjali
>>>
>>>
>>> On 10/14/10 2:57 PM, "Anthony Liguori"<anthony@codemonkey.ws> wrote:
>>>
>>>
>>>
>>>> On 10/14/2010 04:42 PM, Richard W.M. Jones wrote:
>>>>
>>>>
>>>>> On Thu, Oct 14, 2010 at 01:10:47PM +0100, Daniel P. Berrange wrote:
>>>>>
>>>>>
>>>>>
>>>>>> Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
>>>>>> any type of device we emulate.
>>>>>>
>>>>>>
>>>>>>
>>>>> Break the 29/30/31 virtio-blk limit ... please!
>>>>>
>>>>>
>>>>>
>>>> It was broken ages ago:
>>>>
>>>> anthony@howler:~$ wc -l /proc/partitions; tail /proc/partitions
>>>> 422 /proc/partitions
>>>> 251 1618 1 vdcx2
>>>> 251 1621 489951 vdcx5
>>>> 251 1632 10485760 vdcy
>>>> 251 1633 9992398 vdcy1
>>>> 251 1634 1 vdcy2
>>>> 251 1637 489951 vdcy5
>>>> 251 1648 10485760 vdcz
>>>> 251 1649 9992398 vdcz1
>>>> 251 1650 1 vdcz2
>>>> 251 1653 489951 vdcz5
>>>>
>>>> This is what makes qdev so useful.
>>>>
>>>> args=""
>>>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>>>> for fn in 0 1 2 3 4 5 6 7; do
>>>> args="$args -drive
>>>>
>>>>
> file=/home/anthony/images/linux.img,if=none,snapshot=on,id=disk${slot}_${fn}>>>
> "
>
>>>> args="$args -device
>>>> virtio-blk-pci,addr=${slot}.${fn},drive=disk${slot}_${fn},multifunction=on"
>>>> done
>>>> done
>>>>
>>>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>>>> -enable-kvm -serial stdio
>>>>
>>>> Regards,
>>>>
>>>> Anthony Liguori
>>>>
>>>>
>>>>
>>>>> Rich.
>>>>>
>>>>>
>>>>>
>>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>
>>>>
>>>
>>>
>>
>
^ permalink raw reply [flat|nested] 72+ messages in thread* Re: [Qemu-devel] Hitting 29 NIC limit
@ 2010-10-14 22:17 ` Anthony Liguori
0 siblings, 0 replies; 72+ messages in thread
From: Anthony Liguori @ 2010-10-14 22:17 UTC (permalink / raw)
To: Anjali Kulkarni; +Cc: qemu-devel, Richard W.M. Jones, kvm, Avi Kivity
On 10/14/2010 05:12 PM, Anjali Kulkarni wrote:
> Thanks. Does this work for e1000 as well?
>
Haven't tried. I don't know how various e1000 drivers would react.
> Also, does it support pci hotplug?
>
No, but that's fixable down the road.
Regards,
Anthony Liguori
> Anjali
>
> On 10/14/10 3:09 PM, "Anthony Liguori"<anthony@codemonkey.ws> wrote:
>
>
>> On 10/14/2010 05:00 PM, Anjali Kulkarni wrote:
>>
>>> Can you send me pointers to the qdev documentation? How can I use it? Will
>>> it allow us to scale above the 32 PCI limit?
>>>
>>>
>> It's all below. You just have to create a PCI device and mark the
>> multifunction flag to on and then assign it a PCI address that includes
>> a function number. Then you can pack 8 virtio PCI devices into a single
>> slot.
>>
>> Regards,
>>
>> Anthony Liguori
>>
>>
>>> Anjali
>>>
>>>
>>> On 10/14/10 2:57 PM, "Anthony Liguori"<anthony@codemonkey.ws> wrote:
>>>
>>>
>>>
>>>> On 10/14/2010 04:42 PM, Richard W.M. Jones wrote:
>>>>
>>>>
>>>>> On Thu, Oct 14, 2010 at 01:10:47PM +0100, Daniel P. Berrange wrote:
>>>>>
>>>>>
>>>>>
>>>>>> Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
>>>>>> any type of device we emulate.
>>>>>>
>>>>>>
>>>>>>
>>>>> Break the 29/30/31 virtio-blk limit ... please!
>>>>>
>>>>>
>>>>>
>>>> It was broken ages ago:
>>>>
>>>> anthony@howler:~$ wc -l /proc/partitions; tail /proc/partitions
>>>> 422 /proc/partitions
>>>> 251 1618 1 vdcx2
>>>> 251 1621 489951 vdcx5
>>>> 251 1632 10485760 vdcy
>>>> 251 1633 9992398 vdcy1
>>>> 251 1634 1 vdcy2
>>>> 251 1637 489951 vdcy5
>>>> 251 1648 10485760 vdcz
>>>> 251 1649 9992398 vdcz1
>>>> 251 1650 1 vdcz2
>>>> 251 1653 489951 vdcz5
>>>>
>>>> This is what makes qdev so useful.
>>>>
>>>> args=""
>>>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>>>> for fn in 0 1 2 3 4 5 6 7; do
>>>> args="$args -drive
>>>>
>>>>
> file=/home/anthony/images/linux.img,if=none,snapshot=on,id=disk${slot}_${fn}>>>
> "
>
>>>> args="$args -device
>>>> virtio-blk-pci,addr=${slot}.${fn},drive=disk${slot}_${fn},multifunction=on"
>>>> done
>>>> done
>>>>
>>>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>>>> -enable-kvm -serial stdio
>>>>
>>>> Regards,
>>>>
>>>> Anthony Liguori
>>>>
>>>>
>>>>
>>>>> Rich.
>>>>>
>>>>>
>>>>>
>>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>
>>>>
>>>
>>>
>>
>
^ permalink raw reply [flat|nested] 72+ messages in thread* Re: Hitting 29 NIC limit
2010-10-14 22:17 ` Anthony Liguori
@ 2010-11-16 17:23 ` Michael S. Tsirkin
-1 siblings, 0 replies; 72+ messages in thread
From: Michael S. Tsirkin @ 2010-11-16 17:23 UTC (permalink / raw)
To: Anthony Liguori
Cc: Anjali Kulkarni, Richard W.M. Jones, Daniel P. Berrange,
Avi Kivity, kvm, qemu-devel
On Thu, Oct 14, 2010 at 05:17:36PM -0500, Anthony Liguori wrote:
> On 10/14/2010 05:12 PM, Anjali Kulkarni wrote:
> >Thanks. Does this work for e1000 as well?
>
> Haven't tried. I don't know how various e1000 drivers would react.
>
> >Also, does it support pci hotplug?
>
> No, but that's fixable down the road.
>
> Regards,
>
> Anthony Liguori
Probably not.
^ permalink raw reply [flat|nested] 72+ messages in thread
* [Qemu-devel] Re: Hitting 29 NIC limit
@ 2010-11-16 17:23 ` Michael S. Tsirkin
0 siblings, 0 replies; 72+ messages in thread
From: Michael S. Tsirkin @ 2010-11-16 17:23 UTC (permalink / raw)
To: Anthony Liguori
Cc: Anjali Kulkarni, kvm, qemu-devel, Richard W.M. Jones, Avi Kivity
On Thu, Oct 14, 2010 at 05:17:36PM -0500, Anthony Liguori wrote:
> On 10/14/2010 05:12 PM, Anjali Kulkarni wrote:
> >Thanks. Does this work for e1000 as well?
>
> Haven't tried. I don't know how various e1000 drivers would react.
>
> >Also, does it support pci hotplug?
>
> No, but that's fixable down the road.
>
> Regards,
>
> Anthony Liguori
Probably not.
^ permalink raw reply [flat|nested] 72+ messages in thread
* Re: Hitting 29 NIC limit
2010-11-16 17:23 ` [Qemu-devel] " Michael S. Tsirkin
@ 2010-11-16 19:00 ` Anthony Liguori
-1 siblings, 0 replies; 72+ messages in thread
From: Anthony Liguori @ 2010-11-16 19:00 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Anjali Kulkarni, Richard W.M. Jones, Daniel P. Berrange,
Avi Kivity, kvm, qemu-devel
On 11/16/2010 11:23 AM, Michael S. Tsirkin wrote:
> On Thu, Oct 14, 2010 at 05:17:36PM -0500, Anthony Liguori wrote:
>
>> On 10/14/2010 05:12 PM, Anjali Kulkarni wrote:
>>
>>> Thanks. Does this work for e1000 as well?
>>>
>> Haven't tried. I don't know how various e1000 drivers would react.
>>
>>
>>> Also, does it support pci hotplug?
>>>
>> No, but that's fixable down the road.
>>
>> Regards,
>>
>> Anthony Liguori
>>
> Probably not.
>
FWIW, by fixable, I meant, we could add the ability to hot plug a
multi-port NIC device. For instance, we could have -device
virtio-net-pci-x8 which would be an 8-port virtio-net card.
Regards,
Anthony Liguori
^ permalink raw reply [flat|nested] 72+ messages in thread
* [Qemu-devel] Re: Hitting 29 NIC limit
@ 2010-11-16 19:00 ` Anthony Liguori
0 siblings, 0 replies; 72+ messages in thread
From: Anthony Liguori @ 2010-11-16 19:00 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Anjali Kulkarni, kvm, qemu-devel, Richard W.M. Jones, Avi Kivity
On 11/16/2010 11:23 AM, Michael S. Tsirkin wrote:
> On Thu, Oct 14, 2010 at 05:17:36PM -0500, Anthony Liguori wrote:
>
>> On 10/14/2010 05:12 PM, Anjali Kulkarni wrote:
>>
>>> Thanks. Does this work for e1000 as well?
>>>
>> Haven't tried. I don't know how various e1000 drivers would react.
>>
>>
>>> Also, does it support pci hotplug?
>>>
>> No, but that's fixable down the road.
>>
>> Regards,
>>
>> Anthony Liguori
>>
> Probably not.
>
FWIW, by fixable, I meant, we could add the ability to hot plug a
multi-port NIC device. For instance, we could have -device
virtio-net-pci-x8 which would be an 8-port virtio-net card.
Regards,
Anthony Liguori
^ permalink raw reply [flat|nested] 72+ messages in thread
* Re: [Qemu-devel] Hitting 29 NIC limit
2010-10-14 21:57 ` Anthony Liguori
@ 2010-10-14 22:08 ` Richard W.M. Jones
-1 siblings, 0 replies; 72+ messages in thread
From: Richard W.M. Jones @ 2010-10-14 22:08 UTC (permalink / raw)
To: Anthony Liguori
Cc: Daniel P. Berrange, Avi Kivity, Anjali Kulkarni, kvm, qemu-devel
On Thu, Oct 14, 2010 at 04:57:36PM -0500, Anthony Liguori wrote:
> On 10/14/2010 04:42 PM, Richard W.M. Jones wrote:
> >On Thu, Oct 14, 2010 at 01:10:47PM +0100, Daniel P. Berrange wrote:
> >>Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
> >>any type of device we emulate.
> >Break the 29/30/31 virtio-blk limit ... please!
>
> It was broken ages ago:
[...]
Excellent news indeeed.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
^ permalink raw reply [flat|nested] 72+ messages in thread
* Re: [Qemu-devel] Hitting 29 NIC limit
@ 2010-10-14 22:08 ` Richard W.M. Jones
0 siblings, 0 replies; 72+ messages in thread
From: Richard W.M. Jones @ 2010-10-14 22:08 UTC (permalink / raw)
To: Anthony Liguori; +Cc: Anjali Kulkarni, Avi Kivity, kvm, qemu-devel
On Thu, Oct 14, 2010 at 04:57:36PM -0500, Anthony Liguori wrote:
> On 10/14/2010 04:42 PM, Richard W.M. Jones wrote:
> >On Thu, Oct 14, 2010 at 01:10:47PM +0100, Daniel P. Berrange wrote:
> >>Or a PCI bridge to wire up more PCI buses, so we raise the max limit for
> >>any type of device we emulate.
> >Break the 29/30/31 virtio-blk limit ... please!
>
> It was broken ages ago:
[...]
Excellent news indeeed.
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
^ permalink raw reply [flat|nested] 72+ messages in thread
* Re: [Qemu-devel] Hitting 29 NIC limit
2010-10-14 12:07 ` Avi Kivity
@ 2010-10-14 12:36 ` Markus Armbruster
-1 siblings, 0 replies; 72+ messages in thread
From: Markus Armbruster @ 2010-10-14 12:36 UTC (permalink / raw)
To: Avi Kivity; +Cc: Anthony Liguori, Anjali Kulkarni, qemu-devel, kvm
Avi Kivity <avi@redhat.com> writes:
> On 10/14/2010 12:54 AM, Anthony Liguori wrote:
>> On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
>>>
>>> Hi,
>>>
>>> Using the legacy way of starting up NICs, I am hitting a limitation
>>> after 29
>>> NICs ie no more than 29 are detected (that's because of the 32 PCI slot
>>> limit on a single bus- 3 are already taken up)
>>> I had initially increased the MAX_NICS to 48, just on my tree, to get to
>>> more, but ofcource that wont work.
>>> Is there any way to go beyond 29 NICs the legacy way? What is the
>>> maximum
>>> that can be supported by the qdev mothod?
>>
>> I got up to 104 without trying very hard using the following script:
>>
>> args=""
>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>> for fn in 0 1 2 3 4 5 6 7; do
>> args="$args -netdev user,id=eth${slot}_${fn}"
>> args="$args -device
>> virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
>> done
>> done
>>
>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>> -enable-kvm
>>
>> The key is to make the virtio-net devices multifunction and to fill
>> out all 8 functions for each slot.
I'm amazed that works. Can't see how creating another qdev in the same
slot makes a proper multifunction device.
> This is unlikely to work right wrt pci hotplug. If we want to support
> a large number of interfaces, we need true multiport cards.
Indeed. As far as I know, we can't hot plug multifunction PCI devices.
> What's the motivation for such a huge number of interfaces?
^ permalink raw reply [flat|nested] 72+ messages in thread* Re: [Qemu-devel] Hitting 29 NIC limit
@ 2010-10-14 12:36 ` Markus Armbruster
0 siblings, 0 replies; 72+ messages in thread
From: Markus Armbruster @ 2010-10-14 12:36 UTC (permalink / raw)
To: Avi Kivity; +Cc: Anjali Kulkarni, qemu-devel, kvm
Avi Kivity <avi@redhat.com> writes:
> On 10/14/2010 12:54 AM, Anthony Liguori wrote:
>> On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
>>>
>>> Hi,
>>>
>>> Using the legacy way of starting up NICs, I am hitting a limitation
>>> after 29
>>> NICs ie no more than 29 are detected (that's because of the 32 PCI slot
>>> limit on a single bus- 3 are already taken up)
>>> I had initially increased the MAX_NICS to 48, just on my tree, to get to
>>> more, but ofcource that wont work.
>>> Is there any way to go beyond 29 NICs the legacy way? What is the
>>> maximum
>>> that can be supported by the qdev mothod?
>>
>> I got up to 104 without trying very hard using the following script:
>>
>> args=""
>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>> for fn in 0 1 2 3 4 5 6 7; do
>> args="$args -netdev user,id=eth${slot}_${fn}"
>> args="$args -device
>> virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
>> done
>> done
>>
>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>> -enable-kvm
>>
>> The key is to make the virtio-net devices multifunction and to fill
>> out all 8 functions for each slot.
I'm amazed that works. Can't see how creating another qdev in the same
slot makes a proper multifunction device.
> This is unlikely to work right wrt pci hotplug. If we want to support
> a large number of interfaces, we need true multiport cards.
Indeed. As far as I know, we can't hot plug multifunction PCI devices.
> What's the motivation for such a huge number of interfaces?
^ permalink raw reply [flat|nested] 72+ messages in thread* Re: [Qemu-devel] Hitting 29 NIC limit
2010-10-14 12:36 ` Markus Armbruster
@ 2010-10-14 12:59 ` Anthony Liguori
-1 siblings, 0 replies; 72+ messages in thread
From: Anthony Liguori @ 2010-10-14 12:59 UTC (permalink / raw)
To: Markus Armbruster; +Cc: Avi Kivity, Anjali Kulkarni, qemu-devel, kvm
On 10/14/2010 07:36 AM, Markus Armbruster wrote:
> Avi Kivity<avi@redhat.com> writes:
>
>
>> On 10/14/2010 12:54 AM, Anthony Liguori wrote:
>>
>>> On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
>>>
>>>> Hi,
>>>>
>>>> Using the legacy way of starting up NICs, I am hitting a limitation
>>>> after 29
>>>> NICs ie no more than 29 are detected (that's because of the 32 PCI slot
>>>> limit on a single bus- 3 are already taken up)
>>>> I had initially increased the MAX_NICS to 48, just on my tree, to get to
>>>> more, but ofcource that wont work.
>>>> Is there any way to go beyond 29 NICs the legacy way? What is the
>>>> maximum
>>>> that can be supported by the qdev mothod?
>>>>
>>> I got up to 104 without trying very hard using the following script:
>>>
>>> args=""
>>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>>> for fn in 0 1 2 3 4 5 6 7; do
>>> args="$args -netdev user,id=eth${slot}_${fn}"
>>> args="$args -device
>>> virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
>>> done
>>> done
>>>
>>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>>> -enable-kvm
>>>
>>> The key is to make the virtio-net devices multifunction and to fill
>>> out all 8 functions for each slot.
>>>
> I'm amazed that works. Can't see how creating another qdev in the same
> slot makes a proper multifunction device.
>
multifunction=on sets the multifunction bit for the PCI device. Then
it's a matter of setting the address to be a specific function.
Our default platform devices are actually multifunction.
>> This is unlikely to work right wrt pci hotplug. If we want to support
>> a large number of interfaces, we need true multiport cards.
>>
> Indeed. As far as I know, we can't hot plug multifunction PCI devices.
>
Yup.
Regards,
Anthony Liguori
>> What's the motivation for such a huge number of interfaces?
>>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 72+ messages in thread* Re: [Qemu-devel] Hitting 29 NIC limit
@ 2010-10-14 12:59 ` Anthony Liguori
0 siblings, 0 replies; 72+ messages in thread
From: Anthony Liguori @ 2010-10-14 12:59 UTC (permalink / raw)
To: Markus Armbruster; +Cc: Anjali Kulkarni, Avi Kivity, kvm, qemu-devel
On 10/14/2010 07:36 AM, Markus Armbruster wrote:
> Avi Kivity<avi@redhat.com> writes:
>
>
>> On 10/14/2010 12:54 AM, Anthony Liguori wrote:
>>
>>> On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
>>>
>>>> Hi,
>>>>
>>>> Using the legacy way of starting up NICs, I am hitting a limitation
>>>> after 29
>>>> NICs ie no more than 29 are detected (that's because of the 32 PCI slot
>>>> limit on a single bus- 3 are already taken up)
>>>> I had initially increased the MAX_NICS to 48, just on my tree, to get to
>>>> more, but ofcource that wont work.
>>>> Is there any way to go beyond 29 NICs the legacy way? What is the
>>>> maximum
>>>> that can be supported by the qdev mothod?
>>>>
>>> I got up to 104 without trying very hard using the following script:
>>>
>>> args=""
>>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>>> for fn in 0 1 2 3 4 5 6 7; do
>>> args="$args -netdev user,id=eth${slot}_${fn}"
>>> args="$args -device
>>> virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
>>> done
>>> done
>>>
>>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>>> -enable-kvm
>>>
>>> The key is to make the virtio-net devices multifunction and to fill
>>> out all 8 functions for each slot.
>>>
> I'm amazed that works. Can't see how creating another qdev in the same
> slot makes a proper multifunction device.
>
multifunction=on sets the multifunction bit for the PCI device. Then
it's a matter of setting the address to be a specific function.
Our default platform devices are actually multifunction.
>> This is unlikely to work right wrt pci hotplug. If we want to support
>> a large number of interfaces, we need true multiport cards.
>>
> Indeed. As far as I know, we can't hot plug multifunction PCI devices.
>
Yup.
Regards,
Anthony Liguori
>> What's the motivation for such a huge number of interfaces?
>>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 72+ messages in thread* Re: Hitting 29 NIC limit
2010-10-14 12:59 ` Anthony Liguori
@ 2010-10-15 9:11 ` Markus Armbruster
-1 siblings, 0 replies; 72+ messages in thread
From: Markus Armbruster @ 2010-10-15 9:11 UTC (permalink / raw)
To: Anthony Liguori; +Cc: Anjali Kulkarni, Avi Kivity, kvm, qemu-devel
Anthony Liguori <anthony@codemonkey.ws> writes:
> On 10/14/2010 07:36 AM, Markus Armbruster wrote:
>> Avi Kivity<avi@redhat.com> writes:
>>
>>
>>> On 10/14/2010 12:54 AM, Anthony Liguori wrote:
>>>
>>>> On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Using the legacy way of starting up NICs, I am hitting a limitation
>>>>> after 29
>>>>> NICs ie no more than 29 are detected (that's because of the 32 PCI slot
>>>>> limit on a single bus- 3 are already taken up)
>>>>> I had initially increased the MAX_NICS to 48, just on my tree, to get to
>>>>> more, but ofcource that wont work.
>>>>> Is there any way to go beyond 29 NICs the legacy way? What is the
>>>>> maximum
>>>>> that can be supported by the qdev mothod?
>>>>>
>>>> I got up to 104 without trying very hard using the following script:
>>>>
>>>> args=""
>>>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>>>> for fn in 0 1 2 3 4 5 6 7; do
>>>> args="$args -netdev user,id=eth${slot}_${fn}"
>>>> args="$args -device
>>>> virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
>>>> done
>>>> done
>>>>
>>>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>>>> -enable-kvm
>>>>
>>>> The key is to make the virtio-net devices multifunction and to fill
>>>> out all 8 functions for each slot.
>>>>
>> I'm amazed that works. Can't see how creating another qdev in the same
>> slot makes a proper multifunction device.
>>
>
> multifunction=on sets the multifunction bit for the PCI device. Then
> it's a matter of setting the address to be a specific function.
I missed that bit, or perhaps forgot it already. Neat!
Unfortunately, it's not yet good enough for hot plug. We need to plug
the complete device in one go, not function by function. This is harder
than hot unplug, where we can just add a command "unplug this PCI
device", then device_del its functions at our leisure. The symmetric
way to plug would be device_add the functions, then plug the device.
Backward compatibility headache: current device_add/del plug/unplug
automatically.
> Our default platform devices are actually multifunction.
Yes, but they're built "by hand".
[...]
^ permalink raw reply [flat|nested] 72+ messages in thread* Re: [Qemu-devel] Hitting 29 NIC limit
@ 2010-10-15 9:11 ` Markus Armbruster
0 siblings, 0 replies; 72+ messages in thread
From: Markus Armbruster @ 2010-10-15 9:11 UTC (permalink / raw)
To: Anthony Liguori; +Cc: Anjali Kulkarni, Avi Kivity, kvm, qemu-devel
Anthony Liguori <anthony@codemonkey.ws> writes:
> On 10/14/2010 07:36 AM, Markus Armbruster wrote:
>> Avi Kivity<avi@redhat.com> writes:
>>
>>
>>> On 10/14/2010 12:54 AM, Anthony Liguori wrote:
>>>
>>>> On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Using the legacy way of starting up NICs, I am hitting a limitation
>>>>> after 29
>>>>> NICs ie no more than 29 are detected (that's because of the 32 PCI slot
>>>>> limit on a single bus- 3 are already taken up)
>>>>> I had initially increased the MAX_NICS to 48, just on my tree, to get to
>>>>> more, but ofcource that wont work.
>>>>> Is there any way to go beyond 29 NICs the legacy way? What is the
>>>>> maximum
>>>>> that can be supported by the qdev mothod?
>>>>>
>>>> I got up to 104 without trying very hard using the following script:
>>>>
>>>> args=""
>>>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>>>> for fn in 0 1 2 3 4 5 6 7; do
>>>> args="$args -netdev user,id=eth${slot}_${fn}"
>>>> args="$args -device
>>>> virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
>>>> done
>>>> done
>>>>
>>>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>>>> -enable-kvm
>>>>
>>>> The key is to make the virtio-net devices multifunction and to fill
>>>> out all 8 functions for each slot.
>>>>
>> I'm amazed that works. Can't see how creating another qdev in the same
>> slot makes a proper multifunction device.
>>
>
> multifunction=on sets the multifunction bit for the PCI device. Then
> it's a matter of setting the address to be a specific function.
I missed that bit, or perhaps forgot it already. Neat!
Unfortunately, it's not yet good enough for hot plug. We need to plug
the complete device in one go, not function by function. This is harder
than hot unplug, where we can just add a command "unplug this PCI
device", then device_del its functions at our leisure. The symmetric
way to plug would be device_add the functions, then plug the device.
Backward compatibility headache: current device_add/del plug/unplug
automatically.
> Our default platform devices are actually multifunction.
Yes, but they're built "by hand".
[...]
^ permalink raw reply [flat|nested] 72+ messages in thread* Passing in additional info to guest OS and e1000 test suite?
2010-10-15 9:11 ` [Qemu-devel] " Markus Armbruster
@ 2010-10-15 18:49 ` Anjali Kulkarni
-1 siblings, 0 replies; 72+ messages in thread
From: Anjali Kulkarni @ 2010-10-15 18:49 UTC (permalink / raw)
To: kvm, qemu-devel
Hi,
- If I want to pass in additional arguments to the guest OS while booting(in
particular which slot I want to map a nic to) - is there any way to do it?
Some kind of configuration file that I can pass in would also be ok for me.
- Is there a KVM/Qemu/e1000 test suite that is already available and I can
us?
Thanks
Anjali
^ permalink raw reply [flat|nested] 72+ messages in thread
* [Qemu-devel] Passing in additional info to guest OS and e1000 test suite?
@ 2010-10-15 18:49 ` Anjali Kulkarni
0 siblings, 0 replies; 72+ messages in thread
From: Anjali Kulkarni @ 2010-10-15 18:49 UTC (permalink / raw)
To: kvm, qemu-devel
Hi,
- If I want to pass in additional arguments to the guest OS while booting(in
particular which slot I want to map a nic to) - is there any way to do it?
Some kind of configuration file that I can pass in would also be ok for me.
- Is there a KVM/Qemu/e1000 test suite that is already available and I can
us?
Thanks
Anjali
^ permalink raw reply [flat|nested] 72+ messages in thread
* Re: Passing in additional info to guest OS and e1000 test suite?
2010-10-15 18:49 ` [Qemu-devel] " Anjali Kulkarni
@ 2010-10-15 20:02 ` Anthony Liguori
-1 siblings, 0 replies; 72+ messages in thread
From: Anthony Liguori @ 2010-10-15 20:02 UTC (permalink / raw)
To: Anjali Kulkarni; +Cc: kvm, qemu-devel
On 10/15/2010 01:49 PM, Anjali Kulkarni wrote:
> Hi,
>
> - If I want to pass in additional arguments to the guest OS while booting(in
> particular which slot I want to map a nic to) - is there any way to do it?
> Some kind of configuration file that I can pass in would also be ok for me.
> - Is there a KVM/Qemu/e1000 test suite that is already available and I can
> us?
>
I'm a little confused about what you're really asking. Can you give a
more illustrated example of what you're asking? Are you using libvirt?
> Thanks
> Anjali
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 72+ messages in thread
* [Qemu-devel] Re: Passing in additional info to guest OS and e1000 test suite?
@ 2010-10-15 20:02 ` Anthony Liguori
0 siblings, 0 replies; 72+ messages in thread
From: Anthony Liguori @ 2010-10-15 20:02 UTC (permalink / raw)
To: Anjali Kulkarni; +Cc: qemu-devel, kvm
On 10/15/2010 01:49 PM, Anjali Kulkarni wrote:
> Hi,
>
> - If I want to pass in additional arguments to the guest OS while booting(in
> particular which slot I want to map a nic to) - is there any way to do it?
> Some kind of configuration file that I can pass in would also be ok for me.
> - Is there a KVM/Qemu/e1000 test suite that is already available and I can
> us?
>
I'm a little confused about what you're really asking. Can you give a
more illustrated example of what you're asking? Are you using libvirt?
> Thanks
> Anjali
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 72+ messages in thread
* Re: [Qemu-devel] Hitting 29 NIC limit
2010-10-14 12:07 ` Avi Kivity
@ 2010-10-14 12:54 ` Anthony Liguori
-1 siblings, 0 replies; 72+ messages in thread
From: Anthony Liguori @ 2010-10-14 12:54 UTC (permalink / raw)
To: Avi Kivity; +Cc: Anjali Kulkarni, kvm, qemu-devel
On 10/14/2010 07:07 AM, Avi Kivity wrote:
> On 10/14/2010 12:54 AM, Anthony Liguori wrote:
>> On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
>>>
>>> Hi,
>>>
>>> Using the legacy way of starting up NICs, I am hitting a limitation
>>> after 29
>>> NICs ie no more than 29 are detected (that's because of the 32 PCI slot
>>> limit on a single bus- 3 are already taken up)
>>> I had initially increased the MAX_NICS to 48, just on my tree, to
>>> get to
>>> more, but ofcource that wont work.
>>> Is there any way to go beyond 29 NICs the legacy way? What is the
>>> maximum
>>> that can be supported by the qdev mothod?
>>
>> I got up to 104 without trying very hard using the following script:
>>
>> args=""
>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>> for fn in 0 1 2 3 4 5 6 7; do
>> args="$args -netdev user,id=eth${slot}_${fn}"
>> args="$args -device
>> virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
>>
>> done
>> done
>>
>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>> -enable-kvm
>>
>> The key is to make the virtio-net devices multifunction and to fill
>> out all 8 functions for each slot.
>
> This is unlikely to work right wrt pci hotplug.
Yes. Our hotplug design is based on devices.. This is wrong, it should
be based on bus-level concepts (like PCI slots).
> If we want to support a large number of interfaces, we need true
> multiport cards.
This magic here creates a multiport virtio-net card so I'm not really
sure what you're suggesting. It would certainly be nice to make this
all more user friendly (and make hotplug work).
Regards,
Anthony Liguori
> What's the motivation for such a huge number of interfaces?
^ permalink raw reply [flat|nested] 72+ messages in thread* Re: [Qemu-devel] Hitting 29 NIC limit
@ 2010-10-14 12:54 ` Anthony Liguori
0 siblings, 0 replies; 72+ messages in thread
From: Anthony Liguori @ 2010-10-14 12:54 UTC (permalink / raw)
To: Avi Kivity; +Cc: Anjali Kulkarni, qemu-devel, kvm
On 10/14/2010 07:07 AM, Avi Kivity wrote:
> On 10/14/2010 12:54 AM, Anthony Liguori wrote:
>> On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
>>>
>>> Hi,
>>>
>>> Using the legacy way of starting up NICs, I am hitting a limitation
>>> after 29
>>> NICs ie no more than 29 are detected (that's because of the 32 PCI slot
>>> limit on a single bus- 3 are already taken up)
>>> I had initially increased the MAX_NICS to 48, just on my tree, to
>>> get to
>>> more, but ofcource that wont work.
>>> Is there any way to go beyond 29 NICs the legacy way? What is the
>>> maximum
>>> that can be supported by the qdev mothod?
>>
>> I got up to 104 without trying very hard using the following script:
>>
>> args=""
>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
>> for fn in 0 1 2 3 4 5 6 7; do
>> args="$args -netdev user,id=eth${slot}_${fn}"
>> args="$args -device
>> virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
>>
>> done
>> done
>>
>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
>> -enable-kvm
>>
>> The key is to make the virtio-net devices multifunction and to fill
>> out all 8 functions for each slot.
>
> This is unlikely to work right wrt pci hotplug.
Yes. Our hotplug design is based on devices.. This is wrong, it should
be based on bus-level concepts (like PCI slots).
> If we want to support a large number of interfaces, we need true
> multiport cards.
This magic here creates a multiport virtio-net card so I'm not really
sure what you're suggesting. It would certainly be nice to make this
all more user friendly (and make hotplug work).
Regards,
Anthony Liguori
> What's the motivation for such a huge number of interfaces?
^ permalink raw reply [flat|nested] 72+ messages in thread* Re: [Qemu-devel] Hitting 29 NIC limit
2010-10-14 12:54 ` Anthony Liguori
@ 2010-10-14 13:23 ` Avi Kivity
-1 siblings, 0 replies; 72+ messages in thread
From: Avi Kivity @ 2010-10-14 13:23 UTC (permalink / raw)
To: Anthony Liguori; +Cc: Anjali Kulkarni, kvm, qemu-devel
On 10/14/2010 02:54 PM, Anthony Liguori wrote:
>>> The key is to make the virtio-net devices multifunction and to fill
>>> out all 8 functions for each slot.
>>
>> This is unlikely to work right wrt pci hotplug.
>
>
> Yes. Our hotplug design is based on devices.. This is wrong, it
> should be based on bus-level concepts (like PCI slots).
>
>> If we want to support a large number of interfaces, we need true
>> multiport cards.
>
> This magic here creates a multiport virtio-net card so I'm not really
> sure what you're suggesting. It would certainly be nice to make this
> all more user friendly (and make hotplug work).
>
The big issue is to fix hotplug.
I don't see how we can make it user friendly, without making the
ordinary case even more unfriendly. Looks like we need yet another
level of indirection here.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply [flat|nested] 72+ messages in thread
* Re: [Qemu-devel] Hitting 29 NIC limit
@ 2010-10-14 13:23 ` Avi Kivity
0 siblings, 0 replies; 72+ messages in thread
From: Avi Kivity @ 2010-10-14 13:23 UTC (permalink / raw)
To: Anthony Liguori; +Cc: Anjali Kulkarni, qemu-devel, kvm
On 10/14/2010 02:54 PM, Anthony Liguori wrote:
>>> The key is to make the virtio-net devices multifunction and to fill
>>> out all 8 functions for each slot.
>>
>> This is unlikely to work right wrt pci hotplug.
>
>
> Yes. Our hotplug design is based on devices.. This is wrong, it
> should be based on bus-level concepts (like PCI slots).
>
>> If we want to support a large number of interfaces, we need true
>> multiport cards.
>
> This magic here creates a multiport virtio-net card so I'm not really
> sure what you're suggesting. It would certainly be nice to make this
> all more user friendly (and make hotplug work).
>
The big issue is to fix hotplug.
I don't see how we can make it user friendly, without making the
ordinary case even more unfriendly. Looks like we need yet another
level of indirection here.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply [flat|nested] 72+ messages in thread
* Re: [Qemu-devel] Hitting 29 NIC limit
2010-10-14 13:23 ` Avi Kivity
@ 2010-10-14 14:11 ` Anthony Liguori
-1 siblings, 0 replies; 72+ messages in thread
From: Anthony Liguori @ 2010-10-14 14:11 UTC (permalink / raw)
To: Avi Kivity; +Cc: Anjali Kulkarni, kvm, qemu-devel
On 10/14/2010 08:23 AM, Avi Kivity wrote:
> On 10/14/2010 02:54 PM, Anthony Liguori wrote:
>>>> The key is to make the virtio-net devices multifunction and to fill
>>>> out all 8 functions for each slot.
>>>
>>> This is unlikely to work right wrt pci hotplug.
>>
>>
>> Yes. Our hotplug design is based on devices.. This is wrong, it
>> should be based on bus-level concepts (like PCI slots).
>>
>>> If we want to support a large number of interfaces, we need true
>>> multiport cards.
>>
>> This magic here creates a multiport virtio-net card so I'm not really
>> sure what you're suggesting. It would certainly be nice to make this
>> all more user friendly (and make hotplug work).
>>
>
> The big issue is to fix hotplug.
Yes, but this is entirely independent of multifunction devices.
Today we shoe-horn hot remove into device_del. Instead, we should have
explicit bus-level interfaces for hot remove.
Regards,
Anthony Liguori
> I don't see how we can make it user friendly, without making the
> ordinary case even more unfriendly. Looks like we need yet another
> level of indirection here.
>
^ permalink raw reply [flat|nested] 72+ messages in thread
* Re: [Qemu-devel] Hitting 29 NIC limit
@ 2010-10-14 14:11 ` Anthony Liguori
0 siblings, 0 replies; 72+ messages in thread
From: Anthony Liguori @ 2010-10-14 14:11 UTC (permalink / raw)
To: Avi Kivity; +Cc: Anjali Kulkarni, qemu-devel, kvm
On 10/14/2010 08:23 AM, Avi Kivity wrote:
> On 10/14/2010 02:54 PM, Anthony Liguori wrote:
>>>> The key is to make the virtio-net devices multifunction and to fill
>>>> out all 8 functions for each slot.
>>>
>>> This is unlikely to work right wrt pci hotplug.
>>
>>
>> Yes. Our hotplug design is based on devices.. This is wrong, it
>> should be based on bus-level concepts (like PCI slots).
>>
>>> If we want to support a large number of interfaces, we need true
>>> multiport cards.
>>
>> This magic here creates a multiport virtio-net card so I'm not really
>> sure what you're suggesting. It would certainly be nice to make this
>> all more user friendly (and make hotplug work).
>>
>
> The big issue is to fix hotplug.
Yes, but this is entirely independent of multifunction devices.
Today we shoe-horn hot remove into device_del. Instead, we should have
explicit bus-level interfaces for hot remove.
Regards,
Anthony Liguori
> I don't see how we can make it user friendly, without making the
> ordinary case even more unfriendly. Looks like we need yet another
> level of indirection here.
>
^ permalink raw reply [flat|nested] 72+ messages in thread
* Re: [Qemu-devel] Hitting 29 NIC limit
2010-10-14 14:11 ` Anthony Liguori
@ 2010-10-14 14:53 ` Avi Kivity
-1 siblings, 0 replies; 72+ messages in thread
From: Avi Kivity @ 2010-10-14 14:53 UTC (permalink / raw)
To: Anthony Liguori; +Cc: Anjali Kulkarni, kvm, qemu-devel
On 10/14/2010 04:11 PM, Anthony Liguori wrote:
> On 10/14/2010 08:23 AM, Avi Kivity wrote:
>> On 10/14/2010 02:54 PM, Anthony Liguori wrote:
>>>>> The key is to make the virtio-net devices multifunction and to
>>>>> fill out all 8 functions for each slot.
>>>>
>>>> This is unlikely to work right wrt pci hotplug.
>>>
>>>
>>> Yes. Our hotplug design is based on devices.. This is wrong, it
>>> should be based on bus-level concepts (like PCI slots).
>>>
>>>> If we want to support a large number of interfaces, we need true
>>>> multiport cards.
>>>
>>> This magic here creates a multiport virtio-net card so I'm not
>>> really sure what you're suggesting. It would certainly be nice to
>>> make this all more user friendly (and make hotplug work).
>>>
>>
>> The big issue is to fix hotplug.
>
> Yes, but this is entirely independent of multifunction devices.
>
> Today we shoe-horn hot remove into device_del. Instead, we should
> have explicit bus-level interfaces for hot remove.
I'm not saying multiplug is not the right way to approach this (it is).
The only concern is to get hotplug right.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply [flat|nested] 72+ messages in thread
* Re: [Qemu-devel] Hitting 29 NIC limit
@ 2010-10-14 14:53 ` Avi Kivity
0 siblings, 0 replies; 72+ messages in thread
From: Avi Kivity @ 2010-10-14 14:53 UTC (permalink / raw)
To: Anthony Liguori; +Cc: Anjali Kulkarni, qemu-devel, kvm
On 10/14/2010 04:11 PM, Anthony Liguori wrote:
> On 10/14/2010 08:23 AM, Avi Kivity wrote:
>> On 10/14/2010 02:54 PM, Anthony Liguori wrote:
>>>>> The key is to make the virtio-net devices multifunction and to
>>>>> fill out all 8 functions for each slot.
>>>>
>>>> This is unlikely to work right wrt pci hotplug.
>>>
>>>
>>> Yes. Our hotplug design is based on devices.. This is wrong, it
>>> should be based on bus-level concepts (like PCI slots).
>>>
>>>> If we want to support a large number of interfaces, we need true
>>>> multiport cards.
>>>
>>> This magic here creates a multiport virtio-net card so I'm not
>>> really sure what you're suggesting. It would certainly be nice to
>>> make this all more user friendly (and make hotplug work).
>>>
>>
>> The big issue is to fix hotplug.
>
> Yes, but this is entirely independent of multifunction devices.
>
> Today we shoe-horn hot remove into device_del. Instead, we should
> have explicit bus-level interfaces for hot remove.
I'm not saying multiplug is not the right way to approach this (it is).
The only concern is to get hotplug right.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply [flat|nested] 72+ messages in thread
* Re: [Qemu-devel] Hitting 29 NIC limit (+Intel VT-c)
2010-10-14 12:07 ` Avi Kivity
` (3 preceding siblings ...)
(?)
@ 2010-10-28 20:52 ` linux_kvm
-1 siblings, 0 replies; 72+ messages in thread
From: linux_kvm @ 2010-10-28 20:52 UTC (permalink / raw)
To: Avi Kivity, anthony
On Thu, 14 Oct 2010 14:07 +0200, "Avi Kivity" <avi@redhat.com> wrote:
> On 10/14/2010 12:54 AM, Anthony Liguori wrote:
> > On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
> What's the motivation for such a huge number of interfaces?
Ultimately to bring multiple 10Gb bonds into a Vyatta guest.
---
> BTW, I don't think it's possible to hot-add physical functions. I
> believe I know of a card that supports dynamic add of physical functions
> (pre-dating SR-IOV)
I don't know what you're talking about, but it seems you have a better
handle than I on this VT-c stuff, so perhaps misguidedly I'll direct my
next question to you.
Is additional configuration required to make use of SR-IOV & VTq?
I don't immediateley understand how the queueing knows who is who in the
absense of eth.vlan- or if I need to for that matter.
My hope is that this is something like plug n play as long as kernel,
host & driver versions are >foo, but I haven't yet found documentation
to confirm it.
For the sake of future queries, I've come across these references so
far:
http://download.intel.com/design/network/applnots/321211.pdf
http://www.linux-kvm.org/wiki/images/6/6a/KvmForum2008%24kdf2008_7.pdf
http://www.mail-archive.com/kvm@vger.kernel.org/msg27860.html
http://www.mail-archive.com/kvm@vger.kernel.org/msg22721.html
http://thread.gmane.org/gmane.linux.kernel.mm/38508
http://ark.intel.com/Product.aspx?id=36918
^ permalink raw reply [flat|nested] 72+ messages in thread* Re: Hitting 29 NIC limit
2010-10-14 12:07 ` Avi Kivity
@ 2010-11-16 17:21 ` Michael S. Tsirkin
-1 siblings, 0 replies; 72+ messages in thread
From: Michael S. Tsirkin @ 2010-11-16 17:21 UTC (permalink / raw)
To: Avi Kivity; +Cc: Anthony Liguori, Anjali Kulkarni, kvm, qemu-devel
On Thu, Oct 14, 2010 at 02:07:17PM +0200, Avi Kivity wrote:
> On 10/14/2010 12:54 AM, Anthony Liguori wrote:
> >On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
> >>
> >>Hi,
> >>
> >>Using the legacy way of starting up NICs, I am hitting a
> >>limitation after 29
> >>NICs ie no more than 29 are detected (that's because of the 32 PCI slot
> >>limit on a single bus- 3 are already taken up)
> >>I had initially increased the MAX_NICS to 48, just on my tree, to get to
> >>more, but ofcource that wont work.
> >>Is there any way to go beyond 29 NICs the legacy way? What is
> >>the maximum
> >>that can be supported by the qdev mothod?
> >
> >I got up to 104 without trying very hard using the following script:
> >
> >args=""
> >for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
> >for fn in 0 1 2 3 4 5 6 7; do
> > args="$args -netdev user,id=eth${slot}_${fn}"
> > args="$args -device virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
> >done
> >done
> >
> >x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
> >-enable-kvm
> >
> >The key is to make the virtio-net devices multifunction and to
> >fill out all 8 functions for each slot.
>
> This is unlikely to work right wrt pci hotplug. If we want to
> support a large number of interfaces, we need true multiport cards.
Bridge support seems to be working mostly fine here.
That will let you go up to 256 devices without multifunction.
--
MST
^ permalink raw reply [flat|nested] 72+ messages in thread* [Qemu-devel] Re: Hitting 29 NIC limit
@ 2010-11-16 17:21 ` Michael S. Tsirkin
0 siblings, 0 replies; 72+ messages in thread
From: Michael S. Tsirkin @ 2010-11-16 17:21 UTC (permalink / raw)
To: Avi Kivity; +Cc: Anjali Kulkarni, kvm, qemu-devel
On Thu, Oct 14, 2010 at 02:07:17PM +0200, Avi Kivity wrote:
> On 10/14/2010 12:54 AM, Anthony Liguori wrote:
> >On 10/13/2010 05:32 PM, Anjali Kulkarni wrote:
> >>
> >>Hi,
> >>
> >>Using the legacy way of starting up NICs, I am hitting a
> >>limitation after 29
> >>NICs ie no more than 29 are detected (that's because of the 32 PCI slot
> >>limit on a single bus- 3 are already taken up)
> >>I had initially increased the MAX_NICS to 48, just on my tree, to get to
> >>more, but ofcource that wont work.
> >>Is there any way to go beyond 29 NICs the legacy way? What is
> >>the maximum
> >>that can be supported by the qdev mothod?
> >
> >I got up to 104 without trying very hard using the following script:
> >
> >args=""
> >for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do
> >for fn in 0 1 2 3 4 5 6 7; do
> > args="$args -netdev user,id=eth${slot}_${fn}"
> > args="$args -device virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile="
> >done
> >done
> >
> >x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args}
> >-enable-kvm
> >
> >The key is to make the virtio-net devices multifunction and to
> >fill out all 8 functions for each slot.
>
> This is unlikely to work right wrt pci hotplug. If we want to
> support a large number of interfaces, we need true multiport cards.
Bridge support seems to be working mostly fine here.
That will let you go up to 256 devices without multifunction.
--
MST
^ permalink raw reply [flat|nested] 72+ messages in thread[parent not found: <C9081994.3517C%anjali@juniper.net>]
