From mboxrd@z Thu Jan 1 00:00:00 1970 From: Markus Armbruster Subject: Re: Hitting 29 NIC limit Date: Fri, 15 Oct 2010 11:11:24 +0200 Message-ID: References: <4CB6388A.30006@codemonkey.ws> <4CB6F275.2060204@redhat.com> <4CB6FE94.4030203@codemonkey.ws> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Anjali Kulkarni , Avi Kivity , "kvm@vger.kernel.org" , "qemu-devel@nongnu.org" To: Anthony Liguori Return-path: In-Reply-To: <4CB6FE94.4030203@codemonkey.ws> (Anthony Liguori's message of "Thu, 14 Oct 2010 07:59:00 -0500") List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: qemu-devel-bounces+gceq-qemu-devel=gmane.org@nongnu.org Errors-To: qemu-devel-bounces+gceq-qemu-devel=gmane.org@nongnu.org List-Id: kvm.vger.kernel.org Anthony Liguori writes: > On 10/14/2010 07:36 AM, Markus Armbruster wrote: >> Avi Kivity writes: >> >> >>> On 10/14/2010 12:54 AM, Anthony Liguori wrote: >>> >>>> On 10/13/2010 05:32 PM, Anjali Kulkarni wrote: >>>> >>>>> Hi, >>>>> >>>>> Using the legacy way of starting up NICs, I am hitting a limitation >>>>> after 29 >>>>> NICs ie no more than 29 are detected (that's because of the 32 PCI slot >>>>> limit on a single bus- 3 are already taken up) >>>>> I had initially increased the MAX_NICS to 48, just on my tree, to get to >>>>> more, but ofcource that wont work. >>>>> Is there any way to go beyond 29 NICs the legacy way? What is the >>>>> maximum >>>>> that can be supported by the qdev mothod? >>>>> >>>> I got up to 104 without trying very hard using the following script: >>>> >>>> args="" >>>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do >>>> for fn in 0 1 2 3 4 5 6 7; do >>>> args="$args -netdev user,id=eth${slot}_${fn}" >>>> args="$args -device >>>> virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile=" >>>> done >>>> done >>>> >>>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args} >>>> -enable-kvm >>>> >>>> The key is to make the virtio-net devices multifunction and to fill >>>> out all 8 functions for each slot. >>>> >> I'm amazed that works. Can't see how creating another qdev in the same >> slot makes a proper multifunction device. >> > > multifunction=on sets the multifunction bit for the PCI device. Then > it's a matter of setting the address to be a specific function. I missed that bit, or perhaps forgot it already. Neat! Unfortunately, it's not yet good enough for hot plug. We need to plug the complete device in one go, not function by function. This is harder than hot unplug, where we can just add a command "unplug this PCI device", then device_del its functions at our leisure. The symmetric way to plug would be device_add the functions, then plug the device. Backward compatibility headache: current device_add/del plug/unplug automatically. > Our default platform devices are actually multifunction. Yes, but they're built "by hand". [...] From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=40353 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1P6gKx-0007Pb-TA for qemu-devel@nongnu.org; Fri, 15 Oct 2010 05:12:28 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1P6gKw-0004RL-Te for qemu-devel@nongnu.org; Fri, 15 Oct 2010 05:12:27 -0400 Received: from mx1.redhat.com ([209.132.183.28]:55796) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1P6gKw-0004QQ-NH for qemu-devel@nongnu.org; Fri, 15 Oct 2010 05:12:26 -0400 From: Markus Armbruster Subject: Re: [Qemu-devel] Hitting 29 NIC limit References: <4CB6388A.30006@codemonkey.ws> <4CB6F275.2060204@redhat.com> <4CB6FE94.4030203@codemonkey.ws> Date: Fri, 15 Oct 2010 11:11:24 +0200 In-Reply-To: <4CB6FE94.4030203@codemonkey.ws> (Anthony Liguori's message of "Thu, 14 Oct 2010 07:59:00 -0500") Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Anthony Liguori Cc: Anjali Kulkarni , Avi Kivity , "kvm@vger.kernel.org" , "qemu-devel@nongnu.org" Anthony Liguori writes: > On 10/14/2010 07:36 AM, Markus Armbruster wrote: >> Avi Kivity writes: >> >> >>> On 10/14/2010 12:54 AM, Anthony Liguori wrote: >>> >>>> On 10/13/2010 05:32 PM, Anjali Kulkarni wrote: >>>> >>>>> Hi, >>>>> >>>>> Using the legacy way of starting up NICs, I am hitting a limitation >>>>> after 29 >>>>> NICs ie no more than 29 are detected (that's because of the 32 PCI slot >>>>> limit on a single bus- 3 are already taken up) >>>>> I had initially increased the MAX_NICS to 48, just on my tree, to get to >>>>> more, but ofcource that wont work. >>>>> Is there any way to go beyond 29 NICs the legacy way? What is the >>>>> maximum >>>>> that can be supported by the qdev mothod? >>>>> >>>> I got up to 104 without trying very hard using the following script: >>>> >>>> args="" >>>> for slot in 5 6 7 8 9 10 11 12 13 14 15 16 17; do >>>> for fn in 0 1 2 3 4 5 6 7; do >>>> args="$args -netdev user,id=eth${slot}_${fn}" >>>> args="$args -device >>>> virtio-net-pci,addr=${slot}.${fn},netdev=eth${slot}_${fn},multifunction=on,romfile=" >>>> done >>>> done >>>> >>>> x86_64-softmmu/qemu-system-x86_64 -hda ~/images/linux.img ${args} >>>> -enable-kvm >>>> >>>> The key is to make the virtio-net devices multifunction and to fill >>>> out all 8 functions for each slot. >>>> >> I'm amazed that works. Can't see how creating another qdev in the same >> slot makes a proper multifunction device. >> > > multifunction=on sets the multifunction bit for the PCI device. Then > it's a matter of setting the address to be a specific function. I missed that bit, or perhaps forgot it already. Neat! Unfortunately, it's not yet good enough for hot plug. We need to plug the complete device in one go, not function by function. This is harder than hot unplug, where we can just add a command "unplug this PCI device", then device_del its functions at our leisure. The symmetric way to plug would be device_add the functions, then plug the device. Backward compatibility headache: current device_add/del plug/unplug automatically. > Our default platform devices are actually multifunction. Yes, but they're built "by hand". [...]