From mboxrd@z Thu Jan 1 00:00:00 1970 From: Daniel Stodden Subject: Re: blktap2 device creation failing after 162 devices w/Xen4.0 + linux-2.6.31.13 Date: Wed, 14 Apr 2010 01:24:06 -0700 Message-ID: <1271233446.2449.59.camel@ramone.somacoma.net> References: <4BC50A5A.1020501@cs.ucsd.edu> <1271216280.25413.1782.camel@agari.van.xensource.com> <4BC55531.7080005@cs.ucsd.edu> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <4BC55531.7080005@cs.ucsd.edu> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: John McCullough Cc: Ian Pratt , "xen-devel@lists.xensource.com" List-Id: xen-devel@lists.xenproject.org On Wed, 2010-04-14 at 01:40 -0400, John McCullough wrote: > Daniel, > > That did the trick and got us up to 256, Thanks! > > Out of curiosity, what's standing in the way of more devices? I must admit I never tried. Lack of maybe a couple sparse tables here and there? > We tried raising the MAX_*_DEVICES constants in these files to 512, but > didn't have any luck: > linux-2.6-pvops.git/drivers/xen/blktap/blktap.h > tools/blktap2/include/blktaplib.h > tools/blktap/lib/blktaplib.h > > (The error is now "vbd open failed: -6") That would be an ENXIO, probably while trying to open the ring (can you verify that with an strace -f?) Should indeed work for up to 2^20, or MAX_BLKTAP_DEVICEs. I don't think this fails in the ring code, we return ENODEV there (which is a bug) and the kernel won't correct that code. Daniel > I noticed an artificial limit of 26*26 in the tapdev naming scheme, but > I didn't look very thoroughly. > > Thanks again, > John > > > Daniel Stodden wrote: > > Hi. > > > > Please echo $((N * (32 * 11 + 50) + SOME_HEADROOM)) > > to /proc/sys/fs/aio-max-nr. Or set it up in sysctl.conf. > > > > Where N is the number of devices you desire. > > > > As for the apparently missing big fat complaint you should have seen pop > > in syslog, I'll keep it in mind for the next update. :} > > > > Cheers, > > Daniel > > > > On Tue, 2010-04-13 at 20:20 -0400, John McCullough wrote: > > > >> I have been working with a colleague to get a large number of small VMs > >> running on a single system. We were hoping for at least 100, but we > >> seem to be topping out around 81. Each VM has a disk image and a swap > >> image. It seemed like we were hitting a blktap limit, so we tried > >> bumping up the MAX macros in tool/blktap2 and the linux driver, with no > >> change. (Though we haven't hit the theoretical 256 blktap devices yet). > >> > >> (Initially we were only able to get 64 VMs until we bumped > >> CONFIG_NR_CPUS from 8 to 64 to increase the number of dynirqs). > >> > >> To isolate the problem, I tried creating a large number of blktap > >> devices in the dom0 with no guests running and I ran into the same > >> ceiling (162 total devices). Commands to reproduce the problem follow: > >> > >> echo 9 > /sys/class/blktap2/verbosity > >> > >> for x in `seq 0 163`; do > >> if ( ! dd if=/dev/zero of=/scratch/test-$x.img bs=1 count=1 > >> seek=1M 2> /dev/null); then > >> echo "Qemu fail on $x"; exit 1 > >> fi > >> if ( ! tapdisk2 -n aio:/scratch/test-$x.img) ; then > >> echo "blktap fail on $x"; exit 1 > >> fi > >> done > >> > >> The result: > >> ... > >> /dev/xen/blktap-2/tapdev159 > >> /dev/xen/blktap-2/tapdev160 > >> /dev/xen/blktap-2/tapdev161 > >> /dev/xen/blktap-2/tapdev162 > >> unrecognized child response > >> blktap fail on 163 > >> > >> Dmesg output associated with 163: > >> [ 1288.839978] blktap_sysfs_create: adding attributes for dev > >> ffff88019e4d1e00 > >> [ 1288.840947] blktap_sysfs_destroy > >> > >> (Output for the prior devices includes processing a request, and a > >> blktap_device_finish_request) > >> > >> No related xm dmesg output. > >> > >> $ hg tip > >> changeset: 21091:f28f1ee587c8 > >> tag: tip > >> user: Keir Fraser > >> date: Wed Apr 07 12:38:28 2010 +0100 > >> summary: Added signature for changeset 484179b2be5d > >> > >> $ uname -a > >> Linux sysnet121 2.6.32-3-amd64 #1 SMP Wed Feb 24 18:07:42 UTC 2010 > >> x86_64 GNU/Linux > >> > >> Has anyone had contrary experience? Does anyone know where the 162 max > >> is coming from? > >> > >> Thanks, > >> John > >> > >> _______________________________________________ > >> Xen-devel mailing list > >> Xen-devel@lists.xensource.com > >> http://lists.xensource.com/xen-devel > >> > > > > > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel