Re: blktap2 device creation failing after 162 devices w/Xen4.0 + linux-2.6.31.13

[Daniel Stodden <daniel.stodden@citrix.com>]

Hi.

Please echo $((N * (32 * 11 + 50) + SOME_HEADROOM))
to /proc/sys/fs/aio-max-nr. Or set it up in sysctl.conf.

Where N is the number of devices you desire.

As for the apparently missing big fat complaint you should have seen pop
in syslog, I'll keep it in mind  for the next update. :}

Cheers,
Daniel

On Tue, 2010-04-13 at 20:20 -0400, John McCullough wrote:
> I have been working with a colleague to get a large number of small VMs 
> running on a single system.  We were hoping for at least 100, but we 
> seem to be topping out around 81.  Each VM has a disk image and a swap 
> image.  It seemed like we were hitting a blktap limit, so we tried 
> bumping up the MAX macros in tool/blktap2 and the linux driver, with no 
> change.  (Though we haven't hit the theoretical 256 blktap devices yet).
> 
> (Initially we were only able to get 64 VMs until we bumped 
> CONFIG_NR_CPUS from 8 to 64 to increase the number of dynirqs).
> 
> To isolate the problem, I tried creating a large number of blktap 
> devices in the dom0 with no guests running and I ran into the same 
> ceiling (162 total devices).   Commands to reproduce the problem follow:
> 
> echo 9 > /sys/class/blktap2/verbosity
> 
> for x in `seq 0 163`; do
>          if ( ! dd if=/dev/zero of=/scratch/test-$x.img bs=1 count=1 
> seek=1M 2> /dev/null); then
>                  echo "Qemu fail on $x"; exit 1
>          fi
>          if ( ! tapdisk2 -n aio:/scratch/test-$x.img) ; then
>                  echo "blktap fail on $x"; exit 1
>          fi
> done
> 
> The result:
> ...
> /dev/xen/blktap-2/tapdev159
> /dev/xen/blktap-2/tapdev160
> /dev/xen/blktap-2/tapdev161
> /dev/xen/blktap-2/tapdev162
> unrecognized child response
> blktap fail on 163
> 
> Dmesg output associated with 163:
> [ 1288.839978] blktap_sysfs_create: adding attributes for dev 
> ffff88019e4d1e00
> [ 1288.840947] blktap_sysfs_destroy
> 
> (Output for the prior devices includes processing a request, and a 
> blktap_device_finish_request)
> 
> No related xm dmesg output.
> 
> $ hg tip
> changeset:   21091:f28f1ee587c8
> tag:         tip
> user:        Keir Fraser <keir.fraser@citrix.com>
> date:        Wed Apr 07 12:38:28 2010 +0100
> summary:     Added signature for changeset 484179b2be5d
> 
> $ uname -a
> Linux sysnet121 2.6.32-3-amd64 #1 SMP Wed Feb 24 18:07:42 UTC 2010 
> x86_64 GNU/Linux
> 
> Has anyone had contrary experience? Does anyone know where the 162 max 
> is coming from?
> 
> Thanks,
> John
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel