blktap2 device creation failing after 162 devices w/Xen4.0 + linux-2.6.31.13

[John McCullough <jmccullo@cs.ucsd.edu>]

I have been working with a colleague to get a large number of small VMs 
running on a single system.  We were hoping for at least 100, but we 
seem to be topping out around 81.  Each VM has a disk image and a swap 
image.  It seemed like we were hitting a blktap limit, so we tried 
bumping up the MAX macros in tool/blktap2 and the linux driver, with no 
change.  (Though we haven't hit the theoretical 256 blktap devices yet).

(Initially we were only able to get 64 VMs until we bumped 
CONFIG_NR_CPUS from 8 to 64 to increase the number of dynirqs).

To isolate the problem, I tried creating a large number of blktap 
devices in the dom0 with no guests running and I ran into the same 
ceiling (162 total devices).   Commands to reproduce the problem follow:

echo 9 > /sys/class/blktap2/verbosity

for x in `seq 0 163`; do
         if ( ! dd if=/dev/zero of=/scratch/test-$x.img bs=1 count=1 
seek=1M 2> /dev/null); then
                 echo "Qemu fail on $x"; exit 1
         fi
         if ( ! tapdisk2 -n aio:/scratch/test-$x.img) ; then
                 echo "blktap fail on $x"; exit 1
         fi
done

The result:
...
/dev/xen/blktap-2/tapdev159
/dev/xen/blktap-2/tapdev160
/dev/xen/blktap-2/tapdev161
/dev/xen/blktap-2/tapdev162
unrecognized child response
blktap fail on 163

Dmesg output associated with 163:
[ 1288.839978] blktap_sysfs_create: adding attributes for dev 
ffff88019e4d1e00
[ 1288.840947] blktap_sysfs_destroy

(Output for the prior devices includes processing a request, and a 
blktap_device_finish_request)

No related xm dmesg output.

$ hg tip
changeset:   21091:f28f1ee587c8
tag:         tip
user:        Keir Fraser <keir.fraser@citrix.com>
date:        Wed Apr 07 12:38:28 2010 +0100
summary:     Added signature for changeset 484179b2be5d

$ uname -a
Linux sysnet121 2.6.32-3-amd64 #1 SMP Wed Feb 24 18:07:42 UTC 2010 
x86_64 GNU/Linux

Has anyone had contrary experience? Does anyone know where the 162 max 
is coming from?

Thanks,
John