Re: blktap2 device creation failing after 162 devices w/Xen4.0 + linux-2.6.31.13

[John McCullough <jmccullo@cs.ucsd.edu>]

On 04/14/2010 01:24 AM, Daniel Stodden wrote:
> On Wed, 2010-04-14 at 01:40 -0400, John McCullough wrote:
>    
>> Daniel,
>>
>> That did the trick and got us up to 256, Thanks!
>>
>> Out of curiosity, what's standing in the way of more devices?
>>      
> I must admit I never tried. Lack of maybe a couple sparse tables here
> and there?
>
>    
>> We tried raising the MAX_*_DEVICES constants in these files to 512, but
>> didn't have any luck:
>> linux-2.6-pvops.git/drivers/xen/blktap/blktap.h
>> tools/blktap2/include/blktaplib.h
>> tools/blktap/lib/blktaplib.h
>>
>> (The error is now "vbd open failed: -6")
>>      
> That would be an ENXIO, probably while trying to open the ring (can you
> verify that with an strace -f?)
>    

Looks like it is ENXIO:

...
[pid  4154] open("/dev/xen/blktap-2/blktap256", O_RDWR <unfinished ...>

[pid  4153] close(4)                    = 0
[pid  4153] fcntl(3, F_GETFL)           = 0 (flags O_RDONLY)
[pid  4153] brk(0)                      = 0xa1e000
[pid  4153] brk(0xa3f000)               = 0xa3f000
[pid  4153] fstat(3, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
[pid  4153] mmap(NULL, 4096, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6844b40000
[pid  4153] lseek(3, 0, SEEK_CUR)       = -1 ESPIPE (Illegal seek)
[pid  4153] read(3, <unfinished ...>

[pid  4154] <... open resumed> )        = -1 ENXIO (No such device or 
address)
[pid  4154] gettimeofday({1271262219, 398530}, NULL) = 0
[pid  4154] sendto(0, "<27>Apr 14 09:23:39 tapdisk2[415"..., 115, 
MSG_NOSIGNAL, NULL, 0) = 115
[pid  4154] close(3)                    = 0
[pid  4154] gettimeofday({1271262219, 398836}, NULL) = 0
[pid  4154] sendto(0, "<30>Apr 14 09:23:39 tapdisk2[415"..., 109, 
MSG_NOSIGNAL, NULL, 0) = 109
[pid  4154] gettimeofday({1271262219, 399161}, NULL) = 0
[pid  4154] sendto(0, "<27>Apr 14 09:23:39 tapdisk2[415"..., 86, 
MSG_NOSIGNAL, NULL, 0) = 86
[pid  4154] write(1, "-6: vbd open failed: -6\n", 24) = 24
... (err, ioctl to control, close and exit)


Thanks,
John

> Should indeed work for up to 2^20, or MAX_BLKTAP_DEVICEs.
>
> I don't think this fails in the ring code, we return ENODEV there (which
> is a bug) and the kernel won't correct that code.
>
> Daniel
>
>    
>> I noticed an artificial limit of 26*26 in the tapdev naming scheme, but
>> I didn't look very thoroughly.
>>
>> Thanks again,
>> John
>>
>>
>> Daniel Stodden wrote:
>>      
>>> Hi.
>>>
>>> Please echo $((N * (32 * 11 + 50) + SOME_HEADROOM))
>>> to /proc/sys/fs/aio-max-nr. Or set it up in sysctl.conf.
>>>
>>> Where N is the number of devices you desire.
>>>
>>> As for the apparently missing big fat complaint you should have seen pop
>>> in syslog, I'll keep it in mind  for the next update. :}
>>>
>>> Cheers,
>>> Daniel
>>>
>>> On Tue, 2010-04-13 at 20:20 -0400, John McCullough wrote:
>>>
>>>        
>>>> I have been working with a colleague to get a large number of small VMs
>>>> running on a single system.  We were hoping for at least 100, but we
>>>> seem to be topping out around 81.  Each VM has a disk image and a swap
>>>> image.  It seemed like we were hitting a blktap limit, so we tried
>>>> bumping up the MAX macros in tool/blktap2 and the linux driver, with no
>>>> change.  (Though we haven't hit the theoretical 256 blktap devices yet).
>>>>
>>>> (Initially we were only able to get 64 VMs until we bumped
>>>> CONFIG_NR_CPUS from 8 to 64 to increase the number of dynirqs).
>>>>
>>>> To isolate the problem, I tried creating a large number of blktap
>>>> devices in the dom0 with no guests running and I ran into the same
>>>> ceiling (162 total devices).   Commands to reproduce the problem follow:
>>>>
>>>> echo 9>  /sys/class/blktap2/verbosity
>>>>
>>>> for x in `seq 0 163`; do
>>>>           if ( ! dd if=/dev/zero of=/scratch/test-$x.img bs=1 count=1
>>>> seek=1M 2>  /dev/null); then
>>>>                   echo "Qemu fail on $x"; exit 1
>>>>           fi
>>>>           if ( ! tapdisk2 -n aio:/scratch/test-$x.img) ; then
>>>>                   echo "blktap fail on $x"; exit 1
>>>>           fi
>>>> done
>>>>
>>>> The result:
>>>> ...
>>>> /dev/xen/blktap-2/tapdev159
>>>> /dev/xen/blktap-2/tapdev160
>>>> /dev/xen/blktap-2/tapdev161
>>>> /dev/xen/blktap-2/tapdev162
>>>> unrecognized child response
>>>> blktap fail on 163
>>>>
>>>> Dmesg output associated with 163:
>>>> [ 1288.839978] blktap_sysfs_create: adding attributes for dev
>>>> ffff88019e4d1e00
>>>> [ 1288.840947] blktap_sysfs_destroy
>>>>
>>>> (Output for the prior devices includes processing a request, and a
>>>> blktap_device_finish_request)
>>>>
>>>> No related xm dmesg output.
>>>>
>>>> $ hg tip
>>>> changeset:   21091:f28f1ee587c8
>>>> tag:         tip
>>>> user:        Keir Fraser<keir.fraser@citrix.com>
>>>> date:        Wed Apr 07 12:38:28 2010 +0100
>>>> summary:     Added signature for changeset 484179b2be5d
>>>>
>>>> $ uname -a
>>>> Linux sysnet121 2.6.32-3-amd64 #1 SMP Wed Feb 24 18:07:42 UTC 2010
>>>> x86_64 GNU/Linux
>>>>
>>>> Has anyone had contrary experience? Does anyone know where the 162 max
>>>> is coming from?
>>>>
>>>> Thanks,
>>>> John
>>>>
>>>> _______________________________________________
>>>> Xen-devel mailing list
>>>> Xen-devel@lists.xensource.com
>>>> http://lists.xensource.com/xen-devel
>>>>
>>>>          
>>>
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>>
>>>        
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>      
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>