On Thu, Sep 19, 2013 at 8:03 PM, Somnath R= oy <Somnath.Roy-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org> wrote:

Thanks Josh !
I am able to successfully add this noshare option in the image mapping now.= Looking at dmesg output, I found that was indeed the secret key problem. B= lock performance is scaling now.

Regards
Somnath

-----Original Message-----
From: ceph-devel-owner@= vger.kernel.org [mailto:ceph-devel-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Josh Durgin
Sent: Thursday, September 19, 2013 12:24 PM
To: Somnath Roy
Cc: Sage Weil; ceph-devel@vge= r.kernel.org; Anirban Ray; ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
Subject: Re: [ceph-users] Scaling RBD module

On 09/19/2013 12:04 PM, Somnath Roy wrote:
> Hi Josh,
> Thanks for the information. I am trying to add the following but hitti= ng some permission issue.
>
> root@emsclient:/etc# echo <mon-1>:6789,<mon-2>:6789,<mo= n-3>:6789
> name=3Dadmin,key=3Dclient.admin,noshare test_rbd ceph_block_test' = >
> /sys/bus/rbd/add
> -bash: echo: write error: Operation not permitted

If you check dmesg, it will probably show an error trying to authenticate t= o the cluster.

Instead of key=3Dclient.admin, you can pass the base64 secret value as show= n in 'ceph auth list' with the secret=3DXXXXXXXXXXXXXXXXXXXXX optio= n.

BTW, there's a ticket for adding the noshare option to rbd map so using= the sysfs interface like this is never necessary:

http://tr= acker.ceph.com/issues/6264

Josh

> Here is the contents of rbd directory..
>
> root@emsclient:/sys/bus/rbd# ll
> total 0
> drwxr-xr-x =A04 root root =A0 =A00 Sep 19 11:59 ./
> drwxr-xr-x 30 root root =A0 =A00 Sep 13 11:41 ../
> --w------- =A01 root root 4096 Sep 19 11:59 add
> drwxr-xr-x =A02 root root =A0 =A00 Sep 19 12:03 devices/
> drwxr-xr-x =A02 root root =A0 =A00 Sep 19 12:03 drivers/
> -rw-r--r-- =A01 root root 4096 Sep 19 12:03 drivers_autoprobe
> --w------- =A01 root root 4096 Sep 19 12:03 drivers_probe
> --w------- =A01 root root 4096 Sep 19 12:03 remove
> --w------- =A01 root root 4096 Sep 19 11:59 uevent
>
>
> I checked even if I am logged in as root , I can't write anything = on /sys.
>
> Here is the Ubuntu version I am using..
>
> root@emsclient:/etc# lsb_release -a
> No LSB modules are available.
> Distributor ID: Ubuntu
> Description: =A0 =A0Ubuntu 13.04
> Release: =A0 =A0 =A0 =A013.04
> Codename: =A0 =A0 =A0 raring
>
> Here is the mount information....
>
> root@emsclient:/etc# mount
> /dev/mapper/emsclient--vg-root on / type ext4 (rw,errors=3Dremount-ro)=
> proc on /proc type proc (rw,noexec,nosuid,nodev) sysfs on /sys type > sysfs (rw,noexec,nosuid,nodev) none on /sys/fs/cgroup type tmpfs (rw)<= br> > none on /sys/fs/fuse/connections type fusectl (rw) none on
> /sys/kernel/debug type debugfs (rw) none on /sys/kernel/security type<= br> > securityfs (rw) udev on /dev type devtmpfs (rw,mode=3D0755) devpts on<= br> > /dev/pts type devpts (rw,noexec,nosuid,gid=3D5,mode=3D0620)
> tmpfs on /run type tmpfs (rw,noexec,nosuid,size=3D10%,mode=3D0755)
> none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=3D5242880) > none on /run/shm type tmpfs (rw,nosuid,nodev) none on /run/user type > tmpfs (rw,noexec,nosuid,nodev,size=3D104857600,mode=3D0755)
> /dev/sda1 on /boot type ext2 (rw)
> /dev/mapper/emsclient--vg-home on /home type ext4 (rw)
>
>
> Any idea what went wrong here ?
>
> Thanks & Regards
> Somnath
>
> -----Original Message-----
> From: Josh Durgin [mailto:j= osh.durgin-4GqslpFJ+cxBDgjK7y7TUQ@public.gmane.org]
> Sent: Wednesday, September 18, 2013 6:10 PM
> To: Somnath Roy
> Cc: Sage Weil; ceph-deve= l@vger.kernel.org; Anirban Ray;
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org=
> Subject: Re: [ceph-users] Scaling RBD module
>
> On 09/17/2013 03:30 PM, Somnath Roy wrote:
>> Hi,
>> I am running Ceph on a 3 node cluster and each of my server node i= s running 10 OSDs, one for each disk. I have one admin node and all the nod= es are connected with 2 X 10G network. One network is for cluster and other= one configured as public network.
>>
>> Here is the status of my cluster.
>>
>> ~/fio_test# ceph -s
>>
>> =A0 =A0 cluster b2e0b4db-6342-490e-9c28-0aadf0188023
>> =A0 =A0 =A0health HEALTH_WARN clock skew detected on mon. <serv= er-name-2>, mon. <server-name-3>
>> =A0 =A0 =A0monmap e1: 3 mons at {<server-name-1>=3Dxxx.xxx.x= xx.xxx:6789/0, <server-name-2>=3Dxxx.xxx.xxx.xxx:6789/0, <server-n= ame-3>=3Dxxx.xxx.xxx.xxx:6789/0}, election epoch 64, quorum 0,1,2 <se= rver-name-1>,<server-name-2>,<server-name-3>
>> =A0 =A0 =A0osdmap e391: 30 osds: 30 up, 30 in
>> =A0 =A0 =A0 pgmap v5202: 30912 pgs: 30912 active+clean; 8494 MB da= ta, 27912 MB used, 11145 GB / 11172 GB avail
>> =A0 =A0 =A0mdsmap e1: 0/0/1 up
>>
>>
>> I started with rados bench command to benchmark the read performan= ce of this Cluster on a large pool (~10K PGs) and found that each rados cli= ent has a limitation. Each client can only drive up to a certain mark. Each= server =A0node cpu utilization shows it is =A0around 85-90% idle and the a= dmin node (from where rados client is running) is around ~80-85% idle. I am= trying with 4K object size.
>
> Note that rados bench with 4k objects is different from rbd with 4k-si= zed I/Os - rados bench sends each request to a new object, while rbd object= s are 4M by default.
>
>> Now, I started running more clients on the admin node and the perf= ormance is scaling till it hits the client cpu limit. Server still has the = cpu of 30-35% idle. With small object size I must say that the ceph per osd= cpu utilization is not promising!
>>
>> After this, I started testing the rados block interface with kerne= l rbd module from my admin node.
>> I have created 8 images mapped on the pool having around 10K PGs a= nd I am not able to scale up the performance by running fio (either by crea= ting a software raid or running on individual /dev/rbd* instances). For exa= mple, running multiple fio instances (one in /dev/rbd1 and the other in /de= v/rbd2) =A0the performance I am getting is half of what I am getting if run= ning one instance. Here is my fio job script.
>>
>> [random-reads]
>> ioengine=3Dlibaio
>> iodepth=3D32
>> filename=3D/dev/rbd1
>> rw=3Drandread
>> bs=3D4k
>> direct=3D1
>> size=3D2G
>> numjobs=3D64
>>
>> Let me know if I am following the proper procedure or not.
>>
>> But, If my understanding is correct, kernel rbd module is acting a= s a client to the cluster and in one admin node I can run only one of such = kernel instance.
>> If so, I am then limited to the client bottleneck that I stated ea= rlier. The cpu utilization of the server side is around 85-90% idle, so, it= is clear that client is not driving.
>>
>> My question is, is there any way to hit the cluster =A0with more c= lient from a single box while testing the rbd module ?
>
> You can run multiple librbd instances easily (for example with multipl= e runs of the rbd bench-write command).
>
> The kernel rbd driver uses the same rados client instance for multiple= block devices by default. There's an option (noshare) to use a new rad= os client instance for a newly mapped device, but it's not exposed by t= he rbd cli. You need to use the sysfs interface that 'rbd map' uses= instead.
>
> Once you've used rbd map once on a machine, the kernel will alread= y have the auth key stored, and you can use:
>
> echo '1.2.3.4:67= 89 name=3Dadmin,key=3Dclient.admin,noshare poolname
> imagename' > /sys/bus/rbd/add
>
> Where 1.2.3.4:6789 is the address of a monitor, and you're connecting as client.admin.<= br> >
> You can use 'rbd unmap' as usual.
>
> Josh
>
>
> ________________________________
>
> PLEASE NOTE: The information contained in this electronic mail message= is intended only for the use of the designated recipient(s) named above. I= f the reader of this message is not the intended recipient, you are hereby = notified that you have received this message in error and that any review, = dissemination, distribution, or copying of this message is strictly prohibi= ted. If you have received this communication in error, please notify the se= nder by telephone or e-mail (as shown above) immediately and destroy any an= d all copies of this message in your possession (whether hard copies or ele= ctronically stored copies).
>
>

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel&q= uot; in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at =A0http://vger.kernel= .org/majordomo-info.html

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org<= br> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com