[performance] why rbd_aio_write latency increase from 4ms to 7.3ms after the same test

[performance] why rbd_aio_write latency increase from 4ms to 7.3ms after the same test

[hzwulibin]

Hi, 
same environment, after a test script, the io latency(get from sudo ceph --admin-daemon /run/ceph/guests/ceph-client.*.asok per dump) 
increase from about 4ms to 7.3ms

qemu version: debian 2.1.2
kernel:3.10.45-openstack-amd64
system: debian 7.8
ceph: 0.94.5
VM CPU number: 4  (cpu MHz : 2599.998)
VM memory size: 16GB
9 OSD storage servers, with 4 SSD OSD on each, total 36 OSDs.

Test scripts in VM:
# cat reproduce.sh
#!/bin/bash

times=20
for((i=1;i<=$times;i++))
do
    tmpdate=`date "+%F-%T"`
    echo "=======================$tmpdate($i/$times)======================="
    tmp=$((i%2))
    if [[ $tmp -eq 0 ]];then
        echo "############### fio /root/vdb.cfg ###############"
        fio /root/vdb.cfg
    else
        echo "############### fio /root/vdc.cfg ###############"
        fio /root/vdc.cfg
    fi
done


tmpdate=`date "+%F-%T"`
echo "############### [$tmpdate] fio /root/vde.cfg ###############"
fio /root/vde.cfg


# cat vdb.cfg 
[global]
rw=randwrite
direct=1
numjobs=64
ioengine=sync
bsrange=4k-4k
runtime=180
group_reporting

[disk01]
filename=/dev/vdb


# cat vdc.cfg 
[global]
rw=randwrite
direct=1
numjobs=64
ioengine=sync
bsrange=4k-4k
runtime=180
group_reporting

[disk01]
filename=/dev/vdc

# cat vdd.cfg 
[global]
rw=randwrite
direct=1
numjobs=64
ioengine=sync
bsrange=4k-4k
runtime=180
group_reporting

[disk01]
filename=/dev/vdd

# cat vde.cfg 
[global]
rw=randwrite
direct=1
numjobs=64
ioengine=sync
bsrange=4k-4k
runtime=180
group_reporting

[disk01]
filename=/dev/vde 		

After run the scripts reproduce.sh, the disks in the VM's IOPS cutdown from 12k to 5k, the latency increase from 4ms to 7.3ms.

run steps:
1. create a VM
2. create four volumes and attatch to the VM
3. sh reproduce.sh
4. in the runtime of  reproduce.sh, run "fio vdd.cfg" or "fio vde.cfg" to checkt the performance
	
After reproduce.sh finished, performance down.


Anyone has the same problem or has some ideas about this?

Thanks!
--------------
hzwulibin
2015-11-02

RE: [performance] why rbd_aio_write latency increase from 4ms to 7.3ms after the same test

[Chen, Xiaoxi]

Pre-allocated the volume by "DD" across the entire RBD before you do any performance test:).

In this case, you may want to re-create the RBD, pre-allocate and try again.

> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
> owner@vger.kernel.org] On Behalf Of hzwulibin
> Sent: Monday, November 2, 2015 1:24 PM
> To: ceph-devel; ceph-users
> Subject: [performance] why rbd_aio_write latency increase from 4ms to
> 7.3ms after the same test
> 
> Hi,
> same environment, after a test script, the io latency(get from sudo ceph --
> admin-daemon /run/ceph/guests/ceph-client.*.asok per dump) increase
> from about 4ms to 7.3ms
> 
> qemu version: debian 2.1.2
> kernel:3.10.45-openstack-amd64
> system: debian 7.8
> ceph: 0.94.5
> VM CPU number: 4  (cpu MHz : 2599.998)
> VM memory size: 16GB
> 9 OSD storage servers, with 4 SSD OSD on each, total 36 OSDs.
> 
> Test scripts in VM:
> # cat reproduce.sh
> #!/bin/bash
> 
> times=20
> for((i=1;i<=$times;i++))
> do
>     tmpdate=`date "+%F-%T"`
>     echo
> "=======================$tmpdate($i/$times)===================
> ===="
>     tmp=$((i%2))
>     if [[ $tmp -eq 0 ]];then
>         echo "############### fio /root/vdb.cfg ###############"
>         fio /root/vdb.cfg
>     else
>         echo "############### fio /root/vdc.cfg ###############"
>         fio /root/vdc.cfg
>     fi
> done
> 
> 
> tmpdate=`date "+%F-%T"`
> echo "############### [$tmpdate] fio /root/vde.cfg ###############"
> fio /root/vde.cfg
> 
> 
> # cat vdb.cfg
> [global]
> rw=randwrite
> direct=1
> numjobs=64
> ioengine=sync
> bsrange=4k-4k
> runtime=180
> group_reporting
> 
> [disk01]
> filename=/dev/vdb
> 
> 
> # cat vdc.cfg
> [global]
> rw=randwrite
> direct=1
> numjobs=64
> ioengine=sync
> bsrange=4k-4k
> runtime=180
> group_reporting
> 
> [disk01]
> filename=/dev/vdc
> 
> # cat vdd.cfg
> [global]
> rw=randwrite
> direct=1
> numjobs=64
> ioengine=sync
> bsrange=4k-4k
> runtime=180
> group_reporting
> 
> [disk01]
> filename=/dev/vdd
> 
> # cat vde.cfg
> [global]
> rw=randwrite
> direct=1
> numjobs=64
> ioengine=sync
> bsrange=4k-4k
> runtime=180
> group_reporting
> 
> [disk01]
> filename=/dev/vde
> 
> After run the scripts reproduce.sh, the disks in the VM's IOPS cutdown from
> 12k to 5k, the latency increase from 4ms to 7.3ms.
> 
> run steps:
> 1. create a VM
> 2. create four volumes and attatch to the VM 3. sh reproduce.sh 4. in the
> runtime of  reproduce.sh, run "fio vdd.cfg" or "fio vde.cfg" to checkt the
> performance
> 
> After reproduce.sh finished, performance down.
> 
> 
> Anyone has the same problem or has some ideas about this?
> 
> Thanks!
> --------------
> hzwulibin
> 2015-11-02
> \x04 {.n +       +%  lzwm  b 맲  r  yǩ ׯzX  \x17  ܨ}   Ơz &j:+v        zZ+  +zf   h   ~    i   z \x1e w   ?
> & )ߢ^[f

Re: [performance] why rbd_aio_write latency increase from 4ms to 7.3ms after the same test

[hzwulibin-Re5JQEeQqe8AvxtiuMwx3w]

[-- Attachment #1.1: Type: text/plain, Size: 3391 bytes --]

Hi,

Thank you, that make sense for testing, but i'm afraid not in my case.
Even i test on the volume that already test many times, the IOPS will not growing up 
again. Yeah, i mean, this VM is broken, IOPS of the VM will never growing up..

Thanks!



hzwulibin@gmail.com
 
From: Chen, Xiaoxi
Date: 2015-11-02 14:11
To: hzwulibin; ceph-devel; ceph-users
Subject: RE: [performance] why rbd_aio_write latency increase from 4ms to 7.3ms after the same test
Pre-allocated the volume by "DD" across the entire RBD before you do any performance test:).
 
In this case, you may want to re-create the RBD, pre-allocate and try again.
 
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
> owner@vger.kernel.org] On Behalf Of hzwulibin
> Sent: Monday, November 2, 2015 1:24 PM
> To: ceph-devel; ceph-users
> Subject: [performance] why rbd_aio_write latency increase from 4ms to
> 7.3ms after the same test
> 
> Hi,
> same environment, after a test script, the io latency(get from sudo ceph --
> admin-daemon /run/ceph/guests/ceph-client.*.asok per dump) increase
> from about 4ms to 7.3ms
> 
> qemu version: debian 2.1.2
> kernel:3.10.45-openstack-amd64
> system: debian 7.8
> ceph: 0.94.5
> VM CPU number: 4  (cpu MHz : 2599.998)
> VM memory size: 16GB
> 9 OSD storage servers, with 4 SSD OSD on each, total 36 OSDs.
> 
> Test scripts in VM:
> # cat reproduce.sh
> #!/bin/bash
> 
> times=20
> for((i=1;i<=$times;i++))
> do
>     tmpdate=`date "+%F-%T"`
>     echo
> "=======================$tmpdate($i/$times)===================
> ===="
>     tmp=$((i%2))
>     if [[ $tmp -eq 0 ]];then
>         echo "############### fio /root/vdb.cfg ###############"
>         fio /root/vdb.cfg
>     else
>         echo "############### fio /root/vdc.cfg ###############"
>         fio /root/vdc.cfg
>     fi
> done
> 
> 
> tmpdate=`date "+%F-%T"`
> echo "############### [$tmpdate] fio /root/vde.cfg ###############"
> fio /root/vde.cfg
> 
> 
> # cat vdb.cfg
> [global]
> rw=randwrite
> direct=1
> numjobs=64
> ioengine=sync
> bsrange=4k-4k
> runtime=180
> group_reporting
> 
> [disk01]
> filename=/dev/vdb
> 
> 
> # cat vdc.cfg
> [global]
> rw=randwrite
> direct=1
> numjobs=64
> ioengine=sync
> bsrange=4k-4k
> runtime=180
> group_reporting
> 
> [disk01]
> filename=/dev/vdc
> 
> # cat vdd.cfg
> [global]
> rw=randwrite
> direct=1
> numjobs=64
> ioengine=sync
> bsrange=4k-4k
> runtime=180
> group_reporting
> 
> [disk01]
> filename=/dev/vdd
> 
> # cat vde.cfg
> [global]
> rw=randwrite
> direct=1
> numjobs=64
> ioengine=sync
> bsrange=4k-4k
> runtime=180
> group_reporting
> 
> [disk01]
> filename=/dev/vde
> 
> After run the scripts reproduce.sh, the disks in the VM's IOPS cutdown from
> 12k to 5k, the latency increase from 4ms to 7.3ms.
> 
> run steps:
> 1. create a VM
> 2. create four volumes and attatch to the VM 3. sh reproduce.sh 4. in the
> runtime of  reproduce.sh, run "fio vdd.cfg" or "fio vde.cfg" to checkt the
> performance
> 
> After reproduce.sh finished, performance down.
> 
> 
> Anyone has the same problem or has some ideas about this?
> 
> Thanks!
> --------------
> hzwulibin
> 2015-11-02
> {.n +       +%  lzwm  b 맲  r  yǩ ׯzX    ܨ}   Ơz &j:+v        zZ+  +zf   h   ~    i   z w   ?
> & )ߢ f

[-- Attachment #1.2: Type: text/html, Size: 7035 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: RE: [performance] why rbd_aio_write latency increase from 4ms to 7.3ms after the same test

[hzwulibin]

Hi,

Thank you, that make sense for testing, but i'm afraid not in my case.
Even i test on the volume that already test many times, the IOPS will not growing up 
again. Yeah, i mean, this VM is broken, IOPS of the VM will never growing up..

Thanks!

------------------				 
hzwulibin
2015-11-03

-------------------------------------------------------------
发件人："Chen, Xiaoxi" <xiaoxi.chen@intel.com>
发送日期：2015-11-02 14:11
收件人：hzwulibin,ceph-devel,ceph-users
抄送：
主题：RE: [performance] why rbd_aio_write latency increase from 4ms to
 7.3ms after the same test

Pre-allocated the volume by "DD" across the entire RBD before you do any performance test:).

In this case, you may want to re-create the RBD, pre-allocate and try again.

> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
> owner@vger.kernel.org] On Behalf Of hzwulibin
> Sent: Monday, November 2, 2015 1:24 PM
> To: ceph-devel; ceph-users
> Subject: [performance] why rbd_aio_write latency increase from 4ms to
> 7.3ms after the same test
> 
> Hi,
> same environment, after a test script, the io latency(get from sudo ceph --
> admin-daemon /run/ceph/guests/ceph-client.*.asok per dump) increase
> from about 4ms to 7.3ms
> 
> qemu version: debian 2.1.2
> kernel:3.10.45-openstack-amd64
> system: debian 7.8
> ceph: 0.94.5
> VM CPU number: 4  (cpu MHz : 2599.998)
> VM memory size: 16GB
> 9 OSD storage servers, with 4 SSD OSD on each, total 36 OSDs.
> 
> Test scripts in VM:
> # cat reproduce.sh
> #!/bin/bash
> 
> times=20
> for((i=1;i<=$times;i++))
> do
>     tmpdate=`date "+%F-%T"`
>     echo
> "=======================$tmpdate($i/$times)===================
> ===="
>     tmp=$((i%2))
>     if [[ $tmp -eq 0 ]];then
>         echo "############### fio /root/vdb.cfg ###############"
>         fio /root/vdb.cfg
>     else
>         echo "############### fio /root/vdc.cfg ###############"
>         fio /root/vdc.cfg
>     fi
> done
> 
> 
> tmpdate=`date "+%F-%T"`
> echo "############### [$tmpdate] fio /root/vde.cfg ###############"
> fio /root/vde.cfg
> 
> 
> # cat vdb.cfg
> [global]
> rw=randwrite
> direct=1
> numjobs=64
> ioengine=sync
> bsrange=4k-4k
> runtime=180
> group_reporting
> 
> [disk01]
> filename=/dev/vdb
> 
> 
> # cat vdc.cfg
> [global]
> rw=randwrite
> direct=1
> numjobs=64
> ioengine=sync
> bsrange=4k-4k
> runtime=180
> group_reporting
> 
> [disk01]
> filename=/dev/vdc
> 
> # cat vdd.cfg
> [global]
> rw=randwrite
> direct=1
> numjobs=64
> ioengine=sync
> bsrange=4k-4k
> runtime=180
> group_reporting
> 
> [disk01]
> filename=/dev/vdd
> 
> # cat vde.cfg
> [global]
> rw=randwrite
> direct=1
> numjobs=64
> ioengine=sync
> bsrange=4k-4k
> runtime=180
> group_reporting
> 
> [disk01]
> filename=/dev/vde
> 
> After run the scripts reproduce.sh, the disks in the VM's IOPS cutdown from
> 12k to 5k, the latency increase from 4ms to 7.3ms.
> 
> run steps:
> 1. create a VM
> 2. create four volumes and attatch to the VM 3. sh reproduce.sh 4. in the
> runtime of  reproduce.sh, run "fio vdd.cfg" or "fio vde.cfg" to checkt the
> performance
> 
> After reproduce.sh finished, performance down.
> 
> 
> Anyone has the same problem or has some ideas about this?
> 
> Thanks!
> --------------
> hzwulibin
> 2015-11-02
> \x04 {.n +       +%  lzwm  b 맲  r  yǩ ׯzX  \x17  ܨ}   Ơz &j:+v        zZ+  +zf   h   ~    i   z \x1e w   ?
> & )ߢ^[f

Re: RE: [performance] why rbd_aio_write latency increase from 4ms to 7.3ms after the same test

[hzwulibin]

Hi,

Thank you, that make sense for testing, but i'm afraid not in my case.
Even i test on the volume that already test many times, the IOPS will not growing up 
again. Yeah, i mean, this VM is broken, IOPS of the VM will never growing up..

Thanks!

------------------				 
hzwulibin
2015-11-03

-------------------------------------------------------------
发件人："Chen, Xiaoxi" <xiaoxi.chen@intel.com>
发送日期：2015-11-02 14:11
收件人：hzwulibin,ceph-devel,ceph-users
抄送：
主题：RE: [performance] why rbd_aio_write latency increase from 4ms to
 7.3ms after the same test

Pre-allocated the volume by "DD" across the entire RBD before you do any performance test:).

In this case, you may want to re-create the RBD, pre-allocate and try again.

> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
> owner@vger.kernel.org] On Behalf Of hzwulibin
> Sent: Monday, November 2, 2015 1:24 PM
> To: ceph-devel; ceph-users
> Subject: [performance] why rbd_aio_write latency increase from 4ms to
> 7.3ms after the same test
> 
> Hi,
> same environment, after a test script, the io latency(get from sudo ceph --
> admin-daemon /run/ceph/guests/ceph-client.*.asok per dump) increase
> from about 4ms to 7.3ms
> 
> qemu version: debian 2.1.2
> kernel:3.10.45-openstack-amd64
> system: debian 7.8
> ceph: 0.94.5
> VM CPU number: 4  (cpu MHz : 2599.998)
> VM memory size: 16GB
> 9 OSD storage servers, with 4 SSD OSD on each, total 36 OSDs.
> 
> Test scripts in VM:
> # cat reproduce.sh
> #!/bin/bash
> 
> times=20
> for((i=1;i<=$times;i++))
> do
>     tmpdate=`date "+%F-%T"`
>     echo
> "=======================$tmpdate($i/$times)===================
> ===="
>     tmp=$((i%2))
>     if [[ $tmp -eq 0 ]];then
>         echo "############### fio /root/vdb.cfg ###############"
>         fio /root/vdb.cfg
>     else
>         echo "############### fio /root/vdc.cfg ###############"
>         fio /root/vdc.cfg
>     fi
> done
> 
> 
> tmpdate=`date "+%F-%T"`
> echo "############### [$tmpdate] fio /root/vde.cfg ###############"
> fio /root/vde.cfg
> 
> 
> # cat vdb.cfg
> [global]
> rw=randwrite
> direct=1
> numjobs=64
> ioengine=sync
> bsrange=4k-4k
> runtime=180
> group_reporting
> 
> [disk01]
> filename=/dev/vdb
> 
> 
> # cat vdc.cfg
> [global]
> rw=randwrite
> direct=1
> numjobs=64
> ioengine=sync
> bsrange=4k-4k
> runtime=180
> group_reporting
> 
> [disk01]
> filename=/dev/vdc
> 
> # cat vdd.cfg
> [global]
> rw=randwrite
> direct=1
> numjobs=64
> ioengine=sync
> bsrange=4k-4k
> runtime=180
> group_reporting
> 
> [disk01]
> filename=/dev/vdd
> 
> # cat vde.cfg
> [global]
> rw=randwrite
> direct=1
> numjobs=64
> ioengine=sync
> bsrange=4k-4k
> runtime=180
> group_reporting
> 
> [disk01]
> filename=/dev/vde
> 
> After run the scripts reproduce.sh, the disks in the VM's IOPS cutdown from
> 12k to 5k, the latency increase from 4ms to 7.3ms.
> 
> run steps:
> 1. create a VM
> 2. create four volumes and attatch to the VM 3. sh reproduce.sh 4. in the
> runtime of  reproduce.sh, run "fio vdd.cfg" or "fio vde.cfg" to checkt the
> performance
> 
> After reproduce.sh finished, performance down.
> 
> 
> Anyone has the same problem or has some ideas about this?
> 
> Thanks!
> --------------
> hzwulibin
> 2015-11-02
> \x04 {.n +       +%  lzwm  b 맲  r  yǩ ׯzX  \x17  ܨ}   Ơz &j:+v        zZ+  +zf   h   ~    i   z \x1e w   ?
> & )ߢ^[f