Pre-allocated the volume by "DD" across the entire RBD before you do any performance test:).
In this case, you may want to re-create the RBD, pre-allocate and try again.
> -----Original Message-----
> From: ceph-devel-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:ceph-devel-
> owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of hzwulibin
> Sent: Monday, November 2, 2015 1:24 PM
> To: ceph-devel; ceph-users
> Subject: [performance] why rbd_aio_write latency increase from 4ms to
> 7.3ms after the same test
>
> Hi,
> same environment, after a test script, the io latency(get from sudo ceph --
> admin-daemon /run/ceph/guests/ceph-client.*.asok per dump) increase
> from about 4ms to 7.3ms
>
> qemu version: debian 2.1.2
> kernel:3.10.45-openstack-amd64
> system: debian 7.8
> ceph: 0.94.5
> VM CPU number: 4 (cpu MHz : 2599.998)
> VM memory size: 16GB
> 9 OSD storage servers, with 4 SSD OSD on each, total 36 OSDs.
>
> Test scripts in VM:
> # cat reproduce.sh
> #!/bin/bash
>
> times=20
> for((i=1;i<=$times;i++))
> do
> tmpdate=`date "+%F-%T"`
> echo
> "=======================$tmpdate($i/$times)===================
> ===="
> tmp=$((i%2))
> if [[ $tmp -eq 0 ]];then
> echo "############### fio /root/vdb.cfg ###############"
> fio /root/vdb.cfg
> else
> echo "############### fio /root/vdc.cfg ###############"
> fio /root/vdc.cfg
> fi
> done
>
>
> tmpdate=`date "+%F-%T"`
> echo "############### [$tmpdate] fio /root/vde.cfg ###############"
> fio /root/vde.cfg
>
>
> # cat vdb.cfg
> [global]
> rw=randwrite
> direct=1
> numjobs=64
> ioengine=sync
> bsrange=4k-4k
> runtime=180
> group_reporting
>
> [disk01]
> filename=/dev/vdb
>
>
> # cat vdc.cfg
> [global]
> rw=randwrite
> direct=1
> numjobs=64
> ioengine=sync
> bsrange=4k-4k
> runtime=180
> group_reporting
>
> [disk01]
> filename=/dev/vdc
>
> # cat vdd.cfg
> [global]
> rw=randwrite
> direct=1
> numjobs=64
> ioengine=sync
> bsrange=4k-4k
> runtime=180
> group_reporting
>
> [disk01]
> filename=/dev/vdd
>
> # cat vde.cfg
> [global]
> rw=randwrite
> direct=1
> numjobs=64
> ioengine=sync
> bsrange=4k-4k
> runtime=180
> group_reporting
>
> [disk01]
> filename=/dev/vde
>
> After run the scripts reproduce.sh, the disks in the VM's IOPS cutdown from
> 12k to 5k, the latency increase from 4ms to 7.3ms.
>
> run steps:
> 1. create a VM
> 2. create four volumes and attatch to the VM 3. sh reproduce.sh 4. in the
> runtime of reproduce.sh, run "fio vdd.cfg" or "fio vde.cfg" to checkt the
> performance
>
> After reproduce.sh finished, performance down.
>
>
> Anyone has the same problem or has some ideas about this?
>
> Thanks!
> --------------
> hzwulibin
> 2015-11-02
> {.n + +% lzwm b 맲 r yǩ ׯzX ܨ} Ơz &j:+v zZ+ +zf h ~ i z w ?
> & )ߢf