Re: Poor read performance in KVM

[Josh Durgin <josh.durgin@inktank.com>]

On 07/15/2012 06:13 AM, Vladimir Bashkirtsev wrote:
> Hello,
>
> Lately I was trying to get KVM to perform well on RBD. But it still
> appears elusive.
>
> [root@alpha etc]# rados -p rbd bench 120 seq -t 8
>
> Total time run:        16.873277
> Total reads made:     302
> Read size:            4194304
> Bandwidth (MB/sec):    71.592
>
> Average Latency:       0.437984
> Max latency:           3.26817
> Min latency:           0.015786
>
> Fairly good performance. But when I run in KVM:
>
> [root@mail ~]# hdparm -tT /dev/vda
>
> /dev/vda:
>   Timing cached reads:   8808 MB in  2.00 seconds = 4411.49 MB/sec

This is just the guest page cache - it's reading the first two
megabytes of the device repeatedly.

>   Timing buffered disk reads:  10 MB in  6.21 seconds =   1.61 MB/sec

This is a sequential read, so readahead in the guest should help here.

> Not even close to what rados bench show! I even seen 900KB/sec
> performance. Such slow read performance of course affecting guests.
>
> Any ideas where to start to look for performance boost?

Do you have rbd caching enabled? It would also be interesting to see
how the guest reads are translating to rados reads. hdparm is doing
2MiB sequential reads of the block device. If you add
admin_socket=/var/run/ceph/kvm.asok to the rbd device on the qemu
command line) you can see number of requests, latency, and
request size info while the guest is running via:

ceph --admin-daemon /var/run/ceph/kvm.asok perf dump

Josh