Re: Poor read performance in KVM

[Vladimir Bashkirtsev <vladimir@bashkirtsev.com>]

On 21/07/12 02:12, Tommi Virtanen wrote:
> But it leaves me with very final question: should we rely on btrfs at this
> point given it is having such major faults? What if I will use well tested
> by time ext4?
> You might want to try xfs. We hear/see problems with all three, but
> xfs currently seems to have the best long-term performance and
> reliability.
>
> I'm not sure if anyone's run detailed tests with ext4 after the
> xattrs-in-leveldb feature; before that, we ran into fs limitations.
Just reporting back what was going on for last week. I have rebuilt all 
OSDs with fresh btrfs and leaf size of 64K. Straight after rebuild 
everything was flying! But mysql processing I wrote about continued and 
whole cluster was brought again to a stand still in a week. I have done 
some investigation as to causes and it appears that fragmentation went 
ballistic. Reading somewhere on the net I have seen suggestion that if 
cow is not really needed then btrfs mounted with nocow option less 
likely to get overly fragmented. Haven't tried it actually but wondering 
will ceph cope well with nocow? ie does it rely on cow feature? 
Something tells me that as ceph can run on FS which does not have cow we 
actually can mount nocow. Just need some confirmation from devs.

In the mean time I opted to convert all OSDs to xfs. Even after 
rebuilding only two OSDs performance boost is apparent again. So it 
appears that btrfs as it currently is in 3.4.6 is not up to prime time 
and good number of random writes consistently bring it to a halt.

As xfs apparently have its own share of problems when ageing I think 
that periodic online defragmentation may bring xfs back to reasonable 
condition. Have anyone tried xfs defragmentation while ceph uses it?