Re: [Xen-users] File-based domU - Slow storage write since xen 4.8

[Benoit Depail <benoit.depail@nbs-system.com>]

On 07/20/17 19:36, Keith Busch wrote:
> On Thu, Jul 20, 2017 at 05:12:33PM +0200, Benoit Depail wrote:
>> ##### Dom0 with a kernel >= 4.4.2, or a custom 4.4.1 including commit
>> d2081cfe624b5decaaf68088ca256ed1b140672c
>>
>> On the dom0:
>> # dd if=/dev/zero of=/mnt/d-anb-nab2.img bs=4M count=1280
>> 1280+0 records in
>> 1280+0 records out
>> 5368709120 bytes (5.4 GB) copied, 46.3234 s, 116 MB/s
>>
>> # dd if=/dev/zero of=/dev/loop1 bs=4M count=1280
>> 1280+0 records in
>> 1280+0 records out
>> 5368709120 bytes (5.4 GB) copied, 44.948 s, 119 MB/s
>>
>> On the domU:
>> # dd if=/dev/zero of=/dev/xvdb bs=4M count=1280
>> 1280+0 records in
>> 1280+0 records out
>> 5368709120 bytes (5.4 GB) copied, 102.943 s, 52.2 MB/s
>>
>>
>> For completeness sake, I'll put my findings below:
>>
>>> I used git bisect on the linux-stable source tree to build (a lot of)
>>> tests kernels, and was able to find this commit as the first one
>>> introducing the regression :
>>>
>>> d2081cfe624b5decaaf68088ca256ed1b140672c is the first bad commit
>>> commit d2081cfe624b5decaaf68088ca256ed1b140672c
>>> Author: Keith Busch <keith.busch@intel.com>
>>> Date:   Tue Jan 12 15:08:39 2016 -0700
>>>
>>>     block: split bios to max possible length
>>>
>>> In term of kernel version, the first one showing bad performances in my
>>> case is 4.4.2 (with, obviously, 4.4.1 working as expected).
>>>
>>> Interestingly, this commit is an improvement of
>>> d3805611130af9b911e908af9f67a3f64f4f0914, which is present in 4.4-rc8
>>> but do not show any performance issue in our case.
>>>
>>> I can also confirm that this issue is still present in the latest
>>> unstable kernel (we tested 4.13-rc1).
> 
> I admit I don't have a good working knowledge of xen block. I've spent a
> few minutes looking over the code, and the best I can tell is that this
> patch will build requests up to 88 blocks ((11 segments * 4k) / 512)
> that would have been fewer blocks without this patch. The resulting bio
> vectors may also have offsets that weren't there before.
> 
> I'm not sure what the significance is. Perhaps it's causing the number
> of grants to exceed BLKIF_MAX_SEGMENTS_PER_REQUEST, which appears to take
> a less optimal path in the code, but I'm not sure why that would only
> harm writes.
> 
> As a test, could you throttle the xvdb queue's max_sectors_kb? If I
> followed xen-blkfront correctly, the default should have it set to 44.
> Try setting it to 40.
> 
>   echo 40 > /sys/block/xvdb/queue/max_sectors_kb
> 

The default value on my domU is 128.

I ran a couple of tests with different values, starting from 40 and up
to 128, clearing the cache between each tests.

The only value that showed the issue is 128. Even setting max_sectors_kb
to 127 is enough to get normal behaviour.

Best regards,

-- 
Benoit Depail
Senior Infrastructures Architect
NBS System