Re: [Xen-users] File-based domU - Slow storage write since xen 4.8

Re: [Xen-users] File-based domU - Slow storage write since xen 4.8

[Roger Pau Monné]

I'm afraid I'm also not that knowledgeable, so I'm adding the author
and the Linux block ml.

On Wed, Jul 19, 2017 at 04:25:02PM +0200, Benoit Depail wrote:
> Hello,
> 
> On 07/17/17 18:46, Roger Pau Monn� wrote:
> > There are not a huge amount of changes to blkback, so it would be
> > helpful if you could pinpoint the change that introduced this
> > regression for you, or if you can get a kernel version that introduced
> > the regression that would also be helpful.
> 
> I used git bisect on the linux-stable source tree to build (a lot of)
> tests kernels, and was able to find this commit as the first one
> introducing the regression :
> 
> d2081cfe624b5decaaf68088ca256ed1b140672c is the first bad commit
> commit d2081cfe624b5decaaf68088ca256ed1b140672c
> Author: Keith Busch <keith.busch@intel.com>
> Date:   Tue Jan 12 15:08:39 2016 -0700
> 
>     block: split bios to max possible length
> 
> In term of kernel version, the first one showing bad performances in my
> case is 4.4.2 (with, obviously, 4.4.1 working as expected).
> 
> Interestingly, this commit is an improvement of
> d3805611130af9b911e908af9f67a3f64f4f0914, which is present in 4.4-rc8
> but do not show any performance issue in our case.
> 
> I can also confirm that this issue is still present in the latest
> unstable kernel (we tested 4.13-rc1).
> 
> I think I have reached a point where more competent people should
> investigate, but I'll be glad to help if needed.

For the sake of the people that I've added, would you mind describing
the issues you are seeing and the tests that you performed?

Thanks, Roger.

Re: [Xen-users] File-based domU - Slow storage write since xen 4.8

[Benoit Depail]

[-- Attachment #1.1: Type: text/plain, Size: 3262 bytes --]

On 07/20/17 10:52, Roger Pau Monné wrote:
> For the sake of the people that I've added, would you mind describing
> the issues you are seeing and the tests that you performed?
> 
> Thanks, Roger.
> 

Sure.

The main issue we are seeing is degraded write performance on storage
devices of Xen PV DomUs, about half (or even a third on our production
setup where NFS is involved) of what we used to have.

Our test setup is as follow (we left the NFS out as it has nothing to do
with our problem):

On a physical server running debian stretch as a Xen 4.8 hypervisor, we
create a PV domU, also running debian stretch. The storage of this domU
resides on the server's local disk as raw image files, which will be
setup as loop-devices before being exposed to the domU as /dev/xvd{a,b}.

the domU configuration (relevant part):

disk        = [
   'file:/mnt/domu_root.img,xvda,w',
   'file:/mnt/d-anb-nab2.img,xvdb,w'
]


When the domU is running, a loop-device is created:

/dev/loop1: [65027]:12 (/mnt/d-anb-nab2.img)

We perform a simple dd to test the sequential writing speed:

##### Dom0 with a kernel < 4.4.2 (here, 4.1.42, we had similar results
with 3.16.0)

# dd if=/dev/zero of=/mnt/d-anb-nab2.img bs=4M count=1280
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 47.5052 s, 113 MB/s

# dd if=/dev/zero of=/dev/loop1 bs=4M count=1280
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 46.8227 s, 115 MB/s

On the domU:
# dd if=/dev/zero of=/dev/xvdb bs=4M count=1280
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 46.7245 s, 115 MB/s


##### Dom0 with a kernel >= 4.4.2, or a custom 4.4.1 including commit
d2081cfe624b5decaaf68088ca256ed1b140672c

On the dom0:
# dd if=/dev/zero of=/mnt/d-anb-nab2.img bs=4M count=1280
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 46.3234 s, 116 MB/s

# dd if=/dev/zero of=/dev/loop1 bs=4M count=1280
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 44.948 s, 119 MB/s

On the domU:
# dd if=/dev/zero of=/dev/xvdb bs=4M count=1280
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 102.943 s, 52.2 MB/s


For completeness sake, I'll put my findings below:

> I used git bisect on the linux-stable source tree to build (a lot of)
> tests kernels, and was able to find this commit as the first one
> introducing the regression :
>
> d2081cfe624b5decaaf68088ca256ed1b140672c is the first bad commit
> commit d2081cfe624b5decaaf68088ca256ed1b140672c
> Author: Keith Busch <keith.busch@intel.com>
> Date:   Tue Jan 12 15:08:39 2016 -0700
>
>     block: split bios to max possible length
>
> In term of kernel version, the first one showing bad performances in my
> case is 4.4.2 (with, obviously, 4.4.1 working as expected).
>
> Interestingly, this commit is an improvement of
> d3805611130af9b911e908af9f67a3f64f4f0914, which is present in 4.4-rc8
> but do not show any performance issue in our case.
>
> I can also confirm that this issue is still present in the latest
> unstable kernel (we tested 4.13-rc1).


Thanks,

-- 
Benoit Depail
Senior Infrastructures Architect
NBS System


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 1004 bytes --]

Re: [Xen-users] File-based domU - Slow storage write since xen 4.8

[Keith Busch]

On Thu, Jul 20, 2017 at 05:12:33PM +0200, Benoit Depail wrote:
> 
> The main issue we are seeing is degraded write performance on storage
> devices of Xen PV DomUs, about half (or even a third on our production
> setup where NFS is involved) of what we used to have.

Read performance is unchanged?

Re: [Xen-users] File-based domU - Slow storage write since xen 4.8

[Benoit Depail]

[-- Attachment #1.1: Type: text/plain, Size: 470 bytes --]

On 07/20/17 17:33, Keith Busch wrote:
> On Thu, Jul 20, 2017 at 05:12:33PM +0200, Benoit Depail wrote:
>>
>> The main issue we are seeing is degraded write performance on storage
>> devices of Xen PV DomUs, about half (or even a third on our production
>> setup where NFS is involved) of what we used to have.
> 
> Read performance is unchanged?
> 

Read performance is unaffected indeed.

-- 
Benoit Depail
Senior Infrastructures Architect
NBS System


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 1004 bytes --]

Re: [Xen-users] File-based domU - Slow storage write since xen 4.8

[Keith Busch]

On Thu, Jul 20, 2017 at 05:12:33PM +0200, Benoit Depail wrote:
> ##### Dom0 with a kernel >= 4.4.2, or a custom 4.4.1 including commit
> d2081cfe624b5decaaf68088ca256ed1b140672c
> 
> On the dom0:
> # dd if=/dev/zero of=/mnt/d-anb-nab2.img bs=4M count=1280
> 1280+0 records in
> 1280+0 records out
> 5368709120 bytes (5.4 GB) copied, 46.3234 s, 116 MB/s
> 
> # dd if=/dev/zero of=/dev/loop1 bs=4M count=1280
> 1280+0 records in
> 1280+0 records out
> 5368709120 bytes (5.4 GB) copied, 44.948 s, 119 MB/s
> 
> On the domU:
> # dd if=/dev/zero of=/dev/xvdb bs=4M count=1280
> 1280+0 records in
> 1280+0 records out
> 5368709120 bytes (5.4 GB) copied, 102.943 s, 52.2 MB/s
> 
> 
> For completeness sake, I'll put my findings below:
> 
> > I used git bisect on the linux-stable source tree to build (a lot of)
> > tests kernels, and was able to find this commit as the first one
> > introducing the regression :
> >
> > d2081cfe624b5decaaf68088ca256ed1b140672c is the first bad commit
> > commit d2081cfe624b5decaaf68088ca256ed1b140672c
> > Author: Keith Busch <keith.busch@intel.com>
> > Date:   Tue Jan 12 15:08:39 2016 -0700
> >
> >     block: split bios to max possible length
> >
> > In term of kernel version, the first one showing bad performances in my
> > case is 4.4.2 (with, obviously, 4.4.1 working as expected).
> >
> > Interestingly, this commit is an improvement of
> > d3805611130af9b911e908af9f67a3f64f4f0914, which is present in 4.4-rc8
> > but do not show any performance issue in our case.
> >
> > I can also confirm that this issue is still present in the latest
> > unstable kernel (we tested 4.13-rc1).

I admit I don't have a good working knowledge of xen block. I've spent a
few minutes looking over the code, and the best I can tell is that this
patch will build requests up to 88 blocks ((11 segments * 4k) / 512)
that would have been fewer blocks without this patch. The resulting bio
vectors may also have offsets that weren't there before.

I'm not sure what the significance is. Perhaps it's causing the number
of grants to exceed BLKIF_MAX_SEGMENTS_PER_REQUEST, which appears to take
a less optimal path in the code, but I'm not sure why that would only
harm writes.

As a test, could you throttle the xvdb queue's max_sectors_kb? If I
followed xen-blkfront correctly, the default should have it set to 44.
Try setting it to 40.

  echo 40 > /sys/block/xvdb/queue/max_sectors_kb

Re: [Xen-users] File-based domU - Slow storage write since xen 4.8

[Benoit Depail]

On 07/20/17 19:36, Keith Busch wrote:
> On Thu, Jul 20, 2017 at 05:12:33PM +0200, Benoit Depail wrote:
>> ##### Dom0 with a kernel >= 4.4.2, or a custom 4.4.1 including commit
>> d2081cfe624b5decaaf68088ca256ed1b140672c
>>
>> On the dom0:
>> # dd if=/dev/zero of=/mnt/d-anb-nab2.img bs=4M count=1280
>> 1280+0 records in
>> 1280+0 records out
>> 5368709120 bytes (5.4 GB) copied, 46.3234 s, 116 MB/s
>>
>> # dd if=/dev/zero of=/dev/loop1 bs=4M count=1280
>> 1280+0 records in
>> 1280+0 records out
>> 5368709120 bytes (5.4 GB) copied, 44.948 s, 119 MB/s
>>
>> On the domU:
>> # dd if=/dev/zero of=/dev/xvdb bs=4M count=1280
>> 1280+0 records in
>> 1280+0 records out
>> 5368709120 bytes (5.4 GB) copied, 102.943 s, 52.2 MB/s
>>
>>
>> For completeness sake, I'll put my findings below:
>>
>>> I used git bisect on the linux-stable source tree to build (a lot of)
>>> tests kernels, and was able to find this commit as the first one
>>> introducing the regression :
>>>
>>> d2081cfe624b5decaaf68088ca256ed1b140672c is the first bad commit
>>> commit d2081cfe624b5decaaf68088ca256ed1b140672c
>>> Author: Keith Busch <keith.busch@intel.com>
>>> Date:   Tue Jan 12 15:08:39 2016 -0700
>>>
>>>     block: split bios to max possible length
>>>
>>> In term of kernel version, the first one showing bad performances in my
>>> case is 4.4.2 (with, obviously, 4.4.1 working as expected).
>>>
>>> Interestingly, this commit is an improvement of
>>> d3805611130af9b911e908af9f67a3f64f4f0914, which is present in 4.4-rc8
>>> but do not show any performance issue in our case.
>>>
>>> I can also confirm that this issue is still present in the latest
>>> unstable kernel (we tested 4.13-rc1).
> 
> I admit I don't have a good working knowledge of xen block. I've spent a
> few minutes looking over the code, and the best I can tell is that this
> patch will build requests up to 88 blocks ((11 segments * 4k) / 512)
> that would have been fewer blocks without this patch. The resulting bio
> vectors may also have offsets that weren't there before.
> 
> I'm not sure what the significance is. Perhaps it's causing the number
> of grants to exceed BLKIF_MAX_SEGMENTS_PER_REQUEST, which appears to take
> a less optimal path in the code, but I'm not sure why that would only
> harm writes.
> 
> As a test, could you throttle the xvdb queue's max_sectors_kb? If I
> followed xen-blkfront correctly, the default should have it set to 44.
> Try setting it to 40.
> 
>   echo 40 > /sys/block/xvdb/queue/max_sectors_kb
> 

The default value on my domU is 128.

I ran a couple of tests with different values, starting from 40 and up
to 128, clearing the cache between each tests.

The only value that showed the issue is 128. Even setting max_sectors_kb
to 127 is enough to get normal behaviour.

Best regards,

-- 
Benoit Depail
Senior Infrastructures Architect
NBS System

Re: [Xen-users] File-based domU - Slow storage write since xen 4.8

[Keith Busch]

On Fri, Jul 21, 2017 at 12:19:39PM +0200, Benoit Depail wrote:
> On 07/20/17 19:36, Keith Busch wrote:
> > 
> > As a test, could you throttle the xvdb queue's max_sectors_kb? If I
> > followed xen-blkfront correctly, the default should have it set to 44.
> > Try setting it to 40.
> > 
> >   echo 40 > /sys/block/xvdb/queue/max_sectors_kb
> > 
> 
> The default value on my domU is 128.
> 
> I ran a couple of tests with different values, starting from 40 and up
> to 128, clearing the cache between each tests.
> 
> The only value that showed the issue is 128. Even setting max_sectors_kb
> to 127 is enough to get normal behaviour.

Ok, I don't quite follow how it's initialized to 128k, but your
results appear to confirm the default settings are not optimal for the
interface. The patch you identified just submits commands to the max
size the driver asked to use. If the largest size tanks performance,
I think xen-blkfront should register a smaller transfer size, or maybe
some other constraint needs to be refined.

Roger,

I'm a bit out of my depth here in the xen code. Is this something you
may be able to help clarify?

Thanks,
Keith

Re: [Xen-users] File-based domU - Slow storage write since xen 4.8

[Roger Pau Monné]

On Fri, Jul 21, 2017 at 11:53:33AM -0400, Keith Busch wrote:
> On Fri, Jul 21, 2017 at 12:19:39PM +0200, Benoit Depail wrote:
> > On 07/20/17 19:36, Keith Busch wrote:
> > > 
> > > As a test, could you throttle the xvdb queue's max_sectors_kb? If I
> > > followed xen-blkfront correctly, the default should have it set to 44.
> > > Try setting it to 40.
> > > 
> > >   echo 40 > /sys/block/xvdb/queue/max_sectors_kb
> > > 
> > 
> > The default value on my domU is 128.
> > 
> > I ran a couple of tests with different values, starting from 40 and up
> > to 128, clearing the cache between each tests.
> > 
> > The only value that showed the issue is 128. Even setting max_sectors_kb
> > to 127 is enough to get normal behaviour.
> 
> Ok, I don't quite follow how it's initialized to 128k, but your
> results appear to confirm the default settings are not optimal for the
> interface. The patch you identified just submits commands to the max
> size the driver asked to use. If the largest size tanks performance,
> I think xen-blkfront should register a smaller transfer size, or maybe
> some other constraint needs to be refined.
> 
> Roger,
> 
> I'm a bit out of my depth here in the xen code. Is this something you
> may be able to help clarify?

Hm, I'm not sure I follow either. AFAIK this problem came from
changing the Linux version in the Dom0 (where the backend, blkback is
running), rather than in the DomU right?

Regarding the queue/sectors stuff, blkfront uses several blk_queue
functions to set those parameters, maybe there's something wrong
there, but I cannot really spot what it is:

http://elixir.free-electrons.com/linux/latest/source/drivers/block/xen-blkfront.c#L929

In the past the number of pages that could fit in a single ring
request was limited to 11, but some time ago indirect descriptors
where introduced in order to lift this limit, and now requests can
have a much bigger number of pages.

Could you check the max_sectors_kb of the underlying storage you are
using in Dom0?

Roger.

Re: [Xen-users] File-based domU - Slow storage write since xen 4.8

[Benoit Depail]

On 07/21/17 18:07, Roger Pau Monné wrote:
> On Fri, Jul 21, 2017 at 11:53:33AM -0400, Keith Busch wrote:
>> On Fri, Jul 21, 2017 at 12:19:39PM +0200, Benoit Depail wrote:
>>> On 07/20/17 19:36, Keith Busch wrote:
>>>>
>>>> As a test, could you throttle the xvdb queue's max_sectors_kb? If I
>>>> followed xen-blkfront correctly, the default should have it set to 44.
>>>> Try setting it to 40.
>>>>
>>>>   echo 40 > /sys/block/xvdb/queue/max_sectors_kb
>>>>
>>>
>>> The default value on my domU is 128.
>>>
>>> I ran a couple of tests with different values, starting from 40 and up
>>> to 128, clearing the cache between each tests.
>>>
>>> The only value that showed the issue is 128. Even setting max_sectors_kb
>>> to 127 is enough to get normal behaviour.
>>
>> Ok, I don't quite follow how it's initialized to 128k, but your
>> results appear to confirm the default settings are not optimal for the
>> interface. The patch you identified just submits commands to the max
>> size the driver asked to use. If the largest size tanks performance,
>> I think xen-blkfront should register a smaller transfer size, or maybe
>> some other constraint needs to be refined.
>>
>> Roger,
>>
>> I'm a bit out of my depth here in the xen code. Is this something you
>> may be able to help clarify?
> 
> Hm, I'm not sure I follow either. AFAIK this problem came from
> changing the Linux version in the Dom0 (where the backend, blkback is
> running), rather than in the DomU right?
> 
> Regarding the queue/sectors stuff, blkfront uses several blk_queue
> functions to set those parameters, maybe there's something wrong
> there, but I cannot really spot what it is:
> 
> http://elixir.free-electrons.com/linux/latest/source/drivers/block/xen-blkfront.c#L929
> 
> In the past the number of pages that could fit in a single ring
> request was limited to 11, but some time ago indirect descriptors
> where introduced in order to lift this limit, and now requests can
> have a much bigger number of pages.
> 
> Could you check the max_sectors_kb of the underlying storage you are
> using in Dom0?
> 
> Roger.
> 
I checked the value for the loop device as well

With 4.4.77 (bad write performance)
$ cat /sys/block/sda/queue/max_sectors_kb
1280
$ cat /sys/block/loop1/queue/max_sectors_kb
127


With 4.1.42 (normal write performance)
$ cat /sys/block/sda/queue/max_sectors_kb
4096
$ cat /sys/block/loop1/queue/max_sectors_kb
127

Regards,

-- 
Benoit Depail
Senior Infrastructures Architect
NBS System

Re: [Xen-users] File-based domU - Slow storage write since xen 4.8

[Keith Busch]

On Fri, Jul 21, 2017 at 07:07:06PM +0200, Benoit Depail wrote:
> On 07/21/17 18:07, Roger Pau Monn� wrote:
> > 
> > Hm, I'm not sure I follow either. AFAIK this problem came from
> > changing the Linux version in the Dom0 (where the backend, blkback is
> > running), rather than in the DomU right?
> > 
> > Regarding the queue/sectors stuff, blkfront uses several blk_queue
> > functions to set those parameters, maybe there's something wrong
> > there, but I cannot really spot what it is:
> > 
> > http://elixir.free-electrons.com/linux/latest/source/drivers/block/xen-blkfront.c#L929
> > 
> > In the past the number of pages that could fit in a single ring
> > request was limited to 11, but some time ago indirect descriptors
> > where introduced in order to lift this limit, and now requests can
> > have a much bigger number of pages.
> > 
> > Could you check the max_sectors_kb of the underlying storage you are
> > using in Dom0?
> > 
> > Roger.
> > 
> I checked the value for the loop device as well
> 
> With 4.4.77 (bad write performance)
> $ cat /sys/block/sda/queue/max_sectors_kb
> 1280
> $ cat /sys/block/loop1/queue/max_sectors_kb
> 127
> 
> 
> With 4.1.42 (normal write performance)
> $ cat /sys/block/sda/queue/max_sectors_kb
> 4096
> $ cat /sys/block/loop1/queue/max_sectors_kb
> 127

Thank you for the confirmations so far. Could you confirm performance dom0
running 4.4.77 with domU running 4.1.42, and the other way around? Would
like to verify if this is just isolated to blkfront.

Re: [Xen-users] File-based domU - Slow storage write since xen 4.8

[Benoit Depail]

On 07/26/17 00:25, Keith Busch wrote:
> On Fri, Jul 21, 2017 at 07:07:06PM +0200, Benoit Depail wrote:
>> On 07/21/17 18:07, Roger Pau Monné wrote:
>>>
>>> Hm, I'm not sure I follow either. AFAIK this problem came from
>>> changing the Linux version in the Dom0 (where the backend, blkback is
>>> running), rather than in the DomU right?
>>>
>>> Regarding the queue/sectors stuff, blkfront uses several blk_queue
>>> functions to set those parameters, maybe there's something wrong
>>> there, but I cannot really spot what it is:
>>>
>>> http://elixir.free-electrons.com/linux/latest/source/drivers/block/xen-blkfront.c#L929
>>>
>>> In the past the number of pages that could fit in a single ring
>>> request was limited to 11, but some time ago indirect descriptors
>>> where introduced in order to lift this limit, and now requests can
>>> have a much bigger number of pages.
>>>
>>> Could you check the max_sectors_kb of the underlying storage you are
>>> using in Dom0?
>>>
>>> Roger.
>>>
>> I checked the value for the loop device as well
>>
>> With 4.4.77 (bad write performance)
>> $ cat /sys/block/sda/queue/max_sectors_kb
>> 1280
>> $ cat /sys/block/loop1/queue/max_sectors_kb
>> 127
>>
>>
>> With 4.1.42 (normal write performance)
>> $ cat /sys/block/sda/queue/max_sectors_kb
>> 4096
>> $ cat /sys/block/loop1/queue/max_sectors_kb
>> 127
> 
> Thank you for the confirmations so far. Could you confirm performance dom0
> running 4.4.77 with domU running 4.1.42, and the other way around? Would
> like to verify if this is just isolated to blkfront.
> 
Hi,

I've ran the tests, and I can tell that the domU kernel version have no
influence on the performance.

Dom0 with 4.4.77 always shows bad performance, wether the domU runs
4.1.42 or 4.4.77.

Dom0 with 4.1.42 always shows good performance, wether the domU runs
4.1.42 or 4.4.77.

Thanks,

-- 
Benoit Depail
Senior Infrastructures Architect
NBS System

Re: [Xen-users] File-based domU - Slow storage write since xen 4.8

[Roger Pau Monné]

On Fri, Jul 28, 2017 at 04:50:27PM +0200, Benoit Depail wrote:
> On 07/26/17 00:25, Keith Busch wrote:
> > On Fri, Jul 21, 2017 at 07:07:06PM +0200, Benoit Depail wrote:
> >> On 07/21/17 18:07, Roger Pau Monn� wrote:
> >>>
> >>> Hm, I'm not sure I follow either. AFAIK this problem came from
> >>> changing the Linux version in the Dom0 (where the backend, blkback is
> >>> running), rather than in the DomU right?
> >>>
> >>> Regarding the queue/sectors stuff, blkfront uses several blk_queue
> >>> functions to set those parameters, maybe there's something wrong
> >>> there, but I cannot really spot what it is:
> >>>
> >>> http://elixir.free-electrons.com/linux/latest/source/drivers/block/xen-blkfront.c#L929
> >>>
> >>> In the past the number of pages that could fit in a single ring
> >>> request was limited to 11, but some time ago indirect descriptors
> >>> where introduced in order to lift this limit, and now requests can
> >>> have a much bigger number of pages.
> >>>
> >>> Could you check the max_sectors_kb of the underlying storage you are
> >>> using in Dom0?
> >>>
> >>> Roger.
> >>>
> >> I checked the value for the loop device as well
> >>
> >> With 4.4.77 (bad write performance)
> >> $ cat /sys/block/sda/queue/max_sectors_kb
> >> 1280
> >> $ cat /sys/block/loop1/queue/max_sectors_kb
> >> 127
> >>
> >>
> >> With 4.1.42 (normal write performance)
> >> $ cat /sys/block/sda/queue/max_sectors_kb
> >> 4096
> >> $ cat /sys/block/loop1/queue/max_sectors_kb
> >> 127
> > 
> > Thank you for the confirmations so far. Could you confirm performance dom0
> > running 4.4.77 with domU running 4.1.42, and the other way around? Would
> > like to verify if this is just isolated to blkfront.
> > 
> Hi,
> 
> I've ran the tests, and I can tell that the domU kernel version have no
> influence on the performance.
> 
> Dom0 with 4.4.77 always shows bad performance, wether the domU runs
> 4.1.42 or 4.4.77.
> 
> Dom0 with 4.1.42 always shows good performance, wether the domU runs
> 4.1.42 or 4.4.77.

Hello,

I haven't yet got time to look into this sadly. Can you please try to
use fio [0] in order to run the tests against the loop device in Dom0?

If possible, could you test several combinations of block sizes, I/O
sizes and I/O depths?

Thanks, Roger.

[0] http://git.kernel.dk/?p=fio.git;a=summary

Re: [Xen-users] File-based domU - Slow storage write since xen 4.8

[Benoit Depail]

On 08/01/2017 11:48 AM, Roger Pau Monné wrote:
> On Fri, Jul 28, 2017 at 04:50:27PM +0200, Benoit Depail wrote:
>> On 07/26/17 00:25, Keith Busch wrote:
>>> On Fri, Jul 21, 2017 at 07:07:06PM +0200, Benoit Depail wrote:
>>>> On 07/21/17 18:07, Roger Pau Monné wrote:
>>>>>
>>>>> Hm, I'm not sure I follow either. AFAIK this problem came from
>>>>> changing the Linux version in the Dom0 (where the backend, blkback is
>>>>> running), rather than in the DomU right?
>>>>>
>>>>> Regarding the queue/sectors stuff, blkfront uses several blk_queue
>>>>> functions to set those parameters, maybe there's something wrong
>>>>> there, but I cannot really spot what it is:
>>>>>
>>>>> http://elixir.free-electrons.com/linux/latest/source/drivers/block/xen-blkfront.c#L929
>>>>>
>>>>> In the past the number of pages that could fit in a single ring
>>>>> request was limited to 11, but some time ago indirect descriptors
>>>>> where introduced in order to lift this limit, and now requests can
>>>>> have a much bigger number of pages.
>>>>>
>>>>> Could you check the max_sectors_kb of the underlying storage you are
>>>>> using in Dom0?
>>>>>
>>>>> Roger.
>>>>>
>>>> I checked the value for the loop device as well
>>>>
>>>> With 4.4.77 (bad write performance)
>>>> $ cat /sys/block/sda/queue/max_sectors_kb
>>>> 1280
>>>> $ cat /sys/block/loop1/queue/max_sectors_kb
>>>> 127
>>>>
>>>>
>>>> With 4.1.42 (normal write performance)
>>>> $ cat /sys/block/sda/queue/max_sectors_kb
>>>> 4096
>>>> $ cat /sys/block/loop1/queue/max_sectors_kb
>>>> 127
>>>
>>> Thank you for the confirmations so far. Could you confirm performance dom0
>>> running 4.4.77 with domU running 4.1.42, and the other way around? Would
>>> like to verify if this is just isolated to blkfront.
>>>
>> Hi,
>>
>> I've ran the tests, and I can tell that the domU kernel version have no
>> influence on the performance.
>>
>> Dom0 with 4.4.77 always shows bad performance, wether the domU runs
>> 4.1.42 or 4.4.77.
>>
>> Dom0 with 4.1.42 always shows good performance, wether the domU runs
>> 4.1.42 or 4.4.77.
> 
> Hello,
> 
> I haven't yet got time to look into this sadly. Can you please try to
> use fio [0] in order to run the tests against the loop device in Dom0?
> 
> If possible, could you test several combinations of block sizes, I/O
> sizes and I/O depths?
> 
> Thanks, Roger.
> 
> [0] http://git.kernel.dk/?p=fio.git;a=summary
> 

Ok I'll give a try later when I have more time. Probably next week.

Thanks,

-- 
Benoit Depail
Senior Infrastructures Architect
NBS System

Re: [Xen-users] File-based domU - Slow storage write since xen 4.8

[Benoit Depail]

On 08/02/17 18:29, Benoit Depail wrote:
> On 08/01/2017 11:48 AM, Roger Pau Monné wrote:
>> On Fri, Jul 28, 2017 at 04:50:27PM +0200, Benoit Depail wrote:
>>> On 07/26/17 00:25, Keith Busch wrote:
>>>> On Fri, Jul 21, 2017 at 07:07:06PM +0200, Benoit Depail wrote:
>>>>> On 07/21/17 18:07, Roger Pau Monné wrote:
>>>>>>
>>>>>> Hm, I'm not sure I follow either. AFAIK this problem came from
>>>>>> changing the Linux version in the Dom0 (where the backend, blkback is
>>>>>> running), rather than in the DomU right?
>>>>>>
>>>>>> Regarding the queue/sectors stuff, blkfront uses several blk_queue
>>>>>> functions to set those parameters, maybe there's something wrong
>>>>>> there, but I cannot really spot what it is:
>>>>>>
>>>>>> http://elixir.free-electrons.com/linux/latest/source/drivers/block/xen-blkfront.c#L929
>>>>>>
>>>>>> In the past the number of pages that could fit in a single ring
>>>>>> request was limited to 11, but some time ago indirect descriptors
>>>>>> where introduced in order to lift this limit, and now requests can
>>>>>> have a much bigger number of pages.
>>>>>>
>>>>>> Could you check the max_sectors_kb of the underlying storage you are
>>>>>> using in Dom0?
>>>>>>
>>>>>> Roger.
>>>>>>
>>>>> I checked the value for the loop device as well
>>>>>
>>>>> With 4.4.77 (bad write performance)
>>>>> $ cat /sys/block/sda/queue/max_sectors_kb
>>>>> 1280
>>>>> $ cat /sys/block/loop1/queue/max_sectors_kb
>>>>> 127
>>>>>
>>>>>
>>>>> With 4.1.42 (normal write performance)
>>>>> $ cat /sys/block/sda/queue/max_sectors_kb
>>>>> 4096
>>>>> $ cat /sys/block/loop1/queue/max_sectors_kb
>>>>> 127
>>>>
>>>> Thank you for the confirmations so far. Could you confirm performance dom0
>>>> running 4.4.77 with domU running 4.1.42, and the other way around? Would
>>>> like to verify if this is just isolated to blkfront.
>>>>
>>> Hi,
>>>
>>> I've ran the tests, and I can tell that the domU kernel version have no
>>> influence on the performance.
>>>
>>> Dom0 with 4.4.77 always shows bad performance, wether the domU runs
>>> 4.1.42 or 4.4.77.
>>>
>>> Dom0 with 4.1.42 always shows good performance, wether the domU runs
>>> 4.1.42 or 4.4.77.
>>
>> Hello,
>>
>> I haven't yet got time to look into this sadly. Can you please try to
>> use fio [0] in order to run the tests against the loop device in Dom0?
>>
>> If possible, could you test several combinations of block sizes, I/O
>> sizes and I/O depths?
>>
>> Thanks, Roger.
>>
>> [0] http://git.kernel.dk/?p=fio.git;a=summary
>>
> 
> Ok I'll give a try later when I have more time. Probably next week.
> 
> Thanks,
> 

Hi,

I had some time to play around with fio. I am not really sure what I was
supposed to do with it so I made a wild guess.

Using the setup showing bad write performances (dom0 with v4.4.77), I
ran a fio job with the following parameters :

[global]
size=5g
filename=<device>
direct=1
readwrite=write

[job1]
runtime=1m


I then used a loop to change the blocksize from 512 to 8192, steps 512,
and the io depth from 1 to 32, steps 4.

On the loop device (from the dom0), write speed was about 34MB/s most of
the time, reaching 110MB/s with a blocksize multiple of 4096 (4096 and
8192).

On the domU, write speed was the same, but reaching only 52MB/s when the
blocksize is a multiple of 4096.

Io depth did not show any significant change.

Feel free to suggest any improvment on my bench suit.

Thansk,

-- 
Benoit Depail
Senior Infrastructures Architect
NBS System