btrfs and iostat - how do I measure the live performance of my btrsf filesystems?

btrfs and iostat - how do I measure the live performance of my btrsf filesystems?

[G. Richard Bellamy]

I apologize if this has already been addressed, regardless of whether
or not it's a n00b question, I'm stumped and am looking for some
guidance.

Is iostat a viable measure of btrfs performance?
I've got the following environment:
---------------------------------------------------------------------
2014-08-04 11:56:10
root@a i ~ # uname -a
Linux a 3.14.15-1-lts #1 SMP Thu Jul 31 22:42:44 UTC 2014 x86_64 GNU/Linux
2014-08-04 13:05:44
root@a i ~ # btrfs --version
Btrfs v3.14.2-dirty
2014-08-04 13:05:44
root@a i ~ # btrfs fi show
Label: 'system'  uuid: 06acfd0f-314d-457f-ba00-f96bf5b94a02
Total devices 2 FS bytes used 20.68GiB
devid    1 size 476.94GiB used 30.03GiB path /dev/sda
devid    2 size 476.94GiB used 30.03GiB path /dev/sdb

Label: 'data'  uuid: 10cc711b-8c8c-457c-91b1-d8ea4ef4e8d8
Total devices 4 FS bytes used 1.02TiB
devid    1 size 558.91GiB used 558.91GiB path /dev/sdc
devid    2 size 558.91GiB used 558.89GiB path /dev/sdd
devid    3 size 558.91GiB used 558.89GiB path /dev/sde
devid    4 size 558.91GiB used 558.89GiB path /dev/sdf

Btrfs v3.14.2-dirty
---------------------------------------------------------------------
Here's my iostat output:
---------------------------------------------------------------------
2014-08-04 13:26:39
root@a i ~ # iostat
Linux 3.14.15-1-lts (eanna) 08/04/2014 _x86_64_ (24 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           6.93    0.34    2.82    0.40    0.00   89.50

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sdb              22.40        38.75       540.65     836356   11668224
sda              22.41        40.32       540.65     870084   11668224
sdg               0.02         0.14         0.00       2987         24
sdc              17.42       232.10       368.66    5009076    7956248
sdd              16.64       196.53       357.23    4241392    7709640
sdf              16.30       196.35       367.01    4237644    7920700
sde              17.12       232.16       355.58    5010384    7674092
sdh               0.01         0.22         0.00       4820          0

How do I boil those numbers down to something meaningful, so I can get
an idea of the throughput in situ?

Presumably the answer is based on the normal read/write arithmetic
associated with RAID:
* raid1 read is nx (no lower than 1x), write is 1x
* raid10 read is nx (no lower than (n/spans)x), write is (n/spans)x

But as a practical matter, how does one go from the formulae to iostat
output to a metric? For instance:
Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sdb              22.40        38.75       540.65     836356   11668224
sda              22.41        40.32       540.65     870084   11668224

Do I take this to mean that for my raid1 read throughput is possibly
38.75 + 40.32 = 79.07 kB_read/s, and write is (540.65 + 540.65)/2 =
540.65 kB_write/s?

Though raid5&6 are yet to be fully baked, I imagine the answer for
those levels can become quite complex.

Thanks for your time,
-rb

Re: btrfs and iostat - how do I measure the live performance of my btrsf filesystems?

[G. Richard Bellamy]

Um. Derp. Yeah, it's actually sd[defh].

Thanks for the continuing education.

On Fri, Aug 22, 2014 at 8:24 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> G. Richard Bellamy posted on Fri, 22 Aug 2014 14:36:22 -0700 as excerpted:
>
>> An interesting exercise saw me reading data from my RAID10 to a USB
>> device, which produced the following representative iostat:
>>
>> Linux 3.14.17-1-lts (eanna) 08/22/2014 _x86_64_ (24 CPU)
>>
>> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>>            3.53    0.00    0.50    2.83    0.00   93.14
>>
>> Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
>> sda               1.89         0.01         0.01        839        998
>> sdc               0.00         0.00         0.00          1          0
>> sdb               1.23         0.02         0.01       1254        998
>
>> sdi             175.40         0.00        20.26         39    1454881
>
>> sdd               0.26         0.01         0.00        827         58
>> sde              28.86        12.29         0.00     882447         61
>> sdf               0.00         0.00         0.00          1          0
>> sdh              25.25        12.29         0.00     882448         57
>> sdg               0.25         0.01         0.00        826         60
>>
>> /dev/sdi is the USB drive, and /dev/sd[defg] are the four devices in the
>> raid10 volume. I'm reading a large (1.1T) file from the raid10 volume
>> and writing it to the USB drive.
>>
>> You can see that there are approximately two drives from the raid10
>> which are being read from - I assume this corresponds to the two spans
>> (the 'no lower than the (n/spans)x' speed I mentioned in my original
>> post - and that they aggregate to 24.58MB/s reads. This corresponds to
>> the 20.26MB/s writes to the USB drive.
>>
>> The raid10 volume is only being used for this file operation, nothing
>> else is touching it but the kernel and btrfs.
>>
>> I'm curious how others would read this?
>
> Something's not adding up.  You say sd[defg] are the btrfs raid10, but
> it's sde and sdh that are getting the read traffic.  Are you sure sdh
> isn't part of the raid10 and one of sd[dfg] (perhaps f, seeing d and g
> appear to balance out leaving f the odd one out?) is?
>
> Assuming sdh is indeed part of the raid10, it makes sense, and the fact
> that only two of the four devices are being active read matches what's
> known about btrfs raid1/10 at this point -- it has a relatively dumb read
> allocation algorithm that was good enough for a first implementation but
> obviously isn't optimal, reads are allocated based on the last bit of the
> PID (or TID IDR which), so even/odd.  Since this is a single transfer
> process, all the activity is on one or the other, so it's reading from
> the two device wide stripe, but always from the same one of the two
> mirrors supporting each strip.
>
> If you had a second read process going on and it was the same even/odd
> pid, you'd be doubling up on the same two devices.  Only with a
> relatively even mix of even/odd pid reads will you see things even out
> across all four.  See what I mean about a "relatively dumb" not well
> optimized first implementation?
>
> As they say btrfs is stabilizing now, presumably one of these kernel
> cycles we'll see something better in terms of read mirror allocation
> algorithm, perhaps as part of N-way-mirroring, when that gets implemented
> (roadmapped for after raid5/6 is completed, it's two-way-mirroring only
> now, regardless of the number of devices).
>
> --
> Duncan - List replies preferred.   No HTML msgs.
> "Every nonfree program has a lord, a master --
> and if you use the program, he is your master."  Richard Stallman
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs and iostat - how do I measure the live performance of my btrsf filesystems?

[Duncan]

G. Richard Bellamy posted on Fri, 22 Aug 2014 14:36:22 -0700 as excerpted:

> An interesting exercise saw me reading data from my RAID10 to a USB
> device, which produced the following representative iostat:
> 
> Linux 3.14.17-1-lts (eanna) 08/22/2014 _x86_64_ (24 CPU)
> 
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>            3.53    0.00    0.50    2.83    0.00   93.14
> 
> Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
> sda               1.89         0.01         0.01        839        998
> sdc               0.00         0.00         0.00          1          0
> sdb               1.23         0.02         0.01       1254        998

> sdi             175.40         0.00        20.26         39    1454881

> sdd               0.26         0.01         0.00        827         58
> sde              28.86        12.29         0.00     882447         61
> sdf               0.00         0.00         0.00          1          0
> sdh              25.25        12.29         0.00     882448         57
> sdg               0.25         0.01         0.00        826         60
> 
> /dev/sdi is the USB drive, and /dev/sd[defg] are the four devices in the
> raid10 volume. I'm reading a large (1.1T) file from the raid10 volume
> and writing it to the USB drive.
> 
> You can see that there are approximately two drives from the raid10
> which are being read from - I assume this corresponds to the two spans
> (the 'no lower than the (n/spans)x' speed I mentioned in my original
> post - and that they aggregate to 24.58MB/s reads. This corresponds to
> the 20.26MB/s writes to the USB drive.
> 
> The raid10 volume is only being used for this file operation, nothing
> else is touching it but the kernel and btrfs.
> 
> I'm curious how others would read this?

Something's not adding up.  You say sd[defg] are the btrfs raid10, but 
it's sde and sdh that are getting the read traffic.  Are you sure sdh 
isn't part of the raid10 and one of sd[dfg] (perhaps f, seeing d and g 
appear to balance out leaving f the odd one out?) is?

Assuming sdh is indeed part of the raid10, it makes sense, and the fact 
that only two of the four devices are being active read matches what's 
known about btrfs raid1/10 at this point -- it has a relatively dumb read 
allocation algorithm that was good enough for a first implementation but 
obviously isn't optimal, reads are allocated based on the last bit of the 
PID (or TID IDR which), so even/odd.  Since this is a single transfer 
process, all the activity is on one or the other, so it's reading from 
the two device wide stripe, but always from the same one of the two 
mirrors supporting each strip.

If you had a second read process going on and it was the same even/odd 
pid, you'd be doubling up on the same two devices.  Only with a 
relatively even mix of even/odd pid reads will you see things even out 
across all four.  See what I mean about a "relatively dumb" not well 
optimized first implementation?

As they say btrfs is stabilizing now, presumably one of these kernel 
cycles we'll see something better in terms of read mirror allocation 
algorithm, perhaps as part of N-way-mirroring, when that gets implemented 
(roadmapped for after raid5/6 is completed, it's two-way-mirroring only 
now, regardless of the number of devices).

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: btrfs and iostat - how do I measure the live performance of my btrsf filesystems?

[G. Richard Bellamy]

An interesting exercise saw me reading data from my RAID10 to a USB
device, which produced the following representative iostat:

Linux 3.14.17-1-lts (eanna) 08/22/2014 _x86_64_ (24 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.53    0.00    0.50    2.83    0.00   93.14

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
sda               1.89         0.01         0.01        839        998
sdc               0.00         0.00         0.00          1          0
sdb               1.23         0.02         0.01       1254        998
sdi             175.40         0.00        20.26         39    1454881
sdd               0.26         0.01         0.00        827         58
sde              28.86        12.29         0.00     882447         61
sdf               0.00         0.00         0.00          1          0
sdh              25.25        12.29         0.00     882448         57
sdg               0.25         0.01         0.00        826         60

/dev/sdi is the USB drive, and /dev/sd[defg] are the four devices in
the raid10 volume. I'm reading a large (1.1T) file from the raid10
volume and writing it to the USB drive.

You can see that there are approximately two drives from the raid10
which are being read from - I assume this corresponds to the two spans
(the 'no lower than the (n/spans)x' speed I mentioned in my original
post - and that they aggregate to 24.58MB/s reads. This corresponds to
the 20.26MB/s writes to the USB drive.

The raid10 volume is only being used for this file operation, nothing
else is touching it but the kernel and btrfs.

I'm curious how others would read this?

On Tue, Aug 5, 2014 at 4:39 PM, Tomasz Chmielewski <mangoo@wpkg.org> wrote:
> On 2014-08-06 00:06, G. Richard Bellamy wrote:
>>
>> On Tue, Aug 5, 2014 at 7:14 AM, Tomasz Chmielewski <mangoo@wpkg.org>
>> wrote:
>>>
>>> Simple iostat won't give you meaningful live performance stats.
>>>
>>> You can combine it i.e. like below:
>>>
>>> iostat -x 1
>>> iostat -mx 1
>>> iostat -m 1
>>
>>
>> Thanks for the pointer about viewing the extended stats, and showing
>> them in MB rather than kB.
>>
>> Maybe I'm missing something here. I'm failing to see how adding those
>> additional stats helps me get meaningful throughput information for a
>> multi-device btrfs volume.
>
>
> Well you won't see an aggregate, but you will see individual device
> statistics with "1" at the end of iostat arguments (meaning, dump the stats
> every second).
>
>
>
> --
> Tomasz Chmielewski
> http://www.sslrack.com
>

Re: btrfs and iostat - how do I measure the live performance of my btrsf filesystems?

[Tomasz Chmielewski]

On 2014-08-06 00:06, G. Richard Bellamy wrote:
> On Tue, Aug 5, 2014 at 7:14 AM, Tomasz Chmielewski <mangoo@wpkg.org> 
> wrote:
>> Simple iostat won't give you meaningful live performance stats.
>> 
>> You can combine it i.e. like below:
>> 
>> iostat -x 1
>> iostat -mx 1
>> iostat -m 1
> 
> Thanks for the pointer about viewing the extended stats, and showing
> them in MB rather than kB.
> 
> Maybe I'm missing something here. I'm failing to see how adding those
> additional stats helps me get meaningful throughput information for a
> multi-device btrfs volume.

Well you won't see an aggregate, but you will see individual device 
statistics with "1" at the end of iostat arguments (meaning, dump the 
stats every second).


-- 
Tomasz Chmielewski
http://www.sslrack.com

Re: btrfs and iostat - how do I measure the live performance of my btrsf filesystems?

[G. Richard Bellamy]

On Tue, Aug 5, 2014 at 7:14 AM, Tomasz Chmielewski <mangoo@wpkg.org> wrote:
> Simple iostat won't give you meaningful live performance stats.
>
> You can combine it i.e. like below:
>
> iostat -x 1
> iostat -mx 1
> iostat -m 1

Thanks for the pointer about viewing the extended stats, and showing
them in MB rather than kB.

Maybe I'm missing something here. I'm failing to see how adding those
additional stats helps me get meaningful throughput information for a
multi-device btrfs volume.

-rb

Re: btrfs and iostat - how do I measure the live performance of my btrsf filesystems?

[Tomasz Chmielewski]

> root@a i ~ # iostat
> Linux 3.14.15-1-lts (eanna) 08/04/2014 _x86_64_ (24 CPU)
> 
> (...)
> 
> How do I boil those numbers down to something meaningful, so I can get
> an idea of the throughput in situ?

Simple iostat won't give you meaningful live performance stats.

You can combine it i.e. like below:

iostat -x 1
iostat -mx 1
iostat -m 1

etc.



-- 
Tomasz Chmielewski
http://www.sslrack.com