Scrub priority, am I using it wrong?

Scrub priority, am I using it wrong?

[Gareth Pye]

I've got a btrfs file system set up on 6 drbd disks running on 2Tb
spinning disks. The server is moderately loaded with various regular
tasks that use a fair bit of disk IO, but I've scheduled my weekly
btrfs scrub for the best quiet time in the week.

The command that is run is:
/usr/local/bin/btrfs scrub start -Bd -c idle /data

Which is my best attempt to try and get it to have a low impact on
user operations

But iotop shows me:

1765 be/4 root       14.84 M/s    0.00 B/s  0.00 % 96.65 % btrfs scrub
start -Bd -c idle /data
 1767 be/4 root       14.70 M/s    0.00 B/s  0.00 % 95.35 % btrfs
scrub start -Bd -c idle /data
 1768 be/4 root       13.47 M/s    0.00 B/s  0.00 % 92.59 % btrfs
scrub start -Bd -c idle /data
 1764 be/4 root       12.61 M/s    0.00 B/s  0.00 % 88.77 % btrfs
scrub start -Bd -c idle /data
 1766 be/4 root       11.24 M/s    0.00 B/s  0.00 % 85.18 % btrfs
scrub start -Bd -c idle /data
 1763 be/4 root        7.79 M/s    0.00 B/s  0.00 % 63.30 % btrfs
scrub start -Bd -c idle /data
28858 be/4 root        0.00 B/s  810.50 B/s  0.00 % 61.32 % [kworker/u16:25]


Which doesn't look like an idle priority to me. And the system sure
feels like a system with a lot of heavy io going on. Is there
something I'm doing wrong?

System details:

# uname -a
Linux emile 4.4.3-040403-generic #201602251634 SMP Thu Feb 25 21:36:25
UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

# /usr/local/bin/btrfs --version
btrfs-progs v4.4.1

I'm waiting on the ppa version of 4.5.1 before upgrading, that is my
usual kernel update strategy.

# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.4 LTS"

Any other details that people would like to see that are relevant to
this question?

-- 
Gareth Pye - blog.cerberos.id.au
Level 2 MTG Judge, Melbourne, Australia

Re: Scrub priority, am I using it wrong?

[Duncan]

Gareth Pye posted on Tue, 05 Apr 2016 09:36:48 +1000 as excerpted:

> I've got a btrfs file system set up on 6 drbd disks running on 2Tb
> spinning disks. The server is moderately loaded with various regular
> tasks that use a fair bit of disk IO, but I've scheduled my weekly btrfs
> scrub for the best quiet time in the week.
> 
> The command that is run is:
> /usr/local/bin/btrfs scrub start -Bd -c idle /data
> 
> Which is my best attempt to try and get it to have a low impact on user
> operations
> 
> But iotop shows me:
> 
> 1765 be/4 root       14.84 M/s    0.00 B/s  0.00 % 96.65 % btrfs scrub
> start -Bd -c idle /data
>  1767 be/4 root       14.70 M/s    0.00 B/s  0.00 % 95.35 % btrfs
> scrub start -Bd -c idle /data
>  1768 be/4 root       13.47 M/s    0.00 B/s  0.00 % 92.59 % btrfs
> scrub start -Bd -c idle /data
>  1764 be/4 root       12.61 M/s    0.00 B/s  0.00 % 88.77 % btrfs
> scrub start -Bd -c idle /data
>  1766 be/4 root       11.24 M/s    0.00 B/s  0.00 % 85.18 % btrfs
> scrub start -Bd -c idle /data
>  1763 be/4 root        7.79 M/s    0.00 B/s  0.00 % 63.30 % btrfs
> scrub start -Bd -c idle /data
> 28858 be/4 root        0.00 B/s  810.50 B/s  0.00 % 61.32 % [kworker/
u16:25]
> 
> 
> Which doesn't look like an idle priority to me. And the system sure
> feels like a system with a lot of heavy io going on. Is there something
> I'm doing wrong?

Two points:

1) It appears btrfs scrub start's -c option only takes numeric class, so 
try -c3 instead of -c idle.

Works for me with the numeric class (same results as you with spelled out 
class), tho I'm on ssd with multiple independent btrfs on partitions, the 
biggest of which is 24 GiB, 18.something GiB used, which scrubs in all of 
20 seconds, so I don't need and hadn't tried the -c option at all until 
now. 

2) What a difference an ssd makes!

$$ sudo btrfs scrub start -c3 /p
scrub started on /p, [...]

$$ sudo iotop -obn1
Total DISK READ :     626.53 M/s | Total DISK WRITE :       0.00 B/s
Actual DISK READ:     596.93 M/s | Actual DISK WRITE:       0.00 B/s
 TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
 872 idle root      268.40 M/s    0.00 B/s  0.00 %  0.00 % btrfs scrub 
start -c3 /p
 873 idle root      358.13 M/s    0.00 B/s  0.00 %  0.00 % btrfs scrub 
start -c3 /p

CPU bound, 0% IOWait even at idle IO priority, in addition to the 
hundreds of M/s values per thread/device, here.  You OTOH are showing 
under 20 M/s per thread/device on spinning rust, with an IOWait near 90%, 
thus making it IO bound.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Scrub priority, am I using it wrong?

[Gareth Pye]

On Tue, Apr 5, 2016 at 12:37 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> 1) It appears btrfs scrub start's -c option only takes numeric class, so
> try -c3 instead of -c idle.


Does it count as a bug if it silently accepts the way I was doing it?

I've switched to -c3 and at least now the idle class listed in iotop
is idle, so I hope that means it will be more friendly to other
processes.

-- 
Gareth Pye - blog.cerberos.id.au
Level 2 MTG Judge, Melbourne, Australia

Re: Scrub priority, am I using it wrong?

[Gareth Pye]

On Tue, Apr 5, 2016 at 12:37 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> CPU bound, 0% IOWait even at idle IO priority, in addition to the
> hundreds of M/s values per thread/device, here.  You OTOH are showing
> under 20 M/s per thread/device on spinning rust, with an IOWait near 90%,
> thus making it IO bound.


And yes I'd love to switch to SSD, but 12 2TB drives is a bit pricey still

-- 
Gareth Pye - blog.cerberos.id.au
Level 2 MTG Judge, Melbourne, Australia

Re: Scrub priority, am I using it wrong?

[Duncan]

Gareth Pye posted on Tue, 05 Apr 2016 13:44:05 +1000 as excerpted:

> On Tue, Apr 5, 2016 at 12:37 PM, Duncan <1i5t5.duncan@cox.net> wrote:
>> 1) It appears btrfs scrub start's -c option only takes numeric class,
>> so try -c3 instead of -c idle.
> 
> 
> Does it count as a bug if it silently accepts the way I was doing it?
> 
> I've switched to -c3 and at least now the idle class listed in iotop is
> idle, so I hope that means it will be more friendly to other processes.

I'd say yes, particularly given that the value must be the numeric class 
isn't documented in the manpage at all.

Whether the bug is then one of documentation (say it must be numeric) or 
of implementation (take the class name as well) is then up for debate.  
I'd call fixing either one a fix.  If it must be numeric, document that 
(and optionally change the implementation to error in some way if a 
numeric parameter isn't supplied for -c), else change the implementation 
so the name can be taken as well (and optionally change the documentation 
to explicitly mention that either one can be used).  Doesn't matter to me 
which.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Scrub priority, am I using it wrong?

[Duncan]

Gareth Pye posted on Tue, 05 Apr 2016 13:45:11 +1000 as excerpted:

> On Tue, Apr 5, 2016 at 12:37 PM, Duncan <1i5t5.duncan@cox.net> wrote:
>> CPU bound, 0% IOWait even at idle IO priority, in addition to the
>> hundreds of M/s values per thread/device, here.  You OTOH are showing
>> under 20 M/s per thread/device on spinning rust, with an IOWait near
>> 90%,
>> thus making it IO bound.
> 
> 
> And yes I'd love to switch to SSD, but 12 2TB drives is a bit pricey
> still

No kidding.  That's why my media partition remains spinning rust.  (Tho 
FWIW, not btrfs, I use btrfs only on my ssds, and still use the old and 
stable reiserfs on my spinning rust.)

But my media partition is small enough, and ssd prices now low enough up 
to the 1 TB level, that when I upgrade I'll probably switch to ssd for 
the media partition as well, and leave spinning rust only as second or 
third level backups.

But that's because it all, including first level backups, fits in under a 
TB (and if pressed I could do it under a half TB).  Multi-TB, as you 
have, definitely still spinning rust, for me too.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Scrub priority, am I using it wrong?

[Austin S. Hemmelgarn]

On 2016-04-05 00:19, Duncan wrote:
> Gareth Pye posted on Tue, 05 Apr 2016 13:44:05 +1000 as excerpted:
>
>> On Tue, Apr 5, 2016 at 12:37 PM, Duncan <1i5t5.duncan@cox.net> wrote:
>>> 1) It appears btrfs scrub start's -c option only takes numeric class,
>>> so try -c3 instead of -c idle.
>>
>>
>> Does it count as a bug if it silently accepts the way I was doing it?
>>
>> I've switched to -c3 and at least now the idle class listed in iotop is
>> idle, so I hope that means it will be more friendly to other processes.
>
> I'd say yes, particularly given that the value must be the numeric class
> isn't documented in the manpage at all.
>
> Whether the bug is then one of documentation (say it must be numeric) or
> of implementation (take the class name as well) is then up for debate.
> I'd call fixing either one a fix.  If it must be numeric, document that
> (and optionally change the implementation to error in some way if a
> numeric parameter isn't supplied for -c), else change the implementation
> so the name can be taken as well (and optionally change the documentation
> to explicitly mention that either one can be used).  Doesn't matter to me
> which.
>
In general, I'd say:
1. The documentation needs to be updated.  Even if it's intended to 
accept class names eventually, it should match the implementation.
2. The implementation needs error checking added.  At the very least, it 
should say something to the effect of "Hey, this doesn't work the way 
you appear to think it does.", and possibly bail.

That said, I don't think there's a huge amount of value in adding the 
ability to accept symbolic names.  Nothing that I know of accepts them 
on the command line, because most things pass the integer directly to 
the syscall, thus allowing for people to use new classes (if any are 
ever introduced) with old tools.

Re: Scrub priority, am I using it wrong?

[Henk Slager]

On Tue, Apr 5, 2016 at 4:37 AM, Duncan <1i5t5.duncan@cox.net> wrote:
> Gareth Pye posted on Tue, 05 Apr 2016 09:36:48 +1000 as excerpted:
>
>> I've got a btrfs file system set up on 6 drbd disks running on 2Tb
>> spinning disks. The server is moderately loaded with various regular
>> tasks that use a fair bit of disk IO, but I've scheduled my weekly btrfs
>> scrub for the best quiet time in the week.
>>
>> The command that is run is:
>> /usr/local/bin/btrfs scrub start -Bd -c idle /data
>>
>> Which is my best attempt to try and get it to have a low impact on user
>> operations
>>
>> But iotop shows me:
>>
>> 1765 be/4 root       14.84 M/s    0.00 B/s  0.00 % 96.65 % btrfs scrub
>> start -Bd -c idle /data
>>  1767 be/4 root       14.70 M/s    0.00 B/s  0.00 % 95.35 % btrfs
>> scrub start -Bd -c idle /data
>>  1768 be/4 root       13.47 M/s    0.00 B/s  0.00 % 92.59 % btrfs
>> scrub start -Bd -c idle /data
>>  1764 be/4 root       12.61 M/s    0.00 B/s  0.00 % 88.77 % btrfs
>> scrub start -Bd -c idle /data
>>  1766 be/4 root       11.24 M/s    0.00 B/s  0.00 % 85.18 % btrfs
>> scrub start -Bd -c idle /data
>>  1763 be/4 root        7.79 M/s    0.00 B/s  0.00 % 63.30 % btrfs
>> scrub start -Bd -c idle /data
>> 28858 be/4 root        0.00 B/s  810.50 B/s  0.00 % 61.32 % [kworker/
> u16:25]
>>
>>
>> Which doesn't look like an idle priority to me. And the system sure
>> feels like a system with a lot of heavy io going on. Is there something
>> I'm doing wrong?

When I see the throughput numbers, it lets me think that the
filesystem is raid5 or raid6. On single or raid1 or raid10 one easily
gets around 100M/s without the notice/feeling of heavy IO ongoing,
mostly independent of scrub options.

> Two points:
>
> 1) It appears btrfs scrub start's -c option only takes numeric class, so
> try -c3 instead of -c idle.

Thanks to Duncan for pointing this out. I don't remember exactly, but
I think I also had issues with this in the past, but did not realize
or have a further look at it.

> Works for me with the numeric class (same results as you with spelled out
> class), tho I'm on ssd with multiple independent btrfs on partitions, the
> biggest of which is 24 GiB, 18.something GiB used, which scrubs in all of
> 20 seconds, so I don't need and hadn't tried the -c option at all until
> now.
>
> 2) What a difference an ssd makes!
>
> $$ sudo btrfs scrub start -c3 /p
> scrub started on /p, [...]
>
> $$ sudo iotop -obn1
> Total DISK READ :     626.53 M/s | Total DISK WRITE :       0.00 B/s
> Actual DISK READ:     596.93 M/s | Actual DISK WRITE:       0.00 B/s
>  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
>  872 idle root      268.40 M/s    0.00 B/s  0.00 %  0.00 % btrfs scrub
> start -c3 /p
>  873 idle root      358.13 M/s    0.00 B/s  0.00 %  0.00 % btrfs scrub
> start -c3 /p
>
> CPU bound, 0% IOWait even at idle IO priority, in addition to the
> hundreds of M/s values per thread/device, here.  You OTOH are showing
> under 20 M/s per thread/device on spinning rust, with an IOWait near 90%,
> thus making it IO bound.

This low M/s and high IOWait is the kind of behavior I noticed with 3x
2TB raid5 when scrubbing or balancing (no bcache activated, kernel
4.3.3).

Re: Scrub priority, am I using it wrong?

[Gareth Pye]

Yeah, RAID5. I'm now doing pause and resume on it to let it take
multiple nights, the idle should let other processes complete in
reasonable time.

On Wed, Apr 6, 2016 at 3:34 AM, Henk Slager <eye1tm@gmail.com> wrote:
> On Tue, Apr 5, 2016 at 4:37 AM, Duncan <1i5t5.duncan@cox.net> wrote:
>> Gareth Pye posted on Tue, 05 Apr 2016 09:36:48 +1000 as excerpted:
>>
>>> I've got a btrfs file system set up on 6 drbd disks running on 2Tb
>>> spinning disks. The server is moderately loaded with various regular
>>> tasks that use a fair bit of disk IO, but I've scheduled my weekly btrfs
>>> scrub for the best quiet time in the week.
>>>
>>> The command that is run is:
>>> /usr/local/bin/btrfs scrub start -Bd -c idle /data
>>>
>>> Which is my best attempt to try and get it to have a low impact on user
>>> operations
>>>
>>> But iotop shows me:
>>>
>>> 1765 be/4 root       14.84 M/s    0.00 B/s  0.00 % 96.65 % btrfs scrub
>>> start -Bd -c idle /data
>>>  1767 be/4 root       14.70 M/s    0.00 B/s  0.00 % 95.35 % btrfs
>>> scrub start -Bd -c idle /data
>>>  1768 be/4 root       13.47 M/s    0.00 B/s  0.00 % 92.59 % btrfs
>>> scrub start -Bd -c idle /data
>>>  1764 be/4 root       12.61 M/s    0.00 B/s  0.00 % 88.77 % btrfs
>>> scrub start -Bd -c idle /data
>>>  1766 be/4 root       11.24 M/s    0.00 B/s  0.00 % 85.18 % btrfs
>>> scrub start -Bd -c idle /data
>>>  1763 be/4 root        7.79 M/s    0.00 B/s  0.00 % 63.30 % btrfs
>>> scrub start -Bd -c idle /data
>>> 28858 be/4 root        0.00 B/s  810.50 B/s  0.00 % 61.32 % [kworker/
>> u16:25]
>>>
>>>
>>> Which doesn't look like an idle priority to me. And the system sure
>>> feels like a system with a lot of heavy io going on. Is there something
>>> I'm doing wrong?
>
> When I see the throughput numbers, it lets me think that the
> filesystem is raid5 or raid6. On single or raid1 or raid10 one easily
> gets around 100M/s without the notice/feeling of heavy IO ongoing,
> mostly independent of scrub options.
>
>> Two points:
>>
>> 1) It appears btrfs scrub start's -c option only takes numeric class, so
>> try -c3 instead of -c idle.
>
> Thanks to Duncan for pointing this out. I don't remember exactly, but
> I think I also had issues with this in the past, but did not realize
> or have a further look at it.
>
>> Works for me with the numeric class (same results as you with spelled out
>> class), tho I'm on ssd with multiple independent btrfs on partitions, the
>> biggest of which is 24 GiB, 18.something GiB used, which scrubs in all of
>> 20 seconds, so I don't need and hadn't tried the -c option at all until
>> now.
>>
>> 2) What a difference an ssd makes!
>>
>> $$ sudo btrfs scrub start -c3 /p
>> scrub started on /p, [...]
>>
>> $$ sudo iotop -obn1
>> Total DISK READ :     626.53 M/s | Total DISK WRITE :       0.00 B/s
>> Actual DISK READ:     596.93 M/s | Actual DISK WRITE:       0.00 B/s
>>  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
>>  872 idle root      268.40 M/s    0.00 B/s  0.00 %  0.00 % btrfs scrub
>> start -c3 /p
>>  873 idle root      358.13 M/s    0.00 B/s  0.00 %  0.00 % btrfs scrub
>> start -c3 /p
>>
>> CPU bound, 0% IOWait even at idle IO priority, in addition to the
>> hundreds of M/s values per thread/device, here.  You OTOH are showing
>> under 20 M/s per thread/device on spinning rust, with an IOWait near 90%,
>> thus making it IO bound.
>
> This low M/s and high IOWait is the kind of behavior I noticed with 3x
> 2TB raid5 when scrubbing or balancing (no bcache activated, kernel
> 4.3.3).
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Gareth Pye - blog.cerberos.id.au
Level 2 MTG Judge, Melbourne, Australia