How long should a btrfs scrub with RAID5/6 take?

How long should a btrfs scrub with RAID5/6 take?

[Robert Krig]

Hi, I've got a server where I have 4x8TB Disks in a BTRFS RAID5
(metadata and systemdata as RAID1) configuration.

It's just a backup server with data I can always recreate. 
This server is in the bedroom, so I send it to sleep/suspend when I go
to bed and then wake it up in the morning. 

Since a scrub takes days on such a setup, I issue a btrfs scrub resume
whenever the server wakes up again.

btrfs scrub status shows me that the total data to scrub is 18.67TB,
but it's already scrubbed 36.60TB. Is there any way I can calculate how
much more data is going to be scrubbed? 4x8TB is 32TB, so we're passed
that, but I'm guessing this also has to do with parity data as well.

Re: How long should a btrfs scrub with RAID5/6 take?

[Joshua]

I do not have a direct answer to your question, but I can chime in on your scrub times:

I have 4x8TB, 4x10TB, and 8x3TB drives, all in one huge BTRFS ‘RAID1’.

I can tell you that even in my setup, scrubs usually take less than a day.  So unless it’s the raid5 making yours take so long, that does not sound typical.

> On Jan 8, 2020, at 2:13 AM, Robert Krig <robert.krig@render-wahnsinn.de> wrote:
> 
> Hi, I've got a server where I have 4x8TB Disks in a BTRFS RAID5
> (metadata and systemdata as RAID1) configuration.
> 
> It's just a backup server with data I can always recreate. 
> This server is in the bedroom, so I send it to sleep/suspend when I go
> to bed and then wake it up in the morning. 
> 
> Since a scrub takes days on such a setup, I issue a btrfs scrub resume
> whenever the server wakes up again.
> 
> btrfs scrub status shows me that the total data to scrub is 18.67TB,
> but it's already scrubbed 36.60TB. Is there any way I can calculate how
> much more data is going to be scrubbed? 4x8TB is 32TB, so we're passed
> that, but I'm guessing this also has to do with parity data as well.
> 

Re: How long should a btrfs scrub with RAID5/6 take?

[Chris Murphy]

On Wed, Jan 8, 2020 at 3:19 AM Robert Krig
<robert.krig@render-wahnsinn.de> wrote:
>
> Hi, I've got a server where I have 4x8TB Disks in a BTRFS RAID5
> (metadata and systemdata as RAID1) configuration.
>
> It's just a backup server with data I can always recreate.
> This server is in the bedroom, so I send it to sleep/suspend when I go
> to bed and then wake it up in the morning.
>
> Since a scrub takes days on such a setup, I issue a btrfs scrub resume
> whenever the server wakes up again.
>
> btrfs scrub status shows me that the total data to scrub is 18.67TB,
> but it's already scrubbed 36.60TB. Is there any way I can calculate how
> much more data is going to be scrubbed? 4x8TB is 32TB, so we're passed
> that, but I'm guessing this also has to do with parity data as well.
>

What kernel version and btrfs-progs version? Recent btrfs progs has a
new output that shows more info including a time to completion
estimate. Can you post the output from 'btrfs scrub status
/mountpoint/' ?

-- 
Chris Murphy

Re: How long should a btrfs scrub with RAID5/6 take?

[Robert Krig]

I'm on Debian using Kernel 5.3.0-0.bpo.2-amd64 and btrfs-progs version
v5.2.1.

Here's an output of scrub status:
######################################
UUID:             c4adb372-178b-4398-977d-bc91bef6130f
        no stats available
Time left:        87884330:53:42
ETA:              Thu Oct 26 11:30:49 12045
Total to scrub:   18.67TiB
Bytes scrubbed:   35.70TiB
Rate:             55.60MiB/s
Error summary:    no errors found
#######################################


On Mi, 2020-01-08 at 14:49 -0700, Chris Murphy wrote:
> On Wed, Jan 8, 2020 at 3:19 AM Robert Krig
> <robert.krig@render-wahnsinn.de> wrote:
> > Hi, I've got a server where I have 4x8TB Disks in a BTRFS RAID5
> > (metadata and systemdata as RAID1) configuration.
> > 
> > It's just a backup server with data I can always recreate.
> > This server is in the bedroom, so I send it to sleep/suspend when I
> > go
> > to bed and then wake it up in the morning.
> > 
> > Since a scrub takes days on such a setup, I issue a btrfs scrub
> > resume
> > whenever the server wakes up again.
> > 
> > btrfs scrub status shows me that the total data to scrub is
> > 18.67TB,
> > but it's already scrubbed 36.60TB. Is there any way I can calculate
> > how
> > much more data is going to be scrubbed? 4x8TB is 32TB, so we're
> > passed
> > that, but I'm guessing this also has to do with parity data as
> > well.
> > 
> 
> What kernel version and btrfs-progs version? Recent btrfs progs has a
> new output that shows more info including a time to completion
> estimate. Can you post the output from 'btrfs scrub status
> /mountpoint/' ?
> 

Re: How long should a btrfs scrub with RAID5/6 take?

[Jorge Bastos]

I reported recently to the list there there's a performance problem
with scrubs on raid5/6 pools, initial report and followups here:

https://lore.kernel.org/linux-btrfs/CAHzMYBTXvY1VgcoFDUvc2NFmVKq2HJRHuS0VXzoneUMh79cySA@mail.gmail.com/

Quick recap:

4 disk raid5 (raid1 metadata)

UUID:             b75ee8b5-ae1c-4395-aa39-bebf10993057
Scrub started:    Wed Nov 27 07:32:46 2019
Status:           running
Duration:         7:34:50
Time left:        1:52:37
ETA:              Wed Nov 27 17:00:18 2019
Total to scrub:   1.20TiB
Bytes scrubbed:   982.05GiB
Rate:             36.85MiB/s
Error summary:    no errors found

11 disk raid5 (raid1 metadata)

UUID:             1236acc8-dbd5-41bd-bf3d-872a8fbbce49
Scrub started:    Sun Dec 15 11:23:45 2019
Status:           running
Duration:         0:51:34
Time left:        90:06:52
ETA:              Thu Dec 19 06:22:13 2019
Total to scrub:   18.10TiB
Bytes scrubbed:   175.12GiB
Rate:             57.96MiB/s
Error summary:    no errors found

Same 11 disk pool during a btrfs send to /dev/null confirming it's
capable of much better read speeds:

btrfs send /mnt/disks/Pics/T12_disk1_Pics_2019-12-15-0829/ | pv -bart
> /dev/null
At subvol /mnt/disks/Pics/T12_disk1_Pics_2019-12-15-0829/
 447GiB 0:13:19 [ 635MiB/s] [ 573MiB/s]

Current speed 635MiB/s, average speed so far 573MiB/s

Re: How long should a btrfs scrub with RAID5/6 take?

[Chris Murphy]

On Wed, Jan 8, 2020 at 11:38 PM Robert Krig
<robert.krig@render-wahnsinn.de> wrote:
>
> I'm on Debian using Kernel 5.3.0-0.bpo.2-amd64 and btrfs-progs version
> v5.2.1.
>
> Here's an output of scrub status:
> ######################################
> UUID:             c4adb372-178b-4398-977d-bc91bef6130f
>         no stats available
> Time left:        87884330:53:42
> ETA:              Thu Oct 26 11:30:49 12045
> Total to scrub:   18.67TiB
> Bytes scrubbed:   35.70TiB
> Rate:             55.60MiB/s
> Error summary:    no errors found
> #######################################

10,000 years from now. That's a bug! :D Quick look through btrfs-progs
git log, I'm pretty sure 5.2 has the current scrub related fixes. But
it's best if you can test btrfs-progs 5.4 (or even 5.4.1 if you have
to build it anyway). If it does reproduce still, it might be useful
for a developer to see the following output:

$ cd /sys/fs/btrfs/<fsuuid>/allocation/
$ grep -r .



-- 
Chris Murphy