From: Christopher Baines <mail@cbaines.net>
To: "Austin S. Hemmelgarn" <ahferroin7@gmail.com>
Cc: linux-btrfs@vger.kernel.org
Subject: Re: Slow performance with Btrfs RAID 10 with a failed disk
Date: Mon, 02 Dec 2019 16:08:27 +0100 [thread overview]
Message-ID: <87a78alq8k.fsf@cbaines.net> (raw)
In-Reply-To: <1ac24ca2-4f78-13a8-0b06-8970e8ba6e17@gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2461 bytes --]
Austin S. Hemmelgarn <ahferroin7@gmail.com> writes:
> On 2019-11-27 03:36, Christopher Baines wrote:
>> Hey,
>>
>> I'm using RAID 10, and one of the disks has recently failed [1], and I'm
>> seeing plenty of warning and errors in the dmesg output [2].
>>
>> What kind of performance should be expected from Btrfs when a disk has
>> failed? [3] At the moment, the system seems very slow. One contributing
>> factor may be that all the logging that Btrfs is generating is being
>> written to the btrfs filesystem that's degraded, probably causing more
>> log messages to be produced.
>>
>> I guess that replacing the failed disk is the long term solution to get
>> the filesystem back in to proper operation, but is there anything else
>> that can be done to get it back operating until then?
>>
>> Also, is there anything that can stop btrfs logging so much about the
>> failures, now that I know that a disk has failed?
>
> You can solve both problems by replacing the disc, or if possible,
> just removing it from the array. You should, in theory, be able to
> convert to regular raid1 and then remove the failed disc, though it
> will likely take a while. Given your output below, I'd actually drop
> /dev/sdb as well, and look at replacing both with a single 1TB disc
> like your other three.
>
> The issue here is that BTRFS doesn't see the disc as failed, so it
> keeps trying to access it. That's what's slowing things down (because
> it eventually times out on the access attempt) and why it's logging so
> much (because BTRFS logs every IO error it encounters (like it
> should)).
Thanks for the tips :)
I've now remounted the filesystem with the degraded flag.
However, I haven't managed to remove the disk from the array yet.
$ sudo btrfs filesystem show /
Label: none uuid: 620115c7-89c7-4d79-a0bb-4957057d9991
Total devices 6 FS bytes used 1.08TiB
devid 1 size 72.70GiB used 72.70GiB path /dev/sda3
devid 2 size 72.70GiB used 72.70GiB path /dev/sdb3
devid 3 size 931.48GiB used 530.73GiB path /dev/sdc
devid 4 size 931.48GiB used 530.73GiB path /dev/sdd
devid 5 size 931.48GiB used 530.73GiB path /dev/sde
*** Some devices missing
$ sudo btrfs device delete missing /
ERROR: error removing device 'missing': no missing devices found to remove
So Btrfs knows at some level that a device is missing, from the output
of the first command, but it won't delete the missing device.
Am I missing something?
Thanks,
Chris
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 962 bytes --]
prev parent reply other threads:[~2019-12-02 15:08 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-11-27 8:36 Slow performance with Btrfs RAID 10 with a failed disk Christopher Baines
2019-11-27 14:19 ` Austin S. Hemmelgarn
2019-12-02 15:08 ` Christopher Baines [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87a78alq8k.fsf@cbaines.net \
--to=mail@cbaines.net \
--cc=ahferroin7@gmail.com \
--cc=linux-btrfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).