linux-btrfs.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Nemo <me@captnemo.in>
To: linux-btrfs@vger.kernel.org
Subject: RAID1 device deletion feedback
Date: Sun, 24 Feb 2019 23:02:35 +0530	[thread overview]
Message-ID: <d1eac64c-056b-a95b-7890-875d313a7ab4@captnemo.in> (raw)

Hi,

I had a RAID1 disk failure recently, and a limitation in number of SATA
connectors meant I could not do a live replace. I'm still in the
progress of resolving the issue, but posting some feedback here on the
issues I faced and what could have helped.

**What all I did**

- remove the dying disk and mount in degraded mode. This didn't work,
but _didn't provide me a failure reason_
- swap the dead disk post boot with the new one. Tried a btrfs device
add, but it failed as well because of too many failing reads from the
dead disk (which wasn't even attached).
- Started a btrfs device delete, which finally pointed me to the root
cause:

`ERROR: error removing device '/dev/sdb1': No space left on device`

In short:

1. My 2.7TBx3 setup was storing 2.9TB of data
2. Removing any one the disks was not possible without losing some data

However, getting to this error message took me hours of effort, despite
having read the btrfs wiki.

**Feedback**

1. The `btrfs device delete` command should have failed immediately
instead of making me wait for ages.
2. A delete command should stop writes to a disk, unless avoidable. This
would have left some data on the disk, and a subsequent delete could
have cleaned it up much faster. It failed in the worst manner possible,
because all deletion progress reversed once it failed: it balanced back
to complete usage, causing more strain on a dying disk.
3. A degraded mount failure should provide some error message.

*Status*: I cleared up some disk space, and ran another delete which is
still ongoing. I don't have physical access to the system for a few more
days, so adding the new disk will take me some time.

---

## Debug Details


```
uname -a
Linux tatooine.captnemo.in 4.20.10-arch1-1-ARCH #1 SMP PREEMPT Fri Feb
15 17:49:06 UTC 2019 x86_64 GNU/Linux

btrfs --version
btrfs-progs v4.20.1
```

A few more details are on a reddit post[0] I made to help debug the
issue and a gist[1] has disk usage and filesystem details.

[0]:
https://www.reddit.com/r/archlinux/comments/asrlam/btrfscleaner_at_100_cpu_usage_on_raid1_setup/egwc047/
[1]: https://paste.ubuntu.com/p/GDNKNBqqFy/

             reply	other threads:[~2019-02-24 17:40 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-02-24 17:32 Nemo [this message]
2019-02-25  0:27 ` RAID1 device deletion feedback Qu Wenruo
2019-02-25 13:03 ` Austin S. Hemmelgarn

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=d1eac64c-056b-a95b-7890-875d313a7ab4@captnemo.in \
    --to=me@captnemo.in \
    --cc=linux-btrfs@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).