RAID1 device deletion feedback

RAID1 device deletion feedback

[Nemo]

Hi,

I had a RAID1 disk failure recently, and a limitation in number of SATA
connectors meant I could not do a live replace. I'm still in the
progress of resolving the issue, but posting some feedback here on the
issues I faced and what could have helped.

**What all I did**

- remove the dying disk and mount in degraded mode. This didn't work,
but _didn't provide me a failure reason_
- swap the dead disk post boot with the new one. Tried a btrfs device
add, but it failed as well because of too many failing reads from the
dead disk (which wasn't even attached).
- Started a btrfs device delete, which finally pointed me to the root
cause:

`ERROR: error removing device '/dev/sdb1': No space left on device`

In short:

1. My 2.7TBx3 setup was storing 2.9TB of data
2. Removing any one the disks was not possible without losing some data

However, getting to this error message took me hours of effort, despite
having read the btrfs wiki.

**Feedback**

1. The `btrfs device delete` command should have failed immediately
instead of making me wait for ages.
2. A delete command should stop writes to a disk, unless avoidable. This
would have left some data on the disk, and a subsequent delete could
have cleaned it up much faster. It failed in the worst manner possible,
because all deletion progress reversed once it failed: it balanced back
to complete usage, causing more strain on a dying disk.
3. A degraded mount failure should provide some error message.

*Status*: I cleared up some disk space, and ran another delete which is
still ongoing. I don't have physical access to the system for a few more
days, so adding the new disk will take me some time.

---

## Debug Details


```
uname -a
Linux tatooine.captnemo.in 4.20.10-arch1-1-ARCH #1 SMP PREEMPT Fri Feb
15 17:49:06 UTC 2019 x86_64 GNU/Linux

btrfs --version
btrfs-progs v4.20.1
```

A few more details are on a reddit post[0] I made to help debug the
issue and a gist[1] has disk usage and filesystem details.

[0]:
https://www.reddit.com/r/archlinux/comments/asrlam/btrfscleaner_at_100_cpu_usage_on_raid1_setup/egwc047/
[1]: https://paste.ubuntu.com/p/GDNKNBqqFy/

Re: RAID1 device deletion feedback

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 3133 bytes --]



On 2019/2/25 上午1:32, Nemo wrote:
> Hi,
> 
> I had a RAID1 disk failure recently, and a limitation in number of SATA
> connectors meant I could not do a live replace.

Then you should go USB or other interfaces.
In your case, device deletion is not possible due to reasons explained
below.

> I'm still in the
> progress of resolving the issue, but posting some feedback here on the
> issues I faced and what could have helped.
> 
> **What all I did**
> 
> - remove the dying disk and mount in degraded mode. This didn't work,
> but _didn't provide me a failure reason_

It provides a failure reason, in dmesg.

> - swap the dead disk post boot with the new one. Tried a btrfs device
> add, but it failed as well because of too many failing reads from the
> dead disk (which wasn't even attached).
> - Started a btrfs device delete, which finally pointed me to the root
> cause:
> 
> `ERROR: error removing device '/dev/sdb1': No space left on device`
> 
> In short:
> 
> 1. My 2.7TBx3 setup was storing 2.9TB of data

If using RAID1, it takes 5.8T of your 7.1T total capacity.
Removing one missing device will cause ENOSPC failure, as 2 devices will
only be 5.4TB.

What you should do is replace, as during replace you only need one extra
chunk and your current device free space is completely fine for replacing.

> 2. Removing any one the disks was not possible without losing some data
> 
> However, getting to this error message took me hours of effort, despite
> having read the btrfs wiki.
> 
> **Feedback**
> 
> 1. The `btrfs device delete` command should have failed immediately
> instead of making me wait for ages.

Not that easy, as btrfs replace/delete/relocate all happens at chunk
level, we can only do educated guess to determine if it's possible,
especially all those operations are done online, user can do whatever
they want, initial estimation is not reliable.

Thanks,
Qu

> 2. A delete command should stop writes to a disk, unless avoidable. This
> would have left some data on the disk, and a subsequent delete could
> have cleaned it up much faster. It failed in the worst manner possible,
> because all deletion progress reversed once it failed: it balanced back
> to complete usage, causing more strain on a dying disk.
> 3. A degraded mount failure should provide some error message.
> 
> *Status*: I cleared up some disk space, and ran another delete which is
> still ongoing. I don't have physical access to the system for a few more
> days, so adding the new disk will take me some time.
> 
> ---
> 
> ## Debug Details
> 
> 
> ```
> uname -a
> Linux tatooine.captnemo.in 4.20.10-arch1-1-ARCH #1 SMP PREEMPT Fri Feb
> 15 17:49:06 UTC 2019 x86_64 GNU/Linux
> 
> btrfs --version
> btrfs-progs v4.20.1
> ```
> 
> A few more details are on a reddit post[0] I made to help debug the
> issue and a gist[1] has disk usage and filesystem details.
> 
> [0]:
> https://www.reddit.com/r/archlinux/comments/asrlam/btrfscleaner_at_100_cpu_usage_on_raid1_setup/egwc047/
> [1]: https://paste.ubuntu.com/p/GDNKNBqqFy/
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: RAID1 device deletion feedback

[Austin S. Hemmelgarn]

On 2019-02-24 12:32, Nemo wrote:
> Hi,
> 
> I had a RAID1 disk failure recently, and a limitation in number of SATA
> connectors meant I could not do a live replace. I'm still in the
> progress of resolving the issue, but posting some feedback here on the
> issues I faced and what could have helped.
> 
> **What all I did**
> 
> - remove the dying disk and mount in degraded mode. This didn't work,
> but _didn't provide me a failure reason_
If you don't see a message on the console, check dmesg.

> - swap the dead disk post boot with the new one. Tried a btrfs device
> add, but it failed as well because of too many failing reads from the
> dead disk (which wasn't even attached).
You can take this approach, but you need to use the replace command not 
the add command.  You can't really add devices to a partial array, for 
exactly the reason you saw it fail here.  In the case of replacing a 
device that is missing, you want to do the following (though it only 
works if you have only one missing device):

     btrfs replace start -r missing /path/to/new/dev /path/to/fs

> - Started a btrfs device delete, which finally pointed me to the root
> cause:
> 
> `ERROR: error removing device '/dev/sdb1': No space left on device`

> 
> In short:
> 
> 1. My 2.7TBx3 setup was storing 2.9TB of data
> 2. Removing any one the disks was not possible without losing some data
> 
> However, getting to this error message took me hours of effort, despite
> having read the btrfs wiki.
> 
> **Feedback**
> 
> 1. The `btrfs device delete` command should have failed immediately
> instead of making me wait for ages.
I won't pretend that your statement isn't correct, but I can at least 
explain _why_ it didn't.

Put simply, the device delete command is a special case of a balance 
operation, and balancing doesn't do any check prior to running to make 
sure that the requested state is valid.  Part of this is that it's 
actually useful in some cases to just run the balance until it fails 
(because getting part of the work done is usually better than none), and 
part of it is because it is very much non-trivial in a number of the 
cases that balance needs to handle how the disk will look after running 
the balance if it's successful.

> 2. A delete command should stop writes to a disk, unless avoidable. This
> would have left some data on the disk, and a subsequent delete could
> have cleaned it up much faster. It failed in the worst manner possible,
> because all deletion progress reversed once it failed: it balanced back
> to complete usage, causing more strain on a dying disk.
It essentially operates currently in a manner similar to the LVM2 pvmove 
command WRT writes, just not as smart.  AFAIUI:

* Any writes that go to a chunk that is on the device to be removed 
still go through.
* Any writes that go to a chunk that is not on the device to be removed 
don't touch it.
* Any writes that go to the chunk currently being moved off of the 
device being removed go to the new location for that chunk.

> 3. A degraded mount failure should provide some error message.
Again, if you see a mount failure, check dmesg.  This is the case for 
pretty much any filesystem (hell, mount actually tells you to do this in 
a lot of cases).
> 
> *Status*: I cleared up some disk space, and ran another delete which is
> still ongoing. I don't have physical access to the system for a few more
> days, so adding the new disk will take me some time.
> 
> ---
> 
> ## Debug Details
> 
> 
> ```
> uname -a
> Linux tatooine.captnemo.in 4.20.10-arch1-1-ARCH #1 SMP PREEMPT Fri Feb
> 15 17:49:06 UTC 2019 x86_64 GNU/Linux
> 
> btrfs --version
> btrfs-progs v4.20.1
> ```
> 
> A few more details are on a reddit post[0] I made to help debug the
> issue and a gist[1] has disk usage and filesystem details.
> 
> [0]:
> https://www.reddit.com/r/archlinux/comments/asrlam/btrfscleaner_at_100_cpu_usage_on_raid1_setup/egwc047/
> [1]: https://paste.ubuntu.com/p/GDNKNBqqFy/
>