Spare Volume Features

Spare Volume Features

[Marc Oggier]

Hi All,

I am currently buidling a small data server for an experiment.

I was wondering if the features of the spare volume introduced a couple 
of years ago (ttps://patchwork.kernel.org/patch/8687721/) would be 
release soon. I think this would be awesome to have a drive installed, 
that can be used as a spare if one drive of an array died to avoid downtime.

Does anyone have news about it, and when it will be officially in the 
kernel/btrfs-progs ?

Marc

P.S. It took me a long time to switch to btrfs. I did it less than a 
year ago, and I love it.  Keep the great job going, y'all

Re: Spare Volume Features

[Sean Greenslade]

On August 28, 2019 5:51:02 PM PDT, Marc Oggier <marc.oggier@megavolts.ch> wrote:
>Hi All,
>
>I am currently buidling a small data server for an experiment.
>
>I was wondering if the features of the spare volume introduced a couple
>
>of years ago (ttps://patchwork.kernel.org/patch/8687721/) would be 
>release soon. I think this would be awesome to have a drive installed, 
>that can be used as a spare if one drive of an array died to avoid
>downtime.
>
>Does anyone have news about it, and when it will be officially in the 
>kernel/btrfs-progs ?
>
>Marc
>
>P.S. It took me a long time to switch to btrfs. I did it less than a 
>year ago, and I love it.  Keep the great job going, y'all

I've been thinking about this issue myself, and I have an (untested) idea for how to accomplish something similar. My file server has three disks in a btrfs raid1. I added a fourth disk to the array as just a normal, participating disk. I keep an eye on the usage to make sure that I never exceed 3 disk's worth of usage. That way, if one disk dies, there are still enough disks to mount RW (though I may still need to do an explicit degraded mount, not sure). In that scenario, I can just trigger an online full balance to rebuild the missing raid copies on the remaining disks. In theory, minimal to no downtime.

I'm curious if anyone can see any problems with this idea. I've never tested it, and my offsite backups are thorough enough to survive downtime anyway.

--Sean

Re: Spare Volume Features

[waxhead]

Sean Greenslade wrote:
> On August 28, 2019 5:51:02 PM PDT, Marc Oggier <marc.oggier@megavolts.ch> wrote:
>> Hi All,
>>
>> I am currently buidling a small data server for an experiment.
>>
>> I was wondering if the features of the spare volume introduced a couple
>>
>> of years ago (ttps://patchwork.kernel.org/patch/8687721/) would be
>> release soon. I think this would be awesome to have a drive installed,
>> that can be used as a spare if one drive of an array died to avoid
>> downtime.
>>
>> Does anyone have news about it, and when it will be officially in the
>> kernel/btrfs-progs ?
>>
>> Marc
>>
>> P.S. It took me a long time to switch to btrfs. I did it less than a
>> year ago, and I love it.  Keep the great job going, y'all
> 
> I've been thinking about this issue myself, and I have an (untested) idea for how to accomplish something similar. My file server has three disks in a btrfs raid1. I added a fourth disk to the array as just a normal, participating disk. I keep an eye on the usage to make sure that I never exceed 3 disk's worth of usage. That way, if one disk dies, there are still enough disks to mount RW (though I may still need to do an explicit degraded mount, not sure). In that scenario, I can just trigger an online full balance to rebuild the missing raid copies on the remaining disks. In theory, minimal to no downtime.
> 
> I'm curious if anyone can see any problems with this idea. I've never tested it, and my offsite backups are thorough enough to survive downtime anyway.
> 
> --Sean
> 
I'm just a regular btrfs user, but I see tons of problems with this.

When BTRFS introduce per-subvolume (or even per file) "RAID" or 
redundancy levels a spare device an quickly become a headache. While you 
can argue that a spare device of equal or large size than the largest 
device in the pool would suffice in most cases I don't think it is very 
practical.

What BTRFS needs to do (IMHO) is to reserve spare-space instead. This 
means that many smaller devices can be used in case a large device keels 
over.

The spare space also of course needs to be as large or larger than the 
largest device in the pool, but you would have more flexibility.

For example spare space COULD be pre-populated with the most important 
data (hot data tracking) and serve as a speed-up for read operations. 
What is the point of having idle space just waiting to be used when you 
in fact can just use it for useful things such as obvious ideas like 
increased read speed, extra redundancy for stuff like single, dup or 
even raid0 chunks. Using the spare space for SOME potential for recovery 
is better than not using the spare space for anything.

When the spare space is needed you can either simply discard the data on 
the device that is broken if the spare space already holds the data 
(which makes for superfast recovery) or drop any caches it is used for 
and repopulate by restoring non-redundant data to it as soon as you hit 
a certain error count on another device etc...

Just like Linux uses memory I think that BTRFS is better off using the 
spare space for something rather than nothing. This should of course be 
configurable just for the record.

Anyway - that is how I, a humble user without the detailed know-how 
think it should be implemented... :)

Re: Spare Volume Features

[Sean Greenslade]

On Wed, Aug 28, 2019 at 07:21:14PM -0700, Sean Greenslade wrote:
> On August 28, 2019 5:51:02 PM PDT, Marc Oggier <marc.oggier@megavolts.ch> wrote:
> >Hi All,
> >
> >I am currently buidling a small data server for an experiment.
> >
> >I was wondering if the features of the spare volume introduced a couple
> >
> >of years ago (ttps://patchwork.kernel.org/patch/8687721/) would be 
> >release soon. I think this would be awesome to have a drive installed, 
> >that can be used as a spare if one drive of an array died to avoid
> >downtime.
> >
> >Does anyone have news about it, and when it will be officially in the 
> >kernel/btrfs-progs ?
> >
> >Marc
> >
> >P.S. It took me a long time to switch to btrfs. I did it less than a 
> >year ago, and I love it.  Keep the great job going, y'all
> 
> I've been thinking about this issue myself, and I have an (untested)
> idea for how to accomplish something similar. My file server has three
> disks in a btrfs raid1. I added a fourth disk to the array as just a
> normal, participating disk. I keep an eye on the usage to make sure
> that I never exceed 3 disk's worth of usage. That way, if one disk
> dies, there are still enough disks to mount RW (though I may still
> need to do an explicit degraded mount, not sure). In that scenario, I
> can just trigger an online full balance to rebuild the missing raid
> copies on the remaining disks. In theory, minimal to no downtime.
> 
> I'm curious if anyone can see any problems with this idea. I've never
> tested it, and my offsite backups are thorough enough to survive
> downtime anyway.
> 
> --Sean

I decided to do a bit of experimentation to test this theory. The
primary goal was to see if a filesystem could suffer a failed disk and
have that disk removed and rebalanced among the remaining disks without
the filesystem losing data or going read-only. Tested on kernel
5.2.5-arch1-1-ARCH, progs: v5.2.1.

I was actually quite impressed. When I ripped one of the block devices
out from under btrfs, the kernel started spewing tons of BTRFS errors,
but seemed to keep on trucking. I didn't leave it in this state for too
long, but I was reading, writing, and syncing the fs without issue.
After performing a btrfs device delete <MISSING_DEVID>, the filesystem
rebalanced and stopped reporting errors. Looks like this may be a viable
strategy for high-availability filesystems assuming you have adequate
monitoring in place to catch the disk failures quickly. I personally
wouldn't want to fully automate the disk deletion, but it's certainly
possible.

--Sean

Re: Spare Volume Features

[Andrei Borzenkov]

01.09.2019 6:28, Sean Greenslade пишет:
> 
> I decided to do a bit of experimentation to test this theory. The
> primary goal was to see if a filesystem could suffer a failed disk and
> have that disk removed and rebalanced among the remaining disks without
> the filesystem losing data or going read-only. Tested on kernel
> 5.2.5-arch1-1-ARCH, progs: v5.2.1.
> 
> I was actually quite impressed. When I ripped one of the block devices
> out from under btrfs, the kernel started spewing tons of BTRFS errors,
> but seemed to keep on trucking. I didn't leave it in this state for too
> long, but I was reading, writing, and syncing the fs without issue.
> After performing a btrfs device delete <MISSING_DEVID>, the filesystem
> rebalanced and stopped reporting errors.

How many devices did filesystem have? What profiles did original
filesystem use and what profiles were present after deleting device?
Just to be sure there was no silent downgrade from raid1 to dup or
single as example.


> Looks like this may be a viable
> strategy for high-availability filesystems assuming you have adequate
> monitoring in place to catch the disk failures quickly. I personally
> wouldn't want to fully automate the disk deletion, but it's certainly
> possible.
> 

This would be valid strategy if we could tell btrfs to reserve enough
spare space; but even this is not enough, every allocation btrfs does
must be done so that enough spare space remains to reconstruct every
other missing chunk.

Actually I now ask myself - what happens when btrfs sees unusable disk
sector(s) in some chunk? Will it automatically reconstruct content of
this chunk somewhere else? If not, what is an option besides full device
replacement?

Re: Spare Volume Features

[Sean Greenslade]

On Sun, Sep 01, 2019 at 11:03:59AM +0300, Andrei Borzenkov wrote:
> 01.09.2019 6:28, Sean Greenslade пишет:
> > 
> > I decided to do a bit of experimentation to test this theory. The
> > primary goal was to see if a filesystem could suffer a failed disk and
> > have that disk removed and rebalanced among the remaining disks without
> > the filesystem losing data or going read-only. Tested on kernel
> > 5.2.5-arch1-1-ARCH, progs: v5.2.1.
> > 
> > I was actually quite impressed. When I ripped one of the block devices
> > out from under btrfs, the kernel started spewing tons of BTRFS errors,
> > but seemed to keep on trucking. I didn't leave it in this state for too
> > long, but I was reading, writing, and syncing the fs without issue.
> > After performing a btrfs device delete <MISSING_DEVID>, the filesystem
> > rebalanced and stopped reporting errors.
> 
> How many devices did filesystem have? What profiles did original
> filesystem use and what profiles were present after deleting device?
> Just to be sure there was no silent downgrade from raid1 to dup or
> single as example.

I did the simplest case: raid1 with 3 disks, dropping 1 disk to end up
with raid1 with 2 disks. I did check and btrfs fi usage reported no dup
or single chunks.

> > Looks like this may be a viable
> > strategy for high-availability filesystems assuming you have adequate
> > monitoring in place to catch the disk failures quickly. I personally
> > wouldn't want to fully automate the disk deletion, but it's certainly
> > possible.
> > 
> 
> This would be valid strategy if we could tell btrfs to reserve enough
> spare space; but even this is not enough, every allocation btrfs does
> must be done so that enough spare space remains to reconstruct every
> other missing chunk.
> 
> Actually I now ask myself - what happens when btrfs sees unusable disk
> sector(s) in some chunk? Will it automatically reconstruct content of
> this chunk somewhere else? If not, what is an option besides full device
> replacement?

As far as I can tell, btrfs has no facility for dealing with medium
errors (besides just reporting the error).  I just re-ran a simple test
with a two-device raid1 with one device deleted after mounting. Btrfs
complains loudly every time writes to the missing disk fail, but doesn't
retry or redirect these writes.  One half of the raid1 block group makes
it to disk, the other gets lost to the void. The chunk that makes it to
disk is still of raid1 type.

Essentially, it seems that btrfs currently had no way of marking a disk
as offline / missing / problematic post-mount. Additionally, and
possibly more troubling, is the fact that a failed chunk write will not
get retried, even if there is another disk that could possibly accept
that write. I think that for my fake-hot-spare proposal to be viable as
a fault resiliancy measure, this failed-chunk-retry logic would need to
be implemented. Otherwise you're living without data redundancy for some
old data and some (or potentially all) new data from the moment the
first medium error occurs until the moment the device delete completes
successfully.

--Sean

Re: Spare Volume Features

[Chris Murphy]

I'm still mostly convinced the policy questions and management should
be dealt with a btrfsd userspace daemon.

Btrfs kernel code itself tolerates quite a lot of read and write
errors, where a userspace service could say, yeah forget that we're
moving over to the spare.

Also, that user space daemon could handle the spare device while its
in spare status. I don't really see why btrfs kernel code needs to
know about it. It's reserved for Btrfs but isn't used by Btrfs, until
a policy is triggered. Plausibly one of the policies isn't even device
failure, but the volume is nearly full. Spares should be assignable to
multiple Btrfs volumes. And that too can be managed by this
hypothetical daemon.

--
Chris Murphy

Re: Spare Volume Features

[Austin S. Hemmelgarn]

On 2019-09-01 21:09, Chris Murphy wrote:
> I'm still mostly convinced the policy questions and management should
> be dealt with a btrfsd userspace daemon.
> 
> Btrfs kernel code itself tolerates quite a lot of read and write
> errors, where a userspace service could say, yeah forget that we're
> moving over to the spare.
> 
> Also, that user space daemon could handle the spare device while its
> in spare status. I don't really see why btrfs kernel code needs to
> know about it. It's reserved for Btrfs but isn't used by Btrfs, until
> a policy is triggered. Plausibly one of the policies isn't even device
> failure, but the volume is nearly full. Spares should be assignable to
> multiple Btrfs volumes. And that too can be managed by this
> hypothetical daemon.
Having the kernel know about it means, among other things, that 
switching to actually using the spare when needed is far less likely to 
be delayed by an arbitrarily long time.  Worst case scenario in 
userspace is that the daemon gets paged out, but the executable is on 
the volume it's supposed to be fixing and the volume is in a state that 
it returns read errors until it gets fixed.  In such a case, the volume 
will never get fixed without manual intervention.  Such a situation is 
impossible if it's being handled by the kernel.  This could be mitigated 
by using mlock, but that brings it's own set of issues.

Re: Spare Volume Features

[Anand Jain]

  use-cases in production must need a spare device to maintain data
  redundancy in the event of volume disk failure, so this feature
  should be in btrfs.ko. And nntil we get there, workaround like
  monitor for write_io_errs and call for replace should help.

  HTH.

Thanks, Anand