Some ideas for improvements

Some ideas for improvements

[Dmitry Katsubo]

Dear btrfs community,

I hope btrfs developers are open for suggestions.

btrfs-scrub:

* Would be nice if 'btrfs scrub status' shows estimated finishing time 
(ETA) and throughput (in Mb/s).
* Not possible to start scrub for all devices in the volume without 
mounting it.

btrfs-restore:

* It does not restore special files like named pipes and devices.
* Hard-linked files are not correctly restored (they all turn into 
independent replicas).
* If the file cannot be read / recovered, it is still created with zero 
size (I would expect that the file is not created).
* I think that the options '-xmS' should be enabled by default 
(shouldn't it be a goal to restore as much as possible?).
* Option that applies (y) to all questions (completely unattended 
recovery) is missing.

-- 
With best regards,
Dmitry

Re: Some ideas for improvements

[Duncan]

Dmitry Katsubo posted on Wed, 25 May 2016 16:45:41 +0200 as excerpted:

> Dear btrfs community,
> 
> I hope btrfs developers are open for suggestions.
> 
> btrfs-scrub:
> 
> * Would be nice if 'btrfs scrub status' shows estimated finishing time
> (ETA) and throughput (in Mb/s).

That might not be so easy to implement.  (Caveat, I'm not a dev, just a 
btrfs user and list regular, so if a dev says different...)

Currently, a running scrub simply outputs progress to a file (/var/lib/
btrfs/scrub.status.<UUID>), and scrub status is simply a UI to pretty-
print that file.  Note that there's nothing in there which lists the 
total number of extents or bytes to go -- that's not calculated ahead of 
time.

So implementing some form of percentage done or eta is likely to increase 
the processing time dramatically, as it could involve doing a dry-run 
first, in ordered to get the total figures against which to calculate 
percentage done.

> * Not possible to start scrub for all devices in the volume without
> mounting it.

Interesting.  It's news to me that you can scrub individual devices 
without mounting.  But given that, this would indeed be a useful feature, 
and given that btrfs filesystem show can get the information, scrub 
should be able to get and make use of it as well. =:^)

So (again not a dev, but...) this one's probably just a matter of coding 
it up.  Tho there's lots of ideas and only a comparatively few devs 
working on them, so unless it happens to strike someone's fancy as a nice 
short project, it could be awhile before it's implemented.

> btrfs-restore:
> 
> * It does not restore special files like named pipes and devices.
> * Hard-linked files are not correctly restored (they all turn into
> independent replicas).
> * If the file cannot be read / recovered, it is still created with zero
> size (I would expect that the file is not created).
> * I think that the options '-xmS' should be enabled by default
> (shouldn't it be a goal to restore as much as possible?).
> * Option that applies (y) to all questions (completely unattended
> recovery) is missing.

That latter point is a known sore spot that a lot of people have 
complained about.  So it'll almost certainly be addressed eventually, but 
it's a matter of this project vying with all the others so it could be 
awhile.

As for the other points, restore has rather evolved over time.  Those 
optional features were added on one or two at a time and were at first 
non-default options because as new features there was a non-zero chance 
they could disrupt the entire restore, regressing restore as a useful 
disaster recovery tool.

It's also useful to remember that btrfs is still considered by its devs 
to be very much still stabilizing, not fully stable and mature, and many 
of its tools, including restore, /do/ continue to evolve.  Once the 
general feature set is reasonably full and stable, it's likely some of 
these non-default options will be made default, for a final stable-form 
command-set.  I'd consider it quite likely that some of the restore 
options will indeed be made the default, in final-form btrfs restore.

Meanwhile, that unfinished evolution is most likely behind the zero-sized 
(non-)restore of some files, the missing special files, etc.  Those 
issues will likely be addressed at some point.

In fact, it was someone wanting timestamp restoration for his own use, 
that happened to be a dev that could actually write the code, that 
contributed the metadata restoration option, scratching his own itch.  
And when I saw that, I immediately asked about symlink restoration, as 
that was an itch I had when using restore myself, tho not being a dev I 
couldn't just code up the solution to scratch it as he did.  But I was 
lucky and he had the time and patience to do it while he was working with 
the code, so the symlinks restoration option was added a release or two 
after the metadata restoration option.

But I guess nobody has had and made known an itch for other special file 
restoration yet -- I wasn't even aware they weren't restored, here, as it 
hadn't occurred to me to check.  So it'll probably happen, but having 
missed the opportunity I took advantage of to request symlink restoration 
support while he was working on the metadata support, again, it could be 
awhile until someone actually codes up the special file restoration 
support.

I am, however, not sure how practical hardlink (and of course the btrfs 
specific reflink) restoration, maintaining the link as opposed to 
breaking the link and restoring as separate files actually is, and if 
it's complicated at all, the priority is likely to be quite low.  Because 
after all, btrfs restore isn't intended for routine use, but as a 
disaster recovery tool.  In theory, people should be recovering from 
their backups (and any good sysadmin knows the first rule of backups in 
simple form is that if you don't have at least one level of backup, you 
are by lack thereof declaring the data to be worth less than the time and 
resources necessary to do that backup), and shouldn't really be depending 
on btrfs restore anyway.  As such, just restoring the files at all is 
arguably enough, and all these options like restoring metadata are 
arguably gravy on top already.  If restoring hard-links, and if the 
restore is to btrfs, possibly reflinks as well, as links not independent 
files, isn't too complex, then yeah, restoring them as an option is 
fine.  But if it's adding serious additional complexity, as I suspect it 
might, yes, the option might still be added at some point, but the 
priority to do it will be pretty low, which means the point at which it's 
actually added is likely to be out there five years or more, simply 
because there's so many other ideas at a higher priority to implement as 
well.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Some ideas for improvements

[Nicholas D Steeves]

On 25 May 2016 at 15:03, Duncan <1i5t5.duncan@cox.net> wrote:
> Dmitry Katsubo posted on Wed, 25 May 2016 16:45:41 +0200 as excerpted:
>> btrfs-restore:
>>
>> * It does not restore special files like named pipes and devices.
>> * Hard-linked files are not correctly restored (they all turn into
>> independent replicas).
>> * If the file cannot be read / recovered, it is still created with zero
>> size (I would expect that the file is not created).
>> * I think that the options '-xmS' should be enabled by default
>> (shouldn't it be a goal to restore as much as possible?).
>> * Option that applies (y) to all questions (completely unattended
>> recovery) is missing.
>
> That latter point is a known sore spot that a lot of people have
> complained about.  So it'll almost certainly be addressed eventually, but
> it's a matter of this project vying with all the others so it could be
> awhile.

I'm surprised no one has mentioned, in any of these discussions, what
I believe is the standard method of providing this functionality:
yes | btrfs-restore -options /dev/disk

And if you need it in your initrd on a Debian-like system, put the
following in /etc/initramfs-tools/hooks/yes.hook :

#!/bin/sh

. /usr/share/initramfs-tools/hook-functions

if command -v /usr/bin/yes >/dev/null 2>&1; then
    copy_exec /usr/bin/yes usr/bin/yes
    copy_exec /usr/bin/yes usr/bin/yes
fi

I haven't tested this, but it seems like it would do the trick.

Cheers,
Nicholas

Re: Some ideas for improvements

[Duncan]

Nicholas D Steeves posted on Wed, 25 May 2016 16:36:13 -0400 as excerpted:

> On 25 May 2016 at 15:03, Duncan <1i5t5.duncan@cox.net> wrote:
>> Dmitry Katsubo posted on Wed, 25 May 2016 16:45:41 +0200 as excerpted:

>>> btrfs-restore [needs an o]ption that applies (y) to all questions
>>> (completely unattended recovery)
>>
>> That['s] a known sore spot that a lot of people have complained
>> about.

> I'm surprised no one has mentioned, in any of these discussions, what I
> believe is the standard method of providing this functionality:
> yes | btrfs-restore -options /dev/disk

Good point.

I didn't bring it up because while I've used btrfs restore a few times, 
my btrfs are all on relatively small SSD partitions, so I both needed 
less y's, and the total time per restore is a few minutes, not hours, so 
it wasn't a big deal.  As a result, while I know of yes, I didn't need to 
think about automation, and as I never used it, it didn't occur to me to 
suggest it for others.

Had I needed to use it, however, I expect I would have reached for yes, 
myself, so good point indeed, pointing out its use to others.  I just 
wish I'd thought of mentioning it, now.  =:^)

> And if you need it in your initrd on a Debian-like system, put the
> following in /etc/initramfs-tools/hooks/yes.hook :
> 
> #!/bin/sh
> 
> . /usr/share/initramfs-tools/hook-functions
> 
> if command -v /usr/bin/yes >/dev/null 2>&1; then
>     copy_exec /usr/bin/yes usr/bin/yes copy_exec /usr/bin/yes
>     usr/bin/yes
> fi
> 
> I haven't tested this, but it seems like it would do the trick.

FWIW, for those using a dracut-based initr* (which should be fedora/rh-
based distros as well as those like arch and gentoo, where it'd be an 
option, the recommended option in at least gentoo's case), it's a simple 
config file edit, with dracut taking care of the rest, including pulling 
in libraries for non-static binaries.  I've not used it for yes, but I've 
used it for a few other binaries.

I use the /etc/dracut.conf.d/* style of config here, and here's what I 
have in my specific installitems.conf file in that dir:

install_items+="
	/bin/most
	/bin/nano
"

(I don't have less, which dracut uses by default, on my system, 
preferring most.  And nano is a tiny but useful text editor that doesn't 
require the esoteric pre-use knowledge of something like vi/vim-minimal, 
tho I prefer the larger mc/mcedit in a less size constrained environment.)

Obviously I'd just need to include one more line there to pull in 
/bin/yes, if I wanted/needed it.  (Tho I've not checked, for all I know 
dracut pulls it in on its own or via some other module I'm already using.)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Some ideas for improvements

[Dmitry Katsubo]

On 2016-05-25 21:03, Duncan wrote:
> Dmitry Katsubo posted on Wed, 25 May 2016 16:45:41 +0200 as excerpted:
>> * Would be nice if 'btrfs scrub status' shows estimated finishing time
>> (ETA) and throughput (in Mb/s).
> 
> That might not be so easy to implement.  (Caveat, I'm not a dev, just a 
> btrfs user and list regular, so if a dev says different...)
> 
> Currently, a running scrub simply outputs progress to a file (/var/lib/
> btrfs/scrub.status.<UUID>), and scrub status is simply a UI to pretty-
> print that file.  Note that there's nothing in there which lists the 
> total number of extents or bytes to go -- that's not calculated ahead of 
> time.
> 
> So implementing some form of percentage done or eta is likely to increase 
> the processing time dramatically, as it could involve doing a dry-run 
> first, in ordered to get the total figures against which to calculate 
> percentage done.

Indeed that this cannot (should not) be done on user-space level: kernel
module should provide information about that. I am not a dev :) but I
think module should now number of extents, at least something is shown in
"btrfs fi usage ..." output.

The information shouldn't be 100% exact, but at least some indication
would be great. In worst scenario module can remember the last scrub
time and make estimation based on that (similar how some CD burning
utilities do).

>> * Not possible to start scrub for all devices in the volume without
>> mounting it.
> 
> Interesting.  It's news to me that you can scrub individual devices 
> without mounting.  But given that, this would indeed be a useful feature, 
> and given that btrfs filesystem show can get the information, scrub 
> should be able to get and make use of it as well. =:^)

More over I got into a trap when tried to use "btrfs scrub start /dev/..."
syntax, as I only scrubs the given device. When I scrubbed the whole
volume after mounting it, de result was different. I understood it only
after reading man btrfs-scrub more attentively:

  start ... <path>|<device>

  Start a scrub on all devices of the filesystem identified by <path>
  or on a single <device>.

Other (shorter) forms of help misled me, giving the impression that
it does not matter whether I specify a path or device.

On 2016-05-26 00:05, Duncan wrote:
> Nicholas D Steeves posted on Wed, 25 May 2016 16:36:13 -0400 as excerpted:
>> On 25 May 2016 at 15:03, Duncan <1i5t5.duncan@cox.net> wrote:
>>> Dmitry Katsubo posted on Wed, 25 May 2016 16:45:41 +0200 as excerpted:
>>>> btrfs-restore [needs an o]ption that applies (y) to all questions
>>>> (completely unattended recovery)
>>>
>>> That['s] a known sore spot that a lot of people have complained
>>> about.
> 
>> I'm surprised no one has mentioned, in any of these discussions, what I
>> believe is the standard method of providing this functionality:
>> yes | btrfs-restore -options /dev/disk
> 
> Good point.
> 
> I didn't bring it up because while I've used btrfs restore a few times, 
> my btrfs are all on relatively small SSD partitions, so I both needed 
> less y's, and the total time per restore is a few minutes, not hours, so 
> it wasn't a big deal.  As a result, while I know of yes, I didn't need to 
> think about automation, and as I never used it, it didn't occur to me to 
> suggest it for others.

Thanks for advise, Nicholas. Last time I tried it I used the following
command:

while true; do echo y; done | btrfs restore -voxmSi /dev/sda /mnt/tmp &> btrfs_restore &

which presumably is equivalent to what you suggest. The command was in
"running" state in "jobs" output for a while, but then turned into
"waiting" state and did not progress. I suspect that btrfs-restore
somehow reads directly from terminal, not from stdin. I will try the
solution with "yes | btrfs-restore..." once I get a chance.

-- 
With best regards,
Dmitry