btrfs-progs and btrfs(8) inconsistencies

btrfs-progs and btrfs(8) inconsistencies

[Moviuro]

Hi all,

I should have done this a long time ago, so here goes.

Btrfs is certainly the best FS out there but btrfs(8) is a pain to use in
scripts. That is, it talks way too much, exposes unparseable outputs and
has inconsistent behavior.

I strongly believe that the CLI may use a complete revamp, that is: think
of the needed/wanted output. I actually ran into this frustrating behavior
while writing butter (https://github.com/moviuro/butter).

Eg: # btrfs subvolume create foo
Should be silent. I don't want btrfs(8) to bug me about successful
commands. Everything that is a success is silent. In its current form,
btrfs-subvolume(8) would spat out a complete sentence in English that we
don't need, which however contains a tiny bit of useful info: the path
(though relative) to the new subvolume. A -v/--verbose option might be here
a good idea to implement and the following behavior (just thinking out
loud, I'm sure there would be lots of things to think about more precisely):

# btrfs subvolume create foo # should be silent
# btrfs subvolume create -v foo
/absolute/path/foo
# btrfs subvolume create [--details|-vv] foo
PATH:/absolute/path/foo
UUID:082df386-98c2-44d9-9012-07fb2b22ea20
[And whatnot]

That's a first example. Same should go for all the commands. I have no idea
how/where we could share about good/bad outputs (Google Drive? Framapad?
git[hub|lab]?)

Then come inconsistencies: compare the outputs of
# btrfs sub create foo
# btrfs sub delete foo

This also needs to be taken care of, but it should be a by-product of
thinking the outputs all over again.

Behavior inconsistencies: some seemingly valid commands fail, because of
some code that can be easily changed (I'm working on it in my spare time
already, see https://github.com/moviuro/btrfs-progs/tree/getopts):

The valid '--' doesn't work on every command, see
0aa796cad7ed3fce9d5964646a8f1f5a142b4989 here:
https://github.com/kdave/btrfs-progs/commit/0aa796cad7ed3fce9d5964646a8f1f5a142b4989


That's it. And since AFAICT btrfs is FLOSS, I'm sharing about my experience
as user and hope my opinion will change btrfs(8) for the better.

Cheers,

Re: btrfs-progs and btrfs(8) inconsistencies

[Qu Wenruo]

Moviuro wrote on 2016/02/03 22:54 +0100:
> Hi all,
>
> I should have done this a long time ago, so here goes.
>
> Btrfs is certainly the best FS out there but btrfs(8) is a pain to use in
> scripts. That is, it talks way too much, exposes unparseable outputs and
> has inconsistent behavior.
>
> I strongly believe that the CLI may use a complete revamp, that is: think
> of the needed/wanted output. I actually ran into this frustrating behavior
> while writing butter (https://github.com/moviuro/butter).
>
> Eg: # btrfs subvolume create foo
> Should be silent. I don't want btrfs(8) to bug me about successful
> commands. Everything that is a success is silent. In its current form,
> btrfs-subvolume(8) would spat out a complete sentence in English that we
> don't need, which however contains a tiny bit of useful info: the path
> (though relative) to the new subvolume. A -v/--verbose option might be here
> a good idea to implement and the following behavior (just thinking out
> loud, I'm sure there would be lots of things to think about more precisely):
>
> # btrfs subvolume create foo # should be silent
> # btrfs subvolume create -v foo
> /absolute/path/foo
> # btrfs subvolume create [--details|-vv] foo
> PATH:/absolute/path/foo
> UUID:082df386-98c2-44d9-9012-07fb2b22ea20
> [And whatnot]

The idea itself makes a lot of sense.
But I have at least two things to worry about:

1) Old scripts backward compatibility
    Especially xfstests. Maintainer will hate it a lot.
    As we have changed it several times and broken existing test cases.

    Although personally I like to let all the backward compatibility
    things go hell, but that's definitely not how things work. :(

2) End-user taste.
    Some end-users like such info as feedback of success.
    Of course other users like it act as silent as possible.

>
> That's a first example. Same should go for all the commands. I have no idea
> how/where we could share about good/bad outputs (Google Drive? Framapad?
> git[hub|lab]?)

Maybe it's overkilling, but I like the idea to have a good example about 
how CLI interface should be designed.
But it may be harder for all developer/reviewer to follow that restrict 
example.

And even more, I hope there will be a nice btrfs-progs CLI design 
guideline. (Although it's surely overkilling)

Thanks,
Qu

>
> Then come inconsistencies: compare the outputs of
> # btrfs sub create foo
> # btrfs sub delete foo
>
> This also needs to be taken care of, but it should be a by-product of
> thinking the outputs all over again.
>
> Behavior inconsistencies: some seemingly valid commands fail, because of
> some code that can be easily changed (I'm working on it in my spare time
> already, see https://github.com/moviuro/btrfs-progs/tree/getopts):
>
> The valid '--' doesn't work on every command, see
> 0aa796cad7ed3fce9d5964646a8f1f5a142b4989 here:
> https://github.com/kdave/btrfs-progs/commit/0aa796cad7ed3fce9d5964646a8f1f5a142b4989
>
>
> That's it. And since AFAICT btrfs is FLOSS, I'm sharing about my experience
> as user and hope my opinion will change btrfs(8) for the better.
>
> Cheers,
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: btrfs-progs and btrfs(8) inconsistencies

[Moviuro]

On Thu, Feb 4, 2016 at 2:33 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>
>
> Moviuro wrote on 2016/02/03 22:54 +0100:
>>
>> Hi all,
>>
>> ...
>>
>> # btrfs subvolume create foo # should be silent
>> # btrfs subvolume create -v foo
>> /absolute/path/foo
>> # btrfs subvolume create [--details|-vv] foo
>> PATH:/absolute/path/foo
>> UUID:082df386-98c2-44d9-9012-07fb2b22ea20
>> [And whatnot]
>
>
> The idea itself makes a lot of sense.
> But I have at least two things to worry about:
>
> 1) Old scripts backward compatibility
>    Especially xfstests. Maintainer will hate it a lot.
>    As we have changed it several times and broken existing test cases.
Right now, I can think of the following:
- add a "backward-compatibility switch", like -o|--old, which would
have no effect on the output on version 4.X.
- add a "use the new behavior switch", like -n|--new, which would have
no effect starting with the next big release (5.X)

>    Although personally I like to let all the backward compatibility
>    things go hell, but that's definitely not how things work. :(
>
> 2) End-user taste.
>    Some end-users like such info as feedback of success.
>    Of course other users like it act as silent as possible.
I'm pretty sure that's... not the case. Almost everything on GNU/Linux
is silent. cd(1) is silent, cp(1) is silent, rm(1)...
What they all have though is a -v|--verbose switch.

>>
>> That's a first example. Same should go for all the commands. I have no
>> idea
>> how/where we could share about good/bad outputs (Google Drive? Framapad?
>> git[hub|lab]?)
>
>
> Maybe it's overkilling, but I like the idea to have a good example about how
> CLI interface should be designed.
> But it may be harder for all developer/reviewer to follow that restrict
> example.
>
> And even more, I hope there will be a nice btrfs-progs CLI design guideline.
> (Although it's surely overkilling)
What kind of format do you think we should write this in? drop a
stylesheet.md in the repo?

> Thanks,
> Qu
>> Then come inconsistencies: compare the outputs of
>> ...
>

Re: btrfs-progs and btrfs(8) inconsistencies

[Qu Wenruo]

Moviuro wrote on 2016/02/04 09:57 +0100:
> On Thu, Feb 4, 2016 at 2:33 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>>
>>
>> Moviuro wrote on 2016/02/03 22:54 +0100:
>>>
>>> Hi all,
>>>
>>> ...
>>>
>>> # btrfs subvolume create foo # should be silent
>>> # btrfs subvolume create -v foo
>>> /absolute/path/foo
>>> # btrfs subvolume create [--details|-vv] foo
>>> PATH:/absolute/path/foo
>>> UUID:082df386-98c2-44d9-9012-07fb2b22ea20
>>> [And whatnot]
>>
>>
>> The idea itself makes a lot of sense.
>> But I have at least two things to worry about:
>>
>> 1) Old scripts backward compatibility
>>     Especially xfstests. Maintainer will hate it a lot.
>>     As we have changed it several times and broken existing test cases.
> Right now, I can think of the following:
> - add a "backward-compatibility switch", like -o|--old, which would
> have no effect on the output on version 4.X.
> - add a "use the new behavior switch", like -n|--new, which would have
> no effect starting with the next big release (5.X)
>
>>     Although personally I like to let all the backward compatibility
>>     things go hell, but that's definitely not how things work. :(
>>
>> 2) End-user taste.
>>     Some end-users like such info as feedback of success.
>>     Of course other users like it act as silent as possible.
> I'm pretty sure that's... not the case. Almost everything on GNU/Linux
> is silent. cd(1) is silent, cp(1) is silent, rm(1)...
> What they all have though is a -v|--verbose switch.
>
>>>
>>> That's a first example. Same should go for all the commands. I have no
>>> idea
>>> how/where we could share about good/bad outputs (Google Drive? Framapad?
>>> git[hub|lab]?)
>>
>>
>> Maybe it's overkilling, but I like the idea to have a good example about how
>> CLI interface should be designed.
>> But it may be harder for all developer/reviewer to follow that restrict
>> example.
>>
>> And even more, I hope there will be a nice btrfs-progs CLI design guideline.
>> (Although it's surely overkilling)
> What kind of format do you think we should write this in? drop a
> stylesheet.md in the repo?

Markdown is good enough for me.
But I'm not sure where to put it.
Btrfs wiki? Btrfs-progs repo?

Maybe you can write one and put it in btrfs-progs, and send out as a 
patch for us to review?

Thanks,
Qu

>
>> Thanks,
>> Qu
>>> Then come inconsistencies: compare the outputs of
>>> ...
>>
>
>

Re: btrfs-progs and btrfs(8) inconsistencies

[Martin Steigerwald]

Am Donnerstag, 4. Februar 2016, 09:57:54 CET schrieb Moviuro:
> > Although personally I like to let all the backward compatibility
> > things go hell, but that's definitely not how things work. :(
> > 
> > 2) End-user taste.
> > Some end-users like such info as feedback of success.
> > Of course other users like it act as silent as possible.
> 
> I'm pretty sure that's... not the case. Almost everything on GNU/Linux
> is silent. cd(1) is silent, cp(1) is silent, rm(1)...
> What they all have though is a -v|--verbose switch.

The various mkfs commands are not. Not one of them I know of. Additionally 
each one gives a different output.

pvcreate, vgcreate, lvcreate and as well as the remove commands and probably 
other LVM commands are not (no one could argue, that from their ideas they 
come from HP/UX, but thats a Unix as well):

merkaba:~> lvcreate -L 1G -n bla sata
  Logical volume "bla" created.

And I think, not testing right now, that also mdadm is not silent on creating 
a softraid.

So while I agree with you that regular shell commands (coreutils, util-linux) 
are silent on success usually this does not appear to be the case with storage 
related commands in GNU/Linux.

I don´t have a clear oppinion about it other than I´d like to see some 
standard too. coreutils / util-linux both them to have some kind of standard, 
although not necessarily the same standard I bet. And I am not sure whether it 
is documented somewhere.

-- 
Martin

Re: btrfs-progs and btrfs(8) inconsistencies

[Moviuro]

On Thu, Feb 4, 2016 at 11:14 AM, Martin Steigerwald <martin@lichtvoll.de> wrote:
> Am Donnerstag, 4. Februar 2016, 09:57:54 CET schrieb Moviuro:
>> > Although personally I like to let all the backward compatibility
>> > things go hell, but that's definitely not how things work. :(
>> >
>> > 2) End-user taste.
>> > Some end-users like such info as feedback of success.
>> > Of course other users like it act as silent as possible.
>>
>> I'm pretty sure that's... not the case. Almost everything on GNU/Linux
>> is silent. cd(1) is silent, cp(1) is silent, rm(1)...
>> What they all have though is a -v|--verbose switch.
>
> The various mkfs commands are not. Not one of them I know of. Additionally
> each one gives a different output.
>
> pvcreate, vgcreate, lvcreate and as well as the remove commands and probably
> other LVM commands are not (no one could argue, that from their ideas they
> come from HP/UX, but thats a Unix as well):
>
> merkaba:~> lvcreate -L 1G -n bla sata
>   Logical volume "bla" created.
>
> And I think, not testing right now, that also mdadm is not silent on creating
> a softraid.
>
> So while I agree with you that regular shell commands (coreutils, util-linux)
> are silent on success usually this does not appear to be the case with storage
> related commands in GNU/Linux.
To be perfectly honnest, my inspiration comes from zfs(8) on FreeBSD.
I'm pretty sure there is some inspiration we can find in its behavior.
(see https://www.freebsd.org/cgi/man.cgi?zfs(8) )
The output has both a human-readable form (using column(1)) and a
machine-parseable form (e.g. zfs get -o value used /zroot).
The output is limited to what is truly useful, etc.
I have not had any previous experience with any other FS or softraid
solutions, so I'm not familiar with those commands you gave, Martin.

Also, making silence a default option helps spare some typing:
>/dev/null : 11 char (force silence)
-v : 3 char (force verbose) (it's also far more readable)

I should start working on said guidelines.md later today or this weekend.

> I don´t have a clear oppinion about it other than I´d like to see some
> standard too. coreutils / util-linux both them to have some kind of standard,
> although not necessarily the same standard I bet. And I am not sure whether it
> is documented somewhere.

Re: btrfs-progs and btrfs(8) inconsistencies

[Austin S. Hemmelgarn]

On 2016-02-04 05:14, Martin Steigerwald wrote:
> Am Donnerstag, 4. Februar 2016, 09:57:54 CET schrieb Moviuro:
>>> Although personally I like to let all the backward compatibility
>>> things go hell, but that's definitely not how things work. :(
>>>
>>> 2) End-user taste.
>>> Some end-users like such info as feedback of success.
>>> Of course other users like it act as silent as possible.
>>
>> I'm pretty sure that's... not the case. Almost everything on GNU/Linux
>> is silent. cd(1) is silent, cp(1) is silent, rm(1)...
>> What they all have though is a -v|--verbose switch.
>
> The various mkfs commands are not. Not one of them I know of. Additionally
> each one gives a different output.
This is kind of understandable, it's an inherently destructive 
operation, and the information they spit out is almost always useful. 
On top of that, except for certain very specific usage, you don't run 
mkfs every day, while it's a lot more likely that you use btrfs 
subvolume commands with a high regularity.  There are also a number of 
mkfs commands that let you silence the output without using shell 
redirection, which is a good thing.
>
> pvcreate, vgcreate, lvcreate and as well as the remove commands and probably
> other LVM commands are not (no one could argue, that from their ideas they
> come from HP/UX, but thats a Unix as well):
>
> merkaba:~> lvcreate -L 1G -n bla sata
>    Logical volume "bla" created.
The output from lvcreate really isn't useful for the same reasons that 
the output fro btrfs subvolume create isn't.  If the command isn't 
returning an error message, then it is implied that it succeeded, in 
which case it is not providing anything that wasn't already known unless 
you choose to not specify a LV name.  The fact that there is no way to 
make these commands silent short of redirection is a design flaw IMHO, 
not something we should be looking at seriously as a good example.
>
> And I think, not testing right now, that also mdadm is not silent on creating
> a softraid.
Again, this is a destructive operation, and you get useful information, 
and you can silence it with a command-line switch.
>
> So while I agree with you that regular shell commands (coreutils, util-linux)
> are silent on success usually this does not appear to be the case with storage
> related commands in GNU/Linux.
>
> I don´t have a clear oppinion about it other than I´d like to see some
> standard too. coreutils / util-linux both them to have some kind of standard,
> although not necessarily the same standard I bet. And I am not sure whether it
> is documented somewhere.
The general convention across most UNIX utilities is to print basic info 
about what's going on for complex operations (hence, LVM commands print 
info, but cp and rm don't) so that the user has some idea of progress. 
When -v or --verbose is specified, print lots of info (some commands 
allow multiple levels of verbosity, either with an argument to -v or by 
repeating it).  When -q or --quiet is specified, print almost nothing 
except errors (and possibly nothing at all, in which case errors get 
reported via the exit code).

There are many places that BTRFS doesn't follow this or is inconsistent 
internally.  We have no -q option on anything I can find.  We print out 
information for comparatively trivial operations (like creating a 
subvolume, which is a single ioctl, unlike lvcreate which involves at a 
minimum writing to disk and then calling dmsetup (lvcreate could still 
provide better info though)).  We have inconsistencies in how we handle 
management operations (scrub is asynchronous with the option to run in 
the foreground, balance is forced synchronous and can only be made 
asynchronous by methods external to BTRFS, but both have almost the same 
command syntax and both are kernel based operations).

As of right now, I see a few things to be addressed:
1) Add a -q option that makes commands only output stuff if there's an 
error.
2) Improve consistency (scrub/balance is just one example, there are 
other places where we are inconsistent).
3) Possibly get rid of the message on subvolume create (If a script is 
depending on that output, then it is not well written.  The output 
provides nothing the script shouldn't already know, and we give proper 
return codes for errors.  It should be expected still that output may 
change given the state of BTRFS.).
4) Possibly get rid of the message on subvolume delete (It provides no 
useful information at all, and it has no option to not error out on non 
existence of a subvolume.  In addition, the same arguments as for create 
apply here too.).

Re: btrfs-progs and btrfs(8) inconsistencies

[Goffredo Baroncelli]

On 2016-02-04 02:33, Qu Wenruo wrote:
> The idea itself makes a lot of sense.
> But I have at least two things to worry about:
> 
> 1) Old scripts backward compatibility
>     Especially xfstests. Maintainer will hate it a lot.
>     As we have changed it several times and broken existing test cases.
> 
>     Although personally I like to let all the backward compatibility
>     things go hell, but that's definitely not how things work. :(

we could change the name of the btrfs prog (like bfs or bctl  ?). 

If the command is called with the old name (btrfs) the old behavior is maintained; with the new name the new output is show if the specific sub command was updated; instead if the specific sub-command is not updated, the old output is show. 

We could allow a window of 1-year of transition where the new command will be in the alpha state where there is no any guarantee to be backward compatible, hoping that this time would be sufficient to reshape the output of all commands.

For the old command no update or enhancement should be allowed (with the exception of bugfix of course).

BR
G.Baroncelli

Re: btrfs-progs and btrfs(8) inconsistencies

[Chris Murphy]

On Wed, Feb 3, 2016 at 2:54 PM, Moviuro <moviuro@gmail.com> wrote:
> Hi all,
>
> I should have done this a long time ago, so here goes.
>
> Btrfs is certainly the best FS out there but btrfs(8) is a pain to use in
> scripts. That is, it talks way too much, exposes unparseable outputs and
> has inconsistent behavior.
>
> I strongly believe that the CLI may use a complete revamp, that is: think
> of the needed/wanted output. I actually ran into this frustrating behavior
> while writing butter (https://github.com/moviuro/butter).

What about interfacing with libbtrfs? There's also libblockdev-btrfs,
and storaged-btrfs (both are not upstream, but may benefit from a
strong libbtrfs if they use it.)


-- 
Chris Murphy

Re: btrfs-progs and btrfs(8) inconsistencies

[Chris Murphy]

On Thu, Feb 4, 2016 at 5:53 AM, Austin S. Hemmelgarn
<ahferroin7@gmail.com> wrote:


> 4) Possibly get rid of the message on subvolume delete (It provides no
> useful information at all, and it has no option to not error out on non
> existence of a subvolume.  In addition, the same arguments as for create
> apply here too.).

btrfs sub del has different commit modes that affect the user space
command's completion behavior, and output. So I wouldn't say it isn't
useful.


-- 
Chris Murphy

Re: btrfs-progs and btrfs(8) inconsistencies

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2191 bytes --]

On Thu, Feb 04, 2016 at 06:17:58PM +0100, Goffredo Baroncelli wrote:
> On 2016-02-04 02:33, Qu Wenruo wrote:
> > The idea itself makes a lot of sense.
> > But I have at least two things to worry about:
> > 
> > 1) Old scripts backward compatibility
> >     Especially xfstests. Maintainer will hate it a lot.
> >     As we have changed it several times and broken existing test cases.
> > 
> >     Although personally I like to let all the backward compatibility
> >     things go hell, but that's definitely not how things work. :(
> 
> we could change the name of the btrfs prog (like bfs or bctl  ?). 
> 
> If the command is called with the old name (btrfs) the old behavior is maintained; with the new name the new output is show if the specific sub command was updated; instead if the specific sub-command is not updated, the old output is show. 
> 
> We could allow a window of 1-year of transition where the new command will be in the alpha state where there is no any guarantee to be backward compatible, hoping that this time would be sufficient to reshape the output of all commands.
> 
> For the old command no update or enhancement should be allowed (with the exception of bugfix of course).

   If that's going to happen, can I strongly request that all of the
output selection options in btrfs sub list are dropped, and all of the
possible use cases enumerated, evaluated, and understood, before
producing a set of coherent and functional options that are actually
useful.

   There's a load of different ways that the command can be called,
and several of them aren't well supported. There's also different
questions what can be asked of the tool, and it's hard to work out
which switches to use for any given desired output (if it supports
them at all). This _really_ needs fixing.

   (I'm more than happy to do that hard thinking and write up a
detailed spec for it, BTW).

   Hugo.

-- 
Hugo Mills             | How do you become King? You stand in the marketplace
hugo@... carfax.org.uk | and announce you're going to tax everyone. If you
http://carfax.org.uk/  | get out alive, you're King.
PGP: E2AB1DE4          |                                        Harry Harrison

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: btrfs-progs and btrfs(8) inconsistencies

[Chris Murphy]

On Thu, Feb 4, 2016 at 10:17 AM, Goffredo Baroncelli <kreijack@inwind.it> wrote:
> On 2016-02-04 02:33, Qu Wenruo wrote:
>> The idea itself makes a lot of sense.
>> But I have at least two things to worry about:
>>
>> 1) Old scripts backward compatibility
>>     Especially xfstests. Maintainer will hate it a lot.
>>     As we have changed it several times and broken existing test cases.
>>
>>     Although personally I like to let all the backward compatibility
>>     things go hell, but that's definitely not how things work. :(
>
> we could change the name of the btrfs prog (like bfs or bctl  ?).

btrfsctl? :-)

Is there a big problem with just having a consistent "machine
readable" flag for the existing btrfs that achieves this goal? Why fix
bugs in two different places when this is really just a request to
have consistent output *formatting*?


> If the command is called with the old name (btrfs) the old behavior is maintained; with the new name the new output is show if the specific sub command was updated; instead if the specific sub-command is not updated, the old output is show.

If there's a new command, then it ought to be for scripts, not for
users. We are much better off breaking all scripts with a new way of
doing things going forward, than breaking users (again) with another
command change.


>
> We could allow a window of 1-year of transition where the new command will be in the alpha state where there is no any guarantee to be backward compatible, hoping that this time would be sufficient to reshape the output of all commands.
>

Well quite honestly I think bash scripts depending on user space tools
are subject to no guarantees anyway. How about an actual stable
API/ABI for developers to depend on if they need to interact with
Btrfs, rather than doing it indirectly through user space tools, which
are mainly for users after all.

-- 
Chris Murphy

Re: btrfs-progs and btrfs(8) inconsistencies

[Austin S. Hemmelgarn]

On 2016-02-04 14:40, Chris Murphy wrote:
> On Thu, Feb 4, 2016 at 5:53 AM, Austin S. Hemmelgarn
> <ahferroin7@gmail.com> wrote:
>
>
>> 4) Possibly get rid of the message on subvolume delete (It provides no
>> useful information at all, and it has no option to not error out on non
>> existence of a subvolume.  In addition, the same arguments as for create
>> apply here too.).
>
> btrfs sub del has different commit modes that affect the user space
> command's completion behavior, and output. So I wouldn't say it isn't
> useful.
>
Last I checked, those are controlled by using command line switches, 
which means that anyone invoking them knows what they're invoking 
because it's specified on the command line, therefore all info it 
currently provides in the non-error case is info you should already 
have.  I have no issue with it spitting out errors if something goes 
wrong, but it's just annoying having output that provides no information 
that isn't already known when everything goes right (think atd, cron, 
and almost any other periodic or deferred scheduling system).

Re: btrfs-progs and btrfs(8) inconsistencies

[Moviuro]

On Thu, Feb 4, 2016 at 9:28 PM Austin S. Hemmelgarn
<ahferroin7@gmail.com> wrote:
>
> On 2016-02-04 14:40, Chris Murphy wrote:
> > On Thu, Feb 4, 2016 at 5:53 AM, Austin S. Hemmelgarn
> > <ahferroin7@gmail.com> wrote:
> >
> >
> >> 4) Possibly get rid of the message on subvolume delete (It provides no
> >> useful information at all, and it has no option to not error out on non
> >> existence of a subvolume.  In addition, the same arguments as for create
> >> apply here too.).
> >
> > btrfs sub del has different commit modes that affect the user space
> > command's completion behavior, and output. So I wouldn't say it isn't
> > useful.
> >
> Last I checked, those are controlled by using command line switches,
> which means that anyone invoking them knows what they're invoking
> because it's specified on the command line, therefore all info it
> currently provides in the non-error case is info you should already
> have.  I have no issue with it spitting out errors if something goes
> wrong, but it's just annoying having output that provides no information
> that isn't already known when everything goes right (think atd, cron,
> and almost any other periodic or deferred scheduling system).

OK, so let's put our thinking into words.
But where do we start? the ML won't be a good place to do that IMHO.
Something like github may be easier to contribute to, and since it
won't be any real code at first, it shouldn't be a problem using a
platform like that, right? https://github.com/btrfs8-revamp
If you don't find that to be a good choice, I'm open to alternatives.

Re: btrfs-progs and btrfs(8) inconsistencies

[Anand Jain]

  If you look critically we have been using UI/CLI as API,
  IMO these two class of interfaces be distinct clearly.

  Btrfs needs library functions/APIs which is callable in
  popular scripting language like python.

Thanks, Anand


On 02/04/2016 05:54 AM, Moviuro wrote:
> Hi all,
>
> I should have done this a long time ago, so here goes.
>
> Btrfs is certainly the best FS out there but btrfs(8) is a pain to use in
> scripts. That is, it talks way too much, exposes unparseable outputs and
> has inconsistent behavior.
>
> I strongly believe that the CLI may use a complete revamp, that is: think
> of the needed/wanted output. I actually ran into this frustrating behavior
> while writing butter (https://github.com/moviuro/butter).
>
> Eg: # btrfs subvolume create foo
> Should be silent. I don't want btrfs(8) to bug me about successful
> commands. Everything that is a success is silent. In its current form,
> btrfs-subvolume(8) would spat out a complete sentence in English that we
> don't need, which however contains a tiny bit of useful info: the path
> (though relative) to the new subvolume. A -v/--verbose option might be here
> a good idea to implement and the following behavior (just thinking out
> loud, I'm sure there would be lots of things to think about more precisely):
>
> # btrfs subvolume create foo # should be silent
> # btrfs subvolume create -v foo
> /absolute/path/foo
> # btrfs subvolume create [--details|-vv] foo
> PATH:/absolute/path/foo
> UUID:082df386-98c2-44d9-9012-07fb2b22ea20
> [And whatnot]
>
> That's a first example. Same should go for all the commands. I have no idea
> how/where we could share about good/bad outputs (Google Drive? Framapad?
> git[hub|lab]?)
>
> Then come inconsistencies: compare the outputs of
> # btrfs sub create foo
> # btrfs sub delete foo
>
> This also needs to be taken care of, but it should be a by-product of
> thinking the outputs all over again.
>
> Behavior inconsistencies: some seemingly valid commands fail, because of
> some code that can be easily changed (I'm working on it in my spare time
> already, see https://github.com/moviuro/btrfs-progs/tree/getopts):
>
> The valid '--' doesn't work on every command, see
> 0aa796cad7ed3fce9d5964646a8f1f5a142b4989 here:
> https://github.com/kdave/btrfs-progs/commit/0aa796cad7ed3fce9d5964646a8f1f5a142b4989
>
>
> That's it. And since AFAICT btrfs is FLOSS, I'm sharing about my experience
> as user and hope my opinion will change btrfs(8) for the better.
>
> Cheers,
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: btrfs-progs and btrfs(8) inconsistencies

[Austin S. Hemmelgarn]

On 2016-02-04 22:11, Anand Jain wrote:
>
>
>   If you look critically we have been using UI/CLI as API,
>   IMO these two class of interfaces be distinct clearly.
>
>   Btrfs needs library functions/APIs which is callable in
>   popular scripting language like python.
>
I wholeheartedly agree.  At a minimum, we need to get our C API properly 
in line.  I'm not certain how closely libbtrfs matches the btrfs CLI, 
but I'm pretty sure we have all the functionality needed for language 
bindings there, we just need to solidify the API (or at least figure out 
some way to properly version it).  One thing I would love to see added 
to both libbtrfs and the CLI though is some way to make EXTENT_SAME 
ioctl calls on arbitrary extents, there's no way to do this without 
writing your own program, and calling arbitrary ioctls from Python or 
Java or any number of other languages is decidedly non-trivial compared 
to C or C++.

Re: btrfs-progs and btrfs(8) inconsistencies

[Austin S. Hemmelgarn]

On 2016-02-04 15:40, Moviuro wrote:
> On Thu, Feb 4, 2016 at 9:28 PM Austin S. Hemmelgarn
> <ahferroin7@gmail.com> wrote:
>>
>> On 2016-02-04 14:40, Chris Murphy wrote:
>>> On Thu, Feb 4, 2016 at 5:53 AM, Austin S. Hemmelgarn
>>> <ahferroin7@gmail.com> wrote:
>>>
>>>
>>>> 4) Possibly get rid of the message on subvolume delete (It provides no
>>>> useful information at all, and it has no option to not error out on non
>>>> existence of a subvolume.  In addition, the same arguments as for create
>>>> apply here too.).
>>>
>>> btrfs sub del has different commit modes that affect the user space
>>> command's completion behavior, and output. So I wouldn't say it isn't
>>> useful.
>>>
>> Last I checked, those are controlled by using command line switches,
>> which means that anyone invoking them knows what they're invoking
>> because it's specified on the command line, therefore all info it
>> currently provides in the non-error case is info you should already
>> have.  I have no issue with it spitting out errors if something goes
>> wrong, but it's just annoying having output that provides no information
>> that isn't already known when everything goes right (think atd, cron,
>> and almost any other periodic or deferred scheduling system).
>
> OK, so let's put our thinking into words.
> But where do we start? the ML won't be a good place to do that IMHO.
> Something like github may be easier to contribute to, and since it
> won't be any real code at first, it shouldn't be a problem using a
> platform like that, right? https://github.com/btrfs8-revamp
> If you don't find that to be a good choice, I'm open to alternatives.
>
For the spec itself, I think Github is a good idea.  But I feel we need 
to get actual user input before we start writing the spec.  One of the 
worst things these days about UI design is that quite often the 
designers get no actual user input until after they finalize the design, 
and I would _really_ like to avoid that happening, because when it 
happens it often either kills a project, or makes it so difficult to use 
that people don't want to use it outside of a pre-built system.

Re: btrfs-progs and btrfs(8) inconsistencies

[Chris Murphy]

On Thu, Feb 4, 2016 at 8:11 PM, Anand Jain <anand.jain@oracle.com> wrote:
>
>
>  If you look critically we have been using UI/CLI as API,
>  IMO these two class of interfaces be distinct clearly.
>
>  Btrfs needs library functions/APIs which is callable in
>  popular scripting language like python.

I'm not sure what this is using to get btrfs info, it's written in c.
https://github.com/storaged-project/storaged/tree/master/modules/btrfs

This is python and has a btrfs backed that depends on user space
tools, the comments pretty clearly express the challenges with using
user space tools as a primary means to interface with Btrfs volumes.
http://sourceforge.net/p/storagemanager/code/ci/devel/tree/ssmlib/backends/btrfs.py

Red Hat's installer, Anaconda, has split out its storage configuration
module into python-blivet to enable a bunch of storage work to be done
in Python, including a lot of Btrfs functionality. Some of this just
got too complicated I guess for blivet, and has been split out into
libblockdev, which includes plugin apis, and there happens to be one
for Btrfs. But I have no idea where it goes from there.
https://github.com/rhinstaller/libblockdev
https://github.com/rhinstaller/libblockdev/blob/master/src/lib/plugin_apis/btrfs.api

So yeah, people are trying to do this.


-- 
Chris Murphy

Re: btrfs-progs and btrfs(8) inconsistencies

[Qu Wenruo]

On 02/05/2016 11:11 AM, Anand Jain wrote:
>
>
>   If you look critically we have been using UI/CLI as API,
>   IMO these two class of interfaces be distinct clearly.
>
>   Btrfs needs library functions/APIs which is callable in
>   popular scripting language like python.
>
> Thanks, Anand
>
+1 too.

But first in C, then python wrapper.

Not sure why there is no such libbtrfs for C wrapper of btrfs ioctls.

Maybe just because current btrfs ioctl is too easy to use?

AFAIK, systemd (container storage part) calls btrfs ioctl directly, 
especially for subvolume/snapshot creation mainly because the design of 
them are quite easy.

Thanks,
Qu

>
> On 02/04/2016 05:54 AM, Moviuro wrote:
>> Hi all,
>>
>> I should have done this a long time ago, so here goes.
>>
>> Btrfs is certainly the best FS out there but btrfs(8) is a pain to use in
>> scripts. That is, it talks way too much, exposes unparseable outputs and
>> has inconsistent behavior.
>>
>> I strongly believe that the CLI may use a complete revamp, that is: think
>> of the needed/wanted output. I actually ran into this frustrating
>> behavior
>> while writing butter (https://github.com/moviuro/butter).
>>
>> Eg: # btrfs subvolume create foo
>> Should be silent. I don't want btrfs(8) to bug me about successful
>> commands. Everything that is a success is silent. In its current form,
>> btrfs-subvolume(8) would spat out a complete sentence in English that we
>> don't need, which however contains a tiny bit of useful info: the path
>> (though relative) to the new subvolume. A -v/--verbose option might be
>> here
>> a good idea to implement and the following behavior (just thinking out
>> loud, I'm sure there would be lots of things to think about more
>> precisely):
>>
>> # btrfs subvolume create foo # should be silent
>> # btrfs subvolume create -v foo
>> /absolute/path/foo
>> # btrfs subvolume create [--details|-vv] foo
>> PATH:/absolute/path/foo
>> UUID:082df386-98c2-44d9-9012-07fb2b22ea20
>> [And whatnot]
>>
>> That's a first example. Same should go for all the commands. I have no
>> idea
>> how/where we could share about good/bad outputs (Google Drive? Framapad?
>> git[hub|lab]?)
>>
>> Then come inconsistencies: compare the outputs of
>> # btrfs sub create foo
>> # btrfs sub delete foo
>>
>> This also needs to be taken care of, but it should be a by-product of
>> thinking the outputs all over again.
>>
>> Behavior inconsistencies: some seemingly valid commands fail, because of
>> some code that can be easily changed (I'm working on it in my spare time
>> already, see https://github.com/moviuro/btrfs-progs/tree/getopts):
>>
>> The valid '--' doesn't work on every command, see
>> 0aa796cad7ed3fce9d5964646a8f1f5a142b4989 here:
>> https://github.com/kdave/btrfs-progs/commit/0aa796cad7ed3fce9d5964646a8f1f5a142b4989
>>
>>
>>
>> That's it. And since AFAICT btrfs is FLOSS, I'm sharing about my
>> experience
>> as user and hope my opinion will change btrfs(8) for the better.
>>
>> Cheers,
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs-progs and btrfs(8) inconsistencies

[Goffredo Baroncelli]

On 2016-02-07 11:07, Qu Wenruo wrote:
> +1 too.
> 
> But first in C, then python wrapper.
> 
> Not sure why there is no such libbtrfs for C wrapper of btrfs ioctls.
> 
> Maybe just because current btrfs ioctl is too easy to use?

Unfortunately no. 

I think the main problem of writing a libbtrfs, is the code changes needed. 
In fact in order to avoid code duplication (and to have more testing), the btrfs command has to be a client of libbtrfs. But it is not easy: you have to refactoring the error handling and the message printing...

> AFAIK, systemd (container storage part) calls btrfs ioctl directly, especially for subvolume/snapshot creation mainly > because the design of them are quite easy.

Also udev does the same (I don't remember if this happened before or after it went under the systemd umbrella).

I remember a case where it was impossible to change the semantic of BTRFS_IOC_DEVICES_READY: on the basis of its name, it could be seems that this ioctl should _only_ check if a filesystem is ready. Unfortunately it register also the device [1].
It was discussed in the ML to change its semantic, but having already widespread client  like udev it was impossible to do (even tough we could discuss to change their name)


> Thanks,
> Qu 


BR
G.Baroncelli


[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=772744

Re: btrfs-progs and btrfs(8) inconsistencies

[David Sterba]

Let me answer here for the whole thread. There are lot of issues to
cover, I think the github way would work here, so the project you've
created for the purpose seems ok to me.

https://github.com/btrfs8-revamp

I assume way more dicussions than actual patches and coding, the web
platform offers commenting at the exact locations of files, issue
tracking works better than over mail.

Initially we'd cover all the stuff mentioned in this thread, naming,
conventions etc. After we agree we can start changing code in btrfs-progs.

We can ask for feedback wider audience in the mailinglist if needed,
possibly with a draft that would be reviewed already.

After we write comprehensive guidelines for the UI, we can add it to
btrfs-progs, I really want to avoid high churn in that repository for
things that are not related to code. For that the revamp repository is
good, the only requirement is a github account.

Re: btrfs-progs and btrfs(8) inconsistencies

[Moviuro]

Don't hesitate to drop an email here (with your Github account
name...) to get invited into the Github group. That's because you
can't ask to join a group on GitHub.
At the moment, kdave and myself are administrators of the said group
(https://github.com/btrfs8-revamp). I'm sure we can nominate David
too, once he joined ;-)

PS: I really didn't expect my first mail to cause this movement, I'm
glad it took this shape!