* btrfs-progs and btrfs(8) inconsistencies
@ 2016-02-03 21:54 Moviuro
2016-02-04 1:33 ` Qu Wenruo
` (3 more replies)
0 siblings, 4 replies; 22+ messages in thread
From: Moviuro @ 2016-02-03 21:54 UTC (permalink / raw)
To: linux-btrfs
Hi all,
I should have done this a long time ago, so here goes.
Btrfs is certainly the best FS out there but btrfs(8) is a pain to use in
scripts. That is, it talks way too much, exposes unparseable outputs and
has inconsistent behavior.
I strongly believe that the CLI may use a complete revamp, that is: think
of the needed/wanted output. I actually ran into this frustrating behavior
while writing butter (https://github.com/moviuro/butter).
Eg: # btrfs subvolume create foo
Should be silent. I don't want btrfs(8) to bug me about successful
commands. Everything that is a success is silent. In its current form,
btrfs-subvolume(8) would spat out a complete sentence in English that we
don't need, which however contains a tiny bit of useful info: the path
(though relative) to the new subvolume. A -v/--verbose option might be here
a good idea to implement and the following behavior (just thinking out
loud, I'm sure there would be lots of things to think about more precisely):
# btrfs subvolume create foo # should be silent
# btrfs subvolume create -v foo
/absolute/path/foo
# btrfs subvolume create [--details|-vv] foo
PATH:/absolute/path/foo
UUID:082df386-98c2-44d9-9012-07fb2b22ea20
[And whatnot]
That's a first example. Same should go for all the commands. I have no idea
how/where we could share about good/bad outputs (Google Drive? Framapad?
git[hub|lab]?)
Then come inconsistencies: compare the outputs of
# btrfs sub create foo
# btrfs sub delete foo
This also needs to be taken care of, but it should be a by-product of
thinking the outputs all over again.
Behavior inconsistencies: some seemingly valid commands fail, because of
some code that can be easily changed (I'm working on it in my spare time
already, see https://github.com/moviuro/btrfs-progs/tree/getopts):
The valid '--' doesn't work on every command, see
0aa796cad7ed3fce9d5964646a8f1f5a142b4989 here:
https://github.com/kdave/btrfs-progs/commit/0aa796cad7ed3fce9d5964646a8f1f5a142b4989
That's it. And since AFAICT btrfs is FLOSS, I'm sharing about my experience
as user and hope my opinion will change btrfs(8) for the better.
Cheers,
^ permalink raw reply [flat|nested] 22+ messages in thread* Re: btrfs-progs and btrfs(8) inconsistencies 2016-02-03 21:54 btrfs-progs and btrfs(8) inconsistencies Moviuro @ 2016-02-04 1:33 ` Qu Wenruo 2016-02-04 8:57 ` Moviuro 2016-02-04 17:17 ` Goffredo Baroncelli 2016-02-04 18:22 ` Chris Murphy ` (2 subsequent siblings) 3 siblings, 2 replies; 22+ messages in thread From: Qu Wenruo @ 2016-02-04 1:33 UTC (permalink / raw) To: Moviuro, linux-btrfs Moviuro wrote on 2016/02/03 22:54 +0100: > Hi all, > > I should have done this a long time ago, so here goes. > > Btrfs is certainly the best FS out there but btrfs(8) is a pain to use in > scripts. That is, it talks way too much, exposes unparseable outputs and > has inconsistent behavior. > > I strongly believe that the CLI may use a complete revamp, that is: think > of the needed/wanted output. I actually ran into this frustrating behavior > while writing butter (https://github.com/moviuro/butter). > > Eg: # btrfs subvolume create foo > Should be silent. I don't want btrfs(8) to bug me about successful > commands. Everything that is a success is silent. In its current form, > btrfs-subvolume(8) would spat out a complete sentence in English that we > don't need, which however contains a tiny bit of useful info: the path > (though relative) to the new subvolume. A -v/--verbose option might be here > a good idea to implement and the following behavior (just thinking out > loud, I'm sure there would be lots of things to think about more precisely): > > # btrfs subvolume create foo # should be silent > # btrfs subvolume create -v foo > /absolute/path/foo > # btrfs subvolume create [--details|-vv] foo > PATH:/absolute/path/foo > UUID:082df386-98c2-44d9-9012-07fb2b22ea20 > [And whatnot] The idea itself makes a lot of sense. But I have at least two things to worry about: 1) Old scripts backward compatibility Especially xfstests. Maintainer will hate it a lot. As we have changed it several times and broken existing test cases. Although personally I like to let all the backward compatibility things go hell, but that's definitely not how things work. :( 2) End-user taste. Some end-users like such info as feedback of success. Of course other users like it act as silent as possible. > > That's a first example. Same should go for all the commands. I have no idea > how/where we could share about good/bad outputs (Google Drive? Framapad? > git[hub|lab]?) Maybe it's overkilling, but I like the idea to have a good example about how CLI interface should be designed. But it may be harder for all developer/reviewer to follow that restrict example. And even more, I hope there will be a nice btrfs-progs CLI design guideline. (Although it's surely overkilling) Thanks, Qu > > Then come inconsistencies: compare the outputs of > # btrfs sub create foo > # btrfs sub delete foo > > This also needs to be taken care of, but it should be a by-product of > thinking the outputs all over again. > > Behavior inconsistencies: some seemingly valid commands fail, because of > some code that can be easily changed (I'm working on it in my spare time > already, see https://github.com/moviuro/btrfs-progs/tree/getopts): > > The valid '--' doesn't work on every command, see > 0aa796cad7ed3fce9d5964646a8f1f5a142b4989 here: > https://github.com/kdave/btrfs-progs/commit/0aa796cad7ed3fce9d5964646a8f1f5a142b4989 > > > That's it. And since AFAICT btrfs is FLOSS, I'm sharing about my experience > as user and hope my opinion will change btrfs(8) for the better. > > Cheers, > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: btrfs-progs and btrfs(8) inconsistencies 2016-02-04 1:33 ` Qu Wenruo @ 2016-02-04 8:57 ` Moviuro 2016-02-04 9:15 ` Qu Wenruo 2016-02-04 10:14 ` Martin Steigerwald 2016-02-04 17:17 ` Goffredo Baroncelli 1 sibling, 2 replies; 22+ messages in thread From: Moviuro @ 2016-02-04 8:57 UTC (permalink / raw) To: Qu Wenruo; +Cc: linux-btrfs On Thu, Feb 4, 2016 at 2:33 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote: > > > Moviuro wrote on 2016/02/03 22:54 +0100: >> >> Hi all, >> >> ... >> >> # btrfs subvolume create foo # should be silent >> # btrfs subvolume create -v foo >> /absolute/path/foo >> # btrfs subvolume create [--details|-vv] foo >> PATH:/absolute/path/foo >> UUID:082df386-98c2-44d9-9012-07fb2b22ea20 >> [And whatnot] > > > The idea itself makes a lot of sense. > But I have at least two things to worry about: > > 1) Old scripts backward compatibility > Especially xfstests. Maintainer will hate it a lot. > As we have changed it several times and broken existing test cases. Right now, I can think of the following: - add a "backward-compatibility switch", like -o|--old, which would have no effect on the output on version 4.X. - add a "use the new behavior switch", like -n|--new, which would have no effect starting with the next big release (5.X) > Although personally I like to let all the backward compatibility > things go hell, but that's definitely not how things work. :( > > 2) End-user taste. > Some end-users like such info as feedback of success. > Of course other users like it act as silent as possible. I'm pretty sure that's... not the case. Almost everything on GNU/Linux is silent. cd(1) is silent, cp(1) is silent, rm(1)... What they all have though is a -v|--verbose switch. >> >> That's a first example. Same should go for all the commands. I have no >> idea >> how/where we could share about good/bad outputs (Google Drive? Framapad? >> git[hub|lab]?) > > > Maybe it's overkilling, but I like the idea to have a good example about how > CLI interface should be designed. > But it may be harder for all developer/reviewer to follow that restrict > example. > > And even more, I hope there will be a nice btrfs-progs CLI design guideline. > (Although it's surely overkilling) What kind of format do you think we should write this in? drop a stylesheet.md in the repo? > Thanks, > Qu >> Then come inconsistencies: compare the outputs of >> ... > ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: btrfs-progs and btrfs(8) inconsistencies 2016-02-04 8:57 ` Moviuro @ 2016-02-04 9:15 ` Qu Wenruo 2016-02-04 10:14 ` Martin Steigerwald 1 sibling, 0 replies; 22+ messages in thread From: Qu Wenruo @ 2016-02-04 9:15 UTC (permalink / raw) To: Moviuro; +Cc: linux-btrfs Moviuro wrote on 2016/02/04 09:57 +0100: > On Thu, Feb 4, 2016 at 2:33 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote: >> >> >> Moviuro wrote on 2016/02/03 22:54 +0100: >>> >>> Hi all, >>> >>> ... >>> >>> # btrfs subvolume create foo # should be silent >>> # btrfs subvolume create -v foo >>> /absolute/path/foo >>> # btrfs subvolume create [--details|-vv] foo >>> PATH:/absolute/path/foo >>> UUID:082df386-98c2-44d9-9012-07fb2b22ea20 >>> [And whatnot] >> >> >> The idea itself makes a lot of sense. >> But I have at least two things to worry about: >> >> 1) Old scripts backward compatibility >> Especially xfstests. Maintainer will hate it a lot. >> As we have changed it several times and broken existing test cases. > Right now, I can think of the following: > - add a "backward-compatibility switch", like -o|--old, which would > have no effect on the output on version 4.X. > - add a "use the new behavior switch", like -n|--new, which would have > no effect starting with the next big release (5.X) > >> Although personally I like to let all the backward compatibility >> things go hell, but that's definitely not how things work. :( >> >> 2) End-user taste. >> Some end-users like such info as feedback of success. >> Of course other users like it act as silent as possible. > I'm pretty sure that's... not the case. Almost everything on GNU/Linux > is silent. cd(1) is silent, cp(1) is silent, rm(1)... > What they all have though is a -v|--verbose switch. > >>> >>> That's a first example. Same should go for all the commands. I have no >>> idea >>> how/where we could share about good/bad outputs (Google Drive? Framapad? >>> git[hub|lab]?) >> >> >> Maybe it's overkilling, but I like the idea to have a good example about how >> CLI interface should be designed. >> But it may be harder for all developer/reviewer to follow that restrict >> example. >> >> And even more, I hope there will be a nice btrfs-progs CLI design guideline. >> (Although it's surely overkilling) > What kind of format do you think we should write this in? drop a > stylesheet.md in the repo? Markdown is good enough for me. But I'm not sure where to put it. Btrfs wiki? Btrfs-progs repo? Maybe you can write one and put it in btrfs-progs, and send out as a patch for us to review? Thanks, Qu > >> Thanks, >> Qu >>> Then come inconsistencies: compare the outputs of >>> ... >> > > ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: btrfs-progs and btrfs(8) inconsistencies 2016-02-04 8:57 ` Moviuro 2016-02-04 9:15 ` Qu Wenruo @ 2016-02-04 10:14 ` Martin Steigerwald 2016-02-04 12:04 ` Moviuro 2016-02-04 12:53 ` Austin S. Hemmelgarn 1 sibling, 2 replies; 22+ messages in thread From: Martin Steigerwald @ 2016-02-04 10:14 UTC (permalink / raw) To: Moviuro; +Cc: Qu Wenruo, linux-btrfs Am Donnerstag, 4. Februar 2016, 09:57:54 CET schrieb Moviuro: > > Although personally I like to let all the backward compatibility > > things go hell, but that's definitely not how things work. :( > > > > 2) End-user taste. > > Some end-users like such info as feedback of success. > > Of course other users like it act as silent as possible. > > I'm pretty sure that's... not the case. Almost everything on GNU/Linux > is silent. cd(1) is silent, cp(1) is silent, rm(1)... > What they all have though is a -v|--verbose switch. The various mkfs commands are not. Not one of them I know of. Additionally each one gives a different output. pvcreate, vgcreate, lvcreate and as well as the remove commands and probably other LVM commands are not (no one could argue, that from their ideas they come from HP/UX, but thats a Unix as well): merkaba:~> lvcreate -L 1G -n bla sata Logical volume "bla" created. And I think, not testing right now, that also mdadm is not silent on creating a softraid. So while I agree with you that regular shell commands (coreutils, util-linux) are silent on success usually this does not appear to be the case with storage related commands in GNU/Linux. I don´t have a clear oppinion about it other than I´d like to see some standard too. coreutils / util-linux both them to have some kind of standard, although not necessarily the same standard I bet. And I am not sure whether it is documented somewhere. -- Martin ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: btrfs-progs and btrfs(8) inconsistencies 2016-02-04 10:14 ` Martin Steigerwald @ 2016-02-04 12:04 ` Moviuro 2016-02-04 12:53 ` Austin S. Hemmelgarn 1 sibling, 0 replies; 22+ messages in thread From: Moviuro @ 2016-02-04 12:04 UTC (permalink / raw) To: Martin Steigerwald; +Cc: Qu Wenruo, linux-btrfs On Thu, Feb 4, 2016 at 11:14 AM, Martin Steigerwald <martin@lichtvoll.de> wrote: > Am Donnerstag, 4. Februar 2016, 09:57:54 CET schrieb Moviuro: >> > Although personally I like to let all the backward compatibility >> > things go hell, but that's definitely not how things work. :( >> > >> > 2) End-user taste. >> > Some end-users like such info as feedback of success. >> > Of course other users like it act as silent as possible. >> >> I'm pretty sure that's... not the case. Almost everything on GNU/Linux >> is silent. cd(1) is silent, cp(1) is silent, rm(1)... >> What they all have though is a -v|--verbose switch. > > The various mkfs commands are not. Not one of them I know of. Additionally > each one gives a different output. > > pvcreate, vgcreate, lvcreate and as well as the remove commands and probably > other LVM commands are not (no one could argue, that from their ideas they > come from HP/UX, but thats a Unix as well): > > merkaba:~> lvcreate -L 1G -n bla sata > Logical volume "bla" created. > > And I think, not testing right now, that also mdadm is not silent on creating > a softraid. > > So while I agree with you that regular shell commands (coreutils, util-linux) > are silent on success usually this does not appear to be the case with storage > related commands in GNU/Linux. To be perfectly honnest, my inspiration comes from zfs(8) on FreeBSD. I'm pretty sure there is some inspiration we can find in its behavior. (see https://www.freebsd.org/cgi/man.cgi?zfs(8) ) The output has both a human-readable form (using column(1)) and a machine-parseable form (e.g. zfs get -o value used /zroot). The output is limited to what is truly useful, etc. I have not had any previous experience with any other FS or softraid solutions, so I'm not familiar with those commands you gave, Martin. Also, making silence a default option helps spare some typing: >/dev/null : 11 char (force silence) -v : 3 char (force verbose) (it's also far more readable) I should start working on said guidelines.md later today or this weekend. > I don´t have a clear oppinion about it other than I´d like to see some > standard too. coreutils / util-linux both them to have some kind of standard, > although not necessarily the same standard I bet. And I am not sure whether it > is documented somewhere. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: btrfs-progs and btrfs(8) inconsistencies 2016-02-04 10:14 ` Martin Steigerwald 2016-02-04 12:04 ` Moviuro @ 2016-02-04 12:53 ` Austin S. Hemmelgarn 2016-02-04 19:40 ` Chris Murphy 1 sibling, 1 reply; 22+ messages in thread From: Austin S. Hemmelgarn @ 2016-02-04 12:53 UTC (permalink / raw) To: Martin Steigerwald, Moviuro; +Cc: Qu Wenruo, linux-btrfs On 2016-02-04 05:14, Martin Steigerwald wrote: > Am Donnerstag, 4. Februar 2016, 09:57:54 CET schrieb Moviuro: >>> Although personally I like to let all the backward compatibility >>> things go hell, but that's definitely not how things work. :( >>> >>> 2) End-user taste. >>> Some end-users like such info as feedback of success. >>> Of course other users like it act as silent as possible. >> >> I'm pretty sure that's... not the case. Almost everything on GNU/Linux >> is silent. cd(1) is silent, cp(1) is silent, rm(1)... >> What they all have though is a -v|--verbose switch. > > The various mkfs commands are not. Not one of them I know of. Additionally > each one gives a different output. This is kind of understandable, it's an inherently destructive operation, and the information they spit out is almost always useful. On top of that, except for certain very specific usage, you don't run mkfs every day, while it's a lot more likely that you use btrfs subvolume commands with a high regularity. There are also a number of mkfs commands that let you silence the output without using shell redirection, which is a good thing. > > pvcreate, vgcreate, lvcreate and as well as the remove commands and probably > other LVM commands are not (no one could argue, that from their ideas they > come from HP/UX, but thats a Unix as well): > > merkaba:~> lvcreate -L 1G -n bla sata > Logical volume "bla" created. The output from lvcreate really isn't useful for the same reasons that the output fro btrfs subvolume create isn't. If the command isn't returning an error message, then it is implied that it succeeded, in which case it is not providing anything that wasn't already known unless you choose to not specify a LV name. The fact that there is no way to make these commands silent short of redirection is a design flaw IMHO, not something we should be looking at seriously as a good example. > > And I think, not testing right now, that also mdadm is not silent on creating > a softraid. Again, this is a destructive operation, and you get useful information, and you can silence it with a command-line switch. > > So while I agree with you that regular shell commands (coreutils, util-linux) > are silent on success usually this does not appear to be the case with storage > related commands in GNU/Linux. > > I don´t have a clear oppinion about it other than I´d like to see some > standard too. coreutils / util-linux both them to have some kind of standard, > although not necessarily the same standard I bet. And I am not sure whether it > is documented somewhere. The general convention across most UNIX utilities is to print basic info about what's going on for complex operations (hence, LVM commands print info, but cp and rm don't) so that the user has some idea of progress. When -v or --verbose is specified, print lots of info (some commands allow multiple levels of verbosity, either with an argument to -v or by repeating it). When -q or --quiet is specified, print almost nothing except errors (and possibly nothing at all, in which case errors get reported via the exit code). There are many places that BTRFS doesn't follow this or is inconsistent internally. We have no -q option on anything I can find. We print out information for comparatively trivial operations (like creating a subvolume, which is a single ioctl, unlike lvcreate which involves at a minimum writing to disk and then calling dmsetup (lvcreate could still provide better info though)). We have inconsistencies in how we handle management operations (scrub is asynchronous with the option to run in the foreground, balance is forced synchronous and can only be made asynchronous by methods external to BTRFS, but both have almost the same command syntax and both are kernel based operations). As of right now, I see a few things to be addressed: 1) Add a -q option that makes commands only output stuff if there's an error. 2) Improve consistency (scrub/balance is just one example, there are other places where we are inconsistent). 3) Possibly get rid of the message on subvolume create (If a script is depending on that output, then it is not well written. The output provides nothing the script shouldn't already know, and we give proper return codes for errors. It should be expected still that output may change given the state of BTRFS.). 4) Possibly get rid of the message on subvolume delete (It provides no useful information at all, and it has no option to not error out on non existence of a subvolume. In addition, the same arguments as for create apply here too.). ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: btrfs-progs and btrfs(8) inconsistencies 2016-02-04 12:53 ` Austin S. Hemmelgarn @ 2016-02-04 19:40 ` Chris Murphy 2016-02-04 20:19 ` Austin S. Hemmelgarn 0 siblings, 1 reply; 22+ messages in thread From: Chris Murphy @ 2016-02-04 19:40 UTC (permalink / raw) To: Btrfs BTRFS On Thu, Feb 4, 2016 at 5:53 AM, Austin S. Hemmelgarn <ahferroin7@gmail.com> wrote: > 4) Possibly get rid of the message on subvolume delete (It provides no > useful information at all, and it has no option to not error out on non > existence of a subvolume. In addition, the same arguments as for create > apply here too.). btrfs sub del has different commit modes that affect the user space command's completion behavior, and output. So I wouldn't say it isn't useful. -- Chris Murphy ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: btrfs-progs and btrfs(8) inconsistencies 2016-02-04 19:40 ` Chris Murphy @ 2016-02-04 20:19 ` Austin S. Hemmelgarn 2016-02-04 20:40 ` Moviuro 0 siblings, 1 reply; 22+ messages in thread From: Austin S. Hemmelgarn @ 2016-02-04 20:19 UTC (permalink / raw) To: Chris Murphy, Btrfs BTRFS On 2016-02-04 14:40, Chris Murphy wrote: > On Thu, Feb 4, 2016 at 5:53 AM, Austin S. Hemmelgarn > <ahferroin7@gmail.com> wrote: > > >> 4) Possibly get rid of the message on subvolume delete (It provides no >> useful information at all, and it has no option to not error out on non >> existence of a subvolume. In addition, the same arguments as for create >> apply here too.). > > btrfs sub del has different commit modes that affect the user space > command's completion behavior, and output. So I wouldn't say it isn't > useful. > Last I checked, those are controlled by using command line switches, which means that anyone invoking them knows what they're invoking because it's specified on the command line, therefore all info it currently provides in the non-error case is info you should already have. I have no issue with it spitting out errors if something goes wrong, but it's just annoying having output that provides no information that isn't already known when everything goes right (think atd, cron, and almost any other periodic or deferred scheduling system). ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: btrfs-progs and btrfs(8) inconsistencies 2016-02-04 20:19 ` Austin S. Hemmelgarn @ 2016-02-04 20:40 ` Moviuro 2016-02-05 13:04 ` Austin S. Hemmelgarn 0 siblings, 1 reply; 22+ messages in thread From: Moviuro @ 2016-02-04 20:40 UTC (permalink / raw) To: Austin S. Hemmelgarn; +Cc: Chris Murphy, Btrfs BTRFS On Thu, Feb 4, 2016 at 9:28 PM Austin S. Hemmelgarn <ahferroin7@gmail.com> wrote: > > On 2016-02-04 14:40, Chris Murphy wrote: > > On Thu, Feb 4, 2016 at 5:53 AM, Austin S. Hemmelgarn > > <ahferroin7@gmail.com> wrote: > > > > > >> 4) Possibly get rid of the message on subvolume delete (It provides no > >> useful information at all, and it has no option to not error out on non > >> existence of a subvolume. In addition, the same arguments as for create > >> apply here too.). > > > > btrfs sub del has different commit modes that affect the user space > > command's completion behavior, and output. So I wouldn't say it isn't > > useful. > > > Last I checked, those are controlled by using command line switches, > which means that anyone invoking them knows what they're invoking > because it's specified on the command line, therefore all info it > currently provides in the non-error case is info you should already > have. I have no issue with it spitting out errors if something goes > wrong, but it's just annoying having output that provides no information > that isn't already known when everything goes right (think atd, cron, > and almost any other periodic or deferred scheduling system). OK, so let's put our thinking into words. But where do we start? the ML won't be a good place to do that IMHO. Something like github may be easier to contribute to, and since it won't be any real code at first, it shouldn't be a problem using a platform like that, right? https://github.com/btrfs8-revamp If you don't find that to be a good choice, I'm open to alternatives. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: btrfs-progs and btrfs(8) inconsistencies 2016-02-04 20:40 ` Moviuro @ 2016-02-05 13:04 ` Austin S. Hemmelgarn 0 siblings, 0 replies; 22+ messages in thread From: Austin S. Hemmelgarn @ 2016-02-05 13:04 UTC (permalink / raw) To: Moviuro; +Cc: Chris Murphy, Btrfs BTRFS On 2016-02-04 15:40, Moviuro wrote: > On Thu, Feb 4, 2016 at 9:28 PM Austin S. Hemmelgarn > <ahferroin7@gmail.com> wrote: >> >> On 2016-02-04 14:40, Chris Murphy wrote: >>> On Thu, Feb 4, 2016 at 5:53 AM, Austin S. Hemmelgarn >>> <ahferroin7@gmail.com> wrote: >>> >>> >>>> 4) Possibly get rid of the message on subvolume delete (It provides no >>>> useful information at all, and it has no option to not error out on non >>>> existence of a subvolume. In addition, the same arguments as for create >>>> apply here too.). >>> >>> btrfs sub del has different commit modes that affect the user space >>> command's completion behavior, and output. So I wouldn't say it isn't >>> useful. >>> >> Last I checked, those are controlled by using command line switches, >> which means that anyone invoking them knows what they're invoking >> because it's specified on the command line, therefore all info it >> currently provides in the non-error case is info you should already >> have. I have no issue with it spitting out errors if something goes >> wrong, but it's just annoying having output that provides no information >> that isn't already known when everything goes right (think atd, cron, >> and almost any other periodic or deferred scheduling system). > > OK, so let's put our thinking into words. > But where do we start? the ML won't be a good place to do that IMHO. > Something like github may be easier to contribute to, and since it > won't be any real code at first, it shouldn't be a problem using a > platform like that, right? https://github.com/btrfs8-revamp > If you don't find that to be a good choice, I'm open to alternatives. > For the spec itself, I think Github is a good idea. But I feel we need to get actual user input before we start writing the spec. One of the worst things these days about UI design is that quite often the designers get no actual user input until after they finalize the design, and I would _really_ like to avoid that happening, because when it happens it often either kills a project, or makes it so difficult to use that people don't want to use it outside of a pre-built system. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: btrfs-progs and btrfs(8) inconsistencies 2016-02-04 1:33 ` Qu Wenruo 2016-02-04 8:57 ` Moviuro @ 2016-02-04 17:17 ` Goffredo Baroncelli 2016-02-04 19:48 ` Hugo Mills 2016-02-04 20:10 ` Chris Murphy 1 sibling, 2 replies; 22+ messages in thread From: Goffredo Baroncelli @ 2016-02-04 17:17 UTC (permalink / raw) To: Qu Wenruo, Moviuro, linux-btrfs On 2016-02-04 02:33, Qu Wenruo wrote: > The idea itself makes a lot of sense. > But I have at least two things to worry about: > > 1) Old scripts backward compatibility > Especially xfstests. Maintainer will hate it a lot. > As we have changed it several times and broken existing test cases. > > Although personally I like to let all the backward compatibility > things go hell, but that's definitely not how things work. :( we could change the name of the btrfs prog (like bfs or bctl ?). If the command is called with the old name (btrfs) the old behavior is maintained; with the new name the new output is show if the specific sub command was updated; instead if the specific sub-command is not updated, the old output is show. We could allow a window of 1-year of transition where the new command will be in the alpha state where there is no any guarantee to be backward compatible, hoping that this time would be sufficient to reshape the output of all commands. For the old command no update or enhancement should be allowed (with the exception of bugfix of course). BR G.Baroncelli ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: btrfs-progs and btrfs(8) inconsistencies 2016-02-04 17:17 ` Goffredo Baroncelli @ 2016-02-04 19:48 ` Hugo Mills 2016-02-04 20:10 ` Chris Murphy 1 sibling, 0 replies; 22+ messages in thread From: Hugo Mills @ 2016-02-04 19:48 UTC (permalink / raw) To: Goffredo Baroncelli; +Cc: Qu Wenruo, Moviuro, linux-btrfs [-- Attachment #1: Type: text/plain, Size: 2191 bytes --] On Thu, Feb 04, 2016 at 06:17:58PM +0100, Goffredo Baroncelli wrote: > On 2016-02-04 02:33, Qu Wenruo wrote: > > The idea itself makes a lot of sense. > > But I have at least two things to worry about: > > > > 1) Old scripts backward compatibility > > Especially xfstests. Maintainer will hate it a lot. > > As we have changed it several times and broken existing test cases. > > > > Although personally I like to let all the backward compatibility > > things go hell, but that's definitely not how things work. :( > > we could change the name of the btrfs prog (like bfs or bctl ?). > > If the command is called with the old name (btrfs) the old behavior is maintained; with the new name the new output is show if the specific sub command was updated; instead if the specific sub-command is not updated, the old output is show. > > We could allow a window of 1-year of transition where the new command will be in the alpha state where there is no any guarantee to be backward compatible, hoping that this time would be sufficient to reshape the output of all commands. > > For the old command no update or enhancement should be allowed (with the exception of bugfix of course). If that's going to happen, can I strongly request that all of the output selection options in btrfs sub list are dropped, and all of the possible use cases enumerated, evaluated, and understood, before producing a set of coherent and functional options that are actually useful. There's a load of different ways that the command can be called, and several of them aren't well supported. There's also different questions what can be asked of the tool, and it's hard to work out which switches to use for any given desired output (if it supports them at all). This _really_ needs fixing. (I'm more than happy to do that hard thinking and write up a detailed spec for it, BTW). Hugo. -- Hugo Mills | How do you become King? You stand in the marketplace hugo@... carfax.org.uk | and announce you're going to tax everyone. If you http://carfax.org.uk/ | get out alive, you're King. PGP: E2AB1DE4 | Harry Harrison [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 836 bytes --] ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: btrfs-progs and btrfs(8) inconsistencies 2016-02-04 17:17 ` Goffredo Baroncelli 2016-02-04 19:48 ` Hugo Mills @ 2016-02-04 20:10 ` Chris Murphy 1 sibling, 0 replies; 22+ messages in thread From: Chris Murphy @ 2016-02-04 20:10 UTC (permalink / raw) To: kreijack; +Cc: Qu Wenruo, Moviuro, Btrfs BTRFS On Thu, Feb 4, 2016 at 10:17 AM, Goffredo Baroncelli <kreijack@inwind.it> wrote: > On 2016-02-04 02:33, Qu Wenruo wrote: >> The idea itself makes a lot of sense. >> But I have at least two things to worry about: >> >> 1) Old scripts backward compatibility >> Especially xfstests. Maintainer will hate it a lot. >> As we have changed it several times and broken existing test cases. >> >> Although personally I like to let all the backward compatibility >> things go hell, but that's definitely not how things work. :( > > we could change the name of the btrfs prog (like bfs or bctl ?). btrfsctl? :-) Is there a big problem with just having a consistent "machine readable" flag for the existing btrfs that achieves this goal? Why fix bugs in two different places when this is really just a request to have consistent output *formatting*? > If the command is called with the old name (btrfs) the old behavior is maintained; with the new name the new output is show if the specific sub command was updated; instead if the specific sub-command is not updated, the old output is show. If there's a new command, then it ought to be for scripts, not for users. We are much better off breaking all scripts with a new way of doing things going forward, than breaking users (again) with another command change. > > We could allow a window of 1-year of transition where the new command will be in the alpha state where there is no any guarantee to be backward compatible, hoping that this time would be sufficient to reshape the output of all commands. > Well quite honestly I think bash scripts depending on user space tools are subject to no guarantees anyway. How about an actual stable API/ABI for developers to depend on if they need to interact with Btrfs, rather than doing it indirectly through user space tools, which are mainly for users after all. -- Chris Murphy ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: btrfs-progs and btrfs(8) inconsistencies 2016-02-03 21:54 btrfs-progs and btrfs(8) inconsistencies Moviuro 2016-02-04 1:33 ` Qu Wenruo @ 2016-02-04 18:22 ` Chris Murphy 2016-02-05 3:11 ` Anand Jain 2016-03-08 16:02 ` David Sterba 3 siblings, 0 replies; 22+ messages in thread From: Chris Murphy @ 2016-02-04 18:22 UTC (permalink / raw) To: Moviuro; +Cc: Btrfs BTRFS On Wed, Feb 3, 2016 at 2:54 PM, Moviuro <moviuro@gmail.com> wrote: > Hi all, > > I should have done this a long time ago, so here goes. > > Btrfs is certainly the best FS out there but btrfs(8) is a pain to use in > scripts. That is, it talks way too much, exposes unparseable outputs and > has inconsistent behavior. > > I strongly believe that the CLI may use a complete revamp, that is: think > of the needed/wanted output. I actually ran into this frustrating behavior > while writing butter (https://github.com/moviuro/butter). What about interfacing with libbtrfs? There's also libblockdev-btrfs, and storaged-btrfs (both are not upstream, but may benefit from a strong libbtrfs if they use it.) -- Chris Murphy ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: btrfs-progs and btrfs(8) inconsistencies 2016-02-03 21:54 btrfs-progs and btrfs(8) inconsistencies Moviuro 2016-02-04 1:33 ` Qu Wenruo 2016-02-04 18:22 ` Chris Murphy @ 2016-02-05 3:11 ` Anand Jain 2016-02-05 12:59 ` Austin S. Hemmelgarn ` (2 more replies) 2016-03-08 16:02 ` David Sterba 3 siblings, 3 replies; 22+ messages in thread From: Anand Jain @ 2016-02-05 3:11 UTC (permalink / raw) To: Moviuro, linux-btrfs If you look critically we have been using UI/CLI as API, IMO these two class of interfaces be distinct clearly. Btrfs needs library functions/APIs which is callable in popular scripting language like python. Thanks, Anand On 02/04/2016 05:54 AM, Moviuro wrote: > Hi all, > > I should have done this a long time ago, so here goes. > > Btrfs is certainly the best FS out there but btrfs(8) is a pain to use in > scripts. That is, it talks way too much, exposes unparseable outputs and > has inconsistent behavior. > > I strongly believe that the CLI may use a complete revamp, that is: think > of the needed/wanted output. I actually ran into this frustrating behavior > while writing butter (https://github.com/moviuro/butter). > > Eg: # btrfs subvolume create foo > Should be silent. I don't want btrfs(8) to bug me about successful > commands. Everything that is a success is silent. In its current form, > btrfs-subvolume(8) would spat out a complete sentence in English that we > don't need, which however contains a tiny bit of useful info: the path > (though relative) to the new subvolume. A -v/--verbose option might be here > a good idea to implement and the following behavior (just thinking out > loud, I'm sure there would be lots of things to think about more precisely): > > # btrfs subvolume create foo # should be silent > # btrfs subvolume create -v foo > /absolute/path/foo > # btrfs subvolume create [--details|-vv] foo > PATH:/absolute/path/foo > UUID:082df386-98c2-44d9-9012-07fb2b22ea20 > [And whatnot] > > That's a first example. Same should go for all the commands. I have no idea > how/where we could share about good/bad outputs (Google Drive? Framapad? > git[hub|lab]?) > > Then come inconsistencies: compare the outputs of > # btrfs sub create foo > # btrfs sub delete foo > > This also needs to be taken care of, but it should be a by-product of > thinking the outputs all over again. > > Behavior inconsistencies: some seemingly valid commands fail, because of > some code that can be easily changed (I'm working on it in my spare time > already, see https://github.com/moviuro/btrfs-progs/tree/getopts): > > The valid '--' doesn't work on every command, see > 0aa796cad7ed3fce9d5964646a8f1f5a142b4989 here: > https://github.com/kdave/btrfs-progs/commit/0aa796cad7ed3fce9d5964646a8f1f5a142b4989 > > > That's it. And since AFAICT btrfs is FLOSS, I'm sharing about my experience > as user and hope my opinion will change btrfs(8) for the better. > > Cheers, > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: btrfs-progs and btrfs(8) inconsistencies 2016-02-05 3:11 ` Anand Jain @ 2016-02-05 12:59 ` Austin S. Hemmelgarn 2016-02-06 21:35 ` Chris Murphy 2016-02-07 10:07 ` Qu Wenruo 2 siblings, 0 replies; 22+ messages in thread From: Austin S. Hemmelgarn @ 2016-02-05 12:59 UTC (permalink / raw) To: Anand Jain, Moviuro, linux-btrfs On 2016-02-04 22:11, Anand Jain wrote: > > > If you look critically we have been using UI/CLI as API, > IMO these two class of interfaces be distinct clearly. > > Btrfs needs library functions/APIs which is callable in > popular scripting language like python. > I wholeheartedly agree. At a minimum, we need to get our C API properly in line. I'm not certain how closely libbtrfs matches the btrfs CLI, but I'm pretty sure we have all the functionality needed for language bindings there, we just need to solidify the API (or at least figure out some way to properly version it). One thing I would love to see added to both libbtrfs and the CLI though is some way to make EXTENT_SAME ioctl calls on arbitrary extents, there's no way to do this without writing your own program, and calling arbitrary ioctls from Python or Java or any number of other languages is decidedly non-trivial compared to C or C++. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: btrfs-progs and btrfs(8) inconsistencies 2016-02-05 3:11 ` Anand Jain 2016-02-05 12:59 ` Austin S. Hemmelgarn @ 2016-02-06 21:35 ` Chris Murphy 2016-02-07 10:07 ` Qu Wenruo 2 siblings, 0 replies; 22+ messages in thread From: Chris Murphy @ 2016-02-06 21:35 UTC (permalink / raw) To: Anand Jain; +Cc: Moviuro, Btrfs BTRFS On Thu, Feb 4, 2016 at 8:11 PM, Anand Jain <anand.jain@oracle.com> wrote: > > > If you look critically we have been using UI/CLI as API, > IMO these two class of interfaces be distinct clearly. > > Btrfs needs library functions/APIs which is callable in > popular scripting language like python. I'm not sure what this is using to get btrfs info, it's written in c. https://github.com/storaged-project/storaged/tree/master/modules/btrfs This is python and has a btrfs backed that depends on user space tools, the comments pretty clearly express the challenges with using user space tools as a primary means to interface with Btrfs volumes. http://sourceforge.net/p/storagemanager/code/ci/devel/tree/ssmlib/backends/btrfs.py Red Hat's installer, Anaconda, has split out its storage configuration module into python-blivet to enable a bunch of storage work to be done in Python, including a lot of Btrfs functionality. Some of this just got too complicated I guess for blivet, and has been split out into libblockdev, which includes plugin apis, and there happens to be one for Btrfs. But I have no idea where it goes from there. https://github.com/rhinstaller/libblockdev https://github.com/rhinstaller/libblockdev/blob/master/src/lib/plugin_apis/btrfs.api So yeah, people are trying to do this. -- Chris Murphy ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: btrfs-progs and btrfs(8) inconsistencies 2016-02-05 3:11 ` Anand Jain 2016-02-05 12:59 ` Austin S. Hemmelgarn 2016-02-06 21:35 ` Chris Murphy @ 2016-02-07 10:07 ` Qu Wenruo 2016-02-07 20:26 ` Goffredo Baroncelli 2 siblings, 1 reply; 22+ messages in thread From: Qu Wenruo @ 2016-02-07 10:07 UTC (permalink / raw) To: Anand Jain, Moviuro, linux-btrfs On 02/05/2016 11:11 AM, Anand Jain wrote: > > > If you look critically we have been using UI/CLI as API, > IMO these two class of interfaces be distinct clearly. > > Btrfs needs library functions/APIs which is callable in > popular scripting language like python. > > Thanks, Anand > +1 too. But first in C, then python wrapper. Not sure why there is no such libbtrfs for C wrapper of btrfs ioctls. Maybe just because current btrfs ioctl is too easy to use? AFAIK, systemd (container storage part) calls btrfs ioctl directly, especially for subvolume/snapshot creation mainly because the design of them are quite easy. Thanks, Qu > > On 02/04/2016 05:54 AM, Moviuro wrote: >> Hi all, >> >> I should have done this a long time ago, so here goes. >> >> Btrfs is certainly the best FS out there but btrfs(8) is a pain to use in >> scripts. That is, it talks way too much, exposes unparseable outputs and >> has inconsistent behavior. >> >> I strongly believe that the CLI may use a complete revamp, that is: think >> of the needed/wanted output. I actually ran into this frustrating >> behavior >> while writing butter (https://github.com/moviuro/butter). >> >> Eg: # btrfs subvolume create foo >> Should be silent. I don't want btrfs(8) to bug me about successful >> commands. Everything that is a success is silent. In its current form, >> btrfs-subvolume(8) would spat out a complete sentence in English that we >> don't need, which however contains a tiny bit of useful info: the path >> (though relative) to the new subvolume. A -v/--verbose option might be >> here >> a good idea to implement and the following behavior (just thinking out >> loud, I'm sure there would be lots of things to think about more >> precisely): >> >> # btrfs subvolume create foo # should be silent >> # btrfs subvolume create -v foo >> /absolute/path/foo >> # btrfs subvolume create [--details|-vv] foo >> PATH:/absolute/path/foo >> UUID:082df386-98c2-44d9-9012-07fb2b22ea20 >> [And whatnot] >> >> That's a first example. Same should go for all the commands. I have no >> idea >> how/where we could share about good/bad outputs (Google Drive? Framapad? >> git[hub|lab]?) >> >> Then come inconsistencies: compare the outputs of >> # btrfs sub create foo >> # btrfs sub delete foo >> >> This also needs to be taken care of, but it should be a by-product of >> thinking the outputs all over again. >> >> Behavior inconsistencies: some seemingly valid commands fail, because of >> some code that can be easily changed (I'm working on it in my spare time >> already, see https://github.com/moviuro/btrfs-progs/tree/getopts): >> >> The valid '--' doesn't work on every command, see >> 0aa796cad7ed3fce9d5964646a8f1f5a142b4989 here: >> https://github.com/kdave/btrfs-progs/commit/0aa796cad7ed3fce9d5964646a8f1f5a142b4989 >> >> >> >> That's it. And since AFAICT btrfs is FLOSS, I'm sharing about my >> experience >> as user and hope my opinion will change btrfs(8) for the better. >> >> Cheers, >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: btrfs-progs and btrfs(8) inconsistencies 2016-02-07 10:07 ` Qu Wenruo @ 2016-02-07 20:26 ` Goffredo Baroncelli 0 siblings, 0 replies; 22+ messages in thread From: Goffredo Baroncelli @ 2016-02-07 20:26 UTC (permalink / raw) To: Qu Wenruo, Anand Jain, Moviuro, linux-btrfs On 2016-02-07 11:07, Qu Wenruo wrote: > +1 too. > > But first in C, then python wrapper. > > Not sure why there is no such libbtrfs for C wrapper of btrfs ioctls. > > Maybe just because current btrfs ioctl is too easy to use? Unfortunately no. I think the main problem of writing a libbtrfs, is the code changes needed. In fact in order to avoid code duplication (and to have more testing), the btrfs command has to be a client of libbtrfs. But it is not easy: you have to refactoring the error handling and the message printing... > AFAIK, systemd (container storage part) calls btrfs ioctl directly, especially for subvolume/snapshot creation mainly > because the design of them are quite easy. Also udev does the same (I don't remember if this happened before or after it went under the systemd umbrella). I remember a case where it was impossible to change the semantic of BTRFS_IOC_DEVICES_READY: on the basis of its name, it could be seems that this ioctl should _only_ check if a filesystem is ready. Unfortunately it register also the device [1]. It was discussed in the ML to change its semantic, but having already widespread client like udev it was impossible to do (even tough we could discuss to change their name) > Thanks, > Qu BR G.Baroncelli [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=772744 ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: btrfs-progs and btrfs(8) inconsistencies 2016-02-03 21:54 btrfs-progs and btrfs(8) inconsistencies Moviuro ` (2 preceding siblings ...) 2016-02-05 3:11 ` Anand Jain @ 2016-03-08 16:02 ` David Sterba 2016-03-09 10:02 ` Moviuro 3 siblings, 1 reply; 22+ messages in thread From: David Sterba @ 2016-03-08 16:02 UTC (permalink / raw) To: Moviuro; +Cc: linux-btrfs Let me answer here for the whole thread. There are lot of issues to cover, I think the github way would work here, so the project you've created for the purpose seems ok to me. https://github.com/btrfs8-revamp I assume way more dicussions than actual patches and coding, the web platform offers commenting at the exact locations of files, issue tracking works better than over mail. Initially we'd cover all the stuff mentioned in this thread, naming, conventions etc. After we agree we can start changing code in btrfs-progs. We can ask for feedback wider audience in the mailinglist if needed, possibly with a draft that would be reviewed already. After we write comprehensive guidelines for the UI, we can add it to btrfs-progs, I really want to avoid high churn in that repository for things that are not related to code. For that the revamp repository is good, the only requirement is a github account. ^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: btrfs-progs and btrfs(8) inconsistencies 2016-03-08 16:02 ` David Sterba @ 2016-03-09 10:02 ` Moviuro 0 siblings, 0 replies; 22+ messages in thread From: Moviuro @ 2016-03-09 10:02 UTC (permalink / raw) To: dsterba, Moviuro, Btrfs BTRFS Don't hesitate to drop an email here (with your Github account name...) to get invited into the Github group. That's because you can't ask to join a group on GitHub. At the moment, kdave and myself are administrators of the said group (https://github.com/btrfs8-revamp). I'm sure we can nominate David too, once he joined ;-) PS: I really didn't expect my first mail to cause this movement, I'm glad it took this shape! ^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2016-03-09 10:02 UTC | newest] Thread overview: 22+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-02-03 21:54 btrfs-progs and btrfs(8) inconsistencies Moviuro 2016-02-04 1:33 ` Qu Wenruo 2016-02-04 8:57 ` Moviuro 2016-02-04 9:15 ` Qu Wenruo 2016-02-04 10:14 ` Martin Steigerwald 2016-02-04 12:04 ` Moviuro 2016-02-04 12:53 ` Austin S. Hemmelgarn 2016-02-04 19:40 ` Chris Murphy 2016-02-04 20:19 ` Austin S. Hemmelgarn 2016-02-04 20:40 ` Moviuro 2016-02-05 13:04 ` Austin S. Hemmelgarn 2016-02-04 17:17 ` Goffredo Baroncelli 2016-02-04 19:48 ` Hugo Mills 2016-02-04 20:10 ` Chris Murphy 2016-02-04 18:22 ` Chris Murphy 2016-02-05 3:11 ` Anand Jain 2016-02-05 12:59 ` Austin S. Hemmelgarn 2016-02-06 21:35 ` Chris Murphy 2016-02-07 10:07 ` Qu Wenruo 2016-02-07 20:26 ` Goffredo Baroncelli 2016-03-08 16:02 ` David Sterba 2016-03-09 10:02 ` Moviuro
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.