Re: [proposal] making filesystem tools more machine friendly

[Jan Tulak <jtulak@redhat.com>]

On Fri, Jun 30, 2017 at 5:29 PM, Theodore Ts'o <tytso@mit.edu> wrote:
>
> So a couple of comments.  First of all, this sort of thing has been
> proposed before.  Specifically, this was one of the problems which
> EVMS (from IBM, which ultimately lost out to device mapper) had as
> part of their project.  Their userspace component has a plug-in
> architecture so that file systems could provide a shared library which
> could be used by GUI and CLI tools.
>
> I had that support in e2fsprogs, and it was removed when EVMS was
> ultimately killed off by device-mapper and LVM.  The commit removal
> was in commit 921f4ad53: "Remove support for EVMS 1.x plugin library"
> so if you look at the sources in lib/evms in the e2fsprogs git repo at
> commit 921f4ad53 (aka 921f4ad53^), you can see what it looked like.

Thank you for this info, I wasn't aware of EVMS. It might be useful,
although, from my brief look, it seems that it wouldn't do much for
the structured output, but rather for the library later on.

>
> Secondly, as a file system developer and maintainer for e2fsprogs, I'm
> going to be extremely hesitant to accept patches which radically
> reorganize e2fsprogs and which adds dependencies on third-party
> libraries that emit JSON output before I know whether or not your
> project is going to be successful, or will ultimately end up getting
> abandoned and left for dead, as was the case for EVMS.
>
> That being said, I probably would be willing to accept a drop-in
> library, ala lib/evms, where some of the functions were handled
> directly via accessing the file system directly via the plugin
> library, and where some of the changes were made by wrapping mke2fs
> and e2fsck.  Over time, *if* this interface gained legs, I'd be more
> willing to try to make a library version of mke2fs (and then _maybe_
> e2fsck) so that the plugin library could do more of the work natively,
> rather than by wrapping the mke2fs and e2fsck executables.

Yeah, I understand your reluctance (and expect it), which is why I
opened this so early. About the drop-in in lib/, I have to check what
could or couldn't do this way. I know xfsprogs, but e2fsprogs I saw
only really briefly.

My intention with the structured output is to minimize the amount of
code that has to be kept up-to-date, unlike the library. Ideally and
most of the time, the only difference would be that instead of printf
calls, there would be a function to just store the message or specific
values in a 1:1 substitution, while the part that does the actual
printing would be small, loosely coupled and wouldn't need any changes
during the usual development of e2fsprogs, xfsprogs or anything else.

In any case, starting with a wrapper is something I thought about. It
would mean that it's code has to be kept up-to-date manually all the
time, but if it would catch enough of interest, it could be integrated
with only minimal changes on the outside.

>
> Third, I think you are *massively* underestimating how much work is
> needed to do this in a generic fashion, especially with the file
> system check tool.  In any tool where you need to ask the user for
> permission to make a particular change to the file system, trying to
> do this in a generic way is *hard*, and the "use JSON" is not enough.
>
> At the same time, for the common case, where you just want to "check
> to see if the file system is consistent, and return a boolean", or
> "check the file system and do all of the safe things", you don't need
> all of the complexity of JSON.  So I think JSON is simultaneously too
> much (for the simple stuff, where the user doesn't need to see the
> output anyway), and too little (for the hard cases where you are doing
> a file system repair operation).

I want to limit the capabilities of this interface to non-interactive
only. So, yes, with fsck, JSON would be overkill. But the idea is to
have a single format across all the tools, so you don't need a
standalone parser for every tool, even if some tools don't need
anything more than an exit code and/or one message on stderr. In the
case of ext2/3/4 it is more about resize2fs and tune2fs, where the
JSON would be much more useful, than fsck.

>
> For another matter, how you would pass in mkfs parameters, which are
> very file system specific, in a completely generic way, is also
> completely unspecified in your proposal.  This makes me deeply
> suspicious you haven't thought through the issues sufficiently, and if
> you try to send patches that massively reorganize e2fsprogs and adds
> JSON output in what I suspect will be a horrible, terribly ugly way,
> it's most likely I will decline to accept your patches.

The generic way would be along the lines: These ten common and
frequently used fields are generic and work everywhere, anything else
has a prefix (ext4-, xfs-, btrfs-, ...) or is inside of a fs-specific
list of extensions "ext4":{"some-option":value}. And during the
parsing, all the fields would be mapped to some mkfs arguments,
usually 1:1. Similar to what could be done with output, like putting
volume identifier into a specific field (e.g. "fsid") no matter what
the filesystem is.

But I'm still trying to find out what are the common usage patterns;
whether there is some frequently used subset for which it is worth to
do the unification at all. So, it is quite possible that I will ditch
this particular idea completely (and add the JSON while keeping every
field name and structure fs-specific) if I find out that it is better
to be left on higher-level tools; if the only unified options would be
volume label, force flag, and target device, then the saved time for
other developers is not balancing the effort spend on the unification
of fields.

So, I didn't specify how to do it in detail because I don't know yet
at what point it would be worth of the effort (when it would be useful
enough). But at the same time, this decision should better be made
early, so I mentioned it in my proposal, to see if people are
interested in it at all.

>
> This probably means that you will need to do something which starts by
> using screen scraping at least initially, and if the maintainers
> aren't willing to adopt your code, you're going to have to maintain it
> yourself.  That's how EVMS started, and then they asked if I would be
> willing to integrate the ext2/3 (this was before ext4) evms plugin
> into e2fsprogs, and would I be willing to start making changes to
> integrate it more organically into e2fsprogs.  I suspect that kind of
> intecremental approach is much more likely to be successful in the
> long term.  You may also want to look at the EVMS plugin interfaces,
> since the people who tried this did think about what made sense to try
> to do in a file system independent way, and what would probably have
> to be kept file system specific.
>

Thanks for pointing out EVMS, I will see what I can learn from that
attempt. Starting with screenscraping is certainly an option and might
be the only viable one. This raises some other questions, though:
given the temporality of the wrapper, I would rather use other
languages than C/Bash (e.g. Python) to simplify and speed up the
development. But I have doubts about whether you would be willing to
adopt this into e2fsprogs, which would, in turn, reduce the usability
of this approach.

Anyway, thank you much for your reply. :-)

Cheers,
Jan


-- 
Jan Tulak
jtulak@redhat.com / jan@tulak.me