From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr0-f178.google.com ([209.85.128.178]:34058 "EHLO mail-wr0-f178.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751982AbdGDN6B (ORCPT ); Tue, 4 Jul 2017 09:58:01 -0400 Received: by mail-wr0-f178.google.com with SMTP id 77so249566424wrb.1 for ; Tue, 04 Jul 2017 06:58:00 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <20170703173701.GB5869@birch.djwong.org> References: <20170630152906.ztfpez7lyrnmybyi@thunk.org> <20170703150722.ki4zuih6sklk7f6b@thunk.org> <20170703173701.GB5869@birch.djwong.org> From: Jan Tulak Date: Tue, 4 Jul 2017 15:57:38 +0200 Message-ID: Subject: Re: [proposal] making filesystem tools more machine friendly To: "Darrick J. Wong" Cc: "Theodore Ts'o" , linux-fsdevel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Sender: linux-fsdevel-owner@vger.kernel.org List-ID: Reply to both Darrick and Ted. On Mon, Jul 3, 2017 at 7:37 PM, Darrick J. Wong wrote: > On Mon, Jul 03, 2017 at 11:07:22AM -0400, Theodore Ts'o wrote: >> On Mon, Jul 03, 2017 at 01:52:49PM +0200, Jan Tulak wrote: >> >> I'm not sure what sort use cases you have in mind where structured >> output would be useful. >> >> For mke2fs or resize2fs, in general all you care about is "did the >> operation succeed", right? What did you have in mind where this more >> information than "the operation was successful" did you have in mind? >> >> For tune2fs, what options or output in tune2fs are sufficiently file >> system independent that you think it would be worth exporting to your >> infrastructure? I think I muddied the water with the fs-independent fields... :-/ The main thing I'm trying to achieve is to ditch the separate syntax parsers for every tool and fs and instead have only one format, one parser, and the difference between filesystems and tools is only in the data produced by the parser. If some fields in the data are going to be common, it is nice, but that's just a detail. There are some similar values between filesystems, but they are really basic ones - blocks, inodes, and maybe few more other things, so I admit the usability of common names for these fields is questionable. What I really want is that I can run tune2fs or xfs_info, pass the output through a standard, common parser, and get a structure filled with data. >> >> > The generic way would be along the lines: These ten common and >> > frequently used fields are generic and work everywhere, anything else >> > has a prefix (ext4-, xfs-, btrfs-, ...) or is inside of a fs-specific >> > list of extensions "ext4":{"some-option":value}. And during the >> > parsing, all the fields would be mapped to some mkfs arguments, >> > usually 1:1. Similar to what could be done with output, like putting >> > volume identifier into a specific field (e.g. "fsid") no matter what >> > the filesystem is. >> >> It might be useful if you could give some "user stories" that explain >> at a high level what the user might want to do with the ultimate >> user-visible interface that would require this kind of precision. > > Yes, please. I intend it for more automated or abstract storage management. One where the user doesn't really care about specific values in any given filesystem, but an intermediate layer can read and configure them. Perhaps similar to the "mke2fs -t ext4 -T smr-host-managed /dev/sdXX" Ted mentioned bellow. It is either for tools that group multiple subsystems under one hood, abstracting most of the differences between the various filesystems while still letting the user design the storage (e.g. System Storage Manage - SSM), or for strongly automated solutions, where the user just throws some drives into a pool, configures intended use and maybe few other things like redundancy and the rest happens in the background - LVM, fs formatting, raid... (e.g. fresh project stratis-storage [1]). But GUI like YaST or Anaconda will certainly benefit from it as well. [1] https://github.com/stratis-storage/stratisd > >> Most users don't know how to use the specialized options to >> mkfs. and to be honest, most don't need to. The way I've dealt > > To reinforce that point, in XFS land we tell people to take the defaults > unless they have verified that a specific problem of theirs is fixed by > changing some tuning parameter (and doesn't cause other problems). Many > of the user complaints I see on the (xfs) list are a result of people > engaging in copy-pasta without understanding the tradeoffs they're > implicitly accepting. > > Once users get to the point of having verified their workloads with > non-default options, they often encode a mkfs.xfs command line into > their deployment scripts. So while I can be convinced that there's > value in a(nother) EVMS-like thing encapsulating (generic) mkfs, I'm not > currently persuaded that there's much point in adapting mkfs.$FSTYP to > accept a bunch of geometry/feature options via json. > > (This probably changes for things like btrfs that also manage volumes.) See above, the goal is not to give an end user a way how to fiddle with configuration, but rather to simplify the work for developers of automated tools, so they can tune the filesystem for specific workloads without having to care about syntax details that are different between tools and filesystems. > >> with this for mke2fs.conf is that when someone has come up with a >> specialized recipe for a unique sort of file system type --- say, >> maybe for a Lustre Metadata server, or the back-end storage for some >> kind of clustre file system like Hadoopfs, or specialized options for >> an Android phone --- someone with wizard-level skills will edit >> /etc/mke2fs.conf, with perhaps something like this: >> >> >> ... and then all the user will have to do is run "mke2fs -t ext4 -T >> smr-host-managed /dev/sdXX". (The intended use case for this is to >> support an SMR-aware user-space application that was going to be >> managing the host-managed SMR zones directly.) >> >> I suspect telling the user that they have to type a whole series of >> parameters of the form: >> >> "ext4":{"some-option":value} >> >> into a GUI would not be a particularly user-friendly suggestion. :-) >> >> The other question I'd ask is how many people really are going to want >> to use your infrastructure? Is it only going to be for the "point and >> click" users who will want a simplified interface? Are you trying to >> make something that will be useful for advanced/expert users? What >> value are you going to be able to add that will convince the >> advanced/export users that they should learn your new >> "fstype":{"some-option":value} syntax when typing at the command line >> will probably be ten times faster, easier, and less rage-inducing than >> trying to reverse-engineer out some interface that was designed not to >> scare the civilians? (There's a reason why many drivers prefer manual >> to automatic transmission on their cars. :-) > > I doubt Jan is proposing to eliminate ye olde getopt arguments -- > presumably the GUI (or whatever) will generate the json code and the > library stuffs it into mkfs? > > Or to put it another way, I would not accept a mkfs.xfs patch ripping > out the classic getopt options, and I suspect Eric Sandeen wouldn't > either. The old getopt is going to stay and anyone using mkfs. directly will see everything the same way as it is now. :-) The json way is only to simplify "fs tool <---> abstracting tool" data exchange. Which answers the question "who will use it": The json will see only developers or advanced users who are making some scripts, and who want better compatibility between versions and across filesystems. But it shouldn't be used in lieu of current getopts anywhere else. The json input would probably do an internal transformation from json to getopt anyway unless there is a useful internal infrastructure where the getopt part could be skipped. >> >> The reason why you might want to consider C is because: >> >> * It allows the plugin to be imported into many different >> programming languages: Python, Go, Perl, etc., via using >> something like SWIG. >> >> * Different file system maintainers will be willing to accept >> maintenance of your plugin at different times. For some file >> systems, you may have to wrap the command-line tools forever; for >> one thing the file system may no longer be under active >> maintenance (ex: iso9660) but you still might want to be use it >> in your GUI interface. Other file system developers will be >> willing to take over the plugin and support it as a native part >> of their file system tools more quickly. >> >> * The first operations that you might want to make be native >> instead of being screen scraped (getting the file system size, >> the amount of free space, etc.) are things which are most easily >> done in C. > > Weird ioctls and other syscalls aren't so hard in python. I've written > a clumsy GETFSMAP wrapper: > > https://github.com/djwong/filemapper/blob/master/getfsmap.py > > But you're correct to point out that wrapping libext2fs in Python would > be difficult, and even more invasive if you took the interesting parts > of, say, tune2fs and put them into a separate library and made tune2fs > merely an arguments-parsing wrapper around the separate library. > > I'd occasionally thought about wrapping libext2fs in Python and > concluded that it was too much work for not enough gain, and Ted would > likely NAK it anyway. Understood, anything else than C won't fly. :-) > >> So if you want ask the file system developers to take over the >> plugin, they are much more likely to be willing to say yes if the >> plugin is already in C, as opposed to asking them to take over >> some Python class where trying to integrate python code to call >> into libext2fs is going to be a pain in the ass. For that >> matter, you might want to implement the plugins to call libext2fs >> and libxfs directly for those basic functions. That's what the >> EVMS developers did, and those interfaces in libext2fs are >> guaranteed to have ABI and API stability. Anyway, if your goal >> is to convince file system developers to eventually take >> ownership of the interface/plugin module, it will be much easier >> to do that if it is in C --- trust me on that. > > Yeah, probably. :) > > I don't think we'll ever have A[BP]I stability in "libxfs". > > For that matter, libxfs isn't even a proper library; we statically link > all the object files directly into the tools. But then Jan is (so far) > only asking to wrap the tools, not reimplement them too. Yes. As I mentioned at the end of the very first email, I think that simply adding a structured output (and possibly input) to the tools will help the most. All the other stuff about providing calls as a library is something I consider a far future if it happens at all. > >> * I don't think it's going to be that hard to use C; as I've said, >> I really disbelieve that there are that many places where you >> need to screenscrape. Most of what you will probably need to do >> is to return the exit status of mke2fs, fsck, resize2fs, etc. >> Those programs that you do need to screen scrape will have >> outputs similar to dumpe2fs, which is stupid-easy to parse, and >> are also, as I've noted above, the simplest thing to move to >> being done in native code calling the file system's C library. >> >> Oh, one thing. I'll note that e2fsprogs has progress bar support >> already, and it was designed so it could be easily integrated into a >> GUI. As far as I know Ubuntu was the only distro that used it --- >> progress bars tend not to be high on most distros' product manager's >> feature priority lists --- but it's there. See e2fsck's -C option. >> This support was also plumbed into fsck (see its -C option), so I >> designed it to be something that other file systems could implement. >> >> Also, we're Linux systems programmers, not Web developers writing >> Javascript; why use JSON and require fancy parsing when you can just >> isolate the completion information onto a separate file descriptor? I > > I imagine that json makes it easier to export structured information > than a basic flat file, but otoh my experiences with wrapped filesystem > tools is that the good tools report nonzero error status and point you > at the full text logs of what happened. The bad ones, of course, don't > check the status and don't record the output. > >> don't know how big and complex your JSON parsing library is, but all >> that was needed to parse *my* completion information is the single >> line of C code: >> >> fscanf(progress_f, "%d %lu %lu %ms\n", &pass, &cur, &max, &text). > > FWIW I've been building the same into xfs_scrub if anyone cares... :) > A return code and a log file are not going to help much when I need to know the geometry of a volume. I suggested adding json also to the simple cases (where return code would be enough) only because this change should be almost for free (the infrastructure will be there anyway) and it will add the same output format as the more verbose tools have. That being said, it doesn't have to be json. Another format might work as well. The advantage of json is that it already solved all the special cases that may happen, supports nested structures and is already used by other storage-related tools. If there should be a single format, it needs to handle snapshots and subvolumes, etc, even if not every tool will use its whole power. And by the way, I'm going to be off until Monday. The next two days (Wed/Thu) are public holidays here in CZ and then it is almost a weekend, so I will be off the grid for most of the next five days. Cheers, Jan -- Jan Tulak jtulak@redhat.com / jan@tulak.me