* silent semantic changes with reiser4 @ 2004-08-24 20:25 Christoph Hellwig 2004-08-24 20:35 ` Lee Revell ` (3 more replies) 0 siblings, 4 replies; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-24 20:25 UTC (permalink / raw) To: akpm, reiser; +Cc: linux-fsdevel, linux-kernel After looking trough the code and mailinglists I'm quite unhappy with a bunch of user-visible changes that Hans sneaked in and make reiser4 incompatible with other filesystems and have a slight potential to break even in the kernel. o files as directories - O_DIRECTORY opens succeed on all files on reiser4. Besides breaking .htaccess handling in apache and glibc compilation this also renders this flag entirely useless and opens up the races it tries to prevent against cmpletely useless - meaning of the -x permission. This one has different meanings on directories vs files on UNIX systems. If we want to support directories as files we'll probably have to find a way to work around this. - dentry aliasing. I can't find a formal guarantee in the code this can't happen o metafiles - ..metas as a magic name that's just taken out of the namespace doesn't sound like a good idea. If we want this it should be a VFS-level option and there should be a translation-layer to xattrs. Not doing this will again confuse applications greatly that expect uniform filesystem behaviour. Given these problems I request that these interfaces are removed from reiser4 for the kernel merge, and if added later at the proper VFS level after discussion on linux-kernel and linux-fsdevel, like we did for xattrs. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-24 20:25 silent semantic changes with reiser4 Christoph Hellwig @ 2004-08-24 20:35 ` Lee Revell 2004-08-24 20:38 ` Christoph Hellwig 2004-08-24 20:38 ` Jeff Garzik ` (2 subsequent siblings) 3 siblings, 1 reply; 1392+ messages in thread From: Lee Revell @ 2004-08-24 20:35 UTC (permalink / raw) To: Christoph Hellwig; +Cc: Andrew Morton, reiser, linux-fsdevel, linux-kernel On Tue, 2004-08-24 at 16:25, Christoph Hellwig wrote: > - O_DIRECTORY opens succeed on all files on reiser4. Besides breaking > .htaccess handling in apache and glibc compilation this also renders > this flag entirely useless and opens up the races it tries to > prevent against cmpletely useless So `find -type d' would list every file on the system? Lee ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-24 20:35 ` Lee Revell @ 2004-08-24 20:38 ` Christoph Hellwig 2004-08-24 20:42 ` Lee Revell 2004-08-24 21:18 ` Jamie Lokier 0 siblings, 2 replies; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-24 20:38 UTC (permalink / raw) To: Lee Revell Cc: Christoph Hellwig, Andrew Morton, reiser, linux-fsdevel, linux-kernel On Tue, Aug 24, 2004 at 04:35:18PM -0400, Lee Revell wrote: > On Tue, 2004-08-24 at 16:25, Christoph Hellwig wrote: > > - O_DIRECTORY opens succeed on all files on reiser4. Besides breaking > > .htaccess handling in apache and glibc compilation this also renders > > this flag entirely useless and opens up the races it tries to > > prevent against cmpletely useless > > So `find -type d' would list every file on the system? the find I have here is using lstat and not open with O_DIRECTORY, so no. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-24 20:38 ` Christoph Hellwig @ 2004-08-24 20:42 ` Lee Revell 2004-08-24 21:18 ` Jamie Lokier 1 sibling, 0 replies; 1392+ messages in thread From: Lee Revell @ 2004-08-24 20:42 UTC (permalink / raw) To: Christoph Hellwig; +Cc: Andrew Morton, reiser, linux-fsdevel, linux-kernel On Tue, 2004-08-24 at 16:38, Christoph Hellwig wrote: > On Tue, Aug 24, 2004 at 04:35:18PM -0400, Lee Revell wrote: > > On Tue, 2004-08-24 at 16:25, Christoph Hellwig wrote: > > > - O_DIRECTORY opens succeed on all files on reiser4. Besides breaking > > > .htaccess handling in apache and glibc compilation this also renders > > > this flag entirely useless and opens up the races it tries to > > > prevent against cmpletely useless > > > > So `find -type d' would list every file on the system? > > the find I have here is using lstat and not open with O_DIRECTORY, so > no. > Ugh, how embarrassing, I completely forgot about stat(). Lee ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-24 20:38 ` Christoph Hellwig 2004-08-24 20:42 ` Lee Revell @ 2004-08-24 21:18 ` Jamie Lokier 1 sibling, 0 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-08-24 21:18 UTC (permalink / raw) To: Christoph Hellwig, Lee Revell, Andrew Morton, reiser, linux-fsdevel, linux-kernel Christoph Hellwig wrote: > > > - O_DIRECTORY opens succeed on all files on reiser4. Besides breaking > > > .htaccess handling in apache and glibc compilation this also renders > > > this flag entirely useless and opens up the races it tries to > > > prevent against cmpletely useless > > > > So `find -type d' would list every file on the system? > > the find I have here is using lstat and not open with O_DIRECTORY, so > no. The find-like program I use (called treescan) uses O_DIRECTORY as an optimisation. It assumes that O_DIRECTORY will only open objects which are directories and can be read using readdir(). However, if reiser4 returns d_type values, then it won't even attempt an open on non-DT_DIR objects, and that's a better optimisation. (reiserfs doesn't return d_type values, unfortunately). So the list of files that treescan finds depends on whether reiser4 implements d_type. This is nothing like a POSIX filesystem. You untar a tree, and then listing it recursively shows extra things created by reiser4. I quite like the principle, but because it's not like POSIX and doesn't match some program's expectations, it's a problem in its present form. xattrs aren't a complete solution as you can't store structured data in an xattr. For example, with reiser4's model, you can cd into a .tar, .zip, .mp3 or .xml file and list the internal structure along with the file's metadata. You can't do that with xattrs. Programs exist which quite reasonably assume that when you create a file, you can't opendir() the file, and recursive listings (like find, ls -R et al.) won't automatically traverse into every file. On the other hand, being able to enter a file in a directory-like way allows structured representations of the contents to be accessed in the very useful "everything's a file" way -- i.e. ordinary tools. So here's a semantic proposal: 1. O_DIRECTORY won't open an ordinary file. Corollary: opendir("file") won't open an ordinary file. 2. An ordinary file path followed by "/" won't open an ordinary file. Corollary: opendir("file/") won't open an ordinary file. This is because appending a trailing slash is an alternate way for userspace to get the same results as O_DIRECTORY. 3. An ordinary file path followed by "/" _and_ one or more path components will open the file as a directory and enter it. Corollary: opendir("file/.") will open an ordinary file. 4. The type of "file/." shall be S_IFDIR, _not_ S_IFREG. Corollary: stat("file/.") will return that it's a directory. The intention here is that explicit requests to examine the metadata or alternate structure representations of a file will create such a view, but the view is only available if requested explicitly. When such a view is created, the results of stat(), O_DIRECTORY and opendir() are absolutely consistent. This will minimise confusion. Programs which recurse over a directory tree won't look inside any of the files. However they can be explicitly asked to recurse starting from a path inside a file: then they'll recurse over a single file's metadata and structured data. Regarding the problems of safe locking in the VFS. The VFS assumes that directories are not hard linked: i.e. that they cannot appear at more than one path in a filesystem. Files-as-directories breaks that. However, VFS does support directories on multiple paths, using bind mounts. So it wouldn't be out of the question if entering a file (as described above) effectively auto-mounted a bind mount at that point. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-24 20:25 silent semantic changes with reiser4 Christoph Hellwig 2004-08-24 20:35 ` Lee Revell @ 2004-08-24 20:38 ` Jeff Garzik 2004-08-24 20:53 ` viro 2004-08-25 19:53 ` Hans Reiser 2004-09-14 4:27 ` file-as-dir vs. dir? " Andy Lutomirski 3 siblings, 1 reply; 1392+ messages in thread From: Jeff Garzik @ 2004-08-24 20:38 UTC (permalink / raw) To: Christoph Hellwig; +Cc: akpm, reiser, linux-fsdevel, linux-kernel Christoph Hellwig wrote: > o files as directories > - O_DIRECTORY opens succeed on all files on reiser4. Besides breaking > .htaccess handling in apache and glibc compilation this also renders > this flag entirely useless and opens up the races it tries to > prevent against cmpletely useless Ouch. I would definitely classify this as a security hole, since userland definitely uses O_DIRECTORY to avoid races. Jeff ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-24 20:38 ` Jeff Garzik @ 2004-08-24 20:53 ` viro 2004-08-24 21:22 ` viro 0 siblings, 1 reply; 1392+ messages in thread From: viro @ 2004-08-24 20:53 UTC (permalink / raw) To: Jeff Garzik; +Cc: Christoph Hellwig, akpm, reiser, linux-fsdevel, linux-kernel On Tue, Aug 24, 2004 at 04:38:25PM -0400, Jeff Garzik wrote: > Christoph Hellwig wrote: > > o files as directories > > - O_DIRECTORY opens succeed on all files on reiser4. Besides breaking > > .htaccess handling in apache and glibc compilation this also renders > > this flag entirely useless and opens up the races it tries to > > prevent against cmpletely useless > > > Ouch. > > I would definitely classify this as a security hole, since userland > definitely uses O_DIRECTORY to avoid races. Feh. That's far from the worst parts of the mess introduced by "hybrid" crap - trivial sys_link(2) deadlocks triggerable by any user rate a bit higher on the suckitude scale, IMO. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-24 20:53 ` viro @ 2004-08-24 21:22 ` viro 2004-08-25 18:28 ` Hans Reiser 0 siblings, 1 reply; 1392+ messages in thread From: viro @ 2004-08-24 21:22 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, akpm, reiser, linux-fsdevel, linux-kernel, Jeff Garzik On Tue, Aug 24, 2004 at 09:53:44PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote: > Feh. That's far from the worst parts of the mess introduced by "hybrid" > crap - trivial sys_link(2) deadlocks triggerable by any user rate a bit > higher on the suckitude scale, IMO. While we are at it - consider these hybrids vetoed until a) sys_link()/sys_link() deadlock is fixed b) sys_link()/sys_rename() deadlock is fixed c) correctness proof of the locking scheme (in Documentation/filesystems/directory-locking) is updated to match the presense of the file/directory hybrids. Rationale: (a) and (b) - immediately exploitable by any user, (c) - "convince us that there's no more crap of that kind". IMO a reasonable request, seeing that the first look at the patches in -mm4 had turned up two exploits in that area, despite the *YEARS* of warnings about potential trouble and need to be careful there (actually, I've given Hans too much credit and assumed that link/link never happens since nobody would be dumb enough to provide ->link() method for non-directory inodes; turns out that somebody is dumb enough and link/link is as exploitable as link/rename). ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-24 21:22 ` viro @ 2004-08-25 18:28 ` Hans Reiser 2004-08-25 18:45 ` Christoph Hellwig 0 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-25 18:28 UTC (permalink / raw) To: viro Cc: Linus Torvalds, Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Jeff Garzik, Reiserfs developers mail-list I allowed myself to get talked out of a final top to bottom code audit, and obviously that was a mistake. It will probably take about 6 weeks. Apologies for wasting your time before that was done. Hans viro@parcelfarce.linux.theplanet.co.uk wrote: >On Tue, Aug 24, 2004 at 09:53:44PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote: > > > >>Feh. That's far from the worst parts of the mess introduced by "hybrid" >>crap - trivial sys_link(2) deadlocks triggerable by any user rate a bit >>higher on the suckitude scale, IMO. >> >> > >While we are at it - consider these hybrids vetoed until > a) sys_link()/sys_link() deadlock is fixed > b) sys_link()/sys_rename() deadlock is fixed > c) correctness proof of the locking scheme (in >Documentation/filesystems/directory-locking) is updated to match the >presense of the file/directory hybrids. > >Rationale: (a) and (b) - immediately exploitable by any user, (c) - "convince >us that there's no more crap of that kind". IMO a reasonable request, seeing >that the first look at the patches in -mm4 had turned up two exploits in >that area, despite the *YEARS* of warnings about potential trouble and need >to be careful there (actually, I've given Hans too much credit and assumed >that link/link never happens since nobody would be dumb enough to provide >->link() method for non-directory inodes; turns out that somebody is dumb >enough and link/link is as exploitable as link/rename). >- >To unsubscribe from this list: send the line "unsubscribe linux-kernel" in >the body of a message to majordomo@vger.kernel.org >More majordomo info at http://vger.kernel.org/majordomo-info.html >Please read the FAQ at http://www.tux.org/lkml/ > > > > ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 18:28 ` Hans Reiser @ 2004-08-25 18:45 ` Christoph Hellwig 2004-08-26 9:02 ` Hans Reiser 0 siblings, 1 reply; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-25 18:45 UTC (permalink / raw) To: Hans Reiser Cc: viro, Linus Torvalds, Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Jeff Garzik, Reiserfs developers mail-list On Wed, Aug 25, 2004 at 11:28:56AM -0700, Hans Reiser wrote: > I allowed myself to get talked out of a final top to bottom code audit, > and obviously that was a mistake. > > It will probably take about 6 weeks. Apologies for wasting your time > before that was done. I don't think you'll get anywhere with auditing. We need to write down the semantics you want, define them at the VFS level and make sure they're not conflicting with defined userspace semantics or kernel assumptions. I think you need to learn the basic distinction between the VFS layer and a lowlevel filesystem driver. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 18:45 ` Christoph Hellwig @ 2004-08-26 9:02 ` Hans Reiser 2004-08-26 18:38 ` William Lee Irwin III 0 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 9:02 UTC (permalink / raw) To: Christoph Hellwig Cc: viro, Linus Torvalds, akpm, linux-fsdevel, linux-kernel, Jeff Garzik, Reiserfs developers mail-list Christoph Hellwig wrote: > > >I don't think you'll get anywhere with auditing. We need to write down >the semantics you want, define them at the VFS level and make sure >they're not conflicting with defined userspace semantics or kernel >assumptions. > >I think you need to learn the basic distinction between the VFS layer >and a lowlevel filesystem driver. > > > > > How old are you? I thought you were the guy at Linux Tag with fashion oriented hair who gave a talk on his XFS work? Did I confuse you with someone else? Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 9:02 ` Hans Reiser @ 2004-08-26 18:38 ` William Lee Irwin III 2004-08-26 18:49 ` Linus Torvalds 0 siblings, 1 reply; 1392+ messages in thread From: William Lee Irwin III @ 2004-08-26 18:38 UTC (permalink / raw) To: Hans Reiser Cc: Christoph Hellwig, viro, Linus Torvalds, akpm, linux-fsdevel, linux-kernel, Jeff Garzik, Reiserfs developers mail-list On Thu, Aug 26, 2004 at 02:02:29AM -0700, Hans Reiser wrote: > How old are you? I thought you were the guy at Linux Tag with fashion > oriented hair who gave a talk on his XFS work? Did I confuse you with > someone else? Index: mm1-2.6.9-rc1/CREDITS =================================================================== --- mm1-2.6.9-rc1.orig/CREDITS 2004-08-26 10:37:57.424567967 -0700 +++ mm1-2.6.9-rc1/CREDITS 2004-08-26 11:35:24.977260109 -0700 @@ -2677,6 +2677,14 @@ S: 79098 Freiburg S: Germany +N: Hans Reiser +E: reiser@namesys.com +W: http://www.namesys.com/ +D: official Linux kernel hairstyle critic +S: 6979 Exeter Dr +S: Oakland, CA 94611 +S: USA + N: Joerg Reuter E: jreuter@yaina.de W: http://yaina.de/jreuter/ ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:38 ` William Lee Irwin III @ 2004-08-26 18:49 ` Linus Torvalds 2004-08-26 18:56 ` William Lee Irwin III 0 siblings, 1 reply; 1392+ messages in thread From: Linus Torvalds @ 2004-08-26 18:49 UTC (permalink / raw) To: William Lee Irwin III Cc: Hans Reiser, Christoph Hellwig, viro, akpm, linux-fsdevel, linux-kernel, Jeff Garzik, Reiserfs developers mail-list On Thu, 26 Aug 2004, William Lee Irwin III wrote: > > +N: Hans Reiser > +E: reiser@namesys.com > +W: http://www.namesys.com/ > +D: official Linux kernel hairstyle critic > +S: 6979 Exeter Dr > +S: Oakland, CA 94611 > +S: USA > + LOL. In all fairness, how _would_ you have described it? Besides, I don't think this should go in the CREDITS file, since hair styling criticism is clearly an ongoing MAINTAINERS issue, no? Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:49 ` Linus Torvalds @ 2004-08-26 18:56 ` William Lee Irwin III 2004-08-27 8:14 ` Hans Reiser [not found] ` <MPG.1b9926e15d166dfe9896d8@news.gmane.org> 0 siblings, 2 replies; 1392+ messages in thread From: William Lee Irwin III @ 2004-08-26 18:56 UTC (permalink / raw) To: Linus Torvalds Cc: Hans Reiser, Christoph Hellwig, viro, akpm, linux-fsdevel, linux-kernel, Jeff Garzik, Reiserfs developers mail-list On Thu, 26 Aug 2004, William Lee Irwin III wrote: >> +N: Hans Reiser >> +E: reiser@namesys.com >> +W: http://www.namesys.com/ >> +D: official Linux kernel hairstyle critic >> +S: 6979 Exeter Dr >> +S: Oakland, CA 94611 >> +S: USA On Thu, Aug 26, 2004 at 11:49:19AM -0700, Linus Torvalds wrote: > LOL. > In all fairness, how _would_ you have described it? > Besides, I don't think this should go in the CREDITS file, since hair > styling criticism is clearly an ongoing MAINTAINERS issue, no? As far as maintenance goes, absolutely. As far as describing it goes, I probably would have said it was short, red, and spiky. Index: mm1-2.6.9-rc1/MAINTAINERS =================================================================== --- mm1-2.6.9-rc1.orig/MAINTAINERS 2004-08-26 10:38:09.867927190 -0700 +++ mm1-2.6.9-rc1/MAINTAINERS 2004-08-26 11:58:27.759469733 -0700 @@ -1352,6 +1352,12 @@ L: linuxppc64-dev@lists.linuxppc.org S: Supported +LINUX KERNEL HAIRSTYLE CRITICISM +P: Hans Reiser +M: reiser@namesys.com +W: http://www.namesys.com/ +S: Maintained + LINUX SECURITY MODULE (LSM) FRAMEWORK P: Chris Wright M: chrisw@osdl.org ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:56 ` William Lee Irwin III @ 2004-08-27 8:14 ` Hans Reiser [not found] ` <MPG.1b9926e15d166dfe9896d8@news.gmane.org> 1 sibling, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-27 8:14 UTC (permalink / raw) To: William Lee Irwin III Cc: Linus Torvalds, Christoph Hellwig, viro, akpm, linux-fsdevel, linux-kernel, Jeff Garzik, Reiserfs developers mail-list William Lee Irwin III wrote: > >As far as maintenance goes, absolutely. As far as describing it goes, >I probably would have said it was short, red, and spiky. > > > Ah, so it was him, is he disowning XFS these days or? ^ permalink raw reply [flat|nested] 1392+ messages in thread
[parent not found: <MPG.1b9926e15d166dfe9896d8@news.gmane.org>]
* Re: silent semantic changes with reiser4 [not found] ` <MPG.1b9926e15d166dfe9896d8@news.gmane.org> @ 2004-08-27 15:25 ` William Lee Irwin III [not found] ` <20040827161539.GC28876@khan.acc.umu.se> 0 siblings, 1 reply; 1392+ messages in thread From: William Lee Irwin III @ 2004-08-27 15:25 UTC (permalink / raw) To: Linus Torvalds; +Cc: Giuseppe Bilotta, linux-fsdevel, linux-kernel William Lee Irwin III wrote: >> +LINUX KERNEL HAIRSTYLE CRITICISM On Fri, Aug 27, 2004 at 12:10:12PM +0200, Giuseppe Bilotta wrote: > Hm. He doesn't critic the Linux Kernel hairstyle, but the Linux > Kernel Developers' ... Absolutely right. Index: mm1-2.6.9-rc1/MAINTAINERS =================================================================== --- mm1-2.6.9-rc1.orig/MAINTAINERS 2004-08-26 10:38:09.000000000 -0700 +++ mm1-2.6.9-rc1/MAINTAINERS 2004-08-27 08:27:42.738139199 -0700 @@ -1352,6 +1352,12 @@ L: linuxppc64-dev@lists.linuxppc.org S: Supported +LINUX KERNEL DEVELOPER'S HAIRSTYLE CRITICISM +P: Hans Reiser +M: reiser@namesys.com +W: http://www.namesys.com/ +S: Maintained + LINUX SECURITY MODULE (LSM) FRAMEWORK P: Chris Wright M: chrisw@osdl.org ^ permalink raw reply [flat|nested] 1392+ messages in thread
[parent not found: <20040827161539.GC28876@khan.acc.umu.se>]
* Re: silent semantic changes with reiser4 [not found] ` <20040827161539.GC28876@khan.acc.umu.se> @ 2004-08-27 16:18 ` William Lee Irwin III 0 siblings, 0 replies; 1392+ messages in thread From: William Lee Irwin III @ 2004-08-27 16:18 UTC (permalink / raw) To: David Weinehall; +Cc: akpm, torvalds, linux-fsdevel, linux-kernel On Fri, Aug 27, 2004 at 08:25:39AM -0700, William Lee Irwin III wrote: >> +LINUX KERNEL DEVELOPER'S HAIRSTYLE CRITICISM > Aren't there more than one Linux Kernel Developer? Or is this to > indicate that he indeed only criticised the hairstyle of one developer? Best I repair that, then. Index: mm1-2.6.9-rc1/MAINTAINERS =================================================================== --- mm1-2.6.9-rc1.orig/MAINTAINERS 2004-08-26 10:38:09.000000000 -0700 +++ mm1-2.6.9-rc1/MAINTAINERS 2004-08-27 09:18:59.234195262 -0700 @@ -1352,6 +1352,12 @@ L: linuxppc64-dev@lists.linuxppc.org S: Supported +LINUX KERNEL DEVELOPERS' HAIRSTYLE CRITICISM +P: Hans Reiser +M: reiser@namesys.com +W: http://www.namesys.com/ +S: Maintained + LINUX SECURITY MODULE (LSM) FRAMEWORK P: Chris Wright M: chrisw@osdl.org ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-24 20:25 silent semantic changes with reiser4 Christoph Hellwig 2004-08-24 20:35 ` Lee Revell 2004-08-24 20:38 ` Jeff Garzik @ 2004-08-25 19:53 ` Hans Reiser 2004-08-25 20:06 ` Matthew Wilcox ` (5 more replies) 2004-09-14 4:27 ` file-as-dir vs. dir? " Andy Lutomirski 3 siblings, 6 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-25 19:53 UTC (permalink / raw) To: Christoph Hellwig Cc: akpm, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List I had not intended to respond to this because I have nothing positive to say, but Andrew said I needed to respond and suggested I should copy Linus. Sigh. Dear Christoph, Let me see if I can summarize what you and your contingent are saying, and if I misconstrue anything, let me know.;-) You ignored everything I said during the discussion of xattrs about how there is no need to have attributes when you can just have files and directories, and that xattrs reflected a complete ignorance of name space design principles. When I said we should just add some nice optional features to files and directories so that they can do everything that attributes can do if they are used that way, you just didn't get it. You instead went for the quick ugly hack called xattrs. You then got that ugly hack done first, because quick hacks are, well, quick. I then went about doing it the right way for Reiser4, and got DARPA to fund doing it. I was never silent about it. Making files into directories caused only two applications out of the entire OS to notice the change, and that was because of a bug in what error code we returned that we are going to fix. You think that was a disaster; I think it was a triumph. Now a cleanly architected filesystem with no attributes and just files and directories that can do everything attributes are used for exists. You don't want it to have the competitive advantage. Instead, you want it to have its clean design excised until you have something that duplicates it ready to go, and only then should it be allowed that users will use the features of your competitor's filesystem which you disdained implementing for so long. Since you never studied or understood namespace design principles (or you would not have created and supported xattrs), you want to rename it to be called VFS, rewrite what we have done, and take over as the maintainer, mangling its design in a committee clusterfuck as you go. We have just implemented very trivial semantic enhancements of the FS namespace, nothing like as ambitious as www.namesys.com/whitepaper.html or WinFS, and you are already pissing your pants. Is that a fair summary? Eat my dust. Hans PS I should of course qualify what I have said. The use of files and directories in place of attributes is not a finished work. It has bugs, sys_reiser4() does not yet work, and there are little features still missing like having files readdir ignores. Still, except for the bugs, what we have is usable, and there are a lot of happy reiser4 users right now even with the bugs. It will need a little bit more time, and then all the pieces will be in place. PPS If you implement your filesystems as reiser4 plugins, and rename reiser4's plugin code to be called "vfs", your filesystems will go faster. Not as fast as reiser4 though, because it has a better layout and that affects performance a lot, but faster is faster.... See www.namesys.com/benchmarks.html for details. PPPS Since we have such a performance lead, Namesys is about to change its focus from the storage layer to semantics, look at www.namesys.com/whitepaper.html for details. Semantic enhancements are the important stuff, and finally Namesys is where we have all the storage layer prerequisites done right, and the real work can begin. The gap between us is about to widen further. Christoph Hellwig wrote: >After looking trough the code and mailinglists I'm quite unhappy with >a bunch of user-visible changes that Hans sneaked in and make reiser4 >incompatible with other filesystems > if we leave you in the dust, run faster.... not my problem.... >Given these problems I request that these interfaces are removed from >reiser4 for the kernel merge, and if added later at the proper VFS level >after discussion on linux-kernel and linux-fsdevel, like we did for >xattrs. > > > > If you can't help fight WinFS, then get out of the way. Namesys is on the march. Read www.namesys.com/whitepaper.html. Or, be smart, recognize that reiser4 is faster and more flexible than your storage layers because we are older and wiser and worked harder at it, join the team, and start contributing plugins that tap into the higher performance it offers. Microsoft tried to build a storage layer that could handle small objects without losing performance, failed, and gave up at considerable cost to their architecture and pocketbook. We just broke a hole in the enemy line. You could come swarming through it with us, but it sounds like you prefer complaining to HQ that we are getting too far in front of you. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 19:53 ` Hans Reiser @ 2004-08-25 20:06 ` Matthew Wilcox 2004-08-26 8:41 ` Hans Reiser 2004-08-25 20:08 ` Christoph Hellwig ` (4 subsequent siblings) 5 siblings, 1 reply; 1392+ messages in thread From: Matthew Wilcox @ 2004-08-25 20:06 UTC (permalink / raw) To: Hans Reiser Cc: Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List That's a nice marketing talk. Get back to us when you have some technical contribution to make. -- "Next the statesmen will invent cheap lies, putting the blame upon the nation that is attacked, and every man will be glad of those conscience-soothing falsities, and will diligently study them, and refuse to examine any refutations of them; and thus he will by and by convince himself that the war is just, and will thank God for the better sleep he enjoys after this process of grotesque self-deception." -- Mark Twain ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:06 ` Matthew Wilcox @ 2004-08-26 8:41 ` Hans Reiser 0 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 8:41 UTC (permalink / raw) To: Matthew Wilcox Cc: Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Matthew Wilcox wrote: >That's a nice marketing talk. Get back to us when you have some technical >contribution to make. > > > www.namesys.com/download.html ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 19:53 ` Hans Reiser 2004-08-25 20:06 ` Matthew Wilcox @ 2004-08-25 20:08 ` Christoph Hellwig 2004-08-25 20:19 ` Christoph Hellwig ` (2 more replies) 2004-08-25 20:20 ` Jeremy Allison ` (3 subsequent siblings) 5 siblings, 3 replies; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-25 20:08 UTC (permalink / raw) To: Hans Reiser Cc: linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Wed, Aug 25, 2004 at 12:53:28PM -0700, Hans Reiser wrote: > You ignored everything I said during the discussion of xattrs about how > there is no need to have attributes when you can just have files and > directories, and that xattrs reflected a complete ignorance of name > space design principles. Actually in most of the discussion you simply didn't participate. While xattrs might not be the nicest interface they have the advantag of not breaking the SuS assumption of what directories vs files are, and they do not break the Linux O_DIRECTORY semantics that are defined and need to solve real-world races either. > When I said we should just add some nice > optional features to files and directories so that they can do > everything that attributes can do if they are used that way, you just > didn't get it. You instead went for the quick ugly hack called xattrs. > You then got that ugly hack done first, because quick hacks are, well, > quick. I then went about doing it the right way for Reiser4, and got > DARPA to fund doing it. I was never silent about it. For one thing _I_ didn't decide about xattrs anyway. And I still haven't seen a design from you on -fsdevel how you try to solve the problems with files as directories. > Now a cleanly architected filesystem with no attributes and just files > and directories that can do everything attributes are used for exists. > You don't want it to have the competitive advantage. Instead, you want > it to have its clean design excised until you have something that > duplicates it ready to go, and only then should it be allowed that users > will use the features of your competitor's filesystem which you > disdained implementing for so long. My competitors filesystem? If you look at MAINTAINERS I maintain only vxfs and sysvfs, neither of which I'd suggest anyone to run their system on. > Since you never studied or understood namespace design principles (or > you would not have created and supported xattrs), you want to rename it > to be called VFS, rewrite what we have done, and take over as the > maintainer, mangling its design in a committee clusterfuck as you go. Hans, please stop the personal crap or the black helicopters will kidnap you. When was the last time you actually worked on kernel namespace code instead of talking marketing bullshit and ignoring all real world problems. > If you implement your filesystems as reiser4 plugins, and rename > reiser4's plugin code to be called "vfs", your filesystems will go > faster. Not as fast as reiser4 though, because it has a better layout > and that affects performance a lot, but faster is faster.... See > www.namesys.com/benchmarks.html for details. Could you pass on that crack pipe please? ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:08 ` Christoph Hellwig @ 2004-08-25 20:19 ` Christoph Hellwig 2004-08-25 20:24 ` Linus Torvalds 2004-08-26 8:42 ` Hans Reiser 2004-08-25 20:22 ` Linus Torvalds 2004-08-25 20:35 ` Alex Zarochentsev 2 siblings, 2 replies; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-25 20:19 UTC (permalink / raw) To: Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Btw, I just got reminded you might take my saying as an "piss of you're idea stinks" or similar things. So let me clarify again the actual technical and project managment issues another time before we start getting really personal :) Over the last at least five years we've taken as much as possible semantics out of the filesystems and into the VFS layer, thus having a separation between the semantical layer (VFS) and the low level filesystem. Your attributes are absoultely a VFS thing and as such should not happen at the filesystem layer, and no, that doesn't mean they're bad per se, I just think they are a rather bad fit for Linux. So now go on and try to work together with the other peope doing VFS level work instead of hiding, or if you think you can't work together with us search a nice research OS where you can take over the VFS layer, if your ideas prove to be good I'm sure Linux will pick them up sooner or later. Christoph ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:19 ` Christoph Hellwig @ 2004-08-25 20:24 ` Linus Torvalds 2004-08-25 20:25 ` Christoph Hellwig ` (2 more replies) 2004-08-26 8:42 ` Hans Reiser 1 sibling, 3 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-08-25 20:24 UTC (permalink / raw) To: Christoph Hellwig Cc: Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, 25 Aug 2004, Christoph Hellwig wrote: > > Over the last at least five years we've taken as much as possible > semantics out of the filesystems and into the VFS layer, thus having > a separation between the semantical layer (VFS) and the low level > filesystem. Your attributes are absoultely a VFS thing and as such > should not happen at the filesystem layer, and no, that doesn't mean > they're bad per se, I just think they are a rather bad fit for Linux. Now this I agree with, in the sense that I think that if we want to support this, it should be supported at a VFS layer. On the other hand, I think doing it inside the filesystem with ugly hacks is an acceptable way to prototype the idea before it's been proven to really be workable. Maybe it has more problems with legacy apps than we'd expect.. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:24 ` Linus Torvalds @ 2004-08-25 20:25 ` Christoph Hellwig 2004-08-25 20:59 ` viro 2004-08-26 8:43 ` Hans Reiser 2 siblings, 0 replies; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-25 20:25 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, Aug 25, 2004 at 01:24:36PM -0700, Linus Torvalds wrote: > Now this I agree with, in the sense that I think that if we want to > support this, it should be supported at a VFS layer. > > On the other hand, I think doing it inside the filesystem with ugly hacks > is an acceptable way to prototype the idea before it's been proven to > really be workable. Maybe it has more problems with legacy apps than we'd > expect.. Oh, I'm the last person to tell anyone how to prototype things. I just don't want such inconsistancies in the mainline kernel. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:24 ` Linus Torvalds 2004-08-25 20:25 ` Christoph Hellwig @ 2004-08-25 20:59 ` viro 2004-08-26 8:43 ` Hans Reiser 2 siblings, 0 replies; 1392+ messages in thread From: viro @ 2004-08-25 20:59 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, Aug 25, 2004 at 01:24:36PM -0700, Linus Torvalds wrote: > > > On Wed, 25 Aug 2004, Christoph Hellwig wrote: > > > > Over the last at least five years we've taken as much as possible > > semantics out of the filesystems and into the VFS layer, thus having > > a separation between the semantical layer (VFS) and the low level > > filesystem. Your attributes are absoultely a VFS thing and as such > > should not happen at the filesystem layer, and no, that doesn't mean > > they're bad per se, I just think they are a rather bad fit for Linux. > > Now this I agree with, in the sense that I think that if we want to > support this, it should be supported at a VFS layer. ACK. However, I'm still not seeing *ANYTHING* that would look like a workable scheme in presense of hardlinks. Show me how to make that deadlock- and race-free and we might very well do it in VFS. _That_ is what's missing and it's needed no matter where it's implemented. You want hybrid objects - you want to solve that one. So far I've seen nothing workable. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:24 ` Linus Torvalds 2004-08-25 20:25 ` Christoph Hellwig 2004-08-25 20:59 ` viro @ 2004-08-26 8:43 ` Hans Reiser 2 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 8:43 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Linus Torvalds wrote: >On Wed, 25 Aug 2004, Christoph Hellwig wrote: > > >>Over the last at least five years we've taken as much as possible >>semantics out of the filesystems and into the VFS layer, thus having >>a separation between the semantical layer (VFS) and the low level >>filesystem. Your attributes are absoultely a VFS thing and as such >>should not happen at the filesystem layer, and no, that doesn't mean >>they're bad per se, I just think they are a rather bad fit for Linux. >> >> > >Now this I agree with, in the sense that I think that if we want to >support this, it should be supported at a VFS layer. > >On the other hand, I think doing it inside the filesystem with ugly hacks > > what is ugly? ;-/ >is an acceptable way to prototype the idea before it's been proven to >really be workable. Maybe it has more problems with legacy apps than we'd >expect.. > > Linus > > > > ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:19 ` Christoph Hellwig 2004-08-25 20:24 ` Linus Torvalds @ 2004-08-26 8:42 ` Hans Reiser 2004-08-26 9:24 ` Christoph Hellwig 2004-08-26 13:38 ` Chris Mason 1 sibling, 2 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 8:42 UTC (permalink / raw) To: Christoph Hellwig Cc: linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Christoph Hellwig wrote: > >Over the last at least five years we've taken as much as possible >semantics out of the filesystems and into the VFS layer, thus having >a separation between the semantical layer (VFS) and the low level >filesystem. > VFS is the common filesystem layer. The only reason you think semantics belong in the common filesystem layer is that you are not innovating in your semantics, and feel content with stasis. I don't. I expect that semantics will get radically changed over the next few years as we compete with Giampaolo and whatever lesser lights are working at Microsoft. > Your attributes are absoultely a VFS thing and as such >should not happen at the filesystem layer, and no, that doesn't mean >they're bad per se, I just think they are a rather bad fit for Linux. > >So now go on and try to work together with the other peope doing VFS >level work instead of hiding, or if you think you can't work together >with us search a nice research OS where you can take over the VFS layer, >if your ideas prove to be good I'm sure Linux will pick them up sooner >or later. > > Christoph > > > > I tell you what, use xattrs for all the half speed filesystems, and the users and I will use metafiles. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:42 ` Hans Reiser @ 2004-08-26 9:24 ` Christoph Hellwig 2004-08-26 18:45 ` Hans Reiser 2004-08-26 13:38 ` Chris Mason 1 sibling, 1 reply; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-26 9:24 UTC (permalink / raw) To: Hans Reiser Cc: Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Thu, Aug 26, 2004 at 01:42:06AM -0700, Hans Reiser wrote: > Christoph Hellwig wrote: > > > > >Over the last at least five years we've taken as much as possible > >semantics out of the filesystems and into the VFS layer, thus having > >a separation between the semantical layer (VFS) and the low level > >filesystem. > > > VFS is the common filesystem layer. The only reason you think semantics > belong in the common filesystem layer is that you are not innovating in > your semantics, and feel content with stasis. > > I don't. I expect that semantics will get radically changed over the > next few years as we compete with Giampaolo and whatever lesser lights > are working at Microsoft. Hans, please stop the gooddamn personal attack bullshit. How do you for example suggestion exporting your semantics over the network if they're not done at the VFS level? How do you want some clusterfilesystem support them or tmpfs? ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 9:24 ` Christoph Hellwig @ 2004-08-26 18:45 ` Hans Reiser 0 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 18:45 UTC (permalink / raw) To: Christoph Hellwig Cc: linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Christoph Hellwig wrote: > > >How do you for example suggestion exporting your semantics over the >network if they're not done at the VFS level? How do you want some >clusterfilesystem support them or tmpfs? > > > > by accesses to filename/metas..... That's the beauty of simplicity, not much is needed. xattrs on the other hand would be a lot of work to integrate into a cluster filesystem. Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:42 ` Hans Reiser 2004-08-26 9:24 ` Christoph Hellwig @ 2004-08-26 13:38 ` Chris Mason 2004-08-26 23:53 ` Hans Reiser 1 sibling, 1 reply; 1392+ messages in thread From: Chris Mason @ 2004-08-26 13:38 UTC (permalink / raw) To: Hans Reiser Cc: Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Thu, 2004-08-26 at 04:42, Hans Reiser wrote: > Christoph Hellwig wrote: > > > > >Over the last at least five years we've taken as much as possible > >semantics out of the filesystems and into the VFS layer, thus having > >a separation between the semantical layer (VFS) and the low level > >filesystem. > > > VFS is the common filesystem layer. The only reason you think semantics > belong in the common filesystem layer is that you are not innovating in > your semantics, and feel content with stasis. This misunderstanding is the heart of the problem really. In many ways, the VFS is the semantic layer, it defines the picture of the directory tree and the locking rules the applications see. There's no rule that says you can't innovate in the VFS, in fact it is where the semantic innovations belong. Putting them there is what allows other filesystems to implement your semantics, and this is exactly what linux needs if these new semantics are going to be standardized. In other words, writing a reiser4 plugin is one way to get to the new semantics, but it must not be the only way. Keeping them entirely within reiser4 is good for the research stage, but for actual deployment they need to be finalized and pushed up. If the early linux filesystems had taken the same attitude you have (don't write new filesystems, only write plugins), there would be no framework allowing the wealth of filesystems we do have, including reiser4. -chris ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 13:38 ` Chris Mason @ 2004-08-26 23:53 ` Hans Reiser 2004-08-27 9:16 ` Christoph Hellwig 0 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 23:53 UTC (permalink / raw) To: Chris Mason Cc: Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Chris Mason wrote: > > >If the early linux filesystems had taken the same attitude you have >(don't write new filesystems, only write plugins), there would be no >framework allowing the wealth of filesystems we do have, including >reiser4. > > > Au contraire, with my approach, there would be 10x as many filesystems, because people would be easily able to mix-and-match different methods and plugins from different filesystems together, and if you wanted the MySQL semantics, you would mount mysql, and if you wanted the zope semantics you would mount zope. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 23:53 ` Hans Reiser @ 2004-08-27 9:16 ` Christoph Hellwig 0 siblings, 0 replies; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-27 9:16 UTC (permalink / raw) To: Hans Reiser Cc: Chris Mason, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Thu, Aug 26, 2004 at 04:53:51PM -0700, Hans Reiser wrote: > >If the early linux filesystems had taken the same attitude you have > >(don't write new filesystems, only write plugins), there would be no > >framework allowing the wealth of filesystems we do have, including > >reiser4. > > > > > > > Au contraire, with my approach, there would be 10x as many filesystems, > because people would be easily able to mix-and-match different methods > and plugins from different filesystems together, and if you wanted the > MySQL semantics, you would mount mysql, and if you wanted the zope > semantics you would mount zope. And if you wanted to make free love you'd make free love you'd make free love and there would be piece on all earth.. You seem to have lost contact to reality a little. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:08 ` Christoph Hellwig 2004-08-25 20:19 ` Christoph Hellwig @ 2004-08-25 20:22 ` Linus Torvalds 2004-08-25 20:35 ` Christoph Hellwig ` (7 more replies) 2004-08-25 20:35 ` Alex Zarochentsev 2 siblings, 8 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-08-25 20:22 UTC (permalink / raw) To: Christoph Hellwig Cc: Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, 25 Aug 2004, Christoph Hellwig wrote: > > For one thing _I_ didn't decide about xattrs anyway. And I still > haven't seen a design from you on -fsdevel how you try to solve the > problems with files as directories. Hey, files-as-directories are one of my pet things, so I have to side with Hans on this one. I think it just makes sense. A hell of a lot more sense than xattrs, anyway, since it allows scripts etc standard tools to touch the attributes. It's the UNIX way. And yes, the semantics can _easily_ be solved in very unixy ways. One way to solve it is to just realize that a final slash at the end implies pretty strongly that you want to treat it as a directory. So what you do is: - without the slash, a file-as-dir won't open with O_DIRECTORY (ENOTDIR) - with the slash, it won't open _without_ O_DIRECTORY (EISDIR) Problem solved. Very user-friendly, and very intuitive. Will it potentially break something? Sure. Do we care? Me, I'll take that kind of extension _any_ day over xattrs, that are fundamentally flawed in my opinion and totally useless. The argument that applications like "tar" won't understand the file-as-directory thing is _flawed_, since legacy apps won't understand xattrs either. Oh, add a O_NOXATTRS flag to force a path lookup to only use regular directories, the same way we have O_NOFOLLOW and friends. That allows people to see the difference, if they care (ie a file server might decide that it doesn't want to expose things like this). I never liked the xattr stuff. It makes little sense, and is totally useless for 99.9999% of everything. I still don't see the point of it, except for samba. Ugly. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:22 ` Linus Torvalds @ 2004-08-25 20:35 ` Christoph Hellwig 2004-08-25 20:41 ` Hans Reiser ` (6 subsequent siblings) 7 siblings, 0 replies; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-25 20:35 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, Aug 25, 2004 at 01:22:55PM -0700, Linus Torvalds wrote: > And yes, the semantics can _easily_ be solved in very unixy ways. > > One way to solve it is to just realize that a final slash at the end > implies pretty strongly that you want to treat it as a directory. So what > you do is: > > - without the slash, a file-as-dir won't open with O_DIRECTORY (ENOTDIR) > - with the slash, it won't open _without_ O_DIRECTORY (EISDIR) > > Problem solved. Very user-friendly, and very intuitive. That would solve the O_DIRECTORY issue, the dentry aliasing still needs work though with the semantics for link/unlink/rename. Maybe Hans & you should start 2.7 to work this out? :) ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:22 ` Linus Torvalds 2004-08-25 20:35 ` Christoph Hellwig @ 2004-08-25 20:41 ` Hans Reiser 2004-08-25 20:51 ` Chris Mason ` (2 more replies) 2004-08-25 20:42 ` viro ` (5 subsequent siblings) 7 siblings, 3 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-25 20:41 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List, Nikita Danilov I just want to add that I AM capable of working with the other filesystem developers in a team-player way, and I am happy to cooperate with making portions more reusable where there is serious interest from other filesystems in that, but Christoph is a puppy who has never written or designed a major filesystem from scratch, and Nikita is a big dog who has written stuff very few projects are lucky enough to see the likes of, and when Christoph insults Nikita's code, or my design guidance for that code, it is not going to bring out my good side. The plugin and metafiles code needs many improvements, but Christoph does not have the expertise to understand what those needed improvements are because he hasn't invested the work into understanding the code. Christoph is a bright and clever young fellow, who just hasn't had the years of study of the field yet. I wish him well, and away.;-) Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:41 ` Hans Reiser @ 2004-08-25 20:51 ` Chris Mason 2004-08-25 20:58 ` Markus Törnqvist 2004-08-25 21:03 ` Rik van Riel 2 siblings, 0 replies; 1392+ messages in thread From: Chris Mason @ 2004-08-25 20:51 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List, Nikita Danilov On Wed, 2004-08-25 at 16:41, Hans Reiser wrote: > I just want to add that I AM capable of working with the other > filesystem developers in a team-player way, and I am happy to cooperate > with making portions more reusable where there is serious interest from > other filesystems in that, Prove it. Stop replying for today and come back tomorrow with some useful discussions. Christoph suggested that some of the v4 semantics belong in the VFS and therefore linux as a whole. He's helping you to make sure the semantics and fit nicely with the rest of kernel interfaces and are race free. Take him up on the offer. -chris ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:41 ` Hans Reiser 2004-08-25 20:51 ` Chris Mason @ 2004-08-25 20:58 ` Markus Törnqvist 2004-08-25 21:03 ` Rik van Riel 2 siblings, 0 replies; 1392+ messages in thread From: Markus Törnqvist @ 2004-08-25 20:58 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List, Nikita Danilov On Wed, Aug 25, 2004 at 01:41:14PM -0700, Hans Reiser wrote: >Christoph is a bright and clever young fellow, who just hasn't had the >years of study of the field yet. I wish him well, and away.;-) I see this is as an opportunity where you can share some of your experience to Cristoph and many others and work to get the semantics into VFS. Please make this work :) -- mjt ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:41 ` Hans Reiser 2004-08-25 20:51 ` Chris Mason 2004-08-25 20:58 ` Markus Törnqvist @ 2004-08-25 21:03 ` Rik van Riel 2004-08-26 9:00 ` Hans Reiser 2 siblings, 1 reply; 1392+ messages in thread From: Rik van Riel @ 2004-08-25 21:03 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List, Nikita Danilov On Wed, 25 Aug 2004, Hans Reiser wrote: > other filesystems in that, but Christoph is a puppy who has never It's not Christoph who's shown more bark than bite in this thread. -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 21:03 ` Rik van Riel @ 2004-08-26 9:00 ` Hans Reiser 0 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 9:00 UTC (permalink / raw) To: Rik van Riel Cc: Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List, Nikita Danilov Rik van Riel wrote: >On Wed, 25 Aug 2004, Hans Reiser wrote: > > > >>other filesystems in that, but Christoph is a puppy who has never >> >> > >It's not Christoph who's shown more bark than bite in this thread. > > > The bite is at www.namesys.com/download.html Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:22 ` Linus Torvalds 2004-08-25 20:35 ` Christoph Hellwig 2004-08-25 20:41 ` Hans Reiser @ 2004-08-25 20:42 ` viro 2004-08-25 21:00 ` Christophe Saout ` (2 more replies) 2004-08-25 21:52 ` Matt Mackall ` (4 subsequent siblings) 7 siblings, 3 replies; 1392+ messages in thread From: viro @ 2004-08-25 20:42 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, Aug 25, 2004 at 01:22:55PM -0700, Linus Torvalds wrote: > > > On Wed, 25 Aug 2004, Christoph Hellwig wrote: > > > > For one thing _I_ didn't decide about xattrs anyway. And I still > > haven't seen a design from you on -fsdevel how you try to solve the > > problems with files as directories. > > Hey, files-as-directories are one of my pet things, so I have to side with > Hans on this one. I think it just makes sense. A hell of a lot more sense > than xattrs, anyway, since it allows scripts etc standard tools to touch > the attributes. > > It's the UNIX way. Not if you allow link(2) on them. And not if you design and market your stuff as a general-purpose backdoor into kernel. Note how *EVERY* *DAMN* *OPERATION* is made possible to override by "plugins". Which is the reason for deadlocks in question, BTW. Don't fool yourself - that's what Hans is selling. Target market: ISV. Marketed product: a set of hooks, the wider the better, no matter how little sense it makes. The reason for doing that outside of core kernel: bypassing any review and being able to control the product being sold (see above). Shame that it got an actual filesystem mixed in with the marketing plans and general insanity... ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:42 ` viro @ 2004-08-25 21:00 ` Christophe Saout 2004-08-25 22:59 ` Andrea Arcangeli 2004-08-25 21:00 ` Linus Torvalds 2004-08-26 8:43 ` Hans Reiser 2 siblings, 1 reply; 1392+ messages in thread From: Christophe Saout @ 2004-08-25 21:00 UTC (permalink / raw) To: viro Cc: Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 2418 bytes --] Am Mittwoch, den 25.08.2004, 21:42 +0100 schrieb : > > > For one thing _I_ didn't decide about xattrs anyway. And I still > > > haven't seen a design from you on -fsdevel how you try to solve the > > > problems with files as directories. > > > > Hey, files-as-directories are one of my pet things, so I have to side with > > Hans on this one. I think it just makes sense. A hell of a lot more sense > > than xattrs, anyway, since it allows scripts etc standard tools to touch > > the attributes. > > > > It's the UNIX way. > > Not if you allow link(2) on them. That doesn't make sense anyway. (actually, I tried what happens and the result was an Oops ;)) It should be completely forbidden to link into a meta-directory or out of such a directory. You could think of those meta-directory as a sysfs for that inode. Of course it's not an own filesystem and that means that there need to be a lot of security precautions in the VFS layer. Where something like that belongs anyway, if done correctly. > And not if you design and market your > stuff as a general-purpose backdoor into kernel. Note how *EVERY* *DAMN* > *OPERATION* is made possible to override by "plugins". Which is the reason > for deadlocks in question, BTW. What do you mean? If you tell that file that you want it to be compressed or encrypted or modify some attributes (like ACLs) this isn't necessarily a backdoor. > Don't fool yourself - that's what Hans is selling. Target market: ISV. > Marketed product: a set of hooks, the wider the better, no matter how > little sense it makes. The reason for doing that outside of core kernel: > bypassing any review and being able to control the product being sold (see > above). Yes, I don't think it was a good idea either. Probably someone should remove these features and make it a "normal" filesystem. The people who need it now can turn it on again and a real solution could be worked out in Linux 2.7. I wouldn't use it on a public server anyway now because I'm not convinced some malicious guy could find a way to exploit that. What if you changed into a meta directory using ftp and some manage to break things? This might be very dangerous. I personally think that the idea of doing something like this (I'm not speaking of the current implementation which I think is really bad) is the right way to go in the long term. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 21:00 ` Christophe Saout @ 2004-08-25 22:59 ` Andrea Arcangeli 2004-08-26 8:35 ` Hans Reiser 0 siblings, 1 reply; 1392+ messages in thread From: Andrea Arcangeli @ 2004-08-25 22:59 UTC (permalink / raw) To: Christophe Saout Cc: viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, Aug 25, 2004 at 11:00:00PM +0200, Christophe Saout wrote: > It should be completely forbidden to link into a meta-directory or out > of such a directory. [..] agreed. > Yes, I don't think it was a good idea either. Probably someone should I personally would like plugins only if the API they use wouldn't allow them to corrupt the underlying fs, I'm not sure if this is the case with reiserfs4. About the backwards compatibility, another option is to add a O_FILEDIR and have bash learn about it when you cd into a file. No magic with the slashes then. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 22:59 ` Andrea Arcangeli @ 2004-08-26 8:35 ` Hans Reiser 2004-08-26 11:28 ` Andrea Arcangeli 0 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 8:35 UTC (permalink / raw) To: Andrea Arcangeli Cc: Christophe Saout, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Andrea Arcangeli wrote: >On Wed, Aug 25, 2004 at 11:00:00PM +0200, Christophe Saout wrote: > > >>It should be completely forbidden to link into a meta-directory or out >>of such a directory. [..] >> >> > >agreed. > > > >>Yes, I don't think it was a good idea either. Probably someone should >> >> > >I personally would like plugins only if the API they use wouldn't allow >them to corrupt the underlying fs, I'm not sure if this is the case with >reiserfs4. > > You compile crap into the kernel, you are screwed. We have not and will not change that. Reiser4 plugins are not for end users to download from amazon.com, they are for weekend hackers to send me a cool plugin for me to review, assign a plugin id to, and send to Linus in the next release. Sometimes being less ambitious works better, and dynamically loaded plugins would have made the infrastructure too bulky to be fun and fast. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:35 ` Hans Reiser @ 2004-08-26 11:28 ` Andrea Arcangeli 2004-08-26 11:44 ` Christophe Saout ` (3 more replies) 0 siblings, 4 replies; 1392+ messages in thread From: Andrea Arcangeli @ 2004-08-26 11:28 UTC (permalink / raw) To: Hans Reiser Cc: Christophe Saout, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 01:35:01AM -0700, Hans Reiser wrote: > Reiser4 plugins are not for end users to download from amazon.com, they > are for weekend hackers to send me a cool plugin for me to review, > assign a plugin id to, and send to Linus in the next release. Sometimes then what's the difference in having the plugin fixed in stone into reiserfs? That's my whole point. Get the patch from the weekend hacker, check it, send the patch to Linus to add the new feature to reiser4, just call it "feature" not plugin. That's how it works normally for everything. Many fs have many features, many of them optional. you wouldn't need to build any hook infrastructure either that way. Hooks would be needed if this wasn't open source me thinks. Or if you want people to fetch the module from amazon without your review. Another reason I could see the modularization/hooking useful is if those feature would take lots of kernel space, but this sure isn't the case for reiserfs, infact having modules would waste _more_ ram due the half wasted page allocation for the module text. The only single reason we use modules is to avoid wasting tons of ram by loading every possible device driver on earth, it's not that we use modules because they're more flexible, if something they're more fragile. I never use modules in my test kernel just to go fast (because I self compile them). The last good thing of the modules is during development you don't need to reboot the machine to test a kernel change in a driver, but you don't need that with reiserfs since you can make the fs itself a module (just change the name of the fs, AFIK you do that all the time already). ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:28 ` Andrea Arcangeli @ 2004-08-26 11:44 ` Christophe Saout 2004-08-26 12:16 ` Andrea Arcangeli 2004-08-26 23:53 ` Hans Reiser ` (2 subsequent siblings) 3 siblings, 1 reply; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 11:44 UTC (permalink / raw) To: Andrea Arcangeli Cc: Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 640 bytes --] Am Donnerstag, den 26.08.2004, 13:28 +0200 schrieb Andrea Arcangeli: > then what's the difference in having the plugin fixed in stone into > reiserfs? That's my whole point. I think you're not exactly understanding what a reiser4 "plugin" is. Currently plugins are not modules. It just means that there's a defined interface between the reiser4 core and the plugins. Just like you could see filesystems as "VFS plugins". In fact, reiser4 plugins are - users of the reiser4 core API - some of them are implementing Linux VFS methods (thus being some sort of glue code between the Reiser4 storage tree and the Linux VFS) [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:44 ` Christophe Saout @ 2004-08-26 12:16 ` Andrea Arcangeli 2004-08-26 12:23 ` Christophe Saout 2004-08-27 18:38 ` Alex Zarochentsev 0 siblings, 2 replies; 1392+ messages in thread From: Andrea Arcangeli @ 2004-08-26 12:16 UTC (permalink / raw) To: Christophe Saout Cc: Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 01:44:59PM +0200, Christophe Saout wrote: > I think you're not exactly understanding what a reiser4 "plugin" is. > > Currently plugins are not modules. It just means that there's a defined > interface between the reiser4 core and the plugins. Just like you could > see filesystems as "VFS plugins". but filesystems are exactly always modules (in precompiled kernels at least). Anyways I don't see any visible EXPORT_SYMBOL in reiser4. > > In fact, reiser4 plugins are > - users of the reiser4 core API > - some of them are implementing Linux VFS methods (thus being some sort > of glue code between the Reiser4 storage tree and the Linux VFS) then it seems a bit misleading to call those things plugins. As you wrote those are "users of the reiser4 core API", with plugins I always meant dynamically loadable things, like plugins for web browers (hence kernel modules in kernel space). While here apparently you can't plugin anything at runtime, it's just code that uses the reiser4 core API that can be modified with a kernel patch as usual. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 12:16 ` Andrea Arcangeli @ 2004-08-26 12:23 ` Christophe Saout 2004-08-27 18:38 ` Alex Zarochentsev 1 sibling, 0 replies; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 12:23 UTC (permalink / raw) To: Andrea Arcangeli Cc: Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 1469 bytes --] Am Donnerstag, den 26.08.2004, 14:16 +0200 schrieb Andrea Arcangeli: > > Currently plugins are not modules. It just means that there's a defined > > interface between the reiser4 core and the plugins. Just like you could > > see filesystems as "VFS plugins". > > but filesystems are exactly always modules (in precompiled kernels at > least). Yes, of cousre. > Anyways I don't see any visible EXPORT_SYMBOL in reiser4. Yes, there's no module infrastructure there. These "plugins" need to be linked at compile time. Some plugins need to be anyway since the the filesystem is useless without them. Others whould be optional features. > > In fact, reiser4 plugins are > > - users of the reiser4 core API > > - some of them are implementing Linux VFS methods (thus being some sort > > of glue code between the Reiser4 storage tree and the Linux VFS) > > then it seems a bit misleading to call those things plugins. > > As you wrote those are "users of the reiser4 core API", with plugins I > always meant dynamically loadable things, like plugins for web browers > (hence kernel modules in kernel space). While here apparently you can't > plugin anything at runtime, it's just code that uses the reiser4 core > API that can be modified with a kernel patch as usual. Right. The name is probably at bit misleading. But assuming the compression "plugin" whould be dynamically loadable at some later point it would be a plugin then? [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 12:16 ` Andrea Arcangeli 2004-08-26 12:23 ` Christophe Saout @ 2004-08-27 18:38 ` Alex Zarochentsev 2004-08-28 6:17 ` Hans Reiser 1 sibling, 1 reply; 1392+ messages in thread From: Alex Zarochentsev @ 2004-08-27 18:38 UTC (permalink / raw) To: Andrea Arcangeli Cc: Christophe Saout, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 02:16:30PM +0200, Andrea Arcangeli wrote: > On Thu, Aug 26, 2004 at 01:44:59PM +0200, Christophe Saout wrote: > > I think you're not exactly understanding what a reiser4 "plugin" is. > > > > Currently plugins are not modules. It just means that there's a defined > > interface between the reiser4 core and the plugins. Just like you could > > see filesystems as "VFS plugins". > > but filesystems are exactly always modules (in precompiled kernels at > least). > > Anyways I don't see any visible EXPORT_SYMBOL in reiser4. they would be there, I think. > > > > > In fact, reiser4 plugins are > > - users of the reiser4 core API > > - some of them are implementing Linux VFS methods (thus being some sort > > of glue code between the Reiser4 storage tree and the Linux VFS) > > then it seems a bit misleading to call those things plugins. There should be infracture to change plugin on fly, and choosing object's plugins at object's creation time. Currently only certain object's plugins can be changed through <object>/metas/plugin interface. > > As you wrote those are "users of the reiser4 core API", with plugins I > always meant dynamically loadable things, like plugins for web browers > (hence kernel modules in kernel space). While here apparently you can't > plugin anything at runtime, it's just code that uses the reiser4 core > API that can be modified with a kernel patch as usual. -- Alex. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 18:38 ` Alex Zarochentsev @ 2004-08-28 6:17 ` Hans Reiser 0 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-28 6:17 UTC (permalink / raw) To: Alex Zarochentsev Cc: Andrea Arcangeli, Christophe Saout, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Alex Zarochentsev wrote: > > >There should be infracture to change plugin on fly, and choosing object's >plugins at object's creation time. Currently only certain object's plugins >can be changed through <object>/metas/plugin interface. > > To be more clear for those not familiar with our project, there is much more work to be done, and one of the areas that (extremely) needs more work is changing plugins on the fly and clean interfaces for specifying them at creation time. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:28 ` Andrea Arcangeli 2004-08-26 11:44 ` Christophe Saout @ 2004-08-26 23:53 ` Hans Reiser 2004-08-27 0:18 ` Andrea Arcangeli 2004-08-28 10:49 ` Yury Umanets 2004-08-28 11:11 ` Yury Umanets 3 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 23:53 UTC (permalink / raw) To: Andrea Arcangeli Cc: Christophe Saout, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Andrea Arcangeli wrote: >On Thu, Aug 26, 2004 at 01:35:01AM -0700, Hans Reiser wrote: > > >>Reiser4 plugins are not for end users to download from amazon.com, they >>are for weekend hackers to send me a cool plugin for me to review, >>assign a plugin id to, and send to Linus in the next release. Sometimes >> >> > >then what's the difference in having the plugin fixed in stone into >reiserfs? That's my whole point. Get the patch from the weekend hacker, >check it, send the patch to Linus to add the new feature to reiser4, >just call it "feature" not plugin. > Think of it as being like VFS, where you can plugin new filesystems. Only in this case, you can plugin new kinds of files, and everything you need to implement those new kinds of files (items, nodes, keys, processing before flushing to disk, etc.) also gets implemented as a plugin. It is an Uber-VFS. > The only single reason we >use modules is to avoid wasting tons of ram by loading every possible >device driver on earth, > I am not concerned about ram in this design, I want nifty new kinds of files easy crafted over a weekend by sysadmins working in Canada and Guatemala. Software engineering cost is what matters to me ( I turned 40, so I think different now....;-) ) ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 23:53 ` Hans Reiser @ 2004-08-27 0:18 ` Andrea Arcangeli 0 siblings, 0 replies; 1392+ messages in thread From: Andrea Arcangeli @ 2004-08-27 0:18 UTC (permalink / raw) To: Hans Reiser Cc: Christophe Saout, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 04:53:23PM -0700, Hans Reiser wrote: > Think of it as being like VFS, where you can plugin new filesystems. > Only in this case, you can plugin new kinds of files, and everything you > need to implement those new kinds of files (items, nodes, keys, > processing before flushing to disk, etc.) also gets implemented as a > plugin. It is an Uber-VFS. I'm just confused by the naming, following your comparison with it being like VFS, I don't call reiser4 a "plugin" of the VFS. The vfs offers an interface for the fs to register into it. Like the scsi highlevel code also offers an interface for the lowlevel driver to register into it, but we don't call scsi lowlevel drivers "plugins". the ext3 directory code calling ext3_getblk would be also a plugin in your vocabulary, and the ext3_getblk call would be plugin-hook. > I am not concerned about ram in this design, I want nifty new kinds of > files easy crafted over a weekend by sysadmins working in Canada and > Guatemala. Software engineering cost is what matters to me ( I turned > 40, so I think different now....;-) ) Adding proper layering and abstractions is sure a good thing, if you can turn the street into a crowd of reiser4 kernel hackers because your API is so clean that any sysadmin can craft new kind of files over the weekend you're welcome to go ahead ;). (I just doubt the "plugin" terminology can make a difference by the time people start scratching their head into the code, the design and the clean abstraction of the code will make a difference instead) ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:28 ` Andrea Arcangeli 2004-08-26 11:44 ` Christophe Saout 2004-08-26 23:53 ` Hans Reiser @ 2004-08-28 10:49 ` Yury Umanets 2004-08-28 11:11 ` Yury Umanets 3 siblings, 0 replies; 1392+ messages in thread From: Yury Umanets @ 2004-08-28 10:49 UTC (permalink / raw) To: Andrea Arcangeli Cc: Hans Reiser, Christophe Saout, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Andrea Arcangeli wrote: >On Thu, Aug 26, 2004 at 01:35:01AM -0700, Hans Reiser wrote: > > >>Reiser4 plugins are not for end users to download from amazon.com, they >>are for weekend hackers to send me a cool plugin for me to review, >>assign a plugin id to, and send to Linus in the next release. Sometimes >> >> > >then what's the difference in having the plugin fixed in stone into >reiserfs? That's my whole point. Get the patch from the weekend hacker, >check it, send the patch to Linus to add the new feature to reiser4, >just call it "feature" not plugin. That's how it works normally for >everything. Many fs have many features, many of them optional. you >wouldn't need to build any hook infrastructure either that way. Hooks >would be needed if this wasn't open source me thinks. Or if you want >people to fetch the module from amazon without your review. > >Another reason I could see the modularization/hooking useful is if those >feature would take lots of kernel space, but this sure isn't the case >for reiserfs, infact having modules would waste _more_ ram due the half >wasted page allocation for the module text. The only single reason we >use modules is to avoid wasting tons of ram by loading every possible >device driver on earth, it's not that we use modules because they're >more flexible, if something they're more fragile. I never use modules in >my test kernel just to go fast (because I self compile them). The last >good thing of the modules is during development you don't need to reboot >the machine to test a kernel change in a driver, but you don't need that >with reiserfs since you can make the fs itself a module (just change the >name of the fs, AFIK you do that all the time already). > > There lots of good things about plugins as they are in reiser4. And probably the best one (as for me) is that, reiser4 core code (quite complex thing) does not need to be changed after adding new plugins (read new features). It is written once and provides some basic functionality (balancing, atoms, etc.), which works well with all kinds of plugins if they implement required functions from plugin interface. As for me, this looks like Linux's VFS. Sure, if I want to add some very new filesystem with features which were not foreseen, and VFS does not provide needed interface, it will need some changes. But in simple case, when I just want to develop once more disk filesystem, which say will be different from another disk filesystem in only that respect it manages its metadata in different manner and aims to achieve better performance this way. In this case I will not need to chnage VFS at all. The same about reiser4 core and new plugins. >- >To unsubscribe from this list: send the line "unsubscribe linux-kernel" in >the body of a message to majordomo@vger.kernel.org >More majordomo info at http://vger.kernel.org/majordomo-info.html >Please read the FAQ at http://www.tux.org/lkml/ > > > > -- umka ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:28 ` Andrea Arcangeli ` (2 preceding siblings ...) 2004-08-28 10:49 ` Yury Umanets @ 2004-08-28 11:11 ` Yury Umanets 3 siblings, 0 replies; 1392+ messages in thread From: Yury Umanets @ 2004-08-28 11:11 UTC (permalink / raw) To: Andrea Arcangeli Cc: Hans Reiser, Christophe Saout, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Andrea Arcangeli wrote: >On Thu, Aug 26, 2004 at 01:35:01AM -0700, Hans Reiser wrote: > > >>Reiser4 plugins are not for end users to download from amazon.com, they >>are for weekend hackers to send me a cool plugin for me to review, >>assign a plugin id to, and send to Linus in the next release. Sometimes >> >> > >then what's the difference in having the plugin fixed in stone into >reiserfs? That's my whole point. Get the patch from the weekend hacker, >check it, send the patch to Linus to add the new feature to reiser4, >just call it "feature" not plugin. That's how it works normally for >everything. Many fs have many features, many of them optional. you >wouldn't need to build any hook infrastructure either that way. Hooks >would be needed if this wasn't open source me thinks. Or if you want >people to fetch the module from amazon without your review. > >Another reason I could see the modularization/hooking useful is if those >feature would take lots of kernel space, but this sure isn't the case >for reiserfs, infact having modules would waste _more_ ram due the half >wasted page allocation for the module text. The only single reason we >use modules is to avoid wasting tons of ram by loading every possible >device driver on earth, it's not that we use modules because they're >more flexible, if something they're more fragile. I never use modules in >my test kernel just to go fast (because I self compile them). The last >good thing of the modules is during development you don't need to reboot >the machine to test a kernel change in a driver, but you don't need that >with reiserfs since you can make the fs itself a module (just change the >name of the fs, AFIK you do that all the time already). > > There lots of good things about plugins as they are in reiser4. And probably the best one (as for me) is that, reiser4 core code (quite complex thing) does not need to be changed after adding new plugins (read new features). It is written once and provides some basic functionality (balancing, atoms, etc.), which works well with all kinds of plugins if they implement required functions from plugin interface. As for me, this looks like Linux's VFS. Sure, if I want to add some very new filesystem with features which were not foreseen, and VFS does not provide needed interface, it will need some changes. But in simple case, when I just want to develop once more disk filesystem, which say will be different from another disk filesystem in only that respect it manages its metadata in different manner and aims to achieve better performance this way. In this case I will not need to chnage VFS at all. The same about reiser4 core and new plugins. >- >To unsubscribe from this list: send the line "unsubscribe linux-kernel" in >the body of a message to majordomo@vger.kernel.org >More majordomo info at http://vger.kernel.org/majordomo-info.html >Please read the FAQ at http://www.tux.org/lkml/ > > > > -- umka ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:42 ` viro 2004-08-25 21:00 ` Christophe Saout @ 2004-08-25 21:00 ` Linus Torvalds 2004-08-25 21:25 ` viro 2004-08-26 8:43 ` Hans Reiser 2 siblings, 1 reply; 1392+ messages in thread From: Linus Torvalds @ 2004-08-25 21:00 UTC (permalink / raw) To: viro Cc: Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, 25 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > > On Wed, Aug 25, 2004 at 01:22:55PM -0700, Linus Torvalds wrote: > > > > It's the UNIX way. > > Not if you allow link(2) on them. Heh. I don't think that's a very strong argument against being "unixy", considering how traditional unix _used_ to handle directories. mkdir/rmdir/rename only came later. Now, obviously they did come later for a good reason, but still.. The interesting part is that thanks to the dcache, we should be perfectly able to actually _see_ circular links etc, so some of the problems with linking directories should actually be quite solvable - something that is _not_ true for a traditional UNIX VFS layer. Of course, the dcache introduces some new problems of its own wrt directory aliasing, but I don't actually think that should be fundamental either. Treating them more as a "static mountpoint" from an FS angle and less as a traditional Unix hardlink should be doable, I'd have thought. (Also, it's entirely possible that the filesystem may not support some of the more esoteric linking/renaming operations. For example, in a traditional xattrs setup where the xattr is linked on-disk with the file it is associated with, you simply _can't_ link it somewhere else, or rename it to any other directory. That's not a VFS layer issue, obviously, but I thought I'd bring up the point that file-as-dir cases may have limitations that normal files don't have). > And not if you design and market your stuff as a general-purpose > backdoor into kernel. Now that's a separate argument, and not one I'm personally interested in arguing at least right now. I haven't actually looked at the reiser4 code, so I'm really _only_ arguing against special-case attributes. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 21:00 ` Linus Torvalds @ 2004-08-25 21:25 ` viro 2004-08-26 0:11 ` Jamie Lokier 0 siblings, 1 reply; 1392+ messages in thread From: viro @ 2004-08-25 21:25 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, Aug 25, 2004 at 02:00:01PM -0700, Linus Torvalds wrote: > Of course, the dcache introduces some new problems of its own wrt > directory aliasing, but I don't actually think that should be fundamental > either. Treating them more as a "static mountpoint" from an FS angle and > less as a traditional Unix hardlink should be doable, I'd have thought. Yeah, if we ditch the "mountpoints are busy and untouchable" stuff. Which I'd love to, but it's a hell of a visible (and admin-visible) change. FWIW, current deadlocks are unrelated to actual operation succeeding. Look: we have sys_link() making sure that parent of target is a directory (PATH_LOOKUP, in a "it has ->lookup()" sense), then locking target's parent, then checking that it has ->link() (everyone on reiser4 does) and then checking that source (old link to file) is *not* a directory (in S_ISDIR sense). Then we lock source. Note that currently it's OK - we get "all non-directories are always locked after all directories". With filesystem that provides hybrid objects with non-NULL ->link() it's not true and we are in deadlock country. Before we get anywhere near fs code. I'm not saying that this particular instance is hard to fix, but it wasn't even looked at. All it would take is checking the description of current locking scheme and looking through the proof of correctness (present in the tree). That's the first point where said proof breaks if we have hybrids. And it's what, about 4 screenfuls of text? I have no problems with discussing such stuff and no problems with having it merged if it actually works. But let's start with something better than "let's hope nothing breaks if we just add such objects and do nothing else, 'cause hybridi files/directories are good, mmmkay?" ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 21:25 ` viro @ 2004-08-26 0:11 ` Jamie Lokier 2004-08-26 0:30 ` viro 0 siblings, 1 reply; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 0:11 UTC (permalink / raw) To: viro Cc: Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List This message suggests a way to extend the VFS safe locking rules to include files-as-directories. viro@parcelfarce.linux.theplanet.co.uk wrote: > Note that currently it's OK - we get "all non-directories are always locked > after all directories". With filesystem that provides hybrid objects with > non-NULL ->link() it's not true and we are in deadlock country. Before > we get anywhere near fs code. Is this a problem if we treat entering a file-as-directory as crossing a mount point (i.e. like auto-mounting)? Simply doing a path walk would lock the file and then cross the mount point to a directory. A way to ensure that preserves the lock order is to require that the metadata is in a different filesystem to its file (i.e. not crossing a bind mount to the same filesystem). That has the side effect of preventing hard links between metadata files and non-metadata, which in my opinion is fine. Path walking will lock the file, and then lock the directory on a different filesystem. Lock order is still safe, provided a strict order is maintained between the two filesystems. The strict order is ensured by preventing bind mounts which create a path cycle containing a file->metadata edge. One way to ensure that is to prevent mounts on the metadata filesystems, but the rule doesn't have to be that strict. This condition only needs to be checked in the mount() syscall. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 0:11 ` Jamie Lokier @ 2004-08-26 0:30 ` viro 2004-08-26 1:00 ` Jamie Lokier 2004-08-26 1:13 ` Chris Wright 0 siblings, 2 replies; 1392+ messages in thread From: viro @ 2004-08-26 0:30 UTC (permalink / raw) To: Jamie Lokier Cc: Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 01:11:52AM +0100, Jamie Lokier wrote: > Is this a problem if we treat entering a file-as-directory as crossing > a mount point (i.e. like auto-mounting)? Yes - mountpoints can't be e.g. unlinked. Moreover, having directory mounted on non-directory is also an interesting situation. > Simply doing a path walk would lock the file and then cross the mount > point to a directory. *Ugh* What would happen if you open that directory or chdir there? If it's "underlying file stays locked" - we are in even more obvious deadlocks. > A way to ensure that preserves the lock order is to require that the > metadata is in a different filesystem to its file (i.e. not crossing a > bind mount to the same filesystem). > > That has the side effect of preventing hard links between metadata > files and non-metadata, which in my opinion is fine. We don't actually need a different fs - different vfsmount will do just fine. > The strict order is ensured by preventing bind mounts which create a > path cycle containing a file->metadata edge. One way to ensure that > is to prevent mounts on the metadata filesystems, but the rule doesn't > have to be that strict. This condition only needs to be checked in > the mount() syscall. You really don't want to lock mountpoint on path lookup, so I don't see how that would be relevant - it's a hell to clean up, for one thing (I've crossed ten mountpoints on the way, when do I unlock them and how do I prevent deadlocks from that?) Besides, different namespaces can have completely different mount trees, so tracking down all that stuff would be hell in its own right. The main issue I see with all schemes in that direction (and something like that could be made workable) is the semantics of unlink() on mountpoints. *Especially* with users being able to see attributes of files they do not own (e.g. reiser4 mode/uid/gid stuff). Ability to pin down any damn file on the system and make it impossible to replace is not something you want to give to any user. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 0:30 ` viro @ 2004-08-26 1:00 ` Jamie Lokier 2004-08-26 3:13 ` viro ` (2 more replies) 2004-08-26 1:13 ` Chris Wright 1 sibling, 3 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 1:00 UTC (permalink / raw) To: viro Cc: Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List viro@parcelfarce.linux.theplanet.co.uk wrote: > > Is this a problem if we treat entering a file-as-directory as crossing > > a mount point (i.e. like auto-mounting)? > > Yes - mountpoints can't be e.g. unlinked. Moreover, having directory > mounted on non-directory is also an interesting situation. Ok, so can we make it so mountpoints can be unlinked? :) The mount would continue to exist, but with no name, until its last user disappears. > > Simply doing a path walk would lock the file and then cross the mount > > point to a directory. > > *Ugh* > > What would happen if you open that directory or chdir there? If it's > "underlying file stays locked" - we are in even more obvious deadlocks. I think the underlying file does not stay locked, and once you've entered it as a directory, it can be unlinked. If you have the directory open or chdir into it, then it _may_ have the effect of keeping the file's storage allocated when you unlink it -- just like when a file is unlinked while opened. As that is not a user-visible property, it's a filesystem-specific implementation detail as to whether it keeps the file's data in existence while the metadata directories and/or files are open. > > A way to ensure that preserves the lock order is to require that the > > metadata is in a different filesystem to its file (i.e. not crossing a > > bind mount to the same filesystem). > > > > That has the side effect of preventing hard links between metadata > > files and non-metadata, which in my opinion is fine. > > We don't actually need a different fs - different vfsmount will do just fine. > > The strict order is ensured by preventing bind mounts which create a > > path cycle containing a file->metadata edge. One way to ensure that > > is to prevent mounts on the metadata filesystems, but the rule doesn't > > have to be that strict. This condition only needs to be checked in > > the mount() syscall. > > You really don't want to lock mountpoint on path lookup, so I don't see > how that would be relevant - it's a hell to clean up, for one thing > (I've crossed ten mountpoints on the way, when do I unlock them and > how do I prevent deadlocks from that?) Besides, different namespaces > can have completely different mount trees, so tracking down all that > stuff would be hell in its own right. I didn't mean locking a chain of mountpoints, I meant the temporary state where two dentries and/or inodes are locked, parent and child, during a path walk. However I'm not very familiar with that part of the VFS and I see that the current RCU dcache might not lock that much during a path walk. > The main issue I see with all schemes in that direction (and something > like that could be made workable) is the semantics of unlink() on > mountpoints. *Especially* with users being able to see attributes of > files they do not own (e.g. reiser4 mode/uid/gid stuff). Ability to > pin down any damn file on the system and make it impossible to replace > is not something you want to give to any user. I agree, users shouldn't be able to pin down a file. I think unlink() should succeed on a file while something is visiting inside its metadata directory. It's a filesystem quality-of-implementation feature whether that actually releases the file's data. It's a desirable feature because one user shouldn't be able to pin another user's quota'd data if they don't have permission to open the file, but if it's not implemented by a filesystem then it doesn't break anything fundamental. It's a semantics question whether unlinking a file makes the metadata (i.e. "uid", "mode", "content-type" etc.) disappear at the same time, or if the metadata stays around until the last visitor leaves it. A filesystem might be able to keep the metadata in existence even if it deletes the file's storage on unlink(), but it would be nice for the VFS to declare which semantic is preferred. One of the big potential uses for file-as-directory is to go inside archive files, ELF files, .iso files and so on in a convenient way. In those cases, if you open one of the virtually generated "archive content" files, then you might expect the data to continue to exist after the underlying file is unlinked. I think that's reasonable: being inside an archive is very similar to having it open. There is no quota pinning problem with this, because "archive content" files should inherit permission restrictions from the underlying file. If you can't read the file, then you can't read it's unpacked contents. (reiser4 doesn't offer that last feature, but any of the myriad userspace filesystem hooks could offer it if the VFS has approprate auto-mounting file-as-directory hooks). -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 1:00 ` Jamie Lokier @ 2004-08-26 3:13 ` viro 2004-08-26 8:49 ` Hans Reiser 2004-08-26 10:05 ` Chris Wedgwood 2 siblings, 0 replies; 1392+ messages in thread From: viro @ 2004-08-26 3:13 UTC (permalink / raw) To: Jamie Lokier Cc: Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 02:00:49AM +0100, Jamie Lokier wrote: > viro@parcelfarce.linux.theplanet.co.uk wrote: > > > Is this a problem if we treat entering a file-as-directory as crossing > > > a mount point (i.e. like auto-mounting)? > > > > Yes - mountpoints can't be e.g. unlinked. Moreover, having directory > > mounted on non-directory is also an interesting situation. > > Ok, so can we make it so mountpoints can be unlinked? :) User-visible change of behaviour and IIRC a SuS violation on top of that. > I think the underlying file does not stay locked, and once you've > entered it as a directory, it can be unlinked. So why lock it at all in that case? > I didn't mean locking a chain of mountpoints, I meant the temporary > state where two dentries and/or inodes are locked, parent and child, > during a path walk. However I'm not very familiar with that part of > the VFS and I see that the current RCU dcache might not lock that much > during a path walk. Never had been needed on crossing mountpoints, actually. > I agree, users shouldn't be able to pin down a file. > > I think unlink() should succeed on a file while something is visiting > inside its metadata directory. See above. Again, the fundamental problem with that is allowing unlink and friends on a mountpoint. I would love to do that, but it always generated -EBUSY on all Unices. Linux got a bit more users and userland code than Plan 9 - they can afford such changes, but... And yes, from the kernel POV it's trivial to do - witness the MNT_DETACH codepath in umount - it's much simpler than "normal" umount exactly because it doesn't try to emulate old "it's busy, can't umount" behaviour. With umount we could introduce "don't bother with that shit" flag. With unlink() we would have to make that default behaviour to be useful. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 1:00 ` Jamie Lokier 2004-08-26 3:13 ` viro @ 2004-08-26 8:49 ` Hans Reiser 2004-08-26 14:05 ` Adrian Bunk 2004-08-26 18:35 ` Joel Becker 2004-08-26 10:05 ` Chris Wedgwood 2 siblings, 2 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 8:49 UTC (permalink / raw) To: Jamie Lokier Cc: viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Jamie Lokier wrote: > > >One of the big potential uses for file-as-directory is to go inside >archive files, ELF files, .iso files and so on in a convenient way. > > Yes, this was part of the plan, tar file-directory plugins would be cute. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:49 ` Hans Reiser @ 2004-08-26 14:05 ` Adrian Bunk 2004-08-26 14:25 ` Christophe Saout 2004-08-26 15:02 ` Jamie Lokier 2004-08-26 18:35 ` Joel Becker 1 sibling, 2 replies; 1392+ messages in thread From: Adrian Bunk @ 2004-08-26 14:05 UTC (permalink / raw) To: Hans Reiser Cc: Jamie Lokier, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 01:49:15AM -0700, Hans Reiser wrote: > Jamie Lokier wrote: > > >One of the big potential uses for file-as-directory is to go inside > >archive files, ELF files, .iso files and so on in a convenient way. > > > > > Yes, this was part of the plan, tar file-directory plugins would be cute. Silly question: GNU Midnight Commander allows for ages to go into e.g. tar files, so I know the benefits of this. Additionally, in GNU Midnight Commander, this works no matter which file system I use (e.g. it works on iso9660), and it even works the same way on other OS's like e.g. Solaris and NetBSD. What is the technical reason why a tar plugin should be reiser4 specific, instead of a generic VFS or userspace solution that would allow the same also on other fs like e.g. iso9660? cu Adrian -- "Is there not promise of rain?" Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. "Only a promise," Lao Er said. Pearl S. Buck - Dragon Seed ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 14:05 ` Adrian Bunk @ 2004-08-26 14:25 ` Christophe Saout 2004-08-26 14:45 ` Matthew Wilcox 2004-08-26 15:04 ` Jamie Lokier 2004-08-26 15:02 ` Jamie Lokier 1 sibling, 2 replies; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 14:25 UTC (permalink / raw) To: Adrian Bunk Cc: Hans Reiser, Jamie Lokier, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 1316 bytes --] Am Donnerstag, den 26.08.2004, 16:05 +0200 schrieb Adrian Bunk: > > Yes, this was part of the plan, tar file-directory plugins would be cute. > > Silly question: > > GNU Midnight Commander allows for ages to go into e.g. tar files, so I > know the benefits of this. Additionally, in GNU Midnight Commander, this > works no matter which file system I use (e.g. it works on iso9660), and > it even works the same way on other OS's like e.g. Solaris and NetBSD. > > What is the technical reason why a tar plugin should be reiser4 > specific, instead of a generic VFS or userspace solution that would > allow the same also on other fs like e.g. iso9660? I don't think that this should be done inside the kernel. tar is a perfectly valid userspace application. What reiser4 can do, but the VFS can't is to insert or remove data in the middle of a file. Adding this above the page cache would probably be almost impossible (truncate seems already complicated enough). But below page-cache and thus invisible from the VFS or the applications it is extremely useful for the compression plugin. When changing some data in the middle of the file the compressed data might grow in size. reiser4 can handle this inside the storage layer and doesn't mess with the rest of the filesystem. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 14:25 ` Christophe Saout @ 2004-08-26 14:45 ` Matthew Wilcox 2004-08-26 15:04 ` Jamie Lokier 1 sibling, 0 replies; 1392+ messages in thread From: Matthew Wilcox @ 2004-08-26 14:45 UTC (permalink / raw) To: Christophe Saout Cc: Adrian Bunk, Hans Reiser, Jamie Lokier, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 04:25:13PM +0200, Christophe Saout wrote: > What reiser4 can do, but the VFS can't is to insert or remove data in > the middle of a file. Adding this above the page cache would probably be > almost impossible (truncate seems already complicated enough). > > But below page-cache and thus invisible from the VFS or the applications > it is extremely useful for the compression plugin. When changing some > data in the middle of the file the compressed data might grow in size. > reiser4 can handle this inside the storage layer and doesn't mess with > the rest of the filesystem. Never say never ... regular readers of -fsdevel already know what I'm going to say so can skip this post. When compressing data on an ext2-like filesystem, you can use the property of "holes" to deallocate disc blocks in the middle of a file. It works like this: Split the file into 64k chunks Compress each chunk individually, then write them to disc on 64k boundaries, leaving gaps. Now you can quickly seek to the appropriate 64k block for the data you're looking for and decompress just that 64k block. This is how RISCiX managed to fit an 80MB distro onto a 50MB disc. Someone has written an equivalent filesystem for Linux called SquashFS. -- "Next the statesmen will invent cheap lies, putting the blame upon the nation that is attacked, and every man will be glad of those conscience-soothing falsities, and will diligently study them, and refuse to examine any refutations of them; and thus he will by and by convince himself that the war is just, and will thank God for the better sleep he enjoys after this process of grotesque self-deception." -- Mark Twain ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 14:25 ` Christophe Saout 2004-08-26 14:45 ` Matthew Wilcox @ 2004-08-26 15:04 ` Jamie Lokier 2004-08-26 15:12 ` Christophe Saout ` (2 more replies) 1 sibling, 3 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 15:04 UTC (permalink / raw) To: Christophe Saout Cc: Adrian Bunk, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Christophe Saout wrote: > What reiser4 can do, but the VFS can't is to insert or remove data in > the middle of a file. Adding this above the page cache would probably be > almost impossible (truncate seems already complicated enough). That would be one of those "special features" that a VFS-plus-userspace implementation of archive views could take advantage of on reiser4, while using a slower method (sometimes much slower) on all other filesystems. By the way, can reiser4 share parts of files between different files? -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 15:04 ` Jamie Lokier @ 2004-08-26 15:12 ` Christophe Saout 2004-08-26 16:00 ` Jamie Lokier 2004-08-26 15:41 ` Nikita Danilov 2004-08-29 12:34 ` Alex Zarochentsev 2 siblings, 1 reply; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 15:12 UTC (permalink / raw) To: Jamie Lokier Cc: Adrian Bunk, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 1479 bytes --] Am Donnerstag, den 26.08.2004, 16:04 +0100 schrieb Jamie Lokier: > Christophe Saout wrote: > > What reiser4 can do, but the VFS can't is to insert or remove data in > > the middle of a file. Adding this above the page cache would probably be > > almost impossible (truncate seems already complicated enough). > > That would be one of those "special features" that a > VFS-plus-userspace implementation of archive views could take > advantage of on reiser4, while using a slower method (sometimes much > slower) on all other filesystems. I'm just thinking about something. While you can't cut bytes out of unix iles a lot of filesystems can do this (holes). Most of them only on a block boundary, reiser4 on a byte boundary. If the filesystems could export this functionality using an enhanced API we could implement a compression plugin and other things on the VFS level that works with every filesystem supporting the required mechanisms, not only reiser4. And those features would take advantage of reiser4's storage mechanisms. I think Hans made the plugins reiser4-only because only reiser4 has a similar API at that time (and obviously because he didn't even think about doing it otherwise). > By the way, can reiser4 share parts of files between different files? At the moment a file is exactly one object. But someone could write another file plugin that spans a file across multiple objects, then yes, multiple files could share parts. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 15:12 ` Christophe Saout @ 2004-08-26 16:00 ` Jamie Lokier 0 siblings, 0 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 16:00 UTC (permalink / raw) To: Christophe Saout Cc: Adrian Bunk, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Christophe Saout wrote: > I'm just thinking about something. While you can't cut bytes out of unix > iles a lot of filesystems can do this (holes). Most of them only on a > block boundary, reiser4 on a byte boundary. If the filesystems could > export this functionality using an enhanced API we could implement a > compression plugin and other things on the VFS level that works with > every filesystem supporting the required mechanisms, not only reiser4. > And those features would take advantage of reiser4's storage mechanisms. > I think Hans made the plugins reiser4-only because only reiser4 has a > similar API at that time (and obviously because he didn't even think > about doing it otherwise). Yes. In some cases the compression or re-archiving algorithm would choose different parameters, depending on the alignment capabilities of the underlying filesystem. In some cases, that alignment is "offset 0 is the only one you can have", which is a severe special case where truncate and rewrite is required for changes. > > By the way, can reiser4 share parts of files between different files? > > At the moment a file is exactly one object. But someone could write > another file plugin that spans a file across multiple objects, then yes, > multiple files could share parts. Nice. That's useful for files which appear as themselves and also as part of a container. And also for revision managed filesystems :) -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 15:04 ` Jamie Lokier 2004-08-26 15:12 ` Christophe Saout @ 2004-08-26 15:41 ` Nikita Danilov 2004-08-29 12:34 ` Alex Zarochentsev 2 siblings, 0 replies; 1392+ messages in thread From: Nikita Danilov @ 2004-08-26 15:41 UTC (permalink / raw) To: Jamie Lokier Cc: Christophe Saout, Adrian Bunk, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Jamie Lokier writes: > Christophe Saout wrote: > > What reiser4 can do, but the VFS can't is to insert or remove data in > > the middle of a file. Adding this above the page cache would probably be > > almost impossible (truncate seems already complicated enough). > > That would be one of those "special features" that a > VFS-plus-userspace implementation of archive views could take > advantage of on reiser4, while using a slower method (sometimes much > slower) on all other filesystems. > > By the way, can reiser4 share parts of files between different files? It can, but fsck is supposed to cope with this. :) > > -- Jamie Nikita. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 15:04 ` Jamie Lokier 2004-08-26 15:12 ` Christophe Saout 2004-08-26 15:41 ` Nikita Danilov @ 2004-08-29 12:34 ` Alex Zarochentsev 2004-08-29 15:52 ` Shaya Potter 2004-09-01 22:05 ` Jamie Lokier 2 siblings, 2 replies; 1392+ messages in thread From: Alex Zarochentsev @ 2004-08-29 12:34 UTC (permalink / raw) To: Jamie Lokier Cc: Christophe Saout, Adrian Bunk, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 04:04:34PM +0100, Jamie Lokier wrote: > Christophe Saout wrote: > > What reiser4 can do, but the VFS can't is to insert or remove data in > > the middle of a file. Adding this above the page cache would probably be > > almost impossible (truncate seems already complicated enough). > > That would be one of those "special features" that a > VFS-plus-userspace implementation of archive views could take > advantage of on reiser4, while using a slower method (sometimes much > slower) on all other filesystems. > > By the way, can reiser4 share parts of files between different files? no, those file plugins are not written yet :) Do you mean COW files or some another thing? For COW files, the reiser4 support is not enough, we need to teach cp(1) or another utility to inform the fs layer that copying can be done by creation of COW files. > -- Jamie -- Alex. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 12:34 ` Alex Zarochentsev @ 2004-08-29 15:52 ` Shaya Potter 2004-09-01 22:05 ` Jamie Lokier 1 sibling, 0 replies; 1392+ messages in thread From: Shaya Potter @ 2004-08-29 15:52 UTC (permalink / raw) To: Alex Zarochentsev Cc: Jamie Lokier, Christophe Saout, Adrian Bunk, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Sun, 2004-08-29 at 16:34 +0400, Alex Zarochentsev wrote: > Do you mean COW files or some another thing? For COW files, the reiser4 > support is not enough, we need to teach cp(1) or another utility to inform the > fs layer that copying can be done by creation of COW files. it's easy enough to add that via an fs ioctl(), I did the equivalent last semester for some research I'm doing while using ext3cow (which unfortunately was put on hold for the summer), it's just a take off of sys_link(), a sys_cowpy() of sorts. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 12:34 ` Alex Zarochentsev 2004-08-29 15:52 ` Shaya Potter @ 2004-09-01 22:05 ` Jamie Lokier 1 sibling, 0 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-09-01 22:05 UTC (permalink / raw) To: Alex Zarochentsev Cc: Christophe Saout, Adrian Bunk, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Alex Zarochentsev wrote: > > > What reiser4 can do, but the VFS can't is to insert or remove data in > > > the middle of a file. Adding this above the page cache would probably be > > > almost impossible (truncate seems already complicated enough). > > > > That would be one of those "special features" that a > > VFS-plus-userspace implementation of archive views could take > > advantage of on reiser4, while using a slower method (sometimes much > > slower) on all other filesystems. > > > > By the way, can reiser4 share parts of files between different files? > > no, those file plugins are not written yet :) > > Do you mean COW files or some another thing? COW files would be nice as well, but I meant another thing: for files which are parts of an uncompressed archive, it makes sense, if the fs offers it efficiently, to share the storage. Doing it properly requires that memory is also shared, for example when COW files are used to isolate trees in virtual server jails, you still want the mapped executables to share memory. You can't do that just by writing a plugin. > For COW files, the reiser4 support is not enough, we need to teach > cp(1) or another utility to inform the fs layer that copying can be > done by creation of COW files. Actually we don't. The VM and VFS layers could deduce it with no changes to cp(1), by tracking which pages are identical due to read(3) and write(3). However, changing cp(1) is easier. reiser4 support is obviously the first prerequisite... -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 14:05 ` Adrian Bunk 2004-08-26 14:25 ` Christophe Saout @ 2004-08-26 15:02 ` Jamie Lokier 2004-08-28 23:14 ` Horst von Brand 2004-09-10 2:13 ` silent semantic changes with reiser4 Timothy Miller 1 sibling, 2 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 15:02 UTC (permalink / raw) To: Adrian Bunk Cc: Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Adrian Bunk wrote: > What is the technical reason why a tar plugin should be reiser4 > specific, instead of a generic VFS or userspace solution that would > allow the same also on other fs like e.g. iso9660? It should be a generic VFS plugin, not reiser4 or userspace. The VFS plugin should call out to userspace for most actions (except handling cached data), and it should take advantage of special reiser4 features for storage and performance optimisations. But it should still work over a standard filesystem, when those special features aren't available. I guess FUSE and many earlier projects are heading in this direction. A generic userspace solution doesn't let you "cd" into a tar file from all programs like you can inside Midnight Commander. Gnome and KDE take the approach that every userspace system call should be intercepted and filtered, to create the illusion of virtual data. As a result, different programs see different virtual data: you can't just cut and paste a path from Gnome or KDE into any other program. It's not just a "social problem of libraries" thing: sometimes I have programs which don't link to libc. Sometimes I have programs which mustn't link with anything that calls malloc(). It'd be silly for them to have a different view of the filesystem just because they can't link with some userspace library. The Gnome/KDE/Midnight Commander pure userspace solution is silly: if _every_ program in the system should get the same view, it makes much more sense for the kernel to filter the system calls and redirect the virtual accesses to a userspace daemon, while keeping the real accesses at full speed. Furthermore is makes much more sense for the kernel's page cache to hold those uncompressed pages, than for every userspace application to try and cooperatively manage a cache of uncompressed fragments in the most inefficient way. There's another problem with the Midnight Commander approach. If I "cd" into a tar file, and then a program writes to the tar file, I don't always see the changes straight away. The two views aren't coherent. This isn't an easy problem to solve, but it should be solved. When a simple "cd" into .tar.gz or .iso is implemented properly, it will have _no_ performance penalty after you have first looked in the file, so long as it remains in the on-disk cache. And, the filesystem will manage that cache intelligently. Imagine: for looking at source files and such, you probably won't bother untarring in future, and you won't bother keeping untarred source trees in your home directory for easy access to things you look at often. Why waste the space? You could install whole applications as a .tar and run them from within it, with no performance penalty. Similarly, the filesystem will be able to archive directories automatically that haven't been touched in a long time, with no visible change except increased storage space. "grep" will be a bit slower, but you'll have a useful search tool by then (using coherent indexes) which will be more useful than grep, and much faster. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 15:02 ` Jamie Lokier @ 2004-08-28 23:14 ` Horst von Brand 2004-09-01 20:08 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Jamie Lokier 2004-09-10 2:13 ` silent semantic changes with reiser4 Timothy Miller 1 sibling, 1 reply; 1392+ messages in thread From: Horst von Brand @ 2004-08-28 23:14 UTC (permalink / raw) To: Jamie Lokier Cc: Adrian Bunk, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Jamie Lokier <jamie@shareable.org> said: [...] > When a simple "cd" into .tar.gz or .iso is implemented properly, it > will have _no_ performance penalty after you have first looked in the > file, so long as it remains in the on-disk cache. And, the filesystem > will manage that cache intelligently. Nonsense. The .iso or .tar or whatever would have to be kept un-isoed or un-tarred in memory (or on disk cache) for this to be true, and that takes quite a long time. Each time you want to peek anew at linux/Makefile, the whole tarfile will have to be read and stored somewhere, and that is just too slow for my taste. The .tar format is optimized for compact storage, the on-disk format of a filesystem is optimized for fast access and modifiability. Now go ahead and enlarge a file on your .iso/.tar a bit...it will take ages to rebuild the whole thing. There is a _reason_ why there are filesystems and archives, and they use different formats. If it weren't so, everybody and Aunt Tillie would just carry .ext3's around, and would wonder what the heck all this fuss is about. > Imagine: for looking at source files and such, you probably won't > bother untarring in future, and you won't bother keeping untarred > source trees in your home directory for easy access to things you look > at often. Why waste the space? You could install whole applications > as a .tar and run them from within it, with no performance penalty. Perpetuum mobile is a nice idea too. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-08-28 23:14 ` Horst von Brand @ 2004-09-01 20:08 ` Jamie Lokier 2004-09-01 20:50 ` Linus Torvalds 2004-09-02 14:06 ` Horst von Brand 0 siblings, 2 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-09-01 20:08 UTC (permalink / raw) To: Horst von Brand Cc: Adrian Bunk, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List I'm going to explain why filesystem support for .tar.gz or other "document container"-like formats is useful. This does _not_ mean tar in the kernel (I know someone who can't read will think that if I don't say it isn't); it does mean hooks in the kernel for maintaining coherency between different views and filesystem support for cacheing. The vision I'm going for here is: 1. OpenOffice and similar programs store compound documents in standard file formats, such as .tar.gz, compressed XML and such. Fs support can reduce CPU time handling these sorts of files, as I explain below, while still working with the standard file formats. With appropriate userspace support, programs can be written which have access to the capabilities on all platforms, but reduced CPU time occurs only on platforms with the fs support. 2. Real-time indexing and local search engine tools. This isn't just things like local Google; it's also your MP3 player scanning for titles & artists, your email program scanning for subject lines to display the summary fast, your blog server caching built pages, your programming environment scanning for tags, and your file transfer program scanning for shared deltas to reduce bandwidth. I won't explain how these work as it would make this mail too long. It should be clear to anyone who thinks about it why the coherency mechanism is essential for real-time, and a consistent interface to container internals helps with performance. Horst von Brand wrote: > Jamie Lokier <jamie@shareable.org> said: > > When a simple "cd" into .tar.gz or .iso is implemented properly, it > > will have _no_ performance penalty after you have first looked in the > > file, so long as it remains in the on-disk cache. And, the filesystem > > will manage that cache intelligently. > > Nonsense. The .iso or .tar or whatever would have to be kept un-isoed or > un-tarred in memory (or on disk cache) for this to be true, and that takes > quite a long time. Each time you want to peek anew at linux/Makefile, the > whole tarfile will have to be read and stored somewhere, Wrong. "So long as it remains in the on-disk cache" means each time you peek at linux/Makefile, the tarfile is _not_ read. For a tarfile it's slow the first time, and when it falls out of the on-disk cache, otherwise, for component files you are using regularly (even over a long time) it's as fast as reading a plain file. You obviously know this, as you mentioned on-disk cache in the reply, so I infer from the rest of your mail that what you're trying to say is more about modifications than reading archives. That it would be silly to keep working data in .tar.gz files, because working inside them regularly would be slow. Which means you must be assuming, incorrectly, that these .tar.gz files are really kept up to date on disk with every component file modification. Which is silly. .tar.gz files are suitable for *transport* and *archival*, not regular random access; it's almost rude of you to suggest I didn't know that. The proposal is that .tar.gz files (and others) are analysed on demand and content cached on disk as it is read. Then subsequent reads will be as fast as if you had unpacked the archives by hand, manually using the tar command. This is obviously exactly the same as you do now, with a small bit of added convenience. The other part of the proposal is that when you modify a component, the modifications are stored on disk in the same way as ordinary files, using the regular high performance random access disk structures. Nothing is done to recreate the archive at this point; I think this is where you misunderstood and thus flamed. _If_ after modifying components, you then read the .tar.gz as a file, then (and only then) is it recreated, taking in the worst case the same time as running the tar command. The _only_ times when that occurs are precisely those times when you would have run the tar command manually: because you only read the .tar.gz file when you need the flat file for some purpose, such as attaching it to an email, transferring by FTP or HTTP, or reading it into a program that needs it in that format. If there is anything about that strategy that doesn't make sense, then I suggest I have failed to explain it properly, and you're welcome to demand a clearer explanation. > the .tar format is optimized for compact storage, the on-disk format > of a filesystem is optimized for fast access and modifiability. Actually no, .tar is not compact at all. It's also not optimised for random read access, but after an index is build it is very fast for that. .tar.gz is compact. Although that is not especially fast for random read access, you can build a "compression dictionary index" which optimises random read access even in a .tar.gz, without ever unpacking the whole thing. Some formats like .iso, .zip and .jar are optimised for compact storage _and_ fast random access. They come with an index, and don't need one to be built and cached. These are not filesystem-like formats, obviously, but they are the formats you need to pack and unpack when exchanging data with other people. That's the _only_ reason they're on your disk (virtually; they may not really exist some of the time). > Now go ahead and enlarge a file on your .iso/.tar a bit...it > will take ages to rebuild the whole thing. There is a _reason_ why there > are filesystems and archives, and they use different formats. If it weren't > so, everybody and Aunt Tillie would just carry .ext3's around, and would > wonder what the heck all this fuss is about. If you enlarge a file in your .iso/.tar subdirectory a bit... nothing happens. Why would a smart programmer do anything so silly as rebuild the archive at that point? _If_ you subsequently read the .iso/.tar _file_, then and only then does it rebuild. Once, after lots of component writes. The only time you would ever do that is if you are specifically reading the archive file, which means you actually want to use the repacked file at that point, for example to FTP it somewhere or use as an email attachment. If the filesystem does not do that on demand, then you would have run the tar command manually at that point, precisely because that's a point where you need the repacked archive. So in case where the filesystem repacks the archive, it takes exactly the same time as you would have taken anyway; it's just automatic instead of manual. (As a handy side effect, the automatic method offers lower latency for transmissions). Now, why would we bother with all this? I see three reasons: convenience, time efficiency, and storage efficiency. Convenience is simply that it is handy to be able to look inside archive files, in those situations where we _currently_ use them, without having to manually untar when we need to, and without having to remember to clean up old directories when we discover we aren't using those often any more. This is _not_ an argument for using .tar.gz files in place of ordinary directories! Convenience applies to doing the things you do now. Time efficiency has two angles. A simple one is that accessing .tar.gz contents through any kind of filesystem interface, even pure userspace, can be faster than unpacking whole files, simply because there are ways to decode parts without unpacking it all. However, the main time efficiency that I see comes from the increasing number of applications where the "Open" and "Save" operations store data in *.gz files (e.g. OpenOffice compressed XML documents), or *.tar.gz files (some compound document formats), or other things like that. (If you think about it, quite a lot of things are like that). With these, every "Open" currently has to decompress and maybe unpack an archive format. Every "Save" currently has to pack and then compress. This is done so the user sees a single flat file containing a complex document, but it is a waste of CPU time until the user actually transports the flat file. The lazy proposal, as described earlier in this mail, _removes_ these decompression, unpacking, packing and compression CPU-intensive steps when they are unnecessary. The experience of a single file containing a complex document is maintained, but the CPU time is reduced in many typical operations. "Open" gets faster after you first look at a file, "Save" gets a lot faster for large documents, and the equivalent of grep (or later, real-time local search engine) gets a lot faster too. There is no operation where CPU time is overall increased. This is what I've meant throughout this thread, when I say containers: document files of the kind used to hold text, figures, etc. that are typically transported as a unit, and edited as a unit, but nonetheless at the moment they're stored in somewhat CPU intensive formats, for compactness. That's fine for a 1 page letter, but think of the OpenOffice 500-page book containing a large number of diagrams. However, even simple programs that read & write compressed XML benefit. The proposal allows that sort of thing to be handled more time efficiently that it is today, and in a way that is very practical to use. (It's unthinkable that OpenOffice and similar programs would have a lot of code which stored data in a special way just for Linux, just for these performance benefits which are otherwise user-invisible, but it's thinkable that a general purpose userspace library which is portable to all platforms could be written, which takes advantage of the facility when it's available and does the equivalent of today's "compress on save" when the filesystem facility isn't available). Finally, storage efficiency comes from simply allowing the filesystem and supporting tools to decide when it is best to store data in unpacked, packed & compressed, both at the same time, or another archival form. The filesystem has comparativaly good knowledge of which data to archive and when, but it can only maintain the illusion if there's a mechanism to make archived forms and unpacked forms coherent. Now, I'm sure there is a way to implement this on top of a neat and simple kernel feature involving weird bind mounts, leases, dnotifies and FUSE. But those kernel offering are quite a mess at the moment and don't fit together in a way which can usefully create this effect. Auto-mounting uservfs directories over file-as-directory, using moveable bind mounts _nearly_ offers the kernel primitives we need to build this in userspace and get the all the efficiencies. But not quite. (We could obviously do it all in userspace by putting _everything_ in a userspace filesystem, but that would be silly as it would throw away all of the performance of having a threaded filesystem in the kernel. It might do as a proof of concept though). -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-01 20:08 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Jamie Lokier @ 2004-09-01 20:50 ` Linus Torvalds [not found] ` <20040902002431.GN31934@mail.shareable.org> ` (3 more replies) 2004-09-02 14:06 ` Horst von Brand 1 sibling, 4 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-09-01 20:50 UTC (permalink / raw) To: Jamie Lokier Cc: Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, 1 Sep 2004, Jamie Lokier wrote: > > I'm going to explain why filesystem support for .tar.gz or other > "document container"-like formats is useful. This does _not_ mean tar > in the kernel (I know someone who can't read will think that if I > don't say it isn't); it does mean hooks in the kernel for maintaining > coherency between different views and filesystem support for cacheing. I think that's a valid thing, but there are some fundamental problems with it if you expect it to work on a normal filesystem (ie something that isn't fundamentally designed as a database). For example, _what_ kind of coherency do you think is acceptable? Quite frankly, using standard UNIX interfaces, absolute coherency just isn't an option, because it's just not possible to try to atomically update a view at the same time somebody else is writing to the "main file". "mmap()" is the most obvious example of this, but even the _basic_ notion of multiple "read" calls is not atomic without locking that is _way_ too expensive. A "read()" on a file is not atomic even on the _plain_ file: if somebody does a concurrent "write()", the reader may see a partial update. This becomes a million times more confusing if the reader is seeing a structured view of the file the writer is modifying. Also, it's likely impossible to write() to the view-file, again unless you expect all the underlying filesystems to be something really special. So from a _practical_ standpoint, I suspect that the best you can really do pretty cheaply (and which gets you 90% of what you probably want) is: - open-close consistency: the "validity" of the cache is checked at _open_ time, and no guarantees are given about the cache being updated afterwards. - read-only access to the cache (ie you can only read the view, not write to it). and quite frankly, I think you can do the above pretty much totally in user space with a small library and a daemon (in fact, ignoring security issues you probably don't even need the daemon). And if you can prototype it like that, and people actually find it useful, I suspect kernel support for better performance might be possible. Suggested interface: int open_cached_view(int base_fd, char *type, char *subname); where "type" would be the type of the view (ie "tar" for a tar-file view, "idtag" for a mp3 ID tag, or NULL for "autodetect default view") and "subname" would be the cache entry name (ie the tar-file filename, or the tag type to open). I bet you could write a small library to test this out for a few types. See if it's useful to you. And only if it's useful (and would make a huge performance difference) would it be worth putting in the kernel. Implementation of the _user_space_ library would be something like this: #define MAXNAME 1024 int open_cached_view(int base_fd, char *type, char *subname) { struct stat st; char filename[PATH_MAX]; char name[MAXNAME]; int len, cachefd; if (fstat(base_fd, &st) < 0) return -1; sprintf(name, "/proc/self/fd/%d", base_fd); len = readlink(name, filename, sizeof(filename)-1); if (len < 0) return -1; filename[len] = 0; /* FIXME! Replace '/' with '#' in "type" and "subname" */ len = snprintf(name, sizeof(name), "%04llx/%04llx/%s/%s/%s", (unsigned long long) st.st_dev, (unsigned long long) st.st_ino, type ? : "default", subname, filename); errno = ENAMETOOLONG; if (len >= sizeof(name)) return -1; cachefd = open(name, O_RDONLY); if (cachefd >= 0) { /* Check mtime here - maybe we could have kernel support */ return cachefd; } if (errno != ENOENT) return -1; /* .. try to generate cache file here .. */ see what I'm aiming at? You start out with a generic "attribute cache" library that does some hacky things (like depending on "mtime" for coherency) and then if that works out you can see if it's useful. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
[parent not found: <20040902002431.GN31934@mail.shareable.org>]
[parent not found: <413694E6.7010606@slaphack.com>]
[parent not found: <Pine.LNX.4.58.0409012037300.2295@ppc970.osdl.org>]
* Re: The argument for fs assistance in handling archives [not found] ` <Pine.LNX.4.58.0409012037300.2295@ppc970.osdl.org> @ 2004-09-02 5:53 ` Hans Reiser [not found] ` <4136A14E.9010303@slaphack.com> 1 sibling, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-09-02 5:53 UTC (permalink / raw) To: Linus Torvalds Cc: David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Linus Torvalds wrote: >On Wed, 1 Sep 2004, David Masover wrote: > > >>And that is the right solution. Not the only one, but the right one. >>Caching isn't the only thing sorely in need of transaction support right >>now. Actually, I find it hard to think of anything on Linux which >>shouldn't have transactions -- why should /etc/fstab or >>/home/david/homework be more fragile than /var/lib/mysql? >> >> > >It's easy to talk big. > >It's damn hard to _implement_ a complex system, and make it stable and >bug-free, and support legacy applications. > >There is a reason why we do only what _must_ be done in kernel space. > > Linus > > > > Filesystems need transactions. Given that they need it, it was very tempting to build an infrastructure offering some of the functionality to user space, and so we did in reiser4. What we have is very rudimentary, but I think I am prepared to say that important aspects of high performance transactions are better done in the kernel than in user space. It will be interesting to see what aspects of transaction functionality belong in user space as our transactions API comes into existence. Clearly there are some things about isolation that are better implemented by user space because user space can make optimizations the kernel doesn't know enough to make. Still, it will be interesting to see how low we can make the overhead of isolation. Right now reiser4 supports atomicity but not isolation, which is to say we offer to guarantee that a set of operations will either all survive a crash or none of them will, but we don't let you roll them back without a crash. (Originally I called this a transcrash rather than atomicity, but nobody liked the term.) Being modest in our ambition was important to performance, but maybe in time we will find a high performance way to be more ambitious. ^ permalink raw reply [flat|nested] 1392+ messages in thread
[parent not found: <4136A14E.9010303@slaphack.com>]
* Re: The argument for fs assistance in handling archives [not found] ` <4136A14E.9010303@slaphack.com> @ 2004-09-02 6:04 ` Linus Torvalds 2004-09-02 7:15 ` Hans Reiser 0 siblings, 1 reply; 1392+ messages in thread From: Linus Torvalds @ 2004-09-02 6:04 UTC (permalink / raw) To: David Masover Cc: Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, 1 Sep 2004, David Masover wrote: > > I think transactions should be done in kernel space. It's the only way > to "enforce" them sanely. Well, they clearly can't, since it depends on the filesystem. Also, transactions have a tendency to be expensive, and people have a tendency to ask for more than you give them. Doing transactions on one file is only the beginning - you'll find people who want transactions across file boundaries etc. That's a basic fact when it comes to pretty much anything: you can always find people who want something better. You can't be all things to all people, so what you actually _want_ to do is to expose the _minimal_ set of capabilities that people can build on. So I definitely don't want this discussion to degenerate (any more than it lng since has ;) into what people _wish_ for. No, it should be a "what is the _least_ we can absolutely do" that solves real problems. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 6:04 ` Linus Torvalds @ 2004-09-02 7:15 ` Hans Reiser 2004-09-02 7:36 ` Linus Torvalds 2004-09-02 7:55 ` Robin Rosenberg 0 siblings, 2 replies; 1392+ messages in thread From: Hans Reiser @ 2004-09-02 7:15 UTC (permalink / raw) To: Linus Torvalds Cc: David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Linus Torvalds wrote: > >. Doing transactions on one file is >only the beginning - you'll find people who want transactions across file >boundaries etc. > > Yup, that's what reiser4 is aiming at and where things get exciting for version control systems and the like. We are just missing the api code for it at the moment, all the infrastructure is there, and the api is in progress in sys_reiser4. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 7:15 ` Hans Reiser @ 2004-09-02 7:36 ` Linus Torvalds 2004-09-02 8:58 ` Hans Reiser ` (3 more replies) 2004-09-02 7:55 ` Robin Rosenberg 1 sibling, 4 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-09-02 7:36 UTC (permalink / raw) To: Hans Reiser Cc: David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2 Sep 2004, Hans Reiser wrote: > Linus Torvalds wrote: > > > >. Doing transactions on one file is > >only the beginning - you'll find people who want transactions across file > >boundaries etc. > > Yup, that's what reiser4 is aiming at and where things get exciting for > version control systems and the like. We are just missing the api code > for it at the moment, all the infrastructure is there, and the api is in > progress in sys_reiser4. But _my_ point is, no user program is going to take _advantage_ of anything that only one filesystem on one system offers. So there's no point. It's much saner (from _any_ app standpoint) to roll their own database in user space - that way it just works. In other words, nobody is really ever going to take advantage of this. This is _not_ how technical advanncement happens. The way you get people to take advantage of something is to have a nice gradual ramp-up, not a sudden new feature that they can't realistically use. So before you do transactions in the filesystem, you have to be able to do them in user space - and then make the filesystem version be _compatible_ with that user space library. That way you can get people to use the library ("hey, it works anywhere") and then you can get them to use your filesystem ("hey, if you use our super-duper filesystem, you can do what you are already doing five times faster"). See? You're starting at the wrong end. Give me a portable interface to use _first_. Then do transactions in the filesystem. (Now, as an academic exercise and as a learnign experience and prototyping work it migt make sense to do the filesystem first. But then you have to really _consider_ it a prototype, and realize that to make it useful you'd still have to do the user thing before you can "sell" people on the idea). Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 7:36 ` Linus Torvalds @ 2004-09-02 8:58 ` Hans Reiser 2004-09-02 9:12 ` Oliver Hunt ` (6 more replies) 2004-09-02 9:04 ` Hans Reiser ` (2 subsequent siblings) 3 siblings, 7 replies; 1392+ messages in thread From: Hans Reiser @ 2004-09-02 8:58 UTC (permalink / raw) To: Linus Torvalds Cc: David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Linus Torvalds wrote: > But _my_ point is, no user program is going to take _advantage_ of > >anything that only one filesystem on one system offers. > > Apple does not have this problem.... and yes, the apps will take advantage of it, which is different from depending on it. If you use the wrong fs you will lose some of the features of the app. For 30 years nothing much has happened in Unix filesystem semantics because of sheer cowardice (excepting Clearcase, which priced itself into a niche market). It is 25 years past time for someone to change things. That someone will have first mover advantage, and the more little semantic features possessed the more lure there will be to use it which will increase market share which will lure more apps into depending on it and in a few years the other filesystems will (deservedly) have only a small market share because the apps won't all work on them. Besides, there are enhancements which are simply compelling. You can write a dramatically better performance version control system with a much simpler design if the FS is atomic. Our transaction manager first draft was written by a version control guy, and he would probably be happy to tell you how lack of atomicity other than rename makes version control software design hideous. We have the performance lead. By next year we will be stable enough for mission critical servers, and then we start the serious semantic enhancements. If you don't embrace progress, then you doom Linux to following behind, because the guys at Apple are pretty aggressive now that Jobs is back, and they WILL change the semantics, and they will do so in compelling ways, and Linux will be reduced to aping them when it should be leading them. Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 8:58 ` Hans Reiser @ 2004-09-02 9:12 ` Oliver Hunt 2004-09-02 9:26 ` Helge Hafting 2004-09-05 13:28 ` Tonnerre 2004-09-02 9:52 ` Spam ` (5 subsequent siblings) 6 siblings, 2 replies; 1392+ messages in thread From: Oliver Hunt @ 2004-09-02 9:12 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List How would we go about finding out how many data forks were in a file? Because in order to be able to retrieve data from a fork we would need to know that the fork were there. Currently this would imply that we go looking through mtab or some such to find out what fs we're running on, which seems ugly. Alternatively we go through the _exciting_ task of making every other fs (with the exceptions of ntfs, and whatever it is that macs use, which would need there own custom code) and add code that effectively goes getNumForks(fileref){ return 1;} or add a new open call that can take a fork number... either way we have to add a new syscall, that doesn't conform to any real standard(though i suppose it would be possible to use macOS/windows style fork iopening interface) I personally like the concept of having multiple forks in a file, but in this case I'm inclined towards usermode first, then if it takes off add kernel level support. --Oliver Hunt On Thu, 02 Sep 2004 01:58:30 -0700, Hans Reiser <reiser@namesys.com> wrote: > Linus Torvalds wrote: > > > But _my_ point is, no user program is going to take _advantage_ of > > > >anything that only one filesystem on one system offers. > > > > > Apple does not have this problem.... > > and yes, the apps will take advantage of it, which is different from > depending on it. If you use the wrong fs you will lose some of the > features of the app. > > For 30 years nothing much has happened in Unix filesystem semantics > because of sheer cowardice (excepting Clearcase, which priced itself > into a niche market). It is 25 years past time for someone to change > things. That someone will have first mover advantage, and the more > little semantic features possessed the more lure there will be to use it > which will increase market share which will lure more apps into > depending on it and in a few years the other filesystems will > (deservedly) have only a small market share because the apps won't all > work on them. > > Besides, there are enhancements which are simply compelling. You can > write a dramatically better performance version control system with a > much simpler design if the FS is atomic. Our transaction manager > first draft was written by a version control guy, and he would probably > be happy to tell you how lack of atomicity other than rename makes > version control software design hideous. > > We have the performance lead. By next year we will be stable enough for > mission critical servers, and then we start the serious semantic > enhancements. > > If you don't embrace progress, then you doom Linux to following behind, > because the guys at Apple are pretty aggressive now that Jobs is back, > and they WILL change the semantics, and they will do so in compelling > ways, and Linux will be reduced to aping them when it should be leading > them. > > Hans > > > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 9:12 ` Oliver Hunt @ 2004-09-02 9:26 ` Helge Hafting 2004-09-02 9:45 ` Oliver Hunt 2004-09-05 13:28 ` Tonnerre 1 sibling, 1 reply; 1392+ messages in thread From: Helge Hafting @ 2004-09-02 9:26 UTC (permalink / raw) To: Oliver Hunt Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Oliver Hunt wrote: >How would we go about finding out how many data forks were in a file? >Because in order to be able to retrieve data from a fork we would need >to know that the fork were there. Currently this would imply that we >go looking through mtab or some such to find out what fs we're running >on, which seems ugly. > > > Depends on how the forks eventually get implemented. With the file-as-directory concept, all you need is to look at the file's directory part to see what is there. (The forks, implemented as files in a subdirectory.) It is done the same way as for an ordinary directory, so nothing "new". >Alternatively we go through the _exciting_ task of making every other >fs (with the exceptions of ntfs, and whatever it is that macs use, >which would need there own custom code) and add code that effectively >goes > >getNumForks(fileref){ return 1;} > > > Necessary if some other mechanism is used, sure. Helge Hafting ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 9:26 ` Helge Hafting @ 2004-09-02 9:45 ` Oliver Hunt 2004-09-02 10:02 ` Spam 2004-09-03 8:22 ` Helge Hafting 0 siblings, 2 replies; 1392+ messages in thread From: Oliver Hunt @ 2004-09-02 9:45 UTC (permalink / raw) To: Helge Hafting Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > Depends on how the forks eventually get implemented. > With the file-as-directory concept, all you need is to > look at the file's directory part to see what is there. (The forks, > implemented as files in a subdirectory.) It is done the same way > as for an ordinary directory, so nothing "new". > This still doesn't solve the problem of how we distinguish between a multi-fork file and a directory, for those old programs(ie. almost all that currently exist), to be able to access meaningful data we would need to either return just the primary fork, a serialized version of all forks in the file, or make the file look more or less identical to a directory. The first option could cause problems when transfering files between different filesystems, the second results in programs getting metadata they can't handle, and the third option results in few of the advantages over, well, directories... And even those applications that could handle the fork information nicely would need to fs type to find out whether they were handling a directory or a multi-forked file... As I say I like the idea, but I can't see anyway of implementing it in a way that is useful without first putting considerable effort into at least the VFS if not all the actual fs drivers. --Oliver Hunt ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 9:45 ` Oliver Hunt @ 2004-09-02 10:02 ` Spam 2004-09-02 11:00 ` Giuliano Pochini ` (2 more replies) 2004-09-03 8:22 ` Helge Hafting 1 sibling, 3 replies; 1392+ messages in thread From: Spam @ 2004-09-02 10:02 UTC (permalink / raw) To: Oliver Hunt Cc: Helge Hafting, Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List >> Depends on how the forks eventually get implemented. >> With the file-as-directory concept, all you need is to >> look at the file's directory part to see what is there. (The forks, >> implemented as files in a subdirectory.) It is done the same way >> as for an ordinary directory, so nothing "new". >> > This still doesn't solve the problem of how we distinguish between a > multi-fork file > and a directory, for those old programs(ie. almost all that currently > exist), to be able to access meaningful data we would need to either > return just the primary fork, a serialized version of all forks in the > file, or make the file look more or less identical to a directory. Well. wasn't the idea that unless programs specifically tried to open the file-as-dir as a directory it would look like a file? ls -F would show it as file. Or have I understood wrong? > The first option could cause problems when transfering files between > different filesystems, The meta-data should be deleted if it the file is copied or moved to a medium that doesn't support it. However a _warning_ may be shown to the user if there is risk to loose data. I do not think we should not implement something like this because most other filesystems in Linux doesn't support it. Think of other limitations there is in various ones. Large file support. Limited amount of files, length of file names, etc. > the second results in programs getting metadata they can't handle, and > the third option results in few of the advantages over, well, > directories... And even those applications that could handle the fork > information nicely would need to fs type to find out whether they were > handling a directory or a multi-forked file... No, if an application supported the file-as-dir then it could simply try to open the file as a directory. It would fail on systems that didn't support it. The meta-data and file streams would be seen as ordinary files. If applications support the contents or not isn't really relevant, is it?. > As I say I like the idea, but I can't see anyway of implementing it in > a way that is useful without first putting considerable effort into at > least the VFS if not all the actual fs drivers. Indeed. It is important that something like this gets implemented as a transparent way as possible. If it could be done in a general way so other filesystems like ext3/4 can eventually support it then that would be wonderful. I do not, however, think that we should block it in reiser4 because no other filesystems support it. ~S > --Oliver Hunt ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 10:02 ` Spam @ 2004-09-02 11:00 ` Giuliano Pochini 2004-09-02 14:38 ` Christian Mayrhuber 2004-09-02 22:39 ` Valdis.Kletnieks 2 siblings, 0 replies; 1392+ messages in thread From: Giuliano Pochini @ 2004-09-02 11:00 UTC (permalink / raw) To: Spam; +Cc: linux-kernel On Thu, 2 Sep 2004, Spam wrote: > Well. wasn't the idea that unless programs specifically tried to > open the file-as-dir as a directory it would look like a file? > > ls -F would show it as file. Or have I understood wrong? Yes, otherwise apps that do if (S_ISDIR()) { .. } else if (S_ISREG()) { .. } would behave differently from apps that check the file in different order, and they would probably break because in a regular fs that order is not important. The goal is to keep the semantic changes as hidden as possible for apps that don't know about the new features. -- Giuliano. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 10:02 ` Spam 2004-09-02 11:00 ` Giuliano Pochini @ 2004-09-02 14:38 ` Christian Mayrhuber 2004-09-02 22:39 ` Valdis.Kletnieks 2 siblings, 0 replies; 1392+ messages in thread From: Christian Mayrhuber @ 2004-09-02 14:38 UTC (permalink / raw) To: reiserfs-list, Spam; +Cc: linux-fsdevel, linux-kernel On Thursday 02 September 2004 12:02, Spam wrote: > > As I say I like the idea, but I can't see anyway of implementing it in > > a way that is useful without first putting considerable effort into at > > least the VFS if not all the actual fs drivers. > > Indeed. It is important that something like this gets implemented as > a transparent way as possible. If it could be done in a general way > so other filesystems like ext3/4 can eventually support it then that > would be wonderful. I do not, however, think that we should block it > in reiser4 because no other filesystems support it. What about extending the namespace with leading ".." and "...". In Unix names starting with a "." already have the meaning of being a hidden/config entry. A name starting ".." means streams/metainformation, etc. something belonging to the file/directory that should get backed up, copied, etc. There could be a "..streams", "..metas" or a "..acl" entry. If it starts with "..." it means some system specific information, like the name of the hash algorithm used for the current directory. It's basically information that is not portable and not required by applications, like "...fsplugins". There is still the big problem of how to copy files with their associated streams or meta information onto a standard unix filesystem as the file/directory duality cannot be expressed. (It's forbidden to have a directory + file with the same name in the same directory) Maybe copy could create something like a ".#filename" directory for this kind of information if the advanced features are not supported on the target filesystem. This is neither nice nor clean, but at least you don't loose information. I do not suggest that the kernel should simulate the advanced features with ".#filename" directories, it's more a backup/restore thing that could work if it's written down properly. -- lg, Chris ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 10:02 ` Spam 2004-09-02 11:00 ` Giuliano Pochini 2004-09-02 14:38 ` Christian Mayrhuber @ 2004-09-02 22:39 ` Valdis.Kletnieks 2004-09-02 23:06 ` Spam 2004-09-03 8:28 ` Helge Hafting 2 siblings, 2 replies; 1392+ messages in thread From: Valdis.Kletnieks @ 2004-09-02 22:39 UTC (permalink / raw) To: Spam Cc: Oliver Hunt, Helge Hafting, Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 390 bytes --] On Thu, 02 Sep 2004 12:02:59 +0200, Spam said: > The meta-data should be deleted if it the file is copied or moved to > a medium that doesn't support it. However a _warning_ may be shown > to the user if there is risk to loose data. OK... I'll bite. How do you report the warning to the user if you're using an unenhanced utility to copy a file to a file system that may be lossy? [-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 22:39 ` Valdis.Kletnieks @ 2004-09-02 23:06 ` Spam 2004-09-03 8:28 ` Helge Hafting 1 sibling, 0 replies; 1392+ messages in thread From: Spam @ 2004-09-02 23:06 UTC (permalink / raw) To: Valdis.Kletnieks Cc: Oliver Hunt, Helge Hafting, Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > On Thu, 02 Sep 2004 12:02:59 +0200, Spam said: >> The meta-data should be deleted if it the file is copied or moved to >> a medium that doesn't support it. However a _warning_ may be shown >> to the user if there is risk to loose data. > OK... I'll bite. How do you report the warning to the user if you're using > an unenhanced utility to copy a file to a file system that may be lossy? That I do not really know. But I think this is somewhat the discussion here - how to make the support for metas and streams as good as possible. I suppose it would be difficult to actually query the user about something like this unless the applications themselves do. Perhaps warnings in the syslog is good for now? Already now things like advanced attributes get lost when copying data from a supported fs to a non-supported one. Perhaps documentation around the ways to use streams and meta data should mention that only the supported fs will keep the extra info. As long as the user is aware that whatever information he puts in the meta or streams will be lost then it is ok, isn't it? ~S ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 22:39 ` Valdis.Kletnieks 2004-09-02 23:06 ` Spam @ 2004-09-03 8:28 ` Helge Hafting 1 sibling, 0 replies; 1392+ messages in thread From: Helge Hafting @ 2004-09-03 8:28 UTC (permalink / raw) To: Valdis.Kletnieks Cc: Spam, Oliver Hunt, Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Valdis.Kletnieks@vt.edu wrote: >On Thu, 02 Sep 2004 12:02:59 +0200, Spam said: > > > >> The meta-data should be deleted if it the file is copied or moved to >> a medium that doesn't support it. However a _warning_ may be shown >> to the user if there is risk to loose data. >> >> > >OK... I'll bite. How do you report the warning to the user if you're using >an unenhanced utility to copy a file to a file system that may be lossy? > > Well, _if_ you're using my sort of file-as-dir then you'd expect a plain "cp" to only copy the file. You'd use cp -a to copy the file's subdirectory too. If you're copying to anther filesystem that doesn't support file-as-directory then "cp -a" won't be able to create a directory with the same name as the file, and will do whatever cp does in such a situation. I.e. something like "NAME exists but is not a directory". Helge Hafting ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 9:45 ` Oliver Hunt 2004-09-02 10:02 ` Spam @ 2004-09-03 8:22 ` Helge Hafting 2004-09-03 17:41 ` Horst von Brand 2004-09-05 13:44 ` Tonnerre 1 sibling, 2 replies; 1392+ messages in thread From: Helge Hafting @ 2004-09-03 8:22 UTC (permalink / raw) To: Oliver Hunt Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Oliver Hunt wrote: >>Depends on how the forks eventually get implemented. >>With the file-as-directory concept, all you need is to >>look at the file's directory part to see what is there. (The forks, >>implemented as files in a subdirectory.) It is done the same way >>as for an ordinary directory, so nothing "new". >> >> >> >This still doesn't solve the problem of how we distinguish between a >multi-fork file >and a directory, for those old programs(ie. almost all that currently >exist), to be able to access meaningful data we would need to either >return just the primary fork, a serialized version of all forks in the >file, or make the file look more or less identical to a directory. > > If they open "filename" then they get what you calls the primary fork. If they open "filename/someforkname" then they get some other fork, using exactly the same mechanism as when they opens "directoryname/somefilename" And if you want to use the old tools to look at some forks, just do a $ cd filename It will work if "filename" happens to be a file-as-dir thing, then do $ ls And you'll see the forks by name. Because the "forks" are merely plain files in a subdirectory. The only new thing needed is the ability for something to be both file and directory at the same time. Some tools will need a update - usually only because they blindly assume that a directory isn't a file too, or that a file can't be a directory too. Remove the mistaken assumption and things will work because the underlying system calls (chdir or open) _will_ work. >The first option could cause problems when transfering files between >different filesystems, >the second results in programs getting metadata they can't handle, and >the third option results in few of the advantages over, well, >directories... > Few advantages over directories perhaps, but other implementatins don't have more advantages over directories than this one. Requiring another syscall to open forks other than the primary breaks _all_ existing software because it obviously don't use that syscall yet. And then it doesn't provide any advantages over the file-as-plain-directory way. . . >And even those applications that could handle the fork >information nicely would need to fs type to find out whether they were >handling a directory or a multi-forked file... > > > The distinction might not be important. A directory can be seen as a multi-stream file without the main stream. And there's always "stat" which should be able to tell what you're looking at. >As I say I like the idea, but I can't see anyway of implementing it in >a way that is useful without first putting considerable effort into at >least the VFS if not all the actual fs drivers. > > Sure, implementing it well will take effort, wich is why it is important to find a good way of doing it so the effort won't be wasted. Helge Hafting ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 8:22 ` Helge Hafting @ 2004-09-03 17:41 ` Horst von Brand 2004-09-03 19:30 ` Spam 2004-09-06 13:25 ` Helge Hafting 2004-09-05 13:44 ` Tonnerre 1 sibling, 2 replies; 1392+ messages in thread From: Horst von Brand @ 2004-09-03 17:41 UTC (permalink / raw) To: Helge Hafting Cc: Oliver Hunt, Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Helge Hafting <helge.hafting@hist.no> said: [...] > The only new thing needed is the ability for something to be both > file and directory at the same time. Then why have files and directories in the first place? > Some tools will need > a update - usually only because they blindly assume that a directory > isn't a file too, or that a file can't be a directory too. Remove the > mistaken assumption and things will work because the underlying system > calls (chdir or open) _will_ work. But with some weird restrictions: No moving stuff around between files, no linking, some "files" can't be deleted (how would you handle removing the principal stream of a file?). Some stuff you'd love to do (is, in fact, the reason for this all) just can't be allowed (i.e., J. Random Luser setting his own icon for system-wide emacs). So the tools/scripts/users/sysadmins will have to be painfully aware that some of the files aren't, and some of the directories aren't either. Major pain in the neck to use, if you look closer. Add extra kernel complexity. For little (if any) gain. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 17:41 ` Horst von Brand @ 2004-09-03 19:30 ` Spam 2004-09-03 21:07 ` Chris Dukes 2004-09-04 2:27 ` Horst von Brand 2004-09-06 13:25 ` Helge Hafting 1 sibling, 2 replies; 1392+ messages in thread From: Spam @ 2004-09-03 19:30 UTC (permalink / raw) To: Horst von Brand Cc: Helge Hafting, Oliver Hunt, Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > Helge Hafting <helge.hafting@hist.no> said: > [...] >> The only new thing needed is the ability for something to be both >> file and directory at the same time. > Then why have files and directories in the first place? Good point, we don't need them :) Directories are just a visible grouping of files to make it easier for the user to manage. But some things aren't really that intuitive with todays layout - especially for non-unix users. Just an example where the user needs to edit config file for some program. Where should he look? /etc/app.conf ? /etc/app/app.conf ? /etc/conf.d/app.conf ? /var/lib/app/app.conf ? and so on.... Using file-as-dir isn't really that much of a change. It isn't those that will confuse people anyway. >> Some tools will need >> a update - usually only because they blindly assume that a directory >> isn't a file too, or that a file can't be a directory too. Remove the >> mistaken assumption and things will work because the underlying system >> calls (chdir or open) _will_ work. > But with some weird restrictions: No moving stuff around between files, no > linking, some "files" can't be deleted (how would you handle removing the > principal stream of a file?). Well. there are read-only files. And if you remove the main stream (which is the file, really) then it will all be gone =) > Some stuff you'd love to do (is, in fact, the > reason for this all) just can't be allowed (i.e., J. Random Luser setting > his own icon for system-wide emacs). Users do not normally have write permissions to system-wide applications. Why would it be any different now? > So the tools/scripts/users/sysadmins > will have to be painfully aware that some of the files aren't, and some of > the directories aren't either. Major pain in the neck to use, if you look > closer. Add extra kernel complexity. For little (if any) gain. Not sure what you mean "aren't". Things shouldn't be that much different to administer. Normal permissions should still apply. Sure extra complexity comes _if_ you want to use extended features that are using meta-info. But there is where we need some patches to tar and other backup tools. One other way would be to enter a specific mode (chroot, a bash flag --show-metas, etc) that would allow all streams and metas to be seen as files in directories. Then tar and other tools would back it all up. Restoring will be a little trickier as we don't know if stuff was files or folders before. But I am sure that would be solvable. Perhaps a tool to convert them back to normal files with meta-data and streams. ~S ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 19:30 ` Spam @ 2004-09-03 21:07 ` Chris Dukes 2004-09-03 21:50 ` Spam 2004-09-04 2:27 ` Horst von Brand 1 sibling, 1 reply; 1392+ messages in thread From: Chris Dukes @ 2004-09-03 21:07 UTC (permalink / raw) To: Spam Cc: Horst von Brand, Helge Hafting, Oliver Hunt, Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Fri, Sep 03, 2004 at 09:30:38PM +0200, Spam wrote: > > > > > Helge Hafting <helge.hafting@hist.no> said: > > > [...] > > >> The only new thing needed is the ability for something to be both > >> file and directory at the same time. > > > Then why have files and directories in the first place? > > Good point, we don't need them :) Directories are just a visible > grouping of files to make it easier for the user to manage. But some > things aren't really that intuitive with todays layout - especially > for non-unix users. > > Just an example where the user needs to edit config file for some > program. Where should he look? > /etc/app.conf ? > /etc/app/app.conf ? > /etc/conf.d/app.conf ? > /var/lib/app/app.conf ? > and so on.... > > Using file-as-dir isn't really that much of a change. It isn't those > that will confuse people anyway. > If the non-computer literate user is expected to edit the config file, a pretty interface that hides the internals from the user should be provided. For the sake of power users, users that believe themselves to be power users, and configuration changes on multiple systems. 1) Track the location of the config file with the native package management system. 2) Keep a human readable/editable format. 3) Provide APIs or schema so that the files may be parsed for correctness, or edited or created programatically. 4) Provide a tool that not only checks for a parseable file, but does sanity checks such as (Does file exist). I'm at a loss as to how any of this has anything to do with filesystems. The closest I see is one implementation of the above (gconf2) happens to use the filesystem for it's default hierarchical database backend. As for developers that are too stupid/lazy to provide the GUI and meet the other criteria... I suggest extensive use of cattle prods on them as they attend an automata class. -- Chris Dukes Warning: Do not use the reflow toaster oven to prepare foods after it has been used for solder paste reflow. http://www.stencilsunlimited.com/stencil_article_page5.htm ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 21:07 ` Chris Dukes @ 2004-09-03 21:50 ` Spam 0 siblings, 0 replies; 1392+ messages in thread From: Spam @ 2004-09-03 21:50 UTC (permalink / raw) To: Chris Dukes Cc: Horst von Brand, Helge Hafting, Oliver Hunt, Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > On Fri, Sep 03, 2004 at 09:30:38PM +0200, Spam wrote: >> >> >> >> > Helge Hafting <helge.hafting@hist.no> said: >> >> > [...] >> >> >> The only new thing needed is the ability for something to be both >> >> file and directory at the same time. >> >> > Then why have files and directories in the first place? >> >> Good point, we don't need them :) Directories are just a visible >> grouping of files to make it easier for the user to manage. But some >> things aren't really that intuitive with todays layout - especially >> for non-unix users. >> >> Just an example where the user needs to edit config file for some >> program. Where should he look? >> /etc/app.conf ? >> /etc/app/app.conf ? >> /etc/conf.d/app.conf ? >> /var/lib/app/app.conf ? >> and so on.... >> >> Using file-as-dir isn't really that much of a change. It isn't those >> that will confuse people anyway. >> > If the non-computer literate user is expected to edit the config file, > a pretty interface that hides the internals from the user should be provided. > For the sake of power users, users that believe themselves to be power users, > and configuration changes on multiple systems. > 1) Track the location of the config file with the native package management > system. > 2) Keep a human readable/editable format. > 3) Provide APIs or schema so that the files may be parsed for correctness, > or edited or created programatically. > 4) Provide a tool that not only checks for a parseable file, but does > sanity checks such as (Does file exist). > I'm at a loss as to how any of this has anything to do with filesystems. > The closest I see is one implementation of the above (gconf2) happens to > use the filesystem for it's default hierarchical database backend. The relevance was as to the presentation of meta-data, streams and such and their implication on users. I just used the example of config files because there are not so much of a standard among them. Administrators still need to learn and remember where and what to backup. This wouldn't be so much of a change if they decide to introduce some usage for meta-data or named streams. Remember, even if the functionality exist you do not have to use everything of it. > As for developers that are too stupid/lazy to provide the GUI and meet > the other criteria... I suggest extensive use of cattle prods on > them as they attend an automata class. Lets all start with the bash(ing) ;). But really. What you say is true. The users do not care so much about what is going on below the curtains. Here is where the clash between different groups of Linux users and developers appear most vivid. Sure KDE and Gnome are on their way, but their efforts still have a long way to go. They still don't talk to eachother to strive for a comon goal - but competes instead (which may even be a good thing). ~S ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 19:30 ` Spam 2004-09-03 21:07 ` Chris Dukes @ 2004-09-04 2:27 ` Horst von Brand 2004-09-04 13:20 ` Alexander G. M. Smith 2004-09-06 13:43 ` Helge Hafting 1 sibling, 2 replies; 1392+ messages in thread From: Horst von Brand @ 2004-09-04 2:27 UTC (permalink / raw) To: Spam Cc: Helge Hafting, Oliver Hunt, Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Spam <spam@tnonline.net> said: > Horst von Brand <vonbrand@inf.utfsm.cl> said: > > Helge Hafting <helge.hafting@hist.no> said: > > > [...] > > >> The only new thing needed is the ability for something to be both > >> file and directory at the same time. > > > Then why have files and directories in the first place? > Good point, we don't need them :) Great. Then everything is a firectory (or dile?). And a firectory points at other firectories and contains data. I just don't see how you are supposed to distinguish the data from further firectories... > Directories are just a visible > grouping of files to make it easier for the user to manage. But some > things aren't really that intuitive with todays layout - especially > for non-unix users. No operating system is "intuitive". A person is able to glance over some 7 items, a filesystem handles tens of thousands of objects. Nothing "intuitive" possible in the layout, everybody has to _learn_ how to handle this. > Just an example where the user needs to edit config file for some > program. Where should he look? Another example: User looks for the sysadmin. Where does she look? Answers to both questions depend on extra data that can't possibly be deduced just from the statement. [...] > Using file-as-dir isn't really that much of a change. It isn't those > that will confuse people anyway. See above. > >> Some tools will need > >> a update - usually only because they blindly assume that a directory > >> isn't a file too, or that a file can't be a directory too. Remove the > >> mistaken assumption and things will work because the underlying system > >> calls (chdir or open) _will_ work. > > > But with some weird restrictions: No moving stuff around between files, no > > linking, some "files" can't be deleted (how would you handle removing the > > principal stream of a file?). > > Well. there are read-only files. And if you remove the main stream > (which is the file, really) then it will all be gone =) While some other process is messing with another stream. Right, all goes away when the last reference goes away. But it stays iff some other process recreates the data stream. Now what if another process goes ahead and creates a new file with the same name in the meanwhile... but this is very diferent from "real" directories, if you delete some particular member it doesn't go away. Explain that to the poor user that can't be trusted to keep the distinction between directories and files straight... > > Some stuff you'd love to do (is, in fact, the > > reason for this all) just can't be allowed (i.e., J. Random Luser setting > > his own icon for system-wide emacs). > Users do not normally have write permissions to system-wide > applications. Exactly. > Why would it be any different now? Because the user wants to be able to associate an icon/description/comment/... with system files too. This is supposed to be one of the key reasons for the feature... > > So the tools/scripts/users/sysadmins > > will have to be painfully aware that some of the files aren't, and some of > > the directories aren't either. Major pain in the neck to use, if you look > > closer. Add extra kernel complexity. For little (if any) gain. > Not sure what you mean "aren't". Things shouldn't be that much > different to administer. They aren't. Some directories are files, and different rules apply to them. Some files have structure, and you have to exercise extra care not to loose that. Very different indeed in my book. > Normal permissions should still apply. Even streams inside a file have permissions? What does it mean if a file is "searchable" (new permission needed to handle "x" on directories vs files!) but not writeable, and the default stream is writeable? Or do they just have the permissions of the container? In that case, the file must be readable so the user can have a nice icon for it. Consider /etc/shadow... And if somebody tries to change permissions on one of the streams? > Sure > extra complexity comes _if_ you want to use extended features that > are using meta-info. But there is where we need some patches to tar > and other backup tools. And new options (where people already get confused by current options). > One other way would be to enter a specific mode (chroot, a bash > flag --show-metas, etc) that would allow all streams and metas to be > seen as files in directories. Then tar and other tools would back it > all up. Restoring will be a little trickier as we don't know if > stuff was files or folders before. But I am sure that would be > solvable. "I'm sure it can be solved somehow" isn't enough. > Perhaps a tool to convert them back to normal files with > meta-data and streams. Yikes! -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-04 2:27 ` Horst von Brand @ 2004-09-04 13:20 ` Alexander G. M. Smith 2004-09-06 13:43 ` Helge Hafting 1 sibling, 0 replies; 1392+ messages in thread From: Alexander G. M. Smith @ 2004-09-04 13:20 UTC (permalink / raw) To: Horst von Brand Cc: helge.hafting, oliverhunt, reiser, torvalds, ninja, jamie, bunk, viro, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list, spam Horst von Brand wrote on Fri, 03 Sep 2004 22:27:07 -0400: > Great. Then everything is a firectory (or dile?). And a firectory points at > other firectories and contains data. I just don't see how you are supposed > to distinguish the data from further firectories... I like to call them fildirutes (file/directory/attribute). A file typing system would tell you the intended purpose of a particular fildurute. So if a fildurute called X has X/..metas/mimefiletype containing "application/x-directory" then you know that it should be treated as being primarily a container for other fildirutes and shown to the user as a folder in a GUI view. If it said it was "application/x-text-document" then the GUI system would default to opening it in a word processor. Either way, that's only a hint about how it should be presented to the user, not something the kernel enforces. For efficiency, the file type might just be stored in the fildirute's inode as a code number (since most things would have a file type) that the file system exposes as a file called ..metas/mimefiletype. - Alex ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-04 2:27 ` Horst von Brand 2004-09-04 13:20 ` Alexander G. M. Smith @ 2004-09-06 13:43 ` Helge Hafting 1 sibling, 0 replies; 1392+ messages in thread From: Helge Hafting @ 2004-09-06 13:43 UTC (permalink / raw) To: Horst von Brand Cc: Spam, Oliver Hunt, Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Horst von Brand wrote: >Spam <spam@tnonline.net> said: > > >>Horst von Brand <vonbrand@inf.utfsm.cl> said: >> >> >>>Helge Hafting <helge.hafting@hist.no> said: >>> >>> >>>[...] >>> >>> >>>>The only new thing needed is the ability for something to be both >>>>file and directory at the same time. >>>> >>>> >>>Then why have files and directories in the first place? >>> >>> > > > >> Good point, we don't need them :) >> >> > >Great. Then everything is a firectory (or dile?). And a firectory points at >other firectories and contains data. I just don't see how you are supposed >to distinguish the data from further firectories... > > By name. fopen("somefile", "r"); //opens somefile data fopen("somefile/substream", "r"); //opens a substream of somefile. Helge Hafting ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 17:41 ` Horst von Brand 2004-09-03 19:30 ` Spam @ 2004-09-06 13:25 ` Helge Hafting 1 sibling, 0 replies; 1392+ messages in thread From: Helge Hafting @ 2004-09-06 13:25 UTC (permalink / raw) To: Horst von Brand Cc: Oliver Hunt, Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Horst von Brand wrote: >Helge Hafting <helge.hafting@hist.no> said: > >[...] > > > >>The only new thing needed is the ability for something to be both >>file and directory at the same time. >> >> > >Then why have files and directories in the first place? > > > >> Some tools will need >>a update - usually only because they blindly assume that a directory >>isn't a file too, or that a file can't be a directory too. Remove the >>mistaken assumption and things will work because the underlying system >>calls (chdir or open) _will_ work. >> >> > >But with some weird restrictions: No moving stuff around between files, no > > I see no need for such a weird restriction. If it is "weird", why have it? If someone want to move stuff from one subdirectory to another, let them. I see no need to stop that just because a file-as-directory was involved. >linking, some "files" can't be deleted (how would you handle removing the > > linking is a problem of course. We can't have hardlinks to directories, so no hardlinks to file-as-dir either unless the general problems with directory links are solved. >principal stream of a file?). > A design decision. Removing the principal stream may delete the entire tree. Or it may delete the principal stream only, turning that file-as-dir back into a plain directory. >Some stuff you'd love to do (is, in fact, the >reason for this all) just can't be allowed (i.e., J. Random Luser setting >his own icon for system-wide emacs). So the tools/scripts/users/sysadmins >will have to be painfully aware that some of the files aren't, and some of >the directories aren't either. Major pain in the neck to use, if you look >closer. Add extra kernel complexity. For little (if any) gain. > > The gain is indeed the interesting question here. It isn't something I desperately need. But if file-as-directory _happens_, then I hope it'll happen in a good way. No unnecessary complications or restrictions, and allowing existing stuff to work as well as possible. There will surely be some tool problems - but we certainly don't need to break every existing tool the way a new form of "open" will. . . Helge Hafting ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 8:22 ` Helge Hafting 2004-09-03 17:41 ` Horst von Brand @ 2004-09-05 13:44 ` Tonnerre 2004-09-05 14:13 ` Spam 2004-09-08 9:24 ` Helge Hafting 1 sibling, 2 replies; 1392+ messages in thread From: Tonnerre @ 2004-09-05 13:44 UTC (permalink / raw) To: Helge Hafting Cc: Oliver Hunt, Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 597 bytes --] Salut, On Fri, Sep 03, 2004 at 10:22:55AM +0200, Helge Hafting wrote: > Requiring another syscall to open forks other than the primary > breaks _all_ existing software because it obviously don't use that > syscall yet. And then it doesn't provide any advantages over the > file-as-plain-directory way. . . Actually... We might tune the sys_open() call to take an additional argument (the stream ID), and introduce a compatibility interface into *libc which chooses strid=0 by default if the plain old open call is being used. Maybe this can be handled transparently by cpp.. Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-05 13:44 ` Tonnerre @ 2004-09-05 14:13 ` Spam 2004-09-08 9:24 ` Helge Hafting 1 sibling, 0 replies; 1392+ messages in thread From: Spam @ 2004-09-05 14:13 UTC (permalink / raw) To: Tonnerre Cc: Helge Hafting, Oliver Hunt, Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > Salut, > On Fri, Sep 03, 2004 at 10:22:55AM +0200, Helge Hafting wrote: >> Requiring another syscall to open forks other than the primary >> breaks _all_ existing software because it obviously don't use that >> syscall yet. And then it doesn't provide any advantages over the >> file-as-plain-directory way. . . > Actually... > We might tune the sys_open() call to take an additional argument (the > stream ID), and introduce a compatibility interface into *libc which > chooses strid=0 by default if the plain old open call is being used. > Maybe this can be handled transparently by cpp.. How are things like copy handled today? Is it just some code linked in during compilation (ie, the application handles the copy itself) or is it a library function in libc? If it is in libc then it ought to be enough if libc is patched to support the new semantics with file-as-dir etc. ~S > Tonnerre ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-05 13:44 ` Tonnerre 2004-09-05 14:13 ` Spam @ 2004-09-08 9:24 ` Helge Hafting 2004-09-08 16:02 ` Tonnerre 1 sibling, 1 reply; 1392+ messages in thread From: Helge Hafting @ 2004-09-08 9:24 UTC (permalink / raw) To: Tonnerre Cc: Oliver Hunt, Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Tonnerre wrote: >Salut, > >On Fri, Sep 03, 2004 at 10:22:55AM +0200, Helge Hafting wrote: > > >>Requiring another syscall to open forks other than the primary >>breaks _all_ existing software because it obviously don't use that >>syscall yet. And then it doesn't provide any advantages over the >>file-as-plain-directory way. . . >> >> > >Actually... > >We might tune the sys_open() call to take an additional argument (the >stream ID), and introduce a compatibility interface into *libc which >chooses strid=0 by default if the plain old open call is being used. > > But this isn't necessary, as I pointed out above. There is no _need_ for a new kind of open(), so why make one in libc? If you can open a fork/substream/whatever by issuing open("filename/forkname", ... then the old-fashioned open() works with multi-fork files too. An tools based on "open() something, then work with the resulting file descriptor" will work _unchanged_ with such a multi-fork fs. Tools that rely on "stat" or directory traversals might need some updating to work perfectly with such a fs, but not something that just opens by name. Helge Hafting ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-08 9:24 ` Helge Hafting @ 2004-09-08 16:02 ` Tonnerre 2004-09-09 11:22 ` Helge Hafting 0 siblings, 1 reply; 1392+ messages in thread From: Tonnerre @ 2004-09-08 16:02 UTC (permalink / raw) To: Helge Hafting Cc: Oliver Hunt, Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 713 bytes --] Salut, On Wed, Sep 08, 2004 at 11:24:40AM +0200, Helge Hafting wrote: > If you can open a fork/substream/whatever by issuing > open("filename/forkname", ... > then the old-fashioned open() works with multi-fork files too. > An tools based on "open() something, then work with > the resulting file descriptor" will work _unchanged_ > with such a multi-fork fs. In my version they'd run unchanged as well. And BTW, I wasn't talking about introducing a new open at libc level. I was talking about modifying the open system call in the kernel and having libc provide compatibility for the old call. Since I'm not sure how much breakage it takes to make a file a directory. Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-08 16:02 ` Tonnerre @ 2004-09-09 11:22 ` Helge Hafting 0 siblings, 0 replies; 1392+ messages in thread From: Helge Hafting @ 2004-09-09 11:22 UTC (permalink / raw) To: Tonnerre Cc: Oliver Hunt, Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Tonnerre wrote: >Salut, > >On Wed, Sep 08, 2004 at 11:24:40AM +0200, Helge Hafting wrote: > > >>If you can open a fork/substream/whatever by issuing >>open("filename/forkname", ... >>then the old-fashioned open() works with multi-fork files too. >>An tools based on "open() something, then work with >>the resulting file descriptor" will work _unchanged_ >>with such a multi-fork fs. >> >> > >In my version they'd run unchanged as well. > >And BTW, I wasn't talking about introducing a new open at libc >level. I was talking about modifying the open system call in the >kernel and having libc provide compatibility for the old call. > >Since I'm not sure how much breakage it takes to make a file a >directory. > I believe most of the changes necessary for supporting file-as-dir will be in the internal workings of filesystems, not so much in the interfaces. I don't think it is necessary to change the open system call, because opening a file in a subdirectory or a named substream is the same thing. The _uses_ people have suggested for file-as-dir (complex file attributes and different views of the data) tends to be different from what a directory usually does, but the implementation doesn't have to be. There's no need to add extra interfaces or restrictions. File-as-dir that really is a directory, so that it may contain a whole tree is more flexible. And the underlying fs can use existing directory code instead of something extra. So the only change needed for the open system call is that it will suddenly succeed in the open("filename/forkname") case that used to complain about an invalid path because "filename" isn't a directory. "stat" may have to change though, so it can identify a file-as-dir. Helge Hafting ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 9:12 ` Oliver Hunt 2004-09-02 9:26 ` Helge Hafting @ 2004-09-05 13:28 ` Tonnerre 1 sibling, 0 replies; 1392+ messages in thread From: Tonnerre @ 2004-09-05 13:28 UTC (permalink / raw) To: Oliver Hunt Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 449 bytes --] Salut, On Thu, Sep 02, 2004 at 09:12:56PM +1200, Oliver Hunt wrote: > getNumForks(fileref){ return 1;} Actually it might be even cooler to have a generic function which you can apply to any file on a file system (or whatever) and which returns you the file system's capabilities (can write, has xattrs, has acl, has multiple streams, *not* has btree lookups etc, as that is *not* interesting to userland anyway.) Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 8:58 ` Hans Reiser 2004-09-02 9:12 ` Oliver Hunt @ 2004-09-02 9:52 ` Spam 2004-09-02 9:32 ` Alan Cox ` (5 more replies) 2004-09-02 14:24 ` John Stoffel ` (4 subsequent siblings) 6 siblings, 6 replies; 1392+ messages in thread From: Spam @ 2004-09-02 9:52 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > Linus Torvalds wrote: >> But _my_ point is, no user program is going to take _advantage_ of >> >>anything that only one filesystem on one system offers. >> >> > Apple does not have this problem.... > and yes, the apps will take advantage of it, which is different from > depending on it. If you use the wrong fs you will lose some of the > features of the app. > For 30 years nothing much has happened in Unix filesystem semantics > because of sheer cowardice (excepting Clearcase, which priced itself > into a niche market). It is 25 years past time for someone to change > things. That someone will have first mover advantage, and the more > little semantic features possessed the more lure there will be to use it > which will increase market share which will lure more apps into > depending on it and in a few years the other filesystems will > (deservedly) have only a small market share because the apps won't all > work on them. > Besides, there are enhancements which are simply compelling. You can > write a dramatically better performance version control system with a > much simpler design if the FS is atomic. Our transaction manager > first draft was written by a version control guy, and he would probably > be happy to tell you how lack of atomicity other than rename makes > version control software design hideous. Btw, version control for ordinary files would be a great feature. I think something like it is available through Windows 2000/3 server. Isn't it called "Shadow Copies". It works over network shares. :) It allows you to restore previous versions of the file even if you delete or overwrite it. Features like this do make a good point and helps protecting data - something that is important IMHO. http://www.microsoft.com/resources/documentation/windowsserv/2003/standard/proddocs/en-us/overview_snapshot.asp > We have the performance lead. By next year we will be stable enough for > mission critical servers, and then we start the serious semantic > enhancements. > If you don't embrace progress, then you doom Linux to following behind, > because the guys at Apple are pretty aggressive now that Jobs is back, > and they WILL change the semantics, and they will do so in compelling > ways, and Linux will be reduced to aping them when it should be leading > them. > Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 9:52 ` Spam @ 2004-09-02 9:32 ` Alan Cox 2004-09-02 10:27 ` V13 ` (4 subsequent siblings) 5 siblings, 0 replies; 1392+ messages in thread From: Alan Cox @ 2004-09-02 9:32 UTC (permalink / raw) To: Spam Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Iau, 2004-09-02 at 10:52, Spam wrote: > Btw, version control for ordinary files would be a great feature. I > think something like it is available through Windows 2000/3 server. > Isn't it called "Shadow Copies". It works over network shares. :) Netapps have done this for years with NFS, Subversion does it with DAV, clearcase does it with NFS. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 9:52 ` Spam 2004-09-02 9:32 ` Alan Cox @ 2004-09-02 10:27 ` V13 2004-09-02 14:35 ` John Stoffel 2004-09-02 11:09 ` Oliver Neukum ` (3 subsequent siblings) 5 siblings, 1 reply; 1392+ messages in thread From: V13 @ 2004-09-02 10:27 UTC (permalink / raw) To: Spam Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thursday 02 September 2004 12:52, Spam wrote: > > Linus Torvalds wrote: > > Besides, there are enhancements which are simply compelling. You can > > write a dramatically better performance version control system with a > > much simpler design if the FS is atomic. Our transaction manager > > first draft was written by a version control guy, and he would probably > > be happy to tell you how lack of atomicity other than rename makes > > version control software design hideous. > > Btw, version control for ordinary files would be a great feature. I > think something like it is available through Windows 2000/3 server. > Isn't it called "Shadow Copies". It works over network shares. :) > > It allows you to restore previous versions of the file even if you > delete or overwrite it. > > Features like this do make a good point and helps protecting data - > something that is important IMHO. I believe you mean something simillar to: file1.txt;1 file1.txt;2 file1.txt;3 (yeap, it's VMS) where you'll have to cleanup old versions when you don't need them any more... AFAIK that this is older than HDDs <<V13>> ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 10:27 ` V13 @ 2004-09-02 14:35 ` John Stoffel 0 siblings, 0 replies; 1392+ messages in thread From: John Stoffel @ 2004-09-02 14:35 UTC (permalink / raw) To: V13 Cc: Spam, Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List V13> I believe you mean something simillar to: V13> file1.txt;1 V13> file1.txt;2 V13> file1.txt;3 (yeap, it's VMS) Or TOPS-20, a precursor to VMS in some ways. It was a nice feature. V13> where you'll have to cleanup old versions when you don't need V13> them any more... AFAIK that this is older than HDDs It was usually an automatic cleanup past a certain point or if you went over disk quota. In any case, while I do like this feature, I'm not sure how we would cleanly implement this inside the unix namespace, or if inside a new namespace, how that new namespace would be joined with a standard Unix one. John ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 9:52 ` Spam 2004-09-02 9:32 ` Alan Cox 2004-09-02 10:27 ` V13 @ 2004-09-02 11:09 ` Oliver Neukum 2004-09-02 11:11 ` Spam 2004-09-03 0:43 ` David Masover 2004-09-02 15:40 ` Horst von Brand ` (2 subsequent siblings) 5 siblings, 2 replies; 1392+ messages in thread From: Oliver Neukum @ 2004-09-02 11:09 UTC (permalink / raw) To: Spam Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Am Donnerstag, 2. September 2004 11:52 schrieb Spam: > Btw, version control for ordinary files would be a great feature. I > think something like it is available through Windows 2000/3 server. > Isn't it called "Shadow Copies". It works over network shares. :) > > It allows you to restore previous versions of the file even if you > delete or overwrite it. There's no need to do that in kernel, unless you want to be able to force it unto users. Regards Oliver ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 11:09 ` Oliver Neukum @ 2004-09-02 11:11 ` Spam 2004-09-03 0:43 ` David Masover 1 sibling, 0 replies; 1392+ messages in thread From: Spam @ 2004-09-02 11:11 UTC (permalink / raw) To: Oliver Neukum Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > Am Donnerstag, 2. September 2004 11:52 schrieb Spam: >> Btw, version control for ordinary files would be a great feature. I >> think something like it is available through Windows 2000/3 server. >> Isn't it called "Shadow Copies". It works over network shares. :) >> >> It allows you to restore previous versions of the file even if you >> delete or overwrite it. > There's no need to do that in kernel, unless you want to be able > to force it unto users. Exactly ;) Difference with having it in just certain applications like Gnome-VFS etc is that it would work with all applications, over SMB shares etc. This is a great advantage and benefit for many desktop users but also in a corporate environment. Even though there are backups of users documents it is a pain to restore them every time they loose the original file. Besides, the backup is never as fresh either. > Regards > Oliver ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 11:09 ` Oliver Neukum 2004-09-02 11:11 ` Spam @ 2004-09-03 0:43 ` David Masover 2004-09-03 0:47 ` Spam ` (2 more replies) 1 sibling, 3 replies; 1392+ messages in thread From: David Masover @ 2004-09-03 0:43 UTC (permalink / raw) To: Oliver Neukum Cc: Spam, Hans Reiser, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Oliver Neukum wrote: | Am Donnerstag, 2. September 2004 11:52 schrieb Spam: | |> Btw, version control for ordinary files would be a great feature. I |> think something like it is available through Windows 2000/3 server. |> Isn't it called "Shadow Copies". It works over network shares. :) |> |> It allows you to restore previous versions of the file even if you |> delete or overwrite it. | | | There's no need to do that in kernel, unless you want to be able | to force it unto users. And on apps. Should I teach OpenOffice.org to do version control? Seems a lot easier to just do it in the kernel, and teach everything to do version control in one fell swoop. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTe+NXgHNmZLgCUhAQLW+RAAnoQIQLXSd2vQudOKAaLgXXCxoa+gEWIq zcNd+bIWPuQxwcZYlqbs/pDFiDky3XqLRaZv8PZx++rs9mMnv36eAtMX7uUp3egz MxymeDDI796wtsDDBGcZtO+WwmqJ0fPgQMN/Q237s2yccPGJoDIVle9mzZAfNIdO RDdLq142D+0t1iqfiVqaA+NgkArrGhcr0Hs/3TdviJX30gPL/Jm9eOvc3H3NgIQu XToDQB1UnLoyiIwc16IY4ZxYoxf1YnFtrivZb+YgC61mIQLeqJMjixCAtBVzAfKN 2GmlsnfWcMCgXSEX8imkoNSLcmkv886+esGXZVzBbY1/Qg/f5MCXZilbb8e6+4I9 U4ReEUg5fGJW8JyKAXqKGyvbC/lim55Vgcjp/J8mrhUa3Q3cwPKoi2GKNRcbdpT/ pPB2SADA1eNrmFyGB7kLcqt0F9i4fIyTrItpHq9+m72OdFEze2OnWn+eIY8dRut7 SaGTq6ZEYtkH8oWT9xSXp1d2TUxeSVTdx8EffIIdwXAVOO0Z9xd71MJPLMp3VOMH Qld4gjYjKFzwj3QlcWcsiVhk6Zq2fmqoOCQNfxhBzH1Le3v0F0ugammkQUXtRAub dKEWxGfkDGOwviJXKeckDNn98x88vh7C/7gxpAOIcVSRDhEV+MDWIfMMvotE/lxe 393hENsXOgQ= =g+3J -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 0:43 ` David Masover @ 2004-09-03 0:47 ` Spam 2004-09-03 1:36 ` David Masover 2004-09-03 1:18 ` Valdis.Kletnieks 2004-09-03 8:13 ` Christoph Hellwig 2 siblings, 1 reply; 1392+ messages in thread From: Spam @ 2004-09-03 0:47 UTC (permalink / raw) To: David Masover Cc: Oliver Neukum, Hans Reiser, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > Oliver Neukum wrote: > | Am Donnerstag, 2. September 2004 11:52 schrieb Spam: > | |>> Btw, version control for ordinary files would be a great feature. I |>> think something like it is available through Windows 2000/3 server. |>> Isn't it called "Shadow Copies". It works over network shares. :) |>> |>> It allows you to restore previous versions of the file even if you |>> delete or overwrite it. > | > | > | There's no need to do that in kernel, unless you want to be able > | to force it unto users. > And on apps. Should I teach OpenOffice.org to do version control? > Seems a lot easier to just do it in the kernel, and teach everything to > do version control in one fell swoop. Do you mean in the kernel or as a filesystem/VFS plugin that would extend the functionality to include version control? ~S ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 0:47 ` Spam @ 2004-09-03 1:36 ` David Masover 0 siblings, 0 replies; 1392+ messages in thread From: David Masover @ 2004-09-03 1:36 UTC (permalink / raw) To: Spam Cc: Oliver Neukum, Hans Reiser, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Spam wrote: [...] | Do you mean in the kernel or as a filesystem/VFS plugin that would | extend the functionality to include version control? The filesystem plugin. But this needs some kernel support, which scares people. Read the archives if you don't believe me. Besides, we'd want per-block copy-on-write support anyway to make it sane. That has to go in the filesystem. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTfKhXgHNmZLgCUhAQIR5w//Z6hg58uVvIJuwNCb764lFj22iHwLvg0T JvjQeOSCz7p2/uv+dHN2MRNDeezmvGfH49Y7RE78Q9e1xrfZFGZKyybHVisfea1t f9Bk8M6ZtykKnUdPhq5MsDH9Oo98dXMNEWhDw5WUYMzvglitMUcd4np3gGEtr539 yKHuxZhqjc/lQ9rd6PFKJYab1m9RAabQnGMHRIvx0T98iW3EI9pyOd0u1oUVCiRi RxllyP0KCg7wxFv93b4zYHsv5BO8E7MWVI4aKuBUCqrc8ObJnDLDbO/8FfMvDt/1 Do9nlikFh4IaRvwZtmQA0nVv80uxtjnSOCs7F2r+qqNyprcHIdj2m+3x5zE8Tbfz YCkI5BopHub94yBvhlcFGiKBtigLDsH3zHc66dVDl4SreJPKo+yirjjZ+RrU3HN1 iK+aFp5GxapcYDkMNOS0ab9TPejqLVyJ+Ci5mSNckcMARC9v+2mT7JTCBupyXZar LSLFYef+W7ipKq5ZMurDB17I5y7VXfwOTd+oSwQdMwlsQlJs/AFOOGdJywtLGLy1 6jhejQlziPLWB9hz9wFAjWyHPrS0XsCNbtGbBESIVmqvLsZmpcAULpMn7TGc8aJg 0r79ODKlGK8lYJRIquRhZzd51oqiBfFJcX969XGPMPUBX4x3GZjaOlj79OQ7Ih3s Qd8PSfUVzGU= =Cw9B -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 0:43 ` David Masover 2004-09-03 0:47 ` Spam @ 2004-09-03 1:18 ` Valdis.Kletnieks 2004-09-03 1:39 ` David Masover 2004-09-03 8:13 ` Christoph Hellwig 2 siblings, 1 reply; 1392+ messages in thread From: Valdis.Kletnieks @ 2004-09-03 1:18 UTC (permalink / raw) To: David Masover Cc: Oliver Neukum, Spam, Hans Reiser, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 1157 bytes --] On Thu, 02 Sep 2004 19:43:34 CDT, David Masover said: > And on apps. Should I teach OpenOffice.org to do version control? > Seems a lot easier to just do it in the kernel, and teach everything to > do version control in one fell swoop. Including files you didn't really want to keep version control of? How many temp files does gcc create and unlink in the course of a kernel build? (And remember, you can't say "don't enable that on /tmp" - gcc respects the setting of $TMPDIR - so an 'export TMPDIR=~/tmp' confuses things quite nicely...) And it's hard for the kernel to know that an unlink() done by gcc should be treated differently than the "recover the last version" you *want* it do be able to do after you work on a source file for a long while, save it, and then fumble-finger a 'rm * .o' - you can't even use a heuristic like "don't version control it unless it's N seconds or more old" (Note that the "obvious" solution of creating a chattr flag has its own complexity issues - should versioning be turned on by default for some types and not others, etc...) There be dragons here - it's not as simple as "drop in a plugin and be happy". [-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 1:18 ` Valdis.Kletnieks @ 2004-09-03 1:39 ` David Masover 0 siblings, 0 replies; 1392+ messages in thread From: David Masover @ 2004-09-03 1:39 UTC (permalink / raw) To: Valdis.Kletnieks Cc: Oliver Neukum, Spam, Hans Reiser, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Valdis.Kletnieks@vt.edu wrote: | On Thu, 02 Sep 2004 19:43:34 CDT, David Masover said: | | |>And on apps. Should I teach OpenOffice.org to do version control? |>Seems a lot easier to just do it in the kernel, and teach everything to |>do version control in one fell swoop. | | | Including files you didn't really want to keep version control of? No one said you have to apply it to everything. Only that you can. Right now vim cannot do version control, AFAIK. But you could do something like: echo 1 > foo/version_control vim foo Default would be off, of course. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTfLR3gHNmZLgCUhAQKybRAAkF4sqLUCJEfpDNvKEyv8MJGcy0w+qV12 3Rf50OQeeurZMZrnli0vNIuSIWwIsGI+j7qbj1mlfi+Ps3dAjOlKXav3tGoGshLl HH/Wl9XvGPOAWlIQOqqBGAyno8sDIvHAjh5lH3C7m+9/2Ao7s1G/bQ+hm/deYPJK WPZxExd/1BdchrbjNjbfWHKrD6LdF/GBa/Vbj1F8m95WEvxsfbDnSTwb5HhvcUxD TtqwkINmx7Tle2p9q7PcgR7y76dmcyoWw82ST3BfF/AbGauMQAr0h6jNjvzclfgJ yrIOwGXdR3pn8ZK8dHVct79N0f+PYFdeAzLYxMZR2vefB2enQolbbs6Lf7X60g2g 5hrH/ezenFJIZMPhQtlJqBaFuA5GHw9B2An6VbOmDm7sSOFyHSlKoCUFlJ/I5nZd Bcn46lhNxvTJhu3tvnvDqTB8f0/yVYt5QfoBFA1mzHKig2iNN6vr2xLGdWSLU6XD vRjy+KyVC+PvjL13E0JrjXy9UuqY7CH2xAunSgLJ2cvfRmNRoVoV1hkp3McQcUqF GiD3s0REcztlvsrP+DxKLRtPpnOJi5By5NmZpJkiPdVB9zpIsL/Ia7KL/A7IGGh3 IF3t/xSUmjgrek01SeadHqsIngeWQ+F8fiars7NtzNhrvRgPVfDVheey3vhQjFxM 3kDKH+I9rGk= =3+nq -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 0:43 ` David Masover 2004-09-03 0:47 ` Spam 2004-09-03 1:18 ` Valdis.Kletnieks @ 2004-09-03 8:13 ` Christoph Hellwig 2004-09-03 8:30 ` Oliver Neukum 2 siblings, 1 reply; 1392+ messages in thread From: Christoph Hellwig @ 2004-09-03 8:13 UTC (permalink / raw) To: David Masover Cc: Oliver Neukum, Spam, Hans Reiser, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Sep 02, 2004 at 07:43:34PM -0500, David Masover wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Oliver Neukum wrote: > | Am Donnerstag, 2. September 2004 11:52 schrieb Spam: > | > |> Btw, version control for ordinary files would be a great feature. I > |> think something like it is available through Windows 2000/3 server. > |> Isn't it called "Shadow Copies". It works over network shares. :) > |> > |> It allows you to restore previous versions of the file even if you > |> delete or overwrite it. > | > | > | There's no need to do that in kernel, unless you want to be able > | to force it unto users. > > And on apps. Should I teach OpenOffice.org to do version control? > Seems a lot easier to just do it in the kernel, and teach everything to > do version control in one fell swoop. Just add a post-save trigger that can check it into any SCM you want. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 8:13 ` Christoph Hellwig @ 2004-09-03 8:30 ` Oliver Neukum 0 siblings, 0 replies; 1392+ messages in thread From: Oliver Neukum @ 2004-09-03 8:30 UTC (permalink / raw) To: Christoph Hellwig Cc: David Masover, Spam, Hans Reiser, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Am Freitag, 3. September 2004 10:13 schrieb Christoph Hellwig: > On Thu, Sep 02, 2004 at 07:43:34PM -0500, David Masover wrote: > > | There's no need to do that in kernel, unless you want to be able > > | to force it unto users. > > > > And on apps. Should I teach OpenOffice.org to do version control? > > Seems a lot easier to just do it in the kernel, and teach everything to > > do version control in one fell swoop. > > Just add a post-save trigger that can check it into any SCM you want. That depends on whom you refer to. If you want to impose the RCS on the users as an administrative measure and force compliance, then you'll need to do it in kernel. I see some nasty issues with disk quotas there. Regards Oliver ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 9:52 ` Spam ` (2 preceding siblings ...) 2004-09-02 11:09 ` Oliver Neukum @ 2004-09-02 15:40 ` Horst von Brand 2004-09-02 18:21 ` Lee Revell 2004-09-03 8:30 ` Helge Hafting 5 siblings, 0 replies; 1392+ messages in thread From: Horst von Brand @ 2004-09-02 15:40 UTC (permalink / raw) To: Spam Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Spam <spam@tnonline.net> said: [...] > Btw, version control for ordinary files would be a great feature. I > think something like it is available through Windows 2000/3 server. > Isn't it called "Shadow Copies". It works over network shares. :) Right. And when space runs low, the old copies get the axe. And then you need the old version... and are royally screwed because you trusted the system to save it for you while willy-nilly fooling around. I never liked the MS-DOS "undelete" feature, because I saw too many people burned. It is _less_ data security than "deleted file is gone for good", in the end sum. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 9:52 ` Spam ` (3 preceding siblings ...) 2004-09-02 15:40 ` Horst von Brand @ 2004-09-02 18:21 ` Lee Revell 2004-09-03 8:30 ` Helge Hafting 5 siblings, 0 replies; 1392+ messages in thread From: Lee Revell @ 2004-09-02 18:21 UTC (permalink / raw) To: Spam Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2004-09-02 at 05:52, Spam wrote: > Btw, version control for ordinary files would be a great feature. I > think something like it is available through Windows 2000/3 server. > Isn't it called "Shadow Copies". It works over network shares. :) > > It allows you to restore previous versions of the file even if you > delete or overwrite it. > > Features like this do make a good point and helps protecting data - > something that is important IMHO. > > http://www.microsoft.com/resources/documentation/windowsserv/2003/standard/proddocs/en-us/overview_snapshot.asp > Netapp's WAFL has had this for ages, they are called snapshots, and they are one of the most useful file system features ever. But Microsoft certainly did not invent them. Lee ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 9:52 ` Spam ` (4 preceding siblings ...) 2004-09-02 18:21 ` Lee Revell @ 2004-09-03 8:30 ` Helge Hafting 5 siblings, 0 replies; 1392+ messages in thread From: Helge Hafting @ 2004-09-03 8:30 UTC (permalink / raw) To: Spam Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Spam wrote: > > Btw, version control for ordinary files would be a great feature. I > think something like it is available through Windows 2000/3 server. > Isn't it called "Shadow Copies". It works over network shares. :) > > It allows you to restore previous versions of the file even if you > delete or overwrite it. > > Feel free to use cvs on your home directory. :-) Helge Hafting ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 8:58 ` Hans Reiser 2004-09-02 9:12 ` Oliver Hunt 2004-09-02 9:52 ` Spam @ 2004-09-02 14:24 ` John Stoffel 2004-09-02 15:36 ` Alan Cox 2004-09-02 15:32 ` Horst von Brand ` (3 subsequent siblings) 6 siblings, 1 reply; 1392+ messages in thread From: John Stoffel @ 2004-09-02 14:24 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hans> For 30 years nothing much has happened in Unix filesystem Hans> semantics because of sheer cowardice (excepting Clearcase, which Hans> priced itself into a niche market). Keeping a consistent namespace is cowardice? And ClearCase is an example of an application which sets up it's own namespace and grafts it onto the standard Unix one. Most standard Unix tools work just fine inside clearcase Views, but to manage the metadata (xattrs, or whatever you want to call them), you need to use the 'cleartool' command. Hmmm... sounds like 'runat' to me. Hans> It is 25 years past time for someone to change things. That Hans> someone will have first mover advantage, and the more little Hans> semantic features possessed the more lure there will be to use Hans> it which will increase market share which will lure more apps Hans> into depending on it and in a few years the other filesystems Hans> will (deservedly) have only a small market share because the Hans> apps won't all work on them. This is all pure marketing speak and economic theory. Show us the *technical* advantages, not just wishful thinking. Hans> Besides, there are enhancements which are simply compelling. Hans> You can write a dramatically better performance version control Hans> system with a much simpler design if the FS is atomic. Define atomic please, with state diagrams and clear examples. Hans> We have the performance lead. By next year we will be stable Hans> enough for mission critical servers, and then we start the Hans> serious semantic enhancements. I think you've got it backwards. Make your serious semantic enhancements first, then make it stable, then make it fast. Because when you change the semantics, you break all kinds of things and then it doesn't matter how fast you are. Hans> If you don't embrace progress, then you doom Linux to following Hans> behind, because the guys at Apple are pretty aggressive now that Hans> Jobs is back, and they WILL change the semantics, and they will Hans> do so in compelling ways, and Linux will be reduced to aping Hans> them when it should be leading them. Monkey see, Monkey do then. I'd like to point out another successful company which has extended the standard Unix namespace and that's 'Network Appliance' with it's .snapshot directory structure. It's a great idea and allows my users to restore files from snapshots without me having to think about it. But it still causes problems since when I use rsync to move data, I need to put in stuff like: rsync -az --exclude ".snapshot" --delete --delete-excluded <src> <dest> to make sure it doesn't descend into that directory of previous versions and try to copy them over as well. And of course now now other apps can use the .snapshot name. But what if other vendors aped (ook ook) this decided to use their own names? Who decides which gets priority? I think you need to go back and re-read Pike's paper on namespaces that you pointed to before and mull it over. And look at how simplicity is inherently powerful. If the design is too complicated, you're probably doing it wrong. John ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 14:24 ` John Stoffel @ 2004-09-02 15:36 ` Alan Cox 0 siblings, 0 replies; 1392+ messages in thread From: Alan Cox @ 2004-09-02 15:36 UTC (permalink / raw) To: John Stoffel Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Iau, 2004-09-02 at 15:24, John Stoffel wrote: > This is all pure marketing speak and economic theory. I wouldn't grace it with the label economic theory. Lets be honest about this.. its pure -research-. Research is good but don't screw around with Linux semantics until you've done the research, figured out how to also make it useful to real users and proved its a good idea despite the past evidence from things like MacOS Alan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 8:58 ` Hans Reiser ` (2 preceding siblings ...) 2004-09-02 14:24 ` John Stoffel @ 2004-09-02 15:32 ` Horst von Brand 2004-09-05 15:16 ` Tonnerre 2004-09-02 17:43 ` Linus Torvalds ` (2 subsequent siblings) 6 siblings, 1 reply; 1392+ messages in thread From: Horst von Brand @ 2004-09-02 15:32 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hans Reiser <reiser@namesys.com> said: > Linus Torvalds wrote: > > But _my_ point is, no user program is going to take _advantage_ of > >anything that only one filesystem on one system offers. > Apple does not have this problem.... ... because Apple customers are a captive lot... > and yes, the apps will take advantage of it, which is different from > depending on it. If you use the wrong fs you will lose some of the > features of the app. ... which is undesirable to say the very least for cross-platform Unix (even worse, Unix + Windows) applications, so applications won't ever use it (until Linux + super-duper filesystem is more than, say, 80% of the market). I.e., it won't happen soon enough for this to make sense right now. Perhaps in 13 more years... > For 30 years nothing much has happened in Unix filesystem semantics > because of sheer cowardice Careful threading there... calling respected people names without any proof isn't exactly great PR. > (excepting Clearcase, which priced itself > into a niche market). It is 25 years past time for someone to change > things. Your opinion. Some people disagree. > That someone will have first mover advantage, and the more > little semantic features possessed the more lure there will be to use it > which will increase market share which will lure more apps into depending > on it and in a few years the other filesystems will (deservedly) have > only a small market share because the apps won't all work on them. You seem to be blinded by the "success effect": You only ever see the ones who succeed, you don't see the masses who tried to be "first movers" and failed utterly. > Besides, there are enhancements which are simply compelling. You can > write a dramatically better performance version control system with a > much simpler design if the FS is atomic. Iff the performance of the FS is better _at doing what the system wants than the system itself_. That the CVS is simpler/faster is no use whatsoever if the underlying OS is more complex/slower. Besides, what is ideal for CVS isn't necesarily good for word processing or databases, and probably irrelevant or at least overkill for a lot of other applications, which all end up paying the costs. Concentrate on giving what _most_ applications want, make it possible for applications with weird requirements to do their job. > Our transaction manager > first draft was written by a version control guy, and he would probably > be happy to tell you how lack of atomicity other than rename makes > version control software design hideous. Right. But it doesn't make the design of text editing, or word processing, or mail handling, or... hideous, does it? Having a strong background in an area does put blinds on one's eyes. > We have the performance lead. I recently saw (preliminary) numbers directly contradicting this somewhere. > By next year we will be stable enough for > mission critical servers, and then we start the serious semantic > enhancements. By which the performance lead (if any) will do down the drain. There ain't no free lunch. > If you don't embrace progress, then you doom Linux to following behind, > because the guys at Apple are pretty aggressive now that Jobs is back, > and they WILL change the semantics, and they will do so in compelling > ways, and Linux will be reduced to aping them when it should be leading > them. Why on earth should Linux copy a niche system? By now Linux's marketshare is _much_ larger than Apple's... I'd say it is the other way around: Apple tries desperately to differentiate itself from MSFT (and traditional Unix + Linux) in order not to dissapear (and keep their grip on their users). If they are doing it right, only time will tell. But that is no reason to screw up Linux' strengths. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 15:32 ` Horst von Brand @ 2004-09-05 15:16 ` Tonnerre 0 siblings, 0 replies; 1392+ messages in thread From: Tonnerre @ 2004-09-05 15:16 UTC (permalink / raw) To: Horst von Brand; +Cc: linux-kernel, ReiserFS List [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain; charset=unknown-8bit, Size: 878 bytes --] Salut, On Thu, Sep 02, 2004 at 11:32:08AM -0400, Horst von Brand wrote: > Hans Reiser <reiser@namesys.com> said: > > Linus Torvalds wrote: > > > > But _my_ point is, no user program is going to take _advantage_ of > > >anything that only one filesystem on one system offers. > > > Apple does not have this problem.... > > ... because Apple customers are a captive lot... ça suffit! Stop that damn evangelization. I've seen lots of content MacOS/X users. Personally I use MacOS/X, NetBSD and Linux to develop my software, and I quite like all the three. I'm actively developing Linux and MacOS/X. Why I do that? Because I'm a technician. Linux users are no better people whatsoever. Restricting yourself to one OS is bad. I think most of the FS developers here will agree on that. Tonnerre (who's a technician, not an evangelist) [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 8:58 ` Hans Reiser ` (3 preceding siblings ...) 2004-09-02 15:32 ` Horst von Brand @ 2004-09-02 17:43 ` Linus Torvalds 2004-09-02 18:22 ` Martin J. Bligh 2004-09-15 17:23 ` Timothy Miller 6 siblings, 0 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-09-02 17:43 UTC (permalink / raw) To: Hans Reiser Cc: David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2 Sep 2004, Hans Reiser wrote: > > Apple does not have this problem.... Deep, Hans. Deep. You're ignoring the issue. You're flaming about not "embracing progress", when I was talking about how progress happens. You don't seem to understand _how_ people embrace progress, Hans. Railing at windmills isn't real smart. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 8:58 ` Hans Reiser ` (4 preceding siblings ...) 2004-09-02 17:43 ` Linus Torvalds @ 2004-09-02 18:22 ` Martin J. Bligh 2004-09-02 18:38 ` Christer Weinigel ` (2 more replies) 2004-09-15 17:23 ` Timothy Miller 6 siblings, 3 replies; 1392+ messages in thread From: Martin J. Bligh @ 2004-09-02 18:22 UTC (permalink / raw) To: Hans Reiser, Linus Torvalds Cc: David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > For 30 years nothing much has happened in Unix filesystem semantics > because of sheer cowardice Or because it works fine, and isn't broken. M. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 18:22 ` Martin J. Bligh @ 2004-09-02 18:38 ` Christer Weinigel 2004-09-03 13:16 ` Brian Beattie 2004-09-02 19:52 ` Steve Bergman 2004-09-03 0:41 ` David Masover 2 siblings, 1 reply; 1392+ messages in thread From: Christer Weinigel @ 2004-09-02 18:38 UTC (permalink / raw) To: Martin J. Bligh Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List "Martin J. Bligh" <mbligh@aracnet.com> writes: > > For 30 years nothing much has happened in Unix filesystem semantics > > because of sheer cowardice > > Or because it works fine, and isn't broken. I've heard the same argument a lot of times. People complaining that Unix is so seventies because it sticks to the old boring philosophy of everything is a file and that a file is a stream of bytes, nothing more. Modern operating systems such as VMS with basic database handling in the OS itself, or MacOS or NT with named streams is so much more modern. Why don't we get with the times? It may be because just because of the simplicity it's fairly easy to use, harder to break och does one thing well. If you want structured storage, use a database, on top of the low level primitives, or use multiple files in a directory. Why complicate things? /Christer -- "Just how much can I get away with and still go to heaven?" Freelance consultant specializing in device driver programming for Linux Christer Weinigel <christer@weinigel.se> http://www.weinigel.se ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 18:38 ` Christer Weinigel @ 2004-09-03 13:16 ` Brian Beattie 0 siblings, 0 replies; 1392+ messages in thread From: Brian Beattie @ 2004-09-03 13:16 UTC (permalink / raw) To: Christer Weinigel Cc: Martin J. Bligh, Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2004-09-02 at 14:38, Christer Weinigel wrote: > "Martin J. Bligh" <mbligh@aracnet.com> writes: > > > > For 30 years nothing much has happened in Unix filesystem semantics > > > because of sheer cowardice > > > > Or because it works fine, and isn't broken. > > I've heard the same argument a lot of times. People complaining that > Unix is so seventies because it sticks to the old boring philosophy of > everything is a file and that a file is a stream of bytes, nothing > more. Modern operating systems such as VMS with basic database > handling in the OS itself, or MacOS or NT with named streams is so > much more modern. Why don't we get with the times? Actually the handleing of special filesystem features in the OS is an OLD idea. VMS is actually not more modern than Unix, it is older than Unix owing much to earlier OS's such as RSX-11. It was a great insight on the part of Ritche and Thompson that the kernel should present a file as a byte stream and that the interpretation of the contents of a file is the province of the application, not the operating system. Now it maybe, that there are features that can only be provided by adding support in the kernel, but we should be very careful not to be distracted by what we see as new wizbang features. The filesystem as a database has been tried before, ISAM is a very old concept. > > It may be because just because of the simplicity it's fairly easy to > use, harder to break och does one thing well. If you want structured > storage, use a database, on top of the low level primitives, or use > multiple files in a directory. Why complicate things? > > /Christer -- Brian Beattie LFS12947 | "Honor isn't about making the right choices. beattie@beattie-home.net | It's about dealing with the consequences." www.beattie-home.net | -- Midori Koto ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 18:22 ` Martin J. Bligh 2004-09-02 18:38 ` Christer Weinigel @ 2004-09-02 19:52 ` Steve Bergman 2004-09-02 20:06 ` Spam ` (2 more replies) 2004-09-03 0:41 ` David Masover 2 siblings, 3 replies; 1392+ messages in thread From: Steve Bergman @ 2004-09-02 19:52 UTC (permalink / raw) To: Martin J. Bligh Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, reiserfs On Thu, 2004-09-02 at 11:22 -0700, Martin J. Bligh wrote: > > For 30 years nothing much has happened in Unix filesystem semantics > > because of sheer cowardice > > Or because it works fine, and isn't broken. OK. I'm not a kernel hacker. I'm not a crack C programmer. Nor am I a designer of filesystems. I'm just a guy that puts together and supports Linux systems for my customers. In following this thread, I may be missing huge chunks of concept. However, a few things are becoming clear to me: 1. The file as directory thing adds complexity that the administrator has to deal with. Symlinks are useful, but it's still aggravating to tar off a directory structure, take it somewhere, and then realize that all you have is links to something not in the archive because you didn't get your tar switches just right. Now we're talking about adding another set of "files which are not really files" to the semantics. More complexity. I'll take simplicity over some ivory tower ideal of "unified name space" any day. 2. The use of multiple streams within files by Linux apps would make Linux as cross-platform unfriendly as MS is trying to be. Say these features start getting used and you copy an OO.org document from a Linux box to a BSD box. It's broken. Of course, OO.org wouldn't use the streams in the first place because it would destroy their cross platform portability. So what's the point? No one who cares about cross platform portability can use it. Everyone who doesn't care about cross platform portability please raise your hand. 3. MS does require attributes and multiple streams, which makes these features important (even essential) to Samba, and Samba alone. Samba is important to Linux, so this can't be ignored. (Here I am implicitly assuming that Samba will need kernel support for this to do it right.) So it seems to me that the only real consideration is giving Samba what it needs without making the semantics one bit more complex than absolutely necessary. It might even be wise to discourage use of these ambiguous new objects by the casual application programmer. Then again, maybe I just have tunnel vision... -Steve Bergman ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 19:52 ` Steve Bergman @ 2004-09-02 20:06 ` Spam 2004-09-02 20:16 ` Martin J. Bligh ` (2 more replies) 2004-09-02 20:35 ` Erik Hensema 2004-09-03 0:35 ` David Masover 2 siblings, 3 replies; 1392+ messages in thread From: Spam @ 2004-09-02 20:06 UTC (permalink / raw) To: Steve Bergman Cc: Martin J. Bligh, Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, reiserfs > On Thu, 2004-09-02 at 11:22 -0700, Martin J. Bligh wrote: >> > For 30 years nothing much has happened in Unix filesystem semantics >> > because of sheer cowardice >> >> Or because it works fine, and isn't broken. > OK. I'm not a kernel hacker. I'm not a crack C programmer. Nor am I a > designer of filesystems. I'm just a guy that puts together and supports > Linux systems for my customers. > In following this thread, I may be missing huge chunks of concept. > However, a few things are becoming clear to me: > 1. The file as directory thing adds complexity that the administrator > has to deal with. Symlinks are useful, but it's still aggravating to > tar off a directory structure, take it somewhere, and then realize that > all you have is links to something not in the archive because you didn't > get your tar switches just right. Now we're talking about adding > another set of "files which are not really files" to the semantics. > More complexity. I'll take simplicity over some ivory tower ideal of > "unified name space" any day. Are you afraid to learn something new? ;) Just joking. But really, it doesn't have to be very difficult. The extra streams etc would just be saved as files. If tar is patched then it would be no problem and no extra stuff but perhaps a switch --save-metas. > 2. The use of multiple streams within files by Linux apps would make > Linux as cross-platform unfriendly as MS is trying to be. Say these > features start getting used and you copy an OO.org document from a Linux > box to a BSD box. It's broken. Of course, OO.org wouldn't use the > streams in the first place because it would destroy their cross platform > portability. So what's the point? No one who cares about cross > platform portability can use it. Everyone who doesn't care about cross > platform portability please raise your hand. Actually MacOS and Windows support multiple streams, only Linux doesn't. But of course there are BSD's etc too. I'd say to leave them behind. The file streams would make my day a lot easier. The idea to split up contents of OO.org files into streams is bad. But that doesn't make the file streams bad. I see many uses that would make my every day life easier. It isn't about cross plat form compatibility, but to add features that are useful and meta-data and file streams are. Also. No one forces you to use either meta-data or streams, just as no one forces you to use ACLs or other things. > 3. MS does require attributes and multiple streams, which makes these > features important (even essential) to Samba, and Samba alone. Samba is > important to Linux, so this can't be ignored. (Here I am implicitly > assuming that Samba will need kernel support for this to do it right.) I do not think the Samba would really require the streams support but it would certainly make life easier for Samba. Not to mention that these files would also be natively viewed on the Linux host. > So it seems to me that the only real consideration is giving Samba what > it needs without making the semantics one bit more complex than > absolutely necessary. It might even be wise to discourage use of these > ambiguous new objects by the casual application programmer. > Then again, maybe I just have tunnel vision... I would agree with tunnel vision. The kernel should provide the tools and options. Users and developers can then invent new things to use them. :) ~S > -Steve Bergman ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 20:06 ` Spam @ 2004-09-02 20:16 ` Martin J. Bligh 2004-09-03 4:07 ` Hacksaw 2004-09-03 8:49 ` Bernd Petrovitsch 2 siblings, 0 replies; 1392+ messages in thread From: Martin J. Bligh @ 2004-09-02 20:16 UTC (permalink / raw) To: Spam, Steve Bergman Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, reiserfs >> 1. The file as directory thing adds complexity that the administrator >> has to deal with. Symlinks are useful, but it's still aggravating to >> tar off a directory structure, take it somewhere, and then realize that >> all you have is links to something not in the archive because you didn't >> get your tar switches just right. Now we're talking about adding >> another set of "files which are not really files" to the semantics. >> More complexity. I'll take simplicity over some ivory tower ideal of >> "unified name space" any day. > > Are you afraid to learn something new? ;) Just joking. But really, > it doesn't have to be very difficult. The extra streams etc would > just be saved as files. If tar is patched then it would be no > problem and no extra stuff but perhaps a switch --save-metas. If they're saved as files, and the app has to be changed to use them anyway, then what's the point? Just change the app to use new files instead. > I would agree with tunnel vision. The kernel should provide the > tools and options. Users and developers can then invent new things > to use them. :) Ugh. Change for changes sake is not a good thing. There's enough real problems in the world without inventing random features. More complexity without gain is a Bad Thing (tm). I'm not saying streams is bad ... just that there don't seem to have been very many convincing (to me) arguments raised for it yet. The versioning stuff would be nice, IMHO, because the stuff mainly using it wouldn't need to be modified in many instances ... "vi /etc/configfile" would keep old copies for you (only the recovery tool would need to understand it). Saving icons seems easy enough to do with "foo.icon" as another file. M. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 20:06 ` Spam 2004-09-02 20:16 ` Martin J. Bligh @ 2004-09-03 4:07 ` Hacksaw 2004-09-03 8:49 ` Bernd Petrovitsch 2 siblings, 0 replies; 1392+ messages in thread From: Hacksaw @ 2004-09-03 4:07 UTC (permalink / raw) To: Spam; +Cc: linux-kernel >If tar is patched then it would be no >problem and no extra stuff but perhaps a switch --save-metas. Aaargh. How about a tar that recognizes when the filesystem has metas and saves/restores them by default, with a warning when it's losings data because it's restoring to a simpler filesystem? And maybe a switch --forget-metas. Thousands of sys-admins will thank you to not make it easier to lose data with the scripts they have already written. -- You are in a maze of twisty passages, all alike. Again. http://www.hacksaw.org -- http://www.privatecircus.com -- KB1FVD ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 20:06 ` Spam 2004-09-02 20:16 ` Martin J. Bligh 2004-09-03 4:07 ` Hacksaw @ 2004-09-03 8:49 ` Bernd Petrovitsch 2004-09-03 9:53 ` Spam 2 siblings, 1 reply; 1392+ messages in thread From: Bernd Petrovitsch @ 2004-09-03 8:49 UTC (permalink / raw) To: Spam Cc: Steve Bergman, Martin J. Bligh, Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, reiserfs On Thu, 2004-09-02 at 22:06 +0200, Spam wrote: [...] > The file streams would make my day a lot easier. The idea to > split up contents of OO.org files into streams is bad. But that > doesn't make the file streams bad. I see many uses that would make > my every day life easier. When can we expect patches? > It isn't about cross plat form compatibility, but to add features > that are useful and meta-data and file streams are. Putting everything (including web server and browser) in the kernel is not a good (or even desireable) target - perhaps that's the reason for Win* to be where there are. And it *is* done in user-space (ok, there more than one apprach and implementation but this may change. And you want to invent the next version.). > Also. No one forces you to use either meta-data or streams, just as > no one forces you to use ACLs or other things. Which is probably an argument against such features. How long do ACLs exist - as well as standard and in implementation? How many people and/or apps are using it? No more questions. > > 3. MS does require attributes and multiple streams, which makes these > > features important (even essential) to Samba, and Samba alone. Samba is > > important to Linux, so this can't be ignored. (Here I am implicitly > > assuming that Samba will need kernel support for this to do it right.) > > I do not think the Samba would really require the streams support > but it would certainly make life easier for Samba. Not to mention > that these files would also be natively viewed on the Linux host. It is not in the interest of MSFT that their aplication's files are viewable on other OSes. Expect incompatible changes to formats, streams and metadata contents (and surely patents on them - even if the idea is not new) if it starts to be really useful. It happended already in the past with the SMB protocol IIRC .... Bernd -- Firmix Software GmbH http://www.firmix.at/ mobil: +43 664 4416156 fax: +43 1 7890849-55 Embedded Linux Development and Services ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 8:49 ` Bernd Petrovitsch @ 2004-09-03 9:53 ` Spam 0 siblings, 0 replies; 1392+ messages in thread From: Spam @ 2004-09-03 9:53 UTC (permalink / raw) To: Bernd Petrovitsch Cc: Steve Bergman, Martin J. Bligh, Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, reiserfs > On Thu, 2004-09-02 at 22:06 +0200, Spam wrote: > [...] >> The file streams would make my day a lot easier. The idea to >> split up contents of OO.org files into streams is bad. But that >> doesn't make the file streams bad. I see many uses that would make >> my every day life easier. > When can we expect patches? >> It isn't about cross plat form compatibility, but to add features >> that are useful and meta-data and file streams are. > Putting everything (including web server and browser) in the kernel is > not a good (or even desireable) target - perhaps that's the reason for > Win* to be where there are. > And it *is* done in user-space (ok, there more than one apprach and > implementation but this may change. And you want to invent the next > version.). As I said before. It doesn't matter if it is in the kernel or user space. Not to me at least - as long as I could use extend the usage of my filesystem with meta-data, my own descriptions, perhaps streams, etc in a transparent way. I do not want to be confined to only use future Gnome apps. >> Also. No one forces you to use either meta-data or streams, just as >> no one forces you to use ACLs or other things. > Which is probably an argument against such features. How long do ACLs > exist - as well as standard and in implementation? > How many people and/or apps are using it? > No more questions. >> > 3. MS does require attributes and multiple streams, which makes these >> > features important (even essential) to Samba, and Samba alone. Samba is >> > important to Linux, so this can't be ignored. (Here I am implicitly >> > assuming that Samba will need kernel support for this to do it right.) >> >> I do not think the Samba would really require the streams support >> but it would certainly make life easier for Samba. Not to mention >> that these files would also be natively viewed on the Linux host. > It is not in the interest of MSFT that their aplication's files are > viewable on other OSes. Expect incompatible changes to formats, streams > and metadata contents (and surely patents on them - even if the idea is > not new) if it starts to be really useful. It happended already in the > past with the SMB protocol IIRC .... No perhaps. But it doesn't make a difference to the ideas of having plugins for reiser4 that can give such features. ~S > Bernd ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 19:52 ` Steve Bergman 2004-09-02 20:06 ` Spam @ 2004-09-02 20:35 ` Erik Hensema 2004-09-06 14:26 ` Tonnerre 2004-09-03 0:35 ` David Masover 2 siblings, 1 reply; 1392+ messages in thread From: Erik Hensema @ 2004-09-02 20:35 UTC (permalink / raw) To: linux-kernel Steve Bergman (steve@rueb.com) wrote: > However, a few things are becoming clear to me: > > 1. The file as directory thing adds complexity that the administrator > has to deal with. Symlinks are useful, but it's still aggravating to > tar off a directory structure, take it somewhere, and then realize that > all you have is links to something not in the archive because you didn't > get your tar switches just right. Now we're talking about adding > another set of "files which are not really files" to the semantics. > More complexity. I'll take simplicity over some ivory tower ideal of > "unified name space" any day. As with any addition of complexity we have to ask ourselves: do the advantages outweigh the disadvantages? Adding complexity into the kernel may simplify userspace. Adding the complexity of an IP layer into the kernel simplifies networking applications, for example (they won't have to talk IP to the kernel). In this case it's hard to imagine what advantages a multistream file will bring, simply because we aren't used to working with them. Or maybe there aren't any advantages at all. Point is: we don't know yet. Maybe we have to toy with the idea for some years before we really learn why multistream files are so great. I do realize that this isn't how Linux development is done. First we need a problem, then somebody sits down and codes a solution. With reiser4 we've got a solution but we haven't figured out what the problem is yet. > 2. The use of multiple streams within files by Linux apps would make > Linux as cross-platform unfriendly as MS is trying to be. Say these > features start getting used and you copy an OO.org document from a Linux > box to a BSD box. It's broken. Of course, OO.org wouldn't use the > streams in the first place because it would destroy their cross platform > portability. So what's the point? No one who cares about cross > platform portability can use it. Everyone who doesn't care about cross > platform portability please raise your hand. But do we really want to be the lowest common denominator? If you want to lead, you can't be compatible (in some cases). Also, if the reiser4 ideas catch on, they may spark the development of other, similar, filesystems. Maybe we'll end up with a ext4 in the 2.8 kernel... > 3. MS does require attributes and multiple streams, which makes these > features important (even essential) to Samba, and Samba alone. Samba is > important to Linux, so this can't be ignored. (Here I am implicitly > assuming that Samba will need kernel support for this to do it right.) > > So it seems to me that the only real consideration is giving Samba what > it needs without making the semantics one bit more complex than > absolutely necessary. It might even be wise to discourage use of these > ambiguous new objects by the casual application programmer. > > Then again, maybe I just have tunnel vision... Not tunnel vision. You're just looking at what we need *now*, and that happens to be exactly what we *have* now. One possible use for multistream files that I see, is storage of 'not so important' data alongside of the real data of a file. Think: the mimetype of a file; some metadata like author, keywords, etc. When you copy the file and lose all metadata except the file itself, no real harm is done. This could be used by the new mime system Gnome uses, for example. On regular filesystems, the mimetype of a file is determined by its extension. On reiser4(-like filesystems), the mimetype is determined by the 'mimetype' stream. This would clearly simplify the gnome userspace, as there is no need anymore for a seperate extension<->mimetype conversion anymore. It would also be compatible with other tools or desktop environements, we only have to agree on what stream we're going to use for what metadata. IMHO reiser4 is a toy we should give the Linux users to play with. Let's see what they can come up with. With the talented hackers in the community, it should be great. -- Erik Hensema <erik@hensema.net> (attempting to write english while drinking beer) ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 20:35 ` Erik Hensema @ 2004-09-06 14:26 ` Tonnerre 0 siblings, 0 replies; 1392+ messages in thread From: Tonnerre @ 2004-09-06 14:26 UTC (permalink / raw) To: Erik Hensema; +Cc: linux-kernel [-- Attachment #1: Type: text/plain, Size: 1175 bytes --] Salut, On Thu, Sep 02, 2004 at 08:35:58PM +0000, Erik Hensema wrote: > In this case it's hard to imagine what advantages a multistream file > will bring, simply because we aren't used to working with them. Or > maybe there aren't any advantages at all. Point is: we don't know > yet. Maybe we have to toy with the idea for some years before we > really learn why multistream files are so great. Thus I recommend doing the same thing everything does to get his stuff implemented. How did User Mode Linux get accepted into the kernel? In the beginning, people couldn't imagine any advantage of it. Then, someone started coding it and provided patches. People tried it, improved it (important! It got to work on many platforms while being out of the main tree!) and some day we noticed that it was really cool for kernel hacking and virtualization. So when it approached a mature level where we finally saw it was cool, it went into the mainline kernel, where it is now. For multistream files, what is that development cycle going to be? Get the idea, flame, flame, flame... ? Code talks, bullshit walks. Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 19:52 ` Steve Bergman 2004-09-02 20:06 ` Spam 2004-09-02 20:35 ` Erik Hensema @ 2004-09-03 0:35 ` David Masover 2004-09-03 6:35 ` Hans Reiser 2 siblings, 1 reply; 1392+ messages in thread From: David Masover @ 2004-09-03 0:35 UTC (permalink / raw) To: Steve Bergman Cc: Martin J. Bligh, Hans Reiser, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, reiserfs -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Steve Bergman wrote: [...] | In following this thread, I may be missing huge chunks of concept. You are, but don't worry. You aren't the only one. (Before I get assaulted for this, let me say that there are people on this list smarter than me and I miss concept chunks quite frequently.) | 1. The file as directory thing adds complexity that the administrator | has to deal with. Symlinks are useful, but it's still aggravating to | tar off a directory structure, take it somewhere, and then realize that | all you have is links to something not in the archive because you didn't | get your tar switches just right. Now we're talking about adding | another set of "files which are not really files" to the semantics. | More complexity. I'll take simplicity over some ivory tower ideal of | "unified name space" any day. You want simplicity? How does this strike you: scp file/serialize daemon@bsd:~/file.img You don't have to worry about "file is not a file". You don't have to worry about fifos or device nodes. You don't have to worry about files in files. In fact, you don't have to worry about any new kind of file added, ever. | 2. The use of multiple streams within files by Linux apps would make | Linux as cross-platform unfriendly as MS is trying to be. Say these The use of ext3 as a filesystem isn't cross-platform. Every disk-write is platform-specific! We should all be using captive-ntfs instead! Seriously, this isn't hard -- it's called a "library". And anything as bloated as OO.org has got to already be modular enough to add this sort of thing. | features start getting used and you copy an OO.org document from a Linux | box to a BSD box. It's broken. Of course, OO.org wouldn't use the | streams in the first place because it would destroy their cross platform If that was so, OO.org wouldn't use a GUI, because that isn't cross-platform. They'd use a purely commandline interface, because that is universal. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTe8PHgHNmZLgCUhAQKiNQ//Zo6BLeGK6/GBbYzcgyp51Ra4fRf2xoVo 0aOve2vinVl+RRzFGWSa66p+/rHd339BhxTPYhlLkm9SWbU9Et80J0R/bySYtc17 RmUHhmukSEXhJx+py7Vc99jXr03zr2fNaA3UBIfpMiAlj1UB2yOls5iwJgqfPVnX +YwYcVRfZXc3KWqH9SGcY9q+NX38wXoQh3rbeJ0OEzaIxnvPWlMvUFEN4OF92cub on+n0h+JKptwtXOhLh9HaKAzG9fDhy4tJwmN3GIemSzKpEZ0ZGS4+Pz4dcnRXP+S C6cMxpsmCnm5WVzmeoyBYBY28Fmnji00ShvxDGRFYOp48RPFhNaoMbVhLkQFoYfX a5k3iedhbjaWOLmpfNGaDNdpKFtqD3LJO1xOSkXz3sndAiLW9ArKhuu2+63ASAQy wlFknP09k13UHzD2iL5kPUPzy0A/BLqZPSpX4tXnF0xcSp4X8RoLnyu8BmrqaTjG fi7FkEsoVEB7BWslNNtKrH596sDhWCP+BQdkAYqOBh0fzSIJ+0uyo2nJRw3d8FEi /ynNVUqZEN8Xj08WHI+ucwUmRpTnh6spwXZC42K2tokAQa+95nSTegx3uNkypcGU l12s+BtIHFB25BKDaBReyGoBqUj0sdFvoHVkRNc/3DNY6MfRHV3rdgjz5NieT4gH uwMwLNstv8c= =O3r8 -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 0:35 ` David Masover @ 2004-09-03 6:35 ` Hans Reiser 2004-09-03 16:02 ` Chris Mason 2004-09-03 17:26 ` Horst von Brand 0 siblings, 2 replies; 1392+ messages in thread From: Hans Reiser @ 2004-09-03 6:35 UTC (permalink / raw) To: David Masover Cc: Steve Bergman, Martin J. Bligh, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, reiserfs David Masover wrote: > > The use of ext3 as a filesystem isn't cross-platform. Every disk-write > is platform-specific! We should all be using captive-ntfs instead! ;-) All this stuff about how no filesystem should be allowed to have semantic features others don't, it seems very Bolshevist to me. Let Linux have an ecosystem with a diverse ecology of filesystems, and the features that work will reproduce to other filesystems. I thought that was the Linus way? If not, why did I spend 10 years laying the storage layer groundwork for semantic enhancements when I could have taken that job at Sun as filesystems architect and made a lot more money? I want to tinker. Let me play in my sandbox, and if you don't like what I do, don't imitate it..... I think there are plenty of users who like reiser4 though.... Linus, trying to outguess someone who has spent 2 decades studying namespace design as to what will be useful to users is risky. Look at reiser4's performance, see if it obsoletes V3, and if it does then let me play a bit. Objecting on the grounds that it causes VFS bugs is reasonable, but I answered those questions and you did not respond (I can resend if asked). If you really really don't like what we do to VFS, well, we can confine ourselves to sys_reiser4(), but that is only a last resort from my view. Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 6:35 ` Hans Reiser @ 2004-09-03 16:02 ` Chris Mason 2004-09-03 17:26 ` Horst von Brand 1 sibling, 0 replies; 1392+ messages in thread From: Chris Mason @ 2004-09-03 16:02 UTC (permalink / raw) To: Hans Reiser Cc: David Masover, Steve Bergman, Martin J. Bligh, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, reiserfs On Fri, 2004-09-03 at 02:35, Hans Reiser wrote: > All this stuff about how no filesystem should be allowed to have > semantic features others don't, it seems very Bolshevist to me. > > Let Linux have an ecosystem with a diverse ecology of filesystems, and > the features that work will reproduce to other filesystems. I thought > that was the Linus way? > There's no reason you can't have semantics the other filesystems don't. There's also no reason you shouldn't continue researching in any direction you choose. When you submit to the kernel (even just -mm), call the FS stable and suggest people start broadly using it, you're telling them the parts you have submitted are in fact fixed and reliable. This implies you are done changing the semantics you have submitted, which also implies you would like applications to start seriously using it. Those application writers will want other filesystems to be able to support these new semantics, which means they should have a race free framework vetted by the kernel community and available for reuse by the other filesystems. In other words, the semantics should be in the VFS, without any requirement other FS's implement them, but the possibility for every FS to implement them. If you don't want the framework to be agreed on by the community and available for other implementations, you either haven't stabilized the semantics, or you don't want anyone else using them. Either way, it precludes inclusion in the kernel. There are simply higher expectations for a filesystem in these cases, and a higher burden for stable semantics and interfaces. This doesn't mean new semantics are impossible, it just means that constantly changing semantics of the filesystem core are bad for users and applications. They belong in patches that live outside the kernel until they are sufficiently researched to justify their inclusion. The biggest victim of this discussion is the non-semantic part of reiser4. You've done a huge amount of research that is being held back because you haven't separated the semantic research from the storage layer areas. If the storage layers are ready now, please please just submit them. -chris ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 6:35 ` Hans Reiser 2004-09-03 16:02 ` Chris Mason @ 2004-09-03 17:26 ` Horst von Brand 1 sibling, 0 replies; 1392+ messages in thread From: Horst von Brand @ 2004-09-03 17:26 UTC (permalink / raw) To: Hans Reiser Cc: David Masover, Steve Bergman, Martin J. Bligh, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, reiserfs Hans Reiser <reiser@namesys.com> said: > David Masover wrote: > > The use of ext3 as a filesystem isn't cross-platform. Every disk-write > > is platform-specific! We should all be using captive-ntfs instead! > ;-) > > All this stuff about how no filesystem should be allowed to have > semantic features others don't, it seems very Bolshevist to me. Haven seen that. Just requests that _experimental_ features be prototyped in ReiserFS, and _if_ they show they are worth the hassle, _then_ design a nice VFS interface on them. > Let Linux have an ecosystem with a diverse ecology of filesystems, and > the features that work will reproduce to other filesystems. I thought > that was the Linus way? It is. > If not, why did I spend 10 years laying the storage layer groundwork for > semantic enhancements when I could have taken that job at Sun as > filesystems architect and made a lot more money? Dunno. That _you_ choose to do this is no reason _others_ will have to take the results over lock, stock and barrel. Linux' development works because there is a lot of "wasted" effort on alternatives that don't pan out. Many don't ever leave the drawing board, others (like the (now defunct) Xiafs, devfs, and volume management code) went far, even got fully implemented, only to be scrapped later. It might just be ReiserFS' fate too.. > I want to tinker. Let me play in my sandbox, and if you don't like what > I do, don't imitate it..... I think there are plenty of users who like > reiser4 though.... Go tinker and play. But don't aggravate non-ReiserFS-users, please. Come up with solutions to the issues Al Viro and Linus raised. > Linus, trying to outguess someone who has spent 2 decades studying > namespace design as to what will be useful to users is risky. I'd guess LKML adds up to a few thousand years mulling over this, so this isn't a great point. > Look at > reiser4's performance, see if it obsoletes V3, and if it does then let > me play a bit. I'd say it is a free world. > Objecting on the grounds that it causes VFS bugs is reasonable, but I > answered those questions and you did not respond (I can resend if > asked). If you really really don't like what we do to VFS, well, we can > confine ourselves to sys_reiser4(), but that is only a last resort from > my view. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 18:22 ` Martin J. Bligh 2004-09-02 18:38 ` Christer Weinigel 2004-09-02 19:52 ` Steve Bergman @ 2004-09-03 0:41 ` David Masover 2004-09-03 0:46 ` Spam 2 siblings, 1 reply; 1392+ messages in thread From: David Masover @ 2004-09-03 0:41 UTC (permalink / raw) To: Martin J. Bligh Cc: Hans Reiser, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Martin J. Bligh wrote: |>For 30 years nothing much has happened in Unix filesystem semantics |>because of sheer cowardice | | | Or because it works fine, and isn't broken. Ok, maybe it wasn't cowardice. Maybe it was laziness. Because cowardice tells me that data corruption is bad. Data corruption kills kittens. End data corruption now! Unfortunately, the only way to allow files to survive a crash is to implement transactions. The simplest way is to do it in the filesystem and to export an interface to userland. This is what Hans is doing, though AFAIK it's not done yet. Laziness tells me that I should just use Windows anyway, because "it works". Switching to Linux requires training, porting apps requires work, and using Linux instead of Windows is a hell of a lot more work than just patching programs to do something like + start_transaction() ~ write() ~ blah() ~ write() + end_transaction() How'd you like to be able to tell users "If a file gets corrupted, it's either a bug in OpenOffice or your hardware eats data." -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTe9pngHNmZLgCUhAQLTZQ/5AZq31sPKOWB/eI4WtVhM76V2+pmQdfNk aZkPtwhZlVrGDJscp45OVNthwMiSskD07om4obUMac79BCmA5805clZ7i0X1uzQB SecyX/7XITWz3iwF4VIaxJgDoo7YPFfbpWS0NTmhDJPt6h5Z+goEccLdTABj5kVk gbex4xNgXzPrnopvjnWx+K2K3ydvXiX+8bWV200F9S+j0uCObcfSZ357uQPqlReP gKQhEs0pCcoQQua81rXJfUhP8hCbegCpUmhJNI0MwoTUvHTnBU/+99+cCsTLR4gN 1XlbS9SIaaBXG7/AV9L7iQO5GF1t8mTMSe163EOzZ7ypLDb2km5e7zZ7t2DrA6SE Jgd0xT3cHjDgp1b3qhdeWYvt0QVfz5CGaiHukkT4pEOpsrM09BOhDhY72B74swuV byDIJ+y4X3J08i/a0zt15hBlVz1FUq7ac7doZZDt3orj93t3B/y4Xpb7tgzVGoWP T5AkR+uS5dGRUoMSI/btnPIK4ERlhbwFzCB5lgkas5kjp4dnawIKAk1wxPdhvEle g3SsNqSfBsA9hs6tzVw341GaWj6AWV7oOfg3j0c9I+7nt8SX5UeuTCwBWtCuwd/H +nnBulusiMnAm3tzQxq8aMpSVAqsZDywjgXSler1mNik4Ow5I5JdXl+Bf2ZWHF+G VHk4JVxMt/c= =3z3Z -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 0:41 ` David Masover @ 2004-09-03 0:46 ` Spam 2004-09-03 1:34 ` David Masover 0 siblings, 1 reply; 1392+ messages in thread From: Spam @ 2004-09-03 0:46 UTC (permalink / raw) To: David Masover Cc: Martin J. Bligh, Hans Reiser, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > Martin J. Bligh wrote: |>>For 30 years nothing much has happened in Unix filesystem semantics |>>because of sheer cowardice > | > | > | Or because it works fine, and isn't broken. > Ok, maybe it wasn't cowardice. Maybe it was laziness. > Because cowardice tells me that data corruption is bad. Data corruption > kills kittens. End data corruption now! > Unfortunately, the only way to allow files to survive a crash is to > implement transactions. The simplest way is to do it in the filesystem > and to export an interface to userland. This is what Hans is doing, > though AFAIK it's not done yet. I thought reiser4 had its journaling and atomic commits. Am I mistaken? I run reiser4 as primary fs on my test systems and it seem to work as expected. > Laziness tells me that I should just use Windows anyway, because "it > works". Switching to Linux requires training, porting apps requires > work, and using Linux instead of Windows is a hell of a lot more work > than just patching programs to do something like > + start_transaction() > ~ write() > ~ blah() > ~ write() > + end_transaction() > How'd you like to be able to tell users "If a file gets corrupted, it's > either a bug in OpenOffice or your hardware eats data." > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.2.4 (GNU/Linux) > Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org > iQIVAwUBQTe9pngHNmZLgCUhAQLTZQ/5AZq31sPKOWB/eI4WtVhM76V2+pmQdfNk > aZkPtwhZlVrGDJscp45OVNthwMiSskD07om4obUMac79BCmA5805clZ7i0X1uzQB > SecyX/7XITWz3iwF4VIaxJgDoo7YPFfbpWS0NTmhDJPt6h5Z+goEccLdTABj5kVk > gbex4xNgXzPrnopvjnWx+K2K3ydvXiX+8bWV200F9S+j0uCObcfSZ357uQPqlReP > gKQhEs0pCcoQQua81rXJfUhP8hCbegCpUmhJNI0MwoTUvHTnBU/+99+cCsTLR4gN > 1XlbS9SIaaBXG7/AV9L7iQO5GF1t8mTMSe163EOzZ7ypLDb2km5e7zZ7t2DrA6SE > Jgd0xT3cHjDgp1b3qhdeWYvt0QVfz5CGaiHukkT4pEOpsrM09BOhDhY72B74swuV > byDIJ+y4X3J08i/a0zt15hBlVz1FUq7ac7doZZDt3orj93t3B/y4Xpb7tgzVGoWP > T5AkR+uS5dGRUoMSI/btnPIK4ERlhbwFzCB5lgkas5kjp4dnawIKAk1wxPdhvEle > g3SsNqSfBsA9hs6tzVw341GaWj6AWV7oOfg3j0c9I+7nt8SX5UeuTCwBWtCuwd/H > +nnBulusiMnAm3tzQxq8aMpSVAqsZDywjgXSler1mNik4Ow5I5JdXl+Bf2ZWHF+G > VHk4JVxMt/c= > =3z3Z > -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 0:46 ` Spam @ 2004-09-03 1:34 ` David Masover 2004-09-03 5:59 ` Hans Reiser 0 siblings, 1 reply; 1392+ messages in thread From: David Masover @ 2004-09-03 1:34 UTC (permalink / raw) To: Spam Cc: Martin J. Bligh, Hans Reiser, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Spam wrote: [...] |> I thought reiser4 had its journaling and atomic commits. Am I |> mistaken? I run reiser4 as primary fs on my test systems and it seem |> to work as expected. consider this: save_file () { write() /* what if the write flushes halfway through * then crashes? */ blah() /* what if "blah" crashes? */ write() } Some apps need consistency across multiple files, but we don't even have it on a single file. You need a new interface to do that. As you can see, reiser4 has absolutely no way of knowing, anywhere in the above code, when you're done writing -- and when the file is consistent. AFAIK, all that has to be done now for this to work is for them to finish the userland interface to the journalling and atomic commits that already exist for kernel space. But so far, all that is truly atomic is metadata operations -- chmod, mv, mkdir, touch, and rm/rmdir are all atomic, so long as you only use them on a single file/dir. But this has been true in reiserfs3, xfs, ext3, and others. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTfKF3gHNmZLgCUhAQKp3Q//WcijAx8VVdKjJFHENZvrA2P3/wjdCLIn OeIGi9VxbcQhS1cBM+BU6nBN6wz8S59o9hEM/F8L5oDxto8YW6sfyceoanbtVkLo r7k4i39A4Ao1UQFzYEzxWzgCA607WvGBboNN1J/5pYe+bWATWBw+oQgc6XikQofQ QwFX+0KxEDUva5Xkei/wBNhvpSveFBjAm/t79nVkANOVLji2KYALtewps+7QDiGc PNwfjSUcAbNgXrvuhwpocFgAGEhgS/Y7ANR4DqFAiUseDwY6WgkZqhTaxC8t9ZzF /Hd82sEb/J+hGpMfrTIe+D5VYhnjX+bPPs/U0OcfoH4CfkgOFgGOeuzlWg3xeTyX vgtgYGwuGQuWux7WNte0GOCNxfaFDE8lLGexdQYLkdVCSsLmzACA17H1bQdeswgH UnWL3oBoBx5BcdpkW1y1ZeoJv4xInazkzeU3GtEOd50tlJmOXmdjRNfj5LTrTh+1 CUpEKVvSjQ8MY70uQW6XKEUAkCYyBmSVeMay5SSQS/1Q0ISLEG3gMqPdsL2BOBO/ LkkQKjLe3LQS913wUtibL1CCfJCS/BrDswNCMsboivcg1SnwY+xJb9ffFMW0blBL kQ0B1AGZCjWHDfxXKIlC31PJYKzgE5vaKZ6+INj8mG8t5PzqgikBDQStvsXqczJq VgvvOVCjvho= =I+Xw -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 1:34 ` David Masover @ 2004-09-03 5:59 ` Hans Reiser 0 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-09-03 5:59 UTC (permalink / raw) To: David Masover Cc: Spam, Martin J. Bligh, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List David Masover wrote: > Spam wrote: > [...] > |> I thought reiser4 had its journaling and atomic commits. Am I > |> mistaken? I run reiser4 as primary fs on my test systems and it seem > |> to work as expected. > > consider this: > > save_file () { > write() /* what if the write flushes halfway through > * then crashes? reiser4 does protect from this. reiserfs v3 does not. > */ > > blah() /* what if "blah" crashes? */ > write() > } > > Some apps need consistency across multiple files, but we don't even have > it on a single file. You need a new interface to do that. As you can > see, reiser4 has absolutely no way of knowing, anywhere in the above > code, when you're done writing -- and when the file is consistent. > > AFAIK, all that has to be done now for this to work is for them to > finish the userland interface to the journalling and atomic commits that > already exist for kernel space. But so far, all that is truly atomic is > metadata operations -- chmod, mv, mkdir, touch, and rm/rmdir are all > atomic, so long as you only use them on a single file/dir. But this has > been true in reiserfs3, xfs, ext3, and others. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 8:58 ` Hans Reiser ` (5 preceding siblings ...) 2004-09-02 18:22 ` Martin J. Bligh @ 2004-09-15 17:23 ` Timothy Miller 2004-09-16 8:29 ` Helge Hafting 6 siblings, 1 reply; 1392+ messages in thread From: Timothy Miller @ 2004-09-15 17:23 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hey, you know how device nodes have a bit set, indicating that they're device nodes and not regular files? Can a set of such properties be defined for reiser4 metadata properties? Like a "metadata" bit so you can distinguish (if you wish) between regular files and metadata objects, and in addition an "archivable metadata" bit which indicates that the given piece of metadata is not automatically generated and should be archived during backup (some manually-generated metadata which does not need to be backed up will not have this bit set -- perhaps add another flag indicating that it's not automatic but unnecessary to archive). ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-15 17:23 ` Timothy Miller @ 2004-09-16 8:29 ` Helge Hafting 0 siblings, 0 replies; 1392+ messages in thread From: Helge Hafting @ 2004-09-16 8:29 UTC (permalink / raw) To: Timothy Miller Cc: Hans Reiser, Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Timothy Miller wrote: > > Hey, you know how device nodes have a bit set, indicating that they're > device nodes and not regular files? Can a set of such properties be > defined for reiser4 metadata properties? > > Like a "metadata" bit so you can distinguish (if you wish) between > regular files and metadata objects, and in addition an "archivable > metadata" bit which indicates that the given piece of metadata is not > automatically generated and should be archived during backup (some > manually-generated metadata which does not need to be backed up will > not have this bit set -- perhaps add another flag indicating that it's > not automatic but unnecessary to archive). Interesting idea, particularly for autogenerated metadata. Some metadata, like thumbnails and icons might as well be ordinary files though. (Like some icon and want to use it as background bitmap? Just copy it, because it is a plain file. Or want to use it as clipart - fine, the word processor supports bitmap files. . .) This is a case where something is "metadata" only because it is used as such, not because it has to be. File-as-dir is an interesting concept for such cases, because it is convenient to group the "metadata" with the file, have it move with the file and so on. But still have this metadata accessible as a file because it isn't all that special. Helge Hafting ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 7:36 ` Linus Torvalds 2004-09-02 8:58 ` Hans Reiser @ 2004-09-02 9:04 ` Hans Reiser 2004-09-02 12:50 ` James Bruce 2004-09-03 1:11 ` David Masover 3 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-09-02 9:04 UTC (permalink / raw) To: Linus Torvalds Cc: David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Linus Torvalds wrote: > >But _my_ point is, no user program is going to take _advantage_ of >anything that only one filesystem on one system offers. > >So there's no point. > >I > Filesystem diversity will only be crippling for Linux if you choose for it to be so. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 7:36 ` Linus Torvalds 2004-09-02 8:58 ` Hans Reiser 2004-09-02 9:04 ` Hans Reiser @ 2004-09-02 12:50 ` James Bruce 2004-09-02 17:50 ` Linus Torvalds 2004-09-05 15:12 ` Tonnerre 2004-09-03 1:11 ` David Masover 3 siblings, 2 replies; 1392+ messages in thread From: James Bruce @ 2004-09-02 12:50 UTC (permalink / raw) To: Linus Torvalds Cc: Hans Reiser, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Linus Torvalds wrote: >... >But _my_ point is, no user program is going to take _advantage_ of >anything that only one filesystem on one system offers. > >So there's no point. > >It's much saner (from _any_ app standpoint) to roll their own database in >user space - that way it just works. > >In other words, nobody is really ever going to take advantage of this. >This is _not_ how technical advanncement happens. The way you get people >to take advantage of something is to have a nice gradual ramp-up, not a >sudden new feature that they can't realistically use. >... > Sure, but there are plenty of existing interfaces that you could emulate. One could make a small library to use a transactional filesystem to implement the Berkely DB interface (libdb) for example. So on filesystems without such support your app could use the regular userspace database, but on a transactional filesystem it'd just use regular files, which would simplify database management and likely increase performance over the userspace-only version (libdb is pretty slow). In terms of functionality it'd just be a drop-in replacement, just like math libraries that use MMX/SSE when available. - Jim Bruce ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 12:50 ` James Bruce @ 2004-09-02 17:50 ` Linus Torvalds 2004-09-05 15:12 ` Tonnerre 1 sibling, 0 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-09-02 17:50 UTC (permalink / raw) To: James Bruce Cc: Hans Reiser, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2 Sep 2004, James Bruce wrote: > Linus Torvalds wrote: > > >In other words, nobody is really ever going to take advantage of this. > >This is _not_ how technical advanncement happens. The way you get people > >to take advantage of something is to have a nice gradual ramp-up, not a > >sudden new feature that they can't realistically use. > > Sure, but there are plenty of existing interfaces that you could > emulate. Absolutely. We should look at them, and whether they solve any issues. I'm not saying that Hans would have to make up a new interface. Quite the reverse. One interface I suggested to use for attributes (and everybody hates ;) was the existing one from Solaris. Similarly, there are probably perfectly fine interfaces for the issue of transactions.. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 12:50 ` James Bruce 2004-09-02 17:50 ` Linus Torvalds @ 2004-09-05 15:12 ` Tonnerre 1 sibling, 0 replies; 1392+ messages in thread From: Tonnerre @ 2004-09-05 15:12 UTC (permalink / raw) To: James Bruce Cc: Linus Torvalds, Hans Reiser, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 222 bytes --] Salut, On Thu, Sep 02, 2004 at 08:50:10AM -0400, James Bruce wrote: > filesystem to implement the Berkely DB interface (libdb) for example. Problem is that BerkeleyDB changes its API all the time. Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 7:36 ` Linus Torvalds ` (2 preceding siblings ...) 2004-09-02 12:50 ` James Bruce @ 2004-09-03 1:11 ` David Masover 2004-09-03 1:35 ` Valdis.Kletnieks 2004-09-03 5:57 ` Hans Reiser 3 siblings, 2 replies; 1392+ messages in thread From: David Masover @ 2004-09-03 1:11 UTC (permalink / raw) To: Linus Torvalds Cc: Hans Reiser, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Linus Torvalds wrote: [...] | It's much saner (from _any_ app standpoint) to roll their own database in | user space - that way it just works. Then why do we have apps which use Windows' Internet Options where it can, and otherwise ask the user for proxy settings? | | In other words, nobody is really ever going to take advantage of this. | This is _not_ how technical advanncement happens. The way you get people | to take advantage of something is to have a nice gradual ramp-up, not a | sudden new feature that they can't realistically use. | | So before you do transactions in the filesystem, you have to be able to do | them in user space - and then make the filesystem version be _compatible_ | with that user space library. That way you can get people to use the | library ("hey, it works anywhere") and then you can get them to use your | filesystem ("hey, if you use our super-duper filesystem, you can do what | you are already doing five times faster"). Why not: do transactions in the filesystem write a library to talk to that filesystem have the library do emulation on other systems in that order. | See? You're starting at the wrong end. Give me a portable interface to use | _first_. Then do transactions in the filesystem. If you're going to have to do both the portable interface and the fs support, does it really matter which order you do them in? Again, this is curiosity, not sarcasm. Here my reasons for wanting the fs support first: You don't always know exactly how the filesystem transactions will work. ~ You don't know until it's done whether you'll deviate from your design, cut certain features, replace them with others... And if you write the portable library first, and it ends up supporting features a, b, and c, while the kernel supports features b, c, and d, which obsoletes a, you have to add some features and remove some features from the portable library to make it sane. It'd be like writing OpenGL entirely in software, before hardware accelerators work, and at the last minute have to change the library to use triangles instead of splines. What's more, every single app now has to be rewritten or use the library's emulation -- that's assuming many apps would consider using a 3D library that doesn't have hardware accel support yet. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTfEsHgHNmZLgCUhAQK1ORAAiSQ6UHSYc/PPMMOv2VjTkSuK2MgPylKP d2jEitIuvWx01xAa++CLcUS6ncaiLuFYDA7TTHjhKYKAj+zuNN9OTI9EMGc6D+Ua lFcbcRFapqbT9HPrUnGPM8VQIR8UrBt54hRAGm0/jOCc8DKLa3d6xnRfqmKQhnkv w3ut4FiNo0GCd19hF82NxEA1YtwbOIgOIUxVlbADGI9QT/d+I3ZMA8QUNCrgrudn e9g7o0yTa2c16t/rrqkiYCqSHIhCgY8ZH8VQzG4uYSjU+qYS+QbpJakD82EyuF/k 4T4jyJOqqSv6tLwE1j2y0gteXDYOdmfda/TpDeBSaRvGf+SmNxfSDFL448B2QJgA e3V0HZHVVums1Vl6kX4pBaWM01V3n9BGeLaF2S9XSP7WCSWgW7t4ONRSLE3Bf2mm 1e5cMJ63mrtImb8Jyd8GhyCQ8/tkGnEYOIAHL3laGOjuc2ARBeyqmWrGKJsQ8iwu i9NsyukQC3NP2FY1nBSmKs9J2bWtixqk5OXpzupkqBkbpeKL1H/b9Mxpnn5+Aomi PjI/5i4QcaHMNBggxCG3B5zGJIWmvMkmZmHrHxvbLeENGvS5fiZ0Kscq7PL0FnE0 /2etDgjg65I66SqguXMzvtZdQ0hYgdwvoR83nZymHMSLYrFJ8FOxuB7kYQBeKZnu QpUz0UDTUZA= =jVvF -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 1:11 ` David Masover @ 2004-09-03 1:35 ` Valdis.Kletnieks 2004-09-03 5:57 ` Hans Reiser 1 sibling, 0 replies; 1392+ messages in thread From: Valdis.Kletnieks @ 2004-09-03 1:35 UTC (permalink / raw) To: David Masover Cc: Linus Torvalds, Hans Reiser, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 448 bytes --] On Thu, 02 Sep 2004 20:11:13 CDT, David Masover said: > It'd be like writing OpenGL entirely in software, before hardware > accelerators work, and at the last minute have to change the library to > use triangles instead of splines. I expect that SGI did a software-only version of IrisGL first, so they could figure out what the hardware accelerators needed to support. And even then, the API for IrisGL got modified when it became OpenGL..... [-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 1:11 ` David Masover 2004-09-03 1:35 ` Valdis.Kletnieks @ 2004-09-03 5:57 ` Hans Reiser 1 sibling, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-09-03 5:57 UTC (permalink / raw) To: David Masover Cc: Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List The reason for supporting transactions in the kernel is that there is a whole slew of security bugs that have their origins in fs semantics being non-transactional. rename does not cut it as a transactions API. rename results in horribly performing apps that engage in contortions that cost a lot of programmer time to code them to stuff a transaction into a file that can be renamed. Most of the time programmers who should use a transaction/rename are just too busy to do it, and they take the risk that the user will lose data and be less secure rather than write the code to do it right. When they do do it, because it is complex code, they often screw it up. That said, there is a whole slew of reasons for not supporting transactions in the kernel: * the history of the literature stars transactional filesystems that paid too high a cost in performance * isolation is very tricky to do without killing performance * transactions which are kept open too long are a problem * avoiding problems with locking shared data structures long enough to cause painful exclusion of things that should be able to run in parallel just fine Reiser4, by being very modest in its ambitions, manages to not pay a performance cost. We say, isolation and rollback are for user space to manage until we can find a clever implementation, if such an implementation can even be found. Even if such an implementation can be found, there are plenty of cases where special case knowledge of what the app is doing can improve performance a lot, and thus for that app it will belong in user space. All reiser4 does, is allow you to guarantee (using sys_reiser4) that a limited size set of fs operations will be fused into one atom together, and therefor will either all survive a crash or none will survive. This does however meet a lot of apps needs. Hans David Masover wrote: > Linus Torvalds wrote: > [...] > | It's much saner (from _any_ app standpoint) to roll their own > database in > | user space - that way it just works. > > Then why do we have apps which use Windows' Internet Options where it > can, and otherwise ask the user for proxy settings? > > | > | In other words, nobody is really ever going to take advantage of this. > | This is _not_ how technical advanncement happens. The way you get people > | to take advantage of something is to have a nice gradual ramp-up, not a > | sudden new feature that they can't realistically use. > | > | So before you do transactions in the filesystem, you have to be able > to do > | them in user space - and then make the filesystem version be > _compatible_ > | with that user space library. That way you can get people to use the > | library ("hey, it works anywhere") and then you can get them to use your > | filesystem ("hey, if you use our super-duper filesystem, you can do what > | you are already doing five times faster"). > > Why not: > do transactions in the filesystem > write a library to talk to that filesystem > have the library do emulation on other systems > > in that order. > > | See? You're starting at the wrong end. Give me a portable interface to > use > | _first_. Then do transactions in the filesystem. > > If you're going to have to do both the portable interface and the fs > support, does it really matter which order you do them in? > > Again, this is curiosity, not sarcasm. Here my reasons for wanting the > fs support first: > > You don't always know exactly how the filesystem transactions will work. > ~ You don't know until it's done whether you'll deviate from your design, > cut certain features, replace them with others... > > And if you write the portable library first, and it ends up supporting > features a, b, and c, while the kernel supports features b, c, and d, > which obsoletes a, you have to add some features and remove some > features from the portable library to make it sane. > > It'd be like writing OpenGL entirely in software, before hardware > accelerators work, and at the last minute have to change the library to > use triangles instead of splines. What's more, every single app now has > to be rewritten or use the library's emulation -- that's assuming many > apps would consider using a 3D library that doesn't have hardware accel > support yet. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 7:15 ` Hans Reiser 2004-09-02 7:36 ` Linus Torvalds @ 2004-09-02 7:55 ` Robin Rosenberg 1 sibling, 0 replies; 1392+ messages in thread From: Robin Rosenberg @ 2004-09-02 7:55 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, David Masover, Jamie Lokier, Horst von Brand, Adrian Bunk, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thursday 02 September 2004 09.15, Hans Reiser wrote: > Linus Torvalds wrote: > >. Doing transactions on one file is > >only the beginning - you'll find people who want transactions across file > >boundaries etc. > > Yup, that's what reiser4 is aiming at and where things get exciting for > version control systems and the like. We are just missing the api code > for it at the moment, all the infrastructure is there, and the api is in > progress in sys_reiser4. Hope that is the same as or implied XA-support. Having to syncrhonized file updates with database transactions is not an uncommon problem. -- robin ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-01 20:50 ` Linus Torvalds [not found] ` <20040902002431.GN31934@mail.shareable.org> @ 2004-09-02 9:46 ` Alan Cox 2004-09-02 16:11 ` Jamie Lokier ` (2 more replies) 2004-09-03 15:44 ` Pavel Machek 2004-09-05 13:16 ` Tonnerre 3 siblings, 3 replies; 1392+ messages in thread From: Alan Cox @ 2004-09-02 9:46 UTC (permalink / raw) To: Linus Torvalds Cc: Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Mer, 2004-09-01 at 21:50, Linus Torvalds wrote: > and quite frankly, I think you can do the above pretty much totally in > user space with a small library and a daemon (in fact, ignoring security > issues you probably don't even need the daemon). And if you can prototype > it like that, and people actually find it useful, I suspect kernel support > for better performance might be possible. Gnome already supports this in the gnome-vfs2 layer. "MC" has supported it since the late 1990's. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 9:46 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Alan Cox @ 2004-09-02 16:11 ` Jamie Lokier 2004-09-02 17:41 ` Dave Kleikamp 2004-09-02 23:33 ` Paul Jakma 2004-09-02 17:46 ` Linus Torvalds 2004-09-02 20:38 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Frank van Maarseveen 2 siblings, 2 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-09-02 16:11 UTC (permalink / raw) To: Alan Cox Cc: Linus Torvalds, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List Alan Cox wrote: > On Mer, 2004-09-01 at 21:50, Linus Torvalds wrote: > > and quite frankly, I think you can do the above pretty much totally in > > user space with a small library and a daemon (in fact, ignoring security > > issues you probably don't even need the daemon). And if you can prototype > > it like that, and people actually find it useful, I suspect kernel support > > for better performance might be possible. > > Gnome already supports this in the gnome-vfs2 layer. "MC" has supported > it since the late 1990's. Firstly, if I have to do it from a Gnome program, about the only program where looking in a tar file is visibly useful is Nautilus. Ironically, clicking on a tar file in Nautilus doesn't work, despite having a dependency on gnome-vfs2. :/ Secondly, no, Gnome and MC don't support entering a container file, letting you make changes in it, and remembering those changes to _lazily_ regenerate the container file when you need it linearized, possibly months later or never, by some unrelated program. Thirdly, you must be referring to the Gnome versions of Bash, Make, GCC, coreutils and Perl which I haven't found. Perhaps we have a different idea of what "supports this" means :) uservfs, which is based on gnome-vfs and getting a bit rusty due to disuse, does try to solve the last problem. Unfortunately it needs further work to have a nicer interface, and the second problem is still not solved by gnome-vfs. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 16:11 ` Jamie Lokier @ 2004-09-02 17:41 ` Dave Kleikamp 2004-09-03 0:25 ` The argument for fs assistance in handling archives David Masover 2004-09-03 15:48 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Pavel Machek 2004-09-02 23:33 ` Paul Jakma 1 sibling, 2 replies; 1392+ messages in thread From: Dave Kleikamp @ 2004-09-02 17:41 UTC (permalink / raw) To: Jamie Lokier Cc: Alan Cox, Linus Torvalds, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2004-09-02 at 11:11, Jamie Lokier wrote: > Firstly, if I have to do it from a Gnome program, about the only > program where looking in a tar file is visibly useful is Nautilus. > Ironically, clicking on a tar file in Nautilus doesn't work, despite > having a dependency on gnome-vfs2. :/ This should be fixed in Nautilus, not the kernel. > Secondly, no, Gnome and MC don't support entering a container file, > letting you make changes in it, and remembering those changes to > _lazily_ regenerate the container file when you need it linearized, > possibly months later or never, by some unrelated program. Why do this in a tar file? tar = "tape archive". It isn't designed to be a file system. Sure, it's nice to have tools that make it easier to access files in a tar file, but to this isn't a job for the kernel. > Thirdly, you must be referring to the Gnome versions of Bash, Make, > GCC, coreutils and Perl which I haven't found. Perhaps we have a > different idea of what "supports this" means :) Please don't tell me that we have expectations to run make from within a tar file. This is getting silly. tar does a pretty good job of extracting files into real directories, and putting them back into an archive. I don't see a need to teach the kernel how to deal with compound files when user space can do it very easily. Shaggy ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 17:41 ` Dave Kleikamp @ 2004-09-03 0:25 ` David Masover 2004-09-03 12:50 ` Dave Kleikamp 2004-09-05 14:42 ` Tonnerre 2004-09-03 15:48 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Pavel Machek 1 sibling, 2 replies; 1392+ messages in thread From: David Masover @ 2004-09-03 0:25 UTC (permalink / raw) To: Dave Kleikamp Cc: Jamie Lokier, Alan Cox, Linus Torvalds, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dave Kleikamp wrote: [...] | Please don't tell me that we have expectations to run make from within a | tar file. This is getting silly. tar does a pretty good job of | extracting files into real directories, and putting them back into an | archive. I don't see a need to teach the kernel how to deal with | compound files when user space can do it very easily. Suppose I've got a tar file with an index attached. Suppose it's something like /usr/src/linux. Am I expected to extract all code for all architectures, with all drivers, all docs, etc? Now, yes -- or I have to figure out exactly which ones I need before I extract them manually, one by one. But with tar support for make (and so on), files can be extracted on demand. It's possible to do this in userspace, with named pipes, but that's much slower and insanely clumsy. This has further implications -- imagine a desktop, binary distro shipped with all files except the very most basic stuff as package archives. They can all be extracted, on demand -- the first time I run OpenOffice.org, it's installed. If there needs to be post-installation, that's handled by the .deb plugin (or whatever). I don't know offhand how big OOo is. I think it's something like this: ~ The installer is at most half (and maybe only a third) of the full installation. That's a HUGE optimization! -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTe5/HgHNmZLgCUhAQI5vQ/+NyU/tbW1Dyaf/OlDUEScH8jHghdcMPQQ qcyBbzid9hMT0pm4fRX4CQJ/vm+VLhfvYzEmgRUCyNY3JybCKeS/EynRt/ybdblu aB+hO8meFitBmAa7kYrj1UhWvoSvDSZgAwC9k50DYPuQO1kVZFjFYcPee1P54iwJ UMn9RE01aeufCt1+jWFxhsEZKfNWvXDCaQtqa483A2AWWzklwF25ZW2kSfp6G+i0 g1jND8pPDkQcP8ujGTuDxEI8LsN62glNzVZ8MhPa65lZI1vO5Ll2dDL2QKgNwziK MqtMMJD1d3HWa7QBHwMegJ0teR/hiqJ62SgQr3QpW4Xy9Ss0VUVH1HNuhxwPB2rl YYomqw2yO/GGSDs5XuXm/cRM5E9d+nvu1V8bsrSa5LK/64Vlp6huLkLNvOZ3y6vK 38ELPBxbmIA3iWTgaYDPANX/vrpnA0K8JQU9M4LMveaHhxfEcDbH+iZHtpjsYqF3 allfHH2SEZRFlXGxKBNZsXTcrudAHjoyEOQ+UiI9QLCM83G4bFGr1WEGOEHmD0ry hBETe8GkwuQK1CfxFm5obgFUmE4TwVRIWVD71EvoJFuBS+dlezO6GOZ5mDf91tSe goPS2f/9XKwyYfOnEnnfXT17k5SYjTB9m0upi6q7dJpxvg1535E6N1nCqdLZzTcJ mVkdhfFsUqI= =YjJH -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 0:25 ` The argument for fs assistance in handling archives David Masover @ 2004-09-03 12:50 ` Dave Kleikamp 2004-09-04 0:25 ` David Masover 2004-09-05 14:42 ` Tonnerre 1 sibling, 1 reply; 1392+ messages in thread From: Dave Kleikamp @ 2004-09-03 12:50 UTC (permalink / raw) To: David Masover Cc: Jamie Lokier, Alan Cox, Linus Torvalds, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2004-09-02 at 19:25, David Masover wrote: > Dave Kleikamp wrote: > [...] > | Please don't tell me that we have expectations to run make from within a > | tar file. This is getting silly. tar does a pretty good job of > | extracting files into real directories, and putting them back into an > | archive. I don't see a need to teach the kernel how to deal with > | compound files when user space can do it very easily. > > Suppose I've got a tar file with an index attached. Suppose it's > something like /usr/src/linux. Am I expected to extract all code for > all architectures, with all drivers, all docs, etc? Now, yes -- or I > have to figure out exactly which ones I need before I extract them > manually, one by one. I don't think it's unreasonable to expect someone to either extract the whole tar file, or identify what files they want from it. If you think there is too much in the tar file, roll your own with only the files you need. > But with tar support for make (and so on), files can be extracted on > demand. It's possible to do this in userspace, with named pipes, but > that's much slower and insanely clumsy. This doesn't justify bloating the kernel. untar the darn thing and user space does fine. > This has further implications -- imagine a desktop, binary distro > shipped with all files except the very most basic stuff as package > archives. They can all be extracted, on demand -- the first time I run > OpenOffice.org, it's installed. If there needs to be post-installation, > that's handled by the .deb plugin (or whatever). Are you saying install it on demand the first time it's run? This doesn't take any new kernel function. Or are you saying that the files are never installed on the filesystem, but always accessed from the package archives? In this case, why not ship each package as a compressed iso, and have the system mount the iso to run the app. I really don't see the point though, in that disk space is very cheap these days. > I don't know offhand how big OOo is. I think it's something like this: > ~ The installer is at most half (and maybe only a third) of the full > installation. That's a HUGE optimization! Optimization? How much performance are you willing to give up to save a little bit of disk space? -- David Kleikamp IBM Linux Technology Center ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 12:50 ` Dave Kleikamp @ 2004-09-04 0:25 ` David Masover 0 siblings, 0 replies; 1392+ messages in thread From: David Masover @ 2004-09-04 0:25 UTC (permalink / raw) To: Dave Kleikamp Cc: Jamie Lokier, Alan Cox, Linus Torvalds, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dave Kleikamp wrote: | On Thu, 2004-09-02 at 19:25, David Masover wrote: | |>Dave Kleikamp wrote: |>[...] |>| Please don't tell me that we have expectations to run make from within a |>| tar file. This is getting silly. tar does a pretty good job of |>| extracting files into real directories, and putting them back into an |>| archive. I don't see a need to teach the kernel how to deal with |>| compound files when user space can do it very easily. |> |>Suppose I've got a tar file with an index attached. Suppose it's |>something like /usr/src/linux. Am I expected to extract all code for |>all architectures, with all drivers, all docs, etc? Now, yes -- or I |>have to figure out exactly which ones I need before I extract them |>manually, one by one. | | | I don't think it's unreasonable to expect someone to either extract the | whole tar file, or identify what files they want from it. If you think | there is too much in the tar file, roll your own with only the files you | need. That is not a solution, that is exactly the opposite of a solution. "Roll my own" implies that I must download the tar file, extract the whole thing, grab what I need, and then tar it up again. Then I extract it again. Doh! More seriously, suppose it's a format like zip, where I don't have to decompress the whole file to get a listing. And suppose I want to compile some little example from it, but the file itself is huge -- half a gig, say, and I only need ten megabytes. But the only way I'm going to find out _which_ ten megabytes I need is to extract the Makefile, read it, then go find all the other little Makefiles, not to mention the configure script, the header files, and so on... But suppose that make can _transparently_ extract _only_ the files I need for this? |>But with tar support for make (and so on), files can be extracted on |>demand. It's possible to do this in userspace, with named pipes, but |>that's much slower and insanely clumsy. | | | This doesn't justify bloating the kernel. untar the darn thing and user | space does fine. Does it really bloat the kernel? My kernel doesn't feel bloated, and it's got reiser4 and much of what's needed for this. Remember, saying "I can access foo.zip/contents" doesn't mean "zip is in the kernel". |>This has further implications -- imagine a desktop, binary distro |>shipped with all files except the very most basic stuff as package |>archives. They can all be extracted, on demand -- the first time I run |>OpenOffice.org, it's installed. If there needs to be post-installation, |>that's handled by the .deb plugin (or whatever). | | | Are you saying install it on demand the first time it's run? This | doesn't take any new kernel function. It does for it to appear installed. And to only install the pieces that are needed at that time. | Or are you saying that the files are never installed on the filesystem, | but always accessed from the package archives? In this case, why not No, only that if I only wanted OpenOffice writer, why should I install all of OpenOffice? Yes, it could be broken into smaller packages. What I'm talking about is exactly the pieces of an installation that you need - -- insanely fine granularity -- done automatically, on demand. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTkLiHgHNmZLgCUhAQK7+A//TATQvQ3U61VA/mdVqylnrkWC7tNOewwq MqF+KKU3Cc/n54mOlGTpph2qpPJTzv1y4KlVgcDM/d0bn1cDPx41n//xEe9QXlqu vOywb8g11HSlAhKmbl4APwCHFHr1HibHgYqM7PmeVSD+Xfy5gJvIW5Oc44f16+q9 agEXWAk0EgM0WCAKEQFxN56i8e7qHq28PPGzcpGcn08xmBmD9Ik71jjpLY88csYy kjH32ExEy+uABq+Tglfr0EBZR4RDuqkxsei7cL3Rn58O8twJtn8UP3VcukTLciZw jb7If3ekuO7BXYJbwB/foFEESFql68jNKH7c7+Bzeb5pREloreVine/2rRM1iekD FUeTv78kn+6G/INl9XwUB2ER0KZOy8n2wZut35T5w94GtZgmdHpm+3mCOscS6BdG JNx/HRGJJXfm0P/7tKbgZ/3wjQlFzbC5HcByn9Ocfm8qrNsoLxwtF/8aId/9ctD9 lEmDMUHYuVC51/m5ka+i/XUQeuzgbtY5QKoNsxWXYZfeBNQfMqfMOsVWP1wFMlpB mPmf6w+4idp1aIYwgvPyQee1BZiXmkbncglcnY+J4y46AZ+tDEO5eqjJrEU9kA7v NNVfCehCZ2IIo1TMHcQizsHQ53NEmZ5gwCYXvymGS1+6fyE0SqcP9EolVuUfnRN9 yNdulMJ0gDg= =TeOM -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 0:25 ` The argument for fs assistance in handling archives David Masover 2004-09-03 12:50 ` Dave Kleikamp @ 2004-09-05 14:42 ` Tonnerre 1 sibling, 0 replies; 1392+ messages in thread From: Tonnerre @ 2004-09-05 14:42 UTC (permalink / raw) To: David Masover Cc: Dave Kleikamp, Jamie Lokier, Alan Cox, Linus Torvalds, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 458 bytes --] Salut, On Thu, Sep 02, 2004 at 07:25:32PM -0500, David Masover wrote: > This has further implications -- imagine a desktop, binary distro > shipped with all files except the very most basic stuff as package > archives. They can all be extracted, on demand -- the first time I run > OpenOffice.org, it's installed. If there needs to be post-installation, > that's handled by the .deb plugin (or whatever). zsh using apt extensions does that. Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 17:41 ` Dave Kleikamp 2004-09-03 0:25 ` The argument for fs assistance in handling archives David Masover @ 2004-09-03 15:48 ` Pavel Machek 1 sibling, 0 replies; 1392+ messages in thread From: Pavel Machek @ 2004-09-03 15:48 UTC (permalink / raw) To: Dave Kleikamp Cc: Jamie Lokier, Alan Cox, Linus Torvalds, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List Hi! > > Thirdly, you must be referring to the Gnome versions of Bash, Make, > > GCC, coreutils and Perl which I haven't found. Perhaps we have a > > different idea of what "supports this" means :) > > Please don't tell me that we have expectations to run make from within a > tar file. This is getting silly. tar does a pretty good job of > extracting files into real directories, and putting them back into an > archive. I don't see a need to teach the kernel how to deal with > compound files when user space can do it very easily. Actually its not easy. User has to manually extract it and manually delete it when he's done. Not nice. Pavel -- When do you have heart between your knees? ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 16:11 ` Jamie Lokier 2004-09-02 17:41 ` Dave Kleikamp @ 2004-09-02 23:33 ` Paul Jakma 2004-09-02 23:49 ` Spam 1 sibling, 1 reply; 1392+ messages in thread From: Paul Jakma @ 2004-09-02 23:33 UTC (permalink / raw) To: Jamie Lokier Cc: Alan Cox, Linus Torvalds, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2 Sep 2004, Jamie Lokier wrote: > Firstly, if I have to do it from a Gnome program, about the only > program where looking in a tar file is visibly useful is Nautilus. > Ironically, clicking on a tar file in Nautilus doesn't work, > despite having a dependency on gnome-vfs2. :/ Do you have file-roller installed? I can open tar/zip/rar/etc.. files from anywhere in gnome2, eg in Galeon I can click on a tar.gz URL (http or whatever) and have it open it in file-roller, from where i can browse the files to my hearts content. regards, -- Paul Jakma paul@clubi.ie paul@jakma.org Key ID: 64A2FF6A Fortune: Why isn't there a special name for the tops of your feet? -- Lily Tomlin ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 23:33 ` Paul Jakma @ 2004-09-02 23:49 ` Spam 2004-09-02 22:55 ` Alan Cox 0 siblings, 1 reply; 1392+ messages in thread From: Spam @ 2004-09-02 23:49 UTC (permalink / raw) To: Paul Jakma Cc: Jamie Lokier, Alan Cox, Linus Torvalds, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List > On Thu, 2 Sep 2004, Jamie Lokier wrote: >> Firstly, if I have to do it from a Gnome program, about the only >> program where looking in a tar file is visibly useful is Nautilus. >> Ironically, clicking on a tar file in Nautilus doesn't work, >> despite having a dependency on gnome-vfs2. :/ > Do you have file-roller installed? > I can open tar/zip/rar/etc.. files from anywhere in gnome2, eg in > Galeon I can click on a tar.gz URL (http or whatever) and have it > open it in file-roller, from where i can browse the files to my > hearts content. But can you actually do things with these files? Can you run applications or edit files directly, or is there need for temporary unzip first? ~S > regards, ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 23:49 ` Spam @ 2004-09-02 22:55 ` Alan Cox 2004-09-03 0:03 ` Spam 0 siblings, 1 reply; 1392+ messages in thread From: Alan Cox @ 2004-09-02 22:55 UTC (permalink / raw) To: Spam Cc: Paul Jakma, Jamie Lokier, Linus Torvalds, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Gwe, 2004-09-03 at 00:49, Spam wrote: > But can you actually do things with these files? Can you run > applications or edit files directly, or is there need for temporary > unzip first? You always need that for zip files. Firstly because executables are paged so you need an accessible random access copy of the bits. Secondly because data may be paged, and also for seek performance. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 22:55 ` Alan Cox @ 2004-09-03 0:03 ` Spam 2004-09-03 0:14 ` Paul Jakma 0 siblings, 1 reply; 1392+ messages in thread From: Spam @ 2004-09-03 0:03 UTC (permalink / raw) To: Alan Cox Cc: Paul Jakma, Jamie Lokier, Linus Torvalds, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List > On Gwe, 2004-09-03 at 00:49, Spam wrote: >> But can you actually do things with these files? Can you run >> applications or edit files directly, or is there need for temporary >> unzip first? > You always need that for zip files. Firstly because executables are > paged so you need an accessible random access copy of the bits. Secondly > because data may be paged, and also for seek performance. Yes, some archive types can't be partially unzipped either. But my point is that it wouldn't be transparent to the application/user in the same way. ~S ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-03 0:03 ` Spam @ 2004-09-03 0:14 ` Paul Jakma 2004-09-03 0:39 ` Spam 0 siblings, 1 reply; 1392+ messages in thread From: Paul Jakma @ 2004-09-03 0:14 UTC (permalink / raw) To: Spam Cc: Alan Cox, Jamie Lokier, Linus Torvalds, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Fri, 3 Sep 2004, Spam wrote: > Yes, some archive types can't be partially unzipped either. But my > point is that it wouldn't be transparent to the application/user in > the same way. It doesnt matter whether it is transparent to the application. It can be the application which implements the required level of transparency. User doesnt care what provides the transparency or how it's implemented. > ~S regards, -- Paul Jakma paul@clubi.ie paul@jakma.org Key ID: 64A2FF6A Fortune: A committee is a group that keeps the minutes and loses hours. -- Milton Berle ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-03 0:14 ` Paul Jakma @ 2004-09-03 0:39 ` Spam 2004-09-03 13:05 ` Dave Kleikamp ` (2 more replies) 0 siblings, 3 replies; 1392+ messages in thread From: Spam @ 2004-09-03 0:39 UTC (permalink / raw) To: Paul Jakma Cc: Alan Cox, Jamie Lokier, Linus Torvalds, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List > On Fri, 3 Sep 2004, Spam wrote: >> Yes, some archive types can't be partially unzipped either. But my >> point is that it wouldn't be transparent to the application/user in >> the same way. > It doesnt matter whether it is transparent to the application. It can > be the application which implements the required level of > transparency. > User doesnt care what provides the transparency or how it's > implemented. Indeed. I hope I didn't say otherwise :). Just that I think it will be very difficult to have this transparency in all apps. Just thinking of "nano file.jpg/description.txt" or "ls file.tar/untar/*.doc". Sure in some environments like Gnome it could work, but it still doesn't for the rest of the flora of Linux programs. ~S > regards, ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-03 0:39 ` Spam @ 2004-09-03 13:05 ` Dave Kleikamp 2004-09-03 13:16 ` Spam 2004-09-04 0:13 ` The argument for fs assistance in handling archives David Masover 2004-09-03 16:43 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Paul Jakma 2004-09-06 15:07 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Tonnerre 2 siblings, 2 replies; 1392+ messages in thread From: Dave Kleikamp @ 2004-09-03 13:05 UTC (permalink / raw) To: Spam Cc: Paul Jakma, Alan Cox, Jamie Lokier, Linus Torvalds, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2004-09-02 at 19:39, Spam wrote: > > > > On Fri, 3 Sep 2004, Spam wrote: > > >> Yes, some archive types can't be partially unzipped either. But my > >> point is that it wouldn't be transparent to the application/user in > >> the same way. > > > It doesnt matter whether it is transparent to the application. It can > > be the application which implements the required level of > > transparency. > > > User doesnt care what provides the transparency or how it's > > implemented. > > Indeed. I hope I didn't say otherwise :). Just that I think it will > be very difficult to have this transparency in all apps. You're missing the point. We don't need transparency in all apps. You can write an application to be as transparent as you want, but you don't need every app to to understand every file format. > Just > thinking of "nano file.jpg/description.txt" or "ls > file.tar/untar/*.doc". I don't do much image editting, but I'm sure there are applications that let you edit the description in a text file. You can even create a script that extracts it, runs nano, and puts it back into the jpeg. This works for me: tar -tf file.tar | grep '\.doc' There are userland tools that deal with hundreds of file formats. Use the tool you need, rather than try to have the kernel do everything. > Sure in some environments like Gnome it could > work, but it still doesn't for the rest of the flora of Linux > programs. Just choose the right program. tar groks tar files, not ls. -- David Kleikamp IBM Linux Technology Center ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-03 13:05 ` Dave Kleikamp @ 2004-09-03 13:16 ` Spam 2004-09-03 13:33 ` Dave Kleikamp 2004-09-03 21:45 ` Horst von Brand 2004-09-04 0:13 ` The argument for fs assistance in handling archives David Masover 1 sibling, 2 replies; 1392+ messages in thread From: Spam @ 2004-09-03 13:16 UTC (permalink / raw) To: Dave Kleikamp Cc: Paul Jakma, Alan Cox, Jamie Lokier, Linus Torvalds, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List > On Thu, 2004-09-02 at 19:39, Spam wrote: >> >> >> > On Fri, 3 Sep 2004, Spam wrote: >> >> >> Yes, some archive types can't be partially unzipped either. But my >> >> point is that it wouldn't be transparent to the application/user in >> >> the same way. >> >> > It doesnt matter whether it is transparent to the application. It can >> > be the application which implements the required level of >> > transparency. >> >> > User doesnt care what provides the transparency or how it's >> > implemented. >> >> Indeed. I hope I didn't say otherwise :). Just that I think it will >> be very difficult to have this transparency in all apps. > You're missing the point. We don't need transparency in all apps. You > can write an application to be as transparent as you want, but you don't > need every app to to understand every file format. No, but not every user "can write an application" either, or even have the skills to apply patches. What I was talking about wasn't just tar, which itself isn't the best example anyway, but the idea that users can load plugins that will extend the functionality of their filesystems. That idea seem to be to be _much_ better than trying to teach every user how to write applications or patch existing ones. >> Just >> thinking of "nano file.jpg/description.txt" or "ls >> file.tar/untar/*.doc". > I don't do much image editting, but I'm sure there are applications that > let you edit the description in a text file. You can even create a > script that extracts it, runs nano, and puts it back into the jpeg. > This works for me: > tar -tf file.tar | grep '\.doc' > There are userland tools that deal with hundreds of file formats. Use > the tool you need, rather than try to have the kernel do everything. No, but if I wanted to have an encryption plugin active for some of my files or directories then why should I not be able to? I still want to edit, view and save my encrypted files. Again, this was just an example of what could be done with plugins. It is not said that every conceivable plugin will be written, nor loaded per default. Even though plugins cannot today be dynamically used, they will be eventually. Reiser4 is still very young. Please separate your thoughts for specific plugins from those of the idea to have plugins at all. ~S >> Sure in some environments like Gnome it could >> work, but it still doesn't for the rest of the flora of Linux >> programs. > Just choose the right program. tar groks tar files, not ls. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-03 13:16 ` Spam @ 2004-09-03 13:33 ` Dave Kleikamp 2004-09-03 21:45 ` Horst von Brand 1 sibling, 0 replies; 1392+ messages in thread From: Dave Kleikamp @ 2004-09-03 13:33 UTC (permalink / raw) To: Spam Cc: Paul Jakma, Alan Cox, Jamie Lokier, Linus Torvalds, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Fri, 2004-09-03 at 08:16, Spam wrote: > > You're missing the point. We don't need transparency in all apps. You > > can write an application to be as transparent as you want, but you don't > > need every app to to understand every file format. > > No, but not every user "can write an application" either, or even > have the skills to apply patches. What I was talking about wasn't > just tar, which itself isn't the best example anyway, That was one of the examples you gave, that and .jpg. I believe they are both ridiculous. > but the idea > that users can load plugins that will extend the functionality of > their filesystems. That idea seem to be to be _much_ better than > trying to teach every user how to write applications or patch > existing ones. If I understand Hans' plugins, they are not user-loadable, but rather a statically built part of the kernel. > > No, but if I wanted to have an encryption plugin active for some of > my files or directories then why should I not be able to? I still > want to edit, view and save my encrypted files. I would not argue against an encryption plugin. > Again, this was just an example of what could be done with plugins. > It is not said that every conceivable plugin will be written, nor > loaded per default. This I agree with. > Even though plugins cannot today be dynamically > used, they will be eventually. Reiser4 is still very young. As kernel modules, this would make sense. I don't see just any unprivileged user being able to add code into the file system, though. > Please separate your thoughts for specific plugins from those of the > idea to have plugins at all. I'm not against reiser4 plugins. I don't think file system code should care about the type of data in a file, and do any interpretation based on it. Shaggy -- David Kleikamp IBM Linux Technology Center ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-03 13:16 ` Spam 2004-09-03 13:33 ` Dave Kleikamp @ 2004-09-03 21:45 ` Horst von Brand 2004-09-04 5:37 ` The argument for fs assistance in handling archives David Masover 2004-09-04 11:07 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Spam 1 sibling, 2 replies; 1392+ messages in thread From: Horst von Brand @ 2004-09-03 21:45 UTC (permalink / raw) To: Spam Cc: Dave Kleikamp, Paul Jakma, Alan Cox, Jamie Lokier, Linus Torvalds, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List Spam <spam@tnonline.net> said: > Dave Kleikamp <shaggy@austin.ibm.com> said: > >Spam <spam@tnonline.net> said: [...] > > You're missing the point. We don't need transparency in all apps. You > > can write an application to be as transparent as you want, but you don't > > need every app to to understand every file format. > No, but not every user "can write an application" either, or even > have the skills to apply patches. What I was talking about wasn't > just tar, which itself isn't the best example anyway, but the idea > that users can load plugins that will extend the functionality of > their filesystems. That idea seem to be to be _much_ better than > trying to teach every user how to write applications or patch > existing ones. Why compare "write application or apply patches" to "load plugin"? It would be locical to compare running applications with loading plugins (and even so, loading plugins is presumably root-only). [...] > > There are userland tools that deal with hundreds of file formats. Use > > the tool you need, rather than try to have the kernel do everything. > No, but if I wanted to have an encryption plugin active for some of > my files or directories then why should I not be able to? I still > want to edit, view and save my encrypted files. Use an editor that knows about encrypted files. Decrypt/edit/encrypt if no other option (I'm sure emacs can be coerced to do that transparently ;-). I for one would be _way_ more confortable with my users doing that than them loading random modules into the kernel. Besides, if one of my users doesn't trust the system encryption programs, and prefers hers, she can be happy. > Again, this was just an example of what could be done with plugins. > It is not said that every conceivable plugin will be written, nor > loaded per default. Even though plugins cannot today be dynamically > used, they will be eventually. Reiser4 is still very young. Modules of the kernel were supposed to have all those magic properties too, until there were nasty races... and it was _seriously_ considered to take them out. They stayed because they are root-only business, and (un)loading is rare. FS plugins are kernel modules, AFAIU, and are subject to the same problems. > Please separate your thoughts for specific plugins from those of the > idea to have plugins at all. If you can't find concrete uses for specific plugins, they are the proverbial solution searching for a problem. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 21:45 ` Horst von Brand @ 2004-09-04 5:37 ` David Masover 2004-09-04 6:21 ` Hans Reiser 2004-09-04 11:07 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Spam 1 sibling, 1 reply; 1392+ messages in thread From: David Masover @ 2004-09-04 5:37 UTC (permalink / raw) To: Horst von Brand Cc: Spam, Dave Kleikamp, Paul Jakma, Alan Cox, Jamie Lokier, Linus Torvalds, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Horst von Brand wrote: [...] | Use an editor that knows about encrypted files. Decrypt/edit/encrypt if no You use emacs. I use vim. My brother uses gedit. My parents use abiword. Perhaps I should patch them all? If that was so easy, why is there cryptoloop/dm-crypt? | is rare. FS plugins are kernel modules, AFAIU, and are subject to the same | problems. Actually, FS plugins currently cannot be modules. They are currently called "plugins" because they share some concepts with browser plugins, and it sounds great in marketing. | If you can't find concrete uses for specific plugins, they are the | proverbial solution searching for a problem. Fine, let them be. They are just very well structured code. If you use reiser4 and don't look in the source or the "metas" dir, they are completely invisible to you. I used the betas for months, and "metas" never burned me. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTlUt3gHNmZLgCUhAQJVGhAAki5RLZckm5jiZnw7MXQLqURBvm+cz7wU eFPfGyOE0SpFKJLFntWr0zrAgyK5ClgwqB7wsJEldWUfGBPdYpH5lroOQEHVEGBs 4X+ze/xyUOL6z3S07a85jNibYamDeoCDc5P0Vc6GWrdpsU7FQGXrHykNyglDxFJ1 MiYEQkB8NYDzQukl+7HPR3qPhQpAl5hx3XtmOcC5w0/88ATMqXg81DoVzPAPlsrL IPu4ai7KjXRaY1sKo8SU4orj7iQHmmkiFJJg+QwVU9sO2GMBGpXZRSr3KcUL3ux5 nr+++ceVXyLADZaJRYp5LoTxL0KPJUKhaa9ABLmN2zQ5hT/v6AlQmKKD3s6ca02a A8MQxy69hG50RVSeJm9yjRYQQBvATEXslCQXPXSAlLJGrPZ1FZgQdYyo2wNboD23 ep+JP2qTPdyTFFl2TOtoeR7fIsjg6DF5Bq0uh0maqC0UXXIo1GRO/OQGsNMfCN88 pevDg0GvE+bdeL8CEZYfDzu4aaUs+ltzZSPEKlXHCGFORL9iSuYhqdUCRPbcKYOy uyhE7fgZoPYsOZLfChmXllEF69Cs5Vm5R0ymIgHqprfAjfqYf4ypbU0fukFDQ5dS GGFTfxketTHdhYr7ATAfZg08ZMP819UnvcwISflLlDBwvpL4BDAWOrnDFnbHSugk lPAxyPnfECM= =h24Y -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-04 5:37 ` The argument for fs assistance in handling archives David Masover @ 2004-09-04 6:21 ` Hans Reiser 0 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-09-04 6:21 UTC (permalink / raw) To: David Masover Cc: Horst von Brand, Spam, Dave Kleikamp, Paul Jakma, Alan Cox, Jamie Lokier, Linus Torvalds, Adrian Bunk, viro, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List David Masover wrote: > I used the betas for months, and "metas" > never burned me. metas is much less likely than what clearcase uses to be hit accidentally (After all these years I forget what exactly clearcase special cases, maybe it was "@@"). When people using clearcase suffer a namespace collision, life goes on, no big deal, they structure a filename slightly differently and so what? I mean, just how much do we suffer from not being able to use '/' in filenames? Every once in a while it is annoying, but not that much.... ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-03 21:45 ` Horst von Brand 2004-09-04 5:37 ` The argument for fs assistance in handling archives David Masover @ 2004-09-04 11:07 ` Spam 1 sibling, 0 replies; 1392+ messages in thread From: Spam @ 2004-09-04 11:07 UTC (permalink / raw) To: Horst von Brand Cc: Dave Kleikamp, Paul Jakma, Alan Cox, Jamie Lokier, Linus Torvalds, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List > Spam <spam@tnonline.net> said: >> Dave Kleikamp <shaggy@austin.ibm.com> said: >> >Spam <spam@tnonline.net> said: > [...] >> > You're missing the point. We don't need transparency in all apps. You >> > can write an application to be as transparent as you want, but you don't >> > need every app to to understand every file format. >> No, but not every user "can write an application" either, or even >> have the skills to apply patches. What I was talking about wasn't >> just tar, which itself isn't the best example anyway, but the idea >> that users can load plugins that will extend the functionality of >> their filesystems. That idea seem to be to be _much_ better than >> trying to teach every user how to write applications or patch >> existing ones. > Why compare "write application or apply patches" to "load plugin"? It > would be locical to compare running applications with loading plugins (and > even so, loading plugins is presumably root-only). Doesn't matter if it is root-only. Ask the admin about the specific plugin the user wants. For most other users they can su to root and do this on their own computers. Not everyone is using Linux is a corporate world. > [...] >> > There are userland tools that deal with hundreds of file formats. Use >> > the tool you need, rather than try to have the kernel do everything. >> No, but if I wanted to have an encryption plugin active for some of >> my files or directories then why should I not be able to? I still >> want to edit, view and save my encrypted files. > Use an editor that knows about encrypted files. Decrypt/edit/encrypt if no > other option (I'm sure emacs can be coerced to do that transparently ;-). I > for one would be _way_ more confortable with my users doing that than them > loading random modules into the kernel. Besides, if one of my users doesn't > trust the system encryption programs, and prefers hers, she can be happy. That just doesn't do it. I doubt there will be an option to save encrypted Word and Excel files and be able to open them in Abiword, StarOffice or OpenOffice unless the decryption/encryption is done on a lower level. Also, eventually there may be a userland interface for loading certain modules without root access.? >> Again, this was just an example of what could be done with plugins. >> It is not said that every conceivable plugin will be written, nor >> loaded per default. Even though plugins cannot today be dynamically >> used, they will be eventually. Reiser4 is still very young. > Modules of the kernel were supposed to have all those magic properties too, > until there were nasty races... and it was _seriously_ considered to take > them out. They stayed because they are root-only business, and (un)loading > is rare. FS plugins are kernel modules, AFAIU, and are subject to the same > problems. >> Please separate your thoughts for specific plugins from those of the >> idea to have plugins at all. > If you can't find concrete uses for specific plugins, they are the > proverbial solution searching for a problem. To me there are fine and good uses for plugins. Everyone seem to think different though. Sure many things can perhaps be done by linking and piping lots of programs, just to access some data. But that certainly is far from user friendly. ~S ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 13:05 ` Dave Kleikamp 2004-09-03 13:16 ` Spam @ 2004-09-04 0:13 ` David Masover 2004-09-04 11:42 ` Stephan von Krawczynski 1 sibling, 1 reply; 1392+ messages in thread From: David Masover @ 2004-09-04 0:13 UTC (permalink / raw) To: Dave Kleikamp Cc: Spam, Paul Jakma, Alan Cox, Jamie Lokier, Linus Torvalds, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dave Kleikamp wrote: | On Thu, 2004-09-02 at 19:39, Spam wrote: | |> |> |> |>>On Fri, 3 Sep 2004, Spam wrote: |> |>>> Yes, some archive types can't be partially unzipped either. But my |>>> point is that it wouldn't be transparent to the application/user in |>>> the same way. |> |>>It doesnt matter whether it is transparent to the application. It can |>>be the application which implements the required level of |>>transparency. |> |>>User doesnt care what provides the transparency or how it's |>>implemented. |> |> Indeed. I hope I didn't say otherwise :). Just that I think it will |> be very difficult to have this transparency in all apps. | | | You're missing the point. We don't need transparency in all apps. You | can write an application to be as transparent as you want, but you don't | need every app to to understand every file format. You don't need every app to understand every filesystem, either. Come to think of it, why would I want anything other than CD burning software and file browsers to understand a CD drive? Let's all just wait -- quietly -- for the powers that be to work out whether it's feasable and sane to support it in all apps. Because wouldn't that be better? |> Just |> thinking of "nano file.jpg/description.txt" or "ls |> file.tar/untar/*.doc". | | | I don't do much image editting, but I'm sure there are applications that | let you edit the description in a text file. You can even create a | script that extracts it, runs nano, and puts it back into the jpeg. That is a PITA. Because you'll need to make more scripts. And still more. Say you want to grep for a jpeg with a particular description. Say you want to copy the description from one jpeg into another. Maybe you could make a general-purpose command, like "jpeg_run", which runs a command on the jpeg description, but it'd still be hackish, slow, and more to type and remember. Consider that you might have no idea what "jpeg_run" is called, but you can always do "ls file.jpeg/metas" to find out how to edit it. | | This works for me: | tar -tf file.tar | grep '\.doc' And then you need to run "tar -tf" a minute later, this time looking for *.xls. Maybe file.tar is actually a several gigabyte file.tar.bz2. |> Sure in some environments like Gnome it could |> work, but it still doesn't for the rest of the flora of Linux |> programs. | | | Just choose the right program. tar groks tar files, not ls. tar groks tar. bzip2 groks bz2. gzip groks gz. "mount -o loop" groks images. zip groks zip. rar groks rar. openoffice groks .sxw, unless staroffice does. xmms can read id3 info from mp3s. gcc compiles C. If the file is an object, it's easier. You can ask the file what it is, and what you can do with it. And you can tell it to do certain standard things. You can do all of this while reading almost no documentation on what the file is. At the end of the day, it won't matter to you whether a file is zip, rar, or tar -- it's an archive, and you can extract it by copying files out of its /contents directory. Your way (the traditional way) means you have to learn to use which program is the right program and how to use that program, and you'll have to remember it constantly. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTkIq3gHNmZLgCUhAQJaNRAAnp/o0Wr+SJK5tkgYFCuc+CKIP+eALJo/ 9wHnqK0nLdnji+vG0Czd9TUj1vWtoMrUichAwFoguMHHg3VeZGu61YwZoZ4idLNM QbZ+CuQdUygNmyT0byGMFemP+cSbyvff1PRMy2BlSHKW3gUhvnQyggLGVKxpMRWf VNnEHvkvJeA9PpEm6QGi1VRNp5bc0+Ocl4kO4CJk5ZYZ9D+BV6NwN/MZwqwlsu+Y RZsYYEa6mLiCnU4rEo0tEAvvwMdC0e/9s9TQMcmJbT6JnybkWIRFMrTS4pmSabKg uYGG9p1WrX8/V8WRNnaodlvx35gRPQj5S5SWDBoSWr999nmq31Y5RZ9QwbVj0d+U yB4yNu0NpvFBJfwg8nVIKUa+bhPLCkdY5w+GnlEYGweSN20FYOaLiqiPtXBqJm8a PP+8OCL35zy+1X7t/tq+JG/K91fYbECPR/qrAyHDXzNSuxdqidvBjfsEPBHOGwu6 RpsPoKyEvkzdXgauotWbVzWLt1ijGmYd/8Uk19OnmiloggViQhUWAJTIVGcvMNt8 Tnk8ESQMzGHbbVOIu3gB6DZD6P0IEEN4L6+gOZWoYqe2nvaYKvIL1My/i4BMU5Ml cxDkzzeq58iVK07Has8lwgFp0U9iD1LrbIT5p0ZJn/52EXqlX4dJCYSSNaHzUqE8 hcLG4aDELEI= =Wcyv -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-04 0:13 ` The argument for fs assistance in handling archives David Masover @ 2004-09-04 11:42 ` Stephan von Krawczynski 2004-09-04 18:27 ` David Masover 0 siblings, 1 reply; 1392+ messages in thread From: Stephan von Krawczynski @ 2004-09-04 11:42 UTC (permalink / raw) To: David Masover Cc: shaggy, spam, paul, alan, jamie, torvalds, vonbrand, bunk, reiser, viro, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, 03 Sep 2004 19:13:31 -0500 David Masover <ninja@slaphack.com> wrote: > | Just choose the right program. tar groks tar files, not ls. > > tar groks tar. bzip2 groks bz2. gzip groks gz. "mount -o loop" groks > images. zip groks zip. rar groks rar. openoffice groks .sxw, unless > staroffice does. xmms can read id3 info from mp3s. gcc compiles C. > > If the file is an object, it's easier. You can ask the file what it is, > and what you can do with it. And you can tell it to do certain standard > things. You can do all of this while reading almost no documentation on > what the file is. At the end of the day, it won't matter to you whether > a file is zip, rar, or tar -- it's an archive, and you can extract it by > copying files out of its /contents directory. > > Your way (the traditional way) means you have to learn to use which > program is the right program and how to use that program, and you'll > have to remember it constantly. Just a short input from someone listening quite a while to this thread: I think your approach to the problem leads actually nowhere. The reason for this is implicit in your own explanation. _Currently_ you talk of archives, but as soon as you got that, what's next? Your idea needs abstraction, then you see the problem more clearly: An archive is only some sort of file type, just as (you already named) mp3 or .sxw is another type. If you really want to do something _generally_ useful you should think of a method to parse and use all kinds of filetypes and create an interface for that. And one thing is clear: as there are numerous different types you cannot pull all that code inside the kernel. Obviously there should be a way some application can install a hook, a helper, a plugin or whatever to provide extended functionality on its special filetypes. If you don't want to use tar, you don't need the plugin either. If you want tar, you should (as a user) be able to install the "fs-plugin" (just a name, do not shoot me for it) together with tar as an application. You get the idea? Obviously this must be possible during runtime, everything else is senseless. So, please, if you go on investigating this whole stuff do it in a more general way, because "tar inside kernel" is not really what your idea is all about, no? And one last obvious thing: if one doesn't want this kind of file-handling, he should not need it. It has to be an add-on functionality, not a replacement. Regards, Stephan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-04 11:42 ` Stephan von Krawczynski @ 2004-09-04 18:27 ` David Masover 0 siblings, 0 replies; 1392+ messages in thread From: David Masover @ 2004-09-04 18:27 UTC (permalink / raw) To: Stephan von Krawczynski Cc: shaggy, spam, paul, alan, jamie, torvalds, vonbrand, bunk, reiser, viro, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Stephan von Krawczynski wrote: | On Fri, 03 Sep 2004 19:13:31 -0500 | David Masover <ninja@slaphack.com> wrote: | | |>| Just choose the right program. tar groks tar files, not ls. |> |>tar groks tar. bzip2 groks bz2. gzip groks gz. "mount -o loop" groks |>images. zip groks zip. rar groks rar. openoffice groks .sxw, unless |>staroffice does. xmms can read id3 info from mp3s. gcc compiles C. |> |>If the file is an object, it's easier. You can ask the file what it is, |>and what you can do with it. And you can tell it to do certain standard |>things. You can do all of this while reading almost no documentation on |>what the file is. At the end of the day, it won't matter to you whether |>a file is zip, rar, or tar -- it's an archive, and you can extract it by |>copying files out of its /contents directory. |> |>Your way (the traditional way) means you have to learn to use which |>program is the right program and how to use that program, and you'll |>have to remember it constantly. | | | Just a short input from someone listening quite a while to this thread: | I think your approach to the problem leads actually nowhere. The reason for | this is implicit in your own explanation. _Currently_ you talk of archives, but | as soon as you got that, what's next? Anything sane. Currently I talk of archives, because I need examples, and archives have been a good example for awhile of something that can gain things from plugins which they cannot get anywhere else, which have implications immediately after the archive plugin is wrote, without requiring any application support. | Your idea needs abstraction, then you see the problem more clearly: | An archive is only some sort of file type, just as (you already named) mp3 or | .sxw is another type. If you really want to do something _generally_ useful you | should think of a method to parse and use all kinds of filetypes and create an | interface for that. Fine. It'd certainly be nice to use vim to edit an id3 tag, or a text portion of a .sxw document (openoffice is HUGE compared to vim). | And one thing is clear: as there are numerous different types you cannot pull | all that code inside the kernel. Obviously there should be a way some I never said it should all go in the kernel. I repeatedly said that most of it should not -- all that goes in the kernel is the most generic interface that is sane. Obviously an archive plugin is different than a compression plugin, but once you've got the archive plugin, the rest (individual format support) can go in userland. | application can install a hook, a helper, a plugin or whatever to provide | extended functionality on its special filetypes. If you don't want to use tar, | you don't need the plugin either. So you disable it at compile time. I don't want to use XFS, and I certainly don't want to use ATI's video drivers. The latter I choose not to download, and the former I choose not to activate. | If you want tar, you should (as a user) be able to install the "fs-plugin" If by "as a user" you mean "not root", I have to disagree. I think it'd have security implications. It'd be great if I'm wrong. | (just a name, do not shoot me for it) together with tar as an application. You | get the idea? Obviously this must be possible during runtime, everything else | is senseless. I would love for this same attitude of "runtime or bust" to be more common. I would love for there to be a generic way to write a userland fs plugin, just as there's a generic way to write a userland filesystem (lufs). Just understand that some plugins will be in userland, and some will be in the kernel. And, in fact, the part of the archive plugin that knows about tar/zip/rar/whatever goes in userspace. But the part that does caching probably goes in kernel space, even if it becomes a more generic "caching plugin". | So, please, if you go on investigating this whole stuff do it in a more general | way, because "tar inside kernel" is not really what your idea is all about, no? Absolutely not. | And one last obvious thing: if one doesn't want this kind of file-handling, he | should not need it. It has to be an add-on functionality, not a replacement. For awhile, yes. But I can imagine that sometime in the future, an app would need this kind of functionality. And why not? Is it so much worse for an app to insist on "mp3 metadata plugin" than to insist on "libid3"? In any case, look at the current implementation. I can do echo 750 > file/metas/rwx but I can also use chmod, which is currently much more efficient. I think Hans wants to eventually create an efficient interface to access lots of small files at once (or maybe this is done?) -- and his dream is to see things like chmod replaced with a call to this interface. But at best, chmod will become a library call sometime years from now. It will never go away entirely. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQToI+3gHNmZLgCUhAQKEWg/9F0FZGxB2yXoo9l5UKI7Fvh19RlnpJvZE D3DajNqxXZ5smYIba7QxATEwMLWNMwOdiJTDIcQ7jylbD3hTkp0uFQLXcS5FeWt5 yxA0oDYEHsW14aQov4pHGplHVvCHSkm6vJn/zU91QdJAZVYnH4na5NFuWsAIoUWi 5vigYTefavabFv0/RnyPCN6zFfn3hrjOnKqSELUXXleG/fV+XJVw3Nyg4DSe1hK+ xmXHFWkGC832tH052EH/CFlE17/K05QdpfBKYNzCFA5dvTtCs11OQZSggtGVipmn cSycyDHtJraMdF78Qa8LZzjGiZbT/3befM6wrnekcwhjxCqok8CSbOvbVh2ENtiu klNtHeQEEtm+4MhbEEkdiiuWaU7EHIFJFGYC6LJE0B4DAKDdi4TM9QbqxLW8HghQ WUJfMh8H2FtqotjMsdDW0OCmSuzh0B4ZZ9EXvSMIttXBJ7T2+q442hm4PAFQWXSO FPCAiUgGZ+ZCbyZI1/pjM+CGblMTCfCBOm8UI3MSjTTI7cVzWFrQHJSAUhCPIsxA 9RZlEvmg+ElbnGrJkpDggpToDFN5uHXj41r4CZBnXgjfUdA0jmHnWpzO29Smfu7i cJPtN+LzR1Se+fz4IvYamWd1p/Mi3vQACPt4r3IGVKa27tYXzDw+CNa+CliHnczt lAhISMnhoVo= =7rV6 -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-03 0:39 ` Spam 2004-09-03 13:05 ` Dave Kleikamp @ 2004-09-03 16:43 ` Paul Jakma 2004-09-03 23:45 ` The argument for fs assistance in handling archives David Masover 2004-09-06 15:07 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Tonnerre 2 siblings, 1 reply; 1392+ messages in thread From: Paul Jakma @ 2004-09-03 16:43 UTC (permalink / raw) To: Spam Cc: Alan Cox, Jamie Lokier, Linus Torvalds, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Fri, 3 Sep 2004, Spam wrote: > Indeed. I hope I didn't say otherwise :). Sure. > Just that I think it will > be very difficult to have this transparency in all apps. Just > thinking of "nano file.jpg/description.txt" or "ls > file.tar/untar/*.doc". Sure in some environments like Gnome it could > work, but it still doesn't for the rest of the flora of Linux > programs. "will it be transparent for all apps?", whether that's worth doing depends on the technical implications. Thankfully we have Al and Linus to make the judgement call on that ;) Personally, I think that if GNOME can provide transparency for GNOME users, I think that's probably enough - unless there are literally no issues in adding some kind of VFS support. The nano / ls /tar user is likely a very different user to the GNOME user. That user is also likely to appreciate the problems with backups and such more. Anyway, userspace transparency is sufficient for most classes of users. Only reason to provide some kernel support is if it makes sense ("but not all apps can use GNOME transparency" not being one of those reasons). regards, -- Paul Jakma paul@clubi.ie paul@jakma.org Key ID: 64A2FF6A Fortune: Beat your son every day; you may not know why, but he will. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 16:43 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Paul Jakma @ 2004-09-03 23:45 ` David Masover 0 siblings, 0 replies; 1392+ messages in thread From: David Masover @ 2004-09-03 23:45 UTC (permalink / raw) To: Paul Jakma Cc: Spam, Alan Cox, Jamie Lokier, Linus Torvalds, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Paul Jakma wrote: | On Fri, 3 Sep 2004, Spam wrote: | |> Indeed. I hope I didn't say otherwise :). | | | Sure. | |> Just that I think it will |> be very difficult to have this transparency in all apps. Just |> thinking of "nano file.jpg/description.txt" or "ls |> file.tar/untar/*.doc". Sure in some environments like Gnome it could |> work, but it still doesn't for the rest of the flora of Linux |> programs. | | | "will it be transparent for all apps?", whether that's worth doing | depends on the technical implications. Thankfully we have Al and Linus | to make the judgement call on that ;) So far, the technical implications are mainly "does it create a serious issue in the distant future" or "does some exotic new feature that doesn't exist yet cause problems" and not "it's currently broken". Right now, I can edit a file's permissions, transparently, from any app, and I haven't noticed any stability issues at all. | Personally, I think that if GNOME can provide transparency for GNOME | users, I think that's probably enough - unless there are literally no | issues in adding some kind of VFS support. Only it can't. Especially if it's a typical GNOME user, who gets used to having their files encrypted in foo.tar.pgp, say. Works in abiword, works in gedit, breaks in OpenOffice. Not good. | The nano / ls /tar user is likely a very different user to the GNOME | user. That user is also likely to appreciate the problems with backups | and such more. I'm a vim/ls/tar user. And I believe the problems with backups to be very minor. There are other issues that I don't understand as thoroughly, so I'll leave them to Al and Linus. | Anyway, userspace transparency is sufficient for most classes of users. | Only reason to provide some kernel support is if it makes sense ("but | not all apps can use GNOME transparency" not being one of those reasons). Can you justify why "not all apps can use GNOME transparency" is not a valid reason? Would you still think so if GNOME transparency was the only way to read isofs? -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTkCL3gHNmZLgCUhAQJVSxAAiFd+e/Q0VjAvPaQL1IDwA56SP7LF2M1v jnk5wehjvicPEfSn2HpWik7Pr55K2/7dUn6mEJO4N43x/zmfyiA2wXLibfWJtD3X VgtFMgbOeKu+WobfdnvlBPAKYoZ+MYsC2jyDYwOHguJbR9epmFtHv0mMhmD60s+B 3JDLaYXwPjmjs7zezy2B/Xc3mgm060VgDZRVOnrYZLzHAi3dLvZLzGjORf5kBZtd NtsiPUZw3oIqORBHqfWDWUUi4irfzklZkiuYp44RNFZs0z0xvPrUJ1klXiJpbQ9t HTkz9qRe0sxjRYfLcJzWxfahaNN+2SXyR5FFqkpzfUfyNuoOUMZDtin+ZM83qyVc J/zRCZvWQOaZUtgas9KCCYnbwgCFLcDEE0xLLDRMGAMKeHOwCImU55ChTC/0HTas Q7Bd0df456dAv+ktbdMRaznWeNOpW5rJWXtxNeWnncTYKG0ogXct9mD0+BEIssaK qaEYUousp6ZRvOSlCCNCF5hOvfEuZLuAZe2Q8zKH39B4Hg7BMnmIOw6VPLmlLKds pGNuIKfKjyzkxcdHwqGp36e27wizyKtCbZ58Zd0x1wnf47gkm8kgkY38tb0ct7Xf Ch+s3+zLsOtVnnipnlGhIPEtxYv78DiqhGUc+pdZisi5VlfA4PTXZAAjoICeXA/H EqfVU6yYsu8= =ggxZ -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-03 0:39 ` Spam 2004-09-03 13:05 ` Dave Kleikamp 2004-09-03 16:43 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Paul Jakma @ 2004-09-06 15:07 ` Tonnerre 2004-09-06 15:15 ` Spam 2 siblings, 1 reply; 1392+ messages in thread From: Tonnerre @ 2004-09-06 15:07 UTC (permalink / raw) To: Spam Cc: Paul Jakma, Alan Cox, Jamie Lokier, Linus Torvalds, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 148 bytes --] Salut, On Fri, Sep 03, 2004 at 02:39:06AM +0200, Spam wrote: > "nano file.jpg/description.txt" Is that supposed to do steganography? Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-06 15:07 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Tonnerre @ 2004-09-06 15:15 ` Spam 0 siblings, 0 replies; 1392+ messages in thread From: Spam @ 2004-09-06 15:15 UTC (permalink / raw) To: Tonnerre Cc: Paul Jakma, Alan Cox, Jamie Lokier, Linus Torvalds, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List > Salut, > On Fri, Sep 03, 2004 at 02:39:06AM +0200, Spam wrote: >> "nano file.jpg/description.txt" > Is that supposed to do steganography? No, not at all. I already use similar things in Windows this is why I suggested it here. Having a separate file for descriptions are not as convenient. Especially if you have a few hundred JPEGs and I want to quickly view some properties of the one you have just looked at. ~S > Tonnerre ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 9:46 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Alan Cox 2004-09-02 16:11 ` Jamie Lokier @ 2004-09-02 17:46 ` Linus Torvalds 2004-09-02 17:50 ` Christoph Hellwig ` (2 more replies) 2004-09-02 20:38 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Frank van Maarseveen 2 siblings, 3 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-09-02 17:46 UTC (permalink / raw) To: Alan Cox Cc: Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2 Sep 2004, Alan Cox wrote: > > On Mer, 2004-09-01 at 21:50, Linus Torvalds wrote: > > and quite frankly, I think you can do the above pretty much totally in > > user space with a small library and a daemon (in fact, ignoring security > > issues you probably don't even need the daemon). And if you can prototype > > it like that, and people actually find it useful, I suspect kernel support > > for better performance might be possible. > > Gnome already supports this in the gnome-vfs2 layer. "MC" has supported > it since the late 1990's. And nobody has asked for kernel support that I know of. So either "it just works" in user space, or people haven't figured out the kernel could help them. Or decided it's not worth it, exactly because they'd still have to support systems/filesystems that can't be converted. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 17:46 ` Linus Torvalds @ 2004-09-02 17:50 ` Christoph Hellwig 2004-09-02 18:03 ` Linus Torvalds 2004-09-02 18:46 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Alan Cox 2004-09-02 19:50 ` Spam 2 siblings, 1 reply; 1392+ messages in thread From: Christoph Hellwig @ 2004-09-02 17:50 UTC (permalink / raw) To: Linus Torvalds Cc: Alan Cox, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, Sep 02, 2004 at 10:46:32AM -0700, Linus Torvalds wrote: > > On Mer, 2004-09-01 at 21:50, Linus Torvalds wrote: > > > and quite frankly, I think you can do the above pretty much totally in > > > user space with a small library and a daemon (in fact, ignoring security > > > issues you probably don't even need the daemon). And if you can prototype > > > it like that, and people actually find it useful, I suspect kernel support > > > for better performance might be possible. > > > > Gnome already supports this in the gnome-vfs2 layer. "MC" has supported > > it since the late 1990's. > > And nobody has asked for kernel support that I know of. > > So either "it just works" in user space, or people haven't figured out the > kernel could help them. Or decided it's not worth it, exactly because > they'd still have to support systems/filesystems that can't be converted. http://oss.oracle.com/projects/userfs/ has code that clues gnomevfs onto a kernel filesystem. The code is horrible, but it shows that it can be done. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 17:50 ` Christoph Hellwig @ 2004-09-02 18:03 ` Linus Torvalds 2004-09-03 0:57 ` The argument for fs assistance in handling archives Linh Dang 0 siblings, 1 reply; 1392+ messages in thread From: Linus Torvalds @ 2004-09-02 18:03 UTC (permalink / raw) To: Christoph Hellwig Cc: Alan Cox, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2 Sep 2004, Christoph Hellwig wrote: > > http://oss.oracle.com/projects/userfs/ has code that clues gnomevfs onto > a kernel filesystem. The code is horrible, but it shows that it can > be done. I do like the setup where the extended features are done as a "view" on top of some other filesystem, so that you can choose to _either_ access the raw (and supposedly stable, simply by virtue of simplicity) or the "fancy" interface. Without having to reformat the disk to a filesystem you don't trust, or you have other reasons you can't use (disk sharing with other systems, whatever). It doesn't have to be "user", btw, in the sense that a lot of the normal code could be in kernel mode. Same way as Tux handling all the regular static requests entirely in kernel mode, but having the ability for calling down to apache.. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 18:03 ` Linus Torvalds @ 2004-09-03 0:57 ` Linh Dang 2004-09-04 2:58 ` Giuseppe Bilotta 0 siblings, 1 reply; 1392+ messages in thread From: Linh Dang @ 2004-09-03 0:57 UTC (permalink / raw) To: linux-kernel Linus Torvalds <torvalds@osdl.org> wrote: > > > On Thu, 2 Sep 2004, Christoph Hellwig wrote: >> >> http://oss.oracle.com/projects/userfs/ has code that clues gnomevfs >> onto a kernel filesystem. The code is horrible, but it shows that >> it can be done. > > I do like the setup where the extended features are done as a "view" > on top of some other filesystem, so that you can choose to _either_ > access the raw (and supposedly stable, simply by virtue of > simplicity) or the "fancy" interface. Without having to reformat the > disk to a filesystem you don't trust, or you have other reasons you > can't use (disk sharing with other systems, whatever). It'd be something similar to what clearcase does (not that I like clearcase, I hate it with a passion for other reasons!) On such a system, one would have multiple virtual views mounted (by root) under: /view/tar, /view/dpkg, /view/rpm, etc. for every regular file /home/joe/blah.tar the path /view/tar/home/joe/blah.tar/ is a directory where member of the archives directly accessible. old tools continue work as is. new tools can take a look on virtual views for virtual access. Not sure how such a system would work with the dentry cache. -- Linh Dang ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 0:57 ` The argument for fs assistance in handling archives Linh Dang @ 2004-09-04 2:58 ` Giuseppe Bilotta 0 siblings, 0 replies; 1392+ messages in thread From: Giuseppe Bilotta @ 2004-09-04 2:58 UTC (permalink / raw) To: linux-kernel Linh Dang wrote: > On such a system, one would have multiple virtual views mounted (by > root) under: > > /view/tar, /view/dpkg, /view/rpm, etc. > > for every regular file /home/joe/blah.tar > > the path /view/tar/home/joe/blah.tar/ is a directory where member of > the archives directly accessible. > > old tools continue work as is. new tools can take a look on virtual > views for virtual access. > > Not sure how such a system would work with the dentry cache. How does it cope with 'view withint view'? Say you have a .zipped ISO. Can you do cd /view/iso/view/zip/home/joe/mycd.zip/mycd.iso/ ? -- Giuseppe "Oblomov" Bilotta Can't you see It all makes perfect sense Expressed in dollar and cents Pounds shillings and pence (Roger Waters) ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 17:46 ` Linus Torvalds 2004-09-02 17:50 ` Christoph Hellwig @ 2004-09-02 18:46 ` Alan Cox 2004-09-02 20:22 ` Linus Torvalds 2004-09-02 21:56 ` Bill Huey 2004-09-02 19:50 ` Spam 2 siblings, 2 replies; 1392+ messages in thread From: Alan Cox @ 2004-09-02 18:46 UTC (permalink / raw) To: Linus Torvalds Cc: Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Iau, 2004-09-02 at 18:46, Linus Torvalds wrote: > > Gnome already supports this in the gnome-vfs2 layer. "MC" has supported > > it since the late 1990's. > > And nobody has asked for kernel support that I know of. I asked our desktop people. They want something like inotify because dontify doesn't cut it. They have zero interest in the multiple streams and hiding icons in streams type stuff. Alan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 18:46 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Alan Cox @ 2004-09-02 20:22 ` Linus Torvalds 2004-09-05 7:27 ` Stephen Rothwell 2004-09-02 21:56 ` Bill Huey 1 sibling, 1 reply; 1392+ messages in thread From: Linus Torvalds @ 2004-09-02 20:22 UTC (permalink / raw) To: Alan Cox Cc: Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2 Sep 2004, Alan Cox wrote: > > I asked our desktop people. They want something like inotify because > dontify doesn't cut it. Well, dnotify() really _is_ inotify(), since it does actually work on inodes, not dentries. I think what they are really complaining about is that dnotify() only notifies the _directory_ when a file is changed, and they'd like it to notify the file itself too. Which is a one-liner, really. Does the following make sense? (Totally untested, use-at-your-own-risk, I've-never-actually-used-dnotify-in-user-space, whatever). Linus ===== fs/dnotify.c 1.17 vs edited ===== --- 1.17/fs/dnotify.c 2004-08-09 18:45:22 -07:00 +++ edited/fs/dnotify.c 2004-09-02 13:21:26 -07:00 @@ -160,6 +160,8 @@ if (!dir_notify_enable) return; + __inode_dir_notify(dentry->d_inode, event); + spin_lock(&dentry->d_lock); parent = dentry->d_parent; if (parent->d_inode->i_dnotify_mask & event) { ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 20:22 ` Linus Torvalds @ 2004-09-05 7:27 ` Stephen Rothwell 0 siblings, 0 replies; 1392+ messages in thread From: Stephen Rothwell @ 2004-09-05 7:27 UTC (permalink / raw) To: Linus Torvalds; +Cc: linux-fsdevel, LKML [-- Attachment #1: Type: text/plain, Size: 1236 bytes --] On Thu, 2 Sep 2004 13:22:41 -0700 (PDT) Linus Torvalds <torvalds@osdl.org> wrote: > > Well, dnotify() really _is_ inotify(), since it does actually work on > inodes, not dentries. The "d" stands for directory not dentry :-) > I think what they are really complaining about is that dnotify() only > notifies the _directory_ when a file is changed, and they'd like it to > notify the file itself too. Which is a one-liner, really. I don't think so, since this notify will only happen if the process has registered for the notification and there is no way to register unless the file is a directory ... > Does the following make sense? (Totally untested, use-at-your-own-risk, > I've-never-actually-used-dnotify-in-user-space, whatever). I had intended to extend dnotify to do file notifies, but I think the real killer is needing the keep the file open that you want to be notified about when you want to be notified about lots of files ... I think that is what inotify was trying to fix (but I haven't had a chance to look at it recently). It reminds me of omirr that we had many years ago - I wonder what happened to it? -- Cheers, Stephen Rothwell sfr@canb.auug.org.au http://www.canb.auug.org.au/~sfr/ [-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 18:46 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Alan Cox 2004-09-02 20:22 ` Linus Torvalds @ 2004-09-02 21:56 ` Bill Huey 2004-09-02 22:48 ` Alan Cox 1 sibling, 1 reply; 1392+ messages in thread From: Bill Huey @ 2004-09-02 21:56 UTC (permalink / raw) To: Alan Cox Cc: Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, Sep 02, 2004 at 07:46:05PM +0100, Alan Cox wrote: > On Iau, 2004-09-02 at 18:46, Linus Torvalds wrote: > > > Gnome already supports this in the gnome-vfs2 layer. "MC" has supported > > > it since the late 1990's. > > > > And nobody has asked for kernel support that I know of. > > I asked our desktop people. They want something like inotify because > dontify doesn't cut it. They have zero interest in the multiple streams > and hiding icons in streams type stuff. It also depends on who you ask. I can't take a lot of the mainstream X folks serious since they are still using integer math as parameters to half broken drawing primitives and barely discovered things like OpenGL. Their attitude doesn't treat these things as first class citizens in what ever software system they create. They also haven't create a modern and highly dynamic structured document system that's in wide use yet, so this problem space hasn't really been pushed as hard as other much more dynamic systems. And the advent of XML (basically a primitive and flat model of what Hans is doing) for .NET style systems are going to push these systems into those areas in new and unique ways. (Actually retro Smalltalk-ish) It seems that many of the original ideas about "why" GUI systems exist have been lost to older commericial interests (Microsoft Win32) and that has wiped out the fundamental classic computer science backing this from history. This simple "MP3 metadata" stuff is a very superficial example of how something like this is used. The problems are fundamentally about data representation in a manner so simple that its "expressive power" (Hans here) can extend itself to even the dorkiest of shell scripts. To have that power immediately available as network/local objects and to have their relationships clearly defined is a very powerful manner to build software systems. Unix folks tend to forget that since they either have never done this kind of programming or never understood why this existed in the first place. It's about a top-down methodology effecting the entire design of the software system, not just purity Unix. If it can be integrate smoothly into the system, then it should IMO. The folks against this system forget about how important the context of all this is work is set in... The mindset is fundamentally different and I'm quite sick of hearing "It's not Unix" over and over again. And notion of Linux being marginalize to a minority OS over this stuff is just plain crazy. bill ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 21:56 ` Bill Huey @ 2004-09-02 22:48 ` Alan Cox 2004-09-07 21:44 ` Bill Huey 0 siblings, 1 reply; 1392+ messages in thread From: Alan Cox @ 2004-09-02 22:48 UTC (permalink / raw) To: Bill Huey Cc: Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Iau, 2004-09-02 at 22:56, Bill Huey wrote: > It also depends on who you ask. I can't take a lot of the mainstream > X folks serious since they are still using integer math as parameters The X folks know what they are doing. Modern X has a complete compositing model. Modern X has a superb font handling system. Nobody broke anything along the way. The new API's can be mixed with the old, there are good fallbacks for old servers. In fact they are so good at it that most people don't notice beyond the fact their UI looks better than before. That is how you do change *right* > more dynamic systems. And the advent of XML (basically a primitive and > flat model of what Hans is doing) for .NET style systems are going to I see you don't really get XML either. XML is just an encoding. Its larger and prettier than ASN.1 and easier to hack about with perl. You can do the same thing with lisp lists for that matter. > have been lost to older commericial interests (Microsoft Win32) and that > has wiped out the fundamental classic computer science backing this from > history. This simple "MP3 metadata" stuff is a very superficial example > of how something like this is used. The trouble with computer science is that most of it sucks in the real world. We don't write our OS's in Standard ML, we don't implement some of the provably secure capability computing models. At the end of the day they are neat, elegant and useless to real people. > Unix folks tend to forget that since they either have never done this > kind of programming or never understood why this existed in the first > place. It's about a top-down methodology effecting the entire design of > the software system, not just purity Unix. If it can be integrate > smoothly into the system, then it should IMO. The Unix world succeeded because Unix (at least in v7 days) was the other way around to every other grungy OS on the planet. It had only thing things it needed. I've used some of the grungy crawly horrors that were its rivals and there is a reason they don't exist any more. I would sum up the essence of the unix kernel side as - Does only what it must do - "Makes the usual easy makes the unusual possible" - Has an API that is small enough for developers to learn easily (an API so good every other OS promptly ripped it off) People forget the worlds of SYS$QIO, RMS, FCB's and the like Its worked remarkably well for a very very long time, and most of the nasties have come from people trying to break that model or not understanding it. Alan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 22:48 ` Alan Cox @ 2004-09-07 21:44 ` Bill Huey 0 siblings, 0 replies; 1392+ messages in thread From: Bill Huey @ 2004-09-07 21:44 UTC (permalink / raw) To: Alan Cox Cc: Bill Huey, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, Sep 02, 2004 at 11:48:10PM +0100, Alan Cox wrote: > On Iau, 2004-09-02 at 22:56, Bill Huey wrote: > > It also depends on who you ask. I can't take a lot of the mainstream > > X folks serious since they are still using integer math as parameters > > The X folks know what they are doing. Modern X has a complete > compositing model. Modern X has a superb font handling system. Nobody > broke anything along the way. The new API's can be mixed with the old, > there are good fallbacks for old servers. Keith Packard knows what he's doing. The rest that don't follow his lead on these issues are useless or detracting and redirecting resources away from critical problems that have been solved under other, more mature, operating systems, specifically Apple's OS X. Notice, that you didn't deflect the claim that folks are still using integer math to draw things. This is critical... > In fact they are so good at it that most people don't notice beyond the > fact their UI looks better than before. > > That is how you do change *right* It's one way, but it's not top-level redesign heavily enough that you still have a substandard rendering system. Notice that were wasn't a mention of device independent rendering, WYSIWYG (what you get is what you see) in their rasterization model, nor was there any mention of how a scalable unified rasterization model used in an application. OpenStep ruled during it's time and rendered and generate better PostScript than Macintosh applications because of this unified rasterization model. Split development and the lack of "connecting the dots" across systems is one of the reasons why X has such poor app development. It takes way more time to do things in this environment than in other system. What folks X land are missing here is a notion of design verses piecewise hacks to half emulate a critical function. The fault in this matter is really spread out to multipule camps not doing their homework regarding these matters. Also, there's no mention of making OpenGL a first class citizen in this system so that all compositing is hardware accelerate. What OS has this ? You guess it Apple's OS X has this with full transparency and bitmap backstore for region invalidating and redrawing. X currently has none of this stuff. You could say that this is planned in the future, but there has been no push for doing this in this aggressive a manner which is a leadership failure of that community as a whole. Also, the notion of color space, not just RGB, but CMYK is a first class citizen in Display Postscript. X is also missing this. These aren't things that you can gradually add in. It's about looking at the system from the top-down and solving these problems using other systems that. X is at best piecewise and substandard to just about any modern rendering system that I've used. > > more dynamic systems. And the advent of XML (basically a primitive and > > flat model of what Hans is doing) for .NET style systems are going to > > I see you don't really get XML either. XML is just an encoding. Its > larger and prettier than ASN.1 and easier to hack about with perl. You > can do the same thing with lisp lists for that matter. I get it fine. What you don't get is something called Model View Controller in that .NET applications have the ability to take that serialized structure, apart of their entire object store/serialization system and transform it into many views. The same widget data can be used not only to build HTML for a web page, but also a standard component driven GUI system a native windowing kit. This exploits polymorphism and how OOP should have been done in the first place. This use of it is very important and obviously useful in this case. In MVC, all data has relationships and XML s the kind of DB format that these systems use for object serialization. XML is the storage layer in these systems. It's used for every object, remote or local. This is why something like Reiser FS 4 with it's persistent object store capability may have a critical influence on how these systems are built. These systems are returning nothing but structured storage in the form of XML analougs. Your Perl/Python, what ever example here doesn't cut it and explicate how XML based system are being generically used in this sense. It's too technically shallow for this context. > > have been lost to older commericial interests (Microsoft Win32) and that > > has wiped out the fundamental classic computer science backing this from > > history. This simple "MP3 metadata" stuff is a very superficial example > > of how something like this is used. > > The trouble with computer science is that most of it sucks in the real > world. We don't write our OS's in Standard ML, we don't implement some > of the provably secure capability computing models. At the end of the > day they are neat, elegant and useless to real people. Java and .NET are just those things that you have been talking about. As these systems become more and more prevalent (they will) they're influence is just the beginning. These system have pretty much happened overnight and can't be ignored by kernel folks. > > Unix folks tend to forget that since they either have never done this > > kind of programming or never understood why this existed in the first > > place. It's about a top-down methodology effecting the entire design of > > the software system, not just purity Unix. If it can be integrate > > smoothly into the system, then it should IMO. > > The Unix world succeeded because Unix (at least in v7 days) was the > other way around to every other grungy OS on the planet. It had only > thing things it needed. I've used some of the grungy crawly horrors that > were its rivals and there is a reason they don't exist any more. Look, I agree with you for the most part about Unix, but these issues have to be reexamined again after some period of time has elapsed. I can only guess that some of these system are horribly implemented minimally, but I can't comment about their original intentions since I've personally never used them. > I would sum up the essence of the unix kernel side as > - Does only what it must do > - "Makes the usual easy makes the unusual possible" > - Has an API that is small enough for developers to learn > easily (an API so good every other OS promptly ripped it off) > People forget the worlds of SYS$QIO, RMS, FCB's and the like > > Its worked remarkably well for a very very long time, and most of the > nasties have come from people trying to break that model or not > understanding it. This is a difficult topic, but I don't how folks can dimiss something like Reiser FS semantics and the importance of systems like this for future software design across all application boundaries. There has to be a way of getting this stuff in and keeping the traditional semantics in place. I was thinking about have a dual mounted file system, one with traditional Unix semantics with read-only permissions, but then having it also being mounted in a special namespace directory structure (something like /proc) that would reveal other streams for "stream aware" applications with read/write permissions. It might solve both problems in this case. Don't know. I've been on vacation for the last couple of days at Burning Man, which is why this response has been delayed. :) bill ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 17:46 ` Linus Torvalds 2004-09-02 17:50 ` Christoph Hellwig 2004-09-02 18:46 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Alan Cox @ 2004-09-02 19:50 ` Spam 2004-09-03 0:17 ` The argument for fs assistance in handling archives David Masover 2004-09-06 7:45 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Tonnerre 2 siblings, 2 replies; 1392+ messages in thread From: Spam @ 2004-09-02 19:50 UTC (permalink / raw) To: Linus Torvalds Cc: Alan Cox, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List > On Thu, 2 Sep 2004, Alan Cox wrote: >> >> On Mer, 2004-09-01 at 21:50, Linus Torvalds wrote: >> > and quite frankly, I think you can do the above pretty much totally in >> > user space with a small library and a daemon (in fact, ignoring security >> > issues you probably don't even need the daemon). And if you can prototype >> > it like that, and people actually find it useful, I suspect kernel support >> > for better performance might be possible. >> >> Gnome already supports this in the gnome-vfs2 layer. "MC" has supported >> it since the late 1990's. > And nobody has asked for kernel support that I know of. Actually. Doesn't matter if it is in kernel or not for the users as long as it works. The problem is that I do not see either Gnome or KDE to ever get along to form one standard that everyone will use. Their libraries are huge and memory hogging which so many Linux users just do not like. What if a user doesn't want KDE or Gnome? Would all files created with either be broken? I doubt that something like file streams and meta-data can successfully be implemented purely in user-space and get the same support (ie be used by many programs) if this change doesn't come from the kernel. I just do not see it happen. > So either "it just works" in user space, or people haven't figured out the > kernel could help them. Or decided it's not worth it, exactly because > they'd still have to support systems/filesystems that can't be converted. > Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 19:50 ` Spam @ 2004-09-03 0:17 ` David Masover 2004-09-06 7:45 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Tonnerre 1 sibling, 0 replies; 1392+ messages in thread From: David Masover @ 2004-09-03 0:17 UTC (permalink / raw) To: Spam Cc: Linus Torvalds, Alan Cox, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Spam wrote: [...] | I doubt that something like file streams and meta-data can | successfully be implemented purely in user-space and get the same | support (ie be used by many programs) if this change doesn't come | from the kernel. I just do not see it happen. The issue is not "many programs". The issue is "all programs". Even if the political issues were solved -- even if Linus said to us all "Thou shalt use this library or suffer my wrath!" -- it'd have to be everywhere. Bash. Perl. Make. Gcc. Vim. And btw, if it was political, you'd get no sympathy here. ("Oh no, everyone's using their own game engine! We need to put the doom3 engine in the kernel, now!") No, _people_ solve political problems. Kernels don't. Kernel support automatically adds support for a lot of features without patching a thing. There should be an interface in the filesystem. And for certain things, uservfs will be incredibly slower than reiser4 as that interface. (Remember Linus' point about TUX and Apache.) I'll say this again: most of it -- all but the bare interface stuff -- should be in userspace. In fact, let's all add this to our signature so no one brings it up again. ("Oh no, they want to put tar support in the kernel!" no, we don't.) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTe4BXgHNmZLgCUhAQKH1g//WXaL5xdLpX37TdgFhXidiRJGe/ojehj3 CZ7kseI9GkOBSxHt/yb5/xC6r+XT7JLlvJZybT7HLIRIxGp+WQHHBOD/xezWC+eX OyRaOlkZ7o9HRQKhKNRIAwI4jgftpLhFUhePgibubS4UdxtzN2FWuULfKvMKIGHn L4Zv4Dpje5ld7l7ce8jhfcURJ7AgAPwja3Tc7C38pmG+dSo2mj0I+YlCUED7mx3R ZSv6WtdAUCZjnKv9hSQVruk3fjYZc4dLEGzGH1ZJsD1ZkH5wNmWds5gHGEvQrc4Z 9reNanTxy+0ECxndk2H/ukw5Wv011rJWubLy/CnaPakPrSvrsmmoEs8ZcVZavlg9 ABJX/NtyBVl/y8+6Eh6/BdAhQr30U+c/UZLNbOflmcPGPiJCiXfBuaX1OF+qffQ1 QQvAGPgO2R9egHJWqFhBaLHtBAmiXSRWUU4+4nPBYZ/X5dCmGuV46knQGHdqoAQc l/qILh+spY09q9g118QbdnXBseiuVh/a+vf2GrbxbEMuWQu1kAI0DJbN0KKgUdtE ZkmIqXYULO6QZsYk3L41ZyKyE7oFMUqbT0uxSQZUCjOcnuBpMn/PzwM0yMJeDUKx 295Yzq5lkqkGmHJHi7XGOOI5XVIPb++DWXXv9E6Bfgoj4TSZscfwM69PQRoSq0hu iL9VOiyLRBU= =zrvZ -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 19:50 ` Spam 2004-09-03 0:17 ` The argument for fs assistance in handling archives David Masover @ 2004-09-06 7:45 ` Tonnerre 2004-09-06 8:05 ` Spam 1 sibling, 1 reply; 1392+ messages in thread From: Tonnerre @ 2004-09-06 7:45 UTC (permalink / raw) To: Spam Cc: Linus Torvalds, Alan Cox, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 1399 bytes --] Salut, On Thu, Sep 02, 2004 at 09:50:31PM +0200, Spam wrote: > Their libraries are huge and memory hogging which so many Linux > users just do not like. This is rather a fud argument: both the gnome VFS code and the KIOserver/KIOslave code aren't really large. You don't want to use them for a busybox/tinylogin system, however. > What if a user doesn't want KDE or Gnome? Would all files created > with either be broken? The files still work well, just that you can't access them over the old fancy URL schemes. > I doubt that something like file streams and meta-data can > successfully be implemented purely in user-space and get the same > support (ie be used by many programs) if this change doesn't come > from the kernel. I just do not see it happen. Actually, practical discordianism. If you develop a common API, there'll always be people disagreeing. GTK+ with all its features is just cool. Desktop warping is a really nice thing. But there are people out there who don't want to use it. They use QT, or even plain old Athena Widgets. So what? Will we be implementing the X toolkits into the kernel? In case of marketing it's up to the distributions to provide something concise so everyone can use their programs through a coherent namespace. (I.e. port all the apps they ship to gnome-vfs or kio). Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-06 7:45 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Tonnerre @ 2004-09-06 8:05 ` Spam 2004-09-06 8:56 ` Tonnerre 0 siblings, 1 reply; 1392+ messages in thread From: Spam @ 2004-09-06 8:05 UTC (permalink / raw) To: Tonnerre Cc: Linus Torvalds, Alan Cox, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List > Salut, > On Thu, Sep 02, 2004 at 09:50:31PM +0200, Spam wrote: >> Their libraries are huge and memory hogging which so many Linux >> users just do not like. > This is rather a fud argument: both the gnome VFS code and the > KIOserver/KIOslave code aren't really large. You don't want to use > them for a busybox/tinylogin system, however. Then it is good. Just I see no programs other than Gnome or KDE apps that are using them. >> What if a user doesn't want KDE or Gnome? Would all files created >> with either be broken? > The files still work well, just that you can't access them over the > old fancy URL schemes. No only, but if you cp then with a non KDE/Gnome app then you will loose the meta-data and extra info too. That was my point. >> I doubt that something like file streams and meta-data can >> successfully be implemented purely in user-space and get the same >> support (ie be used by many programs) if this change doesn't come >> from the kernel. I just do not see it happen. > Actually, practical discordianism. If you develop a common API, > there'll always be people disagreeing. > GTK+ with all its features is just cool. Desktop warping is a really > nice thing. But there are people out there who don't want to use > it. They use QT, or even plain old Athena Widgets. So what? Will we be > implementing the X toolkits into the kernel? This is certainly not what I said or wanted. > In case of marketing it's up to the distributions to provide something > concise so everyone can use their programs through a coherent > namespace. (I.e. port all the apps they ship to gnome-vfs or kio). Do you really believe this will happen? Good if it did. I do not believe it. And I certainly do not see the thousands of man hours needed to actually provide all the patches as a benefit. ~S > Tonnerre ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-06 8:05 ` Spam @ 2004-09-06 8:56 ` Tonnerre 2004-09-06 9:15 ` The argument for fs assistance in handling archives Dr. Giovanni A. Orlando 0 siblings, 1 reply; 1392+ messages in thread From: Tonnerre @ 2004-09-06 8:56 UTC (permalink / raw) To: Spam Cc: Linus Torvalds, Alan Cox, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 1257 bytes --] Salut, On Mon, Sep 06, 2004 at 10:05:34AM +0200, Spam wrote: > Then it is good. Just I see no programs other than Gnome or KDE apps > that are using them. Because KDE people hate Gnome people and vice versa, and because the rest of the world just neglects the two races for political reasons. Maybe the Freedesktop project should provide some convenient specification/code to do it. Like they do for HAL and DBUS (Please note that this is something interesting because it does clever things on hardware without requiring to patch the kernel.) > > In case of marketing it's up to the distributions to provide > > something concise so everyone can use their programs through a > > coherent namespace. (I.e. port all the apps they ship to gnome-vfs > > or kio). > > Do you really believe this will happen? If the distributors really want to be able to gain money, and if the Free Unix community wants to gain a significant market share, this is supposed to happen. It's the question of whether we can ignore our childish concept wars, or if we're always going to stay at that low level we're at now. Actually, this can't be fixed by putting everything into the kernel. Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-06 8:56 ` Tonnerre @ 2004-09-06 9:15 ` Dr. Giovanni A. Orlando 2004-09-06 12:55 ` Grzegorz Jaśkiewicz 0 siblings, 1 reply; 1392+ messages in thread From: Dr. Giovanni A. Orlando @ 2004-09-06 9:15 UTC (permalink / raw) To: Tonnerre Cc: Spam, Linus Torvalds, Alan Cox, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List Tonnerre wrote: >Salut, > >On Mon, Sep 06, 2004 at 10:05:34AM +0200, Spam wrote: > > >>Then it is good. Just I see no programs other than Gnome or KDE apps >>that are using them. >> >> > >Because KDE people hate Gnome people and vice versa, and because the >rest of the world just neglects the two races for political reasons. > > Hi, This is completely wrong. Neither KDE people hate GNOME people nor viceversa. Time ago, I repeat in a conference the GNOME people need to jump on the KDE wagon, so the train will move to a single direction, and people does not approve this comment. More than on the license, QPL, there are a developer problem. Develop in C (Gtk, GNOME) is a lot more easy than develop in C++ (Qt, KDE) But, for the GUI, the C++ approach is a lot superior ... a lot. The problem is this. They start to develop on Gtk to create GNOME, in C for a license problem. Now, they said why I need to change? But they need to change, because it is supeiror. Everyone may of course wants to do what prefer. >Maybe the Freedesktop project should provide some convenient >specification/code to do it. Like they do for HAL and DBUS (Please >note that this is something interesting because it does clever things >on hardware without requiring to patch the kernel.) > > > I don't agree FreeDesktop.org, because it is handled by a RedHat employee and the code is made on Gtk2, generally. ... They are not neutral. Thanks, Giovanni >>>In case of marketing it's up to the distributions to provide >>>something concise so everyone can use their programs through a >>>coherent namespace. (I.e. port all the apps they ship to gnome-vfs >>>or kio). >>> >>> >>Do you really believe this will happen? >> >> > >If the distributors really want to be able to gain money, and if the >Free Unix community wants to gain a significant market share, this is >supposed to happen. It's the question of whether we can ignore our >childish concept wars, or if we're always going to stay at that low >level we're at now. > >Actually, this can't be fixed by putting everything into the kernel. > > Tonnerre > > -- -- -- Check FT Websites ... http://www.futuretg.com - ftp://ftp.futuretg.com http://www.FTLinuxCourse.com http://www.FTLinuxCourse.com/Certification http://www.rpmparadaise.org http://GNULinuxUtilities.com http://www.YourPersonalOperatingSystem.com -- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-06 9:15 ` The argument for fs assistance in handling archives Dr. Giovanni A. Orlando @ 2004-09-06 12:55 ` Grzegorz Jaśkiewicz 0 siblings, 0 replies; 1392+ messages in thread From: Grzegorz Jaśkiewicz @ 2004-09-06 12:55 UTC (permalink / raw) To: Dr. Giovanni A. Orlando Cc: Tonnerre, Spam, Linus Torvalds, Alan Cox, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Mon, 06 Sep 2004 11:15:59 +0200, Dr. Giovanni A. Orlando <gorlando@futuretg.com> wrote: > Tonnerre wrote: > >Because KDE people hate Gnome people and vice versa, and because the > >rest of the world just neglects the two races for political reasons. > This is completely wrong. Neither KDE people hate GNOME people > nor viceversa. > > Time ago, I repeat in a conference the GNOME people need to jump > on the KDE wagon, so the train will move to a single direction, and > people does not approve this comment. > > More than on the license, QPL, there are a developer problem. KDE is covered with LGPL/GPL licence, so is QT, so I don't see a problem here. > Develop in C (Gtk, GNOME) is a lot more easy than develop in C++ > (Qt, KDE) Well, some ppl knowing well C++ will say something completly different. Since I've started QT/KDE development, my life as developer has changed. I am developing everything in QT, as far as desktop stuff goes. And it's much more clear and easier. You just need to lear C++ well ;) > But, for the GUI, the C++ approach is a lot superior ... a lot. > > The problem is this. They start to develop on Gtk to create GNOME, in C > for a license problem. > I don't agree FreeDesktop.org, because it is handled by a RedHat employee > and the code is made on Gtk2, generally. > > ... They are not neutral. Agreed. -- GJ ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 9:46 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Alan Cox 2004-09-02 16:11 ` Jamie Lokier 2004-09-02 17:46 ` Linus Torvalds @ 2004-09-02 20:38 ` Frank van Maarseveen 2004-09-02 21:36 ` Dave Kleikamp ` (2 more replies) 2 siblings, 3 replies; 1392+ messages in thread From: Frank van Maarseveen @ 2004-09-02 20:38 UTC (permalink / raw) To: Alan Cox Cc: Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, Sep 02, 2004 at 10:46:05AM +0100, Alan Cox wrote: > On Mer, 2004-09-01 at 21:50, Linus Torvalds wrote: > > and quite frankly, I think you can do the above pretty much totally in > > user space with a small library and a daemon (in fact, ignoring security > > issues you probably don't even need the daemon). And if you can prototype > > it like that, and people actually find it useful, I suspect kernel support > > for better performance might be possible. > > Gnome already supports this in the gnome-vfs2 layer. "MC" has supported > it since the late 1990's. Can it do this: cd FC2-i386-disc1.iso ls or this: cd /dev/sda1 ls cd /dev/floppy ls cd /dev/cdrom ls ? -- Frank ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 20:38 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Frank van Maarseveen @ 2004-09-02 21:36 ` Dave Kleikamp 2004-09-02 21:48 ` Frank van Maarseveen 2004-09-02 23:19 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Valdis.Kletnieks 2004-09-06 7:56 ` Tonnerre 2 siblings, 1 reply; 1392+ messages in thread From: Dave Kleikamp @ 2004-09-02 21:36 UTC (permalink / raw) To: Frank van Maarseveen Cc: Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2004-09-02 at 15:38, Frank van Maarseveen wrote: > Can it do this: > > cd FC2-i386-disc1.iso > ls > > or this: > > cd /dev/sda1 > ls > cd /dev/floppy > ls > cd /dev/cdrom > ls > > ? We have the mount command for that. :^) -- David Kleikamp IBM Linux Technology Center ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 21:36 ` Dave Kleikamp @ 2004-09-02 21:48 ` Frank van Maarseveen 2004-09-02 22:00 ` viro 2004-09-06 12:53 ` Clemens Schwaighofer 0 siblings, 2 replies; 1392+ messages in thread From: Frank van Maarseveen @ 2004-09-02 21:48 UTC (permalink / raw) To: Dave Kleikamp Cc: Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, Sep 02, 2004 at 04:36:34PM -0500, Dave Kleikamp wrote: > On Thu, 2004-09-02 at 15:38, Frank van Maarseveen wrote: > > Can it do this: > > > > cd FC2-i386-disc1.iso > > ls > > > > or this: > > > > cd /dev/sda1 > > ls > > cd /dev/floppy > > ls > > cd /dev/cdrom > > ls > > > > ? > > We have the mount command for that. :^) mount is nice for root, clumsy for user. And a rather complicated way of accessing data the kernel has knowledge about in the first place. For filesystem images, cd'ing into the file is the most obvious concept for file-as-a-dir IMHO. -- Frank ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 21:48 ` Frank van Maarseveen @ 2004-09-02 22:00 ` viro 2004-09-02 22:02 ` Frank van Maarseveen 2004-09-03 0:08 ` The argument for fs assistance in handling archives David Masover 2004-09-06 12:53 ` Clemens Schwaighofer 1 sibling, 2 replies; 1392+ messages in thread From: viro @ 2004-09-02 22:00 UTC (permalink / raw) To: Frank van Maarseveen Cc: Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, Sep 02, 2004 at 11:48:06PM +0200, Frank van Maarseveen wrote: > mount is nice for root, clumsy for user. And a rather complicated > way of accessing data the kernel has knowledge about in the first > place. For filesystem images, cd'ing into the file is the most > obvious concept for file-as-a-dir IMHO. The hell it is. a) kernel has *NO* *FUCKING* *KNOWLEDGE* of fs type contained on a device. b) kernel has no way to guess which options to use c) fs _type_ is a fundamental part of mount - device(s) (if any) involved are arguments to be interpreted by that particular fs driver. d) permissions required for that lovely operation (and questions like whether we force nosuid/noexec, etc.) are nightmare to define. Frankly, the longer that thread grows, the more obvious it becomes that file-as-a-dir is a solution in search of problem. Desperate search, at that. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 22:00 ` viro @ 2004-09-02 22:02 ` Frank van Maarseveen 2004-09-02 22:03 ` Christoph Hellwig 2004-09-02 22:06 ` viro 2004-09-03 0:08 ` The argument for fs assistance in handling archives David Masover 1 sibling, 2 replies; 1392+ messages in thread From: Frank van Maarseveen @ 2004-09-02 22:02 UTC (permalink / raw) To: viro Cc: Frank van Maarseveen, Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, Sep 02, 2004 at 11:00:27PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote: > > The hell it is. > > a) kernel has *NO* *FUCKING* *KNOWLEDGE* of fs type contained on a device. excuse me, but how does the kernel mount the root fs? -- Frank ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 22:02 ` Frank van Maarseveen @ 2004-09-02 22:03 ` Christoph Hellwig 2004-09-02 22:04 ` Christoph Hellwig 2004-09-02 22:06 ` viro 1 sibling, 1 reply; 1392+ messages in thread From: Christoph Hellwig @ 2004-09-02 22:03 UTC (permalink / raw) To: Frank van Maarseveen Cc: viro, Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Fri, Sep 03, 2004 at 12:02:42AM +0200, Frank van Maarseveen wrote: > On Thu, Sep 02, 2004 at 11:00:27PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote: > > > > The hell it is. > > > > a) kernel has *NO* *FUCKING* *KNOWLEDGE* of fs type contained on a device. > > excuse me, but how does the kernel mount the root fs? trial and error. That's why you see all thos ext3 mounted as ext2 problems, or $RANDOMFS as fat. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 22:03 ` Christoph Hellwig @ 2004-09-02 22:04 ` Christoph Hellwig 0 siblings, 0 replies; 1392+ messages in thread From: Christoph Hellwig @ 2004-09-02 22:04 UTC (permalink / raw) To: Christoph Hellwig, Frank van Maarseveen, viro, Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Fri, Sep 03, 2004 at 12:03:54AM +0200, Christoph Hellwig wrote: > On Fri, Sep 03, 2004 at 12:02:42AM +0200, Frank van Maarseveen wrote: > > On Thu, Sep 02, 2004 at 11:00:27PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote: > > > > > > The hell it is. > > > > > > a) kernel has *NO* *FUCKING* *KNOWLEDGE* of fs type contained on a device. > > > > excuse me, but how does the kernel mount the root fs? > > trial and error. That's why you see all thos ext3 mounted as ext2 > problems, or $RANDOMFS as fat. Andb btw, for an lkml discussion RTFS wouldn't hurt. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 22:02 ` Frank van Maarseveen 2004-09-02 22:03 ` Christoph Hellwig @ 2004-09-02 22:06 ` viro 2004-09-02 22:11 ` Frank van Maarseveen 1 sibling, 1 reply; 1392+ messages in thread From: viro @ 2004-09-02 22:06 UTC (permalink / raw) To: Frank van Maarseveen Cc: Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Fri, Sep 03, 2004 at 12:02:42AM +0200, Frank van Maarseveen wrote: > On Thu, Sep 02, 2004 at 11:00:27PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote: > > > > The hell it is. > > > > a) kernel has *NO* *FUCKING* *KNOWLEDGE* of fs type contained on a device. > > excuse me, but how does the kernel mount the root fs? By trying all fs types it has registered in a more or less random (OK, defined by order of fs type registration, which is kinda-sorta deterministic at boot time) order. With no flags, unless you pass them explicitly in kernel command line. Fs types list can also be set explicitly in the command line. Next question? ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 22:06 ` viro @ 2004-09-02 22:11 ` Frank van Maarseveen 2004-09-02 22:17 ` viro 0 siblings, 1 reply; 1392+ messages in thread From: Frank van Maarseveen @ 2004-09-02 22:11 UTC (permalink / raw) To: viro Cc: Frank van Maarseveen, Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, Sep 02, 2004 at 11:06:40PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote: > On Fri, Sep 03, 2004 at 12:02:42AM +0200, Frank van Maarseveen wrote: > > > a) kernel has *NO* *FUCKING* *KNOWLEDGE* of fs type contained on a device. > > > > excuse me, but how does the kernel mount the root fs? > > By trying all fs types it has registered in a more or less random (OK, defined > by order of fs type registration, which is kinda-sorta deterministic at > boot time) order. With no flags, unless you pass them explicitly in kernel > command line. Fs types list can also be set explicitly in the command line. Of course I know that: the point is, the kernel _has_ knowlegde contrary to what you blatantly said. -- Frank ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 22:11 ` Frank van Maarseveen @ 2004-09-02 22:17 ` viro 2004-09-02 22:26 ` Frank van Maarseveen 0 siblings, 1 reply; 1392+ messages in thread From: viro @ 2004-09-02 22:17 UTC (permalink / raw) To: Frank van Maarseveen Cc: Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Fri, Sep 03, 2004 at 12:11:33AM +0200, Frank van Maarseveen wrote: > On Thu, Sep 02, 2004 at 11:06:40PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote: > > On Fri, Sep 03, 2004 at 12:02:42AM +0200, Frank van Maarseveen wrote: > > > > a) kernel has *NO* *FUCKING* *KNOWLEDGE* of fs type contained on a device. > > > > > > excuse me, but how does the kernel mount the root fs? > > > > By trying all fs types it has registered in a more or less random (OK, defined > > by order of fs type registration, which is kinda-sorta deterministic at > > boot time) order. With no flags, unless you pass them explicitly in kernel > > command line. Fs types list can also be set explicitly in the command line. > > Of course I know that: the point is, the kernel _has_ knowlegde contrary > to what you blatantly said. What knowledge does the kernel have about fs type that could deal with the contents of given device? Details, please. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 22:17 ` viro @ 2004-09-02 22:26 ` Frank van Maarseveen 2004-09-02 22:33 ` viro 2004-09-06 8:04 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Tonnerre 0 siblings, 2 replies; 1392+ messages in thread From: Frank van Maarseveen @ 2004-09-02 22:26 UTC (permalink / raw) To: viro Cc: Frank van Maarseveen, Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, Sep 02, 2004 at 11:17:22PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote: > > What knowledge does the kernel have about fs type that could deal with the > contents of given device? Details, please. Try a "make tags;grep SUPER_MAGIC tags". Or is it there for a different purpose? -- Frank ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 22:26 ` Frank van Maarseveen @ 2004-09-02 22:33 ` viro 2004-09-02 22:56 ` Frank van Maarseveen 2004-09-06 8:04 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Tonnerre 1 sibling, 1 reply; 1392+ messages in thread From: viro @ 2004-09-02 22:33 UTC (permalink / raw) To: Frank van Maarseveen Cc: Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Fri, Sep 03, 2004 at 12:26:50AM +0200, Frank van Maarseveen wrote: > On Thu, Sep 02, 2004 at 11:17:22PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote: > > > > What knowledge does the kernel have about fs type that could deal with the > > contents of given device? Details, please. > > Try a "make tags;grep SUPER_MAGIC tags". > Or is it there for a different purpose? RTFS and you'll see. Individual fs generally knows how to check if it would be immediately unhappy with given image (not all types do, BTW). Exact form of checks depends on fs type; for crying out loud, there's not even a promise that they are mutually exclusive! Read the fucking source. Read through the code that "chooses" fs type of root fs. Look at it. Then use whatever you have between your ears. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 22:33 ` viro @ 2004-09-02 22:56 ` Frank van Maarseveen 2004-09-02 23:02 ` viro 2004-09-03 8:43 ` Helge Hafting 0 siblings, 2 replies; 1392+ messages in thread From: Frank van Maarseveen @ 2004-09-02 22:56 UTC (permalink / raw) To: viro Cc: Frank van Maarseveen, Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, Sep 02, 2004 at 11:33:24PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote: > > RTFS and you'll see. Individual fs generally knows how to check if it > would be immediately unhappy with given image (not all types do, BTW). > Exact form of checks depends on fs type; for crying out loud, there's > not even a promise that they are mutually exclusive! so? A user can stick an USB memory card with _any_ malformed fs data and make troubles via the automounter or user mounts. Yes, mount might do some more checks but it sure won't do an fsck. The user gets what he deserves when sticking crap in an USB port. And that doesn't mean that the kernel should accept any fs image when a user tries to cd into the file. -- Frank ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 22:56 ` Frank van Maarseveen @ 2004-09-02 23:02 ` viro 2004-09-03 8:43 ` Helge Hafting 1 sibling, 0 replies; 1392+ messages in thread From: viro @ 2004-09-02 23:02 UTC (permalink / raw) To: Frank van Maarseveen Cc: Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Fri, Sep 03, 2004 at 12:56:34AM +0200, Frank van Maarseveen wrote: > And that doesn't mean that the kernel should accept any fs image > when a user tries to cd into the file. No, it shouldn't. It should say "it's not a directory, bugger off". Which it does. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 22:56 ` Frank van Maarseveen 2004-09-02 23:02 ` viro @ 2004-09-03 8:43 ` Helge Hafting 2004-09-03 8:50 ` Frank van Maarseveen 2004-09-03 23:55 ` The argument for fs assistance in handling archives David Masover 1 sibling, 2 replies; 1392+ messages in thread From: Helge Hafting @ 2004-09-03 8:43 UTC (permalink / raw) To: Frank van Maarseveen Cc: viro, Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List Frank van Maarseveen wrote: >On Thu, Sep 02, 2004 at 11:33:24PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote: > > >>RTFS and you'll see. Individual fs generally knows how to check if it >>would be immediately unhappy with given image (not all types do, BTW). >>Exact form of checks depends on fs type; for crying out loud, there's >>not even a promise that they are mutually exclusive! >> >> > >so? > >A user can stick an USB memory card with _any_ malformed fs data and >make troubles via the automounter or user mounts. Yes, mount might do >some more checks but it sure won't do an fsck. > >The user gets what he deserves when sticking crap in an USB port. > >And that doesn't mean that the kernel should accept any fs image >when a user tries to cd into the file. > > You don't need kernel support for cd'ing into fs images. You need a shell (or GUI app) that: 1. notices that user tries to CD into a file, not a directory 2. Attempts fs type detection and do a loop mount. 3. Give error message if it wasn't a supported fs image. Helge Hafting ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-03 8:43 ` Helge Hafting @ 2004-09-03 8:50 ` Frank van Maarseveen 2004-09-03 8:55 ` Frank van Maarseveen 2004-09-03 23:55 ` The argument for fs assistance in handling archives David Masover 1 sibling, 1 reply; 1392+ messages in thread From: Frank van Maarseveen @ 2004-09-03 8:50 UTC (permalink / raw) To: Helge Hafting Cc: Frank van Maarseveen, viro, Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Fri, Sep 03, 2004 at 10:43:44AM +0200, Helge Hafting wrote: > > > You don't need kernel support for cd'ing into fs images. > You need a shell (or GUI app) that: > 1. notices that user tries to CD into a file, not a directory > 2. Attempts fs type detection and do a loop mount. > 3. Give error message if it wasn't a supported fs image. Ok, and right now I'm in vim typing this message and want to ":new /tmp/backup.iso/etc/fstab" modifying 1000+ applications is not an option IMO. Putting it in a preloaded library might be doable except maybe for permission problems but this is incredibly clumsy and lacks the possibility to implement sane caching behavior because it is process bound. -- Frank ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-03 8:50 ` Frank van Maarseveen @ 2004-09-03 8:55 ` Frank van Maarseveen 0 siblings, 0 replies; 1392+ messages in thread From: Frank van Maarseveen @ 2004-09-03 8:55 UTC (permalink / raw) To: Frank van Maarseveen Cc: Helge Hafting, viro, Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Fri, Sep 03, 2004 at 10:50:18AM +0200, I wrote: > and lacks the possibility > to implement sane caching behavior because it is process bound. sorry, this part is crap. -- Frank ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 8:43 ` Helge Hafting 2004-09-03 8:50 ` Frank van Maarseveen @ 2004-09-03 23:55 ` David Masover 2004-09-08 9:56 ` Helge Hafting 1 sibling, 1 reply; 1392+ messages in thread From: David Masover @ 2004-09-03 23:55 UTC (permalink / raw) To: Helge Hafting Cc: Frank van Maarseveen, viro, Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Helge Hafting wrote: [...] | You don't need kernel support for cd'ing into fs images. | You need a shell (or GUI app) that: | 1. notices that user tries to CD into a file, not a directory | 2. Attempts fs type detection and do a loop mount. | 3. Give error message if it wasn't a supported fs image. You can argue this about FS plugins, too. You don't need kernel support for cryptocompress, for example. Just have a fifo farm. Every file that has compression turned on is actually a named pipe, the original file is hidden (starts with a dot), and you have a script running to each fifo which runs zcat. The kernel support (read: interface) is just cleaner in some ways. In the loop mount example, it isn't universal -- the user can't CD into a file if they use the wrong shell, and two different shells might make two different mounts. How many mounts before we run out of loop devices? In the cryptocompress example, there's no caching or compress-on-flush optimization, and it isn't entirely transparent to anything. It doesn't look like a file, it looks like a fifo. ls will make it look yellow. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTkEWngHNmZLgCUhAQKmMhAAkECgB9hUCxM2vNn0NgzQmkuz028QsAEN 6B1ej6tCbVyT8whsVD2YA7rQH5UavUeUKjuFNJNwelXeEYqG8tbv4vcE1Z65U5tR GUJkDxWwgjfx4JtE137VmWXcMWW/6FeIBbHuKFohDeBz2nBYmc3w/b4NjgT//JOG 0iUQGRDQcGxDuCYqMqCO8+m/LOSL9wbYDG+bXa4NxVAcghmxAhAJFiDpRDQHcYxX CeUI7ZkFnNX/4A52YTAFyHQmeoQRX1SLxJsn0GH9mcf2Gsxs8AtoTQgtedDsz9I2 Ct6Dil6NfI94VfOx0EWa1y4I0p7UnFjAsye9zeElhumhocFvEcd5Lzn6SnCY/rQh o4VukNKUtDE92noL62BZGgCc9ek3/YZivMMFfmQOrjKjfQxEMwSvyjKWIj9XPATT FWkN7mZn7JSxluB1n1BtHTYvdieD65RGWF1aW1atJZ1r0z0xzy95LF/kqm9Rkigr zv6+1j9ND79pBikYuq4ol916R9bW93TkriSYbENBP5dtMx4nXUYI6kRU53Y1UxZH +mstouIpGr9MB7kHLLDYFFFV8upRJi2EQHA2sAtHOaMmjZUoJrbsIdA9n1r1LsKt 6Wjq8MofdyvZWXbzh9aGD4QkovNf6tw4UHAGzVf9Uh3n6mwh7yw4qyDC5h8dqMEe P4wWkRawtHc= =GC9p -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 23:55 ` The argument for fs assistance in handling archives David Masover @ 2004-09-08 9:56 ` Helge Hafting 0 siblings, 0 replies; 1392+ messages in thread From: Helge Hafting @ 2004-09-08 9:56 UTC (permalink / raw) To: David Masover Cc: Frank van Maarseveen, viro, Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List David Masover wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Helge Hafting wrote: > [...] > | You don't need kernel support for cd'ing into fs images. > | You need a shell (or GUI app) that: > | 1. notices that user tries to CD into a file, not a directory > | 2. Attempts fs type detection and do a loop mount. > | 3. Give error message if it wasn't a supported fs image. > > You can argue this about FS plugins, too. You don't need kernel support > for cryptocompress, for example. Just have a fifo farm. Every file > that has compression turned on is actually a named pipe, the original > file is hidden (starts with a dot), and you have a script running to > each fifo which runs zcat. Eww. A shell/filemanager supporting fs images is one running program. The fifo farm is one per file. Ouch. > > The kernel support (read: interface) is just cleaner in some ways. > > In the loop mount example, it isn't universal -- the user can't CD into > a file if they use the wrong shell, and two different shells might make > two different mounts. How many mounts before we run out of loop devices? This limitation apply to the kernel interface too. But you can configure for an arbitrary number of loop mounts anyway. The user will have to mount using the correct tool - after that the door is opened for anything. > > In the cryptocompress example, there's no caching or compress-on-flush > optimization, and it isn't entirely transparent to anything. It doesn't > look like a file, it looks like a fifo. ls will make it look yellow. Even the kernel supported "cd into fs images" has problems. How should the kernel know which files you want to support with mounting, and which ones are the wast majority of plain files? A file manager doesn't have to do this fully automatic, it can have a button/hotkey for "loop mount this". So the user will have to tell it what to do, but in a much easier way than typing "mount -o loop . . . " The file manager can also ask for encryption keys - something the kernel cannot do because it doesn't know what user interface(s) is in use by the process that attempts the "cd". There may be one interface (stdin, X) there may be both, i.e. something started from an xterm can use both, there may be none, (the web/ftp/file-server got a request for something inside a encrypted fs image.) And even if the kernel can figure out the correct user interface, do we want it to contain an X client for asking for keys? :-/ Helge Hafting ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 22:26 ` Frank van Maarseveen 2004-09-02 22:33 ` viro @ 2004-09-06 8:04 ` Tonnerre 2004-09-06 8:41 ` Frank van Maarseveen 1 sibling, 1 reply; 1392+ messages in thread From: Tonnerre @ 2004-09-06 8:04 UTC (permalink / raw) To: Frank van Maarseveen Cc: viro, Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 787 bytes --] Salut, On Fri, Sep 03, 2004 at 12:26:50AM +0200, Frank van Maarseveen wrote: > Try a "make tags;grep SUPER_MAGIC tags". > Or is it there for a different purpose? Problem is: There are cool superblock magics for reiserfs. And for ext[23]. And even for good old Minix. Cool. However, there are also ugly file systems, such as fat for example. Fat has been defined as "something that can be read by a fat driver", which can be pretty much anything. You can't really detect whether it is fat. Even Microsoft only guess whether some FS is fat or not. And don't say detect it via a partition table. There are a lot of cases where you get absolutely no or wrong information out of it, and there are also lots of cases where you plainly have none. Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-06 8:04 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Tonnerre @ 2004-09-06 8:41 ` Frank van Maarseveen 0 siblings, 0 replies; 1392+ messages in thread From: Frank van Maarseveen @ 2004-09-06 8:41 UTC (permalink / raw) To: Tonnerre Cc: Frank van Maarseveen, viro, Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Mon, Sep 06, 2004 at 10:04:24AM +0200, Tonnerre wrote: > > Problem is: > > There are cool superblock magics for reiserfs. And for ext[23]. And > even for good old Minix. Cool. > > However, there are also ugly file systems, such as fat for > example. Fat has been defined as "something that can be read by a fat This problem is not new. Kernel probes for filesystems in a particular order for mounting the root fs. And mount understands the fstype=auto in /etc/fstab. There is no perfect solution but it's sure possible to come up with something acceptable and workable with not too much effort, for a configuratble subset of file-systems. This is not that much different from an automounter/usermount mounting a USB storage device or cdrom: ext3, ext2, udf, iso9660, vfat, read-only or not, just to name a few things. This should work anyway. -- Frank ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 22:00 ` viro 2004-09-02 22:02 ` Frank van Maarseveen @ 2004-09-03 0:08 ` David Masover 2004-09-03 0:18 ` Linus Torvalds ` (4 more replies) 1 sibling, 5 replies; 1392+ messages in thread From: David Masover @ 2004-09-03 0:08 UTC (permalink / raw) To: viro Cc: Frank van Maarseveen, Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 viro@parcelfarce.linux.theplanet.co.uk wrote: | On Thu, Sep 02, 2004 at 11:48:06PM +0200, Frank van Maarseveen wrote: | |>mount is nice for root, clumsy for user. And a rather complicated |>way of accessing data the kernel has knowledge about in the first |>place. For filesystem images, cd'ing into the file is the most |>obvious concept for file-as-a-dir IMHO. | | | The hell it is. | | a) kernel has *NO* *FUCKING* *KNOWLEDGE* of fs type contained on a device. reiser4 kernel will contain knowledge of fs type contained in a file. "file/..metas/type" might contain a mime type. Mime type might have to be guessed, but at least if it's made by a local "mkisofs" then we're fine. Indeed, that's not the only interface that's been discussed. "file/..metas/is_isofs" might be consulted. | b) kernel has no way to guess which options to use | c) fs _type_ is a fundamental part of mount - device(s) (if any) involved | are arguments to be interpreted by that particular fs driver. Unless there's some severe security issues with "mount this iso as fat and you get root access", this should work fine. I see no reason why there can't be a global setting of the mount commandline to use. And it doesn't all have to be in the kernel. Only it'd be nice to have some of it there because the kernel knows how to deal with an isofs, even if it won't know what it looks like. | d) permissions required for that lovely operation (and questions like | whether we force nosuid/noexec, etc.) are nightmare to define. They are quite simple, actually. Just set them globally -- some admins would force nosetuid/noexec, some wouldn't. And the operation happens transparently -- you need no "permissions" other than to read the directory which would contain the mount. | Frankly, the longer that thread grows, the more obvious it becomes that | file-as-a-dir is a solution in search of problem. Desperate search, at | that. Actually, the longer this thread grows, the more obvious it is how when there's a hot issue, everyone has an opinion, even if the same opinion has been expressed ten or twenty times already. File-as-a-dir has numerous advantages, but enough have been discussed. Short list is image mounts, tarballs, streams, metas, and namespace unification. Longer list and explanations can be found if you RTFA. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTe19XgHNmZLgCUhAQJdow/+Knsw1GgpauqDUcg8sKtxzgXZ18OxMQ3Q By8sRrSTuKAzI5A3BtYIzndsj1veP+7wndG7nYPz8NS1fU2+xWSIhoGq/YMaQsu4 70uMLu448PFXZua4hZMk2w4mkULXbGyYHJ1Bf+2Z7QkQ/8W08hozC8QQynxMXIkX SrcWCS5hK8Nh7Ol691sDpPqexH7F1GwUyoslNGj63U5r6ViLAawt2ZKDYdT7ZPo8 0a/pWUHoHMPbv/KwqZZxRr1/qncA9QYQo6JqQBPPCr+tWNJs/ei3nAKGi58iOt1M DK1TEKd2lpbmwiK5pWDwGz+nwWmaFTAyfTEEEcP4gZedSJtRXaxyNh0jRl1iLATB SCO5Eb4jkQs8hdjHqQcQ1q7XKFX9eSXWeDdrGrtWaYC/QYOHxT+ci3lnKBKCG99Y YTqg3sNEZlV1N0jIcNvFSDEYbbX12v1Y6xbwvUx48+sMyUj3suT76niTRbwEydfO MA9y+wE2k4wF+h+sJCbTjimCNFvvuFTTJuBCbQTpfY4eOYBAFalxnWmrpTEfVzka 4iAqAYygWObGDkFFy/rp1HEVZPIKM0NwGLOsRwJsgyUMOsccBrEc0bg8sgMECVfs 5qNIb27tLokh8NBR6RodAv2NZYKC+foM0T+PC5bZMFD/Q7f6yDklqK4C4RCIYaXj xO9z1C6FPcM= =/gmK -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 0:08 ` The argument for fs assistance in handling archives David Masover @ 2004-09-03 0:18 ` Linus Torvalds 2004-09-03 1:25 ` David Masover 2004-09-03 5:22 ` Hans Reiser 2004-09-03 0:39 ` viro ` (3 subsequent siblings) 4 siblings, 2 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-09-03 0:18 UTC (permalink / raw) To: David Masover Cc: viro, Frank van Maarseveen, Dave Kleikamp, Alan Cox, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2 Sep 2004, David Masover wrote: > > reiser4 kernel will contain knowledge of fs type contained in a file. That's a disaster, btw. There is no one "fs type" of a file. Files have at _least_ one type (bytestream), but most have more. Which is why automatically doing the right thing (in the sense you seem to want) in kernel space is simply not possible. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 0:18 ` Linus Torvalds @ 2004-09-03 1:25 ` David Masover 2004-09-03 5:22 ` Hans Reiser 1 sibling, 0 replies; 1392+ messages in thread From: David Masover @ 2004-09-03 1:25 UTC (permalink / raw) To: Linus Torvalds Cc: viro, Frank van Maarseveen, Dave Kleikamp, Alan Cox, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Linus Torvalds wrote: | | On Thu, 2 Sep 2004, David Masover wrote: | |>reiser4 kernel will contain knowledge of fs type contained in a file. | | | That's a disaster, btw. | | There is no one "fs type" of a file. Files have at _least_ one type | (bytestream), but most have more. Which is why automatically doing the | right thing (in the sense you seem to want) in kernel space is simply not | possible. Oops, my bad. The interface is kernel, the file type database is user. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTfH7XgHNmZLgCUhAQL0aw//SG/RDCy30xcVh36FVVAOG7GjP6UkHBeo 14/O3hSELxwm9r8z0YnGnbpoJj9atbOCu6VJhiumtjeyikdH5OaOrcMo7DQYp47V holyEKvjd+YUkCBJSjFBmG279Ac7wuk2orFlG76gCkFjNk11W/FQsjEM3gi8yKa0 ZODFmh0XJa2p0mKRnApwynO1ma7HgyqNRKrjmxMUK2VHpOsbqjVe1+Gc2lk0E9gv xc2PIdCgi3yDEpDpxSY8LEBXad7GTwa2VEID5G7C6Z0wOH38YyMEpoHQz03se9zO aCnVg8LPdFkZveWXPiiyJrwDERCyKt/yrRxrVznWyNVaWmhoxXmw7TmQexvGiDD2 E6+XT8uWfdeMZRPhv284qcBegIDw5c1wHgz+GRso61T6x04hfJyY/onbF5lHz661 lfys4JRT0zmYbrG1b3GVgmfKDv7t3V6DkY+Pi1E5raeZr2F0idiZo+7uKh2NZAML PLhb8LLr/lbHOSRG3Rq0YY/l+Q6wZXy770qF/51jp/c3UXXJyVusK+bCLrVZ3VPa 0Uf3KkuFLcoD4jDTNjt79QVJXIvEHmNOxp/g80nJUFM0WDj7u4a7pFpLCHZzFeBl 5uoS9c7dYjuV+/EkQ2S8YwryeyCHQNt381icb8HtbdlQOCVuK0v9OspA3IZmphcB oD1RBCkVTYQ= =l8Vz -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 0:18 ` Linus Torvalds 2004-09-03 1:25 ` David Masover @ 2004-09-03 5:22 ` Hans Reiser 1 sibling, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-09-03 5:22 UTC (permalink / raw) To: Linus Torvalds Cc: David Masover, viro, Frank van Maarseveen, Dave Kleikamp, Alan Cox, Jamie Lokier, Horst von Brand, Adrian Bunk, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List Linus Torvalds wrote: >On Thu, 2 Sep 2004, David Masover wrote: > > >>reiser4 kernel will contain knowledge of fs type contained in a file. >> >> Whoa! No it won't. It will allow you to create a metafile named type if you choose. Or maybe a metadirectory named types, I don't really care much about this stuff yet, we need to do other things to the semantics more urgently than this, Typing requires a lot of cooperation from user space apps. I don't really expect to have that much social pull on the issue, but if someone chooses to design their app to use a types metadirectory, I don't mind accomodating on it. I apologize if from my saying that I gave the impression that I think files should be strongly typed. I am quite well aware of the disadvantages of OS/400 hindering usability with type information, as well the advantages of having types available for those that want to look at them..... > >That's a disaster, btw. > > In every implementation so far it has been a net disaster, because OS designers who type things have been willing to make users pay attention to type when they don't want to. Sometimes trying to make too much out of something makes it a mistake when being a bit more laid back would make it ok. Me, I am so laid back, I prefer to work on other features for a few years first.....;-) >There is no one "fs type" of a file. Files have at _least_ one type >(bytestream), but most have more. Which is why automatically doing the >right thing (in the sense you seem to want) in kernel space is simply not >possible. > > Linus > > > > ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 0:08 ` The argument for fs assistance in handling archives David Masover 2004-09-03 0:18 ` Linus Torvalds @ 2004-09-03 0:39 ` viro 2004-09-03 0:44 ` Spam ` (2 subsequent siblings) 4 siblings, 0 replies; 1392+ messages in thread From: viro @ 2004-09-03 0:39 UTC (permalink / raw) To: David Masover Cc: Frank van Maarseveen, Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, Sep 02, 2004 at 07:08:21PM -0500, David Masover wrote: > | a) kernel has *NO* *FUCKING* *KNOWLEDGE* of fs type contained on a device. > > reiser4 kernel will contain knowledge of fs type contained in a file. Right. And when I decide to talk to delusional nutcases I will go to talk.origin and find a creationist to chat with. Time to extend the killfile... *plonk* ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 0:08 ` The argument for fs assistance in handling archives David Masover 2004-09-03 0:18 ` Linus Torvalds 2004-09-03 0:39 ` viro @ 2004-09-03 0:44 ` Spam 2004-09-03 1:28 ` David Masover 2004-09-03 6:16 ` Hans Reiser 2004-09-03 13:55 ` John Stoffel 4 siblings, 1 reply; 1392+ messages in thread From: Spam @ 2004-09-03 0:44 UTC (permalink / raw) To: David Masover Cc: viro, Frank van Maarseveen, Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > viro@parcelfarce.linux.theplanet.co.uk wrote: > | On Thu, Sep 02, 2004 at 11:48:06PM +0200, Frank van Maarseveen wrote: > | |>>mount is nice for root, clumsy for user. And a rather complicated |>>way of accessing data the kernel has knowledge about in the first |>>place. For filesystem images, cd'ing into the file is the most |>>obvious concept for file-as-a-dir IMHO. > | > | > | The hell it is. > | > | a) kernel has *NO* *FUCKING* *KNOWLEDGE* of fs type contained on a device. > reiser4 kernel will contain knowledge of fs type contained in a file. > "file/..metas/type" might contain a mime type. Mime type might have to > be guessed, but at least if it's made by a local "mkisofs" then we're fine. > Indeed, that's not the only interface that's been discussed. > "file/..metas/is_isofs" might be consulted. What you are talking about isn't the kernel or such, but plugins that could extend the filesystem. Plugins could store information about contents, encodings, formatting, filesystems, etc, as meta-info. If you have a plugin that would allow you to traverse files as disk images then it could read those meta-data. But before those plugins exist then there is no such standard for info stored as meta-data and the kernel wouldn't know anything about this to begin with. > | b) kernel has no way to guess which options to use > | c) fs _type_ is a fundamental part of mount - device(s) (if any) involved > | are arguments to be interpreted by that particular fs driver. > Unless there's some severe security issues with "mount this iso as fat > and you get root access", this should work fine. > I see no reason why there can't be a global setting of the mount > commandline to use. > And it doesn't all have to be in the kernel. Only it'd be nice to have > some of it there because the kernel knows how to deal with an isofs, > even if it won't know what it looks like. > | d) permissions required for that lovely operation (and questions like > | whether we force nosuid/noexec, etc.) are nightmare to define. > They are quite simple, actually. Just set them globally -- some admins > would force nosetuid/noexec, some wouldn't. And the operation happens > transparently -- you need no "permissions" other than to read the > directory which would contain the mount. > | Frankly, the longer that thread grows, the more obvious it becomes that > | file-as-a-dir is a solution in search of problem. Desperate search, at > | that. > Actually, the longer this thread grows, the more obvious it is how when > there's a hot issue, everyone has an opinion, even if the same opinion > has been expressed ten or twenty times already. > File-as-a-dir has numerous advantages, but enough have been discussed. > Short list is image mounts, tarballs, streams, metas, and namespace > unification. Longer list and explanations can be found if you RTFA. > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.2.4 (GNU/Linux) > Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org > iQIVAwUBQTe19XgHNmZLgCUhAQJdow/+Knsw1GgpauqDUcg8sKtxzgXZ18OxMQ3Q > By8sRrSTuKAzI5A3BtYIzndsj1veP+7wndG7nYPz8NS1fU2+xWSIhoGq/YMaQsu4 > 70uMLu448PFXZua4hZMk2w4mkULXbGyYHJ1Bf+2Z7QkQ/8W08hozC8QQynxMXIkX > SrcWCS5hK8Nh7Ol691sDpPqexH7F1GwUyoslNGj63U5r6ViLAawt2ZKDYdT7ZPo8 > 0a/pWUHoHMPbv/KwqZZxRr1/qncA9QYQo6JqQBPPCr+tWNJs/ei3nAKGi58iOt1M > DK1TEKd2lpbmwiK5pWDwGz+nwWmaFTAyfTEEEcP4gZedSJtRXaxyNh0jRl1iLATB > SCO5Eb4jkQs8hdjHqQcQ1q7XKFX9eSXWeDdrGrtWaYC/QYOHxT+ci3lnKBKCG99Y > YTqg3sNEZlV1N0jIcNvFSDEYbbX12v1Y6xbwvUx48+sMyUj3suT76niTRbwEydfO > MA9y+wE2k4wF+h+sJCbTjimCNFvvuFTTJuBCbQTpfY4eOYBAFalxnWmrpTEfVzka > 4iAqAYygWObGDkFFy/rp1HEVZPIKM0NwGLOsRwJsgyUMOsccBrEc0bg8sgMECVfs > 5qNIb27tLokh8NBR6RodAv2NZYKC+foM0T+PC5bZMFD/Q7f6yDklqK4C4RCIYaXj > xO9z1C6FPcM= > =/gmK > -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 0:44 ` Spam @ 2004-09-03 1:28 ` David Masover 2004-09-03 3:44 ` Chris Dukes 0 siblings, 1 reply; 1392+ messages in thread From: David Masover @ 2004-09-03 1:28 UTC (permalink / raw) To: Spam Cc: viro, Frank van Maarseveen, Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Spam wrote: [...] | Indeed, that's not the only interface that's been discussed. | "file/..metas/is_isofs" might be consulted. | | |> What you are talking about isn't the kernel or such, but plugins that Plugins are kernel-space. As Linus points out, dealing with mime-types in the kernel is uncool. |> could extend the filesystem. Plugins could store information about |> contents, encodings, formatting, filesystems, etc, as meta-info. If |> you have a plugin that would allow you to traverse files as disk |> images then it could read those meta-data. But before those plugins |> exist then there is no such standard for info stored as meta-data and |> the kernel wouldn't know anything about this to begin with. So implement a plugin which knows how to talk to a userland program which knows about metadata. The plugin controls access to file-type. Maybe there ought to be a general-purpose userland plugin interface? So that the only things left in the kernel are things that have to be there for speed and/or sanity reasons? (Things like cryptocompress and standard file/directory plugins.) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTfIs3gHNmZLgCUhAQJCqA//cPGI/TPfgtwovk5a8HdcY9TbjJELGtIb CHcNa2LKGUB8uVvoL9M0Pe9ei4iVK2QOL1QjUAbIE0Dx7t18KR/5qEIfCHTSw0sJ 8u/r3aaJhFjFwMcVLQOZWJYuTeodgMwkV96GgViGoHoiqDiV7BzZgjd43qwiH8rW FUTKlPn2VmijyTTbf5VfX4hvmsU/He+5W0t08/xe3vpCa+ihNFLJQAwGioo/wzFq aHl9jBT1esFLxONd7OxQpgVl/2uHx+rSAY6F5RyBqL/Tpm1ZKlrMdAzmdDWcAJA9 KnOLN8ltcPmjP0eCzgCO/iq8yczcwcagfmbD+WcYOmQbTXMTjxktrZqRuLrUHU+7 tl8JISKch5epHfOnQ/RTMEgotlcQ0SCoE7K5lIUuyheMYRWoVDJSvy3okET6ZxQL NP3PVHguQbu1Bo2X8LNsrnU0KFT7XrejpXzKalPWVbQxayEEWUwLXBdNIpgBoYwX FZXQKEpS4MmB/8kEC9xuQ077PfAclcjcHSj4B7phbSVMioFy4/HojOLOLA8gjHyr 7p+wimfDpsuioDLlccwc1b+fLB2eJM5G+zfzh//uy7LdfFxuI+074FJYxebRI2eL N/rpjT+S0S7L0/k71Rwp/5y4YgDgdQqU0wyGaBLRjP0qD7k2S41g+yrua6qArhzP baR9dCu1zAQ= =Nrye -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 1:28 ` David Masover @ 2004-09-03 3:44 ` Chris Dukes 2004-09-03 4:37 ` David Masover 0 siblings, 1 reply; 1392+ messages in thread From: Chris Dukes @ 2004-09-03 3:44 UTC (permalink / raw) To: David Masover Cc: Spam, viro, Frank van Maarseveen, Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, Sep 02, 2004 at 08:28:20PM -0500, David Masover wrote: > > So implement a plugin which knows how to talk to a userland program > which knows about metadata. The plugin controls access to file-type. > > Maybe there ought to be a general-purpose userland plugin interface? So > that the only things left in the kernel are things that have to be there > for speed and/or sanity reasons? (Things like cryptocompress and > standard file/directory plugins.) Ahem, Wasn't this the goal of GNU HURD? I really think you should ask them why they haven't delivered something useful, then come back to this thread. Thanks. -- Chris Dukes Warning: Do not use the reflow toaster oven to prepare foods after it has been used for solder paste reflow. http://www.stencilsunlimited.com/stencil_article_page5.htm ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 3:44 ` Chris Dukes @ 2004-09-03 4:37 ` David Masover 2004-09-03 13:09 ` Horst von Brand 0 siblings, 1 reply; 1392+ messages in thread From: David Masover @ 2004-09-03 4:37 UTC (permalink / raw) To: Chris Dukes Cc: Spam, viro, Frank van Maarseveen, Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Chris Dukes wrote: | On Thu, Sep 02, 2004 at 08:28:20PM -0500, David Masover wrote: | |>So implement a plugin which knows how to talk to a userland program |>which knows about metadata. The plugin controls access to file-type. |> |>Maybe there ought to be a general-purpose userland plugin interface? So |>that the only things left in the kernel are things that have to be there |>for speed and/or sanity reasons? (Things like cryptocompress and |>standard file/directory plugins.) | | | Ahem, | Wasn't this the goal of GNU HURD? The goal of GNU HURD was to take everything out of the kernel and make it entirely daemons. That's a far cry from keeping a file-type database (historically the realm of file managers) out of the kernel. Most people hate putting things in the kernel that don't belong there. Most of what's in the kernel is for speed and/or sanity -- filesystems are there for speed, device drivers (and scheduling, and vm, and so on) are there for sanity. Correct me if I'm wrong on this. | I really think you should ask them why they haven't delivered | something useful, then come back to this thread. Honestly? I think it's mostly got nothing to do with architecture. I think it's mostly got to do with politics. Most people would rather hack on Linux, which is already done, than try to develop HURD, which is something new. Most people also enjoy working with Linus (or prefer Linus to the FSF). I do not like how Linux is monolithic. I do not like having to reboot to upgrade the kernel, and I do not like having to run closed software (the nvidia drivers) in the kernel (as in, full privelages, can crash entire system, yadda yadda). But Linux is the best we have. The HURD people have delivered at least something. I think there's even a Debian/HURD distro. Whether it's useful probably has to do with whether it's stable/fast, which isn't likely. You hear of Linux news every day, you hear of HURD maybe once in a lifetime -- "Hey, HURD exists!" But you are right -- I need to take a break from this thread. It's becoming addictive, and it's hard keeping up with so many people who are so much smarter than I am. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTf0/3gHNmZLgCUhAQKGrw//XMBsWBo7uqlocjdvSati4GkxsB+f2GPW hI3JboYhWVMB7Ya+mS9VbrruxVn7uCCUwLwNDDnMncHTf09Em+WhzwJCQXZ14Y2r AxZ8iaVR9Qv8hIHGw59uJYWvtTSHgSmWiFFVa4UB4D3NneSFr2vgXXDbU8LjhHQB 46qs8IM6s+RCb0L608uGml6AmyrxFvDgWz9p+++tWicLjQPVhBnLQQnqYy92we7x qUwxCyGQywmdcSsJlkcjKzLQ/dCDwQRq7LXmSCx3qbSRr27D6zpE6l9He62ZuoBR I9NmO12S6VgYifO5fnq6pxSu2+GuJVW469tm7EiXmKZ/rjfMJKp0cqg+mtGTU91m gk7cZTBbe4id9+IQPJPJJ08IMH5XTI4DI6dXCkKIB0TZ8teFnU2+DsUdZ50ZXwiL etFUjortzjPjEqZoKfvK+4n7HOMu1w1og/7aE6P5UNTbj5ViwLQB9KjwLBDyBMJr dTmcLRyxzHtDZJMQuLwvU/SsdF86CdlR4spkEI6CitRMiiWvYghOeOkasOO8qxVj ckzZzUCRHNF/PTPSAEiIaw2HyjVzf0Nru96Y80/hik3hRYO0XuTP/3K/twHoebkz Fo1NLtGU3g631pDYcpqnkkAcQN+3fiyZ3VEiNoYYLhZpkQiEH8goRv5lof9XJPAC krhwM8lNHoM= =JJEe -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 4:37 ` David Masover @ 2004-09-03 13:09 ` Horst von Brand 0 siblings, 0 replies; 1392+ messages in thread From: Horst von Brand @ 2004-09-03 13:09 UTC (permalink / raw) To: David Masover Cc: Chris Dukes, Spam, viro, Frank van Maarseveen, Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List David Masover <ninja@slaphack.com> said: > Chris Dukes wrote: > | On Thu, Sep 02, 2004 at 08:28:20PM -0500, David Masover wrote: > | > |>So implement a plugin which knows how to talk to a userland program > |>which knows about metadata. The plugin controls access to file-type. > |> > |>Maybe there ought to be a general-purpose userland plugin interface? So > |>that the only things left in the kernel are things that have to be there > |>for speed and/or sanity reasons? (Things like cryptocompress and > |>standard file/directory plugins.) > | > | > | Ahem, > | Wasn't this the goal of GNU HURD? > The goal of GNU HURD was to take everything out of the kernel and make > it entirely daemons. That's a far cry from keeping a file-type database > (historically the realm of file managers) out of the kernel. Yep. Traditional microkernel. [...] > | I really think you should ask them why they haven't delivered > | something useful, then come back to this thread. > Honestly? I think it's mostly got nothing to do with architecture. I > think it's mostly got to do with politics. Most people would rather > hack on Linux, which is already done, than try to develop HURD, which is > something new. Most people also enjoy working with Linus (or prefer > Linus to the FSF). Partly right. Hurd is _much_ older than Linux, and didn't start from scratch (Mach was around, and worked). Yet I remember hearing "any day now" for Hurd from 86 or 87... and as of now, AFAIU even the people hacking on Hurd prefer Linux (because Linux does work, Hurd doesn't). > I do not like how Linux is monolithic. I do not like having to reboot > to upgrade the kernel, Tough luck. An upgrade of the core will _always_ mean rebooting. Plus you can replace modules in Linux without a reboot. > and I do not like having to run closed software > (the nvidia drivers) in the kernel (as in, full privelages, can crash > entire system, yadda yadda). Me neither. But that is not exactly Linux' fault, binary drivers are quite strongly discouraged. > But Linux is the best we have. By far. > The HURD people have delivered at least something. I think there's even > a Debian/HURD distro. Whether it's useful probably has to do with > whether it's stable/fast, which isn't likely. You hear of Linux news > every day, you hear of HURD maybe once in a lifetime -- "Hey, HURD > exists!" It is still vaporware, some 20 years later. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 0:08 ` The argument for fs assistance in handling archives David Masover ` (2 preceding siblings ...) 2004-09-03 0:44 ` Spam @ 2004-09-03 6:16 ` Hans Reiser 2004-09-03 13:55 ` John Stoffel 4 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-09-03 6:16 UTC (permalink / raw) To: David Masover Cc: viro, Frank van Maarseveen, Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List I don't think streams are a good thing, I think that all of the different pieces of additional functionality necessary to emulate them with files and directories are a good thing. Keeping streams out of linux was one of the (less important) ideas behind reiser4. Streams are a rigid hack. The toolkit that can emulate them, is useful, and the perceived importance of that emulation will fade as people start to use the toolkit for things that are much more fun than streams. I agree with most of the rest of what you say though David. Hans David Masover wrote: > File-as-a-dir has numerous advantages, but enough have been discussed. > Short list is image mounts, tarballs, streams, metas, and namespace > unification. Longer list and explanations can be found if you RTFA. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 0:08 ` The argument for fs assistance in handling archives David Masover ` (3 preceding siblings ...) 2004-09-03 6:16 ` Hans Reiser @ 2004-09-03 13:55 ` John Stoffel 2004-09-03 14:01 ` Spam 4 siblings, 1 reply; 1392+ messages in thread From: John Stoffel @ 2004-09-03 13:55 UTC (permalink / raw) To: David Masover Cc: viro, Frank van Maarseveen, Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List >>>>> "David" == David Masover <ninja@slaphack.com> writes: David> File-as-a-dir has numerous advantages, but enough have been David> discussed. Short list is image mounts, tarballs, streams, David> metas, and namespace unification. Longer list and explanations David> can be found if you RTFA. And it has numerous dis-advantages as well, since it doesn't have a good set of semantics and syntax defined yet, nor does it explain except by vigorous handwaving the performance and security impacts it can have. My personal feeling is that the mount(8) command should be the tool used to extract and expose the internal namespace of files like this and to then graft it onto the standard Unix namespace with gross Unix semantics, but it's own wacky internal semantics. This way, standard tools don't care, but special tools which know how to handle it can do what they want. > mount -t tarfs /some/place/on/disk/foo.tar.gz /mnt/tar > cp /var/tmp/img.gif . > umount /mnt/tar Oops! Someone did a rm /some/place/on/disk/foo.tar.gz between steps one and two. Now what happens? Please define those semantics... John ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 13:55 ` John Stoffel @ 2004-09-03 14:01 ` Spam 2004-09-05 11:37 ` Michelle Konzack 2004-09-06 8:58 ` Tonnerre 0 siblings, 2 replies; 1392+ messages in thread From: Spam @ 2004-09-03 14:01 UTC (permalink / raw) To: John Stoffel Cc: David Masover, viro, Frank van Maarseveen, Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List >>>>>> "David" == David Masover <ninja@slaphack.com> writes: David>> File-as-a-dir has numerous advantages, but enough have been David>> discussed. Short list is image mounts, tarballs, streams, David>> metas, and namespace unification. Longer list and explanations David>> can be found if you RTFA. > And it has numerous dis-advantages as well, since it doesn't have a > good set of semantics and syntax defined yet, nor does it explain > except by vigorous handwaving the performance and security impacts it > can have. > My personal feeling is that the mount(8) command should be the tool > used to extract and expose the internal namespace of files like this > and to then graft it onto the standard Unix namespace with gross Unix > semantics, but it's own wacky internal semantics. This way, standard > tools don't care, but special tools which know how to handle it can do > what they want. >> mount -t tarfs /some/place/on/disk/foo.tar.gz /mnt/tar >> cp /var/tmp/img.gif . >> umount /mnt/tar > Oops! Someone did a rm /some/place/on/disk/foo.tar.gz between steps > one and two. Now what happens? Please define those semantics... Uhm, can you delete a file (loop) that is mounted? > John ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 14:01 ` Spam @ 2004-09-05 11:37 ` Michelle Konzack 2004-09-06 8:58 ` Tonnerre 1 sibling, 0 replies; 1392+ messages in thread From: Michelle Konzack @ 2004-09-05 11:37 UTC (permalink / raw) To: Linux Kernel Mailing List, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 761 bytes --] Am 2004-09-03 16:01:17, schrieb Spam: > >>>>>> "David" == David Masover <ninja@slaphack.com> writes: > >> mount -t tarfs /some/place/on/disk/foo.tar.gz /mnt/tar > >> cp /var/tmp/img.gif . > >> umount /mnt/tar > > > Oops! Someone did a rm /some/place/on/disk/foo.tar.gz between steps > > one and two. Now what happens? Please define those semantics... > > Uhm, can you delete a file (loop) that is mounted? Not as $USER but root... :-/ > > John Greetings Michelle -- Linux-User #280138 with the Linux Counter, http://counter.li.org/ Michelle Konzack Apt. 917 ICQ #328449886 50, rue de Soultz MSM LinuxMichi 0033/3/88452356 67100 Strasbourg/France IRC #Debian (irc.icq.com) [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-03 14:01 ` Spam 2004-09-05 11:37 ` Michelle Konzack @ 2004-09-06 8:58 ` Tonnerre 1 sibling, 0 replies; 1392+ messages in thread From: Tonnerre @ 2004-09-06 8:58 UTC (permalink / raw) To: Spam Cc: John Stoffel, David Masover, viro, Frank van Maarseveen, Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 582 bytes --] Salut, On Fri, Sep 03, 2004 at 04:01:17PM +0200, Spam wrote: > >> mount -t tarfs /some/place/on/disk/foo.tar.gz /mnt/tar > >> cp /var/tmp/img.gif . > >> umount /mnt/tar > > > Oops! Someone did a rm /some/place/on/disk/foo.tar.gz between steps > > one and two. Now what happens? Please define those semantics... > > Uhm, can you delete a file (loop) that is mounted? "Text file busy" But for files this isn't true. Currently, we have another method assuring data coherency: we remove the inode only when the last reference goes away. Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-02 21:48 ` Frank van Maarseveen 2004-09-02 22:00 ` viro @ 2004-09-06 12:53 ` Clemens Schwaighofer 2004-09-06 12:57 ` Spam 2004-09-06 13:45 ` Tonnerre 1 sibling, 2 replies; 1392+ messages in thread From: Clemens Schwaighofer @ 2004-09-06 12:53 UTC (permalink / raw) To: Frank van Maarseveen Cc: Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Frank van Maarseveen wrote: |> |>We have the mount command for that. :^) | | | mount is nice for root, clumsy for user. And a rather complicated | way of accessing data the kernel has knowledge about in the first | place. For filesystem images, cd'ing into the file is the most | obvious concept for file-as-a-dir IMHO. thats why we have automount. lg, clemens -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFBPF3djBz/yQjBxz8RAmVhAKCODKNKOV9V0I59SUfQ1pp+qk88xwCg6+4p Q+JcbO+W0+6cUfSEf3z/Iwk= =IwZQ -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-06 12:53 ` Clemens Schwaighofer @ 2004-09-06 12:57 ` Spam 2004-09-06 12:59 ` Clemens Schwaighofer 2004-09-06 13:45 ` Tonnerre 1 sibling, 1 reply; 1392+ messages in thread From: Spam @ 2004-09-06 12:57 UTC (permalink / raw) To: Clemens Schwaighofer Cc: Frank van Maarseveen, Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > Frank van Maarseveen wrote: |>> |>>We have the mount command for that. :^) > | > | > | mount is nice for root, clumsy for user. And a rather complicated > | way of accessing data the kernel has knowledge about in the first > | place. For filesystem images, cd'ing into the file is the most > | obvious concept for file-as-a-dir IMHO. > thats why we have automount. Which still needs to be setup in fstab, right? > lg, clemens > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.2.5 (GNU/Linux) > Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org > iD8DBQFBPF3djBz/yQjBxz8RAmVhAKCODKNKOV9V0I59SUfQ1pp+qk88xwCg6+4p > Q+JcbO+W0+6cUfSEf3z/Iwk= > =IwZQ > -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-06 12:57 ` Spam @ 2004-09-06 12:59 ` Clemens Schwaighofer 2004-09-06 13:01 ` Frank van Maarseveen 0 siblings, 1 reply; 1392+ messages in thread From: Clemens Schwaighofer @ 2004-09-06 12:59 UTC (permalink / raw) To: Spam Cc: Frank van Maarseveen, Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Spam wrote: | thats why we have automount. | | |> Which still needs to be setup in fstab, right? no, in /etc/automount* actually. I have no fstab entry here and still I can mount various samba shares, cdroms, etc ... lg, clemens even my usbhd is automount :) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFBPF8tjBz/yQjBxz8RApB5AKDOjvCM2foy0lTXAy3IOVXXxdBcUACgxUyn 1WlAMXdnleNZOTOwvVRQCz0= =LX+8 -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-06 12:59 ` Clemens Schwaighofer @ 2004-09-06 13:01 ` Frank van Maarseveen 2004-09-06 13:03 ` Spam 2004-09-06 13:17 ` Clemens Schwaighofer 0 siblings, 2 replies; 1392+ messages in thread From: Frank van Maarseveen @ 2004-09-06 13:01 UTC (permalink / raw) To: Clemens Schwaighofer Cc: Spam, Frank van Maarseveen, Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Mon, Sep 06, 2004 at 09:59:25PM +0900, Clemens Schwaighofer wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Spam wrote: > > | thats why we have automount. > | > | > |> Which still needs to be setup in fstab, right? > > no, in /etc/automount* actually. I have no fstab entry here and still I > can mount various samba shares, cdroms, etc ... but can you access images that way? -- Frank ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-06 13:01 ` Frank van Maarseveen @ 2004-09-06 13:03 ` Spam 2004-09-06 13:17 ` Clemens Schwaighofer 1 sibling, 0 replies; 1392+ messages in thread From: Spam @ 2004-09-06 13:03 UTC (permalink / raw) To: Frank van Maarseveen Cc: Clemens Schwaighofer, Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List > On Mon, Sep 06, 2004 at 09:59:25PM +0900, Clemens Schwaighofer wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> Spam wrote: >> >> | thats why we have automount. >> | >> | >> |> Which still needs to be setup in fstab, right? >> >> no, in /etc/automount* actually. I have no fstab entry here and still I >> can mount various samba shares, cdroms, etc ... > but can you access images that way? Probably, but only predefined ones.. IE, no random automount? ~S ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-06 13:01 ` Frank van Maarseveen 2004-09-06 13:03 ` Spam @ 2004-09-06 13:17 ` Clemens Schwaighofer 1 sibling, 0 replies; 1392+ messages in thread From: Clemens Schwaighofer @ 2004-09-06 13:17 UTC (permalink / raw) To: Frank van Maarseveen Cc: Spam, Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Frank van Maarseveen wrote: | On Mon, Sep 06, 2004 at 09:59:25PM +0900, Clemens Schwaighofer wrote: | |>-----BEGIN PGP SIGNED MESSAGE----- |>Hash: SHA1 |> |>Spam wrote: |> |>| thats why we have automount. |>| |>| |>|> Which still needs to be setup in fstab, right? |> |>no, in /etc/automount* actually. I have no fstab entry here and still I |>can mount various samba shares, cdroms, etc ... | | | but can you access images that way? | in theory yes, but only predefined ones. But I don't see automount for this. more for cdroms and other removable medias. I haven't had the need to mount and iso for quite some time, so I don't know if I need root access for that, but I think yes like for all mount things. lg, clemens -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFBPGNgjBz/yQjBxz8RAsm7AJsEwa18ymMBWKf1rJ2PAwPOfCMIJgCglaNS d8OqXmD0jsYezcE8u7BGrkM= =ctEE -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives 2004-09-06 12:53 ` Clemens Schwaighofer 2004-09-06 12:57 ` Spam @ 2004-09-06 13:45 ` Tonnerre 1 sibling, 0 replies; 1392+ messages in thread From: Tonnerre @ 2004-09-06 13:45 UTC (permalink / raw) To: Clemens Schwaighofer Cc: Frank van Maarseveen, Dave Kleikamp, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 227 bytes --] Salut, On Mon, Sep 06, 2004 at 09:53:49PM +0900, Clemens Schwaighofer wrote: > thats why we have automount. Plus the HAL automounting code which dynamically generates mountpoints for HAL detected devices... Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 20:38 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Frank van Maarseveen 2004-09-02 21:36 ` Dave Kleikamp @ 2004-09-02 23:19 ` Valdis.Kletnieks 2004-09-02 23:41 ` Frank van Maarseveen ` (2 more replies) 2004-09-06 7:56 ` Tonnerre 2 siblings, 3 replies; 1392+ messages in thread From: Valdis.Kletnieks @ 2004-09-02 23:19 UTC (permalink / raw) To: Frank van Maarseveen Cc: Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 360 bytes --] On Thu, 02 Sep 2004 22:38:54 +0200, Frank van Maarseveen said: > Can it do this: > > cd FC2-i386-disc1.iso > ls That one's at least theoretically doable, assuming that it really *IS* the Fedora Core disk and an ISO9660 format... > cd /dev/cdrom > ls And the CD in the drive at the moment is AC/DC "Back in Black". What should this produce as output? [-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 23:19 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Valdis.Kletnieks @ 2004-09-02 23:41 ` Frank van Maarseveen 2004-09-02 23:43 ` Spam 2004-09-03 7:13 ` Jan Harkes 2 siblings, 0 replies; 1392+ messages in thread From: Frank van Maarseveen @ 2004-09-02 23:41 UTC (permalink / raw) To: Valdis.Kletnieks Cc: Frank van Maarseveen, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, Sep 02, 2004 at 07:19:47PM -0400, Valdis.Kletnieks@vt.edu wrote: > > > cd FC2-i386-disc1.iso > > ls > > That one's at least theoretically doable, assuming that it really *IS* the > Fedora Core disk and an ISO9660 format... Impressive. But when it comes to file-systems like ext[23] I think an in-kernel solution might be preferable to get the exact semantics wrt locking, atomicity, synchronization, coherency, whatever a kernel is good at for filesystems. > > cd /dev/cdrom > > ls > > And the CD in the drive at the moment is AC/DC "Back in Black". What > should this produce as output? bash: cd: /dev/cdrom: Not a directory When it doesn't match a (sub)set of known file system types I think. Because that is the area the kernel has knowledge about. Nothing else, no tarballs for example unless of course the kernel has "tarfs" :-) -- Frank ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 23:19 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Valdis.Kletnieks 2004-09-02 23:41 ` Frank van Maarseveen @ 2004-09-02 23:43 ` Spam 2004-09-02 23:52 ` Frank van Maarseveen 2004-09-06 14:43 ` Tonnerre 2004-09-03 7:13 ` Jan Harkes 2 siblings, 2 replies; 1392+ messages in thread From: Spam @ 2004-09-02 23:43 UTC (permalink / raw) To: Valdis.Kletnieks Cc: Frank van Maarseveen, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List > On Thu, 02 Sep 2004 22:38:54 +0200, Frank van Maarseveen said: >> Can it do this: >> >> cd FC2-i386-disc1.iso >> ls > That one's at least theoretically doable, assuming that it really *IS* the > Fedora Core disk and an ISO9660 format... >> cd /dev/cdrom >> ls > And the CD in the drive at the moment is AC/DC "Back in Black". What > should this produce as output? Yes why not? If there was any filesystem drivers for the AudioCD format then it could. I had such a driver for Windows 9x which would display several folders and files for inserted AudioCD's: D: (cdrom) Stereo 22050 Track01.wav Track02.wav ... 44100 Track01.wav ... Mono 22050 Track01.wav ... 44100 Track01.wav ... Normal AudioCD players would also work even though this driver was installed. These files were also visible for legacy applications in the command prompt (inside Windows). I do not see why this would not be possible in Linux. Of course, it would perhaps require a filesystem driver/module to be present when you mount. If you just want to do a cd file.iso then it may be a totally different thing. Either you would have a automount feature or a filesystem/vfs plugin that could load secondary modules to support this kind of thing. ~S ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 23:43 ` Spam @ 2004-09-02 23:52 ` Frank van Maarseveen 2004-09-06 14:43 ` Tonnerre 1 sibling, 0 replies; 1392+ messages in thread From: Frank van Maarseveen @ 2004-09-02 23:52 UTC (permalink / raw) To: Spam Cc: Valdis.Kletnieks, Frank van Maarseveen, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Fri, Sep 03, 2004 at 01:43:02AM +0200, Spam wrote: > > Yes why not? If there was any filesystem drivers for the AudioCD > format then it could. > > I had such a driver for Windows 9x which would display several > folders and files for inserted AudioCD's: > > D: (cdrom) > Stereo > 22050 > Track01.wav > Track02.wav > ... > 44100 > Track01.wav > ... ... > > If you just want to do a cd file.iso then it may be a totally > different thing. Either you would have a automount feature or a > filesystem/vfs plugin that could load secondary modules to support > this kind of thing. Why is this so different, compared to your example? They are both filesystem drivers. -- Frank ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 23:43 ` Spam 2004-09-02 23:52 ` Frank van Maarseveen @ 2004-09-06 14:43 ` Tonnerre 2004-09-06 14:59 ` Spam 1 sibling, 1 reply; 1392+ messages in thread From: Tonnerre @ 2004-09-06 14:43 UTC (permalink / raw) To: Spam Cc: Valdis.Kletnieks, Frank van Maarseveen, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 708 bytes --] Salut, On Fri, Sep 03, 2004 at 01:43:02AM +0200, Spam wrote: > Yes why not? If there was any filesystem drivers for the AudioCD > format then it could. > > I had such a driver for Windows 9x which would display several > folders and files for inserted AudioCD's: > > D: (cdrom) > Stereo > 22050 > Track01.wav > Track02.wav > ... > 44100 > Track01.wav > ... > Mono > 22050 > Track01.wav > ... > 44100 > Track01.wav > ... So you'd like the kernel to know about raw CD PCM and RIFF PCM format and conversion? Great.. That's really Solarisish! Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-06 14:43 ` Tonnerre @ 2004-09-06 14:59 ` Spam 0 siblings, 0 replies; 1392+ messages in thread From: Spam @ 2004-09-06 14:59 UTC (permalink / raw) To: Tonnerre Cc: Valdis.Kletnieks, Frank van Maarseveen, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List > Salut, > On Fri, Sep 03, 2004 at 01:43:02AM +0200, Spam wrote: >> Yes why not? If there was any filesystem drivers for the AudioCD >> format then it could. >> >> I had such a driver for Windows 9x which would display several >> folders and files for inserted AudioCD's: >> >> D: (cdrom) >> Stereo >> 22050 >> Track01.wav >> Track02.wav >> ... >> 44100 >> Track01.wav >> ... >> Mono >> 22050 >> Track01.wav >> ... >> 44100 >> Track01.wav >> ... > So you'd like the kernel to know about raw CD PCM and RIFF PCM format > and conversion? Great.. That's really Solarisish! If it could be done in userland, and still be accessible for most applications then that is great. If the user things that such a plugin is of value for him then I do not see why he should not be able to use it? Again these are example of stuff that could be done if there was a plugin/extensible interface. You make it sound like everything on the planet should be predefined and included in the kernel. This is not what I was saying or wanted. From what I understood from Hans, there will be a way to load plugins without having to recompile reiser4 module (or the kernel). This is what I'd like to see. Some things/plugins may even be partially user-space. ~S > Tonnerre ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 23:19 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Valdis.Kletnieks 2004-09-02 23:41 ` Frank van Maarseveen 2004-09-02 23:43 ` Spam @ 2004-09-03 7:13 ` Jan Harkes 2004-09-03 19:30 ` Valdis.Kletnieks 2 siblings, 1 reply; 1392+ messages in thread From: Jan Harkes @ 2004-09-03 7:13 UTC (permalink / raw) To: linux-fsdevel, Linux Kernel Mailing List On Thu, Sep 02, 2004 at 07:19:47PM -0400, Valdis.Kletnieks@vt.edu wrote: > On Thu, 02 Sep 2004 22:38:54 +0200, Frank van Maarseveen said: > > cd /dev/cdrom > > ls > > And the CD in the drive at the moment is AC/DC "Back in Black". What > should this produce as output? Hehe, cdfs already figured that one out. Ofcourse you show the individual tracks as .wav files. Jan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-03 7:13 ` Jan Harkes @ 2004-09-03 19:30 ` Valdis.Kletnieks 0 siblings, 0 replies; 1392+ messages in thread From: Valdis.Kletnieks @ 2004-09-03 19:30 UTC (permalink / raw) To: Jan Harkes; +Cc: linux-fsdevel, Linux Kernel Mailing List [-- Attachment #1: Type: text/plain, Size: 857 bytes --] On Fri, 03 Sep 2004 03:13:35 EDT, Jan Harkes said: > On Thu, Sep 02, 2004 at 07:19:47PM -0400, Valdis.Kletnieks@vt.edu wrote: > > On Thu, 02 Sep 2004 22:38:54 +0200, Frank van Maarseveen said: > > > cd /dev/cdrom > > > ls > > > > And the CD in the drive at the moment is AC/DC "Back in Black". What > > should this produce as output? > > Hehe, cdfs already figured that one out. Ofcourse you show the > individual tracks as .wav files. That's sidestepping the *real* issue - which is that you need reasonable semantics even if you don't have cdfs or whatever special gee-wizz-bang driver handy. Consider an embedded system - it may have iso9660 support, and boot off a CD like Knoppix, but not have cdfs. What do you do then? (For bonus points, figure out the security issues involved in dealing with an intentionall corrupted image on a CD......) [-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 20:38 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Frank van Maarseveen 2004-09-02 21:36 ` Dave Kleikamp 2004-09-02 23:19 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Valdis.Kletnieks @ 2004-09-06 7:56 ` Tonnerre 2004-09-06 8:08 ` Frank van Maarseveen 2 siblings, 1 reply; 1392+ messages in thread From: Tonnerre @ 2004-09-06 7:56 UTC (permalink / raw) To: Frank van Maarseveen Cc: Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 784 bytes --] Salut, On Thu, Sep 02, 2004 at 10:38:54PM +0200, Frank van Maarseveen wrote: > Can it do this: > > cd FC2-i386-disc1.iso > ls > > or this: > > cd /dev/sda1 > ls > cd /dev/floppy > ls > cd /dev/cdrom > ls > > ? Actually I see some small-security-hole-you-can-drive-a-big-yellow-truck-with-flashes-on-through problem. $ cat fs_header owner_root flags_with_suid evil_program > evil.iso $ ls -l evil.iso/evil_program -rwsr-xr-x 1 root root 24 2003-11-09 16:41 evil.iso/evil_program $ evil.iso/evil_program Mwahaha, I got root! Copy-right violating IP... done. Hijacking administrators wife... done. Overwriting OS with Parachute 2010... done. etc. pp. Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-06 7:56 ` Tonnerre @ 2004-09-06 8:08 ` Frank van Maarseveen 2004-09-06 12:43 ` Herbert Poetzl 0 siblings, 1 reply; 1392+ messages in thread From: Frank van Maarseveen @ 2004-09-06 8:08 UTC (permalink / raw) To: Tonnerre Cc: Frank van Maarseveen, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Mon, Sep 06, 2004 at 09:56:03AM +0200, Tonnerre wrote: > > $ cat fs_header owner_root flags_with_suid evil_program > evil.iso > $ ls -l evil.iso/evil_program It should of course be equivalent to a user mount: nodev nosuid etc. -- Frank ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-06 8:08 ` Frank van Maarseveen @ 2004-09-06 12:43 ` Herbert Poetzl 2004-09-06 12:54 ` Frank van Maarseveen 0 siblings, 1 reply; 1392+ messages in thread From: Herbert Poetzl @ 2004-09-06 12:43 UTC (permalink / raw) To: Frank van Maarseveen Cc: Tonnerre, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Mon, Sep 06, 2004 at 10:08:45AM +0200, Frank van Maarseveen wrote: > On Mon, Sep 06, 2004 at 09:56:03AM +0200, Tonnerre wrote: > > > > $ cat fs_header owner_root flags_with_suid evil_program > evil.iso > > $ ls -l evil.iso/evil_program > > It should of course be equivalent to a user mount: nodev nosuid etc. hmm, sounds reasonable, but what if root accesses it? (or somebody with the 'right' capability) - it might be strange if even root is not able to open device nodes or execute files from an archive - it might lead to interesting situations if the archive is opened by root, but accessed by an user (thinking of caches and such) best, Herbert > -- > Frank > - > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-06 12:43 ` Herbert Poetzl @ 2004-09-06 12:54 ` Frank van Maarseveen 0 siblings, 0 replies; 1392+ messages in thread From: Frank van Maarseveen @ 2004-09-06 12:54 UTC (permalink / raw) To: Frank van Maarseveen, Tonnerre, Alan Cox, Linus Torvalds, Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Mon, Sep 06, 2004 at 02:43:57PM +0200, Herbert Poetzl wrote: > hmm, sounds reasonable, but what if root accesses it? > (or somebody with the 'right' capability) > > - it might be strange if even root is not able to > open device nodes or execute files from an archive Yes, but if the file is owned by or writable for non-root then you've got a security problem. So, unless owned by root and not writable (readable, executable?) for anyone else "nodev" and 'nosuid" are mandatory. > > - it might lead to interesting situations if the > archive is opened by root, but accessed by an user > (thinking of caches and such) See the above. Alternatively, each process could have its own vfsmount (please don't shoot me for suggesting this ;-) -- Frank ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-01 20:50 ` Linus Torvalds [not found] ` <20040902002431.GN31934@mail.shareable.org> 2004-09-02 9:46 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Alan Cox @ 2004-09-03 15:44 ` Pavel Machek 2004-09-05 13:16 ` Tonnerre 3 siblings, 0 replies; 1392+ messages in thread From: Pavel Machek @ 2004-09-03 15:44 UTC (permalink / raw) To: Linus Torvalds Cc: Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hi! > I bet you could write a small library to test this out for a few types. > See if it's useful to you. And only if it's useful (and would make a huge > performance difference) would it be worth putting in the kernel. It seemed really usefull in uservfs incarnation. Unfortunately the daemon was not multihreaded at that time, so it was not really usefull on multiuser systems :-( > Implementation of the _user_space_ library would be something like this: > > #define MAXNAME 1024 > int open_cached_view(int base_fd, char *type, char *subname) Well, you'd need more than simple open. For caching tar (etc), you'd need stat_cached_view and opendir_cached_view and ... And this really works, only that its called mc_open(), mc_stat() etc. Gnome actually uses newer incarnation of mc_open etc, but they had to introduce rather ugly interface to make it asynchronous. Pavel -- When do you have heart between your knees? ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-01 20:50 ` Linus Torvalds ` (2 preceding siblings ...) 2004-09-03 15:44 ` Pavel Machek @ 2004-09-05 13:16 ` Tonnerre 3 siblings, 0 replies; 1392+ messages in thread From: Tonnerre @ 2004-09-05 13:16 UTC (permalink / raw) To: Linus Torvalds Cc: Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 1631 bytes --] Salut, On Wed, Sep 01, 2004 at 01:50:56PM -0700, Linus Torvalds wrote: > #define MAXNAME 1024 > int open_cached_view(int base_fd, char *type, char *subname) > { > struct stat st; > char filename[PATH_MAX]; > char name[MAXNAME]; > int len, cachefd; > > if (fstat(base_fd, &st) < 0) > return -1; > sprintf(name, "/proc/self/fd/%d", base_fd); > len = readlink(name, filename, sizeof(filename)-1); > if (len < 0) > return -1; > filename[len] = 0; > > /* FIXME! Replace '/' with '#' in "type" and "subname" */ > len = snprintf(name, sizeof(name), > "%04llx/%04llx/%s/%s/%s", > (unsigned long long) st.st_dev, > (unsigned long long) st.st_ino, > type ? : "default", > subname, > filename); > errno = ENAMETOOLONG; > if (len >= sizeof(name)) > return -1; > cachefd = open(name, O_RDONLY); > if (cachefd >= 0) { > /* Check mtime here - maybe we could have kernel support */ > return cachefd; > } > if (errno != ENOENT) > return -1; > /* > .. try to generate cache file here .. > */ Around of what I've had in mind. Only that one might use libmagic instead of the type argument. The rest can be done by a corresponding MIME plugin. Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-01 20:08 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Jamie Lokier 2004-09-01 20:50 ` Linus Torvalds @ 2004-09-02 14:06 ` Horst von Brand 2004-09-02 17:32 ` Jamie Lokier 1 sibling, 1 reply; 1392+ messages in thread From: Horst von Brand @ 2004-09-02 14:06 UTC (permalink / raw) To: Jamie Lokier Cc: Adrian Bunk, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Jamie Lokier <jamie@shareable.org> said: > I'm going to explain why filesystem support for .tar.gz or other > "document container"-like formats is useful. Nobody disagrees there (I think), the disagreement is on whether the usefullness is worth the hassle. > This does _not_ mean tar > in the kernel (I know someone who can't read will think that if I > don't say it isn't); it does mean hooks in the kernel for maintaining > coherency between different views and filesystem support for cacheing. "Coherency" and "different views" implies atomic transactions, and being able to tell that an access to the file requieres updating random junk about it. It requires being able to guess if it is worth updating now (given that the file might be modified a dozen times more before the junk being checked). > The vision I'm going for here is: > > 1. OpenOffice and similar programs store compound documents in > standard file formats, such as .tar.gz, compressed XML and such. And they are doing fine AFAICS. Besides, they won't exactly jump on the possibility of leaving behind all other OSes on which they run to become a Linux-only format. > Fs support can reduce CPU time handling these sorts of files, as > I explain below, while still working with the standard file formats. I don't buy this one. A tar.gz must be uncompressed and unpacked, and whatever you could save is surely dwarfed by those costs. > With appropriate userspace support, programs can be written which > have access to the capabilities on all platforms, but reduced CPU > time occurs only on platforms with the fs support. Userspace support isn't there on any of the platforms right now, if ever it will be a strange-Linux-installation thing for quite some time to come. Not exactly attractive for application writers. > 2. Real-time indexing and local search engine tools. Sure! Gimme the CPU power and disk throughput for that, pretty please. [No, I won't tell I have better use for those right now...] > This isn't > just things like local Google; it's also your MP3 player scanning > for titles & artists, your email program scanning for subject > lines to display the summary fast, your blog server caching built > pages, your programming environment scanning for tags, and your > file transfer program scanning for shared deltas to reduce bandwidth. With no description on how this is supposed to work, this is pure science fiction/wet dreams. > I won't explain how these work as it would make this mail too > long. It should be clear to anyone who thinks about it why the > coherency mechanism is essential for real-time, and a consistent > interface to container internals helps with performance. Coherency is essential, but it isn't free. Far from it. The easiest way of getting coherency is having _one_ authoritative source. That way you don't need coherency, and don't pay for it. Anything in this class must by force be just hints, to be recomputed at a moment's notice. I.e., let the application who might use it check and recompute as needed. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 14:06 ` Horst von Brand @ 2004-09-02 17:32 ` Jamie Lokier 2004-09-02 18:23 ` Christer Weinigel 0 siblings, 1 reply; 1392+ messages in thread From: Jamie Lokier @ 2004-09-02 17:32 UTC (permalink / raw) To: Horst von Brand Cc: Adrian Bunk, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Horst von Brand wrote: > > This does _not_ mean tar > > in the kernel (I know someone who can't read will think that if I > > don't say it isn't); it does mean hooks in the kernel for maintaining > > coherency between different views and filesystem support for cacheing. > > "Coherency" and "different views" implies atomic transactions, Wrong. They imply ordering. > and being able to tell that an access to the file requieres updating > random junk about it. Correct. > It requires being able to guess if it is worth updating now > (given that the file might be modified a dozen times more before the junk > being checked). Wrong. Lazy and deterministic are compatible. > > > The vision I'm going for here is: > > > > 1. OpenOffice and similar programs store compound documents in > > standard file formats, such as .tar.gz, compressed XML and such. > > And they are doing fine AFAICS. Besides, they won't exactly jump on the > possibility of leaving behind all other OSes on which they run to become a > Linux-only format. You replied before reading the rest of the mail again, didn't you? The whole purpose of the idea is to work with formats which are _not_ Linux-only. > > Fs support can reduce CPU time handling these sorts of files, as > > I explain below, while still working with the standard file formats. > > I don't buy this one. A tar.gz must be uncompressed and unpacked, and > whatever you could save is surely dwarfed by those costs. Didn't they explain what "lazy" means in your computer science class? Google for "lazy evaluation". In this context it means _unnecessary_ uncomressions and compressions, i.e. repeated ones, or ones where you never really needed the compressed form, are eliminated. Necessary ones are of course done. Those are things which are currently wasting your CPU because apps cannot eliminate them: to do so implies synchronisation between different apps running at different times. Surprise surprise, that's something filesystems and kernels are good for. > > With appropriate userspace support, programs can be written which > > have access to the capabilities on all platforms, but reduced CPU > > time occurs only on platforms with the fs support. > > Userspace support isn't there on any of the platforms right now, if ever it > will be a strange-Linux-installation thing for quite some time to come. Not > exactly attractive for application writers. Again, you have a strange vision of what userspace support means. I say it means a portable and simple library for accessing components inside a compressed container file in a standard format. Something I believe OpenOffice et al. already has - so to say it doesn't exist is both incorrect and missing the point. The point is to change that library which already exists, so it uses the filesystem facility when available. And to make the library better in standalone form, so that other tools which manipulate container-like formats are inclined to work with it as plugins, instead of creating their own interface as cli tools. _At no point is portability an issue_ - that library is already portable to everything - and it's certainly not a strange-Linux thing nor meant to become one. > > 2. Real-time indexing and local search engine tools. > > Sure! Gimme the CPU power and disk throughput for that, pretty please. [No, > I won't tell I have better use for those right now...] You still haven't grasped the idea of an algorithm which is more complex but reduces CPU time and disk throughput, have you? *Today*, when I open Rhythmbox I have to wait 5 minutes while it wastes my CPU power and disk throughput scanning all the files in my Music directory. The entire point of these indexing schemes is so that programs like Rhythmbox will display the same data without having to scan the disk for 5 minutes every time they start up. That's a near infinite _saving_ of CPU power and time. If you don't see that I'm just going to have to suggest you google for "cache" and learn a bit about them. > > This isn't > > just things like local Google; it's also your MP3 player scanning > > for titles & artists, your email program scanning for subject > > lines to display the summary fast, your blog server caching built > > pages, your programming environment scanning for tags, and your > > file transfer program scanning for shared deltas to reduce bandwidth. > > With no description on how this is supposed to work, this is pure science > fiction/wet dreams. Sigh. If you must continue to hurl your blunt instruments around, here is a description. I didn't want to write this becuase it is off topic, and you insist on not understanding the basic of algorithm complexity and CPU usage, so you'll probably not understand this either, but I'll give it a go. 1. Local Google (by which I mean a search engine on your local machine), Real-time (by which I mean the results are always up to date): Every file modified since last search is known to the query engine. This is a reality: BeOS does it; WinFS is expected to do it. We know that it's possible to update free text indexes with small amounts of known changes quickly, at the time of a query. Thus we have real-time local free text search engine, and other features like searching inside files and for file names. The point is the real-time nature of it: the results you get correspond to exactly the contents of the filesystem at the time of the query (writes which occur _during_ a query are obviously not coherent with this, but writes which complete before the query, even immediately before, appear in the results). Note that file write timing need not be affected - all the work can happen during queries (although it is best to have a nightly re-index or whatever so that the delay at query times is kept moderate). 2. MP3 player scanning artists & titles: Easy. MP3 player does what it does, shows the extracted ID tags from all .mp3 files in your home directory. They do this already! The difference is that with fs coherency hooks, they can _store_ that ID information, retrieve it later without having to scan all the .mp3s again (see Rhythmbox earlier), and keep their lists updated on the screen as soon as any .mp3 is changed or even any new ones created anywhere. Technically it's a simpler subset of real-time queries in 1. 3. Email program scanning for subject lines fast: See Evolution; the only difference is stat() on a thousand files won't be required, it'll be exposed through a standard query instead of an Evolution-only cache method so other mail programs may use it as well as shell commands, and you can pretend you have mbox instead of maildir (mbox is a container format...). 4. Blog server caching built pages: It's a reality already, obviously. The difference is it'll make sense to built the pages through an indexed query on files instead of a database, e.g. one file per article, and (independent of the first point) the process of building can keep track of all the prerequisite files and scripts and templates used to produce the page, and actually expect to know, _with coherent accuracy_, if any of those prerequisites is different the next time the page is needed. In other words, complex script-generated web pages, cached output, and with no overhead when serving each page to check prerequisites (i.e. no 100 stat() calls), because we know the filesystem would have told us if any prerequisite had changed prior to the moment we begin serving the cached output. You can do something close to this already with dnotify (if you ignore that it doesn't tell you about new hard links, which is a dnotify bug), although doing the query part is unrealistic with dnotify and stat() prerequisites unless the directory names are structured thoughtfully. 5. Programming environment scanning for tags: By now this should be obvious. No need to run "exuberant ctags" or whatever class-hierarchy-extraction and documentation-extraction program every so often after making changes, and yet your IDE's clickable list of tags and notes stays up to date with file modifications in real time. I think some IDEs do this already with moderate size trees, using stat(), but it's not realistic when you have tens of thousands of source files. 6. File transfer program scanning for shared deltas. This is nothing more than searching all files for common subsequences which match strong hashes which are expected to be common during file transfer operations. E.g. the GPL header at the start of many source files would have one such hash. A hash of every whole file is another one, along with the name of the file (it is a key which indicates a "likely to match the corresponding hash" condition), as is a hash of every aligned 64k subsequence, or whatever is appropriate to reduce disk I/O. Having an index of these can speed up some file transfer protocols, in the manner of rsync but comparing among a large group of files instead of just two. The point is that kind of protocol can be more efficient than rsync (sometimes dramatically so), but it's only a net gain to use that kind of algorithm if you have a handy _and reliable_ index of likely common subsequences and whole-file hashes, otherwise it uses far too much disk I/O to check the group. The index needs filesystem coherency support, otherwise it is not reliable enough for this bandwidth optimisation. > > I won't explain how these work as it would make this mail too > > long. It should be clear to anyone who thinks about it why the > > coherency mechanism is essential for real-time, and a consistent > > interface to container internals helps with performance. > > Coherency is essential, but it isn't free. Far from it. The easiest way of > getting coherency is having _one_ authoritative source. That way you don't > need coherency, and don't pay for it. Anything in this class must by force > be just hints, to be recomputed at a moment's notice. I.e., let the > application who might use it check and recompute as needed. (a) Without some kind of minimal hook, there is no way for the application to check without re-reading all the files it has used for a computation, starting at the moment where it's about to potentially recompute the result. Sometimes you can get away with stat() on the files (e.g. Evolution and fontconfig do that), but often you can't because that's not trustworthy enough (security checks, protocol optimisations, transparent "as if I didn't cache anything" semantics), and sometimes stat() calls are far too slow anyway (anything that involves a lot of files). (The trustworthiness problem is likely to be solved, for some applications, by an xattr or such which is guaranteed to be deleted or changed when a file is modified. That is really a very minimal hook, but it is one). (b) Even with all the misgivings of stat(), you can't realistically update real time displays, e.g. the lists of artists & albums on the screen in your MP3 player whenever you modify anything in your music collection using another program. At best, you have to expect the display to not notice the change for a while (minutes in the mp3 case), or until you click an "Explicit Refresh" button. Neither should be necessary. It's true that the simplest coherency is one authoratitive source which you have to check all the time (although even that doesn't work for some things). But checking all the time does rule out, due to its high algorithmic complexity, a lot of very interesting applications and even some interesting simple unixish tools which you might like, such as real-time "locate". -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 17:32 ` Jamie Lokier @ 2004-09-02 18:23 ` Christer Weinigel 2004-09-02 21:47 ` Jamie Lokier 0 siblings, 1 reply; 1392+ messages in thread From: Christer Weinigel @ 2004-09-02 18:23 UTC (permalink / raw) To: Jamie Lokier Cc: Horst von Brand, Adrian Bunk, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Jamie Lokier <jamie@shareable.org> writes: > 1. Local Google (by which I mean a search engine on your local machine), > Real-time (by which I mean the results are always up to date): > > Every file modified since last search is known to the query engine. > This is a reality: BeOS does it; WinFS is expected to do it. > > Thus we have real-time local free text search engine, and other > features like searching inside files and for file names. The > point is the real-time nature of it: the results you get > correspond to exactly the contents of the filesystem at the time of > the query (writes which occur _during_ a query are obviously > not coherent with this, but writes which complete before the > query, even immediately before, appear in the results). Can be done with dnotify/inotify and a cache daemon keeping track of mtime. Yes, this will need a kernel change to make sure mtime always changed when the file changes, but it does not require anything else. > 2. MP3 player scanning artists & titles: Same. > 3. Email program scanning for subject lines fast: Same here. > 4. Blog server caching built pages: > 5. Programming environment scanning for tags: > 6. File transfer program scanning for shared deltas. And so on. /Christer -- "Just how much can I get away with and still go to heaven?" Freelance consultant specializing in device driver programming for Linux Christer Weinigel <christer@weinigel.se> http://www.weinigel.se ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 18:23 ` Christer Weinigel @ 2004-09-02 21:47 ` Jamie Lokier 2004-09-02 22:27 ` Alan Cox 2004-09-06 15:55 ` Christer Weinigel 0 siblings, 2 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-09-02 21:47 UTC (permalink / raw) To: Christer Weinigel Cc: Horst von Brand, Adrian Bunk, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Christer Weinigel wrote: > Can be done with dnotify/inotify and a cache daemon keeping track of > mtime. Yes, this will need a kernel change to make sure mtime always > changed when the file changes, but it does not require anything else. - Can the daemon keep track of _every_ file on my disk like this? That's more than a million files, and about 10^5 directories. dnotify would require the daemon to open all the directories. I'm not sure what inotify offers. - What happens at reboot - I guess the daemon has to call stat() on every file to verify its indexes? Have you any idea how long it takes to call stat() on every file in my home directory? - The ordering problem: I write to a file, then the program returns. System is very busy compiling. 2 minutes later, I execute a search query. The file I wrote two minute ago doesn't appear in the search results. What's wrong? Due to scheduling, the daemon hasn't caught up yet. Ok, we can accept that's just hard life. Sometimes it takes a while for something I write to appear in search results. But! That means I can't use these optimised queries as drop-in replacements for calling grep and find, or for making Make-like programs run faster (by eliminating parsing and stat() calls). That's a shame, it would have been nice to have a mechanism that could transparently optimise prorgrams that do calculations.... Do you see what I'm getting at? There's building some nice GUI and search engine like functionality, where changes made by one program _eventually_ show up in another (i.e. not synchronously). That's easy. And then there's optimising things like grep, find, perl, gcc, make, httpd, rsync, in a way that's semantically transparent, but executes faster _as if_ they had recalculated everything they need to every time. That's harder. > > 3. Email program scanning for subject lines fast: > > Same here. > > > 4. Blog server caching built pages: > > 5. Programming environment scanning for tags: > > 6. File transfer program scanning for shared deltas. > > And so on. No, not 3, 4 or 6. For correct behaviour those require synchronous query results. Think about 6, where one important cached query is "what is the MD5 sum of this file", and another critical one, which can only work through indexing, is "give me the name of any file whose MD5 sum matches $A_SPECIFIC_MD5". Trusting the async results for those kind of queries from your daemon would occasionally result in data loss due to race conditions. So you wouldn't trust the async results, and you fail to get those CPU-saving and bandwidth-saving optimisations. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 21:47 ` Jamie Lokier @ 2004-09-02 22:27 ` Alan Cox 2004-09-03 15:17 ` Jon Smirl 2004-09-06 15:55 ` Christer Weinigel 1 sibling, 1 reply; 1392+ messages in thread From: Alan Cox @ 2004-09-02 22:27 UTC (permalink / raw) To: Jamie Lokier Cc: Christer Weinigel, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Iau, 2004-09-02 at 22:47, Jamie Lokier wrote: > - Can the daemon keep track of _every_ file on my disk like this? > That's more than a million files, and about 10^5 directories. > dnotify would require the daemon to open all the directories. > I'm not sure what inotify offers. This is currently a real issue for both desktop search and for virus scanners. They want a "what changed and where" system wide (or at least per namespace/mount). ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 22:27 ` Alan Cox @ 2004-09-03 15:17 ` Jon Smirl 0 siblings, 0 replies; 1392+ messages in thread From: Jon Smirl @ 2004-09-03 15:17 UTC (permalink / raw) To: Alan Cox Cc: Jamie Lokier, Christer Weinigel, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, 02 Sep 2004 23:27:06 +0100, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote: > On Iau, 2004-09-02 at 22:47, Jamie Lokier wrote: > > - Can the daemon keep track of _every_ file on my disk like this? > > That's more than a million files, and about 10^5 directories. > > dnotify would require the daemon to open all the directories. > > I'm not sure what inotify offers. > > This is currently a real issue for both desktop search and for virus > scanners. They want a "what changed and where" system wide (or at least > per namespace/mount). In the database work everything is a transaction and the transactions are logged. Reiser4 is fully atomic and logged. So to get the "what changed and where" you just process the transaction logs from the point of your last marked checkpoint. Hot backup db servers work this way too by listening to the transaction log stream. You don't need a daemon in this model. Jon Smirl jonsmirl@gmail.com ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-02 21:47 ` Jamie Lokier 2004-09-02 22:27 ` Alan Cox @ 2004-09-06 15:55 ` Christer Weinigel 2004-09-06 15:56 ` Chris Dukes 1 sibling, 1 reply; 1392+ messages in thread From: Christer Weinigel @ 2004-09-06 15:55 UTC (permalink / raw) To: Jamie Lokier Cc: Christer Weinigel, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Jamie Lokier <jamie@shareable.org> writes: > Christer Weinigel wrote: > > Can be done with dnotify/inotify and a cache daemon keeping track of > > mtime. Yes, this will need a kernel change to make sure mtime always > > changed when the file changes, but it does not require anything else. > > - Can the daemon keep track of _every_ file on my disk like this? > That's more than a million files, and about 10^5 directories. > dnotify would require the daemon to open all the directories. > I'm not sure what inotify offers. I don't think dnotify/inotify handles subdirectories well yet, so I suppose that they would have to be extended. > - What happens at reboot - I guess the daemon has to call stat() > on every file to verify its indexes? Have you any idea how long > it takes to call stat() on every file in my home directory? The daemon saves state before it shuts down and reloads the state after a reboot. You have to make sure that it is started first and stopped last during the boot process. How would a kernel plugin handle things that happen before or after the plugin module has been loaded? It's the same problem. > - The ordering problem: I write to a file, then the program > returns. System is very busy compiling. 2 minutes later, I > execute a search query. The file I wrote two minute ago doesn't > appear in the search results. What's wrong? > > Due to scheduling, the daemon hasn't caught up yet. Ok, we can > accept that's just hard life. Sometimes it takes a while for > something I write to appear in search results. > > But! That means I can't use these optimised queries as drop-in > replacements for calling grep and find, or for making Make-like > programs run faster (by eliminating parsing and stat() calls). > That's a shame, it would have been nice to have a mechanism that > could transparently optimise prorgrams that do calculations.... Sure you can. First of all, you can just wait for the daemon to finish indexing any files that it has been notified about changes in. This is no different from you having to wait for the kernel to finish indexing the files. Or are you suggesting that the kernel should stop all other processes until the indexing is done? > Do you see what I'm getting at? There's building some nice GUI > and search engine like functionality, where changes made by one > program _eventually_ show up in another (i.e. not synchronously). > > That's easy. > > And then there's optimising things like grep, find, perl, gcc, > make, httpd, rsync, in a way that's semantically transparent, but > executes faster _as if_ they had recalculated everything they > need to every time. That's harder. If you have a good notify, it's not harder. > No, not 3, 4 or 6. For correct behaviour those require synchronous > query results. Think about 6, where one important cached query is > "what is the MD5 sum of this file", and another critical one, which > can only work through indexing, is "give me the name of any file whose > MD5 sum matches $A_SPECIFIC_MD5". Trusting the async results for > those kind of queries from your daemon would occasionally result in > data loss due to race conditions. So you wouldn't trust the async > results, and you fail to get those CPU-saving and bandwidth-saving > optimisations. So how do you calculate the MD5 sum of a file that is in the process of being modified? It's not possible to do that unless you block all other access to that file and recalculate the MD5 sum after each write. With a notifier that tells the daemon that it has stale data and needs to reprocess the file, it's no different. /Christer -- "Just how much can I get away with and still go to heaven?" Freelance consultant specializing in device driver programming for Linux Christer Weinigel <christer@weinigel.se> http://www.weinigel.se ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) 2004-09-06 15:55 ` Christer Weinigel @ 2004-09-06 15:56 ` Chris Dukes 0 siblings, 0 replies; 1392+ messages in thread From: Chris Dukes @ 2004-09-06 15:56 UTC (permalink / raw) To: Christer Weinigel Cc: Jamie Lokier, Horst von Brand, Adrian Bunk, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Mon, Sep 06, 2004 at 05:55:21PM +0200, Christer Weinigel wrote: Could ya'll take this to slashdot for a bit just to improve the level of commentary there? http://slashdot.org/article.pl?sid=04/09/06/1235236 -- Chris Dukes Warning: Do not use the reflow toaster oven to prepare foods after it has been used for solder paste reflow. http://www.stencilsunlimited.com/stencil_article_page5.htm ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 15:02 ` Jamie Lokier 2004-08-28 23:14 ` Horst von Brand @ 2004-09-10 2:13 ` Timothy Miller 2004-09-10 5:22 ` Hans Reiser 1 sibling, 1 reply; 1392+ messages in thread From: Timothy Miller @ 2004-09-10 2:13 UTC (permalink / raw) To: Jamie Lokier Cc: Adrian Bunk, Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Jamie Lokier wrote: [snip > When a simple "cd" into .tar.gz or .iso is implemented properly, it > will have _no_ performance penalty after you have first looked in the > file, so long as it remains in the on-disk cache. And, the filesystem > will manage that cache intelligently. > > Imagine: for looking at source files and such, you probably won't > bother untarring in future, and you won't bother keeping untarred > source trees in your home directory for easy access to things you look > at often. Why waste the space? You could install whole applications > as a .tar and run them from within it, with no performance penalty. > > Similarly, the filesystem will be able to archive directories > automatically that haven't been touched in a long time, with no > visible change except increased storage space. "grep" will be a bit > slower, but you'll have a useful search tool by then (using coherent > indexes) which will be more useful than grep, and much faster. > [Again, pardon me for being 5000 messages behind.] Anyhow, I recall reading an article about 'unified name spaces', and I'm not referring to what's on namesys.com. It mentioned how putting device nodes into the file system is a very powerful innovation of UNIX. Having files and devices in the same name space simplifies tools and makes the environment much more powerful, because you can connect data sources and targets together more arbitarily. If I recall correctly, the article mentioned something about Plan9 taking things to a greater extreme than UNIX. Going along with what you said, Jamie, if "containers" like tar files could be accessed like directories without being first extracted, it would increase the power and flexibility of the whole system. Windows XP does something like this with the way it presents ZIP files as directories, and although I'm sure the Windows way isn't nearly as efficient and universal as how Linux would do it, I find the feature to be INCREDIBLY useful. Of course, we don't necessarily want codecs for every archive format ever invented to be embedded in the kernel. Instead, we need to do something like how you can mount an ISO on a loopback device, only more transparently and more flexibly, and with the less performance-critical codecs being implemented as daemons in userspace. Also, there would be limitations. For instance, many such pseudo-directories would be read-only, and some would be writable only when on some file systems (like writing to a compressed archive might be much easier to implement on Reiser4 than something else, or perhaps the ext3 version would have to do a lot more copying, while the Reiser version could take advantage of Reiser-specific features like inserting in the middle of a file). ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-10 2:13 ` silent semantic changes with reiser4 Timothy Miller @ 2004-09-10 5:22 ` Hans Reiser 2004-09-10 6:32 ` Peter Foldiak 2004-09-10 15:33 ` Horst von Brand 0 siblings, 2 replies; 1392+ messages in thread From: Hans Reiser @ 2004-09-10 5:22 UTC (permalink / raw) To: Timothy Miller Cc: Jamie Lokier, Adrian Bunk, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List, Peter Foldiak A friend asked me a question, and because he is very bright it reminded me that I have not done a good job of reviewing the history of the design's evolution. He asked me, why not just access a filename's size as filename/size? So, the original idea was to access metafiles as just files within a directory, and it actually remains that way. However, I first decided to make the standard unix metafiles less intrusive on the namespace. So that led to calling it filename/..size and filename/..owner, etc. In this scheme, the use of '..' was just a style convention for metafiles, and not a requirement in anyway. This was actually pretty decent as a design, but then a user on the mailing list suggested replacing the '..' prefix with a subdirectory prefix. I forget who suggested this and what the prefix was exactly. So we then created a '..metas' subdirectory, and this had the advantage that one could ls it to see all the builtins supported by a given plugin. This is not an important advantage, and I encourage others to critique it. So, instead of filename/size we have filename/..metas/size. The only thing gained by this is that 'size' has a rarer name of '..metas/size'. The use of the '..metas/' prefix is purely a non-mandated style convention. File plugins that dislike it are free to violate the convention. There is no deep semantic to it, just a cowardly aversion to intruding on current namespace usage with something as common as 'size'. It has the significant disadvantage of being a longer name than 'size' or '..size'. I could be talked out of it. Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-10 5:22 ` Hans Reiser @ 2004-09-10 6:32 ` Peter Foldiak 2004-09-10 6:57 ` Hans Reiser 2004-09-10 15:33 ` Horst von Brand 1 sibling, 1 reply; 1392+ messages in thread From: Peter Foldiak @ 2004-09-10 6:32 UTC (permalink / raw) To: Hans Reiser Cc: Timothy Miller, Jamie Lokier, Adrian Bunk, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Fri, 2004-09-10 at 06:22, Hans Reiser wrote: > He asked me, why not just access a filename's size as filename/size? I now understand that you need a way to distinguish between something like shoe/size and shoe/.../size (or shoe/..size) The first one is the size of the shoe, the second is the automatically generated size of the file (object). You would get into trouble if you would not allow the user to use shoe/size for shoe size. Peter ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-10 6:32 ` Peter Foldiak @ 2004-09-10 6:57 ` Hans Reiser 2004-09-10 15:47 ` Timothy Miller 0 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-09-10 6:57 UTC (permalink / raw) To: Peter Foldiak Cc: Timothy Miller, Jamie Lokier, Adrian Bunk, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Peter Foldiak wrote: >On Fri, 2004-09-10 at 06:22, Hans Reiser wrote: > > >>He asked me, why not just access a filename's size as filename/size? >> >> > >I now understand that you need a way to distinguish between something >like > >shoe/size > >and > >shoe/.../size (or shoe/..size) > >The first one is the size of the shoe, the second is the automatically >generated size of the file (object). You would get into trouble if you >would not allow the user to use shoe/size for shoe size. Peter > > > > > > exactly. Of course, problem/shoe/size could refer to shoe size in centimeters of a problem shoe or the size of the problem relating to a shoe in units of reporters providing press coverage of it or.... So there are lots of opportunities for ambiguity in semantics.... Still, widely used builtins seem like they should be moderately evasive of commonly used names. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-10 6:57 ` Hans Reiser @ 2004-09-10 15:47 ` Timothy Miller 2004-09-10 15:50 ` Wayne Scott 2004-09-10 17:51 ` Hans Reiser 0 siblings, 2 replies; 1392+ messages in thread From: Timothy Miller @ 2004-09-10 15:47 UTC (permalink / raw) To: Hans Reiser Cc: Peter Foldiak, Jamie Lokier, Adrian Bunk, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hans Reiser wrote: > Peter Foldiak wrote: > >> On Fri, 2004-09-10 at 06:22, Hans Reiser wrote: >> >> >>> He asked me, why not just access a filename's size as filename/size? >>> >> >> >> I now understand that you need a way to distinguish between something >> like >> >> shoe/size >> >> and >> >> shoe/.../size (or shoe/..size) >> >> The first one is the size of the shoe, the second is the automatically >> generated size of the file (object). You would get into trouble if you >> would not allow the user to use shoe/size for shoe size. Peter >> >> >> >> >> >> > exactly. > > Of course, problem/shoe/size could refer to shoe size in centimeters of > a problem shoe or the size of the problem relating to a shoe in units of > reporters providing press coverage of it or.... > > So there are lots of opportunities for ambiguity in semantics.... > > Still, widely used builtins seem like they should be moderately evasive > of commonly used names. You know, if tools all need to be rewritten anyway to deal with the file metadata "directory", then why not change the symbol that delimits the metadata key? Everyone likes ':', so we'd have "problem/shoe:size". (Don't bother to complain about files which have : in them, because I already know it sucks, but it's an example.) See, unless you can come up with a way to seamlessly make old tools work with the new semantics, then there's no reason not to make more than one change to tools at at the same time. Also, if you take the ':' example literally, then the file system would need to be able to figure out that a file whose literal name is "C:\MYPR0NDIR\BODY_PARTS.JPG" isn't referring to metadata for a file named "C". :) ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-10 15:47 ` Timothy Miller @ 2004-09-10 15:50 ` Wayne Scott 2004-09-10 17:50 ` Horst von Brand 2004-09-10 19:54 ` Uwe Ohse 2004-09-10 17:51 ` Hans Reiser 1 sibling, 2 replies; 1392+ messages in thread From: Wayne Scott @ 2004-09-10 15:50 UTC (permalink / raw) To: miller Cc: reiser, Peter.Foldiak, jamie, bunk, viro, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list From: Timothy Miller <miller@techsource.com> > Everyone likes ':', so we'd have "problem/shoe:size". (Don't bother to > complain about files which have : in them, because I already know it > sucks, but it's an example.) [[ I just joined this discussion, so pardon if this is already known.]] One advantage of ':' is that portable programs already have to avoid it because of NTFS alternate data streams: http://www.diamondcs.com.au/index.php?page=archive&id=ntfs-streams For example on an XP box with NTFS: $ mkdir j $ cd j $ echo hi > foo:bar $ ls -l total 0 -rw-r--r-- 1 wscott Administ 0 Sep 10 10:45 foo $ cat foo $ cat foo:bar hi $ rm foo $ cat foo:bar cat: foo:bar: No such file or directory -Wayne ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-10 15:50 ` Wayne Scott @ 2004-09-10 17:50 ` Horst von Brand 2004-09-10 19:54 ` Uwe Ohse 1 sibling, 0 replies; 1392+ messages in thread From: Horst von Brand @ 2004-09-10 17:50 UTC (permalink / raw) To: Wayne Scott Cc: miller, reiser, Peter.Foldiak, jamie, bunk, viro, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Timothy Miller <miller@techsource.com> said: > > Everyone likes ':', so we'd have "problem/shoe:size". (Don't bother to > > complain about files which have : in them, because I already know it > > sucks, but it's an example.) > > [[ I just joined this discussion, so pardon if this is already known.]] > One advantage of ':' is that portable programs already have to avoid > it because of NTFS alternate data streams: Portable programs working on POSIX don't give a damn that C:, CON:, etc are reserved in MS-DOS. Windows warts shouldn't infect Linux. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-10 15:50 ` Wayne Scott 2004-09-10 17:50 ` Horst von Brand @ 2004-09-10 19:54 ` Uwe Ohse 1 sibling, 0 replies; 1392+ messages in thread From: Uwe Ohse @ 2004-09-10 19:54 UTC (permalink / raw) To: linux-kernel Wayne Scott wrote: >One advantage of ':' is that portable programs already have to avoid >it because of NTFS alternate data streams: this is not going to work: lynx www.site.tld:8080 ls /usr/share/man/man3/*::* ## perl stuff http://cr.yp.to/proto/maildir.html Regards, Uwe ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-10 15:47 ` Timothy Miller 2004-09-10 15:50 ` Wayne Scott @ 2004-09-10 17:51 ` Hans Reiser 2004-09-10 18:15 ` Horst von Brand 1 sibling, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-09-10 17:51 UTC (permalink / raw) To: Timothy Miller Cc: Peter Foldiak, Jamie Lokier, Adrian Bunk, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Timothy Miller wrote: > > > Hans Reiser wrote: > >> Peter Foldiak wrote: >> >>> On Fri, 2004-09-10 at 06:22, Hans Reiser wrote: >>> >>> >>>> He asked me, why not just access a filename's size as filename/size? >>>> >>> >>> >>> >>> I now understand that you need a way to distinguish between something >>> like >>> >>> shoe/size >>> >>> and >>> >>> shoe/.../size (or shoe/..size) >>> >>> The first one is the size of the shoe, the second is the automatically >>> generated size of the file (object). You would get into trouble if you >>> would not allow the user to use shoe/size for shoe size. Peter >>> >>> >>> >>> >>> >>> >> exactly. >> >> Of course, problem/shoe/size could refer to shoe size in centimeters >> of a problem shoe or the size of the problem relating to a shoe in >> units of reporters providing press coverage of it or.... >> >> So there are lots of opportunities for ambiguity in semantics.... >> >> Still, widely used builtins seem like they should be moderately >> evasive of commonly used names. > > > You know, if tools all need to be rewritten anyway to deal with the > file metadata "directory", then why not change the symbol that > delimits the metadata key? because it is useful that it is only a style convention. Changing the symbol makes it mandatory to distinguish metafiles from files. > > Everyone likes ':', so we'd have "problem/shoe:size". (Don't bother > to complain about files which have : in them, because I already know > it sucks, but it's an example.) > > See, unless you can come up with a way to seamlessly make old tools > work with the new semantics, then there's no reason not to make more > than one change to tools at at the same time. > > Also, if you take the ':' example literally, then the file system > would need to be able to figure out that a file whose literal name is > "C:\MYPR0NDIR\BODY_PARTS.JPG" isn't referring to metadata for a file > named "C". :) > > > ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-10 17:51 ` Hans Reiser @ 2004-09-10 18:15 ` Horst von Brand 2004-09-11 0:00 ` Timothy Miller 0 siblings, 1 reply; 1392+ messages in thread From: Horst von Brand @ 2004-09-10 18:15 UTC (permalink / raw) To: Hans Reiser Cc: Timothy Miller, Peter Foldiak, Jamie Lokier, Adrian Bunk, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hans Reiser <reiser@namesys.com> said: > Timothy Miller wrote: [...] > > You know, if tools all need to be rewritten anyway to deal with the > > file metadata "directory", then why not change the symbol that > > delimits the metadata key? > because it is useful that it is only a style convention. Changing the > symbol makes it mandatory to distinguish metafiles from files. If they are really different than directories, it should be clear what is what, IMVHO. If they aren't different, they have no place here. Trying to break hoary Unix tradition (files and directories are separate) while pretending nothing has changed (see, a file can be handled just like a directory) just doesn't cut it. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-10 18:15 ` Horst von Brand @ 2004-09-11 0:00 ` Timothy Miller 0 siblings, 0 replies; 1392+ messages in thread From: Timothy Miller @ 2004-09-11 0:00 UTC (permalink / raw) To: Horst von Brand Cc: Hans Reiser, Peter Foldiak, Jamie Lokier, Adrian Bunk, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Horst von Brand wrote: > Hans Reiser <reiser@namesys.com> said: > >>Timothy Miller wrote: > > > [...] > > >>>You know, if tools all need to be rewritten anyway to deal with the >>>file metadata "directory", then why not change the symbol that >>>delimits the metadata key? > > >>because it is useful that it is only a style convention. Changing the >>symbol makes it mandatory to distinguish metafiles from files. > > > If they are really different than directories, it should be clear what is > what, IMVHO. If they aren't different, they have no place here. Trying to > break hoary Unix tradition (files and directories are separate) while > pretending nothing has changed (see, a file can be handled just like a > directory) just doesn't cut it. I think that sums up what I was trying to say better than what I said did. :) ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-10 5:22 ` Hans Reiser 2004-09-10 6:32 ` Peter Foldiak @ 2004-09-10 15:33 ` Horst von Brand 1 sibling, 0 replies; 1392+ messages in thread From: Horst von Brand @ 2004-09-10 15:33 UTC (permalink / raw) To: Hans Reiser Cc: Timothy Miller, Jamie Lokier, Adrian Bunk, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List, Peter Foldiak Hans Reiser <reiser@namesys.com> said: > A friend asked me a question, and because he is very bright it reminded > me that I have not done a good job of reviewing the history of the > design's evolution. The design's evolution, and your friend's question, are irrelevant. Might be a nice story, but not relevant to discuss the _current_ design, its shortcommings, and possible solutions. > He asked me, why not just access a filename's size as filename/size? Urgh. > So, the original idea was to access metafiles as just files within a > directory, and it actually remains that way. However, I first decided to > make the standard unix metafiles less intrusive on the namespace. So > that led to calling it filename/..size and filename/..owner, etc. In > this scheme, the use of '..' was just a style convention for metafiles, > and not a requirement in anyway. Oh, so metafiles could have any names at all. Awful. > This was actually pretty decent as a design, Could have been worse, true. Not very much, anyway. > but then a user on the > mailing list suggested replacing the '..' prefix with a subdirectory > prefix. I forget who suggested this and what the prefix was exactly. So > we then created a '..metas' subdirectory, Bad design. As Ted explained, _any_ "normal" name in there will lead to legacy applications do the wrong thing unaware of the special status of this stuff, and this is sure to break things and/or to lead to nasty security problems. > and this had the advantage > that one could ls it to see all the builtins supported by a given > plugin. What kind of builtins? What is a "plugin"? > This is not an important advantage, and I encourage others to > critique it. If it is not important, let's postpone any discussion on it until (if ever) the important problems are solved. > So, instead of filename/size we have filename/..metas/size. The only > thing gained by this is that 'size' has a rarer name of '..metas/size'. English, please. > The use of the '..metas/' prefix is purely a non-mandated style > convention. So it isn't mandated, and I could decide placing my own stuff under other names, willy-nilly? See the comment above on "normal names" being dangerous; if it hasn't a builtin way of ensuring legacy (or even new applications) don't trip over this stuff, it is a moby nightmare security and correctness wise. > File plugins that dislike it are free to violate the > convention. There is no deep semantic to it, just a cowardly aversion to > intruding on current namespace usage "Common" and "current usage" is important to the average user, "normal" is crucial for correctness and security, > with something as common as 'size'. > It has the significant disadvantage of being a longer name than 'size' > or '..size'. I could be talked out of it. So far, I've seen no real reason to introduce this. If it is scrapped, there will be no need to talk anybody out of silly ideas. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:49 ` Hans Reiser 2004-08-26 14:05 ` Adrian Bunk @ 2004-08-26 18:35 ` Joel Becker 2004-08-26 19:53 ` Jamie Lokier 2004-08-27 9:19 ` Markus Törnqvist 1 sibling, 2 replies; 1392+ messages in thread From: Joel Becker @ 2004-08-26 18:35 UTC (permalink / raw) To: Hans Reiser Cc: Jamie Lokier, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 01:49:15AM -0700, Hans Reiser wrote: > Yes, this was part of the plan, tar file-directory plugins would be cute. Question: Is "cat /foo/bar/baz.tar.gz/metas" the attribute directory or a directory in the tarball named "metas"? Joel -- "I'm so tired of being tired, Sure as night will follow day. Most things I worry about Never happen anyway." Joel Becker Senior Member of Technical Staff Oracle Corporation E-mail: joel.becker@oracle.com Phone: (650) 506-8127 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:35 ` Joel Becker @ 2004-08-26 19:53 ` Jamie Lokier 2004-08-27 9:19 ` Markus Törnqvist 1 sibling, 0 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 19:53 UTC (permalink / raw) To: Hans Reiser, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Joel Becker wrote: > On Thu, Aug 26, 2004 at 01:49:15AM -0700, Hans Reiser wrote: > > Yes, this was part of the plan, tar file-directory plugins would be cute. > > Question: Is "cat /foo/bar/baz.tar.gz/metas" the attribute > directory or a directory in the tarball named "metas"? This needs to be designed. Perhaps /foo/bar/baz.tar.gz/tar/metas is the directory in the tarball named "metas". Or perhaps /foo/bar/baz.tar.gz/x/metas is: it's independent of archive format, and I personally tend to extract things into a directory called "x". [*] Or perhaps /foo/bar/baz.tar.gz/metas is, and the attribute directory is /foo/bar/baz.tar.gz/../metas, to be perverse ;) I prefer the second one, ("x/metas"), but not with any conviction. -- Jamie [*] Actually I prefer: /foo/bar/baz.tar.gz/content/metas /foo/bar/baz-0.01.tar.gz/content/baz-0.01/metas Archives always in "content". One layer of decompression always tried for .tar files and other uncompressed archive formats. /foo/bar/baz.tar.gz/x -> content/ /foo/bar/baz-0.01.tar.gz/x -> content/baz-0.01/ If the root of the archive contains a single directory, "x" is a symlink to it. Otherwise "x" is a symlink to the root directory of the archive. This is comfortable with the common practice by which archives are distributed, without making a mess when someone forgets to put everything in a top-level directory. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:35 ` Joel Becker 2004-08-26 19:53 ` Jamie Lokier @ 2004-08-27 9:19 ` Markus Törnqvist 2004-08-27 12:26 ` Matthew Wilcox 1 sibling, 1 reply; 1392+ messages in thread From: Markus Törnqvist @ 2004-08-27 9:19 UTC (permalink / raw) To: Hans Reiser, Jamie Lokier, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 11:35:32AM -0700, Joel Becker wrote: > Question: Is "cat /foo/bar/baz.tar.gz/metas" the attribute >directory or a directory in the tarball named "metas"? This has been fought over on the reiserfs-list ad nauseaum, but it's a valid point. That's why I tend to rename metas/ into ..metas/ to avoid name clashes, even if I've never had a directory named metas/ apart from what Reiser4 ships. It is then debatable if it should be renamed before it's too late, have it renamable in the kernel configs (and the name exported via /sys or something) or just leave it be. -- mjt ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 9:19 ` Markus Törnqvist @ 2004-08-27 12:26 ` Matthew Wilcox 0 siblings, 0 replies; 1392+ messages in thread From: Matthew Wilcox @ 2004-08-27 12:26 UTC (permalink / raw) To: Markus Törnqvist Cc: Hans Reiser, Jamie Lokier, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Fri, Aug 27, 2004 at 12:19:54PM +0300, Markus Törnqvist wrote: > That's why I tend to rename metas/ into ..metas/ to avoid > name clashes, even if I've never had a directory named metas/ > apart from what Reiser4 ships. http://packages.debian.org/cgi-bin/search_contents.pl?word=metas&searchmode=searchfilesanddirs&case=insensitive&version=unstable&arch=i386 OK, those are capital METAS rather than junior metas, but it does show this is not a unique word to reiser4. -- "Next the statesmen will invent cheap lies, putting the blame upon the nation that is attacked, and every man will be glad of those conscience-soothing falsities, and will diligently study them, and refuse to examine any refutations of them; and thus he will by and by convince himself that the war is just, and will thank God for the better sleep he enjoys after this process of grotesque self-deception." -- Mark Twain ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 1:00 ` Jamie Lokier 2004-08-26 3:13 ` viro 2004-08-26 8:49 ` Hans Reiser @ 2004-08-26 10:05 ` Chris Wedgwood 2004-08-26 11:02 ` Jamie Lokier 2004-08-27 21:01 ` Pavel Machek 2 siblings, 2 replies; 1392+ messages in thread From: Chris Wedgwood @ 2004-08-26 10:05 UTC (permalink / raw) To: Jamie Lokier Cc: viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 02:00:49AM +0100, Jamie Lokier wrote: > One of the big potential uses for file-as-directory is to go inside > archive files, ELF files, .iso files and so on in a convenient way. Arguably this belongs in userspace --- and people have put it there. --cw ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:05 ` Chris Wedgwood @ 2004-08-26 11:02 ` Jamie Lokier 2004-08-26 11:19 ` Spam ` (2 more replies) 2004-08-27 21:01 ` Pavel Machek 1 sibling, 3 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 11:02 UTC (permalink / raw) To: Chris Wedgwood Cc: viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Chris Wedgwood wrote: > On Thu, Aug 26, 2004 at 02:00:49AM +0100, Jamie Lokier wrote: > > > One of the big potential uses for file-as-directory is to go inside > > archive files, ELF files, .iso files and so on in a convenient way. > > Arguably this belongs in userspace --- and people have put it there. I agree that these belong in userspace, and that there's plenty* of userspace code doing a similar thing already. I don't think there's any argument over it. However, as far as I know it's not accessible in a file-as-directory form as yet. In my opinion that is the most natural form and it would be very intuitive to use. I hope we can pick a useful semantics for them, and also provide filesystem-independent plugins with GNU Hurd-like per-user extensibility. -- Jamie * plenty == too much. Gnome, KDE, Emacs and Bash all see different virtual filesystems. (All but Bash implement their own virtual filesystem extensions). That makes them much less useful than they could be. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:02 ` Jamie Lokier @ 2004-08-26 11:19 ` Spam 2004-08-28 0:28 ` Horst von Brand 2004-08-26 15:50 ` Will Dyson 2004-08-27 21:06 ` Pavel Machek 2 siblings, 1 reply; 1392+ messages in thread From: Spam @ 2004-08-26 11:19 UTC (permalink / raw) To: Jamie Lokier Cc: Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > Chris Wedgwood wrote: >> On Thu, Aug 26, 2004 at 02:00:49AM +0100, Jamie Lokier wrote: >> >> > One of the big potential uses for file-as-directory is to go inside >> > archive files, ELF files, .iso files and so on in a convenient way. >> >> Arguably this belongs in userspace --- and people have put it there. > I agree that these belong in userspace, and that there's plenty* of > userspace code doing a similar thing already. I don't think there's > any argument over it. > However, as far as I know it's not accessible in a file-as-directory > form as yet. In my opinion that is the most natural form and it would > be very intuitive to use. I hope we can pick a useful semantics for > them, and also provide filesystem-independent plugins with GNU > Hurd-like per-user extensibility. > -- Jamie > * plenty == too much. > Gnome, KDE, Emacs and Bash all see different virtual filesystems. > (All but Bash implement their own virtual filesystem extensions). > That makes them much less useful than they could be. Exactly, and I doubt they have the ability to join together and create a common uniform standard out of it either. They have just far to different views on Linux and how they want things to be done. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:19 ` Spam @ 2004-08-28 0:28 ` Horst von Brand 0 siblings, 0 replies; 1392+ messages in thread From: Horst von Brand @ 2004-08-28 0:28 UTC (permalink / raw) To: Spam Cc: Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Spam <spam@tnonline.net> said: > > Chris Wedgwood wrote: [...] > > Gnome, KDE, Emacs and Bash all see different virtual filesystems. > > (All but Bash implement their own virtual filesystem extensions). > > That makes them much less useful than they could be. > Exactly, and I doubt they have the ability to join together and > create a common uniform standard out of it either. They have just > far to different views on Linux and how they want things to be done. And the solution is... inventing yet another, incompatible way of doing things that nobody outside its followers will ever use. Inside the kernel. Should have thought of that before. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:02 ` Jamie Lokier 2004-08-26 11:19 ` Spam @ 2004-08-26 15:50 ` Will Dyson 2004-08-26 20:38 ` Lee Revell 2004-08-27 21:06 ` Pavel Machek 2 siblings, 1 reply; 1392+ messages in thread From: Will Dyson @ 2004-08-26 15:50 UTC (permalink / raw) To: Jamie Lokier Cc: Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Jamie Lokier wrote: > However, as far as I know it's not accessible in a file-as-directory > form as yet. In my opinion that is the most natural form and it would > be very intuitive to use. I hope we can pick a useful semantics for > them, and also provide filesystem-independent plugins with GNU > Hurd-like per-user extensibility. > > -- Jamie > > * plenty == too much. > Gnome, KDE, Emacs and Bash all see different virtual filesystems. > (All but Bash implement their own virtual filesystem extensions). > That makes them much less useful than they could be. It has always bugged me that Gnome and KDE implement their own VFS layers. It seems to me that having a standard userspace filesystem layer like FUSE (http://sourceforge.net/projects/avf) would provide a lot of the benefit that HURD users (all 5 of them) see from filesystem "translators". Now, safely allowing unprivileged users to run arbitrary userspace filesystems would be a real trick. But, if it were possible, it could be combined with files-as-directories in some rather neat ways. -- Will Dyson "Back off man, I'm a scientist!" -Dr. Peter Venkman ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 15:50 ` Will Dyson @ 2004-08-26 20:38 ` Lee Revell 2004-08-26 20:50 ` Christophe Saout 2004-08-26 21:00 ` Jamie Lokier 0 siblings, 2 replies; 1392+ messages in thread From: Lee Revell @ 2004-08-26 20:38 UTC (permalink / raw) To: Will Dyson Cc: Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2004-08-26 at 11:50, Will Dyson wrote: > Jamie Lokier wrote: > > > However, as far as I know it's not accessible in a file-as-directory > > form as yet. In my opinion that is the most natural form and it would > > be very intuitive to use. I hope we can pick a useful semantics for > > them, and also provide filesystem-independent plugins with GNU > > Hurd-like per-user extensibility. > > > > -- Jamie > > > > * plenty == too much. > > Gnome, KDE, Emacs and Bash all see different virtual filesystems. > > (All but Bash implement their own virtual filesystem extensions). > > That makes them much less useful than they could be. > > It has always bugged me that Gnome and KDE implement their own VFS layers. > Same here. This always seemed like something the kernel should be able to handle. It seems to me that if reiser4 had been available at the time the Gnome and KDE developers would not have needed to do this. Lee ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:38 ` Lee Revell @ 2004-08-26 20:50 ` Christophe Saout 2004-08-26 20:57 ` Lee Revell 2004-08-26 21:00 ` Jamie Lokier 1 sibling, 1 reply; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 20:50 UTC (permalink / raw) To: Lee Revell Cc: Will Dyson, Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 801 bytes --] Am Donnerstag, den 26.08.2004, 16:38 -0400 schrieb Lee Revell: > > It has always bugged me that Gnome and KDE implement their own VFS layers. > > Same here. This always seemed like something the kernel should be able > to handle. It seems to me that if reiser4 had been available at the > time the Gnome and KDE developers would not have needed to do this. Well, the kernel doesn't have a filesystem that speaks http, scp and those things. GnomeVFS is URL-based. It has some pseudo-protocols that extract a pseudo directory-tree for all installed applications + the changes the used made, created on the fly from a set of XML files that are read-only and system-wide and the user-overridden changes. I don't know if all of these things would really make sense inside the kernel. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:50 ` Christophe Saout @ 2004-08-26 20:57 ` Lee Revell 2004-08-27 9:21 ` Markus Törnqvist 0 siblings, 1 reply; 1392+ messages in thread From: Lee Revell @ 2004-08-26 20:57 UTC (permalink / raw) To: Christophe Saout Cc: Will Dyson, Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2004-08-26 at 16:50, Christophe Saout wrote: > Am Donnerstag, den 26.08.2004, 16:38 -0400 schrieb Lee Revell: > > > > It has always bugged me that Gnome and KDE implement their own VFS layers. > > > > Same here. This always seemed like something the kernel should be able > > to handle. It seems to me that if reiser4 had been available at the > > time the Gnome and KDE developers would not have needed to do this. > > Well, the kernel doesn't have a filesystem that speaks http, scp and > those things. GnomeVFS is URL-based. It has some pseudo-protocols that > extract a pseudo directory-tree for all installed applications + the > changes the used made, created on the fly from a set of XML files that > are read-only and system-wide and the user-overridden changes. I don't > know if all of these things would really make sense inside the kernel. > True. FWIW, I never use most of those features. It's just too damn slow. Windows seems to implement all of the useful features of GnomeVFS, and they are 10x faster. Lee ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:57 ` Lee Revell @ 2004-08-27 9:21 ` Markus Törnqvist 2004-08-27 11:34 ` Spam 0 siblings, 1 reply; 1392+ messages in thread From: Markus Törnqvist @ 2004-08-27 9:21 UTC (permalink / raw) To: Lee Revell Cc: Christophe Saout, Will Dyson, Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 04:57:26PM -0400, Lee Revell wrote: >On Thu, 2004-08-26 at 16:50, Christophe Saout wrote: >> are read-only and system-wide and the user-overridden changes. I don't >> know if all of these things would really make sense inside the kernel. >True. FWIW, I never use most of those features. It's just too damn >slow. Windows seems to implement all of the useful features of >GnomeVFS, and they are 10x faster. Are they in the kernel in Windows? -- mjt ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 9:21 ` Markus Törnqvist @ 2004-08-27 11:34 ` Spam 0 siblings, 0 replies; 1392+ messages in thread From: Spam @ 2004-08-27 11:34 UTC (permalink / raw) To: Markus Törnqvist Cc: Lee Revell, Christophe Saout, Will Dyson, Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hello Markus, Friday, August 27, 2004, 11:21:31 AM, you wrote: > On Thu, Aug 26, 2004 at 04:57:26PM -0400, Lee Revell wrote: >>On Thu, 2004-08-26 at 16:50, Christophe Saout wrote: >>> are read-only and system-wide and the user-overridden changes. I don't >>> know if all of these things would really make sense inside the kernel. >>True. FWIW, I never use most of those features. It's just too damn >>slow. Windows seems to implement all of the useful features of >>GnomeVFS, and they are 10x faster. > Are they in the kernel in Windows? Some are, some aren't :) Many of the features are between the FS and the application layers. Therefore they will work with all programs that use the Windows API to access files etc. I do not think that many programs actually access disks directly at the FS or below level. Exceptions would perhaps be file-recovery tools etc. Others are simply plugins to Windows Explorer. I don't know if all, any, or none, are installed at the kernel level. I doubt they are. But the problem with Linux seem to be that programs actually do access the FS directly. Therefore everything will break if you change the FS structures to much. Like I said, this is IMHO something that Linux should somehow solve. Applications should really not access the filesystems directly. Perhaps Grub and stuff like it should, but not many other. ~S ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:38 ` Lee Revell 2004-08-26 20:50 ` Christophe Saout @ 2004-08-26 21:00 ` Jamie Lokier 1 sibling, 0 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 21:00 UTC (permalink / raw) To: Lee Revell Cc: Will Dyson, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Lee Revell wrote: > Same here. This always seemed like something the kernel should be able > to handle. It seems to me that if reiser4 had been available at the > time the Gnome and KDE developers would not have needed to do this. reiser4 has prompted the discussion again, but it doesn't provide the virtual filesystem capabilities that we're talking about, which Gnome and KDE implement in userspace. The FUSE project is much more relevant for that. We had file-as-directory discussions years ago, and 5 years ago I was experimenting with doing virtual filesystems with that capability over NFS. podfuk offered something close to Gnome and KDE but in kernel space: virtual filesytems accessible through the kernel, but it didn't get too popular. It was based on Midnight Commander's VFS (which has evolved into Gnome's VFS), using the CODA filesystem interface to provide the kernel hooks. It had some problems, and was a bit fiddly to use. We still don't have something that's particularly nice to use for ordinary users. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:02 ` Jamie Lokier 2004-08-26 11:19 ` Spam 2004-08-26 15:50 ` Will Dyson @ 2004-08-27 21:06 ` Pavel Machek 2004-08-31 1:00 ` David Masover 2 siblings, 1 reply; 1392+ messages in thread From: Pavel Machek @ 2004-08-27 21:06 UTC (permalink / raw) To: Jamie Lokier Cc: Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hi! > > > One of the big potential uses for file-as-directory is to go inside > > > archive files, ELF files, .iso files and so on in a convenient way. > > > > Arguably this belongs in userspace --- and people have put it there. > > I agree that these belong in userspace, and that there's plenty* of > userspace code doing a similar thing already. I don't think there's > any argument over it. > > However, as far as I know it's not accessible in a file-as-directory > form as yet. In my opinion that is the most natural form and it would > be very intuitive to use. It does not work. .deb file is ar archive. It is also debian package. How do you know if I want to see it as a package or as a archive? How do you identify file types, anyway? Same happens for .tar.gz. uservfs does cd foo.deb#uar vs. cd foo.deb#udeb and cd foo.tgz#utar vs. cat foo.tgz#ugz -- 64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 21:06 ` Pavel Machek @ 2004-08-31 1:00 ` David Masover 2004-08-31 8:21 ` Pavel Machek 0 siblings, 1 reply; 1392+ messages in thread From: David Masover @ 2004-08-31 1:00 UTC (permalink / raw) To: Pavel Machek Cc: Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Pavel Machek wrote: [...] | uservfs does | | cd foo.deb#uar cd foo.deb/ar | vs. | cd foo.deb#udeb cd foo.deb/deb and why would you want that, instead of just: cd foo.deb # for the ar dpkg -i foo.deb # for the deb | | and | | cd foo.tgz#utar cd foo.tgz | vs. | cat foo.tgz#ugz cat foo.tgz Just want to extract the tar file? Maybe something like cat foo.tgz/gunzip In which case (of course) foo.tgz/gunzip has exactly the same directory contents as foo.tgz Looks different, that's all. In fact, for just about any syntax anyone could suggest, I can't really see why you can't just replace all weird symbols with a slash and a symbol. Instead of foo.tgz#utar you have foo.tgz/#/utar Only difference is, some things which used to require special tools can now be serviced by less than what's in busybox. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTPNpngHNmZLgCUhAQJ0PA//RmPgHL4MgiQn7Li5b52DI/0p+Zp9JUs7 OaA+k4wHeXEIdBProtBQ+3541noEg086bVriYRnajsA9xgd/P9ncE7ci/d9Kagdl 8KmH4eWnCjNkL4x8GnDeR+EM5EfcgoSxR7ezO1KO9uMMcPFnCvmeB1yB5U+Fr9Q1 HVPHfGsh+ZWU2CFJHoJx27Q07UK+bFvzGr4JkhLbRMbPeEROrkHJHp3wj/N2yXXb 5icLgFa0g/b6wM8SuBAaO5xVM8jkmECI/P5Jo3n/d/CTUumb/BoKMIgbMurR61CG hrZ40mizpCMFAnLrgm9FIvjKtZUsigG+oM/2xTtku2Z2nSM16/ChKtJOAj6MoMGT xvWzLdTY6kEL30VDaHtcVvIyUFtUe5oYxEasnrcKseV7hJtccgjTYKB21PnWyZFz /KIdh8sBg6i1nuYOtHncL6agD1M2bg1vBYwbMvIRfa1YbAFUbW1A6hDTftxJIhLn Beso6d68SyfNMdQ8yWloR7sIufmbDut2fx1SHS3wPt2Z5W1e0XBexcpSqdD8KPns lvmC0ggPDuQzXjPCIsS17Uk5vrCtVEdrsCCaj62a8UE2d1mCqu4CyLwMYCFv3Sve SwZ36iOv4fzdvgjPOUg27tMwbpa/WiDAcZJ/31BfZfTDl4E9LbkQjWYT/YgsJuaQ wp3Hi/Cgh/4= =8YVp -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 1:00 ` David Masover @ 2004-08-31 8:21 ` Pavel Machek 2004-08-31 19:31 ` Horst von Brand 2004-09-01 1:10 ` David Masover 0 siblings, 2 replies; 1392+ messages in thread From: Pavel Machek @ 2004-08-31 8:21 UTC (permalink / raw) To: David Masover Cc: Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hi! > | uservfs does > | > | cd foo.deb#uar > cd foo.deb/ar > | vs. > | cd foo.deb#udeb > cd foo.deb/deb > > and why would you want that, instead of just: > cd foo.deb # for the ar > dpkg -i foo.deb # for the deb Because I want to see contents of that .deb package, nicely parsed? > Just want to extract the tar file? Maybe something like > cat foo.tgz/gunzip > In which case (of course) foo.tgz/gunzip has exactly the same directory > contents as foo.tgz Yes, that would work. > In fact, for just about any syntax anyone could suggest, I can't really > see why you can't just replace all weird symbols with a slash and a > symbol. Instead of > foo.tgz#utar > you have > foo.tgz/#/utar > Only difference is, some things which used to require special tools can > now be serviced by less than what's in busybox. That would work, too. I do not get your comment about busybox. Pavel -- People were complaining that M$ turns users into beta-testers... ...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl! ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 8:21 ` Pavel Machek @ 2004-08-31 19:31 ` Horst von Brand 2004-08-31 20:05 ` Linus Torvalds 2004-09-01 1:14 ` David Masover 2004-09-01 1:10 ` David Masover 1 sibling, 2 replies; 1392+ messages in thread From: Horst von Brand @ 2004-08-31 19:31 UTC (permalink / raw) To: Pavel Machek Cc: David Masover, Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Pavel Machek <pavel@ucw.cz> said: > David Masover <ninja@slaphack.com> said: [...] > > Just want to extract the tar file? Maybe something like > > cat foo.tgz/gunzip > > In which case (of course) foo.tgz/gunzip has exactly the same directory > > contents as foo.tgz > > Yes, that would work. > > > In fact, for just about any syntax anyone could suggest, I can't really > > see why you can't just replace all weird symbols with a slash and a > > symbol. Instead of > > foo.tgz#utar > > you have > > foo.tgz/#/utar > > Only difference is, some things which used to require special tools can > > now be serviced by less than what's in busybox. > > That would work, too. I do not get your comment about busybox. You do need extra tools anyway, placing them in the kernel is cheating (and absolutely pointless, IMHO). -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 19:31 ` Horst von Brand @ 2004-08-31 20:05 ` Linus Torvalds 2004-08-31 20:29 ` Spam ` (4 more replies) 2004-09-01 1:14 ` David Masover 1 sibling, 5 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-08-31 20:05 UTC (permalink / raw) To: Horst von Brand Cc: Pavel Machek, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Tue, 31 Aug 2004, Horst von Brand wrote: > > You do need extra tools anyway, placing them in the kernel is cheating (and > absolutely pointless, IMHO). I agree. There's no point to having the kernel export information that is already inherent in the main stream. I've seen all these examples of exposing MP3 ID information as a "side stream", and that's TOTALLY POINTLESS! The information is already there, it's in a standard format, and exporting it as a stream buys you absolutely nothing. Where named attributes make sense is when they are _independent_ data. Data that is only tangentially related to the main stream itself, and which the main stream _cannot_ encompass because of some real technical issue. In a graphical environment, the "icon" stream is a good example of this. It literally has _nothing_ to do with the data in the main stream. The only linkage is a totally non-technical one, where the user wanted to associate a secondary stream with the main stream _without_ altering the main one. THAT is where named streams make sense. But if you want to look at one particular file inside a tar-file, do so in user space. There are zero advantages to exposing it as a side-stream, and there are absolutely _tons_ of disadvantages. In short, named streams only make sense if: - they are tied to the file some way that is _independent_ of the file contents (since if it's dependent on the file contents, you're just a ton better off regenerating it with a caching server) - there are serious reasons to keep the lookup synchronized (since if there isn't such a reason, you're just better off with a separate shadow tree in user space) And realize that the "separate shadow tree" actually works very well. That's how version control systems like CVS have always worked. It's certainly how you can make icon information work too. If you use a tool for accessing the data, the tool can maintain coherency and you'll never care about the side stream. Which means that normally we really don't _want_ named streams. In 99% of all cases we can use equally good - and _much_ simpler - tool-based solutions. Which means that the only _real_ technical issue for supporting named streams really ends up being things like samba, which want named streams just because the work they do fundamentally is about them, for externally dictated reasons. Doing named streams for any other reason is likely just being stupid. Once you do decide that you have to do named streams, you might then decide to use them for convenient things like icons. But it should very much be a secondary issue at that point. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 20:05 ` Linus Torvalds @ 2004-08-31 20:29 ` Spam 2004-08-31 20:32 ` Pavel Machek ` (3 subsequent siblings) 4 siblings, 0 replies; 1392+ messages in thread From: Spam @ 2004-08-31 20:29 UTC (permalink / raw) To: Linus Torvalds Cc: Horst von Brand, Pavel Machek, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > On Tue, 31 Aug 2004, Horst von Brand wrote: >> >> You do need extra tools anyway, placing them in the kernel is cheating (and >> absolutely pointless, IMHO). > I agree. > There's no point to having the kernel export information that is already > inherent in the main stream. There are two things you are saying in the same sentence here. One I do not agree with. There is point to export information from the main stream to secondary file streams. Just it doesn't have to be done by the kernel. Just do not limit things to a few defined types of things that can be stored in a stream. IMHO a file stream should be able to contain just about anything the user wishes. That said, the kernel should not have to know about the contents. > I've seen all these examples of exposing MP3 ID information as a "side > stream", and that's TOTALLY POINTLESS! The information is already there, > it's in a standard format, and exporting it as a stream buys you > absolutely nothing. True. But this is the case for this specific example. I have stated previously about what I think would be legitimate uses as examples. Of course these apply to me and what I work with. One of the examples were to add streams that contained descriptions or keywords. > Where named attributes make sense is when they are _independent_ data. > Data that is only tangentially related to the main stream itself, and > which the main stream _cannot_ encompass because of some real technical > issue. > In a graphical environment, the "icon" stream is a good example of this. > It literally has _nothing_ to do with the data in the main stream. The > only linkage is a totally non-technical one, where the user wanted to > associate a secondary stream with the main stream _without_ altering the > main one. THAT is where named streams make sense. > But if you want to look at one particular file inside a tar-file, do so in > user space. There are zero advantages to exposing it as a side-stream, and > there are absolutely _tons_ of disadvantages. The tar archive support seem (is) to be useless because tar is also so supported and implemented in every type of application out there. It is also easy to script and otherwise work with. But the idea to be able to extend the filesystem to show/represent data as file streams and meta directores is very good. Someone might come up with an idea to allow browsing and changing of data inside XML files. Of course this may already be done with sed and other tools. > In short, named streams only make sense if: > - they are tied to the file some way that is _independent_ of the file > contents (since if it's dependent on the file contents, you're just a > ton better off regenerating it with a caching server) > - there are serious reasons to keep the lookup synchronized (since if > there isn't such a reason, you're just better off with a separate > shadow tree in user space) > And realize that the "separate shadow tree" actually works very well. > That's how version control systems like CVS have always worked. It's > certainly how you can make icon information work too. If you use a tool > for accessing the data, the tool can maintain coherency and you'll never > care about the side stream. Just that with shadow trees (I assume they are stored simply in another folder beside the file) you won't be able to "cp filea /mnt/backup" and keep the extra stuff. Sure icons could also easily be stores as file.icon, just like on the Amiga :). It just isn't as nice. > Which means that normally we really don't _want_ named streams. In 99% of > all cases we can use equally good - and _much_ simpler - tool-based > solutions. > Which means that the only _real_ technical issue for supporting named > streams really ends up being things like samba, which want named streams > just because the work they do fundamentally is about them, for externally > dictated reasons. Doing named streams for any other reason is likely just > being stupid. I doubt it would be stupid. Perhaps today just not needed for many things. > Once you do decide that you have to do named streams, you might then > decide to use them for convenient things like icons. But it should very > much be a secondary issue at that point. I think we focus to much on the actual contents of these streams. Do we really have to know what will in the future be stored inside these streams? For example. Applications may use file streams for locking instead of having a .lock file. The future isn't known, and we shouldn't try to limit what we can put in the streams. You said 99% of the uses won't need them. True. The same is with xattr, ACLs and so many other things. Even if 0.1% of the users today would use it I think we should have the support for named streams. ~S > Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 20:05 ` Linus Torvalds 2004-08-31 20:29 ` Spam @ 2004-08-31 20:32 ` Pavel Machek 2004-08-31 20:38 ` Linus Torvalds 2004-08-31 23:02 ` Christer Weinigel ` (2 subsequent siblings) 4 siblings, 1 reply; 1392+ messages in thread From: Pavel Machek @ 2004-08-31 20:32 UTC (permalink / raw) To: Linus Torvalds Cc: Horst von Brand, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hi! > > You do need extra tools anyway, placing them in the kernel is cheating (and > > absolutely pointless, IMHO). > > I agree. > > There's no point to having the kernel export information that is already > inherent in the main stream. > > I've seen all these examples of exposing MP3 ID information as a "side > stream", and that's TOTALLY POINTLESS! The information is already there, > it's in a standard format, and exporting it as a stream buys you > absolutely nothing. It buys me caching. I do quite often bzcat patch.2.6.8.bz2 | less (read the patch) (sometimes repeat that few times because I hit ^c when I should not have etc). cd ...clean; bzcat patch.2.6.8.bz2 | patch -Esp1 cd ...linux; bzcat patch.2.6.8.bz2 | patch -Esp1 Now... that's total waste of cpu. bzip2 decompression takes quite some time. I could do bzcat patch.2.6.8.bz2 > /tmp/delete.me.when.you.are.done ...but I'd probably forget to delete that one and anyway, it requires me to think about it. Nicest way would be cat patch.2.6.8.bz2/ubz | less cd ...clean; cat patch.2.6.8.bz2/ubz | patch -Esp1 cd ...linux; cat patch.2.6.8.bz2/ubz | patch -Esp1 with kernel intelligently caching uncompressed data. I believe this can not be done completely in userspace. Pavel -- People were complaining that M$ turns users into beta-testers... ...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl! ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 20:32 ` Pavel Machek @ 2004-08-31 20:38 ` Linus Torvalds 2004-08-31 20:54 ` Pavel Machek 0 siblings, 1 reply; 1392+ messages in thread From: Linus Torvalds @ 2004-08-31 20:38 UTC (permalink / raw) To: Pavel Machek Cc: Horst von Brand, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Tue, 31 Aug 2004, Pavel Machek wrote: > > It buys me caching. I'll actually buy into that. If only because I consider caching to be one of the more important things that the kernel does (caches are a _classic_ case of "shared data that needs synchronization"). However, that said, user space can trivially cache things in the filesystem, so while this may be a convenient feature, I think you should look at perhaps doing it in the _shell_ instead.. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 20:38 ` Linus Torvalds @ 2004-08-31 20:54 ` Pavel Machek 2004-08-31 20:59 ` Linus Torvalds 0 siblings, 1 reply; 1392+ messages in thread From: Pavel Machek @ 2004-08-31 20:54 UTC (permalink / raw) To: Linus Torvalds Cc: Horst von Brand, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hi! > > It buys me caching. > > I'll actually buy into that. If only because I consider caching to be one > of the more important things that the kernel does (caches are a _classic_ > case of "shared data that needs synchronization"). > > However, that said, user space can trivially cache things in the > filesystem, so while this may be a convenient feature, I think you should > look at perhaps doing it in the _shell_ instead.. That cache should disappear as soon as I need disk space. I.e. userspace should never see -ENOSPC because of this kind of caching. This need some kernel support. Ouch and cached file should atomically go away as soon as main file changes, otherwise I do not see how multiple processes could cooperate on caching... chattr +kill-this-file-when-low-on-disk-space patch.bz2...ubz would solve first problem. Not sure how to do the second one. Pavel -- People were complaining that M$ turns users into beta-testers... ...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl! ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 20:54 ` Pavel Machek @ 2004-08-31 20:59 ` Linus Torvalds 2004-08-31 22:07 ` Pavel Machek 0 siblings, 1 reply; 1392+ messages in thread From: Linus Torvalds @ 2004-08-31 20:59 UTC (permalink / raw) To: Pavel Machek Cc: Horst von Brand, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Tue, 31 Aug 2004, Pavel Machek wrote: > > > > However, that said, user space can trivially cache things in the > > filesystem, so while this may be a convenient feature, I think you should > > look at perhaps doing it in the _shell_ instead.. > > That cache should disappear as soon as I need disk > space. I.e. userspace should never see -ENOSPC because of this kind of > caching. This need some kernel support. Ouch and cached file should > atomically go away as soon as main file changes, otherwise I do not > see how multiple processes could cooperate on caching... Well, what other projects have done is to just reserve a certain amount of disk for caching. See "ccache", which solves both of the above problems (it doesn't shrink the cache on ENOSPC, but reserving diskspace is accepted practice for things like this..) Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 20:59 ` Linus Torvalds @ 2004-08-31 22:07 ` Pavel Machek 2004-08-31 23:19 ` Christer Weinigel 0 siblings, 1 reply; 1392+ messages in thread From: Pavel Machek @ 2004-08-31 22:07 UTC (permalink / raw) To: Linus Torvalds Cc: Horst von Brand, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hi! > > > However, that said, user space can trivially cache things in the > > > filesystem, so while this may be a convenient feature, I think you should > > > look at perhaps doing it in the _shell_ instead.. > > > > That cache should disappear as soon as I need disk > > space. I.e. userspace should never see -ENOSPC because of this kind of > > caching. This need some kernel support. Ouch and cached file should > > atomically go away as soon as main file changes, otherwise I do not > > see how multiple processes could cooperate on caching... > > Well, what other projects have done is to just reserve a certain amount of > disk for caching. See "ccache", which solves both of the above problems > (it doesn't shrink the cache on ENOSPC, but reserving diskspace is > accepted practice for things like this..) Okay, that does work, it just is not really nice. Just as reserving fixed ammount of space for disk cache is bad, reserving fixed ammount of space for ccache (and similar) is bad. When there are few of such caches, balancing between them starts to matter... Pavel -- People were complaining that M$ turns users into beta-testers... ...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl! ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 22:07 ` Pavel Machek @ 2004-08-31 23:19 ` Christer Weinigel 2004-09-01 1:43 ` David Masover ` (2 more replies) 0 siblings, 3 replies; 1392+ messages in thread From: Christer Weinigel @ 2004-08-31 23:19 UTC (permalink / raw) To: Pavel Machek Cc: Linus Torvalds, Horst von Brand, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Pavel Machek <pavel@ucw.cz> writes: > Okay, that does work, it just is not really nice. Just as reserving > fixed ammount of space for disk cache is bad, reserving fixed ammount > of space for ccache (and similar) is bad. When there are few of such > caches, balancing between them starts to matter... So try to convince them to use the same cache daemon or the same shared cache manager library then. It isn't all that different from implementing a kernel interface that everyone is supposed to use. A cache manager daemon could sit and watch the free space on the disk every other second and start deleting the cached files (according to some LCU heuristics or whatever) whenever free space is getting low. To see if the original file is newer than the cached file, good old mtime can be used. I can't remember who right now, but someone mentioned a few problems with timestamps: 1. The resolutions is too low in current kernels, but that just means that someone(tm) needs to implement nanosecond timestamps in the kernel and for the filesystems. 2. Even nanosecond timestams may become to coarse in the future. One way of getting around this is to keep track of the latest timestamp written and make sure the next timestamp is one higher than the latest one. This way timestamps are always increasing, and I doubt that we can have modifications each nanosecond for a long time with the hardware available for the next ten years. 3. If time goes backwards beacuse somebody changes the clock, the caching (or makefile rules) can get confued. This is a hard one, but I actually we can use the doctors response "don't do that then". With NTP we can guarantee a monotonically increasing clock. I belive the kernel could give some assistance to make it easier to see if a file has been modified, I remember that a few suggestions were thrown around the last time Samba and dcache aliases were discussed on l-k. I definitely belive that kind of infrastructure belongs in the kernel. But the cache manager itself, no. /Christer -- "Just how much can I get away with and still go to heaven?" Freelance consultant specializing in device driver programming for Linux Christer Weinigel <christer@weinigel.se> http://www.weinigel.se ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 23:19 ` Christer Weinigel @ 2004-09-01 1:43 ` David Masover 2004-09-01 5:02 ` Pavel Machek 2004-09-05 11:22 ` Tonnerre 2 siblings, 0 replies; 1392+ messages in thread From: David Masover @ 2004-09-01 1:43 UTC (permalink / raw) To: Christer Weinigel Cc: Pavel Machek, Linus Torvalds, Horst von Brand, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Christer Weinigel wrote: | Pavel Machek <pavel@ucw.cz> writes: | | |>Okay, that does work, it just is not really nice. Just as reserving |>fixed ammount of space for disk cache is bad, reserving fixed ammount |>of space for ccache (and similar) is bad. When there are few of such |>caches, balancing between them starts to matter... | | | So try to convince them to use the same cache daemon or the same | shared cache manager library then. It isn't all that different from | implementing a kernel interface that everyone is supposed to use. You're right, it's not. Is there some huge performance loss by having a kernel interface that then turns around and calls a userland daemon? If yes, I can understand the argument. If no, then the kernel interface should ultimately be able to replace gnome-vfs -- and it should rely heavily on userland utilities. Caching is very generic. It's so generic that there should be exactly one cache manager, the way there is exactly one VM manager for Linux. If there's a choice, choose it at boot or compile-time. Abiword uses gzipped files. So does gnumeric, gnucash, cube (a cute little game engine), and many others. There's no way every single one will suddenly use a new library/daemon. But with plugins and intelligent caching, any kind of compression could be supported without the app needing to support it natively -- either it goes in a compressed folder or the user browses to it and changes an attribute. | A cache manager daemon could sit and watch the free space on the disk | every other second and start deleting the cached files (according to | some LCU heuristics or whatever) whenever free space is getting low. Agreed. The daemon should go in userspace. But the interface should be in the kernel. Just as no one suggested putting tar in the kernel, I don't think anyone would really insist that the daemon needs to go in the kernel. That is, unless critical system files start actually being cached streams of the actual package from the distro... But this way leads to madness. Any serious issues here are better solved by an initrd. [...] | I belive the kernel could give some assistance to make it easier to | see if a file has been modified, I remember that a few suggestions | were thrown around the last time Samba and dcache aliases were | discussed on l-k. I definitely belive that kind of infrastructure | belongs in the kernel. But the cache manager itself, no. Isn't there already dnotify? Dnotify is wrong here, because I don't think it would be sane in a sufficiently large setup. But wouldn't it be nice if the kernel not only told you whether a file has been modified, but told you immediately after it was modified? Yet another reason why there has to be a common cache and significant kernel management -- if I've got a drive of less than, say, 5 gigs, I am simply not willing to reserve any large amount of it for cache. Take the example of a source tarball being automatically extracted and the contents cached. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTUpPHgHNmZLgCUhAQLhIg/9G87W4OCt8hOvTK+XniPgWGn+YC7t4fKy mi4QowX1eLmnFEovprjSUzPgKRMeDSChZ8M1pvXXWJQ/iULE44WH8vNiH8H974zw h5O8dNI/AczuzaCalQ+BYXEGM8pasaCGCI/sLrr+WBy97NOpM2VUUKY+qKxSFWFt x7u7/XCGGWSHVG3t3KzpkqD2K8u3IGSrV/vHIR2PLatEXu3G8K8sUAhfXMgFsTFp gQAuQN/YzQKGUphnMzFWLDMYyEYjAA4VACZFqFz9QJIYeWzP+YmGdLfZ5mzN91JF g5ACGHUxbYdjZsjTh6RbalVr1HpAAy8mvGNydbwUvuOxaXBvH44vUfRLnvIUoZs9 BB0Tq4xPqAwAtP0Z7b5Qem1ef3gqGXo1hfsEJWJZPv1lssYzn1+Jpn99p6N6VkXI fbAHnypbHeYbf30lC5lvID0yngb+s5EUaxzxOYqFA8QpEJD/bl3S9Qx6bpom/9AL U8kdK8QjNbhHSMrOpl8mg77VRLXdINoV7R5iSXXUMdIcA1j+4R1AIPh62JAMwQDF /Vzek8KNyxvJDgF/y78LiGETZcG/l2W4szEsJFFF4iIKLri5+OFGiLFz8jVNViKu Lr7Q7u7TgWSSykdTrHFcR/WRupAHeL5RB5DA9HBZErqgAgJWz/Rjv743G8PYFKs8 oPhRyePnFkE= =cQLE -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 23:19 ` Christer Weinigel 2004-09-01 1:43 ` David Masover @ 2004-09-01 5:02 ` Pavel Machek 2004-09-05 11:22 ` Tonnerre 2 siblings, 0 replies; 1392+ messages in thread From: Pavel Machek @ 2004-09-01 5:02 UTC (permalink / raw) To: Christer Weinigel Cc: Linus Torvalds, Horst von Brand, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hi! > I belive the kernel could give some assistance to make it easier to > see if a file has been modified, I remember that a few suggestions > were thrown around the last time Samba and dcache aliases were > discussed on l-k. I definitely belive that kind of infrastructure > belongs in the kernel. But the cache manager itself, no. Well, callback "filesystem full" would be nice, too; but I guess that is "support user cache", too... Pavel -- When do you have heart between your knees? ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 23:19 ` Christer Weinigel 2004-09-01 1:43 ` David Masover 2004-09-01 5:02 ` Pavel Machek @ 2004-09-05 11:22 ` Tonnerre 2 siblings, 0 replies; 1392+ messages in thread From: Tonnerre @ 2004-09-05 11:22 UTC (permalink / raw) To: Christer Weinigel Cc: Pavel Machek, Linus Torvalds, Horst von Brand, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 416 bytes --] Salut, On Wed, Sep 01, 2004 at 01:19:00AM +0200, Christer Weinigel wrote: > To see if the original file is newer than the cached file, good old > mtime can be used. I'd not use a daemon or any mtime for it. I'd just put it into xattrs and let it up to the applications to poke the values in case they changed. And copy them on cp, since it would have to be changed either way. Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 20:05 ` Linus Torvalds 2004-08-31 20:29 ` Spam 2004-08-31 20:32 ` Pavel Machek @ 2004-08-31 23:02 ` Christer Weinigel 2004-08-31 23:20 ` Spam 2004-09-05 11:17 ` Tonnerre 2004-09-01 12:10 ` Alan Cox 2004-09-01 19:44 ` Markus Törnqvist 4 siblings, 2 replies; 1392+ messages in thread From: Christer Weinigel @ 2004-08-31 23:02 UTC (permalink / raw) To: Linus Torvalds Cc: Horst von Brand, Pavel Machek, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Linus Torvalds <torvalds@osdl.org> writes: > In a graphical environment, the "icon" stream is a good example of this. > It literally has _nothing_ to do with the data in the main stream. The > only linkage is a totally non-technical one, where the user wanted to > associate a secondary stream with the main stream _without_ altering the > main one. THAT is where named streams make sense. I think that the "icon" argument for named streams is a silly argument, since different users may want to have different icons for the same file. Say that I want /usr/bin/emacs to have the enterprise icon and someone else wants the gnu head icon. And besides, root owns the file anyways, so neither of us mortal users should be able to add a stream to it. Another reason for named streams that usually crops up is the ability set a "preferred application" for a certain file, so that when I double click on a document I want to open it with antiword instead of openoffice. But the same contra-argument applies here, different users have different preferences. I can see the argument for having the equivalent of Content-type or Content-transfer-encoding as a named stream though. /Christer -- "Just how much can I get away with and still go to heaven?" Freelance consultant specializing in device driver programming for Linux Christer Weinigel <christer@weinigel.se> http://www.weinigel.se ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 23:02 ` Christer Weinigel @ 2004-08-31 23:20 ` Spam 2004-09-05 11:17 ` Tonnerre 1 sibling, 0 replies; 1392+ messages in thread From: Spam @ 2004-08-31 23:20 UTC (permalink / raw) To: Christer Weinigel Cc: Linus Torvalds, Horst von Brand, Pavel Machek, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > Linus Torvalds <torvalds@osdl.org> writes: >> In a graphical environment, the "icon" stream is a good example of this. >> It literally has _nothing_ to do with the data in the main stream. The >> only linkage is a totally non-technical one, where the user wanted to >> associate a secondary stream with the main stream _without_ altering the >> main one. THAT is where named streams make sense. > I think that the "icon" argument for named streams is a silly > argument, since different users may want to have different icons for > the same file. Say that I want /usr/bin/emacs to have the enterprise > icon and someone else wants the gnu head icon. And besides, root owns > the file anyways, so neither of us mortal users should be able to add > a stream to it. Yet again are we thinking in blocking ways. Firstly this was an example. Usually, though, most users accept the default icon for a file. If they do not they can still change the icon for the link they make on their start-menu/home folder/etc. > Another reason for named streams that usually crops up is the ability > set a "preferred application" for a certain file, so that when I > double click on a document I want to open it with antiword instead of > openoffice. But the same contra-argument applies here, different > users have different preferences. I can make the same argument as for the icons. > I can see the argument for having the equivalent of Content-type or > Content-transfer-encoding as a named stream though. That would be a nice thing. > /Christer ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 23:02 ` Christer Weinigel 2004-08-31 23:20 ` Spam @ 2004-09-05 11:17 ` Tonnerre 2004-09-05 11:40 ` Grzegorz Jaśkiewicz 2004-09-05 11:57 ` Spam 1 sibling, 2 replies; 1392+ messages in thread From: Tonnerre @ 2004-09-05 11:17 UTC (permalink / raw) To: Christer Weinigel Cc: Linus Torvalds, Horst von Brand, Pavel Machek, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 407 bytes --] Salut, On Wed, Sep 01, 2004 at 01:02:24AM +0200, Christer Weinigel wrote: > I can see the argument for having the equivalent of Content-type or > Content-transfer-encoding as a named stream though. Why having them as named streams if we can get them as xattrs for no additional pain? (Since fileutils would have to be changed anyway, we can even make cp copy and emacs update xattrs.) Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-05 11:17 ` Tonnerre @ 2004-09-05 11:40 ` Grzegorz Jaśkiewicz 2004-09-05 11:57 ` Spam 1 sibling, 0 replies; 1392+ messages in thread From: Grzegorz Jaśkiewicz @ 2004-09-05 11:40 UTC (permalink / raw) To: Tonnerre Cc: Christer Weinigel, Linus Torvalds, Horst von Brand, Pavel Machek, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Sun, 5 Sep 2004 13:17:43 +0200, Tonnerre <tonnerre@thundrix.ch> wrote: > Salut, > > On Wed, Sep 01, 2004 at 01:02:24AM +0200, Christer Weinigel wrote: > > I can see the argument for having the equivalent of Content-type or > > Content-transfer-encoding as a named stream though. > > Why having them as named streams if we can get them as xattrs for no > additional pain? (Since fileutils would have to be changed anyway, we > can even make cp copy and emacs update xattrs.) Because some of as requested to be able to open tar,iso, and few other formats this way. So I can simply access it with cat (as a dir/file). -- GJ ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-05 11:17 ` Tonnerre 2004-09-05 11:40 ` Grzegorz Jaśkiewicz @ 2004-09-05 11:57 ` Spam 2004-09-05 11:58 ` Tonnerre 1 sibling, 1 reply; 1392+ messages in thread From: Spam @ 2004-09-05 11:57 UTC (permalink / raw) To: Tonnerre Cc: Christer Weinigel, Linus Torvalds, Horst von Brand, Pavel Machek, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > Salut, > On Wed, Sep 01, 2004 at 01:02:24AM +0200, Christer Weinigel wrote: >> I can see the argument for having the equivalent of Content-type or >> Content-transfer-encoding as a named stream though. > Why having them as named streams if we can get them as xattrs for no > additional pain? (Since fileutils would have to be changed anyway, we > can even make cp copy and emacs update xattrs.) What if I do not use emacs, but vim, mcedit, gedit, or some other editor? It doesn't seem logical to have to patch every application that uses files. ~S > Tonnerre ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-05 11:57 ` Spam @ 2004-09-05 11:58 ` Tonnerre 2004-09-05 12:23 ` Spam 2004-09-05 12:30 ` Spam 0 siblings, 2 replies; 1392+ messages in thread From: Tonnerre @ 2004-09-05 11:58 UTC (permalink / raw) To: Spam Cc: Christer Weinigel, Linus Torvalds, Horst von Brand, Pavel Machek, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 483 bytes --] Salut, On Sun, Sep 05, 2004 at 01:57:49PM +0200, Spam wrote: > What if I do not use emacs, but vim, mcedit, gedit, or some other > editor? It doesn't seem logical to have to patch every application > that uses files. We would have to do that in either case, so let's patch them to do it in a nonintrusive way. And as to reading and writing inside tar files, write and/or use a really nice userspace library to do it. (As does MacOS/X, as does KDE, etc.) Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-05 11:58 ` Tonnerre @ 2004-09-05 12:23 ` Spam 2004-09-05 12:30 ` Spam 1 sibling, 0 replies; 1392+ messages in thread From: Spam @ 2004-09-05 12:23 UTC (permalink / raw) To: Tonnerre Cc: Christer Weinigel, Linus Torvalds, Horst von Brand, Pavel Machek, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > Salut, > On Sun, Sep 05, 2004 at 01:57:49PM +0200, Spam wrote: >> What if I do not use emacs, but vim, mcedit, gedit, or some other >> editor? It doesn't seem logical to have to patch every application >> that uses files. > We would have to do that in either case, so let's patch them to do it > in a nonintrusive way. And as to reading and writing inside tar files, > write and/or use a really nice userspace library to do it. (As does > MacOS/X, as does KDE, etc.) No, not with the current setup with file-as-dir. It also works in Gnome/Natilus. Just tested: mkdir test chmod +x test cd test dir metas echo 0777>rwx In Nautilus I just typed in in the address bar /test/metas/ I could open meta-data and change with GEdit too. I do not see that I have had to patch any programs to get this to work. ~S > Tonnerre ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-05 11:58 ` Tonnerre 2004-09-05 12:23 ` Spam @ 2004-09-05 12:30 ` Spam 2004-09-06 10:50 ` Pavel Machek 2004-09-06 18:14 ` Horst von Brand 1 sibling, 2 replies; 1392+ messages in thread From: Spam @ 2004-09-05 12:30 UTC (permalink / raw) To: Tonnerre Cc: Christer Weinigel, Linus Torvalds, Horst von Brand, Pavel Machek, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > Salut, > On Sun, Sep 05, 2004 at 01:57:49PM +0200, Spam wrote: >> What if I do not use emacs, but vim, mcedit, gedit, or some other >> editor? It doesn't seem logical to have to patch every application >> that uses files. > We would have to do that in either case, so let's patch them to do it > in a nonintrusive way. And as to reading and writing inside tar files, > write and/or use a really nice userspace library to do it. (As does > MacOS/X, as does KDE, etc.) The problem with the userspace library is standardization. What would be needed is a userspace library that has a extensible plugin interface that is standardized. Otherwise we would need lots of different libraries, and I seriously doubt that 1) this will happen and 2) we get all Linux programs to be patched to use it. ~S > Tonnerre ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-05 12:30 ` Spam @ 2004-09-06 10:50 ` Pavel Machek 2004-09-06 12:32 ` Spam 2004-09-06 18:14 ` Horst von Brand 1 sibling, 1 reply; 1392+ messages in thread From: Pavel Machek @ 2004-09-06 10:50 UTC (permalink / raw) To: Spam Cc: Tonnerre, Christer Weinigel, Linus Torvalds, Horst von Brand, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hi! > >> What if I do not use emacs, but vim, mcedit, gedit, or some other > >> editor? It doesn't seem logical to have to patch every application > >> that uses files. > > > We would have to do that in either case, so let's patch them to do it > > in a nonintrusive way. And as to reading and writing inside tar files, > > write and/or use a really nice userspace library to do it. (As does > > MacOS/X, as does KDE, etc.) > > The problem with the userspace library is standardization. What > would be needed is a userspace library that has a extensible plugin > interface that is standardized. Otherwise we would need lots of > different libraries, and I seriously doubt that 1) this will happen > and 2) we get all Linux programs to be patched to use it. libvfs from midnight commander (and anything build on the top of it) already is very extensible. Pavel ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-06 10:50 ` Pavel Machek @ 2004-09-06 12:32 ` Spam 2004-09-06 14:52 ` Christer Weinigel 0 siblings, 1 reply; 1392+ messages in thread From: Spam @ 2004-09-06 12:32 UTC (permalink / raw) To: Pavel Machek Cc: Tonnerre, Christer Weinigel, Linus Torvalds, Horst von Brand, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > Hi! >> >> What if I do not use emacs, but vim, mcedit, gedit, or some other >> >> editor? It doesn't seem logical to have to patch every application >> >> that uses files. >> >> > We would have to do that in either case, so let's patch them to do it >> > in a nonintrusive way. And as to reading and writing inside tar files, >> > write and/or use a really nice userspace library to do it. (As does >> > MacOS/X, as does KDE, etc.) >> >> The problem with the userspace library is standardization. What >> would be needed is a userspace library that has a extensible plugin >> interface that is standardized. Otherwise we would need lots of >> different libraries, and I seriously doubt that 1) this will happen >> and 2) we get all Linux programs to be patched to use it. > libvfs from midnight commander (and anything build on the top of it) > already is very extensible. This wasn't my point. My point was about all applications using it. Most aren't. > Pavel ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-06 12:32 ` Spam @ 2004-09-06 14:52 ` Christer Weinigel 2004-09-06 15:13 ` Spam 0 siblings, 1 reply; 1392+ messages in thread From: Christer Weinigel @ 2004-09-06 14:52 UTC (permalink / raw) To: Spam Cc: Pavel Machek, Tonnerre, Christer Weinigel, Linus Torvalds, Horst von Brand, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Spam <spam@tnonline.net> writes: > >> The problem with the userspace library is standardization. What > >> would be needed is a userspace library that has a extensible plugin > >> interface that is standardized. Otherwise we would need lots of > >> different libraries, and I seriously doubt that 1) this will happen > >> and 2) we get all Linux programs to be patched to use it. > > > libvfs from midnight commander (and anything build on the top of it) > > already is very extensible. > > This wasn't my point. My point was about all applications using it. > Most aren't. So why do you think that just because you put an interface into the everyone will automatically start using it and use it without any conflicts (i.e. is foo/icon a windows icon, a gif, png, or jpeg file, what size is it, 16x16, 48x48...). When you add a new shared namespace, userspace must be taught about it anyways. So start with a userspace library, convince people to use it, and when you know what people want, _then_ put it into the kernel to increase performance or security. Don't start with "wouldn't it be nice if" without any thought behind it, it will just mean that we get another useless and misdesigned interface in the kernel that we have to live with for years. Do you know how long it took to get rid of /dev/cua0 and the locking mess associated with it? (Deity! I just checked on my FC2 system, it's still there. I thought it would be dead by now.) /Christer -- "Just how much can I get away with and still go to heaven?" Freelance consultant specializing in device driver programming for Linux Christer Weinigel <christer@weinigel.se> http://www.weinigel.se ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-06 14:52 ` Christer Weinigel @ 2004-09-06 15:13 ` Spam 2004-09-06 15:44 ` Christer Weinigel 0 siblings, 1 reply; 1392+ messages in thread From: Spam @ 2004-09-06 15:13 UTC (permalink / raw) To: Christer Weinigel Cc: Pavel Machek, Tonnerre, Linus Torvalds, Horst von Brand, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > Spam <spam@tnonline.net> writes: >> >> The problem with the userspace library is standardization. What >> >> would be needed is a userspace library that has a extensible plugin >> >> interface that is standardized. Otherwise we would need lots of >> >> different libraries, and I seriously doubt that 1) this will happen >> >> and 2) we get all Linux programs to be patched to use it. >> >> > libvfs from midnight commander (and anything build on the top of it) >> > already is very extensible. >> >> This wasn't my point. My point was about all applications using it. >> Most aren't. > So why do you think that just because you put an interface into the > everyone will automatically start using it and use it without any > conflicts (i.e. is foo/icon a windows icon, a gif, png, or jpeg file, > what size is it, 16x16, 48x48...). > When you add a new shared namespace, userspace must be taught about it > anyways. So start with a userspace library, convince people to use > it, and when you know what people want, _then_ put it into the kernel > to increase performance or security. Several aspects of this; 1) Programs will need to actually be coded against this shared library which probably will be more advanced that just usning normal file access code. 2) Then we redo it all again to fit in the kernel, as a kernel plugin or user-level plugin. > Don't start with "wouldn't it be > nice if" without any thought behind it, it will just mean that we get > another useless and misdesigned interface in the kernel that we have > to live with for years. I am not saying that there should not be any thought behind implementing something. But from many hear I just hear a big no, without any thought about it either. Named streams and meta files are just one example of a general access to extra data that belongs to files. It seem to me as a much simpler way for program to use those than to link in several various libraries to do the same thing. Plugins were just another thing that could extend the functionality of these streams and meta data. Reiser4 has a plugin architecture, although not yet any run-time support for loading them. Is this so bad that we have to prevent it? No one has to use them. maybe at some point it would be possible to specify which classes of plugins that could be loaded by a user and his files. What are the complaints about. The technical challenge to make things like this secure, or the concept to have this functionality at all. I do not think that because something is hard to do, it should be dismissed. ~S > Do you know how long it took to get rid of /dev/cua0 and the locking > mess associated with it? (Deity! I just checked on my FC2 system, > it's still there. I thought it would be dead by now.) > /Christer ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-06 15:13 ` Spam @ 2004-09-06 15:44 ` Christer Weinigel 2004-09-06 15:54 ` Pavel Machek ` (2 more replies) 0 siblings, 3 replies; 1392+ messages in thread From: Christer Weinigel @ 2004-09-06 15:44 UTC (permalink / raw) To: Spam Cc: Christer Weinigel, Pavel Machek, Tonnerre, Linus Torvalds, Horst von Brand, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Spam <spam@tnonline.net> writes: > > When you add a new shared namespace, userspace must be taught about it > > anyways. So start with a userspace library, convince people to use > > it, and when you know what people want, _then_ put it into the kernel > > to increase performance or security. > > Several aspects of this; > > 1) Programs will need to actually be coded against this shared > library which probably will be more advanced that just usning normal > file access code. Not at all. As many others have already shown, trying to use files as directories won't work since then you can't have named streams or attributes for real directories. What does foo/icon refer to when foo is a directory? There must be some way of differentiating between them, which in turn means that we need a new way of referring to the stream. So the simple, hackish, solution really won't fly. Besides, it will probably break a lot of security critical assumptions in todays software. What if you have a filter in your web server that excludes executable files or files that end with .php, can you imagine the mess if you could just open "foo.php/main-stream" instead? > 2) Then we redo it all again to fit in the kernel, as a kernel > plugin or user-level plugin. No, we use the same user space library, but let the user space library use helper functions in the kernel. We could for example add a function that opens a named stream based on a filename which uses a .xattr directory in the same directory as the file, or in database in our home directory. If we add a solaris-like openat(3) syscall we modify the library to take advantage of it. > Plugins were just another thing that could extend the functionality > of these streams and meta data. Reiser4 has a plugin architecture, > although not yet any run-time support for loading them. Is this so > bad that we have to prevent it? Take an example: "expose a tar-file as streams below the file" which as been suggested here is IMNSHO plain silly. I'm not saying anything about mounting a tar file via userfs somewhere else in the file system, that is quite ok, but trying to mount it on top of the same file which suddenly and automagically turns into a sort-of-directory breaks too many thing. Let your file manager do the choice instead, based on the users preferences. For example, with a tar.gz-file, I'd like to have a choice of "open file as a seekable file" which would do: mount -t userfs -o driver=gunzip foo.tar.gz /tmp/xyzzy Then I can work with /tmp/xyzzy as a normal file (maybe even with write access if the userfs driver can handle this). Another choice in the same file manager would be "open file as a directory" which would mount the same file in _another place_ as a directory, and I can even have different views of the same file mounted at the same time. With the named streams junk that have been suggested here, having two different views would be impossible. Sure, we could say that we add another level of indirection to the named streams, so that we specify the driver as the first component of te magic file-as-directory, i.e. foo.tar.gz/ungzipped would refer to the ungzipped stream and foo.tar.gz/ungzipped-and-untarred would show the tar file as a directory, but really, this isn't any more useful than doing a userfs mount. The userfs mount does not break existing semantics (anymore than mount -o loop does today), and it will work with the existing infrastructure in the linux kernel. The only advantage of files-as-directories with magic plugins in the kernel is that one can look at it and say "look, how neat, the filenames look almost the same". /Christer -- "Just how much can I get away with and still go to heaven?" Freelance consultant specializing in device driver programming for Linux Christer Weinigel <christer@weinigel.se> http://www.weinigel.se ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-06 15:44 ` Christer Weinigel @ 2004-09-06 15:54 ` Pavel Machek 2004-09-06 16:07 ` Christer Weinigel 2004-09-08 6:11 ` Tonnerre 2004-09-06 16:14 ` Spam 2004-09-06 19:14 ` David Masover 2 siblings, 2 replies; 1392+ messages in thread From: Pavel Machek @ 2004-09-06 15:54 UTC (permalink / raw) To: Christer Weinigel Cc: Spam, Tonnerre, Linus Torvalds, Horst von Brand, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hi! > Sure, we could say that we add another level of indirection to the > named streams, so that we specify the driver as the first component of > te magic file-as-directory, i.e. foo.tar.gz/ungzipped would refer to > the ungzipped stream and foo.tar.gz/ungzipped-and-untarred would show > the tar file as a directory, but really, this isn't any more useful > than doing a userfs mount. The userfs mount does not break existing > semantics (anymore than mount -o loop does today), and it will work > with the existing infrastructure in the linux kernel. The only > advantage of files-as-directories with magic plugins in the kernel is > that one can look at it and say "look, how neat, the filenames look > almost the same". Who is going to umount it when application crashes, etc? Plus mount required root priviledges last time I checked. Pavel ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-06 15:54 ` Pavel Machek @ 2004-09-06 16:07 ` Christer Weinigel 2004-09-06 16:56 ` Pavel Machek 2004-09-08 6:11 ` Tonnerre 1 sibling, 1 reply; 1392+ messages in thread From: Christer Weinigel @ 2004-09-06 16:07 UTC (permalink / raw) To: Pavel Machek Cc: Christer Weinigel, Spam, Tonnerre, Linus Torvalds, Horst von Brand, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Pavel Machek <pavel@ucw.cz> writes: > Who is going to umount it when application crashes, etc? Who removes temporary files left behind when an application crashes? I'd guess a daemon such as autofs could do a very good job here. >Plus mount required root priviledges last time I checked. bash$ ls -l /bin/mount -rwsr-xr-x 1 root root 78504 May 4 23:34 /bin/mount with proper policies in userspace it allows users to perform mounts. I'm not suggesting that the kernel should be unchanged, but really, some of the proposals here, to put a hell of a lot of complexity into the kernel it just wet dreams with not much thought of how it affects the kernel. What happened to the philosophy of putting complexity and policy in userspace? Look at khttpd and tux, they were hacks in the kernel to try things out. But what ended up in the kernel is generic infrastructure that is useful for a lot more applications than just a web server. That is the right way to do things. /Christer -- "Just how much can I get away with and still go to heaven?" Freelance consultant specializing in device driver programming for Linux Christer Weinigel <christer@weinigel.se> http://www.weinigel.se ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-06 16:07 ` Christer Weinigel @ 2004-09-06 16:56 ` Pavel Machek 0 siblings, 0 replies; 1392+ messages in thread From: Pavel Machek @ 2004-09-06 16:56 UTC (permalink / raw) To: Christer Weinigel Cc: Spam, Tonnerre, Linus Torvalds, Horst von Brand, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hi! > >Plus mount required root priviledges last time I checked. > > bash$ ls -l /bin/mount > -rwsr-xr-x 1 root root 78504 May 4 23:34 /bin/mount > > with proper policies in userspace it allows users to perform mounts. You want to do exec() each time you enter a archive? I do not think that is good idea. > I'm not suggesting that the kernel should be unchanged, but really, > some of the proposals here, to put a hell of a lot of complexity into > the kernel it just wet dreams with not much thought of how it affects > the kernel. What happened to the philosophy of putting complexity and > policy in userspace? Look at khttpd and tux, they were hacks in the > kernel to try things out. But what ended up in the kernel is generic > infrastructure that is useful for a lot more applications than just a > web server. That is the right way to do things. Well, generic infrastructure for uservfs would be enough for me... And it is few lines in kernel. Pavel ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-06 15:54 ` Pavel Machek 2004-09-06 16:07 ` Christer Weinigel @ 2004-09-08 6:11 ` Tonnerre 2004-09-08 12:57 ` Alan Cox 1 sibling, 1 reply; 1392+ messages in thread From: Tonnerre @ 2004-09-08 6:11 UTC (permalink / raw) To: Pavel Machek Cc: Christer Weinigel, Spam, Linus Torvalds, Horst von Brand, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 617 bytes --] Salut, On Mon, Sep 06, 2004 at 05:54:56PM +0200, Pavel Machek wrote: > Who is going to umount it when application crashes, etc? This has been discussed along with the HAL people a while ago. Actually, file systems can introduce a refcount, where we need a decrement function which automatically unmounts the filesystem if we decrement the use count to zero. Kind of an automatic umount. How do you tell which file systems shall be autoumounted? HAL suggestion: introduce a no-op mount option. For some classes of fs'es we might as well make that obligatory, such as tar. Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-08 6:11 ` Tonnerre @ 2004-09-08 12:57 ` Alan Cox 0 siblings, 0 replies; 1392+ messages in thread From: Alan Cox @ 2004-09-08 12:57 UTC (permalink / raw) To: Tonnerre Cc: Pavel Machek, Christer Weinigel, Spam, Linus Torvalds, Horst von Brand, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Mer, 2004-09-08 at 07:11, Tonnerre wrote: > This has been discussed along with the HAL people a while > ago. Actually, file systems can introduce a refcount, where we need a > decrement function which automatically unmounts the filesystem if we > decrement the use count to zero. Kind of an automatic umount. We've supported file system garbage collection when they become unused since umm about 2.4.10 I think. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-06 15:44 ` Christer Weinigel 2004-09-06 15:54 ` Pavel Machek @ 2004-09-06 16:14 ` Spam 2004-09-06 19:14 ` David Masover 2 siblings, 0 replies; 1392+ messages in thread From: Spam @ 2004-09-06 16:14 UTC (permalink / raw) To: Christer Weinigel Cc: Pavel Machek, Tonnerre, Linus Torvalds, Horst von Brand, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List >> Plugins were just another thing that could extend the functionality >> of these streams and meta data. Reiser4 has a plugin architecture, >> although not yet any run-time support for loading them. Is this so >> bad that we have to prevent it? > Take an example: "expose a tar-file as streams below the file" which > as been suggested here is IMNSHO plain silly. I'm not saying anything > about mounting a tar file via userfs somewhere else in the file > system, that is quite ok, but trying to mount it on top of the same > file which suddenly and automagically turns into a sort-of-directory > breaks too many thing. Let your file manager do the choice instead, > based on the users preferences. For example, with a tar.gz-file, I'd > like to have a choice of "open file as a seekable file" which would > do: > mount -t userfs -o driver=gunzip foo.tar.gz /tmp/xyzzy Where is the difference? Both would be handled by a specific driver or module and export the same semantics (files+dirs) with permissions etc to the user. With your idea you still would need the userfs kernel module, and with "magic plugins", as you said, you will need a vfs/reiser4 kernel plugin. > Then I can work with /tmp/xyzzy as a normal file (maybe even with > write access if the userfs driver can handle this). Another choice in > the same file manager would be "open file as a directory" which would > mount the same file in _another place_ as a directory, and I can even > have different views of the same file mounted at the same time. With > the named streams junk that have been suggested here, having two > different views would be impossible. > Sure, we could say that we add another level of indirection to the > named streams, so that we specify the driver as the first component of > te magic file-as-directory, i.e. foo.tar.gz/ungzipped would refer to > the ungzipped stream and foo.tar.gz/ungzipped-and-untarred would show > the tar file as a directory, but really, this isn't any more useful > than doing a userfs mount. The userfs mount does not break existing > semantics (anymore than mount -o loop does today), and it will work > with the existing infrastructure in the linux kernel. The only > advantage of files-as-directories with magic plugins in the kernel is > that one can look at it and say "look, how neat, the filenames look > almost the same". No there are _usability_ differences. I cirtanly do not want to mount lots of files with mount -t userfs, just to see extra meta-data that I want to quickly be able to use. And it also wouldn't work generically (searchable) with tools like find, grep, etc either. ~S > /Christer ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-06 15:44 ` Christer Weinigel 2004-09-06 15:54 ` Pavel Machek 2004-09-06 16:14 ` Spam @ 2004-09-06 19:14 ` David Masover 2 siblings, 0 replies; 1392+ messages in thread From: David Masover @ 2004-09-06 19:14 UTC (permalink / raw) To: Christer Weinigel Cc: Spam, Pavel Machek, Tonnerre, Linus Torvalds, Horst von Brand, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Christer Weinigel wrote: | Spam <spam@tnonline.net> writes: | | |>>When you add a new shared namespace, userspace must be taught about it |>>anyways. So start with a userspace library, convince people to use |>>it, and when you know what people want, _then_ put it into the kernel |>>to increase performance or security. |> |> Several aspects of this; |> |> 1) Programs will need to actually be coded against this shared |> library which probably will be more advanced that just usning normal |> file access code. | | | Not at all. As many others have already shown, trying to use files as | directories won't work since then you can't have named streams or | attributes for real directories. What does foo/icon refer to when foo Oh yes, you can. That's what "metas" is for. I wanted to call it ... | is a directory? There must be some way of differentiating between When foo is a directory, foo/icon is a normal file. But foo/metas/icon (or foo/.../icon if you like) is the metadata. This already works. I can use it right now for file permissions: echo 755 > /bin/bash/metas/rwx echo 755 > ./metas/rwx | Besides, it will probably break a lot of security critical assumptions | in todays software. What if you have a filter in your web server that | excludes executable files or files that end with .php, can you imagine | the mess if you could just open "foo.php/main-stream" instead? Why would you have a foo.php/main-stream? And what would be so hard about allowing read access to foo, but not access to foo's metas? This is essentially the same as having mode 6 on a directory. [...] |> Plugins were just another thing that could extend the functionality |> of these streams and meta data. Reiser4 has a plugin architecture, |> although not yet any run-time support for loading them. Is this so |> bad that we have to prevent it? | | | Take an example: "expose a tar-file as streams below the file" which | as been suggested here is IMNSHO plain silly. I'm not saying anything | about mounting a tar file via userfs somewhere else in the file | system, that is quite ok, but trying to mount it on top of the same | file which suddenly and automagically turns into a sort-of-directory | breaks too many thing. Let your file manager do the choice instead, What things? It's just dangerously different thinking. I don't see it breaking anything on my system right now. If anything, the nvidia binary drivers are a bigger issue... | based on the users preferences. For example, with a tar.gz-file, I'd | like to have a choice of "open file as a seekable file" which would | do: | | mount -t userfs -o driver=gunzip foo.tar.gz /tmp/xyzzy | | Then I can work with /tmp/xyzzy as a normal file (maybe even with | write access if the userfs driver can handle this). Another choice in | the same file manager would be "open file as a directory" which would | mount the same file in _another place_ as a directory, and I can even | have different views of the same file mounted at the same time. With | the named streams junk that have been suggested here, having two | different views would be impossible. | | Sure, we could say that we add another level of indirection to the | named streams, so that we specify the driver as the first component of | te magic file-as-directory, i.e. foo.tar.gz/ungzipped would refer to | the ungzipped stream and foo.tar.gz/ungzipped-and-untarred would show | the tar file as a directory, but really, this isn't any more useful | than doing a userfs mount. The userfs mount does not break existing Yes, it is more useful than doing a userfs mount. The userfs mount has to be done manually or by a file manager. I do not use a file manager. ~ I am not alone here. Unless you're going to have the entire FS mounted userfs and have that be the file manager, which wouldn't be nearly as fast as plugins. I've heard autofs mentioned -- which only works on specific files which it's been preconfigured for. Not good here. | semantics (anymore than mount -o loop does today), and it will work | with the existing infrastructure in the linux kernel. The only That's true -- it's easier to implement if you ignore the number of apps which need patching. There's also the fact that most people here won't be patching apps, they'll just be breathing a sigh of relief that they didn't have to patch the kernel. Just because someone else has to do it doesn't make it any less work. | advantage of files-as-directories with magic plugins in the kernel is | that one can look at it and say "look, how neat, the filenames look | almost the same". And this is a usability issue. Not all newbies use Nautilus. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTy3MngHNmZLgCUhAQJ/lQ//UmQOJLZAmsFFD8TJehD8WitMrImrcrBC SJ/mCkIaPq5JNo9R/ovd1UoGlf47JUvaaT5G1d/yOMA+/0kPNqjJL49Km9A3Ze+C 00Uq12yP8yp6NjBzq9HKAjvyHVghl1A5ZllIdiwuc8Fywbl6UKUHlN1D4PpUp1uA znKPjmGpBOsTzgNWJYbAfjns60C7DHZoD/S+ldt3af2PsJKOpx0v4fusV8Y2+XRG kWybZLl1YPwPLfJLCaa3KpbY1KkSuk6TSeut90+zRV6s1WJ8LE98NpDBGUDt/+cm AXcp/zyrUmbpvQtZBaUFYwZ8imizspiSHYnBwdhVm9tfERFOFI9diBidvanHk/s0 D+FSJhnQSeo/hMnbHlfv4E6p4k1jDvJk06XJ/U45sEym4hCp9roDPB0tUic7bOnE mxNxzogp92KHfsVPuqkidq255ztAjRbzdnt8zsBOPImqTgfsO3xVfT3svIixNxWv +qaAlrR7dAri9bi3wNLCjzRhp3id0fspetLJrrCHI/8hMlu7a6nsIHLocuvG069W OzdXtTPRYsV8YlElsdL5d/3grZSL3CWNpQxoRb9gzj0hKHHPjrXw9ACcQthiknNc DkjjcLxV2gPqhKq3WgMTKXHhLDWEfJpFVQ8PccJHsKaaUgGtDUg0ETBT54h8Mo8E I0bkcYOLtHk= =o3vQ -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-05 12:30 ` Spam 2004-09-06 10:50 ` Pavel Machek @ 2004-09-06 18:14 ` Horst von Brand 2004-09-06 18:53 ` David Masover 1 sibling, 1 reply; 1392+ messages in thread From: Horst von Brand @ 2004-09-06 18:14 UTC (permalink / raw) To: Spam Cc: Tonnerre, Christer Weinigel, Linus Torvalds, Horst von Brand, Pavel Machek, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Spam <spam@tnonline.net> said: [...] > The problem with the userspace library is standardization. What > would be needed is a userspace library that has a extensible plugin > interface that is standardized. Otherwise we would need lots of > different libraries, and I seriously doubt that 1) this will happen > and 2) we get all Linux programs to be patched to use it. What is the difference with a kernel implementation? Not by being in-kernel will it make all the incompatible ways of doing this magically vanish, and give outstanding performance. Plus handling and maintaining the in-kernel stuff is _much_ harder than userspace libraries. I'd go the other way around: Get userspace to agree on a common framework, make it work in userspace; if (extensive, hopefully) experience shows that a pure userspace solution has issues that can't be solved except by kernel assistance, so be it. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-06 18:14 ` Horst von Brand @ 2004-09-06 18:53 ` David Masover 2004-09-06 22:37 ` Christer Weinigel ` (2 more replies) 0 siblings, 3 replies; 1392+ messages in thread From: David Masover @ 2004-09-06 18:53 UTC (permalink / raw) To: Horst von Brand Cc: Spam, Tonnerre, Christer Weinigel, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Horst von Brand wrote: | Spam <spam@tnonline.net> said: | | [...] | | |> The problem with the userspace library is standardization. What |> would be needed is a userspace library that has a extensible plugin |> interface that is standardized. Otherwise we would need lots of |> different libraries, and I seriously doubt that 1) this will happen |> and 2) we get all Linux programs to be patched to use it. | | | What is the difference with a kernel implementation? Not by being in-kernel | will it make all the incompatible ways of doing this magically vanish, and | give outstanding performance. Plus handling and maintaining the in-kernel | stuff is _much_ harder than userspace libraries. First of all, only the interface has to be in the kernel. I haven't heard anyone suggest otherwise. Second, there are quite a few things which I might want to do, which can be done with this interface and without patching programs, but would require massive patches to userspace. There have been numerous examples. There are some things which can't be solved without patching. Version control is one such thing. But then there can be more generic patches - -- as soon as the transaction API is done, you only have to patch apps to use that, and have a version control reiser4 plugin. | I'd go the other way around: Get userspace to agree on a common framework, | make it work in userspace; if (extensive, hopefully) experience shows that | a pure userspace solution has issues that can't be solved except by kernel | assistance, so be it. We already have such a framework -- it's called "VFS". -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTyyGXgHNmZLgCUhAQLUeQ/8D3tWL9l/zQeGylpVbOe6SkcSPOOmlIFR tcjh0y1Y4ET17ATFKbKbzQYYDgd49AqU/gZnro27jYun3Yi6U0fWGGQFfi1A9O5E gSCmsjSWjfDfx4gu3EU1x0Bhkd6Mo8GCrC7L5gj5C+L4c5ZAnffeGloF8nM4hCex Wsb0PgOSxXuoQcd2EELVEXYdq0RCnxrmuszl1B2SE6w1ImONWMoXJ9fDGDf0aIUu rwrrZnlH4zp0bQ0dXDGXqUYYT5h5DAhbh96IWLrPbWMB0vLBqIP+95P2/vTHb7EL RwVKBV1UuuZ2ANPbImoIuxHWF+PCx/HwFs/mUolw0D2Yn3u2HgmPVFemyPnlCfeX yGPhJgnieRuGntgUZcfbqk7ZO3y0y5eRDq6N4eMHMlWYV9LC5kyP7OxcQ8SAF8P6 Sk4iylYN1AMiy5Bp9odScauST0NT9CLmi1Ps0DYwgVN1H+ldS1l+4ITokb1Ex1+4 ZLq1HhPNaYYWoA4VPuwxl0XrB4wGrMbOt1w4+TNM3AG9MvzqTGgSrh2rXfXkPHGZ 7LNHuinRyJt3dcF0vPS4WHG6FtVsO8XVPaY55tYQIYZEBtZl3mattBb9gM3WDJmw M/pxbAQTDZHloR9/7TGEF8gD3AjPBexTfvojqVHK2VvRu4/2Ku17wvK82v68LuyU bFxxxgj9IgY= =BxAT -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-06 18:53 ` David Masover @ 2004-09-06 22:37 ` Christer Weinigel 2004-09-07 2:06 ` Horst von Brand 2004-09-07 6:32 ` Hans Reiser 2 siblings, 0 replies; 1392+ messages in thread From: Christer Weinigel @ 2004-09-06 22:37 UTC (permalink / raw) To: David Masover Cc: Horst von Brand, Spam, Tonnerre, Christer Weinigel, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List David Masover <ninja@slaphack.com> writes: > Second, there are quite a few things which I might want to do, which can > be done with this interface and without patching programs, but would > require massive patches to userspace. There have been numerous > examples. Please give one example? There is a lot of handwaving here on all sides. > | I'd go the other way around: Get userspace to agree on a common framework, > | make it work in userspace; if (extensive, hopefully) experience shows that > | a pure userspace solution has issues that can't be solved except by kernel > | assistance, so be it. > > We already have such a framework -- it's called "VFS". That you'd like to break the semantics of. Great. /Christer -- "Just how much can I get away with and still go to heaven?" Freelance consultant specializing in device driver programming for Linux Christer Weinigel <christer@weinigel.se> http://www.weinigel.se ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-06 18:53 ` David Masover 2004-09-06 22:37 ` Christer Weinigel @ 2004-09-07 2:06 ` Horst von Brand 2004-09-07 5:50 ` David Masover 2004-09-07 6:32 ` Hans Reiser 2 siblings, 1 reply; 1392+ messages in thread From: Horst von Brand @ 2004-09-07 2:06 UTC (permalink / raw) To: David Masover Cc: Spam, Tonnerre, Christer Weinigel, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List David Masover <ninja@slaphack.com> said: > Horst von Brand wrote: > | Spam <spam@tnonline.net> said: > | > | [...] > | > | > |> The problem with the userspace library is standardization. What > |> would be needed is a userspace library that has a extensible plugin > |> interface that is standardized. Otherwise we would need lots of > |> different libraries, and I seriously doubt that 1) this will happen > |> and 2) we get all Linux programs to be patched to use it. > | > | > | What is the difference with a kernel implementation? Not by being > | in-kernel will it make all the incompatible ways of doing this > | magically vanish, and give outstanding performance. Plus handling > | and maintaining the in-kernel stuff is _much_ harder than userspace > | libraries. > First of all, only the interface has to be in the kernel. I haven't > heard anyone suggest otherwise. Right. But it is _another_ interface in the kernel, plus special userland code supporting it. > Second, there are quite a few things which I might want to do, which can > be done with this interface and without patching programs, Such as? > but would > require massive patches to userspace. There have been numerous examples. Haven't seen any that made sense to me, sorry. > There are some things which can't be solved without patching. Maybe. Question is, is it worth it (kernel modifications + userland support, or just userland support, or leave it alone). Sure, it might make your particular application easier to write (at a cost for _all_ filesystem hackers!), perhaps even a bit faster; but is _your_ particular convenience worth the cost for _everybody_? > Version > control is one such thing. bk, cvs, svn, rcs, ... are working just fine here, thank you so much. Used to work on SunOS and Solaris, even SCO Unix (I used at least rcs and cvs there). No Reiser4 in sight. > But then there can be more generic patches > - -- as soon as the transaction API is done, you only have to patch apps > to use that, and have a version control reiser4 plugin. Again, _what_ version control exactly? Will the above packages be able to make use of it (remember they all are cross-platform (at least cross-Unix), and so quite unlikely to make use of a Reiser4 on Linux whackiness...)? > | I'd go the other way around: Get userspace to agree on a common framework, > | make it work in userspace; if (extensive, hopefully) experience shows that > | a pure userspace solution has issues that can't be solved except by kernel > | assistance, so be it. > We already have such a framework -- it's called "VFS". Right. It offers what applications need to build their own stuff. It is minimalistic (well, sort of) and time-proven. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 2:06 ` Horst von Brand @ 2004-09-07 5:50 ` David Masover 2004-09-07 6:02 ` Hans Reiser ` (2 more replies) 0 siblings, 3 replies; 1392+ messages in thread From: David Masover @ 2004-09-07 5:50 UTC (permalink / raw) To: Horst von Brand Cc: Spam, Tonnerre, Christer Weinigel, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Horst von Brand wrote: [...] |>Second, there are quite a few things which I might want to do, which can |>be done with this interface and without patching programs, | | | Such as? They've been mentioned. | | |> but would |>require massive patches to userspace. There have been numerous examples. | | | Haven't seen any that made sense to me, sorry. Sorry if they don't make sense to you, but I don't feel like discussing them now. Either you get it or you don't, either you agree or you don't. Read the archives. | | |>There are some things which can't be solved without patching. | | | Maybe. Question is, is it worth it (kernel modifications + userland | support, or just userland support, or leave it alone). Sure, it might make | your particular application easier to write (at a cost for _all_ filesystem | hackers!), perhaps even a bit faster; but is _your_ particular convenience | worth the cost for _everybody_? There are far more userland developers than filesystem hackers. If you're a filesystem hacker, it's easier to understand how much work it is to do something in the filesystem/kernel, and harder to understand how it works in userland, or for the actual user. | | |> Version |>control is one such thing. | | | bk, cvs, svn, rcs, ... are working just fine here, thank you so much. Used | to work on SunOS and Solaris, even SCO Unix (I used at least rcs and cvs | there). No Reiser4 in sight. Transparently? It _works_ to do zcat file.gz > /tmp/file vim /tmp/file gzip -c /tmp/file > file.gz rm /tmp/file You can even do that as a script, call it zvim. You could even do it as a generic script, where "vim" is replaced with "$1". But is it as elegent as transparently compressed files? I'm not sure about the version control thing -- I don't think people have hit every conceptual issue about it yet. But the point is: Moving complexity from kernel space to user space (or the other way) doesn't make it any less complex. |> But then there can be more generic patches |>- -- as soon as the transaction API is done, you only have to patch apps |>to use that, and have a version control reiser4 plugin. | | | Again, _what_ version control exactly? Will the above packages be able to | make use of it (remember they all are cross-platform (at least cross-Unix), | and so quite unlikely to make use of a Reiser4 on Linux whackiness...)? Probably. It wasn't specified what version control would be used -- that's currently just as abstract as what compression algorithm to use to transparently compress files. Maybe something new. Version control just doesn't look that complex once you've got the interface and the data storage done. I do incremental backups using rsync and hardlinks -- why would version control be so much more complex than that? Said backup system uses hardlinks, btw, which not all systems support. And rsync is fairly broken on some platforms. |>| I'd go the other way around: Get userspace to agree on a common framework, |>| make it work in userspace; if (extensive, hopefully) experience shows that |>| a pure userspace solution has issues that can't be solved except by kernel |>| assistance, so be it. | | |>We already have such a framework -- it's called "VFS". | | | Right. It offers what applications need to build their own stuff. It is | minimalistic (well, sort of) and time-proven. x86 assemby is minimalistic and time-proven. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQT1MGHgHNmZLgCUhAQLxoQ/+Mglz6tr0A4wQmAnP1y9q7WVqQQ8s3Cwg b3VyOHaIIZSw3lod+bWDgE9LJJsQOxO4SThlWLyMEHipFJT0iHVIYjLSrC5e4DOb Wl5E51yz3ZVx11mRTM7uEw+Ez6wOcaWIFLTvarDxzmDaxS6oh2ZpN2Ibnf8h/1lT 0YJnz6C964TkaA8jYIsQljIWkMMk7EGzP6UlrQOBbo4xMJNT+wIZMJWX5JAsULc2 EGzTO8dgZB0MbJ5STufS4h5tudny/lz4TO9iTv9CRGhusf2am7k7PVFPG4RcjXKU iMgLTHKsXbDjncj39JZqwQtBZP8nuf72pMzbEe5iiYYIHGWu4Mm/JSxUmuJZ/YXX yZIS6DKwSrKzDf/p1chvLVCaScxsaIWuetoe4ODFUoWMRUfCdxK+r7+6j9tKn+hn LK5iYVAs3/xpU64jpBKvrKlRlISm8++GvGD+tdnZKAnetmaS0QHb2DrbwSbONvC1 4RvABUYC2IoVoDuAsueRDQxqTJjGPWn7DBvVUI5SCQZVlJPMavHmrv9qRVtNA4Y4 LBzfYK6aCprbmmX2Axs8FS4ptNGdwGpUhwuVKfSzlOgUS5gIaWlMKOGxWOTgDx7U Q53mVhdZIinHH+/h4xBpcP3Q1fk8nTQ1gqfYVTseNlrMZvgFAax7E2VINiJWohis KH6z9bFgkZA= =aNM7 -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 5:50 ` David Masover @ 2004-09-07 6:02 ` Hans Reiser 2004-09-07 6:28 ` William Lee Irwin III 2004-09-07 11:55 ` Christer Weinigel 2004-09-08 6:20 ` Tonnerre 2 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-09-07 6:02 UTC (permalink / raw) To: David Masover Cc: Horst von Brand, Spam, Tonnerre, Christer Weinigel, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List I just want to remind that no one has been able to offer a good way of handling attributes/streams/metafiles other than reiser4 in which CTRL-XCTRL-Ffilename within emacs to edit the stream/attribute/metafile will work without modifying the emacs source. David Masover is right that there are far more application writers than kernel hackers, and we should make the kernel more complicated if it makes a few thousand apps simpler. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 6:02 ` Hans Reiser @ 2004-09-07 6:28 ` William Lee Irwin III 2004-09-07 22:38 ` William Lee Irwin III 0 siblings, 1 reply; 1392+ messages in thread From: William Lee Irwin III @ 2004-09-07 6:28 UTC (permalink / raw) To: Hans Reiser Cc: David Masover, Horst von Brand, Spam, Tonnerre, Christer Weinigel, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Mon, Sep 06, 2004 at 11:02:01PM -0700, Hans Reiser wrote: > I just want to remind that no one has been able to offer a good way of > handling attributes/streams/metafiles other than reiser4 in which > CTRL-XCTRL-Ffilename within emacs to edit the stream/attribute/metafile > will work without modifying the emacs source. David Masover is right > that there are far more application writers than kernel hackers, and we > should make the kernel more complicated if it makes a few thousand apps > simpler. This thread is getting a bit soft on technical details and/or value. Hans, please post the results of fsstress (the bits from ext3 cvs) on reiser4 on SMP machines of non-i386 architectures (e.g. ppc64, sparc64, ia64) as well as the results of transferring reiser4 filesystems between machines whose PAGE_SIZE, wordsize and endianness vary. -- wli ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 6:28 ` William Lee Irwin III @ 2004-09-07 22:38 ` William Lee Irwin III 2004-09-08 2:43 ` William Lee Irwin III 0 siblings, 1 reply; 1392+ messages in thread From: William Lee Irwin III @ 2004-09-07 22:38 UTC (permalink / raw) To: Hans Reiser Cc: David Masover, Horst von Brand, Spam, Tonnerre, Christer Weinigel, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Mon, Sep 06, 2004 at 11:28:06PM -0700, William Lee Irwin III wrote: > This thread is getting a bit soft on technical details and/or value. > Hans, please post the results of fsstress (the bits from ext3 cvs) on > reiser4 on SMP machines of non-i386 architectures (e.g. ppc64, sparc64, > ia64) as well as the results of transferring reiser4 filesystems > between machines whose PAGE_SIZE, wordsize and endianness vary. Okay, sounds like time to run these tests and post the results myself. Hans, these things should be part of your standard QA. It would likely make a better impression if there were some record of these kinds of things having been successfully tested prior to your releases. For future reference, Andrew, Christoph, myself, and others can provide more detailed references to suites of stress tests and various kinds of tests filesystems should pass before being considered stable, and we (it's a relatively safe presumption that I speak for all of us when I say this) would appreciate this kind of testing in the future. -- wli ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 22:38 ` William Lee Irwin III @ 2004-09-08 2:43 ` William Lee Irwin III 2004-09-08 3:03 ` William Lee Irwin III 0 siblings, 1 reply; 1392+ messages in thread From: William Lee Irwin III @ 2004-09-08 2:43 UTC (permalink / raw) To: Hans Reiser Cc: David Masover, Horst von Brand, Spam, Tonnerre, Christer Weinigel, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 5599 bytes --] On Tue, Sep 07, 2004 at 03:38:01PM -0700, William Lee Irwin III wrote: > Okay, sounds like time to run these tests and post the results myself. > Hans, these things should be part of your standard QA. It would likely > make a better impression if there were some record of these kinds of > things having been successfully tested prior to your releases. For > future reference, Andrew, Christoph, myself, and others can provide > more detailed references to suites of stress tests and various kinds of > tests filesystems should pass before being considered stable, and we > (it's a relatively safe presumption that I speak for all of us when I > say this) would appreciate this kind of testing in the future. Step 1, the tools are very broken. This level of nonfunctionality of the reiser4 toolchain precludes any kind of exposure to the kind of testing I've asked about. I would very strongly prefer not to have to become a reiser4 implementor and furthermore fix numerous bugs just to have the smallest bit of assurance that this thing won't generate bug reports en masse once merged. The following is from an UltraSPARC system (64-bit wordsize, big-endian, 8KB pagesize, 32-bit userspace, including 32-bit compiled reiser4 tools) running 2.6.9-rc1-mm3: # /usr/local/sbin/mkfs.reiser4 -f -b 4096 /dev/loop0 /usr/local/sioctl32(mkfs.reiser4:1949): Unknown cmd fd(3) cmd(40081272){00} arg(efffc810) on /dev/loop0 4boicnt/lm3k2f(sm.krfesi.sreeri4s e1r.40:.109 9C)o:p yUrnikgnhotw n( Cc)m d2 0f0d1(,3 )2 0c0m2d,( 420000831,2 7220)0{40 0b}y raHragn(se fRfefics8e1r0,) loinc e/ndseivn/gl ogoopv0e ned by reiser4progs/COPYING. Block size 4096 will be used. Linux 2.6.9-rc1-mm3 is detected. Reiser4 is going to be created on /dev/loop0. (Yes/No): Yes ioctl32(mkfs.reiser4:1949): Unknown cmd fd(3) cmd(40081272){00} arg(efffc738) on /dev/loop0 i o c t l 3 2 ( m k f s . r e i s e r 4 : 1 9 4 9 ) : U n k n o w n c m d sC r/edaetvi/nlgo orpe0i0 8 1 2 7 2 ) { 0 0 } a r g ( e f f f c 7 3 8 ) er4 on /dev/loop0 ... Bus error # With the below patch, I get: # /usr/local/sbin/mkfs.reiser4 -f -b 4096 /dev/loop0 /usr/local/sbin/mkfs.reiser4 1.0.0 Copyright (C) 2001, 2002, 2003, 2004 by Hans Reiser, licensing governed by reiser4progs/COPYING. Block size 4096 will be used. Linux 2.6.9-rc1-mm3 is detected. Reiser4 is going to be created on /dev/loop0. (Yes/No): Yes Creating reiser4 on /dev/loop0 ... Bus error # The S_ISBLK() check is so that other assumptions about the file being a block device elsewhere won't be tripped up; it's not strictly necessary, only retained so as not to expose unaudited code to a new situation for which it's not prepared. strace(1) before and after the patch is included here. The backtrace from the core dump is: #0 0x7005e464 in cde40_insert_units () from /usr/local/lib/libreiser4-1.0.so.0 (gdb) bt #0 0x7005e464 in cde40_insert_units () from /usr/local/lib/libreiser4-1.0.so.0 #1 0x7004fa44 in node40_modify () from /usr/local/lib/libreiser4-1.0.so.0 #2 0x7003c608 in cb_node_insert () from /usr/local/lib/libreiser4-1.0.so.0 #3 0x7003c58c in reiser4_node_modify () from /usr/local/lib/libreiser4-1.0.so.0 #4 0x7003ee7c in reiser4_tree_modify () from /usr/local/lib/libreiser4-1.0.so.0 #5 0x7006bc10 in obj40_insert () from /usr/local/lib/libreiser4-1.0.so.0 #6 0x7006d9e8 in dir40_create () from /usr/local/lib/libreiser4-1.0.so.0 #7 0x70041610 in reiser4_object_create () from /usr/local/lib/libreiser4-1.0.so.0 #8 0x00012160 in main () -- wli Index: libaal-1.0.0/src/file.c =================================================================== --- libaal-1.0.0.orig/src/file.c 2004-01-08 06:49:40.000000000 -0800 +++ libaal-1.0.0/src/file.c 2004-09-07 19:36:49.593844072 -0700 @@ -193,59 +193,31 @@ return !aal_strncmp(file1, file2, aal_strlen(file1)); } -#if defined(__linux__) && defined(_IOR) && !defined(BLKGETSIZE64) -# define BLKGETSIZE64 _IOR(0x12, 114, uint64_t) -#endif - /* Handler for "len" operation for use with file device. See bellow for understanding where it is used. */ static count_t file_len( aal_device_t *device) /* file device, lenght will be obtained from */ { - uint64_t size; - off_t max_off = 0; + off_t off, size; + struct stat stat_buf; + int fd; - if (!device) + if (!device) return INVAL_BLK; - -#ifdef BLKGETSIZE64 - if ((int)ioctl(*((int *)device->entity), BLKGETSIZE64, &size) >= (int)0) { - uint32_t block_count; - - size = (size / 4096) * 4096 / device->blksize; - block_count = size; - - if ((uint64_t)block_count != size) { - aal_fatal("The partition size is too big."); - return INVAL_BLK; - } - - return (count_t)block_count; - } - -#endif - -#ifdef BLKGETSIZE - { - unsigned long l_size; - - if (ioctl(*((int *)device->entity), BLKGETSIZE, &l_size) >= 0) { - size = l_size; - return (count_t)((size * 512 / 4096) * 4096 / - device->blksize); - } - } - -#endif - - if ((max_off = lseek(*((int *)device->entity), - 0, SEEK_END)) == (off_t)-1) - { - file_error(device); + fd = *(int *)device->entity; + if (fstat(fd, &stat_buf)) + return INVAL_BLK; + if (!S_ISBLK(stat_buf.st_mode)) + return INVAL_BLK; + off = lseek(fd, 0, SEEK_CUR); + size = lseek(fd, 0, SEEK_END); + if (lseek(fd, off, SEEK_SET) == (off_t)-1) + errno = 0; + if (size == (off_t)-1) { + errno = 0; return INVAL_BLK; } - - return (count_t)(max_off / device->blksize); + return (size & ~4096ULL)/device->blksize; } /* Initializing the file device operations. They are used when any operation of [-- Attachment #2: mkfs.reiser4.log.1.gz --] [-- Type: application/octet-stream, Size: 2695 bytes --] [-- Attachment #3: mkfs.reiser4.log.2.gz --] [-- Type: application/octet-stream, Size: 2679 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-08 2:43 ` William Lee Irwin III @ 2004-09-08 3:03 ` William Lee Irwin III 0 siblings, 0 replies; 1392+ messages in thread From: William Lee Irwin III @ 2004-09-08 3:03 UTC (permalink / raw) To: Hans Reiser Cc: David Masover, Horst von Brand, Spam, Tonnerre, Christer Weinigel, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Tue, Sep 07, 2004 at 07:43:19PM -0700, William Lee Irwin III wrote: > - return (count_t)(max_off / device->blksize); > + return (size & ~4096ULL)/device->blksize; Correcting this to return (size & ~4095ULL)/device->blksize does not fix the coredumps. -- wli ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 5:50 ` David Masover 2004-09-07 6:02 ` Hans Reiser @ 2004-09-07 11:55 ` Christer Weinigel 2004-09-07 12:26 ` Gunnar Ritter ` (3 more replies) 2004-09-08 6:20 ` Tonnerre 2 siblings, 4 replies; 1392+ messages in thread From: Christer Weinigel @ 2004-09-07 11:55 UTC (permalink / raw) To: David Masover Cc: Horst von Brand, Spam, Tonnerre, Christer Weinigel, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List David Masover <ninja@slaphack.com> writes: > |>Second, there are quite a few things which I might want to do, which can > |>be done with this interface and without patching programs, > | Such as? > They've been mentioned. > | Haven't seen any that made sense to me, sorry. > Sorry if they don't make sense to you, but I don't feel like discussing > them now. Either you get it or you don't, either you agree or you > don't. Read the archives. Great argument. Not. There has been so much shit thrown around here so that it's impossible to keep track of all examples. Could you please try summarize a few of the arguments that you find especially compelling? This thread has gotten very confused since there are a bunch of different subjects all being intermixed here. What are we discussing? 1. Do we want support for named streams? I belive the answer is yes, since both NTFS and HFS (that's the MacOS filesystem, isn't it?) supports streams we want Linux to support this if possible. Anyone disagreeing? 2. How do we want to expose named streams? One suggestion is file-as-directory in some form. Another suggestion made is to expose named streams somewhere under /proc/self/fd. Yet another suggestion is to use the openat(3) API from solaris. Some filesystems exposes extra data in a special directory in the same directory as the file, such as netapps .snapshot directories or the extra directories that netatalk expects. This has the advantage that it even works on non-named stream capable filesystems, but it has a lot of problems too. Linux already has limited support for names streams via the xattr interface, but it's not a good interface for people wanting to have large files as named streams. 4. What belongs in the generic VFS, what belongs in Reiser4? Some things reiser4 do, such as files-as-directories need changes to the VFS because it breaks assumptions that the VFS makes (i.e. a deadlock or an oops when doing a hard link out of one). Some other things reiser4 can do would be better if they were in the VFS since other filesystems might want to support the same functionality. Or Linux may not support some of the things reiserfs at all. 5. What belongs in the kernel, what belongs in userspace? This is mostly what I have been trying to argue about. So, to try to summarize my opinion, regarding file-as-directory, I belive it's fatally flawed because it breaks a lot of assumptions that existing code make. One example of an application that will break is a web server that tries to filter out accesses to "bad" files, files-as-directories suddenly means that part of those files will be accessible (and there are a _lot_ of CERT reports on just this kind of problems with Windows web servers due to access to named streams not being restricted or ways to access files with non-canonical names that also managed to bypass access restrictions). Files-as-directories also does not give us named streams on directories. The suggestion to have dir/metas access the named streams means that if someone already has a file named metas in a directory that file will be lost. (Does anyone remember the discussions about the linux kernel having a directory named "core" and the problems this caused for some people?) All this suggests to me that named streams must live in another namespace than the normal one. To me, openat(3) seems like a good choice for an API and it has the advantage that someone else, Solaris, already has implemented it. Additionally, files-as-directores does not solve the problem of "cp a b" losing named streams. There is curently no copyfile syscall in the Linux kernel, "cp a b" essentially does "cat a >b". So unless cp is modified we don't gain anything. If cp is modified to know about named streams, it really does not matter if named streams are accessed as file-as-directories, via openat(3) or via a shared library with some other interface. Regarding the kernel or userspace discussion. In my opinion anything that can be done in user space should be done in userspace. If the performance sucks, or it has security problems, or needs caching that cant be solved in userspace it can be moved to the kernel, but in that case the smallest and cleanest API possible should be implemented. If, for historical reasons, an API must be in the kernel, there is not much we can do about it either. It'll have to stay there, but we can avoid making the same mistakes again. So, for all the examples of the kernel having plugins that automatically lets an application see a tar-file as a directory, I really, really don't belive this belongs in the kernel. First of all, this is the file-as-directory discussion again, I belive it is a mistake to expose the contents as a directory on top of the file because it breaks a lot of assumptions that unix programs make. It's much better to expose the contents at another place in the filesystem by doing a temporary mount of the file with the proper filesystem. As Pavel Machek pointed out, this has the problem of who cleans up the mount if the application crashes. One way to handle this could be something like this: mount -t tarfs -o loop bar.tar /tmp/bar-fabb50509 chdir /tmp/bar-fabb50509 umount -f /tmp/bar-fabb50509 This will require the ability to unmount busy filesystems (but I belive Alexander Viro already has implemented the infrastructure needed for this). Or for files that we don't have a real filesystem driver (or on other systems where userspace mounts are not allowed), we could just unpack the contents into /tmp. For cleanup we could let whatever cleans up /tmp anyways handle it, or have a cache daemon that keeps track of untarred directories and removes them after a while. Another way is to completely forget about presenting the contents of a tar file as a real files, and just use a shared library to get at the contents (now we just have to convince everyone to use the shared library). This would also be portable to other systems. If we do this right, it could all be hidden in a shared library, and if the system below it supports more advanced features, it can use it. Regarding the "I want a realtime index of all files". I belive that a notifier that can tell me when a file has been changed and a userspace daemon ought to handle most of the cases that have been mentioned. The suggested problems of not getting an up to date query response can be handled by just asking the daemon "are you done with indexing yet". The design of such a daemon and the support it needs from the kernel can definitely be discussed. But to put the indexer itself in the kernel sounds like a bad idea. Even adding an API to query the indexer into the kernel sounds pointless, why do that instead of just opening a Unix socket to the indexer and asking it directly? /Christer -- "Just how much can I get away with and still go to heaven?" Freelance consultant specializing in device driver programming for Linux Christer Weinigel <christer@weinigel.se> http://www.weinigel.se ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 11:55 ` Christer Weinigel @ 2004-09-07 12:26 ` Gunnar Ritter 2004-09-07 18:20 ` Hans Reiser 2004-09-07 12:30 ` Herbert Poetzl ` (2 subsequent siblings) 3 siblings, 1 reply; 1392+ messages in thread From: Gunnar Ritter @ 2004-09-07 12:26 UTC (permalink / raw) To: David Masover, Christer Weinigel Cc: Horst von Brand, viro, Linus Torvalds, Tonnerre, Spam, ReiserFS List, Hans Reiser, Pavel Machek, linux-kernel, linux-fsdevel, Jamie Lokier, Christoph Hellwig, Alexander Lyamin aka FLX, Chris Wedgwood, Christer Weinigel Christer Weinigel <christer@weinigel.se> wrote: > Additionally, files-as-directores does not solve the problem of > "cp a b" losing named streams. There is curently no copyfile syscall > in the Linux kernel, "cp a b" essentially does "cat a >b". So unless > cp is modified we don't gain anything. If cp is modified to know > about named streams, it really does not matter if named streams are > accessed as file-as-directories, via openat(3) or via a shared library > with some other interface. You cannot just 'modify cp'. cp is a programming interface standardized in POSIX.1. You can of course add non-standard extensions to some cp implementations, but it seems hardly evitable then that you either have to use cp in a non-standard manner regularly with Linux or risk to lose data. This is even more severe with tar/pax. Just patching GNU tar for file streams, as it was suggested earlier in this discussion, is still far away from a real solution because it neither solves the issues with the POSIX.1 pax standard nor those with other implementations of it. Given these facts, it does not seem so clear to me that adding named streams for Windows and Mac OS interoperability would be a win to Linux in the end. The loss of interoperability to Unix/POSIX/today's Linux might have much worse effects. The current xattr extension is much less of a problem because it only holds metadata, which is mostly not applicable to other environments anyway. Gunnar -- http://omnibus.ruf.uni-freiburg.de/~gritter ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 12:26 ` Gunnar Ritter @ 2004-09-07 18:20 ` Hans Reiser 2004-09-07 19:22 ` Gunnar Ritter 0 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-09-07 18:20 UTC (permalink / raw) To: Gunnar Ritter Cc: David Masover, Christer Weinigel, Horst von Brand, viro, Linus Torvalds, Tonnerre, Spam, ReiserFS List, Pavel Machek, linux-kernel, linux-fsdevel, Jamie Lokier, Christoph Hellwig, Alexander Lyamin aka FLX, Chris Wedgwood Gunnar Ritter wrote: > > >You cannot just 'modify cp'. > People who think that POSIX is the objective rather than the least common denominator of OS design have had their head screwed on backwards to better look at where their competitors used to be. However, I agree that streams suck. That is why reiser4 just has files and directories and not streams. Our files and directories just happen to be able to do all that streams can do. Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 18:20 ` Hans Reiser @ 2004-09-07 19:22 ` Gunnar Ritter 0 siblings, 0 replies; 1392+ messages in thread From: Gunnar Ritter @ 2004-09-07 19:22 UTC (permalink / raw) To: Hans Reiser Cc: Horst von Brand, viro, Linus Torvalds, Tonnerre, Spam, ReiserFS List, Pavel Machek, David Masover, linux-kernel, linux-fsdevel, Jamie Lokier, Christoph Hellwig, Alexander Lyamin aka FLX, Chris Wedgwood, Christer Weinigel Hans Reiser <reiser@namesys.com> wrote: > Gunnar Ritter wrote: > >You cannot just 'modify cp'. > > > People who think that POSIX is the objective rather than the least > common denominator of OS design I am not principally adversed against extensions to POSIX. My mailx implementation 'nail' has e.g. perhaps more extensions than there are commands and options in the POSIX standard for it. POSIX is also not against extensions. In fact, POSIX development generally works as follows: One vendor creates something as an extension, other vendors follow to implement it, and later on it is discussed if it is desirable to integrate the feature into the standard itself. It is absolutely possible that Sun's openat() might be in POSIX.1-2010 one day, for example. Useful extensions are thus welcome to POSIX. This does not mean, however, that one should not clearly distinct between standard and extensions, and that extensions should be created at will without carefully weighting pros and cons. I did not say: POSIX forbids to handle streams or directory/file mixes. This would not even have been true. However, POSIX restricts the choice of possible interfaces for them. One of those restrictions is that 'cp' may not copy streams if used in strict accordance with POSIX. As you acknowledged in your reply, POSIX is the least common denominator. Thus 'cp' implementations should not be modified in a way that violates it. This means, in effect, that a strictly conforming POSIX application (i.e. something like a shell script that uses no POSIX extensions or methods which are not clearly defined in the standard) will very likely be unable to copy streams, unless some other, conforming, method is found. Which is a problem one should know about when discussing this. > have had their head screwed on backwards > to better look at where their competitors used to be. But there are not only forwards and backwards directions. Sideways might lead to nowhere. Gunnar -- http://omnibus.ruf.uni-freiburg.de/~gritter ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 11:55 ` Christer Weinigel 2004-09-07 12:26 ` Gunnar Ritter @ 2004-09-07 12:30 ` Herbert Poetzl 2004-09-07 13:44 ` Oliver Neukum 2004-09-07 12:33 ` Spam 2004-09-07 18:34 ` Hans Reiser 3 siblings, 1 reply; 1392+ messages in thread From: Herbert Poetzl @ 2004-09-07 12:30 UTC (permalink / raw) To: Christer Weinigel Cc: David Masover, Horst von Brand, Spam, Tonnerre, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Tue, Sep 07, 2004 at 01:55:32PM +0200, Christer Weinigel wrote: > David Masover <ninja@slaphack.com> writes: > > > |>Second, there are quite a few things which I might want to do, which can > > |>be done with this interface and without patching programs, > > | Such as? > > They've been mentioned. > > > | Haven't seen any that made sense to me, sorry. > > Sorry if they don't make sense to you, but I don't feel like discussing > > them now. Either you get it or you don't, either you agree or you > > don't. Read the archives. > > Great argument. Not. There has been so much shit thrown around here > so that it's impossible to keep track of all examples. > > Could you please try summarize a few of the arguments that you find > especially compelling? This thread has gotten very confused since > there are a bunch of different subjects all being intermixed here. > > What are we discussing? > > 1. Do we want support for named streams? > > I belive the answer is yes, since both NTFS and HFS (that's the > MacOS filesystem, isn't it?) supports streams we want Linux to > support this if possible. well, yes HFS has this, is it advantageous, no it's kind of heritage ... > Anyone disagreeing? yes, MacOS X allows to use UFS instead of HFS+ which doesn't support the fancy/confusing streams I, for my part, do not like the idea of multiple streams for one file, IMHO all features can be provided by using directories instead, which does not break any userspace tools _and_ sounds natural to me ... best, Herbert > 2. How do we want to expose named streams? > > One suggestion is file-as-directory in some form. > > Another suggestion made is to expose named streams somewhere under > /proc/self/fd. > > Yet another suggestion is to use the openat(3) API from solaris. > > Some filesystems exposes extra data in a special directory in the > same directory as the file, such as netapps .snapshot directories > or the extra directories that netatalk expects. This has the > advantage that it even works on non-named stream capable > filesystems, but it has a lot of problems too. > > Linux already has limited support for names streams via the xattr > interface, but it's not a good interface for people wanting to > have large files as named streams. > > 4. What belongs in the generic VFS, what belongs in Reiser4? > > Some things reiser4 do, such as files-as-directories need changes > to the VFS because it breaks assumptions that the VFS makes > (i.e. a deadlock or an oops when doing a hard link out of one). > > Some other things reiser4 can do would be better if they were in > the VFS since other filesystems might want to support the same > functionality. > > Or Linux may not support some of the things reiserfs at all. > > 5. What belongs in the kernel, what belongs in userspace? > > This is mostly what I have been trying to argue about. > > So, to try to summarize my opinion, regarding file-as-directory, I > belive it's fatally flawed because it breaks a lot of assumptions that > existing code make. One example of an application that will break is > a web server that tries to filter out accesses to "bad" files, > files-as-directories suddenly means that part of those files will be > accessible (and there are a _lot_ of CERT reports on just this kind of > problems with Windows web servers due to access to named streams not > being restricted or ways to access files with non-canonical names that > also managed to bypass access restrictions). > > Files-as-directories also does not give us named streams on > directories. The suggestion to have dir/metas access the named > streams means that if someone already has a file named metas in a > directory that file will be lost. (Does anyone remember the > discussions about the linux kernel having a directory named "core" and > the problems this caused for some people?) > > All this suggests to me that named streams must live in another > namespace than the normal one. To me, openat(3) seems like a good > choice for an API and it has the advantage that someone else, Solaris, > already has implemented it. > > Additionally, files-as-directores does not solve the problem of > "cp a b" losing named streams. There is curently no copyfile syscall > in the Linux kernel, "cp a b" essentially does "cat a >b". So unless > cp is modified we don't gain anything. If cp is modified to know > about named streams, it really does not matter if named streams are > accessed as file-as-directories, via openat(3) or via a shared library > with some other interface. > > Regarding the kernel or userspace discussion. In my opinion anything > that can be done in user space should be done in userspace. If the > performance sucks, or it has security problems, or needs caching that > cant be solved in userspace it can be moved to the kernel, but in that > case the smallest and cleanest API possible should be implemented. > > If, for historical reasons, an API must be in the kernel, there is not > much we can do about it either. It'll have to stay there, but we can > avoid making the same mistakes again. > > So, for all the examples of the kernel having plugins that > automatically lets an application see a tar-file as a directory, I > really, really don't belive this belongs in the kernel. First of all, > this is the file-as-directory discussion again, I belive it is a > mistake to expose the contents as a directory on top of the file > because it breaks a lot of assumptions that unix programs make. > > It's much better to expose the contents at another place in the > filesystem by doing a temporary mount of the file with the proper > filesystem. As Pavel Machek pointed out, this has the problem of who > cleans up the mount if the application crashes. One way to handle > this could be something like this: > > mount -t tarfs -o loop bar.tar /tmp/bar-fabb50509 > chdir /tmp/bar-fabb50509 > umount -f /tmp/bar-fabb50509 > > This will require the ability to unmount busy filesystems (but I > belive Alexander Viro already has implemented the infrastructure > needed for this). > > Or for files that we don't have a real filesystem driver (or on other > systems where userspace mounts are not allowed), we could just unpack > the contents into /tmp. For cleanup we could let whatever cleans up > /tmp anyways handle it, or have a cache daemon that keeps track of > untarred directories and removes them after a while. > > Another way is to completely forget about presenting the contents of a > tar file as a real files, and just use a shared library to get at the > contents (now we just have to convince everyone to use the shared > library). This would also be portable to other systems. > > If we do this right, it could all be hidden in a shared library, and > if the system below it supports more advanced features, it can use it. > > Regarding the "I want a realtime index of all files". I belive that a > notifier that can tell me when a file has been changed and a userspace > daemon ought to handle most of the cases that have been mentioned. > The suggested problems of not getting an up to date query response can > be handled by just asking the daemon "are you done with indexing yet". > The design of such a daemon and the support it needs from the kernel > can definitely be discussed. But to put the indexer itself in the > kernel sounds like a bad idea. Even adding an API to query the > indexer into the kernel sounds pointless, why do that instead of just > opening a Unix socket to the indexer and asking it directly? > > /Christer > > -- > "Just how much can I get away with and still go to heaven?" > > Freelance consultant specializing in device driver programming for Linux > Christer Weinigel <christer@weinigel.se> http://www.weinigel.se > - > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 12:30 ` Herbert Poetzl @ 2004-09-07 13:44 ` Oliver Neukum 0 siblings, 0 replies; 1392+ messages in thread From: Oliver Neukum @ 2004-09-07 13:44 UTC (permalink / raw) To: Herbert Poetzl Cc: Christer Weinigel, David Masover, Horst von Brand, Spam, Tonnerre, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Am Dienstag, 7. September 2004 14:30 schrieb Herbert Poetzl: > > 1. Do we want support for named streams? > > > > I belive the answer is yes, since both NTFS and HFS (that's the > > MacOS filesystem, isn't it?) supports streams we want Linux to > > support this if possible. > > well, yes HFS has this, is it advantageous, no > it's kind of heritage ... HFS doesn't support _named_ streams. There are just two forks per file. Regards Oliver ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 11:55 ` Christer Weinigel 2004-09-07 12:26 ` Gunnar Ritter 2004-09-07 12:30 ` Herbert Poetzl @ 2004-09-07 12:33 ` Spam 2004-09-07 13:40 ` Christer Weinigel ` (2 more replies) 2004-09-07 18:34 ` Hans Reiser 3 siblings, 3 replies; 1392+ messages in thread From: Spam @ 2004-09-07 12:33 UTC (permalink / raw) To: Christer Weinigel Cc: David Masover, Horst von Brand, Tonnerre, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > David Masover <ninja@slaphack.com> writes: >> |>Second, there are quite a few things which I might want to do, which can >> |>be done with this interface and without patching programs, >> | Such as? >> They've been mentioned. >> | Haven't seen any that made sense to me, sorry. >> Sorry if they don't make sense to you, but I don't feel like discussing >> them now. Either you get it or you don't, either you agree or you >> don't. Read the archives. > Great argument. Not. There has been so much shit thrown around here > so that it's impossible to keep track of all examples. > Could you please try summarize a few of the arguments that you find > especially compelling? This thread has gotten very confused since > there are a bunch of different subjects all being intermixed here. Indeed. We are discussion changes to the heart of Linux. It is bound to get a little heated :/ > What are we discussing? > 1. Do we want support for named streams? > I belive the answer is yes, since both NTFS and HFS (that's the > MacOS filesystem, isn't it?) supports streams we want Linux to > support this if possible. > Anyone disagreeing? No :) > 2. How do we want to expose named streams? > One suggestion is file-as-directory in some form. > Another suggestion made is to expose named streams somewhere under > /proc/self/fd. > Yet another suggestion is to use the openat(3) API from solaris. > Some filesystems exposes extra data in a special directory in the > same directory as the file, such as netapps .snapshot directories > or the extra directories that netatalk expects. This has the > advantage that it even works on non-named stream capable > filesystems, but it has a lot of problems too. > Linux already has limited support for names streams via the xattr > interface, but it's not a good interface for people wanting to > have large files as named streams. > 4. What belongs in the generic VFS, what belongs in Reiser4? > Some things reiser4 do, such as files-as-directories need changes > to the VFS because it breaks assumptions that the VFS makes > (i.e. a deadlock or an oops when doing a hard link out of one). > Some other things reiser4 can do would be better if they were in > the VFS since other filesystems might want to support the same > functionality. > Or Linux may not support some of the things reiserfs at all. > 5. What belongs in the kernel, what belongs in userspace? > This is mostly what I have been trying to argue about. > So, to try to summarize my opinion, regarding file-as-directory, I > belive it's fatally flawed because it breaks a lot of assumptions that > existing code make. One example of an application that will break is > a web server that tries to filter out accesses to "bad" files, > files-as-directories suddenly means that part of those files will be > accessible (and there are a _lot_ of CERT reports on just this kind of > problems with Windows web servers due to access to named streams not > being restricted or ways to access files with non-canonical names that > also managed to bypass access restrictions). But restrictions could be controlled. But exactly how depends on what solution for named streams is chosen. For example, there might be separate permissions for named streams other than the main stream. > Files-as-directories also does not give us named streams on > directories. The suggestion to have dir/metas access the named > streams means that if someone already has a file named metas in a > directory that file will be lost. (Does anyone remember the > discussions about the linux kernel having a directory named "core" and > the problems this caused for some people?) Yes, reserved keywords needs to be chosen very carefully if they are used. Other examples here has been ..metas or ... > All this suggests to me that named streams must live in another > namespace than the normal one. To me, openat(3) seems like a good > choice for an API and it has the advantage that someone else, Solaris, > already has implemented it. > Additionally, files-as-directores does not solve the problem of > "cp a b" losing named streams. There is curently no copyfile syscall > in the Linux kernel, "cp a b" essentially does "cat a >b". So unless > cp is modified we don't gain anything. If cp is modified to know > about named streams, it really does not matter if named streams are > accessed as file-as-directories, via openat(3) or via a shared library > with some other interface. One suggestion is missed. It is to provide system calls for copy. That would also solve the problem. Named streams and metas would then be handled correctly. It also allows further changes to filesystems without having to patch applications yet again. A copy system call would also be large beneficial for networked filesystems (NFS, Samba, etc) as data wouldn't have to be transferred over the network and back. > Regarding the kernel or userspace discussion. In my opinion anything > that can be done in user space should be done in userspace. If the > performance sucks, or it has security problems, or needs caching that > cant be solved in userspace it can be moved to the kernel, but in that > case the smallest and cleanest API possible should be implemented. Yes. I agree. I do not think most people arguing for plugins or usage for named streams actually wanted to put everything in the kernel, but rather to have a kernel interface to which user-level modules, plugins and applications could work. > If, for historical reasons, an API must be in the kernel, there is not > much we can do about it either. It'll have to stay there, but we can > avoid making the same mistakes again. > So, for all the examples of the kernel having plugins that > automatically lets an application see a tar-file as a directory, I > really, really don't belive this belongs in the kernel. First of all, > this is the file-as-directory discussion again, I belive it is a > mistake to expose the contents as a directory on top of the file > because it breaks a lot of assumptions that unix programs make. > It's much better to expose the contents at another place in the > filesystem by doing a temporary mount of the file with the proper > filesystem. As Pavel Machek pointed out, this has the problem of who > cleans up the mount if the application crashes. One way to handle > this could be something like this: > mount -t tarfs -o loop bar.tar /tmp/bar-fabb50509 > chdir /tmp/bar-fabb50509 > umount -f /tmp/bar-fabb50509 > This will require the ability to unmount busy filesystems (but I > belive Alexander Viro already has implemented the infrastructure > needed for this). > Or for files that we don't have a real filesystem driver (or on other > systems where userspace mounts are not allowed), we could just unpack > the contents into /tmp. For cleanup we could let whatever cleans up > /tmp anyways handle it, or have a cache daemon that keeps track of > untarred directories and removes them after a while. But now, as you said, we are not talking about named streams, but about a specic type of plugin and its implications. Can we make a plugin infrastructure that will let user-space plugins to be loaded for certain directories or files? If we can, then it would present a much cleaner and easier way for the user to access data he wants. In this particular example it was a tar file. > Another way is to completely forget about presenting the contents of a > tar file as a real files, and just use a shared library to get at the > contents (now we just have to convince everyone to use the shared > library). This would also be portable to other systems. Yes. If it was possible. I do honestly think it will be easier and less man hour work to create a run-time loadable user-space plugin interface than to convince thousands of application developers to use this shared library. > If we do this right, it could all be hidden in a shared library, and > if the system below it supports more advanced features, it can use it. > Regarding the "I want a realtime index of all files". I belive that a > notifier that can tell me when a file has been changed and a userspace > daemon ought to handle most of the cases that have been mentioned. We have FAM (File Alteration Monitor) that notifies applications of changes to files they are interested in. http://oss.sgi.com/projects/fam/ > The suggested problems of not getting an up to date query response can > be handled by just asking the daemon "are you done with indexing yet". > The design of such a daemon and the support it needs from the kernel > can definitely be discussed. But to put the indexer itself in the > kernel sounds like a bad idea. Even adding an API to query the > indexer into the kernel sounds pointless, why do that instead of just > opening a Unix socket to the indexer and asking it directly? I think that we do not need a indexer with Reiser4 anyway. It is already a database that could be queried directly. There shouldn't be any need to build a database on top of Reiser4 (like the updatedb) that holds the same information already existing in Reiser4. Now, it is just a matter of having an interface to query the Reiser4 db. One of the suggested ways were through meta-data. For example: ls -l ..metas/keyword/.pdf ~S > /Christer ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 12:33 ` Spam @ 2004-09-07 13:40 ` Christer Weinigel 2004-09-07 13:52 ` Spam 2004-09-07 18:11 ` Hans Reiser 2004-09-07 15:30 ` Horst von Brand 2004-09-07 15:37 ` Gunnar Ritter 2 siblings, 2 replies; 1392+ messages in thread From: Christer Weinigel @ 2004-09-07 13:40 UTC (permalink / raw) To: Spam Cc: Christer Weinigel, David Masover, Horst von Brand, Tonnerre, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Spam <spam@tnonline.net> writes: > > Additionally, files-as-directores does not solve the problem of > > "cp a b" losing named streams. There is curently no copyfile syscall > > in the Linux kernel, "cp a b" essentially does "cat a >b". So unless > > cp is modified we don't gain anything. If cp is modified to know > > about named streams, it really does not matter if named streams are > > accessed as file-as-directories, via openat(3) or via a shared library > > with some other interface. > > One suggestion is missed. It is to provide system calls for copy. > That would also solve the problem. Named streams and metas would > then be handled correctly. It also allows further changes to > filesystems without having to patch applications yet again. But this still solves only part of the problem. A backup application won't have any use for a copyfile syscall, it will need to be taught about streams. > A copy system call would also be large beneficial for networked > filesystems (NFS, Samba, etc) as data wouldn't have to be > transferred over the network and back. Definitely. > Can we make a plugin infrastructure that will let user-space plugins > to be loaded for certain directories or files? If we can, then it > would present a much cleaner and easier way for the user to access > data he wants. In this particular example it was a tar file. In that case I'd argue that: mount -t userfs -o driver=tarfs foo /tmp/foo is a rather good kernel interface for plugins. userfs (or something based on userfs) is the plugin API and tarfs is a plugin. :-) To make this efficient, well have to allow non-root users to perform the mount syscall (with the limitation that they can only mount on top of directories they own and that the mounts have the nosuid and nodev flags set). /Christer -- "Just how much can I get away with and still go to heaven?" Freelance consultant specializing in device driver programming for Linux Christer Weinigel <christer@weinigel.se> http://www.weinigel.se ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 13:40 ` Christer Weinigel @ 2004-09-07 13:52 ` Spam 2004-09-07 15:37 ` Horst von Brand 2004-09-07 18:11 ` Hans Reiser 1 sibling, 1 reply; 1392+ messages in thread From: Spam @ 2004-09-07 13:52 UTC (permalink / raw) To: Christer Weinigel Cc: David Masover, Horst von Brand, Tonnerre, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > Spam <spam@tnonline.net> writes: >> > Additionally, files-as-directores does not solve the problem of >> > "cp a b" losing named streams. There is curently no copyfile syscall >> > in the Linux kernel, "cp a b" essentially does "cat a >b". So unless >> > cp is modified we don't gain anything. If cp is modified to know >> > about named streams, it really does not matter if named streams are >> > accessed as file-as-directories, via openat(3) or via a shared library >> > with some other interface. >> >> One suggestion is missed. It is to provide system calls for copy. >> That would also solve the problem. Named streams and metas would >> then be handled correctly. It also allows further changes to >> filesystems without having to patch applications yet again. > But this still solves only part of the problem. A backup application > won't have any use for a copyfile syscall, it will need to be taught > about streams. Yes, but backup programs always needed to be taught about new features. Be it new type of files, attributes or meta-data. I think that teaching backup applications is far better than teaching every application. >> A copy system call would also be large beneficial for networked >> filesystems (NFS, Samba, etc) as data wouldn't have to be >> transferred over the network and back. > Definitely. >> Can we make a plugin infrastructure that will let user-space plugins >> to be loaded for certain directories or files? If we can, then it >> would present a much cleaner and easier way for the user to access >> data he wants. In this particular example it was a tar file. > In that case I'd argue that: > mount -t userfs -o driver=tarfs foo /tmp/foo > is a rather good kernel interface for plugins. userfs (or something > based on userfs) is the plugin API and tarfs is a plugin. :-) > To make this efficient, well have to allow non-root users to perform > the mount syscall (with the limitation that they can only mount on top > of directories they own and that the mounts have the nosuid and nodev > flags set). Yes, this seem to be one solution. It isn't very dynamic in usage though. You can't use this directly from applications wihout manually doing the mount. This is only a solution to browsing contents of files. It doesn't provide a solution for using meta-data streams or other things like this. ~S > /Christer ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 13:52 ` Spam @ 2004-09-07 15:37 ` Horst von Brand 0 siblings, 0 replies; 1392+ messages in thread From: Horst von Brand @ 2004-09-07 15:37 UTC (permalink / raw) To: Spam Cc: Christer Weinigel, David Masover, Tonnerre, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Spam <spam@tnonline.net> said: > Christer Weinigel <christer@weinigel.se> said: [...] > > But this still solves only part of the problem. A backup application > > won't have any use for a copyfile syscall, it will need to be taught > > about streams. > Yes, but backup programs always needed to be taught about new > features. Be it new type of files, attributes or meta-data. I think > that teaching backup applications is far better than teaching every > application. Strange... tar(1) is quite capable of backing up .mp3 files, which weren't around when tar was born... -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 13:40 ` Christer Weinigel 2004-09-07 13:52 ` Spam @ 2004-09-07 18:11 ` Hans Reiser 2004-09-07 21:41 ` Christer Weinigel 1 sibling, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-09-07 18:11 UTC (permalink / raw) To: Christer Weinigel Cc: Spam, David Masover, Horst von Brand, Tonnerre, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Christer Weinigel wrote: >Spam <spam@tnonline.net> writes: > > > >>>Additionally, files-as-directores does not solve the problem of >>>"cp a b" losing named streams. >>> reiser4 does not support streams, it supports files that can do what streams do. cp -r does not currently lose files. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 18:11 ` Hans Reiser @ 2004-09-07 21:41 ` Christer Weinigel 0 siblings, 0 replies; 1392+ messages in thread From: Christer Weinigel @ 2004-09-07 21:41 UTC (permalink / raw) To: Hans Reiser Cc: Christer Weinigel, Spam, David Masover, Horst von Brand, Tonnerre, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hans Reiser <reiser@namesys.com> writes: > Christer Weinigel wrote: > > >Spam <spam@tnonline.net> writes: > > > > > >>> Additionally, files-as-directores does not solve the problem of > >>> "cp a b" losing named streams. > >>> > reiser4 does not support streams, it supports files that can do what > streams do. cp -r does not currently lose files. I'm not talking about reiser4 only. I'm interested in semantics that can be sanely apply to reiser4, NTFS, HFS, or any other file system that supports XATTRS. I relize that resier4 is your baby and you're only interested in it, but I'm much more interested in linux in general. /Christer -- "Just how much can I get away with and still go to heaven?" Freelance consultant specializing in device driver programming for Linux Christer Weinigel <christer@weinigel.se> http://www.weinigel.se ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 12:33 ` Spam 2004-09-07 13:40 ` Christer Weinigel @ 2004-09-07 15:30 ` Horst von Brand 2004-09-07 18:05 ` Hans Reiser 2004-09-07 22:28 ` Christer Weinigel 2004-09-07 15:37 ` Gunnar Ritter 2 siblings, 2 replies; 1392+ messages in thread From: Horst von Brand @ 2004-09-07 15:30 UTC (permalink / raw) To: Spam Cc: Christer Weinigel, David Masover, Tonnerre, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Spam <spam@tnonline.net> said: > Christer Weinigel <christer@weinigel.se> said: [...] > > Could you please try summarize a few of the arguments that you find > > especially compelling? This thread has gotten very confused since > > there are a bunch of different subjects all being intermixed here. > Indeed. We are discussion changes to the heart of Linux. It is bound > to get a little heated :/ True. But without any specific applications, just "this would be nice to have", the discussion can't go forward. > > What are we discussing? > > 1. Do we want support for named streams? > > > I belive the answer is yes, since both NTFS and HFS (that's the > > MacOS filesystem, isn't it?) supports streams we want Linux to > > support this if possible. > > > Anyone disagreeing? > > No :) There are many people around here who disagree (that is precisely the heart of the discussion). I for one don't think Linux has to get $RANDOM_FEATURE just because $SOME_OTHER_OS has got it. Either the feature stands on its own _in the context of POSIX/Unix/Linux_ (possibly as an extension or modification of said standards) or it isn't worth it. > > 2. How do we want to expose named streams? > > > One suggestion is file-as-directory in some form. Which is broken, as it forbids hard links to files. It also has quite interesting issues with moving streams around among containers, or renaming them. How would you handle permissions on streams inside containers? Links to streams? What happens if the principal stream is deleted? Renamed? Moved into another container? Now you have 3 principal types of objects: Directories, containers (files with streams), and files (no streams). What is the essential difference between containers and directories? Same for containers vs plain files? If none, containers are just complicating stuff, and should not exist at all. If you add streams to directories, you get 4 types. Again, tell what the exact differences are, and what each one can do none of the others can. In the preceding two exercises, it is not enough to show something that would be dificult or messy to do with what we have now, it must also be some operation(s) that are natural and common enough to warrant the extra complexity (in kernel, in applications, in wetware). Consider support in form of existing or to-be-written tools for the task. What of all this can the GUI hide from (or fake for) the unsuspecting end-user? Also, there have been suggestions of cached data in streams (like thumbnail version of graphic file, contents listing of archive, ...), where losing the stream is of no real consecuence (except performance). There have also been requests for metadata that can't be recovered from the file (author, comments on the file, older versions of the file (i.e., some form of file version control), permissions). And finally there have been requests for structured contents manipulable in other forms (f.ex. .tar.bz2 file in unpacked form, where you can directly read or even modify the contents). Some suggestions go for a limited number of small(ish), even standardized, extra streams with no further structure, others ask for full directories under streams (which could even be much larger than the principal stream, like the tar.bz2 case). And there is the possibility of recursive application: If you want to be consecuent, the files under a .tar.bz2 should again be able to be .tar.bz2'es with their own streams, or files with streams in them. What permissions should apply to streams? The ones of the file (i.e., I can see the icon stream of a file I can't read), or extra permissions (i.e., icon is writable to me (so I can adjust it to taste) but I can't write to the file; now I can screw up the icon for everybody...). Streams on a per-user basis (i.e., icon stream for each user)? We need to sort out exactly how far it makes sense to go, by showing concrete, down to earth uses for whatever substructure we want. Then show the effect can't be easily gotten through tools for power users or faking it for unsuspecting users via GUI, and that overall the complexity and performance cost is less than the win. Note that the success of the Unix way is in large part due to its use of few, simple concepts that can be combined endlessly; and tools following the same strategy. Adding extra concepts that current tools can't naturally handle has to be considered with extreme care. [-- much snippage --] > > The suggested problems of not getting an up to date query response can > > be handled by just asking the daemon "are you done with indexing yet". > > The design of such a daemon and the support it needs from the kernel > > can definitely be discussed. But to put the indexer itself in the > > kernel sounds like a bad idea. Even adding an API to query the > > indexer into the kernel sounds pointless, why do that instead of just > > opening a Unix socket to the indexer and asking it directly? > I think that we do not need a indexer with Reiser4 anyway. It is > already a database that could be queried directly. There shouldn't > be any need to build a database on top of Reiser4 (like the > updatedb) that holds the same information already existing in > Reiser4. Indexing/general queries of the filesystem are _hard_ (you need a complete database behind it for this, plus handling transactions for full functionality). Applications who _really_ need this use DBMSes, and are prepared to pay the ensuing cost. They are far in between, in any case. Besides, this is completely orthogonal to the streams discussion. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 15:30 ` Horst von Brand @ 2004-09-07 18:05 ` Hans Reiser 2004-09-07 18:15 ` Horst von Brand 2004-09-07 22:28 ` Christer Weinigel 1 sibling, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-09-07 18:05 UTC (permalink / raw) To: Horst von Brand Cc: Spam, Christer Weinigel, David Masover, Tonnerre, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Horst von Brand wrote: >Spam <spam@tnonline.net> said: > > >>Christer Weinigel <christer@weinigel.se> said: >> >> > >[...] > > > >>>Could you please try summarize a few of the arguments that you find >>>especially compelling? This thread has gotten very confused since >>>there are a bunch of different subjects all being intermixed here. >>> >>> > > > >> Indeed. We are discussion changes to the heart of Linux. It is bound >> to get a little heated :/ >> >> > >True. But without any specific applications, just "this would be nice to >have", the discussion can't go forward. > > > >>>What are we discussing? >>> >>> > > > >>>1. Do we want support for named streams? >>> >>> >>> I belive the answer is yes, since both NTFS and HFS (that's the >>> MacOS filesystem, isn't it?) supports streams we want Linux to >>> support this if possible. >>> >>> >>> Anyone disagreeing? >>> >>> >> No :) >> >> > >There are many people around here who disagree (that is precisely the heart >of the discussion). I for one don't think Linux has to get $RANDOM_FEATURE >just because $SOME_OTHER_OS has got it. Either the feature stands on its >own _in the context of POSIX/Unix/Linux_ (possibly as an extension or >modification of said standards) or it isn't worth it. > > > >>>2. How do we want to expose named streams? >>> >>> >>> One suggestion is file-as-directory in some form. >>> >>> > >Which is broken, as it forbids hard links to files. > > No, it forbids hard links to the directory aspect of the file-directory duality. > > >Now you have 3 principal types of objects: Directories, containers (files >with streams), and files (no streams). > No, the reiser4 design supports only files and directories, but makes them able to do what people use streams for. The reiser4 design is based on a hatred of streams, and a desire to show that adding more features to files and directories makes streams unnecessary. Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 18:05 ` Hans Reiser @ 2004-09-07 18:15 ` Horst von Brand 2004-09-07 19:14 ` Hans Reiser 0 siblings, 1 reply; 1392+ messages in thread From: Horst von Brand @ 2004-09-07 18:15 UTC (permalink / raw) To: Hans Reiser Cc: Spam, Christer Weinigel, David Masover, Tonnerre, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hans Reiser <reiser@namesys.com> said: > Horst von Brand wrote: > >Spam <spam@tnonline.net> said: > >>Christer Weinigel <christer@weinigel.se> said: [...] > >>>2. How do we want to expose named streams? > >>> One suggestion is file-as-directory in some form. > >Which is broken, as it forbids hard links to files. > No, it forbids hard links to the directory aspect of the file-directory > duality. How do you distinguish a "hard link to the directory personality" from "hard link to the file personality"? And how do you make sure that only one of them can be followed if you go to the directory, while allowing several to the exact same file?! > >Now you have 3 principal types of objects: Directories, containers (files > >with streams), and files (no streams). > No, the reiser4 design supports only files and directories, but makes > them able to do what people use streams for. I.e., you have files/directories with/without "use as streams" stuff. 4 types (or uses) of things. > The reiser4 design is based on a hatred of streams, and a desire to show > that adding more features to files and directories makes streams > unnecessary. Trade one ugly wart for others... -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 18:15 ` Horst von Brand @ 2004-09-07 19:14 ` Hans Reiser 2004-09-07 21:02 ` Horst von Brand 0 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-09-07 19:14 UTC (permalink / raw) To: Horst von Brand Cc: Spam, Christer Weinigel, David Masover, Tonnerre, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Horst von Brand wrote: >Hans Reiser <reiser@namesys.com> said: > > >>Horst von Brand wrote: >> >> >>>Spam <spam@tnonline.net> said: >>> >>> >>>>Christer Weinigel <christer@weinigel.se> said: >>>> >>>> > >[...] > > > >>>>>2. How do we want to expose named streams? >>>>> >>>>> > > > >>>>> One suggestion is file-as-directory in some form. >>>>> >>>>> > > > >>>Which is broken, as it forbids hard links to files. >>> >>> > > > >>No, it forbids hard links to the directory aspect of the file-directory >>duality. >> >> > >How do you distinguish a "hard link to the directory personality" from >"hard link to the file personality"? > Put in (undoubtedly overly) simple terms, if you can do it to a file you can do it to the file personality, but if you currently can only do it to a directory and get an error from attempting it to a file then in the new scheme doing it to the hard link only gives the same error. Or, we can ask Alexander to help us use his deadlock detection algorithm and try to do things right.... ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 19:14 ` Hans Reiser @ 2004-09-07 21:02 ` Horst von Brand 2004-09-07 21:29 ` Spam 2004-09-08 1:38 ` Alexander G. M. Smith 0 siblings, 2 replies; 1392+ messages in thread From: Horst von Brand @ 2004-09-07 21:02 UTC (permalink / raw) To: Hans Reiser Cc: Spam, Christer Weinigel, David Masover, Tonnerre, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hans Reiser <reiser@namesys.com> said: > Horst von Brand wrote: > >Hans Reiser <reiser@namesys.com> said: > >>Horst von Brand wrote: > >>>Spam <spam@tnonline.net> said: > >>>>Christer Weinigel <christer@weinigel.se> said: [...] > >>>>>2. How do we want to expose named streams? > >>>>> One suggestion is file-as-directory in some form. > >>>Which is broken, as it forbids hard links to files. > >>No, it forbids hard links to the directory aspect of the file-directory > >>duality. > >How do you distinguish a "hard link to the directory personality" from > >"hard link to the file personality"? > Put in (undoubtedly overly) simple terms, if you can do it to a file you > can do it to the file personality, but if you currently can only do it > to a directory and get an error from attempting it to a file then in the > new scheme doing it to the hard link only gives the same error. Let me sort this out: If it can't be done POSIXly to a directory, it can't be done in Reiser4 to a file (which really is a directory too). So there can be exactly _one_ hard link to a file. Way borken. > Or, we can ask Alexander to help us use his deadlock detection algorithm > and try to do things right.... Good luck with that one. I'd suspect if it can be made to work, it will have _huge_ overhead, so much that it is useless. I'd love to be proven wrong, but I won't hold my breath. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 21:02 ` Horst von Brand @ 2004-09-07 21:29 ` Spam 2004-09-07 22:01 ` Horst von Brand 2004-09-08 1:38 ` Alexander G. M. Smith 1 sibling, 1 reply; 1392+ messages in thread From: Spam @ 2004-09-07 21:29 UTC (permalink / raw) To: Horst von Brand Cc: Hans Reiser, Christer Weinigel, David Masover, Tonnerre, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > Hans Reiser <reiser@namesys.com> said: >> Horst von Brand wrote: >> >Hans Reiser <reiser@namesys.com> said: >> >>Horst von Brand wrote: >> >>>Spam <spam@tnonline.net> said: >> >>>>Christer Weinigel <christer@weinigel.se> said: > [...] >> >>>>>2. How do we want to expose named streams? >> >>>>> One suggestion is file-as-directory in some form. >> >>>Which is broken, as it forbids hard links to files. >> >>No, it forbids hard links to the directory aspect of the file-directory >> >>duality. >> >How do you distinguish a "hard link to the directory personality" from >> >"hard link to the file personality"? >> Put in (undoubtedly overly) simple terms, if you can do it to a file you >> can do it to the file personality, but if you currently can only do it >> to a directory and get an error from attempting it to a file then in the >> new scheme doing it to the hard link only gives the same error. > Let me sort this out: If it can't be done POSIXly to a directory, it can't > be done in Reiser4 to a file (which really is a directory too). So there > can be exactly _one_ hard link to a file. Way borken. But you can make a hard link to a file in reiser4, and you can access the metadata in both. I did this test: echo "moooo" > test ln test moo chmod +x moo test echo "0700" > test/metas/rwx dir moo test -rwx------ 2 root root 10 Sep 7 23:17 moo* -rwx------ 2 root root 10 Sep 7 23:17 test* echo "0777" > test/metas/rwx dir moo test -rwxrwxrwx 2 root root 10 Sep 7 23:17 moo* -rwxrwxrwx 2 root root 10 Sep 7 23:17 test* rm moo test mkdir test ln test moo ln: `test': hard link not allowed for directory If this is the intended behaviour I do not know, but it shows that hard links works as normal. You can also do linking to meta-data: echo moo > moo ln -s moo/metas/rwx test dir -rwxrwxrwx 1 root root 0 Sep 7 23:24 moo* lrwxrwxrwx 1 root root 13 Sep 7 23:25 test -> moo/metas/rwx echo 0700 > test dir moo -rwx------ 1 root root 0 Sep 7 23:24 moo* >> Or, we can ask Alexander to help us use his deadlock detection algorithm >> and try to do things right.... > Good luck with that one. I'd suspect if it can be made to work, it will > have _huge_ overhead, so much that it is useless. I'd love to be proven > wrong, but I won't hold my breath. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 21:29 ` Spam @ 2004-09-07 22:01 ` Horst von Brand 0 siblings, 0 replies; 1392+ messages in thread From: Horst von Brand @ 2004-09-07 22:01 UTC (permalink / raw) To: Spam Cc: Hans Reiser, Christer Weinigel, David Masover, Tonnerre, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Spam <spam@tnonline.net> said: > Horst von Brand <vonbrand@inf.utfsm.cl> said: > > Hans Reiser <reiser@namesys.com> said: > >> Horst von Brand wrote: > >> >Hans Reiser <reiser@namesys.com> said: > >> >>Horst von Brand wrote: > >> >>>Spam <spam@tnonline.net> said: > >> >>>>Christer Weinigel <christer@weinigel.se> said: [...] > >> >>>>>2. How do we want to expose named streams? > >> >>>>> One suggestion is file-as-directory in some form. > >> >>>Which is broken, as it forbids hard links to files. > >> >>No, it forbids hard links to the directory aspect of the file-directory > >> >>duality. > >> >How do you distinguish a "hard link to the directory personality" from > >> >"hard link to the file personality"? > >> Put in (undoubtedly overly) simple terms, if you can do it to a file you > >> can do it to the file personality, but if you currently can only do it > >> to a directory and get an error from attempting it to a file then in the > >> new scheme doing it to the hard link only gives the same error. > > Let me sort this out: If it can't be done POSIXly to a directory, it can't > > be done in Reiser4 to a file (which really is a directory too). So there > > can be exactly _one_ hard link to a file. Way borken. > But you can make a hard link to a file in reiser4, IIUC, this isn't safe (you might tie your filesystem into a knot this way, as LWN puts it). Sure, you can do it right now (just as I was able to create hard links to directories in some propietary Unix way back; untangling the resulting mess, complete with segfaulting fsck, was a whole world of fun... it just happened rarely enough that it did not matter very much); production versions will have to forbid it (or incurr in _huge_ overheads locking everything in sight, or detecting loops). -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 21:02 ` Horst von Brand 2004-09-07 21:29 ` Spam @ 2004-09-08 1:38 ` Alexander G. M. Smith 1 sibling, 0 replies; 1392+ messages in thread From: Alexander G. M. Smith @ 2004-09-08 1:38 UTC (permalink / raw) To: Horst von Brand Cc: spam, christer, ninja, tonnerre, torvalds, pavel, jamie, cw, viro, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list, reiser Horst von Brand wrote on Tue, 07 Sep 2004 17:02:20 -0400: > Hans Reiser <reiser@namesys.com> said: > > Or, we can ask Alexander to help us use his deadlock detection algorithm > > and try to do things right.... > > Good luck with that one. I'd suspect if it can be made to work, it will > have _huge_ overhead, so much that it is useless. I'd love to be proven > wrong, but I won't hold my breath. Depends if you consider it to be a huge overhead to lock all the objects that are ancestors (possibly through cyclical multiple parents) of the fildirute (file/directory/attribute/whatever) being deleted/renamed. It could definitely be a lot if you have some weird worst case directory layout with lots of parent directories. But that's rare (most users don't have that many directories anyway). By the way, the deadlock system I was using is just a hack - a timeout on the locking semaphores for each file. If it fails to lock everything it needs, it backs off, waits a while, tries again, and eventually reports EDEADLK if it exhausts the retry count. Well, actually it's a bit fancier - it read-locks the file nodes as it is traversing the graph to find all ancestors. Then when it has finished, it write-locks the ones it needs to change. - Alex ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 15:30 ` Horst von Brand 2004-09-07 18:05 ` Hans Reiser @ 2004-09-07 22:28 ` Christer Weinigel 1 sibling, 0 replies; 1392+ messages in thread From: Christer Weinigel @ 2004-09-07 22:28 UTC (permalink / raw) To: Horst von Brand Cc: Spam, Christer Weinigel, David Masover, Tonnerre, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Horst von Brand <vonbrand@inf.utfsm.cl> writes: > > > 1. Do we want support for named streams? > > > > > I belive the answer is yes [...] > > There are many people around here who disagree (that is precisely the heart > of the discussion). I for one don't think Linux has to get $RANDOM_FEATURE > just because $SOME_OTHER_OS has got it. Either the feature stands on its > own _in the context of POSIX/Unix/Linux_ (possibly as an extension or > modification of said standards) or it isn't worth it. Ok, noted. :-) I myself am not very interested in generic named stream support in Linux. But since we have support for filesystems with named streams and people are interested in getting at those streams (if nothing else thanq to serve streams from an NTFS file system via Samba on a dual boot machine), we'd better do it well, instead of with ugly hacks. > We need to sort out exactly how far it makes sense to go, by showing > concrete, down to earth uses for whatever substructure we want. Then show > the effect can't be easily gotten through tools for power users or faking > it for unsuspecting users via GUI, and that overall the complexity and > performance cost is less than the win. Note that the success of the Unix > way is in large part due to its use of few, simple concepts that can be > combined endlessly; and tools following the same strategy. Adding extra > concepts that current tools can't naturally handle has to be considered > with extreme care. Regarding this section and all you wrote before, I definitely agree. Simplicity wins hands down every time. /Christer -- "Just how much can I get away with and still go to heaven?" Freelance consultant specializing in device driver programming for Linux Christer Weinigel <christer@weinigel.se> http://www.weinigel.se ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 12:33 ` Spam 2004-09-07 13:40 ` Christer Weinigel 2004-09-07 15:30 ` Horst von Brand @ 2004-09-07 15:37 ` Gunnar Ritter 2004-09-07 16:25 ` Spam 2004-09-07 21:39 ` Christer Weinigel 2 siblings, 2 replies; 1392+ messages in thread From: Gunnar Ritter @ 2004-09-07 15:37 UTC (permalink / raw) To: Spam, Christer Weinigel Cc: Horst von Brand, viro, Linus Torvalds, Tonnerre, ReiserFS List, Hans Reiser, Pavel Machek, David Masover, linux-kernel, linux-fsdevel, Jamie Lokier, Christoph Hellwig, Alexander Lyamin aka FLX, Chris Wedgwood Spam <spam@tnonline.net> wrote: > One suggestion is missed. It is to provide system calls for copy. > That would also solve the problem. No, it would not. If you read the POSIX.1 specification for cp carefully <http://www.unix.org/version3/online.html>, you will notice that the process for copying a regular file is carefully standardized. A POSIX.1-conforming cp implementation would not be allowed to copy additional streams, unless either additional options are given or the type of the file being copied is other than S_IFREG. And cp is just one example of a standardized file handling program. Gunnar -- http://omnibus.ruf.uni-freiburg.de/~gritter ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 15:37 ` Gunnar Ritter @ 2004-09-07 16:25 ` Spam 2004-09-07 16:55 ` Gunnar Ritter 2004-09-07 21:39 ` Christer Weinigel 1 sibling, 1 reply; 1392+ messages in thread From: Spam @ 2004-09-07 16:25 UTC (permalink / raw) To: Gunnar Ritter; +Cc: ReiserFS List, linux-kernel, linux-fsdevel > Spam <spam@tnonline.net> wrote: >> One suggestion is missed. It is to provide system calls for copy. >> That would also solve the problem. > No, it would not. If you read the POSIX.1 specification for cp > carefully <http://www.unix.org/version3/online.html>, you will > notice that the process for copying a regular file is carefully > standardized. A POSIX.1-conforming cp implementation would not > be allowed to copy additional streams, unless either additional > options are given or the type of the file being copied is other > than S_IFREG. And cp is just one example of a standardized file > handling program. It would solve the problem in Linux. However, it may not be POSIX.1 compatible. On the other hand I read that NTFS 5.0 is POSIX.1 compliant - and Windows uses copy system call. NTFS also has streams support using special character in the file names to select the streams. Surely there must be a solution in Linux that will allow things like streams and meta-data(meta-streams) be visible to the user. ~S > Gunnar ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 16:25 ` Spam @ 2004-09-07 16:55 ` Gunnar Ritter 0 siblings, 0 replies; 1392+ messages in thread From: Gunnar Ritter @ 2004-09-07 16:55 UTC (permalink / raw) To: Spam; +Cc: ReiserFS List, linux-kernel, linux-fsdevel Spam <spam@tnonline.net> wrote: > > Spam <spam@tnonline.net> wrote: > >> One suggestion is missed. It is to provide system calls for copy. > >> That would also solve the problem. > > No, it would not. If you read the POSIX.1 specification for cp > > carefully <http://www.unix.org/version3/online.html>, you will > > notice that the process for copying a regular file is carefully > > standardized. A POSIX.1-conforming cp implementation would not > > be allowed to copy additional streams, unless either additional > > options are given or the type of the file being copied is other > > than S_IFREG. And cp is just one example of a standardized file > > handling program. > It would solve the problem in Linux. However, it may not be POSIX.1 > compatible. On the other hand I read that NTFS 5.0 is POSIX.1 > compliant - and Windows uses copy system call. You should obviously take a thorough read of at least the Base Definitions, section 2, 'Conformance' of POSIX.1 before you further comment on this issue. Gunnar ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 15:37 ` Gunnar Ritter 2004-09-07 16:25 ` Spam @ 2004-09-07 21:39 ` Christer Weinigel 2004-09-07 22:13 ` Gunnar Ritter 1 sibling, 1 reply; 1392+ messages in thread From: Christer Weinigel @ 2004-09-07 21:39 UTC (permalink / raw) To: Gunnar Ritter Cc: Spam, Christer Weinigel, Horst von Brand, viro, Linus Torvalds, Tonnerre, ReiserFS List, Hans Reiser, Pavel Machek, David Masover, linux-kernel, linux-fsdevel, Jamie Lokier, Christoph Hellwig, Alexander Lyamin aka FLX, Chris Wedgwood Gunnar Ritter <Gunnar.Ritter@pluto.uni-freiburg.de> writes: > No, it would not. If you read the POSIX.1 specification for cp > carefully <http://www.unix.org/version3/online.html>, you will > notice that the process for copying a regular file is carefully > standardized. A POSIX.1-conforming cp implementation would not > be allowed to copy additional streams, unless either additional > options are given or the type of the file being copied is other > than S_IFREG. And cp is just one example of a standardized file > handling program. We can safely ignore POSIX when it is too broken. cp could very well be modified to copy named streams except when the option --posix is specified or the environment variale POSIXLY_CORRECT is set. /Christer -- "Just how much can I get away with and still go to heaven?" Freelance consultant specializing in device driver programming for Linux Christer Weinigel <christer@weinigel.se> http://www.weinigel.se ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 21:39 ` Christer Weinigel @ 2004-09-07 22:13 ` Gunnar Ritter 2004-09-07 22:44 ` Christer Weinigel 0 siblings, 1 reply; 1392+ messages in thread From: Gunnar Ritter @ 2004-09-07 22:13 UTC (permalink / raw) To: Christer Weinigel Cc: Horst von Brand, viro, Linus Torvalds, Tonnerre, Spam, ReiserFS List, Hans Reiser, Pavel Machek, David Masover, linux-kernel, linux-fsdevel, Jamie Lokier, Christoph Hellwig, Alexander Lyamin aka FLX, Chris Wedgwood, Christer Weinigel Christer Weinigel <christer@weinigel.se> wrote: > Gunnar Ritter <Gunnar.Ritter@pluto.uni-freiburg.de> writes: > > No, it would not. If you read the POSIX.1 specification for cp > > carefully <http://www.unix.org/version3/online.html>, you will > > notice that the process for copying a regular file is carefully > > standardized. A POSIX.1-conforming cp implementation would not > > be allowed to copy additional streams, unless either additional > > options are given or the type of the file being copied is other > > than S_IFREG. And cp is just one example of a standardized file > > handling program. > We can safely ignore POSIX when it is too broken. Excuse me, but there's really nothing broken here with POSIX and cp. You're just making an insulting talk about a part of the specification which currently serves GNU/Linux and other Unix-like environments very well, and has done so for about twelve years now. > cp could very well be modified to copy named streams except when > the option --posix is specified Hey, you didn't ever even have a look at POSIX Shell & Utilities, did you? Then why are you making derogatory statements about it? > or the environment variale POSIXLY_CORRECT is set. Cool, data loss depending upon an environment variable which is even currently used by many programs unaware of such results. This really sounds like good engineering to me. Gunnar ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 22:13 ` Gunnar Ritter @ 2004-09-07 22:44 ` Christer Weinigel 2004-09-07 23:45 ` Gunnar Ritter 0 siblings, 1 reply; 1392+ messages in thread From: Christer Weinigel @ 2004-09-07 22:44 UTC (permalink / raw) To: Gunnar Ritter Cc: Christer Weinigel, Horst von Brand, viro, Linus Torvalds, Tonnerre, Spam, ReiserFS List, Hans Reiser, Pavel Machek, David Masover, linux-kernel, linux-fsdevel, Jamie Lokier, Christoph Hellwig, Alexander Lyamin aka FLX, Chris Wedgwood Gunnar Ritter <Gunnar.Ritter@pluto.uni-freiburg.de> writes: > Excuse me, but there's really nothing broken here with POSIX and cp. > You're just making an insulting talk about a part of the specification > which currently serves GNU/Linux and other Unix-like environments very > well, and has done so for about twelve years now. "Broken" in the sense "POSIX mandates something that users wouldn't expect". > > cp could very well be modified to copy named streams except when > > the option --posix is specified > > Hey, you didn't ever even have a look at POSIX Shell & Utilities, did > you? Then why are you making derogatory statements about it? Those derogatory statments are really all in your mind. > > or the environment variale POSIXLY_CORRECT is set. > > Cool, data loss depending upon an environment variable which is even > currently used by many programs unaware of such results. This really > sounds like good engineering to me. How would you consider cp to cause "data loss" if it _besides_ copying the normal stream _also_ copied any named streams or xattrs belonging to the stream? How would it cause data loss if cp started using a theoretical copyfile syscall? It may not be 100% according to POSIX, but I'd definitely say that it does what the user expects. Lots of GNU utilities already differ from POSIX mandated behaviour because the authors of those utilities belive that the POSIX mandated behaviour is confusing. http://www.wlug.org.nz/POSIXLY_CORRECT POSIXLY_CORRECT is an environment variable that some programs use to follow strict POSIX standards behaviour, where that isn't the default. Probably the most well-known example of this is that POSIX states that filesystem blocks are 512 bytes per block, so the GNU fileutils such as df(1) and GNU tar(1) use 512 if the variable POSIXLY_CORRECT is set, and 1024 bytes per block by default. Many of the GNU tools comply with POSIX by default, except for where the author thinks the POSIX standard is wrong or dumb. :) As a result, some programs also check if a variable named POSIX_ME_HARDER is set as an acceptable alias for POSIXLY_CORRECT. See Democracy Triumphs in Disk Units. /Christer -- "Just how much can I get away with and still go to heaven?" Freelance consultant specializing in device driver programming for Linux Christer Weinigel <christer@weinigel.se> http://www.weinigel.se ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 22:44 ` Christer Weinigel @ 2004-09-07 23:45 ` Gunnar Ritter 2004-09-08 5:48 ` Christer Weinigel 0 siblings, 1 reply; 1392+ messages in thread From: Gunnar Ritter @ 2004-09-07 23:45 UTC (permalink / raw) To: Christer Weinigel Cc: Horst von Brand, viro, Linus Torvalds, Tonnerre, Spam, ReiserFS List, Hans Reiser, Pavel Machek, David Masover, linux-kernel, linux-fsdevel, Jamie Lokier, Christoph Hellwig, Alexander Lyamin aka FLX, Chris Wedgwood, Christer Weinigel Christer Weinigel <christer@weinigel.se> wrote: > Gunnar Ritter <Gunnar.Ritter@pluto.uni-freiburg.de> writes: > > Excuse me, but there's really nothing broken here with POSIX and cp. > > You're just making an insulting talk about a part of the specification > > which currently serves GNU/Linux and other Unix-like environments very > > well, and has done so for about twelve years now. > "Broken" in the sense "POSIX mandates something that users wouldn't > expect". Only if one breaks it by making extensions in an inappropriate way. This is not a fault of POSIX. POSIX usually allows a lot of sane ways to introduce extensions. There are usually valid interoperability arguments for behavior prescribed by POSIX. It is really not one of those standards where you want to ignore every second word because it is obviously nothing but committee nonsense. > > > or the environment variale POSIXLY_CORRECT is set. > > Cool, data loss depending upon an environment variable which is even > > currently used by many programs unaware of such results. This really > > sounds like good engineering to me. > How would you consider cp to cause "data loss" if it _besides_ copying > the normal stream _also_ copied any named streams or xattrs belonging > to the stream? You are reversing the argument. If additional streams are introduced inappropriately by extending the semantics of S_IFREG files, POSIX requires cp to lose the data. Your proposal would then make this loss of additional stream data dependent on an environment variable that is already in wide use. If it was set by accident, the data would be lost. Besides, copying xattrs is usually permitted (POSIX.1-2004, XCU cp): # If the implementation provides additional or alternate access control # mechanisms (see the Base Definitions volume of IEEE Std 1003.1-2001, # Section 4.4, File Access Permissions), their effect on copies of files # is implementation-defined. It is also permitted to add other S_IFXXX types and then let cp act in an implementation-defined manner on them (cf. my earlier message <413E40D1.nailFBI11XFML@pluto.uni-freiburg.de>). The 'standardized' data loss would only occur if the standardized type of regular file, S_IFREG, was abused. This would really not be a fault of POSIX. > Lots of GNU utilities already differ from POSIX mandated behaviour > because the authors of those utilities belive that the POSIX mandated > behaviour is confusing. Sure, but it is not the preferred method of adding features. In addition, most of the existing POSIXLY_CORRECT influences are nothing but cosmetical details in comparison to copying/not copying stream data. Gunnar ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 23:45 ` Gunnar Ritter @ 2004-09-08 5:48 ` Christer Weinigel 2004-09-08 6:29 ` Gunnar Ritter 0 siblings, 1 reply; 1392+ messages in thread From: Christer Weinigel @ 2004-09-08 5:48 UTC (permalink / raw) To: Gunnar Ritter Cc: Christer Weinigel, Horst von Brand, viro, Linus Torvalds, Tonnerre, Spam, ReiserFS List, Hans Reiser, Pavel Machek, David Masover, linux-kernel, linux-fsdevel, Jamie Lokier, Christoph Hellwig, Alexander Lyamin aka FLX, Chris Wedgwood Gunnar Ritter <Gunnar.Ritter@pluto.uni-freiburg.de> writes: > Besides, copying xattrs is usually permitted (POSIX.1-2004, XCU cp): > > # If the implementation provides additional or alternate access control > # mechanisms (see the Base Definitions volume of IEEE Std 1003.1-2001, > # Section 4.4, File Access Permissions), their effect on copies of files > # is implementation-defined. In <413DD5B4.nailC801GI4E2@pluto.uni-freiburg.de> you wrote: >A POSIX.1-conforming cp implementation would not be allowed to copy >additional streams, unless either additional options are given or the >type of the file being copied is other than S_IFREG. I read this as that POSIX mandates that cp can absolutely not copy anything else but the file contents. That is what I called broken. If we implement named streams as xattrs and that can be accessed with openat(..., O_XATTR) this means that cp is allowed to copy the xattrs (well, named streans don't neccesarily have to be "alternate access control mechanisms", but they can use the same xattr namespace). That's quite ok in that case. /Christer -- "Just how much can I get away with and still go to heaven?" Freelance consultant specializing in device driver programming for Linux Christer Weinigel <christer@weinigel.se> http://www.weinigel.se ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-08 5:48 ` Christer Weinigel @ 2004-09-08 6:29 ` Gunnar Ritter 0 siblings, 0 replies; 1392+ messages in thread From: Gunnar Ritter @ 2004-09-08 6:29 UTC (permalink / raw) To: Christer Weinigel Cc: Horst von Brand, viro, Linus Torvalds, Tonnerre, Spam, ReiserFS List, Hans Reiser, Pavel Machek, David Masover, linux-kernel, linux-fsdevel, Jamie Lokier, Christoph Hellwig, Alexander Lyamin aka FLX, Chris Wedgwood, Christer Weinigel Christer Weinigel <christer@weinigel.se> wrote: > Gunnar Ritter <Gunnar.Ritter@pluto.uni-freiburg.de> writes: > > > Besides, copying xattrs is usually permitted (POSIX.1-2004, XCU cp): > > > > # If the implementation provides additional or alternate access control > > # mechanisms (see the Base Definitions volume of IEEE Std 1003.1-2001, > > # Section 4.4, File Access Permissions), their effect on copies of files > > # is implementation-defined. > > In <413DD5B4.nailC801GI4E2@pluto.uni-freiburg.de> you wrote: > > >A POSIX.1-conforming cp implementation would not be allowed to copy > >additional streams, unless either additional options are given or the > >type of the file being copied is other than S_IFREG. > > I read this as that POSIX mandates that cp can absolutely not copy > anything else but the file contents. That is what I called broken. It would be really helpful if you read the specification before you comment on it or try to interpret my wording further. > If we implement named streams as xattrs and that can be accessed with > openat(..., O_XATTR) this means that cp is allowed to copy the xattrs No. Not if the file type of such beasts remains S_IFREG. > (well, named streans don't neccesarily have to be "alternate access > control mechanisms", but they can use the same xattr namespace). POSIX does not know anything about the 'xattr namespace'. It just allows 'additional or alternate access control mechanisms'. Which are, in turn, well-defined terms in the standard again. You can read about them too <http://www.unix.org/version3/>. Gunnar ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 11:55 ` Christer Weinigel ` (2 preceding siblings ...) 2004-09-07 12:33 ` Spam @ 2004-09-07 18:34 ` Hans Reiser 2004-09-07 22:13 ` Christer Weinigel 3 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-09-07 18:34 UTC (permalink / raw) To: Christer Weinigel Cc: David Masover, Horst von Brand, Spam, Tonnerre, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Christer Weinigel wrote: >David Masover <ninja@slaphack.com> writes: > > > >>|>Second, there are quite a few things which I might want to do, which can >>|>be done with this interface and without patching programs, >>| Such as? >>They've been mentioned. >> >> > > > >>| Haven't seen any that made sense to me, sorry. >>Sorry if they don't make sense to you, but I don't feel like discussing >>them now. Either you get it or you don't, either you agree or you >>don't. Read the archives. >> >> > >Great argument. Not. There has been so much shit thrown around here >so that it's impossible to keep track of all examples. > >Could you please try summarize a few of the arguments that you find >especially compelling? This thread has gotten very confused since >there are a bunch of different subjects all being intermixed here. > >What are we discussing? > >1. Do we want support for named streams? > > I belive the answer is yes, since both NTFS and HFS (that's the > MacOS filesystem, isn't it?) supports streams we want Linux to > support this if possible. > > Anyone disagreeing? > > No, we want files and directories that can do what streams can do. This means files that are also directories, plugins that aggregate the contents of a directory, files that inherit stat data, maybe I forget something ---- it is on my website. Streams themselves are a bad idea because they are a filesystem inside of a file which is double the overhead, and they fragment the namespace. >4. What belongs in the generic VFS, what belongs in Reiser4? > > Some things reiser4 do, such as files-as-directories need changes > to the VFS because it breaks assumptions that the VFS makes > (i.e. a deadlock or an oops when doing a hard link out of one). > > Some other things reiser4 can do would be better if they were in > the VFS since other filesystems might want to support the same > functionality. > > It is always better to lead by example. These ideas are too new for the other fs developers, they need 5 years to get used to them. Reiser4 should create something that works, and let others follow when they will. > Or Linux may not support some of the things reiserfs at all. > >5. What belongs in the kernel, what belongs in userspace? > > This is the wrong question. The right question is, what belongs in a unified namespace? Then having answered that, the extent to which it is easier to have all aspects of name resolution together in one body of code is an implementation detail that can change and evolve with time. A rapidly evolving namespace is easier to modify if it is all one body of code. Furthermore, the people doing the work should really be left to decide such implementation details themselves because they are more expert on their code than anyone else. Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 18:34 ` Hans Reiser @ 2004-09-07 22:13 ` Christer Weinigel 2004-09-07 22:36 ` David Lang 0 siblings, 1 reply; 1392+ messages in thread From: Christer Weinigel @ 2004-09-07 22:13 UTC (permalink / raw) To: Hans Reiser Cc: Christer Weinigel, David Masover, Horst von Brand, Spam, Tonnerre, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hans Reiser <reiser@namesys.com> writes: > No, we want files and directories that can do what streams can do. > This means files that are also directories, plugins that aggregate the > contents of a directory, files that inherit stat data, maybe I forget > something ---- it is on my website. > > Streams themselves are a bad idea because they are a filesystem inside > of a file which is double the overhead, and they fragment the > namespace. There really isn't a difference between your files are directories think and named streams that can be accessed with openat. Both are concepts that won't fit into the classical Unix entity "file". As far as I can tell, windows named streams could be exposed via a file-as-directory model. In that case what windows sees as foo.txt:icon would be accessed as foo.txt/icon instead. The question in my mind is if that is a desirable concept. I can see that it is very a attractive concept because it's easy to explain to users, but at the same time, a lot of things suggest that the files-as- directories concept is a bad idea. I mentioned a few of the possible problems in the grand-grand-parent of this message. Try to address those problems. It may be that your answer is "I don't give a shit about existing backup applications", or "I don't care about security in existing web-servers, they'll have to be taught about reiser4", but in that case say so. Or show how it's not a problem with existing applictions. > It is always better to lead by example. These ideas are too new for > the other fs developers, they need 5 years to get used to them. > Reiser4 should create something that works, and let others follow when > they will. Pretty please, give up the marketing bullshit. I already know that you belive reiser4 is the best thing since sliced bread. You say that "reiser4 should create something that works". So far you haven't convinced me that reiser4 actually "works". It's a nice proof of concept, but it definitely does not feel like something that I'd like in the mainstream kernel until those problems are addressed. Have you looked at the VFS deadlocks that Alexander Viro pointed at? Does tar work with files-as-directories? Can I use tar to make a backup of my resier4 directory. What changes would be required to the tar application or to the tar file format to make this work? Can I do a cp -a /home/wingel to a NFS-mounted drive on some other computer to make a backup of my home directory or will I lose data that way? What changes would be required to cp to make this work? Please remember that reiser4 does not exist in a vaccum. If you want to make a fundamental change to unix concepts that have been with us for three decades, it's really up to you to show that it doesn't break too many existing applications. > This is the wrong question. The right question is, what belongs in a > unified namespace? Then having answered that, the extent to which it > is easier to have all aspects of name resolution together in one body > of code is an implementation detail that can change and evolve with > time. A rapidly evolving namespace is easier to modify if it is all > one body of code. Furthermore, the people doing the work should > really be left to decide such implementation details themselves > because they are more expert on their code than anyone else. I still haven't seen a good answer to the namespace problem. You say that files-as-directories is great, I'm definitely not convinced. The VFS is the body of code that handles most aspects of name resolution today, so it sounds as if reiser4 is supposed to be the new super-VFS, and you want to keep it that way and do not want to integrate with the current VFS. In that case resier4 really does not belong in the mainstream kernel since it's your pet research project. /Christer -- "Just how much can I get away with and still go to heaven?" Freelance consultant specializing in device driver programming for Linux Christer Weinigel <christer@weinigel.se> http://www.weinigel.se ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 22:13 ` Christer Weinigel @ 2004-09-07 22:36 ` David Lang 2004-09-07 22:47 ` Christer Weinigel ` (2 more replies) 0 siblings, 3 replies; 1392+ messages in thread From: David Lang @ 2004-09-07 22:36 UTC (permalink / raw) To: Christer Weinigel Cc: Hans Reiser, David Masover, Horst von Brand, Spam, Tonnerre, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List so far the best answer that I've seen is a slight varient of what Hans is proposing for the 'file-as-a-directory' make the base file itself be a serialized version of all the streams and if you want the 'main' stream open file/. (or some similar varient) this doesn't address the hard-link issue, but it should handle the backup problems (your backup software just goes through the files and what it gets is suitable for backups). you will ask what serializer to user, and my answer is to let one of the streams tell you, and have the kernel make a call out to userspace to execute the appropriate program (note that this means that tar is not put into the kernel) in fact it may make sense to just open file/file to get at the 'main' stream of the file (there may be cases where the concept of a single main stream may not make sense) so if this solves the tool/backup problem then we can look and figure out if there's a reasonable way to solve the hard-link problem David Lang On Tue, 8 Sep 2004, Christer Weinigel wrote: > Date: 08 Sep 2004 00:13:12 +0200 > From: Christer Weinigel <christer@weinigel.se> > To: Hans Reiser <reiser@namesys.com> > Cc: Christer Weinigel <christer@weinigel.se>, > David Masover <ninja@slaphack.com>, > Horst von Brand <vonbrand@inf.utfsm.cl>, Spam <spam@tnonline.net>, > Tonnerre <tonnerre@thundrix.ch>, Linus Torvalds <torvalds@osdl.org>, > Pavel Machek <pavel@ucw.cz>, Jamie Lokier <jamie@shareable.org>, > Chris Wedgwood <cw@f00f.org>, viro@parcelfarce.linux.theplanet.co.uk, > Christoph Hellwig <hch@lst.de>, linux-fsdevel@vger.kernel.org, > linux-kernel@vger.kernel.org, Alexander Lyamin aka FLX <flx@namesys.com>, > ReiserFS List <reiserfs-list@namesys.com> > Subject: Re: silent semantic changes with reiser4 > > Hans Reiser <reiser@namesys.com> writes: > >> No, we want files and directories that can do what streams can do. >> This means files that are also directories, plugins that aggregate the >> contents of a directory, files that inherit stat data, maybe I forget >> something ---- it is on my website. >> >> Streams themselves are a bad idea because they are a filesystem inside >> of a file which is double the overhead, and they fragment the >> namespace. > > There really isn't a difference between your files are directories > think and named streams that can be accessed with openat. Both are > concepts that won't fit into the classical Unix entity "file". > > As far as I can tell, windows named streams could be exposed via a > file-as-directory model. In that case what windows sees as > foo.txt:icon would be accessed as foo.txt/icon instead. The question > in my mind is if that is a desirable concept. I can see that it is > very a attractive concept because it's easy to explain to users, but > at the same time, a lot of things suggest that the files-as- > directories concept is a bad idea. I mentioned a few of the possible > problems in the grand-grand-parent of this message. > > Try to address those problems. It may be that your answer is "I don't > give a shit about existing backup applications", or "I don't care > about security in existing web-servers, they'll have to be taught > about reiser4", but in that case say so. Or show how it's not a > problem with existing applictions. > >> It is always better to lead by example. These ideas are too new for >> the other fs developers, they need 5 years to get used to them. >> Reiser4 should create something that works, and let others follow when >> they will. > > Pretty please, give up the marketing bullshit. I already know that > you belive reiser4 is the best thing since sliced bread. You say that > "reiser4 should create something that works". So far you haven't > convinced me that reiser4 actually "works". It's a nice proof of > concept, but it definitely does not feel like something that I'd like > in the mainstream kernel until those problems are addressed. > > Have you looked at the VFS deadlocks that Alexander Viro pointed at? > > Does tar work with files-as-directories? Can I use tar to make a > backup of my resier4 directory. What changes would be required to the > tar application or to the tar file format to make this work? > > Can I do a cp -a /home/wingel to a NFS-mounted drive on some other > computer to make a backup of my home directory or will I lose data > that way? What changes would be required to cp to make this work? > > Please remember that reiser4 does not exist in a vaccum. If you want > to make a fundamental change to unix concepts that have been with us > for three decades, it's really up to you to show that it doesn't break > too many existing applications. > >> This is the wrong question. The right question is, what belongs in a >> unified namespace? Then having answered that, the extent to which it >> is easier to have all aspects of name resolution together in one body >> of code is an implementation detail that can change and evolve with >> time. A rapidly evolving namespace is easier to modify if it is all >> one body of code. Furthermore, the people doing the work should >> really be left to decide such implementation details themselves >> because they are more expert on their code than anyone else. > > I still haven't seen a good answer to the namespace problem. You say > that files-as-directories is great, I'm definitely not convinced. > > The VFS is the body of code that handles most aspects of name > resolution today, so it sounds as if reiser4 is supposed to be the new > super-VFS, and you want to keep it that way and do not want to > integrate with the current VFS. In that case resier4 really does not > belong in the mainstream kernel since it's your pet research project. > > /Christer > > -- > "Just how much can I get away with and still go to heaven?" > > Freelance consultant specializing in device driver programming for Linux > Christer Weinigel <christer@weinigel.se> http://www.weinigel.se > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 22:36 ` David Lang @ 2004-09-07 22:47 ` Christer Weinigel 2004-09-07 22:56 ` David Lang 2004-09-07 23:14 ` Gunnar Ritter 2004-09-08 0:07 ` Horst von Brand 2 siblings, 1 reply; 1392+ messages in thread From: Christer Weinigel @ 2004-09-07 22:47 UTC (permalink / raw) To: David Lang Cc: Christer Weinigel, Hans Reiser, David Masover, Horst von Brand, Spam, Tonnerre, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List David Lang <david.lang@digitalinsight.com> writes: > so far the best answer that I've seen is a slight varient of what Hans > is proposing for the 'file-as-a-directory' > > make the base file itself be a serialized version of all the streams > and if you want the 'main' stream open file/. (or some similar > varient) > in fact it may make sense to just open file/file to get at the 'main' > stream of the file (there may be cases where the concept of a single > main stream may not make sense) So what happens if I have a text file foo.txt and add an author attribute to it? When I read foo.txt the next time it's supposed to give me a serialized version with both the contents of foo.txt _and_ the author attribute? That would definitely confuse me. Or did I misunderstand something? /Christer -- "Just how much can I get away with and still go to heaven?" Freelance consultant specializing in device driver programming for Linux Christer Weinigel <christer@weinigel.se> http://www.weinigel.se ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 22:47 ` Christer Weinigel @ 2004-09-07 22:56 ` David Lang 2004-09-08 0:15 ` Horst von Brand 0 siblings, 1 reply; 1392+ messages in thread From: David Lang @ 2004-09-07 22:56 UTC (permalink / raw) To: Christer Weinigel Cc: Hans Reiser, David Masover, Horst von Brand, Spam, Tonnerre, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Tue, 8 Sep 2004, Christer Weinigel wrote: > Subject: Re: silent semantic changes with reiser4 > > David Lang <david.lang@digitalinsight.com> writes: > >> so far the best answer that I've seen is a slight varient of what Hans >> is proposing for the 'file-as-a-directory' >> >> make the base file itself be a serialized version of all the streams >> and if you want the 'main' stream open file/. (or some similar >> varient) > >> in fact it may make sense to just open file/file to get at the 'main' >> stream of the file (there may be cases where the concept of a single >> main stream may not make sense) > > So what happens if I have a text file foo.txt and add an author > attribute to it? When I read foo.txt the next time it's supposed to > give me a serialized version with both the contents of foo.txt _and_ > the author attribute? > > That would definitely confuse me. > > Or did I misunderstand something? > good point. under my scheme you would need to access foo.txt/foo.txt or foo.txt/. instead of just foo.txt I guess my way would work if there is a way to know that a file has been extended (or if you just make it a habit of opening the file/file instead of just file) but not for random additions of streams to otherwise normal files. Oh well, it seemed like a easy fix (and turned out to be to easy to be practical) David Lang > /Christer > > -- > "Just how much can I get away with and still go to heaven?" > > Freelance consultant specializing in device driver programming for Linux > Christer Weinigel <christer@weinigel.se> http://www.weinigel.se > -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 22:56 ` David Lang @ 2004-09-08 0:15 ` Horst von Brand 0 siblings, 0 replies; 1392+ messages in thread From: Horst von Brand @ 2004-09-08 0:15 UTC (permalink / raw) To: David Lang Cc: Christer Weinigel, Hans Reiser, David Masover, Spam, Tonnerre, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List David Lang <david.lang@digitalinsight.com> said: > On Tue, 8 Sep 2004, Christer Weinigel wrote: > > David Lang <david.lang@digitalinsight.com> writes: [...] > > So what happens if I have a text file foo.txt and add an author > > attribute to it? When I read foo.txt the next time it's supposed to > > give me a serialized version with both the contents of foo.txt _and_ > > the author attribute? > > > > That would definitely confuse me. > > > > Or did I misunderstand something? > good point. under my scheme you would need to access foo.txt/foo.txt or > foo.txt/. instead of just foo.txt > I guess my way would work if there is a way to know that a file has been > extended Out of the question if you want to use legacy tools. > (or if you just make it a habit of opening the file/file instead > of just file) but not for random additions of streams to otherwise normal > files. And then a file called foo inside directory foo will have to be refered to as foo/foo/foo... > Oh well, it seemed like a easy fix (and turned out to be to easy to be > practical) My gut feeling is that this whole concept is too messed up to be sorted out. That something can easily be done given whatever underlying structure there is doesn't make it worthwhile to do it. Just as it is absurdly easy to allow hard links to directories if you look at how they are implemented, but the consecuences aren't that easy to live with. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 22:36 ` David Lang 2004-09-07 22:47 ` Christer Weinigel @ 2004-09-07 23:14 ` Gunnar Ritter 2004-09-08 1:29 ` Julian Blake Kongslie 2004-09-08 0:07 ` Horst von Brand 2 siblings, 1 reply; 1392+ messages in thread From: Gunnar Ritter @ 2004-09-07 23:14 UTC (permalink / raw) To: David Lang, Christer Weinigel Cc: Horst von Brand, viro, Linus Torvalds, Tonnerre, Spam, ReiserFS List, Hans Reiser, Pavel Machek, David Masover, linux-kernel, linux-fsdevel, Jamie Lokier, Christoph Hellwig, Alexander Lyamin aka FLX, Chris Wedgwood David Lang <david.lang@digitalinsight.com> wrote: > so far the best answer that I've seen is a slight varient of what Hans is > proposing for the 'file-as-a-directory' > > make the base file itself be a serialized version of all the streams and > if you want the 'main' stream open file/. (or some similar varient) As has been said previously, all such proposals except for those with two leading slashes would directly violate POSIX.1-2004, Base Definitions, 4.11 'Pathname Resolution'. In particular, # A pathname that contains at least one non-slash character and that # ends with one or more trailing slashes shall be resolved as if a # single dot character ( '.' ) were appended to the pathname. A regular file name with special semantics, in contrast, would not violate POSIX, in particular if a stat() on it would not return one of the standard S_IFXXX types. Inappropriate operations could just fail with EINVAL on such files, and it would be unspecified by the standard how they were handled by 'cp' or other standardized utilities. Having a separate S_IFXXX type for streamed files would also give attention to portability issues easily. It might in fact be better to have existing applications fail explicitly with them than to make this as transparent as possible, but hope for good luck. Utilities like 'tar' would just say 'Unknown file type' or the like and exclude such files from the archives they create. This would lead the user's explicit attention to the data loss. He might then choose to ignore the error, or to get a special version of 'tar' to handle it. It would also be clear to programmers and users that such files are special things they don't normally need. Regular Linux installations might totally ignore them. If streamed files are only intended for use with CIFS or other special cases, that might be a more clean solution for this issue than watering down the existing portable file semantics. Gunnar ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 23:14 ` Gunnar Ritter @ 2004-09-08 1:29 ` Julian Blake Kongslie 0 siblings, 0 replies; 1392+ messages in thread From: Julian Blake Kongslie @ 2004-09-08 1:29 UTC (permalink / raw) To: Gunnar Ritter Cc: David Lang, Christer Weinigel, Horst von Brand, viro, Linus Torvalds, Tonnerre, Spam, ReiserFS List, Hans Reiser, Pavel Machek, David Masover, linux-kernel, linux-fsdevel, Jamie Lokier, Christoph Hellwig, Alexander Lyamin aka FLX, Chris Wedgwood [-- Attachment #1: Type: text/plain, Size: 1597 bytes --] On Wed, 08 Sep 2004 01:14:25 +0200, Gunnar Ritter <Gunnar.Ritter@pluto.uni-freiburg.de> wrote: > Having a separate S_IFXXX type for streamed files would also give > attention to portability issues easily. It might in fact be better > to have existing applications fail explicitly with them than to > make this as transparent as possible, but hope for good luck. > > Utilities like 'tar' would just say 'Unknown file type' or the like > and exclude such files from the archives they create. This would lead > the user's explicit attention to the data loss. He might then choose > to ignore the error, or to get a special version of 'tar' to handle it. > > It would also be clear to programmers and users that such files are > special things they don't normally need. Regular Linux installations > might totally ignore them. If streamed files are only intended for > use with CIFS or other special cases, that might be a more clean > solution for this issue than watering down the existing portable > file semantics. You could actually use two S_IFXXX types - one for stream files which should be archived by backup programs because they contain real information, and a different one for stream files which should not be archived because they are merely a different "view" of data from the main file. This simplifies the archival problem because legacy programs would complain that the file types aren't handled - alerting the user to potential data loss, and updated programs would automatically backup only the attributes that are required. -- -Julian Blake Kongslie <jblake@omgwallhack.org> [-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 22:36 ` David Lang 2004-09-07 22:47 ` Christer Weinigel 2004-09-07 23:14 ` Gunnar Ritter @ 2004-09-08 0:07 ` Horst von Brand 2004-09-08 6:51 ` David Lang 2 siblings, 1 reply; 1392+ messages in thread From: Horst von Brand @ 2004-09-08 0:07 UTC (permalink / raw) To: David Lang Cc: Christer Weinigel, Hans Reiser, David Masover, Horst von Brand, Spam, Tonnerre, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List David Lang <david.lang@digitalinsight.com> said: > so far the best answer that I've seen is a slight varient of what Hans is > proposing for the 'file-as-a-directory' > make the base file itself be a serialized version of all the streams and > if you want the 'main' stream open file/. (or some similar varient) Serialized how? cpio(1), tar(1), cat(1)ed together, ...? Ship it via FTP or something, how do you unpack on a traditional filesystem? On a wacky filesystem? Note that this has the same performance implications as the targzfilesystem concept, just the other way around: Instead of unpacking on the fly it is packing on the fly. And paying a (possibly huge) cost for "simple, everyday operations" in some cases that can't be distinguished from ones where there is no such cost is very wrong to me. > this doesn't address the hard-link issue, It has to be solved! > but it should handle the backup > problems (your backup software just goes through the files and what it > gets is suitable for backups). But how do you recreate the original "files" later? You only get the serialized version back. Even worse, if you unpack on top of the original wacky file, you'd get a wacky file, where the main stream is the serialized version of the original... Rinse and repeat. [Shudder] > you will ask what serializer to user, and my answer is to let one of the > streams tell you, and have the kernel make a call out to userspace to > execute the appropriate program (note that this means that tar is not put > into the kernel) If I want to see it serialized via tar(1), and you via cpio(1), and somebody else via "take the icon stream, discard everything else", how do you handle that? > in fact it may make sense to just open file/file to get at the 'main' > stream of the file (there may be cases where the concept of a single main > stream may not make sense) What do you do then? > so if this solves the tool/backup problem It makes it much worse, AFAICS. > then we can look and figure out > if there's a reasonable way to solve the hard-link problem Right. And if none is found, can we drop this madness? -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-08 0:07 ` Horst von Brand @ 2004-09-08 6:51 ` David Lang 0 siblings, 0 replies; 1392+ messages in thread From: David Lang @ 2004-09-08 6:51 UTC (permalink / raw) To: Horst von Brand Cc: Christer Weinigel, Hans Reiser, David Masover, Spam, Tonnerre, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Tue, 7 Sep 2004, Horst von Brand wrote: > David Lang <david.lang@digitalinsight.com> said: >> so far the best answer that I've seen is a slight varient of what Hans is >> proposing for the 'file-as-a-directory' > >> make the base file itself be a serialized version of all the streams and >> if you want the 'main' stream open file/. (or some similar varient) > > Serialized how? cpio(1), tar(1), cat(1)ed together, ...? Ship it via FTP or > something, how do you unpack on a traditional filesystem? On a wacky > filesystem? > as I said elsewhere in the origional post the owner of the file sets the tool to use to serialize it. if you want tar set it to use tar, if you want cpio set that. in both cases it would be a call out to userspace > Note that this has the same performance implications as the targzfilesystem > concept, just the other way around: Instead of unpacking on the fly it is > packing on the fly. And paying a (possibly huge) cost for "simple, everyday > operations" in some cases that can't be distinguished from ones where there > is no such cost is very wrong to me. this is a (huge) oppurtunity for the filesystem to provide for an optimization. if it is able to maintain a serialized version then this could turn into a simple read. no it's not always the simple obvious thing, but very few things are. the problem of things having unexpected side effects in an uncontrolled server already are out there. >> this doesn't address the hard-link issue, > > It has to be solved! I never disputed that, in fact I pointed out that that issue remained, with the idea (since disproved) that this suggestion would settle the other issues sufficiantly to let everyone concentrate on the link issue. > >> but it should handle the backup >> problems (your backup software just goes through the files and what it >> gets is suitable for backups). > > But how do you recreate the original "files" later? You only get the > serialized version back. Even worse, if you unpack on top of the original > wacky file, you'd get a wacky file, where the main stream is the serialized > version of the original... Rinse and repeat. [Shudder] how smart if the filesystem plugin? can the filesystem recognise compound filetypes? if so then it can tell what's happening and do the right thing when recreating the file >> you will ask what serializer to user, and my answer is to let one of the >> streams tell you, and have the kernel make a call out to userspace to >> execute the appropriate program (note that this means that tar is not put >> into the kernel) > > If I want to see it serialized via tar(1), and you via cpio(1), and > somebody else via "take the icon stream, discard everything else", how do > you handle that? the owner of the file decides (or whoever that owner grants the permissions to to change it to a different system) >> in fact it may make sense to just open file/file to get at the 'main' >> stream of the file (there may be cases where the concept of a single main >> stream may not make sense) > > What do you do then? the backup doesn't really care. if you refrence files by file/file then you can't really say that file/foo is any more the 'main' stream then file/bar. but if you call it file/. then you do have to define what themain version is. >> so if this solves the tool/backup problem > > It makes it much worse, AFAICS. >> then we can look and figure out >> if there's a reasonable way to solve the hard-link problem > > Right. And if none is found, can we drop this madness? solutions are known , it's just not known if any solution can be implemented with suitable performance (lock the entire filesystem when you go to do a dangerous action goes a _long_ way towards solving the problems, but it's not acceptable due to performance issues) the fundamental problem with hard links and directories (maintaining a sane path) is the exact same problem that the *notify tools need solved so that a program can setup notification on a directory tree that it's interested in without tieing up a large number of file handles. it's a odd problem, but if it can be solved there are several doors that open up. David Lang -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 5:50 ` David Masover 2004-09-07 6:02 ` Hans Reiser 2004-09-07 11:55 ` Christer Weinigel @ 2004-09-08 6:20 ` Tonnerre 2 siblings, 0 replies; 1392+ messages in thread From: Tonnerre @ 2004-09-08 6:20 UTC (permalink / raw) To: David Masover Cc: Horst von Brand, Spam, Christer Weinigel, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 593 bytes --] Salut, On Tue, Sep 07, 2004 at 12:50:16AM -0500, David Masover wrote: > Transparently? > > It _works_ to do > zcat file.gz > /tmp/file > vim /tmp/file > gzip -c /tmp/file > file.gz > rm /tmp/file > You can even do that as a script, call it zvim. You could even do it as > a generic script, where "vim" is replaced with "$1". But is it as > elegent as transparently compressed files? ...or you get yourself a sane editor which supports gzopen/gzread/gzwrite/gzclose. You can do that in userland, no kernel implementation needed. Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-06 18:53 ` David Masover 2004-09-06 22:37 ` Christer Weinigel 2004-09-07 2:06 ` Horst von Brand @ 2004-09-07 6:32 ` Hans Reiser 2 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-09-07 6:32 UTC (permalink / raw) To: David Masover Cc: Horst von Brand, Spam, Tonnerre, Christer Weinigel, Linus Torvalds, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List David Masover wrote: > > There are some things which can't be solved without patching. Version > control is one such thing. You can do most of the version control stuff without patching, for instance selecting a version by name is easy (see Clearcase). Clearcase users mostly do not patch user space binaries. But your point is generally true that there are features that knowing about them allows you to better employ them. > But then there can be more generic patches > -- as soon as the transaction API is done, you only have to patch apps > to use that, and have a version control reiser4 plugin. > > | I'd go the other way around: Get userspace to agree on a common > framework, > | make it work in userspace; if (extensive, hopefully) experience > shows that > | a pure userspace solution has issues that can't be solved except by > kernel > | assistance, so be it. > > We already have such a framework -- it's called "VFS". If doing it in the kernel is so hard, why hasn't it stopped us yet? ;-) I am not asking other people to contibute their labor to making this thing they view as infeasible work, I am just asking them to get out of the way please, and let the users decide whether they like it. Nothing significant about the reiserfs project was thought likely to work by sensible people before it worked. I am a bit used to that. After all, Oracle's IFS and several similar projects proved filesystems on top of balanced trees cannot perform well....;-) Anyone who complains about kernel bloat should first consider that reiser4 is not by any means the largest filesystem in the kernel. Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 20:05 ` Linus Torvalds ` (2 preceding siblings ...) 2004-08-31 23:02 ` Christer Weinigel @ 2004-09-01 12:10 ` Alan Cox 2004-09-01 19:44 ` Markus Törnqvist 4 siblings, 0 replies; 1392+ messages in thread From: Alan Cox @ 2004-09-01 12:10 UTC (permalink / raw) To: Linus Torvalds Cc: Horst von Brand, Pavel Machek, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Maw, 2004-08-31 at 21:05, Linus Torvalds wrote: > In a graphical environment, the "icon" stream is a good example of this. > It literally has _nothing_ to do with the data in the main stream. The > only linkage is a totally non-technical one, where the user wanted to > associate a secondary stream with the main stream _without_ altering the > main one. THAT is where named streams make sense. The icon doesn't belong in the document, that was a catastrophic disaster in early MacOS (although they only use an index). Users want to manage their icon choices, tags, tooltip notes and attached labels, and you cannot do that in the file if you don't own the file. Also the icon is *not* unrelated to the file in a modern GUI, eg rox and nautilus uses scaled versions of the content for many media types and will show you pictures, frames from a movie etc to help you remember the content. On top they then add user specific annotations. The things that are more independant are: "This file was created by OpenOffice 1.2" "This is a text/plain file in UTF-8" "This file has a UUID of ...." The type has dragons because you get heirarchical typing within documents (consider XML containing namespaces) (UUIDs being one really useful thing we don't tag everywhere onto files that would be a godsend on the desktop providing they moved with the file on rename) Alan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 20:05 ` Linus Torvalds ` (3 preceding siblings ...) 2004-09-01 12:10 ` Alan Cox @ 2004-09-01 19:44 ` Markus Törnqvist 2004-09-07 6:43 ` Neil Brown 4 siblings, 1 reply; 1392+ messages in thread From: Markus Törnqvist @ 2004-09-01 19:44 UTC (permalink / raw) To: Linus Torvalds Cc: Horst von Brand, Pavel Machek, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Tue, Aug 31, 2004 at 01:05:40PM -0700, Linus Torvalds wrote: > >There's no point to having the kernel export information that is already >inherent in the main stream. As said before, it can be bounced :) You have support for streams, someone could read the inherent information and write it to the stream. At the risk of these two streams being unsynced temporarily, requiring some syncer. >I've seen all these examples of exposing MP3 ID information as a "side >stream", and that's TOTALLY POINTLESS! The information is already there, No it's not pointless :) >it's in a standard format, and exporting it as a stream buys you >absolutely nothing. There is no nice standard way of finding MP3s by the ID information. find music/ -name *mp3/..metas/id3/ | xargs grep -i "dream theater" or something. Which reveals the metadata to find. The kernel doesn't have to parse the ID3 tags, that can be handled in userspace by a daemon, updated based on mtimes or somesuch, yes? >Which means that normally we really don't _want_ named streams. In 99% of >all cases we can use equally good - and _much_ simpler - tool-based >solutions. Sure, when someone goes through every application from ls to nautilus and rams a tool-based patch down the app maintainer's throat and then we're still dependent on the filesystem layout, which is easier to break than the filesystem itself :) >Which means that the only _real_ technical issue for supporting named >streams really ends up being things like samba, which want named streams So they should be implemented for samba and other guys can come up with alternative means for using them ;) -- mjt ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 19:44 ` Markus Törnqvist @ 2004-09-07 6:43 ` Neil Brown 2004-09-07 6:47 ` Chris Wedgwood 0 siblings, 1 reply; 1392+ messages in thread From: Neil Brown @ 2004-09-07 6:43 UTC (permalink / raw) To: Markus Törnqvist Cc: Linus Torvalds, Horst von Brand, Pavel Machek, David Masover, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wednesday September 1, mjt@nysv.org wrote: > On Tue, Aug 31, 2004 at 01:05:40PM -0700, Linus Torvalds wrote: > > > >There's no point to having the kernel export information that is already > >inherent in the main stream. Having read this quote a few times in this thread I finally figured out what was wrong with it. It seems to imply that "iso9660" shouldn't be in the kernel. After all, it just exports information that is already in the underlying device. It doesn't provide any "mediate multiple access" benefit as a read-only filesystem doesn't require any mediation between users. It might provide some caching benefit, but I somehow don't think that is the reason that it is in the kernel. Now I'm happy to agree that there is a case for "iso9660" but not for "tarfs", but I don't think it is that "there's no point to having the kernel export information that is already [there]" NeilBrown ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-07 6:43 ` Neil Brown @ 2004-09-07 6:47 ` Chris Wedgwood 0 siblings, 0 replies; 1392+ messages in thread From: Chris Wedgwood @ 2004-09-07 6:47 UTC (permalink / raw) To: Neil Brown Cc: Markus Törnqvist, Linus Torvalds, Horst von Brand, Pavel Machek, David Masover, Jamie Lokier, viro, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Tue, Sep 07, 2004 at 04:43:11PM +1000, Neil Brown wrote: > It seems to imply that "iso9660" shouldn't be in the kernel. After > all, it just exports information that is already in the underlying > device. Some applications do isofs/udf in userspace, DVD playback for example. > It doesn't provide any "mediate multiple access" benefit as a > read-only filesystem doesn't require any mediation between users. Umm... you can have permissions for users and you need the kernel to determine who can access what. --cw ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 19:31 ` Horst von Brand 2004-08-31 20:05 ` Linus Torvalds @ 2004-09-01 1:14 ` David Masover 2004-09-01 4:59 ` Pavel Machek ` (2 more replies) 1 sibling, 3 replies; 1392+ messages in thread From: David Masover @ 2004-09-01 1:14 UTC (permalink / raw) To: Horst von Brand Cc: Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Horst von Brand wrote: | Pavel Machek <pavel@ucw.cz> said: | |>David Masover <ninja@slaphack.com> said: | | | [...] [...] | You do need extra tools anyway, placing them in the kernel is cheating (and | absolutely pointless, IMHO). Repeat after me: plugins in kernel does NOT equal tools in kernel. It's not hard to, for instance, imagine a generic plugin for archive manipulation which talks to a userspace daemon/library. The kernel doesn't know anything other than (maybe) a list of extensions that are archives. All else is handled in userspace -- the idea that "this is a zipfile" and "zipfiles can be extracted with the 'zip' command" are all in userspace. It's not about the kernel, it's about the interface. And see my other mail: cat foo.zip/README less foo.zip/contents/bar.c is a lot easier than lynx http://google.com/search?q=zip emerge zip man zip unzip foo.zip cat bar.c which already assumes quite a lot of expertise. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTUieXgHNmZLgCUhAQLj8Q//VEuRWg9RB/PVg/M1t36SraKIRdbrgGq6 0DYLZwwVWkfz+UKGhUkACNGQzhHG3ySn/lRjqXCsJhzMOzfgmte2CwgGxunHYqdp W69OSZ1vv0ZLpdpzEkVdMfRhhg8RBL54Ewj6fQu7G1ZzJ0J0HUFCEZIsVoL4zJfk +GFl4M+vvc+152HipEDsvrrPasc0dSxmryyzPbyABrfPRi3l2E3Okfewn/4MCC1E H/L04nmvJQ2OSZYu1eSMSpnLrgQdZJbYefRuSinpqfKhjpwQXjj2rnfcPm+xc3hh FDMlAUPA/DAlthTuhxd9+8iDPEARY57/FAMFTalYVhBEv0NWieMMn9V6R8BbnhZW 8y9EWu0Nvoowg9qElkVjWEt5M1Rbeuq4TDviyVyPYg/KcTKDGpjLuyQQbWv6hMVM WAx6THtRK8Ji/XLCj4Kh6qDOEYP1nuwqLYSWv13BtNAJ3bVeKryOJ4Er4AmuJomA uydBPQVsrtzTNmk7Eul2ZjrxXdESpUnUIAbu7H1Gbn51uP4rQTdlXkdPTv6s/wvG OSOMWKs28GBFsG3+IJLnoN2wxrffpgTBEAaPwbh9TRFKLIH+pkCFSx2cIkTj93nr RjETFObjYdJGhGygrq0xOmN5e7iprdxhxd3/+fvjQxnZGxXT3fZLUwhVX9/zhaXG cxau2Onr3jc= =MrR9 -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 1:14 ` David Masover @ 2004-09-01 4:59 ` Pavel Machek 2004-09-01 16:14 ` Jamie Lokier 2004-09-01 5:50 ` Hans Reiser 2004-09-05 11:33 ` Tonnerre 2 siblings, 1 reply; 1392+ messages in thread From: Pavel Machek @ 2004-09-01 4:59 UTC (permalink / raw) To: David Masover Cc: Horst von Brand, Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hi! > | You do need extra tools anyway, placing them in the kernel is cheating > (and > | absolutely pointless, IMHO). > > Repeat after me: plugins in kernel does NOT equal tools in kernel. > > It's not hard to, for instance, imagine a generic plugin for archive > manipulation which talks to a userspace daemon/library. The kernel > doesn't know anything other than (maybe) a list of extensions that are > archives. All else is handled in userspace -- the idea that "this is a > zipfile" and "zipfiles can be extracted with the 'zip' command" are all > in userspace. > > It's not about the kernel, it's about the interface. And see my other mail: > cat foo.zip/README > less foo.zip/contents/bar.c > is a lot easier than > lynx http://google.com/search?q=zip > emerge zip > man zip > unzip foo.zip > cat bar.c > which already assumes quite a lot of expertise. If you only want nice user interface, you can have that today. Its done using coda, and hosted at uservfs.sf.net. Pavel -- When do you have heart between your knees? ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 4:59 ` Pavel Machek @ 2004-09-01 16:14 ` Jamie Lokier 2004-09-01 20:18 ` Pavel Machek 2004-09-03 1:25 ` Gianni Tedesco 0 siblings, 2 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-09-01 16:14 UTC (permalink / raw) To: Pavel Machek Cc: David Masover, Horst von Brand, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Pavel Machek wrote: > > It's not about the kernel, it's about the interface. And see my other mail: > > cat foo.zip/README > > less foo.zip/contents/bar.c > > is a lot easier than > > lynx http://google.com/search?q=zip > > emerge zip > > man zip > > unzip foo.zip > > cat bar.c > > which already assumes quite a lot of expertise. > > If you only want nice user interface, you can have that today. Its > done using coda, and hosted at uservfs.sf.net. Can I do these already using uservfs? cd less foo.zip/contents/bar.c less /usr/portage/distfiles/perl-5.8.5.tar.gz/contents/toke.c grep -R obscure_label ~/RPMS No I can't. Even using the right "#" names for uservfs ('foo.zip#zip' etc.), I can only do the above with convenient paths, completion etc. if I mount my home directory and distfiles over uservfs, so that all I/O in those directories including ordinary accesses is through uservfs, and have the _real_ storage elsewhere. That's quite illogical, cumbersome to set up, and slows down normal I/O significantly. I can do some of those things in a similar way if I do them inside Midnight Commander, Gnome, KDE, or Emacs -- but even then a path that works in one of those doesn't work in another, and the extensions don't even work in all programs, plugins or scripts for those systems. As we're adding file-as-directories to VFS, I propose it would be good to have *hooks* in place, and non-metadata namespace reserved inside the directories, so that uservfs-like userspace filesystems can be auto-mounted at those places and synchronised with the file. In other words, something that makes the next version of uservfs a whole lot more useful. The good news is that a lot of the hooks are falling into place already. Kernel support is needed to handle the interface, cacheing to trigger deletions, page cacheing so that I/O is a normal speed and mmap() works, moveable mounts so that renames work and the cached data followed them, and coherency so that if the underlying archive is changed, it invalidates cached data with the appropriate degree of synchronisation. Note that the current dnotify cannot provide the latter with any strength; it's too weak an interface (so is inotify). (You can omit a lot of these things, but then you get a flaky interface). The FUSE project looks like it will provide an efficient interface to the page cache, avoiding round trips where possible and supporting mmap. If it doesn't, it should. (Similar VFS facilities are useful for distributed filesystems). The presently discussed file-as-directory semantics will provide a nice view, and force the necessary minimal tool changes, so _all_ tools will be fine with the new uservfs. So all I am asking for is a facility to auto-mount with file-as-directory, and the ability for a userspace daemon to be notified of regular file modifications synchronously. Both can be added later, once file-as-directory and moveable mounts are established. (fcntl leases and dnotify _almost_ provide this, but they don't. Looks like incoherent hacks keep getting added all over the place for Samba... :). That's all. Kernel space needs to provide very little that it doesn't already provide, for this rather nice and universally usable feature to work. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 16:14 ` Jamie Lokier @ 2004-09-01 20:18 ` Pavel Machek 2004-09-01 21:59 ` Jamie Lokier 2004-09-03 1:25 ` Gianni Tedesco 1 sibling, 1 reply; 1392+ messages in thread From: Pavel Machek @ 2004-09-01 20:18 UTC (permalink / raw) To: Jamie Lokier Cc: David Masover, Horst von Brand, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hi! > > > It's not about the kernel, it's about the interface. And see my other mail: > > > cat foo.zip/README > > > less foo.zip/contents/bar.c > > > is a lot easier than > > > lynx http://google.com/search?q=zip > > > emerge zip > > > man zip > > > unzip foo.zip > > > cat bar.c > > > which already assumes quite a lot of expertise. > > > > If you only want nice user interface, you can have that today. Its > > done using coda, and hosted at uservfs.sf.net. > > Can I do these already using uservfs? > > cd > less foo.zip/contents/bar.c less foo.zip#uzip/contents/bar.c > less /usr/portage/distfiles/perl-5.8.5.tar.gz/contents/toke.c less /usr/portage/distfiles/perl-5.8.5.tar.gz#utar/contents/toke.c > grep -R obscure_label ~/RPMS I do not think you'd want this. How would you tell grep obscure label without entering archives, then? > No I can't. > Even using the right "#" names for uservfs ('foo.zip#zip' etc.), I can > only do the above with convenient paths, completion etc. if I mount > my Well, you can... or at least I had version somewhere that automagically prepended /overlay when file was not found. That way, normal requests were fast and uservfs were used whenever it was needed... At the price of little uglyness and slowing down -ENOENTs. > As we're adding file-as-directories to VFS, I propose it would be good > to have *hooks* in place, and non-metadata namespace reserved inside > the directories, so that uservfs-like userspace filesystems can > be auto-mounted at those places and synchronised with the file. > > In other words, something that makes the next version of uservfs a > whole lot more useful. Yep, something less hacky would be nice. > So all I am asking for is a facility to auto-mount with > file-as-directory, and the ability for a userspace daemon to be > notified of regular file modifications synchronously. Both can be > added later, once file-as-directory and moveable mounts are If the userspace daemon is synchronously notified of file modification... will not that lead to very ugly deadlocks? Pavel -- Horseback riding is like software... ...vgf orggre jura vgf serr. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:18 ` Pavel Machek @ 2004-09-01 21:59 ` Jamie Lokier 2004-09-01 22:51 ` Lee Revell 0 siblings, 1 reply; 1392+ messages in thread From: Jamie Lokier @ 2004-09-01 21:59 UTC (permalink / raw) To: Pavel Machek Cc: David Masover, Horst von Brand, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Pavel Machek wrote: > > Can I do these already using uservfs? > > > > cd > > less foo.zip/contents/bar.c > > less foo.zip#uzip/contents/bar.c > > > less /usr/portage/distfiles/perl-5.8.5.tar.gz/contents/toke.c > > less /usr/portage/distfiles/perl-5.8.5.tar.gz#utar/contents/toke.c > > > grep -R obscure_label ~/RPMS > > I do not think you'd want this. How would you tell grep obscure label > without entering archives, then? Granted, this is where we need "grep --recurse-into-files" or, more generally useful, a find option. > > No I can't. > > > Even using the right "#" names for uservfs ('foo.zip#zip' etc.), I can > > only do the above with convenient paths, completion etc. if I mount > > my > > Well, you can... or at least I had version somewhere that > automagically prepended /overlay when file was not found. That way, > normal requests were fast and uservfs were used whenever it was > needed... At the price of little uglyness and slowing down -ENOENTs. There you go... a kernel hook to prepend /overlay works :) These hook I keep mentioning, they are really very simple :) > > So all I am asking for is a facility to auto-mount with > > file-as-directory, and the ability for a userspace daemon to be > > notified of regular file modifications synchronously. Both can be > > added later, once file-as-directory and moveable mounts are > > If the userspace daemon is synchronously notified of file > modification... will not that lead to very ugly deadlocks? Not if it's written properly. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 21:59 ` Jamie Lokier @ 2004-09-01 22:51 ` Lee Revell 2004-09-02 14:25 ` Horst von Brand 2004-09-05 11:54 ` Tonnerre 0 siblings, 2 replies; 1392+ messages in thread From: Lee Revell @ 2004-09-01 22:51 UTC (permalink / raw) To: Jamie Lokier Cc: Pavel Machek, David Masover, Horst von Brand, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, 2004-09-01 at 17:59, Jamie Lokier wrote: > Pavel Machek wrote: > > > Can I do these already using uservfs? > > > > > > cd > > > less foo.zip/contents/bar.c > > > > less foo.zip#uzip/contents/bar.c > > > > > less /usr/portage/distfiles/perl-5.8.5.tar.gz/contents/toke.c > > > > less /usr/portage/distfiles/perl-5.8.5.tar.gz#utar/contents/toke.c > > > > > grep -R obscure_label ~/RPMS > > > > I do not think you'd want this. How would you tell grep obscure label > > without entering archives, then? > > Granted, this is where we need "grep --recurse-into-files" or, more > generally useful, a find option. > FWIW, this is how Windows does it now. As of XP, 'Find files' has an option, enabled by default, to look inside archives. If you tell it to look for a driver in a given directory it will also look inside .cab and .zip files. It's extremely useful, I would imagine someone who uses XP a lot will come to expect this feature. Of course, no idea how it's implemented, but users like it. Lee ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 22:51 ` Lee Revell @ 2004-09-02 14:25 ` Horst von Brand 2004-09-02 17:47 ` Lee Revell 2004-09-02 19:41 ` Spam 2004-09-05 11:54 ` Tonnerre 1 sibling, 2 replies; 1392+ messages in thread From: Horst von Brand @ 2004-09-02 14:25 UTC (permalink / raw) To: Lee Revell Cc: Jamie Lokier, Pavel Machek, David Masover, Horst von Brand, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Lee Revell <rlrevell@joe-job.com> said: [...] > FWIW, this is how Windows does it now. As of XP, 'Find files' has an > option, enabled by default, to look inside archives. If you tell it to > look for a driver in a given directory it will also look inside .cab > and .zip files. It's extremely useful, I would imagine someone who uses > XP a lot will come to expect this feature. It is trivial to implement this by looking inside the files. I.e., the way mc has done this for ages. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 14:25 ` Horst von Brand @ 2004-09-02 17:47 ` Lee Revell 2004-09-02 19:41 ` Spam 1 sibling, 0 replies; 1392+ messages in thread From: Lee Revell @ 2004-09-02 17:47 UTC (permalink / raw) To: Horst von Brand Cc: Jamie Lokier, Pavel Machek, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2004-09-02 at 10:25, Horst von Brand wrote: > Lee Revell <rlrevell@joe-job.com> said: > > [...] > > > FWIW, this is how Windows does it now. As of XP, 'Find files' has an > > option, enabled by default, to look inside archives. If you tell it to > > look for a driver in a given directory it will also look inside .cab > > and .zip files. It's extremely useful, I would imagine someone who uses > > XP a lot will come to expect this feature. > > It is trivial to implement this by looking inside the files. I.e., the way > mc has done this for ages. This requires a separate, MC-specific namespace. The point is to unify the namespace, not fragment it. If Hans had a comprehensible web page, maybe more people would understand this aspect of his argument. Would you put digressions about chaos theory and King Arthur in a man page? Lee ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 14:25 ` Horst von Brand 2004-09-02 17:47 ` Lee Revell @ 2004-09-02 19:41 ` Spam 2004-09-02 18:55 ` Alan Cox 2004-09-02 19:49 ` Pavel Machek 1 sibling, 2 replies; 1392+ messages in thread From: Spam @ 2004-09-02 19:41 UTC (permalink / raw) To: Horst von Brand Cc: Lee Revell, Jamie Lokier, Pavel Machek, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > Lee Revell <rlrevell@joe-job.com> said: > [...] >> FWIW, this is how Windows does it now. As of XP, 'Find files' has an >> option, enabled by default, to look inside archives. If you tell it to >> look for a driver in a given directory it will also look inside .cab >> and .zip files. It's extremely useful, I would imagine someone who uses >> XP a lot will come to expect this feature. > It is trivial to implement this by looking inside the files. I.e., the way > mc has done this for ages. Difference is that you can't do "locate" or "find" or "Search".. You would have to open the files in an archive-supporting application such as mc. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 19:41 ` Spam @ 2004-09-02 18:55 ` Alan Cox 2004-09-02 20:07 ` Spam 2004-09-02 20:42 ` Pavel Machek 2004-09-02 19:49 ` Pavel Machek 1 sibling, 2 replies; 1392+ messages in thread From: Alan Cox @ 2004-09-02 18:55 UTC (permalink / raw) To: Spam Cc: Horst von Brand, Lee Revell, Jamie Lokier, Pavel Machek, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Iau, 2004-09-02 at 20:41, Spam wrote: > > It is trivial to implement this by looking inside the files. I.e., the way > > mc has done this for ages. > > Difference is that you can't do "locate" or "find" or "Search".. You > would have to open the files in an archive-supporting application > such as mc. And would you rather that logic was running swappable in shared library space or privileged and unswappable in kernel ? Alan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 18:55 ` Alan Cox @ 2004-09-02 20:07 ` Spam 2004-09-02 19:17 ` Alan Cox 2004-09-02 20:42 ` Pavel Machek 1 sibling, 1 reply; 1392+ messages in thread From: Spam @ 2004-09-02 20:07 UTC (permalink / raw) To: Alan Cox Cc: Horst von Brand, Lee Revell, Jamie Lokier, Pavel Machek, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List > On Iau, 2004-09-02 at 20:41, Spam wrote: >> > It is trivial to implement this by looking inside the files. I.e., the way >> > mc has done this for ages. >> >> Difference is that you can't do "locate" or "find" or "Search".. You >> would have to open the files in an archive-supporting application >> such as mc. > And would you rather that logic was running swappable in shared library > space or privileged and unswappable in kernel ? I would rather have it as a filesystem/vfs plugin that would allow all my programs to use the features the plugin gives, even if it would mean being totally in kernel space. > Alan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 20:07 ` Spam @ 2004-09-02 19:17 ` Alan Cox 2004-09-02 20:28 ` Spam 0 siblings, 1 reply; 1392+ messages in thread From: Alan Cox @ 2004-09-02 19:17 UTC (permalink / raw) To: Spam Cc: Horst von Brand, Lee Revell, Jamie Lokier, Pavel Machek, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Iau, 2004-09-02 at 21:07, Spam wrote: > > And would you rather that logic was running swappable in shared library > > space or privileged and unswappable in kernel ? > > I would rather have it as a filesystem/vfs plugin that would allow > all my programs to use the features the plugin gives, even if it > would mean being totally in kernel space. Thats not what I asked. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 19:17 ` Alan Cox @ 2004-09-02 20:28 ` Spam 0 siblings, 0 replies; 1392+ messages in thread From: Spam @ 2004-09-02 20:28 UTC (permalink / raw) To: Alan Cox Cc: Horst von Brand, Lee Revell, Jamie Lokier, Pavel Machek, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List > On Iau, 2004-09-02 at 21:07, Spam wrote: >> > And would you rather that logic was running swappable in shared library >> > space or privileged and unswappable in kernel ? >> >> I would rather have it as a filesystem/vfs plugin that would allow >> all my programs to use the features the plugin gives, even if it >> would mean being totally in kernel space. > Thats not what I asked. I failed to understand your question then. :( ~S ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 18:55 ` Alan Cox 2004-09-02 20:07 ` Spam @ 2004-09-02 20:42 ` Pavel Machek 1 sibling, 0 replies; 1392+ messages in thread From: Pavel Machek @ 2004-09-02 20:42 UTC (permalink / raw) To: Alan Cox Cc: Spam, Horst von Brand, Lee Revell, Jamie Lokier, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List Hi! > > > It is trivial to implement this by looking inside the files. I.e., the way > > > mc has done this for ages. > > > > Difference is that you can't do "locate" or "find" or "Search".. You > > would have to open the files in an archive-supporting application > > such as mc. > > And would you rather that logic was running swappable in shared library > space or privileged and unswappable in kernel ? Well, having it in kernel mean that cache gets actually shared between different processes -- and that's usefull. With coda and similar tools you can put most of the logic into userspace (kernel impact is 20 lines). You *will* want it to run priviledged, because otherwise caches are useless. Pavel ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 19:41 ` Spam 2004-09-02 18:55 ` Alan Cox @ 2004-09-02 19:49 ` Pavel Machek 2004-09-02 20:01 ` Lee Revell 1 sibling, 1 reply; 1392+ messages in thread From: Pavel Machek @ 2004-09-02 19:49 UTC (permalink / raw) To: Spam Cc: Horst von Brand, Lee Revell, Jamie Lokier, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hi! > >> FWIW, this is how Windows does it now. As of XP, 'Find files' has an > >> option, enabled by default, to look inside archives. If you tell it to > >> look for a driver in a given directory it will also look inside .cab > >> and .zip files. It's extremely useful, I would imagine someone who uses > >> XP a lot will come to expect this feature. > > > It is trivial to implement this by looking inside the files. I.e., the way > > mc has done this for ages. > > Difference is that you can't do "locate" or "find" or "Search".. You > would have to open the files in an archive-supporting application > such as mc. You really need archive support in find. At the very least you need option "enter archives" vs. "do not enter archives". Entering archives automagically is seriously wrong. Pavel ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 19:49 ` Pavel Machek @ 2004-09-02 20:01 ` Lee Revell 2004-09-02 19:16 ` Alan Cox ` (3 more replies) 0 siblings, 4 replies; 1392+ messages in thread From: Lee Revell @ 2004-09-02 20:01 UTC (permalink / raw) To: Pavel Machek Cc: Spam, Horst von Brand, Jamie Lokier, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2004-09-02 at 15:49, Pavel Machek wrote: > Hi! > > > >> FWIW, this is how Windows does it now. As of XP, 'Find files' has an > > >> option, enabled by default, to look inside archives. If you tell it to > > >> look for a driver in a given directory it will also look inside .cab > > >> and .zip files. It's extremely useful, I would imagine someone who uses > > >> XP a lot will come to expect this feature. > > > > > It is trivial to implement this by looking inside the files. I.e., the way > > > mc has done this for ages. > > > > Difference is that you can't do "locate" or "find" or "Search".. You > > would have to open the files in an archive-supporting application > > such as mc. > > You really need archive support in find. At the very least you need > option "enter archives" vs. "do not enter archives". Entering archives > automagically is seriously wrong. But is it efficient to make every application that reads files have to know how to get inside a tar file, just to read its contents? That seems like a massive duplication of effort. Better to have the contents accessible via a separate stream, in the same namespace. Fix it once in the kernel vs. fix it in umpteen apps. The key point here is that the expressive power of the system is greatly reduced by having a fragmented namespace. Of course there are any number of ways for an app to find out what is in a tar file, but exporting all of that information in a unified namespace is nontrivial and much more interesting. Lee ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 20:01 ` Lee Revell @ 2004-09-02 19:16 ` Alan Cox 2004-09-05 12:07 ` Tonnerre 2004-09-02 20:04 ` Chris Wedgwood ` (2 subsequent siblings) 3 siblings, 1 reply; 1392+ messages in thread From: Alan Cox @ 2004-09-02 19:16 UTC (permalink / raw) To: Lee Revell Cc: Pavel Machek, Spam, Horst von Brand, Jamie Lokier, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Iau, 2004-09-02 at 21:01, Lee Revell wrote: > But is it efficient to make every application that reads files have to > know how to get inside a tar file, just to read its contents? That > seems like a massive duplication of effort. Better to have the contents > accessible via a separate stream, in the same namespace. Fix it once in > the kernel vs. fix it in umpteen apps. Thats how you get yourself a non useful OS. Fix it in a library and share it between the apps that care. Like say.. gnome-vfs2 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 19:16 ` Alan Cox @ 2004-09-05 12:07 ` Tonnerre 2004-09-05 12:27 ` Spam ` (2 more replies) 0 siblings, 3 replies; 1392+ messages in thread From: Tonnerre @ 2004-09-05 12:07 UTC (permalink / raw) To: Alan Cox Cc: Lee Revell, Pavel Machek, Spam, Horst von Brand, Jamie Lokier, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 717 bytes --] Salut, On Thu, Sep 02, 2004 at 08:16:35PM +0100, Alan Cox wrote: > Thats how you get yourself a non useful OS. Fix it in a library and > share it between the apps that care. Like say.. gnome-vfs2 Even KIOslave has it. They even support sftp and stuff just by using shared files in /tmp in reality. That's a much saner interface than doing it all in the kernel. I mean, the kernel is supposed to support access to the disk drives. Who says that it's got to be the uppermost VFS level? You can be perfectly happy to build your own VFS on top of it (or use other's implementations, that is.) I can already see people moving to FreeBSD if this gets implemented into the kernel... Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-05 12:07 ` Tonnerre @ 2004-09-05 12:27 ` Spam 2004-09-06 10:54 ` Pavel Machek 2004-09-06 18:08 ` Horst von Brand 2 siblings, 0 replies; 1392+ messages in thread From: Spam @ 2004-09-05 12:27 UTC (permalink / raw) To: Tonnerre Cc: Alan Cox, Lee Revell, Pavel Machek, Horst von Brand, Jamie Lokier, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List > Salut, > On Thu, Sep 02, 2004 at 08:16:35PM +0100, Alan Cox wrote: >> Thats how you get yourself a non useful OS. Fix it in a library and >> share it between the apps that care. Like say.. gnome-vfs2 > Even KIOslave has it. They even support sftp and stuff just by using > shared files in /tmp in reality. That's a much saner interface than > doing it all in the kernel. > I mean, the kernel is supposed to support access to the disk > drives. Who says that it's got to be the uppermost VFS level? You can > be perfectly happy to build your own VFS on top of it (or use other's > implementations, that is.) Yes, we have several VFS versions. KDE and Gnome has their own, mc another, and so on. None is compatible with the other and all data is unavailable if you do not use a program specifically coded to use the particular VFS you stored the data with. > I can already see people moving to FreeBSD if this gets implemented > into the kernel... Why? No one has to use it. you do not have to load FS/VFS plugins so you can extend the functionality of your filesystem with stuff like compression, encryption, sorting, filtering, etc etc.. > Tonnerre ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-05 12:07 ` Tonnerre 2004-09-05 12:27 ` Spam @ 2004-09-06 10:54 ` Pavel Machek 2004-09-06 18:08 ` Horst von Brand 2 siblings, 0 replies; 1392+ messages in thread From: Pavel Machek @ 2004-09-06 10:54 UTC (permalink / raw) To: Tonnerre Cc: Alan Cox, Lee Revell, Spam, Horst von Brand, Jamie Lokier, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List Hi! > > Thats how you get yourself a non useful OS. Fix it in a library and > > share it between the apps that care. Like say.. gnome-vfs2 > > Even KIOslave has it. They even support sftp and stuff just by using > shared files in /tmp in reality. That's a much saner interface than > doing it all in the kernel. > > I mean, the kernel is supposed to support access to the disk > drives. Who says that it's got to be the uppermost VFS level? You can > be perfectly happy to build your own VFS on top of it (or use other's > implementations, that is.) You can not reasonably do caching when you are in shared library. And you can not do caching across users at all. Pavel ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-05 12:07 ` Tonnerre 2004-09-05 12:27 ` Spam 2004-09-06 10:54 ` Pavel Machek @ 2004-09-06 18:08 ` Horst von Brand 2004-09-08 6:00 ` Tonnerre 2 siblings, 1 reply; 1392+ messages in thread From: Horst von Brand @ 2004-09-06 18:08 UTC (permalink / raw) To: Tonnerre Cc: Alan Cox, Lee Revell, Pavel Machek, Spam, Horst von Brand, Jamie Lokier, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List Tonnerre <tonnerre@thundrix.ch> said: [...] > I can already see people moving to FreeBSD if this gets implemented > into the kernel... If it works better for them, great! It is after all a free world. If it turns out via *BSD/MacOS XIX/Windows 2010/... that it is really useful, we can add it to Linux. Linux is _not_ in a feeping creaturitis race, thank you very much. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-06 18:08 ` Horst von Brand @ 2004-09-08 6:00 ` Tonnerre 2004-09-08 13:26 ` Horst von Brand 0 siblings, 1 reply; 1392+ messages in thread From: Tonnerre @ 2004-09-08 6:00 UTC (permalink / raw) To: Horst von Brand; +Cc: linux-kernel [-- Attachment #1: Type: text/plain, Size: 815 bytes --] Salut, On Mon, Sep 06, 2004 at 02:08:15PM -0400, Horst von Brand wrote: > Tonnerre <tonnerre@thundrix.ch> said: > > I can already see people moving to FreeBSD if this gets implemented > > into the kernel... > > If it works better for them, great! It is after all a free world. If it > turns out via *BSD/MacOS XIX/Windows 2010/... that it is really useful, we > can add it to Linux. Linux is _not_ in a feeping creaturitis race, thank > you very much. Please read again what I just said. I'm against implementing it, because it is creeping featurism. On the contrary, I said that people might actually change to a less complicated/bloated OS (Whereas I would choose NetBSD over FreeBSD, but that's another story). So you say the wrong things at the wrong people. Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-08 6:00 ` Tonnerre @ 2004-09-08 13:26 ` Horst von Brand 0 siblings, 0 replies; 1392+ messages in thread From: Horst von Brand @ 2004-09-08 13:26 UTC (permalink / raw) To: Tonnerre; +Cc: Horst von Brand, linux-kernel Tonnerre <tonnerre@thundrix.ch> said: > On Mon, Sep 06, 2004 at 02:08:15PM -0400, Horst von Brand wrote: > > Tonnerre <tonnerre@thundrix.ch> said: > > > I can already see people moving to FreeBSD if this gets implemented > > > into the kernel... > > If it works better for them, great! It is after all a free world. If it > > turns out via *BSD/MacOS XIX/Windows 2010/... that it is really useful, we > > can add it to Linux. Linux is _not_ in a feeping creaturitis race, thank > > you very much. > > Please read again what I just said. I'm against implementing it, > because it is creeping featurism. On the contrary, I said that people > might actually change to a less complicated/bloated OS (Whereas I > would choose NetBSD over FreeBSD, but that's another story). Sorry, understood exactly the wrong way around. Fingers faster than the brain, again. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 20:01 ` Lee Revell 2004-09-02 19:16 ` Alan Cox @ 2004-09-02 20:04 ` Chris Wedgwood 2004-09-02 20:10 ` Spam 2004-09-02 20:43 ` Pavel Machek 2004-09-02 22:00 ` Horst von Brand 3 siblings, 1 reply; 1392+ messages in thread From: Chris Wedgwood @ 2004-09-02 20:04 UTC (permalink / raw) To: Lee Revell Cc: Pavel Machek, Spam, Horst von Brand, Jamie Lokier, David Masover, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Sep 02, 2004 at 04:01:17PM -0400, Lee Revell wrote: > Better to have the contents accessible via a separate stream, in the > same namespace. Fix it once in the kernel vs. fix it in umpteen > apps. This is ridiculous. We have shared libraries, 99% of applications manage to use libc for example --- this isn't that different. --cw ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 20:04 ` Chris Wedgwood @ 2004-09-02 20:10 ` Spam 0 siblings, 0 replies; 1392+ messages in thread From: Spam @ 2004-09-02 20:10 UTC (permalink / raw) To: Chris Wedgwood Cc: Lee Revell, Pavel Machek, Horst von Brand, Jamie Lokier, David Masover, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > On Thu, Sep 02, 2004 at 04:01:17PM -0400, Lee Revell wrote: >> Better to have the contents accessible via a separate stream, in the >> same namespace. Fix it once in the kernel vs. fix it in umpteen >> apps. > This is ridiculous. We have shared libraries, 99% of applications > manage to use libc for example --- this isn't that different. Yes if that was the case then programs wouldn't brake if libc and its API was fixed. Sadly, I think it isn't. ~S > --cw ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 20:01 ` Lee Revell 2004-09-02 19:16 ` Alan Cox 2004-09-02 20:04 ` Chris Wedgwood @ 2004-09-02 20:43 ` Pavel Machek 2004-09-02 20:47 ` Lee Revell 2004-09-02 22:00 ` Horst von Brand 3 siblings, 1 reply; 1392+ messages in thread From: Pavel Machek @ 2004-09-02 20:43 UTC (permalink / raw) To: Lee Revell Cc: Spam, Horst von Brand, Jamie Lokier, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hi! > > > >> FWIW, this is how Windows does it now. As of XP, 'Find files' has an > > > >> option, enabled by default, to look inside archives. If you tell it to > > > >> look for a driver in a given directory it will also look inside .cab > > > >> and .zip files. It's extremely useful, I would imagine someone who uses > > > >> XP a lot will come to expect this feature. > > > > > > > It is trivial to implement this by looking inside the files. I.e., the way > > > > mc has done this for ages. > > > > > > Difference is that you can't do "locate" or "find" or "Search".. You > > > would have to open the files in an archive-supporting application > > > such as mc. > > > > You really need archive support in find. At the very least you need > > option "enter archives" vs. "do not enter archives". Entering archives > > automagically is seriously wrong. > > But is it efficient to make every application that reads files have to > know how to get inside a tar file, just to read its contents? That Application does not have to know how to handle tar/zip/etc, but it has to make distinction between "enter archives" and "do not enter archives". See uservfs.sf.net. Pavel ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 20:43 ` Pavel Machek @ 2004-09-02 20:47 ` Lee Revell 2004-09-02 20:49 ` Chris Wedgwood 2004-09-02 20:58 ` Pavel Machek 0 siblings, 2 replies; 1392+ messages in thread From: Lee Revell @ 2004-09-02 20:47 UTC (permalink / raw) To: Pavel Machek Cc: Spam, Horst von Brand, Jamie Lokier, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2004-09-02 at 16:43, Pavel Machek wrote: > Hi! > > > > > >> FWIW, this is how Windows does it now. As of XP, 'Find files' has an > > > > >> option, enabled by default, to look inside archives. If you tell it to > > > > >> look for a driver in a given directory it will also look inside .cab > > > > >> and .zip files. It's extremely useful, I would imagine someone who uses > > > > >> XP a lot will come to expect this feature. > > > > > > > > > It is trivial to implement this by looking inside the files. I.e., the way > > > > > mc has done this for ages. > > > > > > > > Difference is that you can't do "locate" or "find" or "Search".. You > > > > would have to open the files in an archive-supporting application > > > > such as mc. > > > > > > You really need archive support in find. At the very least you need > > > option "enter archives" vs. "do not enter archives". Entering archives > > > automagically is seriously wrong. > > > > But is it efficient to make every application that reads files have to > > know how to get inside a tar file, just to read its contents? That > > Application does not have to know how to handle tar/zip/etc, but it > has to make distinction between "enter archives" and "do not enter > archives". See uservfs.sf.net. But how do you cache the information you had to look in the archive for in a way that other apps can use it? How do you synchronize access to the cache and maintain cache coherency in userspace? Lee ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 20:47 ` Lee Revell @ 2004-09-02 20:49 ` Chris Wedgwood 2004-09-02 20:57 ` Lee Revell 2004-09-02 20:58 ` Pavel Machek 1 sibling, 1 reply; 1392+ messages in thread From: Chris Wedgwood @ 2004-09-02 20:49 UTC (permalink / raw) To: Lee Revell Cc: Pavel Machek, Spam, Horst von Brand, Jamie Lokier, David Masover, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Sep 02, 2004 at 04:47:40PM -0400, Lee Revell wrote: > But how do you cache the information you had to look in the archive > for in a way that other apps can use it? ~/.object-cache/ or whatever > How do you synchronize access to the cache and maintain cache > coherency in userspace? coherency doesn't exist for normal files for the most part anyhow ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 20:49 ` Chris Wedgwood @ 2004-09-02 20:57 ` Lee Revell 2004-09-02 20:58 ` Chris Wedgwood 0 siblings, 1 reply; 1392+ messages in thread From: Lee Revell @ 2004-09-02 20:57 UTC (permalink / raw) To: Chris Wedgwood Cc: Pavel Machek, Spam, Horst von Brand, Jamie Lokier, David Masover, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2004-09-02 at 16:49, Chris Wedgwood wrote: > On Thu, Sep 02, 2004 at 04:47:40PM -0400, Lee Revell wrote: > > > But how do you cache the information you had to look in the archive > > for in a way that other apps can use it? > > ~/.object-cache/ or whatever > How are permissions handled? If root lists the contents of a tar file that is world readable, then joeuser comes along and does the same, can joeuser sees the cached listing? Would it depend on root's umask? Lee ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 20:57 ` Lee Revell @ 2004-09-02 20:58 ` Chris Wedgwood 0 siblings, 0 replies; 1392+ messages in thread From: Chris Wedgwood @ 2004-09-02 20:58 UTC (permalink / raw) To: Lee Revell Cc: Pavel Machek, Spam, Horst von Brand, Jamie Lokier, David Masover, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Sep 02, 2004 at 04:57:06PM -0400, Lee Revell wrote: > How are permissions handled? If root lists the contents of a tar > file that is world readable, then joeuser comes along and does the > same, can joeuser sees the cached listing? everyone has their own cache i guess, works well enough elsewhere also, am i the only person scared by the code complexity when it comes to root and setuid code here? sounds like a disaster waiting to happen ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 20:47 ` Lee Revell 2004-09-02 20:49 ` Chris Wedgwood @ 2004-09-02 20:58 ` Pavel Machek 2004-09-02 22:33 ` Alan Cox 1 sibling, 1 reply; 1392+ messages in thread From: Pavel Machek @ 2004-09-02 20:58 UTC (permalink / raw) To: Lee Revell Cc: Spam, Horst von Brand, Jamie Lokier, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hi! > > Application does not have to know how to handle tar/zip/etc, but it > > has to make distinction between "enter archives" and "do not enter > > archives". See uservfs.sf.net. > > But how do you cache the information you had to look in the archive for > in a way that other apps can use it? How do you synchronize access to > the cache and maintain cache coherency in userspace? Uservfs.sf.net. Unlike alan, I do not think that "do it all in library" is good idea. I put it in the userspace as "codafs" server, and let applications see it as a regular filesystem. Pavel ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 20:58 ` Pavel Machek @ 2004-09-02 22:33 ` Alan Cox 2004-09-03 3:22 ` Gianni Tedesco 2004-09-03 9:42 ` silent semantic changes with reiser4 Pavel Machek 0 siblings, 2 replies; 1392+ messages in thread From: Alan Cox @ 2004-09-02 22:33 UTC (permalink / raw) To: Pavel Machek Cc: Lee Revell, Spam, Horst von Brand, Jamie Lokier, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Iau, 2004-09-02 at 21:58, Pavel Machek wrote: > Uservfs.sf.net. > > Unlike alan, I do not think that "do it all in library" is good > idea. I put it in the userspace as "codafs" server, and let > applications see it as a regular filesystem. That works for me too, providing someone has fixed all the user mode fs deadlocks with paging ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 22:33 ` Alan Cox @ 2004-09-03 3:22 ` Gianni Tedesco 2004-09-03 9:24 ` Userspace framework (was: Re: silent semantic changes with reiser4) Luca Ferroni 2004-09-03 9:42 ` silent semantic changes with reiser4 Pavel Machek 1 sibling, 1 reply; 1392+ messages in thread From: Gianni Tedesco @ 2004-09-03 3:22 UTC (permalink / raw) To: Alan Cox Cc: Pavel Machek, Lee Revell, Spam, Horst von Brand, Jamie Lokier, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Thu, 2004-09-02 at 23:33 +0100, Alan Cox wrote: > On Iau, 2004-09-02 at 21:58, Pavel Machek wrote: > > Uservfs.sf.net. > > > > Unlike alan, I do not think that "do it all in library" is good > > idea. I put it in the userspace as "codafs" server, and let > > applications see it as a regular filesystem. > > That works for me too, providing someone has fixed all the user mode fs > deadlocks with paging Aren't the deadlock scenarios only applicable for read/write mounted filesystems ? -- // Gianni Tedesco (gianni at scaramanga dot co dot uk) lynx --source www.scaramanga.co.uk/scaramanga.asc | gpg --import 8646BE7D: 6D9F 2287 870E A2C9 8F60 3A3C 91B5 7669 8646 BE7D ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Userspace framework (was: Re: silent semantic changes with reiser4) 2004-09-03 3:22 ` Gianni Tedesco @ 2004-09-03 9:24 ` Luca Ferroni 2004-09-03 20:31 ` Gianni Tedesco 0 siblings, 1 reply; 1392+ messages in thread From: Luca Ferroni @ 2004-09-03 9:24 UTC (permalink / raw) To: Gianni Tedesco; +Cc: linux-kernel, miklos, renzo, frederik Il Fri, 03 Sep 2004 04:22:48 +0100, Gianni Tedesco <gianni@scaramanga.co.uk> ha scritto: > On Thu, 2004-09-02 at 23:33 +0100, Alan Cox wrote: > > On Iau, 2004-09-02 at 21:58, Pavel Machek wrote: > > > Uservfs.sf.net. > > > > > > Unlike alan, I do not think that "do it all in library" is good > > > idea. I put it in the userspace as "codafs" server, and let > > > applications see it as a regular filesystem. > > > > That works for me too, providing someone has fixed all the user mode fs > > deadlocks with paging > > Aren't the deadlock scenarios only applicable for read/write mounted > filesystems ? > AFAIK deadlock arises when kernel manages buffers: it has to free a buffer ==> choose a dirty one ==> if cleaning requires to make a call to network server and this last is waiting for a buffer (cleaning accomplished) ==> ==> deadlock. I think buffers are involved even in reading file system access. I think also that NFS is threatened by deadlock, not CodaFS. The problem with CodaFS is that it doesn't support random file access: you have to download a whole file in coda cache directory even to read its first byte. I'm writing my laurea thesis about userspace file system, and two Interesting papers i found about this, are: http://a5.repetae.net/~frederik/avfs/frederik_surf_paper.pdf by Frederik Eaton and the more generic UFO paper http://www.usenix.org/publications/library/proceedings/ana97/alexandrov.html I think that LUFS (Linux User File System) and FUSE (Filesystem in USErspace) kernel modules are solutions to be considered for adding to Linux develop branch. They are similar, but FUSE is still maintained and improved, and it permit to create a custom daemon for each user filesystem (unlinke lufsd daemon which loads a shared library). Communication kernel-user space is done through a special file descriptor, unlike LUFS socket. The only doubt is that FUSE is tuned for local file system performance, but I don't know if it is possible to develop network file system with this framework. LUFS is slower but more generic in this sense. http://sourceforge.net/projects/avf There is more documentation in the tarball than in the home page. Ciao Luca ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: Userspace framework (was: Re: silent semantic changes with reiser4) 2004-09-03 9:24 ` Userspace framework (was: Re: silent semantic changes with reiser4) Luca Ferroni @ 2004-09-03 20:31 ` Gianni Tedesco 0 siblings, 0 replies; 1392+ messages in thread From: Gianni Tedesco @ 2004-09-03 20:31 UTC (permalink / raw) To: Luca Ferroni; +Cc: linux-kernel, miklos, renzo, frederik On Fri, 2004-09-03 at 11:24 +0200, Luca Ferroni wrote: > Il Fri, 03 Sep 2004 04:22:48 +0100, Gianni Tedesco <gianni@scaramanga.co.uk> ha scritto: > > > On Thu, 2004-09-02 at 23:33 +0100, Alan Cox wrote: > > > On Iau, 2004-09-02 at 21:58, Pavel Machek wrote: > > > > Uservfs.sf.net. > > > > > > > > Unlike alan, I do not think that "do it all in library" is good > > > > idea. I put it in the userspace as "codafs" server, and let > > > > applications see it as a regular filesystem. > > > > > > That works for me too, providing someone has fixed all the user mode fs > > > deadlocks with paging > > > > Aren't the deadlock scenarios only applicable for read/write mounted > > filesystems ? > > > > AFAIK deadlock arises when kernel manages buffers: > it has to free a buffer ==> choose a dirty one ==> if cleaning > requires to make a call to > network server and this last is waiting for a buffer (cleaning > accomplished) ==> > ==> deadlock. so during page launder, we need to write to the filesystem, but if thats in userspace there is the possibility the page launder could have been caused in order that the filesystem daemon may run. AFAICS this problem only arises when file is being written. The only deadlock I can think of for read-only filesystems is if the demon inadvertantly accesses one of the files that it is handling. That could be avoided quite simply by preventing the demon from doing that in the kernel. I'm sure I'm missing something, I'd just like to know what ;) -- // Gianni Tedesco (gianni at scaramanga dot co dot uk) lynx --source www.scaramanga.co.uk/scaramanga.asc | gpg --import 8646BE7D: 6D9F 2287 870E A2C9 8F60 3A3C 91B5 7669 8646 BE7D ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 22:33 ` Alan Cox 2004-09-03 3:22 ` Gianni Tedesco @ 2004-09-03 9:42 ` Pavel Machek 1 sibling, 0 replies; 1392+ messages in thread From: Pavel Machek @ 2004-09-03 9:42 UTC (permalink / raw) To: Alan Cox Cc: Lee Revell, Spam, Horst von Brand, Jamie Lokier, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List Hi! > > Uservfs.sf.net. > > > > Unlike alan, I do not think that "do it all in library" is good > > idea. I put it in the userspace as "codafs" server, and let > > applications see it as a regular filesystem. > > That works for me too, providing someone has fixed all the user mode fs > deadlocks with paging Its okay: coda works on whole file granularity, and files are backed by real disk, so there are no deadlocks with paging. We only block open() and similar syscalls, and those are not in paging paths. It should be safe. Pavel ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 20:01 ` Lee Revell ` (2 preceding siblings ...) 2004-09-02 20:43 ` Pavel Machek @ 2004-09-02 22:00 ` Horst von Brand 2004-09-02 23:23 ` Jamie Lokier 3 siblings, 1 reply; 1392+ messages in thread From: Horst von Brand @ 2004-09-02 22:00 UTC (permalink / raw) To: Lee Revell Cc: Pavel Machek, Spam, Horst von Brand, Jamie Lokier, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Lee Revell <rlrevell@joe-job.com> said: > To: Pavel Machek <pavel@ucw.cz> > Cc: Spam <spam@tnonline.net>, Horst von Brand <vonbrand@inf.utfsm.cl>, > Jamie Lokier <jamie@shareable.org>, David Masover <ninja@slaphack.com>, > Chris Wedgwood <cw@f00f.org>, viro@parcelfarce.linux.theplanet.co.uk, > Linus Torvalds <torvalds@osdl.org>, Christoph Hellwig <hch@lst.de>, > Hans Reiser <reiser@namesys.com>, linux-fsdevel@vger.kernel.org, > linux-kernel <linux-kernel@vger.kernel.org>, > Alexander Lyamin aka FLX <flx@namesys.com>, > ReiserFS List <reiserfs-list@namesys.com> > X-Mailer: Ximian Evolution 1.4.6 > Date: Thu, 02 Sep 2004 16:01:17 -0400 > > On Thu, 2004-09-02 at 15:49, Pavel Machek wrote: [...] > > You really need archive support in find. At the very least you need > > option "enter archives" vs. "do not enter archives". Entering archives > > automagically is seriously wrong. I have used find(1) for quite some time now, and have never (or very rarely) missed this. > But is it efficient to make every application that reads files have to > know how to get inside a tar file, just to read its contents? Totally ridiculous, especially if you factor in .gz, .bz2, .zip, .a, .whatever.new.format.they.come.up.with.tomorrow. But then again, this would presumably reside in a (shared) library, so it isn't so bad... > That > seems like a massive duplication of effort. Right. tar, gzip, bunzip, et al are already around. > Better to have the contents > accessible via a separate stream, in the same namespace. Fix it once in > the kernel vs. fix it in umpteen apps. Dead wrong. It is better to fix it in userspace (via a library, if required; could call random unpacking etc programs at will, even be configured on a user-by-user basis through ~/.wacky-file-handling or environment) than force this junk into the kernel. Kernel code is _always_ resident, extremely security critical, and hard to get right. Besides, not everybody will want to carry this around, and so it will forever stay a "weird Linux kernel configuration only" feature, i.e., useless in practice. > The key point here is that the expressive power of the system is greatly > reduced by having a fragmented namespace. Of course there are any > number of ways for an app to find out what is in a tar file, but > exporting all of that information in a unified namespace is nontrivial > and much more interesting. I don't see how it is "nontrivial": "tar tf some.tar" is quite enough to find out what is inside, in a customary format. Placing random junk in the kernel doesn't magically make it fast, right, or useful. Quite a lot of work is required for that, much more than for getting the same in userland. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 22:00 ` Horst von Brand @ 2004-09-02 23:23 ` Jamie Lokier 2004-09-02 22:51 ` Alan Cox 2004-09-03 13:46 ` John Stoffel 0 siblings, 2 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-09-02 23:23 UTC (permalink / raw) To: Horst von Brand Cc: Lee Revell, Pavel Machek, Spam, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Horst von Brand wrote: > > > You really need archive support in find. At the very least you need > > > option "enter archives" vs. "do not enter archives". Entering archives > > > automagically is seriously wrong. > > I have used find(1) for quite some time now, and have never (or very > rarely) missed this. I've occasionally had the need to search all files on my system for the one file which contains a particular phrase -- all I remember is the phrase. Just doing "grep -R" was a tedious job: at least half an hour. Sometimes, I want to search all source files on my system for a particular word, for example to search for uses of a particular system call or library function. That would require something that could search through all the .tar.gz files and .zip files (nested if necessary) as well as plain files. It would take so long -- hours at least, maybe more than a day -- that I've never bothered doing such a thing. "find "that entered archives really wouldn't help (although sometimes "locate" that entered archives would be nice). In other words, I'd use that capability if it was magically fast, but as we expect it to be insanely slow (just grepping gigabytes is slow) that makes it not so useful. However, if we ever see that search engine index thing happen, it would be a most excellent capability if it searched inside archive files too. I would definitely use that. Not often, but occasionally I would. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 23:23 ` Jamie Lokier @ 2004-09-02 22:51 ` Alan Cox 2004-09-03 0:02 ` Spam 2004-09-03 13:46 ` John Stoffel 1 sibling, 1 reply; 1392+ messages in thread From: Alan Cox @ 2004-09-02 22:51 UTC (permalink / raw) To: Jamie Lokier Cc: Horst von Brand, Lee Revell, Pavel Machek, Spam, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Gwe, 2004-09-03 at 00:23, Jamie Lokier wrote: > However, if we ever see that search engine index thing happen, it > would be a most excellent capability if it searched inside archive > files too. I would definitely use that. Not often, but occasionally I would. Thats an indexer decision, the search backend (which is the performance and complexity critical part) doesn't give a damn. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 22:51 ` Alan Cox @ 2004-09-03 0:02 ` Spam 0 siblings, 0 replies; 1392+ messages in thread From: Spam @ 2004-09-03 0:02 UTC (permalink / raw) To: Alan Cox Cc: Jamie Lokier, Horst von Brand, Lee Revell, Pavel Machek, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List > On Gwe, 2004-09-03 at 00:23, Jamie Lokier wrote: >> However, if we ever see that search engine index thing happen, it >> would be a most excellent capability if it searched inside archive >> files too. I would definitely use that. Not often, but occasionally I would. > Thats an indexer decision, the search backend (which is the performance > and complexity critical part) doesn't give a damn. I am just talking general now, but it seems to me that there have been many suggestions on user-land solutions like shared librares and so forth just to say there is no need for file streams and plugins. Many of these ideas do exist in one way or another, but none is truly system wide and and as application independent as file streams+plugins would be. Would it not be much less effort to implement these in a good way, than trying to reinvent lots of new stuff in userland - that wouldn't be systemwide anyway? ~S ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 23:23 ` Jamie Lokier 2004-09-02 22:51 ` Alan Cox @ 2004-09-03 13:46 ` John Stoffel 1 sibling, 0 replies; 1392+ messages in thread From: John Stoffel @ 2004-09-03 13:46 UTC (permalink / raw) To: Jamie Lokier Cc: Horst von Brand, Lee Revell, Pavel Machek, Spam, David Masover, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List >>>>> "Jamie" == Jamie Lokier <jamie@shareable.org> writes: Jamie> Horst von Brand wrote: >> > > You really need archive support in find. At the very least you need >> > > option "enter archives" vs. "do not enter archives". Entering archives >> > > automagically is seriously wrong. >> >> I have used find(1) for quite some time now, and have never (or very >> rarely) missed this. Jamie> I've occasionally had the need to search all files on my system Jamie> for the one file which contains a particular phrase -- all I Jamie> remember is the phrase. So use glimpse and run an indexer once a night to pick up changes. Jamie> Just doing "grep -R" was a tedious job: at least half an hour. Ugh, esp since you're not caching any results of all that work either, when you need to repeat 10 minutes down the line... Jamie> Sometimes, I want to search all source files on my system for a Jamie> particular word, for example to search for uses of a particular Jamie> system call or library function. I think we want to pull this back a bit more and define this better. Instead of 'system' we should be thinking 'namespace' or 'hierarchy'. Jamie> That would require something that could search through all the Jamie> .tar.gz files and .zip files (nested if necessary) as well as Jamie> plain files. It would take so long -- hours at least, maybe Jamie> more than a day -- that I've never bothered doing such a thing. Jamie> In other words, I'd use that capability if it was magically Jamie> fast, but as we expect it to be insanely slow (just grepping Jamie> gigabytes is slow) that makes it not so useful. Jamie> However, if we ever see that search engine index thing happen, Jamie> it would be a most excellent capability if it searched inside Jamie> archive files too. I would definitely use that. Not often, Jamie> but occasionally I would. Is the overhead of pre-indexing all files and their contents worth it though? And would the use of inotify allow us to efficiently update the index in a fairly quick time? All the various arguements have been that we need a filesystem which magically indexes files, does queries and does it all atomically and without any thought on the part of the developer/user. It could happen, but then we'd have such a *slow* system in general, just to make 1% of the usage simpler (for some measure of simple) that it would be a waste. Providing a simple, consistent set of semantics and syntax allows you to build on top of it the layer(s) you need to provide the guarenttees you need for your application. I don't expect that Oracle has the same needs as does my mail spool, or as an image store. Sure, Oracle could do both of them, but are the overheads of oracle worth the slowdown in the common case? John ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 22:51 ` Lee Revell 2004-09-02 14:25 ` Horst von Brand @ 2004-09-05 11:54 ` Tonnerre 1 sibling, 0 replies; 1392+ messages in thread From: Tonnerre @ 2004-09-05 11:54 UTC (permalink / raw) To: Lee Revell Cc: Jamie Lokier, Pavel Machek, David Masover, Horst von Brand, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 669 bytes --] Salut, On Wed, Sep 01, 2004 at 06:51:12PM -0400, Lee Revell wrote: > FWIW, this is how Windows does it now. As of XP, 'Find files' has an > option, enabled by default, to look inside archives. If you tell it to > look for a driver in a given directory it will also look inside .cab > and .zip files. It's extremely useful, I would imagine someone who uses > XP a lot will come to expect this feature. > > Of course, no idea how it's implemented, but users like it. The trick is that their tools are aware of it, and that the library the programs use supports on-the-fly decompression. Same applies to Spotlight for Tiger. Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 16:14 ` Jamie Lokier 2004-09-01 20:18 ` Pavel Machek @ 2004-09-03 1:25 ` Gianni Tedesco 1 sibling, 0 replies; 1392+ messages in thread From: Gianni Tedesco @ 2004-09-03 1:25 UTC (permalink / raw) To: Jamie Lokier; +Cc: linux-kernel On Wed, 2004-09-01 at 17:14 +0100, Jamie Lokier wrote: > So all I am asking for is a facility to auto-mount with > file-as-directory, and the ability for a userspace daemon to be > notified of regular file modifications synchronously. Both can be > added later, once file-as-directory and moveable mounts are > established. (fcntl leases and dnotify _almost_ provide this, but > they don't. Looks like incoherent hacks keep getting added all over > the place for Samba... :). How will synchronous file modification notifications be handled? Programs listening for the notification will want to see the changes when they read the file but the reason this is synchronous is to prevent other applications seeing the changes before they are reflected in these indexing programs etc.. right? -- // Gianni Tedesco (gianni at scaramanga dot co dot uk) lynx --source www.scaramanga.co.uk/scaramanga.asc | gpg --import 8646BE7D: 6D9F 2287 870E A2C9 8F60 3A3C 91B5 7669 8646 BE7D ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 1:14 ` David Masover 2004-09-01 4:59 ` Pavel Machek @ 2004-09-01 5:50 ` Hans Reiser 2004-09-01 6:06 ` Linus Torvalds ` (2 more replies) 2004-09-05 11:33 ` Tonnerre 2 siblings, 3 replies; 1392+ messages in thread From: Hans Reiser @ 2004-09-01 5:50 UTC (permalink / raw) To: David Masover Cc: Horst von Brand, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Linus, you are looking at this like a lieutenant instead of an HQ staffer, which is unusual of you. You are saying, 1-2% simpler and better, no biggie, why work so hard to get it? And we are saying, 1-2% simpler and better, times thousands of applications, wow! That's a lot! Yes, changing cat to use openat() is no big deal. 1-2% additional coding cost for cat, who cares? But if you add 1-2% coding cost to every application which might access an attribute/stream/whatever, well, that totals more than the effort of authoring emacs. Can you see that? Namespace simplifications and empowerments are force multipliers. They don't add to Linux like writing a new app adds to Linux, they add to Linux like adding percentage improvements to every app in Linux adds to Linux. HQ staffer types know that if you collect enough itty bitty little force multipliers, you win the war. Whether the troops have to spend five minutes a day polishing their shoes because their type of shoe needs polishing, that matters more than losing a tank battalion, when you are a major power. Linux is a major power. This is a software engineering issue. We are discussing improvements that because they are diffused throughout the OS in their impact, seem like no big deal. But they are a big deal. One of the major determinants of an organization's efficiency is whether the management can recognize widely diffused inefficiencies as well as it can recognize focused inefficiencies of the same magnitude. David is so very right about the usability issues. Usability is all about trivia. Usability battles feature no dragons, they feature armies of spiders. If you add one step to what a user needs to do to get to the data with an app, that matters a lot. That's doubling the spider army. Making a 2 step process to access data into a one step process to access the data halves the time cost of accessing the data. Lifestyle efficiency is mostly about reducing the cost of trivia, because trivia is most of what we do. Most people have as much interest in reading the man page for tar as you have in learning how to turn the hand crank to start your car. Just the look of "tar -xf" turns them away. Crypto-Geek gobbledy-gook is what it is. Let's value their time, there are a lot of them. Hans David Masover wrote: > > > It's not about the kernel, it's about the interface. And see my other > mail: > cat foo.zip/README > less foo.zip/contents/bar.c > is a lot easier than > lynx http://google.com/search?q=zip > emerge zip > man zip > unzip foo.zip > cat bar.c > which already assumes quite a lot of expertise. > > ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 5:50 ` Hans Reiser @ 2004-09-01 6:06 ` Linus Torvalds 2004-09-01 7:22 ` Hans Reiser 2004-09-01 11:49 ` Alan Cox 2004-09-02 23:40 ` Paul Jakma 2 siblings, 1 reply; 1392+ messages in thread From: Linus Torvalds @ 2004-09-01 6:06 UTC (permalink / raw) To: Hans Reiser Cc: David Masover, Horst von Brand, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Tue, 31 Aug 2004, Hans Reiser wrote: > > You are saying, 1-2% simpler and better, no biggie, why work so hard to > get it? > > And we are saying, 1-2% simpler and better, times thousands of > applications, wow! That's a lot! But would thousands care? Seriously? For example, you could make just _one_ program support "openat()", and you'd get most of the advantages, with no possibility of _breaking_ any of thousands of applications.. I know, you've ignored the "runat" program (which is just a wrapper around the openat() system call), but it _has_ been mentioned several times in this thread. Yes, you'd type a bit more to do runat file ls -l instead of ls -l file/ but at least the openat/runat approach also works for directories, which does actually make it a lot more _generic_ than the "show in the regular filespace" approach. No special cases. So your comparison isn't valid, because you're ignoring the people who shout "runat" at you. You've also not ever actually answered about the problems about directories with attributes. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 6:06 ` Linus Torvalds @ 2004-09-01 7:22 ` Hans Reiser 2004-09-01 11:51 ` Alexander G. M. Smith 2004-09-15 17:06 ` Timothy Miller 0 siblings, 2 replies; 1392+ messages in thread From: Hans Reiser @ 2004-09-01 7:22 UTC (permalink / raw) To: Linus Torvalds Cc: David Masover, Horst von Brand, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List, Alexander Zarochentcev Linus Torvalds wrote: >On Tue, 31 Aug 2004, Hans Reiser wrote: > > >>You are saying, 1-2% simpler and better, no biggie, why work so hard to >>get it? >> >>And we are saying, 1-2% simpler and better, times thousands of >>applications, wow! That's a lot! >> >> > >But would thousands care? Seriously? > > >For example, you could make just _one_ program support "openat()", and >you'd get most of the advantages, with no possibility of _breaking_ any of >thousands of applications.. > >I know, you've ignored the "runat" program (which is just a wrapper around >the openat() system call), but it _has_ been mentioned several times in >this thread. Yes, you'd type a bit more to do > > runat file ls -l > >instead of > > ls -l file/ > >but at least the openat/runat approach also works for directories, which >does actually make it a lot more _generic_ than the "show in the regular >filespace" approach. No special cases. > > So you are saying I can do runat emacs my-compound-document/stream-name? and in place of CTRL-X CTRL-F to open a file in emacs I type what to open an attribute? I think it does not work Linus.... >So your comparison isn't valid, because you're ignoring the people who >shout "runat" at you. You've also not ever actually answered about the >problems about directories with attributes. > > Which problem specifically? I will list them and hope I don't miss what you want addressed. I won't claim to be sure I have the right answers for all of these (because VFS has not been my interest in the past), but these look feasible to me.;-) Cycle detection: We should either 1) make hard links only link to the file aspect of the file-directory duality, and persons who want to link to the directory aspect must use symlinks (best short term answer), or 2) ask Alexander Smith to help us with applying his cycle detection algorithm and gain the benefit of being able to hard link to directories (if it works well, best long term answer). Ambiguity: Linus said: - how do you handle the ambiguity of //attr/usr/bin/emacs/icon (is that the "icon" attribute on "/usr/bin/emacs", or is it perhaps the "emacs/icon" attribute on "/usr/bin"). If you want to avoid crowding the namespace with some piece of meta information that you want to be applicable to both directories and the files in them, and also widely used enough that you care about crowding the namespace, and unambiguous as to what level in the hierarchy it metas, then you put it into the metas subdirectory. There is no principled difference between a file that is a child of a directory and an attribute. The ambiguity is inherent in hiearchical semantics I think, and whether or not you have attributes you have this type of ambiguity. Thus, you have either: /usr/bin/metas/emacs/icon or /usr/bin/emacs/metas/icon and these are easily distinguished and unambiguous. This makes metas a reserved keyword, and there was a long discussion about whether to use "metas" or "..metas" as the reserved subdirectory name which I can summarize for you on request. There is also "pseudos", which contains files implementing special methods of special plugins. Feel free to invent more reserved subdirectories as needed, or to argue for fewer. Seeing two trees, one with attrs and one without: Linus said: If you open a filename in some "secondary" tree (be it /proc or //attr or whatever) based on the filename in the primary one, you have two issues that you need to work out: - how do you handle a name change in the primary tree at the same time as lookup - how do you handle the ambiguity of //attr/usr/bin/emacs/icon (is that the "icon" attribute on "/usr/bin/emacs", or is it perhaps the "emacs/icon" attribute on "/usr/bin"). The ambiguity can be handled by saying that attributes only have one component (ie only the _last_ component of a lookup is the attribute name). But the race between primary tree and secondary tree cannot be handled in a normal name-space. Are you sure that you have to have two trees in order to see two trees? That is, can you not refuse to follow or see a link depending on whether some flag is set and what the type of the link is? Locking: Linus said: let's say that you have filename "a" hard-linked to filename "b", and you have a directory structure of streams under there. So you have a/file1 a/dir1/file2 a/dir2/file3 and (through the hard-link with "b") you have aliases of all these same names available as "b/file1", "b/dir1/file2" etc). Now, imagine that you have two processes doing mv a/dir1 a/dir2/newdir and mv b/dir2 b/dir1/newdir at the same time. Both of them MUST NOT SUCCEED, for pretty obvious reasons (you'd have moved two directories within each other, and now neither would be accessible any more). How do you handle locking for this situation? (Here is where my knowledge of VFS is shakiest.) Making "hard links only link to the file aspect of the file-directory duality, and persons who want to link to the directory aspect must use symlinks" seems to solve this. If we want to use the Alexander Smith solution to cycle detection, and allow hard links to directories, then we must make sure that for the hard linked entity, there is a single lock for it somewhere (e.g. the directory inode) that can be taken. If you create two dentries for the same object, then flag the dentries as needing to lock a lock in their shared inode and not in the dentry. If you don't want to read in the inode from disk to do that, then create a pointer to a lock shared by them both. Linus, is this a start on solving some of the issues? ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 7:22 ` Hans Reiser @ 2004-09-01 11:51 ` Alexander G. M. Smith 2004-09-15 17:06 ` Timothy Miller 1 sibling, 0 replies; 1392+ messages in thread From: Alexander G. M. Smith @ 2004-09-01 11:51 UTC (permalink / raw) To: Hans Reiser Cc: ninja, vonbrand, pavel, jamie, cw, viro, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list, zam, torvalds Hans Reiser wrote on Wed, 01 Sep 2004 00:22:49 -0700: > Making "hard links only link to the file aspect of the file-directory > duality, and persons who want to link to the directory aspect must use > symlinks" seems to solve this. If we want to use the Alexander Smith > solution to cycle detection, and allow hard links to directories, then > we must make sure that for the hard linked entity, there is a single > lock for it somewhere (e.g. the directory inode) that can be taken. Oddly enough that's what I did in my RAM file system. No matter how the file/directory node is found (the path can lead through different parents), it has a unique ID and a corresponding single lock for it. Rename/delete goes and locks all the nodes that might change parents and/or children, then it does its operations (though it has to watch out for deadlocks, which are unavoidable when two rename/deletes both sneak up on the same file by different paths). - Alex ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 7:22 ` Hans Reiser 2004-09-01 11:51 ` Alexander G. M. Smith @ 2004-09-15 17:06 ` Timothy Miller 2004-09-16 8:23 ` Helge Hafting 1 sibling, 1 reply; 1392+ messages in thread From: Timothy Miller @ 2004-09-15 17:06 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, David Masover, Horst von Brand, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List, Alexander Zarochentcev I'm probably not the first to suggest this idea, and it's probably not a very good idea, but here's my idea anyhow: You have a file "/usr/bin/emacs" with a metadata property in the overlaid namespace "/usr/bin/emacs/[[..]metas/]icon" According to some, this could cause some confusion. Howabout instead: You have a file "/usr/bin/emacs" with a metadata property in a slightly separated namespace "/metas/usr/bin/emacs/icon" This has the advantage of still having the metadata in the filesystem namespace but without the confusion of having files-as-directories, ambiguity of filename, backup issues, etc. This is the reverse of having the namespaces overlaid with a "/nometas" view which is separate. Furthermore, you can split things further like this: You have a file "/usr/bin/emacs" with an automatically-generated metadata property that you don't want to back up in "/autometas/usr/bin/emacs/modification_date" and a manually generated metadata property that you MAY want to backup in "/staticmetas/usr/bin/emacs/icon". This is inelegant, I know. But if we do this, we can add the extra features of reiser4 without confusing existing apps or having to modify them to support the new functionality. Furthermore, you can easily hide the extra features by not mounting the meta top-level directories (assuming they're mounted like separate filesystems, rather than just magically appearing there, which is okay too). ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-15 17:06 ` Timothy Miller @ 2004-09-16 8:23 ` Helge Hafting 0 siblings, 0 replies; 1392+ messages in thread From: Helge Hafting @ 2004-09-16 8:23 UTC (permalink / raw) To: Timothy Miller Cc: Hans Reiser, Linus Torvalds, David Masover, Horst von Brand, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List, Alexander Zarochentcev Timothy Miller wrote: > I'm probably not the first to suggest this idea, and it's probably not > a very good idea, but here's my idea anyhow: > > You have a file "/usr/bin/emacs" > with a metadata property in the overlaid namespace > "/usr/bin/emacs/[[..]metas/]icon" > > According to some, this could cause some confusion. Howabout instead: > > You have a file "/usr/bin/emacs" > with a metadata property in a slightly separated namespace > "/metas/usr/bin/emacs/icon" > And the problem with such a "solution" is mv /usr/bin/emacs /usr/bin/old-emacs Do this with an ordinary fs, and the /metas/usr/bin/emacs/icon won't move with it. Now metas might might not be an ordinary fs, so perhaps the move happens automatically there, but if so it will be unexpected. > This has the advantage of still having the metadata in the filesystem > namespace but without the confusion of having files-as-directories, > ambiguity of filename, backup issues, etc. This is the reverse of > having the namespaces overlaid with a "/nometas" view which is separate. > > Furthermore, you can split things further like this: > > You have a file "/usr/bin/emacs" > with an automatically-generated metadata property that you don't want > to back up in "/autometas/usr/bin/emacs/modification_date" > and a manually generated metadata property that you MAY want to backup > in "/staticmetas/usr/bin/emacs/icon". > > This is inelegant, I know. But if we do this, we can add the extra > features of reiser4 without confusing existing apps or having to > modify them to support the new functionality. > > Furthermore, you can easily hide the extra features by not mounting > the meta top-level directories (assuming they're mounted like separate > filesystems, rather than just magically appearing there, which is okay > too). Having to go up to root and then down a similar but different path to reach a file's metadata seems very counterintuitive to me. And you have to update all tools to do this automatically, or it'll be hopeless to actually use. Helge Hafting ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 5:50 ` Hans Reiser 2004-09-01 6:06 ` Linus Torvalds @ 2004-09-01 11:49 ` Alan Cox 2004-09-02 23:40 ` Paul Jakma 2 siblings, 0 replies; 1392+ messages in thread From: Alan Cox @ 2004-09-01 11:49 UTC (permalink / raw) To: Hans Reiser Cc: David Masover, Horst von Brand, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Mer, 2004-09-01 at 06:50, Hans Reiser wrote: > Yes, changing cat to use openat() is no big deal. 1-2% additional coding > cost for cat, who cares? Large cost on its own. The FSF would almost certainly reject such a change until someone had written portable emulation of the feature for every other platform which would mean vendor patches which would essentially mean it wouldn't happen. I know I rarely agree with Hans but I think he's right on this one, it has to work with the existing open() API. Alan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 5:50 ` Hans Reiser 2004-09-01 6:06 ` Linus Torvalds 2004-09-01 11:49 ` Alan Cox @ 2004-09-02 23:40 ` Paul Jakma 2004-09-03 6:06 ` Hans Reiser 2 siblings, 1 reply; 1392+ messages in thread From: Paul Jakma @ 2004-09-02 23:40 UTC (permalink / raw) To: Hans Reiser Cc: David Masover, Horst von Brand, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Tue, 31 Aug 2004, Hans Reiser wrote: > learning how to turn the hand crank to start your car. Just the look of "tar > -xf" turns them away. Crypto-Geek gobbledy-gook is what it is. Let's value > their time, there are a lot of them. These kinds of people would use something like file-roller in GNOME, and they'd care not a jot whether file-rollers used built-in tar support, and/or openat() or via file-as-dir fs support. All they care about is that they get a window with icons for their files that they can click on. How it's done does not matter - it only matters to those who must write the code. > Hans regards, -- Paul Jakma paul@clubi.ie paul@jakma.org Key ID: 64A2FF6A Fortune: I haven't lost my mind; I know exactly where I left it. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 23:40 ` Paul Jakma @ 2004-09-03 6:06 ` Hans Reiser 2004-09-03 7:00 ` viro 0 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-09-03 6:06 UTC (permalink / raw) To: Paul Jakma Cc: David Masover, Horst von Brand, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Linus, did you get the email in which I (I think....) answered all the problems with files as directories that you and Viro had raised? You and Viro did not respond to it.... Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-03 6:06 ` Hans Reiser @ 2004-09-03 7:00 ` viro 2004-09-08 7:51 ` Sriram Karra 0 siblings, 1 reply; 1392+ messages in thread From: viro @ 2004-09-03 7:00 UTC (permalink / raw) To: Hans Reiser Cc: Paul Jakma, David Masover, Horst von Brand, Pavel Machek, Jamie Lokier, Chris Wedgwood, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Sep 02, 2004 at 11:06:39PM -0700, Hans Reiser wrote: > Linus, did you get the email in which I (I think....) answered all the > problems with files as directories that you and Viro had raised? > > You and Viro did not respond to it.... Message-ID, please. There was a lot of mails from you in this thread and I probably have missed one that did address *any* problems. If such mail had ever existed, that is. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-03 7:00 ` viro @ 2004-09-08 7:51 ` Sriram Karra 2004-09-08 9:36 ` viro 0 siblings, 1 reply; 1392+ messages in thread From: Sriram Karra @ 2004-09-08 7:51 UTC (permalink / raw) To: viro Cc: Hans Reiser, Paul Jakma, David Masover, Horst von Brand, Pavel Machek, Jamie Lokier, Chris Wedgwood, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List viro@parcelfarce.linux.theplanet.co.uk writes: > On Thu, Sep 02, 2004 at 11:06:39PM -0700, Hans Reiser wrote: >> Linus, did you get the email in which I (I think....) answered all >> the problems with files as directories that you and Viro had >> raised? >> >> You and Viro did not respond to it.... > > Message-ID, please. There was a lot of mails from you in this > thread and I probably have missed one that did address *any* > problems. If such mail had ever existed, that is. Perhaps this is one? Message-ID: <413578C9.8020305@namesys.com> ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-08 7:51 ` Sriram Karra @ 2004-09-08 9:36 ` viro 0 siblings, 0 replies; 1392+ messages in thread From: viro @ 2004-09-08 9:36 UTC (permalink / raw) To: Sriram Karra Cc: Hans Reiser, Paul Jakma, David Masover, Horst von Brand, Pavel Machek, Jamie Lokier, Chris Wedgwood, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, Sep 08, 2004 at 01:21:45PM +0530, Sriram Karra wrote: > Perhaps this is one? Message-ID: <413578C9.8020305@namesys.com> OK... One note before replying: current code deadlocks even if you make ->link() *ALWAYS* return an error. It doesn't get to calling the method. No amount of "disallow hard links to <something>" is going to help here, obviously. <quote> Cycle detection: We should either 1) make hard links only link to the file aspect of the file-directory duality, and persons who want to link to the directory aspect must use symlinks (best short term answer), or 2) ask Alexander Smith to help us with applying his cycle detection algorithm and gain the benefit of being able to hard link to directories (if it works well, best long term answer). </quote> ... which doesn't address the problem at all. The question is what to do with seeing directory "aspect..." in more than one place when we have many links to file in question. So much for (1). And (2) is not feasible with on-disk fs both due to memory, CPU and IO costs _and_ due to exclusion from hell you'll need to make it safe. Re: ambiguity - lots and lots of handwaving on both sides. FWIW, I agree with Hans in one (and only one) respect here - openat() as a primary API (and not a convenient libc function) is an atrocity. Simply because it doesn't address operations beyond open (unlinkat(2), anyone?). However, I still haven't seen any strong arguments for need of this "metas" stuff _or_ the need to export mode/ownership as files, both for regular files and for directories. Aside of "we can do that" [if we solve the locking issues] and "xattrs are atrocious" [yes, they are; it doesn't make alternative mechanism any better] there was nothing that even pretended to be a technical reason. Note that we also have fun issues with device nodes (Linus' "show partitions" vs. "show metadata from hosting filesystem"), which makes it even more dubious. We also have symlinks to deal with (do they have attributes? where should that be exported?). Reserved names have one more problem: to be useful, they'd have to be hardcoded into applications. And that will create hell with use of such applications on existing filesystems. Again, no feasible scheme to deal with that in userland code had been proposed so far, AFAICS. Locking: see above - links to regular files would create directories seen in many places. With all related issues... ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 1:14 ` David Masover 2004-09-01 4:59 ` Pavel Machek 2004-09-01 5:50 ` Hans Reiser @ 2004-09-05 11:33 ` Tonnerre 2 siblings, 0 replies; 1392+ messages in thread From: Tonnerre @ 2004-09-05 11:33 UTC (permalink / raw) To: David Masover Cc: Horst von Brand, Pavel Machek, Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 485 bytes --] Salut, On Tue, Aug 31, 2004 at 08:14:33PM -0500, David Masover wrote: > It's not about the kernel, it's about the interface. And see my other mail: > cat foo.zip/README > less foo.zip/contents/bar.c > is a lot easier than > lynx http://google.com/search?q=zip > emerge zip > man zip > unzip foo.zip > cat bar.c > which already assumes quite a lot of expertise. Most distros ship Ziv's packer by default. telnet dict.org 2628 DEFINE * "creeping featurism" QUIT Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 8:21 ` Pavel Machek 2004-08-31 19:31 ` Horst von Brand @ 2004-09-01 1:10 ` David Masover 1 sibling, 0 replies; 1392+ messages in thread From: David Masover @ 2004-09-01 1:10 UTC (permalink / raw) To: Pavel Machek Cc: Jamie Lokier, Chris Wedgwood, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Pavel Machek wrote: | Hi! | | |>| uservfs does |>| |>| cd foo.deb#uar |>cd foo.deb/ar |>| vs. |>| cd foo.deb#udeb |>cd foo.deb/deb |> |>and why would you want that, instead of just: |>cd foo.deb # for the ar |>dpkg -i foo.deb # for the deb | | | Because I want to see contents of that .deb package, nicely parsed? | | |>Just want to extract the tar file? Maybe something like |>cat foo.tgz/gunzip |>In which case (of course) foo.tgz/gunzip has exactly the same directory |>contents as foo.tgz | | | Yes, that would work. | | |>In fact, for just about any syntax anyone could suggest, I can't really |>see why you can't just replace all weird symbols with a slash and a |>symbol. Instead of |> foo.tgz#utar |>you have |> foo.tgz/#/utar |>Only difference is, some things which used to require special tools can |>now be serviced by less than what's in busybox. | | | That would work, too. I do not get your comment about busybox. Busybox is a bare minimum of unix tools, yet still includes things like tar/bz2. With such a system, you can use the most basic (sub-GNU) versions of tools like cp, cat, sh, ls, and so on, to do much more complex operations. What's more, you can see everything you can do to a file just by looking at it. I can certainly imagine doing cat foo.tgz/metas/README rather than a google search for tgz, followed by a "man tar" and "man gzip". Assuming I know nothing about tar/gzip files, it's a lot easier as a new user to have one common interface for everything -- including docs. Fewer commands to memorize, faster to learn new stuff. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTUha3gHNmZLgCUhAQKRJw/7BUEm41gKZGoVj7gOkcUVe/SRdTs9WtTL jkjCX8Zzr/1nnAKTHLqNJEiAJ68Ndf+dRhDfL5y6dDHhOy/Dx+38SzsWz+dQnMJ0 Fiq5fFDO/DTd0UPecXYPhBrtVeCdfjyyENhgRgJud6D8F39qvLWrZFUsIkjAA3lj nzZ/48jRPDWT/mF6JJmv0RfVvNtMWh9HUcW/i0+n4WZU1yFp6jyiF5ExCOP84z3x yXQ4Q9hnjgt52MDJTrGsCDvfCSQscQilxvXbl0iaozquIhG9Jz1XNYpij2XUygFi RDc1LA1k74VvbWahlkdfbTI+3P4qeMYGBblmfwXVDRwW0i4NH7X0/6xcnr1QjaK/ f/7EHBz0wnkIG4OpcGWzu9ik4WOR0WXapKrd938XC/mG+WzIluJlQsxG1A2HoRqg R66E/mVMVvHJWnJjrW6s10WIemPiKnuXgeIVPifsw81DLXlZ3wxjlZU3iuo7xked O6wNDFQBWQR8kv+EMDIt3gtbUdB+5KpxGR+Yli0AjxEdpdk3oZTfe5XQiNdJhcl5 xrcomVKXGvR96IlNUwHy1EsQk5LySMl101vJIS4z6i7lAIM5CQ1d6eOZrPp/DBOP Rw0KoQaiPd3SKNMgoyWSKQL6h85i1PRUN62WH0T2NRU1fZ3EazpiRZr+fuaFqFT9 Wk2lkDvm5yw= =ShWo -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:05 ` Chris Wedgwood 2004-08-26 11:02 ` Jamie Lokier @ 2004-08-27 21:01 ` Pavel Machek 1 sibling, 0 replies; 1392+ messages in thread From: Pavel Machek @ 2004-08-27 21:01 UTC (permalink / raw) To: Chris Wedgwood Cc: Jamie Lokier, viro, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hi! > > One of the big potential uses for file-as-directory is to go inside > > archive files, ELF files, .iso files and so on in a convenient way. > > Arguably this belongs in userspace --- and people have put it there. > uservfs.sf.net, using existing coda hooks. There are 2+ very similar projects. Advantage is we can do ftpfs, too. Having ftp done by > /ftp cd /ftp/sunsite would be too ugly for reiser4 plugins... Or not? -- 64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 0:30 ` viro 2004-08-26 1:00 ` Jamie Lokier @ 2004-08-26 1:13 ` Chris Wright 1 sibling, 0 replies; 1392+ messages in thread From: Chris Wright @ 2004-08-26 1:13 UTC (permalink / raw) To: viro Cc: Jamie Lokier, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List * viro@parcelfarce.linux.theplanet.co.uk (viro@parcelfarce.linux.theplanet.co.uk) wrote: > On Thu, Aug 26, 2004 at 01:11:52AM +0100, Jamie Lokier wrote: > > Is this a problem if we treat entering a file-as-directory as crossing > > a mount point (i.e. like auto-mounting)? > > Yes - mountpoints can't be e.g. unlinked. Could it be essentially MNT_DETACH'd? ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:42 ` viro 2004-08-25 21:00 ` Christophe Saout 2004-08-25 21:00 ` Linus Torvalds @ 2004-08-26 8:43 ` Hans Reiser 2 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 8:43 UTC (permalink / raw) To: viro Cc: Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List, Jason Holt viro@parcelfarce.linux.theplanet.co.uk wrote: >On Wed, Aug 25, 2004 at 01:22:55PM -0700, Linus Torvalds wrote: > > >>On Wed, 25 Aug 2004, Christoph Hellwig wrote: >> >> >>>For one thing _I_ didn't decide about xattrs anyway. And I still >>>haven't seen a design from you on -fsdevel how you try to solve the >>>problems with files as directories. >>> >>> >>Hey, files-as-directories are one of my pet things, so I have to side with >>Hans on this one. I think it just makes sense. A hell of a lot more sense >>than xattrs, anyway, since it allows scripts etc standard tools to touch >>the attributes. >> >>It's the UNIX way. >> >> > >Not if you allow link(2) on them. And not if you design and market your >stuff as a general-purpose backdoor into kernel. > What backdoor? Please spell it out. Plugins are not dynamically loadable. > Note how *EVERY* *DAMN* >*OPERATION* is made possible to override by "plugins". Which is the reason >for deadlocks in question, BTW. > >Don't fool yourself - that's what Hans is selling. Target market: ISV. > > Hunh? No, target market is hackers who like to spend a weekend dreaming up funky new kinds of files. One guy (Jason Holt, clever guy) came up with an idea for write only files for which even root cannot read the parts of the file written prior to the time root was achieved, because the encryption key is changed in a forward computable only direction with every write, and the start key is kept on another computer. Lots of folks will have plugins I would never dream of. Think of photoshop plugins, and what plugins did for photoshop. Same thing will happen to reiser4 next year (or earlier). >Marketed product: a set of hooks, the wider the better, no matter how >little sense it makes. The reason for doing that outside of core kernel: >bypassing any review and being able to control the product being sold (see >above). > >Shame that it got an actual filesystem mixed in with the marketing plans >and general insanity... > > > > I am one of those free software hippy-child anarchists who thinks that random people should come up with ideas and contribute them. You understand me well.;-) ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:22 ` Linus Torvalds ` (2 preceding siblings ...) 2004-08-25 20:42 ` viro @ 2004-08-25 21:52 ` Matt Mackall 2004-08-25 22:21 ` Linus Torvalds 2004-08-26 0:18 ` Mikulas Patocka ` (3 subsequent siblings) 7 siblings, 1 reply; 1392+ messages in thread From: Matt Mackall @ 2004-08-25 21:52 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, Aug 25, 2004 at 01:22:55PM -0700, Linus Torvalds wrote: > > > On Wed, 25 Aug 2004, Christoph Hellwig wrote: > > > > For one thing _I_ didn't decide about xattrs anyway. And I still > > haven't seen a design from you on -fsdevel how you try to solve the > > problems with files as directories. > > Hey, files-as-directories are one of my pet things, so I have to side with > Hans on this one. I think it just makes sense. A hell of a lot more sense > than xattrs, anyway, since it allows scripts etc standard tools to touch > the attributes. > > It's the UNIX way. I thought the UNIX way is "everything's a file", not "everything's a directory". > Will it potentially break something? Sure. Do we care? Me, I'll take that > kind of extension _any_ day over xattrs, that are fundamentally flawed in > my opinion and totally useless. There's always the option that they're both broken. -- Mathematics is the supreme nostalgia of our time. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 21:52 ` Matt Mackall @ 2004-08-25 22:21 ` Linus Torvalds 0 siblings, 0 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-08-25 22:21 UTC (permalink / raw) To: Matt Mackall Cc: Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, 25 Aug 2004, Matt Mackall wrote: > > > > It's the UNIX way. > > I thought the UNIX way is "everything's a file", not "everything's a > directory". It really was. Directories were historically largely just files too, although with the special "lookup" operation. Historic unix didn't have readdir/rmdir/mkdir/rename or really much _any_ special directory handling. Directories were just files, and you read them like files. Of course, even in that early unix, "directories" were very much a reality even apart from the fact that they happened to be implemented pretty much like files. Nobody has ever claimed that the UNIX way is "everything is _one_ file", after all ;) > > Will it potentially break something? Sure. Do we care? Me, I'll take that > > kind of extension _any_ day over xattrs, that are fundamentally flawed in > > my opinion and totally useless. > > There's always the option that they're both broken. Yes. Highly likely. However, something like that _does_ end up what a Windows fileserver wants. IOW, even if it's broken, _something_ is likely forced on us by that nasty thing we call "real users". Damn them. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:22 ` Linus Torvalds ` (3 preceding siblings ...) 2004-08-25 21:52 ` Matt Mackall @ 2004-08-26 0:18 ` Mikulas Patocka 2004-08-26 0:27 ` Linus Torvalds 2004-08-26 0:57 ` Rik van Riel 2004-08-26 8:43 ` Hans Reiser ` (2 subsequent siblings) 7 siblings, 2 replies; 1392+ messages in thread From: Mikulas Patocka @ 2004-08-26 0:18 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Wed, 25 Aug 2004, Linus Torvalds wrote: > On Wed, 25 Aug 2004, Christoph Hellwig wrote: > > > > For one thing _I_ didn't decide about xattrs anyway. And I still > > haven't seen a design from you on -fsdevel how you try to solve the > > problems with files as directories. > > Hey, files-as-directories are one of my pet things, so I have to side with > Hans on this one. I think it just makes sense. A hell of a lot more sense > than xattrs, anyway, since it allows scripts etc standard tools to touch > the attributes. > > It's the UNIX way. > > And yes, the semantics can _easily_ be solved in very unixy ways. > > One way to solve it is to just realize that a final slash at the end > implies pretty strongly that you want to treat it as a directory. So what > you do is: > > - without the slash, a file-as-dir won't open with O_DIRECTORY (ENOTDIR) > - with the slash, it won't open _without_ O_DIRECTORY (EISDIR) > > Problem solved. Very user-friendly, and very intuitive. Stupid question: who will use it? And why? Anyone can write an userspace library, that implements function set_attribute(char *file, char *attribute, char *value), that creates directory ".attr/file" in file's directory and stores attribute there. (and you can get list of attributes from shell too: ls `echo "$filename" |sed 's/\/\([^\/]*\)$/\/\.attr\/\1/'` ). There's no need to add extra functionality to kernel and filesystem. Advantage: - you don't add bloat to kernel or filesystem - you don't need to teach tar/cp -a/mc about attributes - you won't lose attributes after editing file in vim (it creates another file and renames it over original one) > Will it potentially break something? Sure. Do we care? Me, I'll take that > kind of extension _any_ day over xattrs, that are fundamentally flawed in > my opinion and totally useless. The argument that applications like "tar" > won't understand the file-as-directory thing is _flawed_, since legacy > apps won't understand xattrs either. The only way xattrs are useful is that backup/restore software doesn't have to know about every filesystem with it's specific attributes and every magic ioctl for setting them. Instead it can save/restore filesystem-specific attributes without understanding what do they mean. However there's no need why application should use them. And no application does. I can't imagine anyone shipping an application with "this app requires reiser4" prerequisite. Why should anyone use it if he can store attributes in ".attr" directory or whereever and make the application work on any OS and any filesystem? Mikulas ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 0:18 ` Mikulas Patocka @ 2004-08-26 0:27 ` Linus Torvalds 2004-08-26 0:51 ` Mikulas Patocka 2004-08-26 8:36 ` Hans Reiser 2004-08-26 0:57 ` Rik van Riel 1 sibling, 2 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-08-26 0:27 UTC (permalink / raw) To: Mikulas Patocka Cc: Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 26 Aug 2004, Mikulas Patocka wrote: > > Stupid question: who will use it? And why? > > Anyone can write an userspace library, that implements function > set_attribute(char *file, char *attribute, char *value), that creates > directory ".attr/file" in file's directory and stores attribute there. > (and you can get list of attributes from shell too: > ls `echo "$filename" |sed 's/\/\([^\/]*\)$/\/\.attr\/\1/'` > ). There's no need to add extra functionality to kernel and filesystem. ..and the above is, roughly, what I understand samba etc falls back on. The problem ends up being that the above isn't in any way safe from people moving files around (oops, where did those attributes go?) nor does it have any consistency guarantees. So it only works well if _one_ application does this, and that application follows all the locking rules. Is it enough? It may have to be. > The only way xattrs are useful is that backup/restore software doesn't > have to know about every filesystem with it's specific attributes and > every magic ioctl for setting them. Instead it can save/restore > filesystem-specific attributes without understanding what do they mean. > However there's no need why application should use them. And no > application does. If no application does, then why back them up? Why implement them in the first place? In other words - some apps obviously do want to use the. Sadly. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 0:27 ` Linus Torvalds @ 2004-08-26 0:51 ` Mikulas Patocka 2004-08-26 8:36 ` Hans Reiser 1 sibling, 0 replies; 1392+ messages in thread From: Mikulas Patocka @ 2004-08-26 0:51 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > > The only way xattrs are useful is that backup/restore software doesn't > > have to know about every filesystem with it's specific attributes and > > every magic ioctl for setting them. Instead it can save/restore > > filesystem-specific attributes without understanding what do they mean. > > However there's no need why application should use them. And no > > application does. > > If no application does, then why back them up? Why implement them in the > first place? > > In other words - some apps obviously do want to use the. Sadly. You can add more functionality to filesystem and use xattrs to control it. For example: - acls - compress file - encrypt file (copy user's password into task_struct and use it to encrypt his files) - preallocate file in 4MB contignuous chunks, becuase it needs real time multimedia access - sync/append-only/immutable etc. However there's no need why an application should care whether the file is compressed, whether it has acls, or so. And applications don't. And I think this is the only legitimate use for xattrs. Who else uses them except samba? I don't see how reiser4's hybrids would help. Mikulas ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 0:27 ` Linus Torvalds 2004-08-26 0:51 ` Mikulas Patocka @ 2004-08-26 8:36 ` Hans Reiser 2004-08-26 10:53 ` Jamie Lokier 1 sibling, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 8:36 UTC (permalink / raw) To: Linus Torvalds Cc: Mikulas Patocka, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Streams are quite ugly. However, if you decompose streams into all of the little pieces that are needed to emulate them, the pieces are quite nice. For instance, inheriting stat data from a common parent is nice, and inheritance is nice, and being able to cat dirname/pseudos/cat and get a concatenation of all of the files is nice, and being able to cat dirname/pseudos/tar and get an archive of the directory is nice, and, well, if you decompose all of the features of streams into little features you get a bunch of fun little features much nicer than streams. Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:36 ` Hans Reiser @ 2004-08-26 10:53 ` Jamie Lokier 2004-08-26 10:59 ` Christoph Hellwig 0 siblings, 1 reply; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 10:53 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, Mikulas Patocka, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Hans Reiser wrote: > being able to cat dirname/pseudos/cat and get a > concatenation of all of the files is nice, and being able to cat > dirname/pseudos/tar and get an archive of the directory is nice Yes. Being able to cd into filename.tar.gz and filename.iso is also nice, but all of these features should be supported by the VFS generically, not in any specific filesystem, and there should be a hook to invoke the various fun filesystem-independent handlers by name. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:53 ` Jamie Lokier @ 2004-08-26 10:59 ` Christoph Hellwig 2004-08-26 11:07 ` Jamie Lokier 2004-08-26 11:17 ` Spam 0 siblings, 2 replies; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-26 10:59 UTC (permalink / raw) To: Jamie Lokier Cc: Hans Reiser, Linus Torvalds, Mikulas Patocka, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 11:53:30AM +0100, Jamie Lokier wrote: > Hans Reiser wrote: > > being able to cat dirname/pseudos/cat and get a > > concatenation of all of the files is nice, and being able to cat > > dirname/pseudos/tar and get an archive of the directory is nice > > Yes. Being able to cd into filename.tar.gz and filename.iso is also > nice, but all of these features should be supported by the VFS > generically, not in any specific filesystem, and there should be a > hook to invoke the various fun filesystem-independent handlers by name. It doesn't belong into the kernel at all. If at all it belongs into a userspace filesystems, but even in that case the magic detection of which one to use is kinda hard. You absoutely don't want to hardcode file formats in the kernel. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:59 ` Christoph Hellwig @ 2004-08-26 11:07 ` Jamie Lokier 2004-08-26 11:17 ` Spam 1 sibling, 0 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 11:07 UTC (permalink / raw) To: Christoph Hellwig, Hans Reiser, Linus Torvalds, Mikulas Patocka, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Christoph Hellwig wrote: > > > being able to cat dirname/pseudos/cat and get a > > > concatenation of all of the files is nice, and being able to cat > > > dirname/pseudos/tar and get an archive of the directory is nice > > > > Yes. Being able to cd into filename.tar.gz and filename.iso is also > > nice, but all of these features should be supported by the VFS > > generically, not in any specific filesystem, and there should be a > > hook to invoke the various fun filesystem-independent handlers by name. > > It doesn't belong into the kernel at all. If at all it belongs into a > userspace filesystems, but even in that case the magic detection of > which one to use is kinda hard. You absoutely don't want to hardcode > file formats in the kernel. Oh, I agree userspace should be involved. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:59 ` Christoph Hellwig 2004-08-26 11:07 ` Jamie Lokier @ 2004-08-26 11:17 ` Spam 1 sibling, 0 replies; 1392+ messages in thread From: Spam @ 2004-08-26 11:17 UTC (permalink / raw) To: Christoph Hellwig Cc: Jamie Lokier, Hans Reiser, Linus Torvalds, Mikulas Patocka, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List > On Thu, Aug 26, 2004 at 11:53:30AM +0100, Jamie Lokier wrote: >> Hans Reiser wrote: >> > being able to cat dirname/pseudos/cat and get a >> > concatenation of all of the files is nice, and being able to cat >> > dirname/pseudos/tar and get an archive of the directory is nice >> >> Yes. Being able to cd into filename.tar.gz and filename.iso is also >> nice, but all of these features should be supported by the VFS >> generically, not in any specific filesystem, and there should be a >> hook to invoke the various fun filesystem-independent handlers by name. > It doesn't belong into the kernel at all. If at all it belongs into a > userspace filesystems, but even in that case the magic detection of > which one to use is kinda hard. You absoutely don't want to hardcode > file formats in the kernel. Do you mean user-level file system as a VFS handled by user applications, or a intermediate file system layer between any application and the real file system? The latter would be good enough as it would still be transparent to the applications. ~S ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 0:18 ` Mikulas Patocka 2004-08-26 0:27 ` Linus Torvalds @ 2004-08-26 0:57 ` Rik van Riel 2004-08-26 7:51 ` Helge Hafting ` (2 more replies) 1 sibling, 3 replies; 1392+ messages in thread From: Rik van Riel @ 2004-08-26 0:57 UTC (permalink / raw) To: Mikulas Patocka Cc: Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 26 Aug 2004, Mikulas Patocka wrote: > On Wed, 25 Aug 2004, Linus Torvalds wrote: > > One way to solve it is to just realize that a final slash at the end > > implies pretty strongly that you want to treat it as a directory. So what > > you do is: > > Stupid question: who will use it? And why? I've got a stupid question too. How do you back up these things ? If your backup program reads them as a file and restores them as a file, you might lose your directory-inside-the-file magic. If your backup program dives into the file despite stat() saying it's a file and you restore your backup, how are the "file is a file" semantics preserved ? Obviously this is something that needs to be sorted out at the VFS layer. A filesystem specific backup and restore program isn't desirable, if only because then there'd be no way for Hans's users to switch to reiser5 in 2010 ;) -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 0:57 ` Rik van Riel @ 2004-08-26 7:51 ` Helge Hafting 2004-08-26 9:21 ` Paul Jackson 2004-08-26 8:40 ` Hans Reiser 2004-08-26 9:44 ` Paul Jackson 2 siblings, 1 reply; 1392+ messages in thread From: Helge Hafting @ 2004-08-26 7:51 UTC (permalink / raw) To: Rik van Riel Cc: Mikulas Patocka, Linus Torvalds, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Rik van Riel wrote: >On Thu, 26 Aug 2004, Mikulas Patocka wrote: > > >>On Wed, 25 Aug 2004, Linus Torvalds wrote: >> >> > > > >>>One way to solve it is to just realize that a final slash at the end >>>implies pretty strongly that you want to treat it as a directory. So what >>>you do is: >>> >>> >>Stupid question: who will use it? And why? >> >> > >I've got a stupid question too. How do you back up these >things ? > >If your backup program reads them as a file and restores >them as a file, you might lose your directory-inside-the-file >magic. > >If your backup program dives into the file despite stat() >saying it's a file and you restore your backup, how are the >"file is a file" semantics preserved ? > > > >Obviously this is something that needs to be sorted out at >the VFS layer. A filesystem specific backup and restore >program isn't desirable, if only because then there'd be >no way for Hans's users to switch to reiser5 in 2010 ;) > > > Sure, this sort of thing must be sorted out at the VFS layer. And a backup program working on such a filesystem will need to know that something can be a file, a directory - or both. So an old "tar" won't get this right as it will assume that an object is either file or directory. The change to get it right won't be that big - just notice that an object is both, then backup the ordinary file contents as usual, before recursing into the directory it also provides and backup stuff there as usual. The resulting .tar can of course only be unpacked properly on a fs supporting file-as-directory, similiar to how a .tar of a fs with links only will unpack properly on a fs supporting links. I don't see much problems for userland. Old apps will keep working, as the new features is a superset. Those who care about file-as-directory extras will provide patches for "tar" and friends, after that the extras become useable. Helge Hafting ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 7:51 ` Helge Hafting @ 2004-08-26 9:21 ` Paul Jackson 2004-08-26 10:47 ` Jamie Lokier 2004-08-27 12:33 ` Helge Hafting 0 siblings, 2 replies; 1392+ messages in thread From: Paul Jackson @ 2004-08-26 9:21 UTC (permalink / raw) To: Helge Hafting Cc: riel, mikulas, torvalds, hch, reiser, linux-fsdevel, linux-kernel, flx, reiserfs-list Helge wrote: > So an old "tar" won't get this right as it will assume that an object > is either file or directory. There are many backup apps, not just one. I've written a few myself, none of which will ever be worthy of notice. The sourceforge Topic.System.Archiving.Backup lists 335 projects at present. I find the idea that most backup tools and scripts will silently stop working correctly to be pretty scary. And then there's archiving, installation, distribution, administration, emulation, file system and partition managers, and on and on. === I wonder if we can make this "modal" somehow. The one consistency I see is that apps that want the "enhanced" view need to ask for it, somehow. It is the new views of the data that are being added - let the app announce to the kernel (usually via specialized code in some shared library that the app is using to get the alternate views) that either per-task, or per-file descriptor, it is to see the "enhanced" view, as a side affect of trying to access it. Old stuff, or even new stuff that is content to work with the "classic" view that a file is a single data stream, and that directories only have pathnames, not data, would by default see that view, and see _all_ the data, presented somehow in that view, perhaps as additional files with magic names. This still leaves the breakage that such tools don't know, and don't preserve, the magic linkage between such magic files. But that is much less of an issue, in my view. Programs such as backups that are manipulating the files of apps they know nothing about already have to presume that all the files are important in inscrutable ways, and just be careful to preserve or copy or backup all of them. Yeah - I realize that there will be a few followups denouncing modal architectures. I might even agree with some of them. If this were easy, it would have been done years ago. The onus should be on the new stuff to request the enhanced view, rather than on the old dogs to learn new tricks. -- I won't rest till it's the best ... Programmer, Linux Scalability Paul Jackson <pj@sgi.com> 1.650.933.1373 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 9:21 ` Paul Jackson @ 2004-08-26 10:47 ` Jamie Lokier 2004-08-26 11:19 ` Paul Jackson 2004-08-27 12:33 ` Helge Hafting 1 sibling, 1 reply; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 10:47 UTC (permalink / raw) To: Paul Jackson Cc: Helge Hafting, riel, mikulas, torvalds, hch, reiser, linux-fsdevel, linux-kernel, flx, reiserfs-list Paul Jackson wrote: > I find the idea that most backup tools and scripts will silently > stop working correctly to be pretty scary. Too late. We have xattrs already; many programs don't store them. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:47 ` Jamie Lokier @ 2004-08-26 11:19 ` Paul Jackson 0 siblings, 0 replies; 1392+ messages in thread From: Paul Jackson @ 2004-08-26 11:19 UTC (permalink / raw) To: Jamie Lokier Cc: helge.hafting, riel, mikulas, torvalds, hch, reiser, linux-fsdevel, linux-kernel, flx, reiserfs-list > Too late. We have xattrs already; many programs don't store them. If by your "too late" you mean we can stop worrying about any more breakage of file system utilities, because there exists an example in which some were already broken, then you are absolutely wrong. Just because we caused some breakage doesn't give us license to cause even more. -- I won't rest till it's the best ... Programmer, Linux Scalability Paul Jackson <pj@sgi.com> 1.650.933.1373 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 9:21 ` Paul Jackson 2004-08-26 10:47 ` Jamie Lokier @ 2004-08-27 12:33 ` Helge Hafting 1 sibling, 0 replies; 1392+ messages in thread From: Helge Hafting @ 2004-08-27 12:33 UTC (permalink / raw) To: Paul Jackson Cc: riel, mikulas, torvalds, hch, reiser, linux-fsdevel, linux-kernel, flx, reiserfs-list Paul Jackson wrote: >Helge wrote: > > >>So an old "tar" won't get this right as it will assume that an object >>is either file or directory. >> >> > >There are many backup apps, not just one. I've written a few myself, >none of which will ever be worthy of notice. The sourceforge >Topic.System.Archiving.Backup lists 335 projects at present. > >I find the idea that most backup tools and scripts will silently >stop working correctly to be pretty scary. > > They won't stop working, they will merely not support the new features. That is only a problem if you actually use those features. If all you have is plain files and plain directories - no problem. Not if files-as-directories are implemented right. Helge Hafting ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 0:57 ` Rik van Riel 2004-08-26 7:51 ` Helge Hafting @ 2004-08-26 8:40 ` Hans Reiser 2004-08-26 14:46 ` Lars Marowsky-Bree 2004-08-26 9:44 ` Paul Jackson 2 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 8:40 UTC (permalink / raw) To: Rik van Riel Cc: Mikulas Patocka, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Rik van Riel wrote: > > >If your backup program dives into the file despite stat() >saying it's a file and you restore your backup, how are the >"file is a file" semantics preserved ? > >Obviously this is something that needs to be sorted out at >the VFS layer. > It needs to be sorted out, whether it is sorted out at the VFS layer is unimportant. > A filesystem specific backup and restore >program isn't desirable, if only because then there'd be >no way for Hans's users to switch to reiser5 in 2010 ;) > > > It might be that we need a filenameA/metas/backup method for all of our file plugins, which if cat'd gives a set of instructions which if executed are adequate for restoring filenameA. Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:40 ` Hans Reiser @ 2004-08-26 14:46 ` Lars Marowsky-Bree 0 siblings, 0 replies; 1392+ messages in thread From: Lars Marowsky-Bree @ 2004-08-26 14:46 UTC (permalink / raw) To: Hans Reiser, Rik van Riel Cc: Mikulas Patocka, Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On 2004-08-26T01:40:29, Hans Reiser <reiser@namesys.com> said: > >Obviously this is something that needs to be sorted out at > >the VFS layer. > > > It needs to be sorted out, whether it is sorted out at the VFS layer is > unimportant. So what exactly is wrong with sorting it out at the VFS layer, and why do you _insist_ on sorting it in the reiserfs4 core? I'm missing something, please fill me in on the details. Sincerely, Lars Marowsky-Brée <lmb@suse.de> -- High Availability & Clustering \\\ /// SUSE Labs, Research and Development \honk/ SUSE LINUX AG - A Novell company \\// ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 0:57 ` Rik van Riel 2004-08-26 7:51 ` Helge Hafting 2004-08-26 8:40 ` Hans Reiser @ 2004-08-26 9:44 ` Paul Jackson 2 siblings, 0 replies; 1392+ messages in thread From: Paul Jackson @ 2004-08-26 9:44 UTC (permalink / raw) To: Rik van Riel Cc: mikulas, torvalds, hch, reiser, linux-fsdevel, linux-kernel, flx, reiserfs-list Rik van Riel writes: > If your backup program reads them as a file and restores > them as a file, you might lose your directory-inside-the-file > magic. Encode the magic in the names, by stealing a bit of the existing filename space to encode it. Such works pretty well as part of the magic to map long filenames into DOS 8.3 names on my FAT partitions. Apps linked with the appropriate Windows library see nice fancy long names. The rest of the world, including DOS apps and my Unix backup scripts, see the primitive 8.3 names, including one or a few extra files per directory, which are nothing special to them. So long as these other apps don't presume to know that they can keep some of the files in an apps directory, and drop others, then it works well enough. And no self-respecting general purpose backup program is going to presume such knowledge anyway. -- I won't rest till it's the best ... Programmer, Linux Scalability Paul Jackson <pj@sgi.com> 1.650.933.1373 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:22 ` Linus Torvalds ` (4 preceding siblings ...) 2004-08-26 0:18 ` Mikulas Patocka @ 2004-08-26 8:43 ` Hans Reiser 2004-08-26 13:53 ` Michael Halcrow 2004-08-29 14:20 ` Alan Cox 7 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 8:43 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Linus Torvalds wrote: >On Wed, 25 Aug 2004, Christoph Hellwig wrote: > > >>For one thing _I_ didn't decide about xattrs anyway. And I still >>haven't seen a design from you on -fsdevel how you try to solve the >>problems with files as directories. >> >> > >Hey, files-as-directories are one of my pet things, so I have to side with >Hans on this one. I think it just makes sense. A hell of a lot more sense >than xattrs, anyway, since it allows scripts etc standard tools to touch >the attributes. > >It's the UNIX way. > >And yes, the semantics can _easily_ be solved in very unixy ways. > >One way to solve it is to just realize that a final slash at the end >implies pretty strongly that you want to treat it as a directory. So what >you do is: > > - without the slash, a file-as-dir won't open with O_DIRECTORY (ENOTDIR) > - with the slash, it won't open _without_ O_DIRECTORY (EISDIR) > >Problem solved. Very user-friendly, and very intuitive. > >Will it potentially break something? Sure. Do we care? Me, I'll take that >kind of extension _any_ day over xattrs, that are fundamentally flawed in >my opinion and totally useless. The argument that applications like "tar" >won't understand the file-as-directory thing is _flawed_, since legacy >apps won't understand xattrs either. > >Oh, add a O_NOXATTRS flag to force a path lookup to only use regular >directories, the same way we have O_NOFOLLOW and friends. That allows >people to see the difference, if they care (ie a file server might decide >that it doesn't want to expose things like this). > > I think we should require people to care enough to supply an O_NOMETAS flag to see the difference. >I never liked the xattr stuff. It makes little sense, and is totally >useless for 99.9999% of everything. I still don't see the point of it, >except for samba. Ugly. > > Linus > > > > ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:22 ` Linus Torvalds ` (5 preceding siblings ...) 2004-08-26 8:43 ` Hans Reiser @ 2004-08-26 13:53 ` Michael Halcrow 2004-08-26 22:26 ` Hans Reiser 2004-08-29 14:20 ` Alan Cox 7 siblings, 1 reply; 1392+ messages in thread From: Michael Halcrow @ 2004-08-26 13:53 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List [-- Attachment #1: Type: text/plain, Size: 5340 bytes --] If I may chime in here... On Wed, Aug 25, 2004 at 01:22:55PM -0700, Linus Torvalds wrote: > On Wed, 25 Aug 2004, Christoph Hellwig wrote: > > > > For one thing _I_ didn't decide about xattrs anyway. And I still > > haven't seen a design from you on -fsdevel how you try to solve > > the problems with files as directories. > > Hey, files-as-directories are one of my pet things, so I have to > side with Hans on this one. I think it just makes sense. A hell of a > lot more sense than xattrs, anyway, since it allows scripts etc > standard tools to touch the attributes. > > It's the UNIX way. This is an issue that directly affects work I am doing in extended cryptfs: http://www.linuxsymposium.org/2004/view_abstract.php?content_key=55 http://halcrow.us/~mhalcrow/ols2004.pdf http://halcrow.us/~mhalcrow/ols_cryptfs.sxi The basic idea is that the cryptographic context for every file is correlated with the individual file via xattr's. A file is a unit of data that should, as it stands, contain all the information requisite for the encrypting filesystem layer to transparently decrypt (and encrypt, when the file is written to). This allows for a key->file granularity, as opposed to a key->block device (dm-crypt) or a key->mount point (CFS) granularity. My grand vision is to have a policy that determines whether or not the encrypted version of the file or the decrypted version of the file is read, dependent on whether or not the file is leaving the security domain (the storage device under the control of the currently running kernel). For example, if the ``cp'' command is copying a file from a filesystem mounted from /dev/hda1 to a filesystem mounted from /dev/fd0, then the policy would indicate that (unless otherwise noted in the .cryptfsrc file in the root of the filesystem mounted from /dev/fd0, which might also contain the default security context for that filesystem or directory - like whose public keys should be used to encrypt the symmetric key for data) the file is leaving the security domain, and the encrypted contents of the file should be given to cp. Same with mutt reading an email attachment (as opposed to, say, .muttrc, where, more likely than not, the unencrypted version is wanted). The goal is to enable an ``encrypted by default'' policy, in which files on the storage devices are independent encrypted units that remain encrypted until an application that actually needs to see the decrypted contents opens them. Then the encryption and decryption is done transparently by the fs layer, as long as the user has the right keys. Extended attributes seem like a natural way to store this context. Once you consider that you can have a crypto context for each file, you can start doing other neat tricks, like keyed hashes over extents within the file, to allow for dynamic integrity verification during the read. If an offset of 1.5 gigabytes into a 2-gigabyte has been tampered with, then that tampering will be caught when that portion of the file is read; you don't have to verify the hash of the entire 2-gigabyte file at the time of the open. Of course, this would very rapidly overrun the available xattr storage size. And so to realistically implement something like this, some new underlying file format is in order. In any case, the issue of userspace applications supporting extended attributes is key to the viability of this approach. If cp, uuencode, tar, or what not do not preserve the extended attributes, then the crypto context is lost, and the file is unreadable. So the $64,000 question is, just how committed is the community to this whole concept of extended attributes? From this point, should I assume that good xattr support is forthcoming, or should I abandon the idea of using xattr's for this altogether? One solution I've been kicking around is to make cryptfs GnuPG-compatible. Not only would this eliminate the need to store some of the crypto context in the xattr set, but it would also preserve the crypto context with apps that don't know about xattr's, and it would be possible for users who are not running cryptfs to read the files with gpg. Keyed hashes over extents would be doable if GnuPG allowed for opaque data blobs in the file that gpg would just ignore when decrypting the file (gnupg-dev list had technical issues last time I tried to post these ideas to it - any gpg guys around that can comment on this?). > I never liked the xattr stuff. It makes little sense, and is totally > useless for 99.9999% of everything. I still don't see the point of it, > except for samba. Ugly. If xattr's wind up getting supported by a certain critical mass of applications, then they are somewhat useful for me, although, as currently implemented, are insufficient for what I really need (keyed hashes over extents require too much space). BTW, early this week I migrated cryptfs over to use David Howell's new keyring, which is working out nicely. Mike .___________________________________________________________________. Michael A. Halcrow Security Software Engineer, IBM Linux Technology Center GnuPG Fingerprint: 05B5 08A8 713A 64C1 D35D 2371 2D3C FDDA 3EB6 601D [-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 13:53 ` Michael Halcrow @ 2004-08-26 22:26 ` Hans Reiser 0 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 22:26 UTC (permalink / raw) To: Michael Halcrow Cc: Linus Torvalds, Christoph Hellwig, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Michael Halcrow wrote: > This is an issue that directly affects work I am doing in extended > >cryptfs: > >http://www.linuxsymposium.org/2004/view_abstract.php?content_key=55 >http://halcrow.us/~mhalcrow/ols2004.pdf >http://halcrow.us/~mhalcrow/ols_cryptfs.sxi > > > Reiser4 has an encryption plugin that will ship sometime this year. You might want to talk to edward@namesys.com about it. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:22 ` Linus Torvalds ` (6 preceding siblings ...) 2004-08-26 13:53 ` Michael Halcrow @ 2004-08-29 14:20 ` Alan Cox 7 siblings, 0 replies; 1392+ messages in thread From: Alan Cox @ 2004-08-29 14:20 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, Hans Reiser, linux-fsdevel, Linux Kernel Mailing List, Alexander Lyamin aka FLX, ReiserFS List On Mer, 2004-08-25 at 21:22, Linus Torvalds wrote: > - without the slash, a file-as-dir won't open with O_DIRECTORY (ENOTDIR) > - with the slash, it won't open _without_ O_DIRECTORY (EISDIR) > > Problem solved. Very user-friendly, and very intuitive. Breaks bash and almost all other file completion 8) ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:08 ` Christoph Hellwig 2004-08-25 20:19 ` Christoph Hellwig 2004-08-25 20:22 ` Linus Torvalds @ 2004-08-25 20:35 ` Alex Zarochentsev 2004-08-25 20:51 ` Christoph Hellwig 2 siblings, 1 reply; 1392+ messages in thread From: Alex Zarochentsev @ 2004-08-25 20:35 UTC (permalink / raw) To: Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Wed, Aug 25, 2004 at 10:08:59PM +0200, Christoph Hellwig wrote: > On Wed, Aug 25, 2004 at 12:53:28PM -0700, Hans Reiser wrote: > > You ignored everything I said during the discussion of xattrs about how > > there is no need to have attributes when you can just have files and > > directories, and that xattrs reflected a complete ignorance of name > > space design principles. > > Actually in most of the discussion you simply didn't participate. While > xattrs might not be the nicest interface they have the advantag of not > breaking the SuS assumption of what directories vs files are, and they > do not break the Linux O_DIRECTORY semantics that are defined and need > to solve real-world races either. Reiser4 may have a mount option for whoose who like or have to use traditional O_DIRECTORY semantics. There would be no /metas under non-directories, if user wants that. [ ... ] -- Alex. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:35 ` Alex Zarochentsev @ 2004-08-25 20:51 ` Christoph Hellwig 2004-08-25 23:54 ` Jamie Lokier 2004-08-26 8:43 ` Hans Reiser 0 siblings, 2 replies; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-25 20:51 UTC (permalink / raw) To: Alex Zarochentsev Cc: Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Thu, Aug 26, 2004 at 12:35:17AM +0400, Alex Zarochentsev wrote: > Reiser4 may have a mount option for whoose who like or have to use traditional > O_DIRECTORY semantics. There would be no /metas under non-directories, if user > wants that. Again, O_DIRECTORY was added to solve a real-world race, not just for the sake of it. If you really want to add that option I'll research the CAN number for you so you can named it after that - or just call it -o insecure directly. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:51 ` Christoph Hellwig @ 2004-08-25 23:54 ` Jamie Lokier 2004-08-26 8:44 ` Hans Reiser 2004-08-26 12:43 ` Christoph Hellwig 2004-08-26 8:43 ` Hans Reiser 1 sibling, 2 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-08-25 23:54 UTC (permalink / raw) To: Christoph Hellwig, Alex Zarochentsev, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Christoph Hellwig wrote: > > Reiser4 may have a mount option for whoose who like or have to use > > traditional O_DIRECTORY semantics. There would be no /metas under > > non-directories, if user wants that. > > Again, O_DIRECTORY was added to solve a real-world race, not just for > the sake of it. If you really want to add that option I'll research the > CAN number for you so you can named it after that - or just call it -o > insecure directly. man open(2) explains that O_DIRECTORY is used by opendir() to prevent blocking when opening pipes and certain devices*, and should only by used by opendir (of course it isn't only used by opendir, as it's a handy optimisation). In fact O_DIRECTORY is also used by Glibc to optimise away stat() before and fstat() after calls. An O_NODEVICE flag would be equally secure, and more generally useful. It's important that device nodes cannot be opened when O_DIRECTORY is set. This is compatible with reiser4 file-as-directory semantics, but I don't know if reiser4 actually implements this. If it does (and it should) then there is no device blocking problem. That leaves only the optimisation of fstat() in opendir(). But that begs the question: do we want opendir() to succeed on a reiser4 file? It's up to us to decide if we like that semantic, or prefer a different one such as the one I described before (path search enters file-as-directory, but opendir() directly on it fails), or the Sun syscalls someone mentioned. -- Jamie * Aside: O_NONBLOCK|O_NOCTTY is an effective way to prevent blocking on many systems. It's best if you can avoid opening devices at all though. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 23:54 ` Jamie Lokier @ 2004-08-26 8:44 ` Hans Reiser 2004-08-26 12:43 ` Christoph Hellwig 1 sibling, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 8:44 UTC (permalink / raw) To: Jamie Lokier Cc: Christoph Hellwig, Alex Zarochentsev, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Jamie Lokier wrote: >Christoph Hellwig wrote: > > >>>Reiser4 may have a mount option for whoose who like or have to use >>>traditional O_DIRECTORY semantics. There would be no /metas under >>>non-directories, if user wants that. >>> >>> >>Again, O_DIRECTORY was added to solve a real-world race, not just for >>the sake of it. If you really want to add that option I'll research the >>CAN number for you so you can named it after that - or just call it -o >>insecure directly. >> >> > >man open(2) explains that O_DIRECTORY is used by opendir() to prevent >blocking when opening pipes and certain devices*, and should only by >used by opendir (of course it isn't only used by opendir, as it's a >handy optimisation). > >In fact O_DIRECTORY is also used by Glibc to optimise away stat() >before and fstat() after calls. > >An O_NODEVICE flag would be equally secure, and more generally useful. > >It's important that device nodes cannot be opened when O_DIRECTORY is >set. This is compatible with reiser4 file-as-directory semantics, but >I don't know if reiser4 actually implements this. If it does (and it >should) then there is no device blocking problem. > >That leaves only the optimisation of fstat() in opendir(). > >But that begs the question: do we want opendir() to succeed on a reiser4 file? > > Yes, we do. Why not? Not is more complex..... >It's up to us to decide if we like that semantic, or prefer a >different one such as the one I described before (path search enters >file-as-directory, but opendir() directly on it fails), or the Sun >syscalls someone mentioned. > >-- Jamie > > >* Aside: O_NONBLOCK|O_NOCTTY is an effective way to prevent blocking on >many systems. It's best if you can avoid opening devices at all though. > > > > ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 23:54 ` Jamie Lokier 2004-08-26 8:44 ` Hans Reiser @ 2004-08-26 12:43 ` Christoph Hellwig 1 sibling, 0 replies; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-26 12:43 UTC (permalink / raw) To: Jamie Lokier Cc: Christoph Hellwig, Alex Zarochentsev, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Thu, Aug 26, 2004 at 12:54:09AM +0100, Jamie Lokier wrote: > Christoph Hellwig wrote: > > > Reiser4 may have a mount option for whoose who like or have to use > > > traditional O_DIRECTORY semantics. There would be no /metas under > > > non-directories, if user wants that. > > > > Again, O_DIRECTORY was added to solve a real-world race, not just for > > the sake of it. If you really want to add that option I'll research the > > CAN number for you so you can named it after that - or just call it -o > > insecure directly. > > man open(2) explains that O_DIRECTORY is used by opendir() to prevent > blocking when opening pipes and certain devices*, and should only by > used by opendir (of course it isn't only used by opendir, as it's a > handy optimisation). > > In fact O_DIRECTORY is also used by Glibc to optimise away stat() > before and fstat() after calls. > > An O_NODEVICE flag would be equally secure, and more generally useful. It would prevent the open device issue but you'd need another syscall. Anyways, this is an existing API and chaging it now would suddenly making everything that relies on it for that purpose insecure, so it's completely out of option. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:51 ` Christoph Hellwig 2004-08-25 23:54 ` Jamie Lokier @ 2004-08-26 8:43 ` Hans Reiser 2004-08-26 12:41 ` Christoph Hellwig 1 sibling, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 8:43 UTC (permalink / raw) To: Christoph Hellwig Cc: Alex Zarochentsev, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Christoph Hellwig wrote: >On Thu, Aug 26, 2004 at 12:35:17AM +0400, Alex Zarochentsev wrote: > > >>Reiser4 may have a mount option for whoose who like or have to use traditional >>O_DIRECTORY semantics. There would be no /metas under non-directories, if user >>wants that. >> >> > >Again, O_DIRECTORY was added to solve a real-world race, not just for >the sake of it. > Can you supply more details and we will try to reply concretely? Thanks. > If you really want to add that option I'll research the >CAN number for you so you can named it after that - or just call it -o >insecure directly. > > > > > ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:43 ` Hans Reiser @ 2004-08-26 12:41 ` Christoph Hellwig 2004-08-26 13:48 ` Jamie Lokier 0 siblings, 1 reply; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-26 12:41 UTC (permalink / raw) To: Hans Reiser Cc: Christoph Hellwig, Alex Zarochentsev, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Thu, Aug 26, 2004 at 01:43:59AM -0700, Hans Reiser wrote: > >Again, O_DIRECTORY was added to solve a real-world race, not just for > >the sake of it. > > > Can you supply more details and we will try to reply concretely? Thanks. It's really userland programming 101 :) you need to fstat after open to really check that no one swtiched the path below you to a fifo or device, O_DIRECTORY gets that directly in the open call, with the additional benefit of making sure you don't get any of the sideeffects at all that would happen if you're opening a device file. the current reiser4 semantics break that and as soon as you're having a world-writeable (e.g. /tmp) dir on it and someone is doing an opendir on it he's lost. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 12:41 ` Christoph Hellwig @ 2004-08-26 13:48 ` Jamie Lokier 2004-08-26 15:57 ` Christoph Hellwig 0 siblings, 1 reply; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 13:48 UTC (permalink / raw) To: Christoph Hellwig, Hans Reiser, Alex Zarochentsev, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Christoph Hellwig wrote: > you need to fstat after open to really check that no one swtiched the > path below you to a fifo or device, O_DIRECTORY gets that directly in > the open call, with the additional benefit of making sure you don't > get any of the sideeffects at all that would happen if you're opening > a device file. > > the current reiser4 semantics break that and as soon as you're having a > world-writeable (e.g. /tmp) dir on it and someone is doing an opendir > on it he's lost. How does the current reiser4 semantics break that? In a reiser4 filesystem, a file _is_ a directory. opendir() is supposed to succeed on it. There's bound to be some security issue, but I'm not sure what you're getting at with /tmp. What sort of sort of security problem arises with a world-writeable directory such as /tmp, that cannot arise with the standard fs semantics? -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 13:48 ` Jamie Lokier @ 2004-08-26 15:57 ` Christoph Hellwig 2004-08-26 16:06 ` Jamie Lokier 0 siblings, 1 reply; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-26 15:57 UTC (permalink / raw) To: Jamie Lokier Cc: Christoph Hellwig, Hans Reiser, Alex Zarochentsev, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Thu, Aug 26, 2004 at 02:48:12PM +0100, Jamie Lokier wrote: > > the current reiser4 semantics break that and as soon as you're having a > > world-writeable (e.g. /tmp) dir on it and someone is doing an opendir > > on it he's lost. > > How does the current reiser4 semantics break that? > > In a reiser4 filesystem, a file _is_ a directory. > opendir() is supposed to succeed on it. > > There's bound to be some security issue, but I'm not sure what you're > getting at with /tmp. What sort of sort of security problem arises > with a world-writeable directory such as /tmp, that cannot arise with > the standard fs semantics? Actually you are right on that issue because it would open the device/fifo as directory and not device/fifo (in fact I'd had to look at the code again to see whether they actually do this only for files or also for special files) ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 15:57 ` Christoph Hellwig @ 2004-08-26 16:06 ` Jamie Lokier 2004-08-26 16:13 ` Christoph Hellwig 0 siblings, 1 reply; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 16:06 UTC (permalink / raw) To: Christoph Hellwig, Hans Reiser, Alex Zarochentsev, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Christoph Hellwig wrote: > > There's bound to be some security issue, but I'm not sure what you're > > getting at with /tmp. What sort of sort of security problem arises > > with a world-writeable directory such as /tmp, that cannot arise with > > the standard fs semantics? > > Actually you are right on that issue because it would open the > device/fifo as directory and not device/fifo (in fact I'd had to look at > the code again to see whether they actually do this only for files or > also for special files) Are you saying that with reiser4, you can open a device or fifo with O_DIRECTORY? -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:06 ` Jamie Lokier @ 2004-08-26 16:13 ` Christoph Hellwig 2004-08-26 16:35 ` Linus Torvalds 2004-08-29 18:43 ` Alex Zarochentsev 0 siblings, 2 replies; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-26 16:13 UTC (permalink / raw) To: Jamie Lokier Cc: Christoph Hellwig, Hans Reiser, Alex Zarochentsev, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Thu, Aug 26, 2004 at 05:06:38PM +0100, Jamie Lokier wrote: > Christoph Hellwig wrote: > > > There's bound to be some security issue, but I'm not sure what you're > > > getting at with /tmp. What sort of sort of security problem arises > > > with a world-writeable directory such as /tmp, that cannot arise with > > > the standard fs semantics? > > > > Actually you are right on that issue because it would open the > > device/fifo as directory and not device/fifo (in fact I'd had to look at > > the code again to see whether they actually do this only for files or > > also for special files) > > Are you saying that with reiser4, you can open a device or fifo with > O_DIRECTORY? That's what I thought, but as far as I can follow the code this is not actually true. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:13 ` Christoph Hellwig @ 2004-08-26 16:35 ` Linus Torvalds 2004-08-26 17:20 ` Jamie Lokier 2004-08-27 9:18 ` Giuliano Pochini 2004-08-29 18:43 ` Alex Zarochentsev 1 sibling, 2 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-08-26 16:35 UTC (permalink / raw) To: Christoph Hellwig Cc: Jamie Lokier, Hans Reiser, Alex Zarochentsev, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 26 Aug 2004, Christoph Hellwig wrote: > > > Are you saying that with reiser4, you can open a device or fifo with > > O_DIRECTORY? > > That's what I thought, but as far as I can follow the code this is not > actually true. It should be possible to do, though. There's nothing really different in making the "default" (unnamed) fork be a special device or a fifo. And it would be perfectly ok for O_DIRECTORY to open such a file, as long as it opens the directory branch, not the special device. I advocated (long ago) something like this for /dev handling, just because I think it would make sense to have /dev/hda <- special file /dev/hda/part1 <- partition 1 (aka /dev/hda1) which just seems like a very obvious and intuitive interface to me. Of course, we have so much legacy in /dev that there's no real point to doing this, but it's still an appealing approach, I think. But I do take Al's concerns seriously. I like the notion of supporting "containers", but there are undoubtedly serious issues in the notion. I don't strictly know exactly _how_ to implement it sanely (I can talk about using the vfsmnt structure all I like, but the fact is, it's a different thing from a normal mount, and there may be serious problems indeed there). Still, I really do like the idea of merging the notion of file and directory into one notion of "container". I absolutely _detest_ files with internal structure that tools have to know about (ie I hate seeing all those embedded formats that I can't use "grep" on - MIME being one case). I'd much rather see a "group of files" and a "file with a grouping of information". (Now, flattening that "group of files" is obviously needed for serial protocols, so I think MIME/tar/xxxx are fine for _transporting_ data, but I'm saying that outside of transport I really prefer a "collection of files" approach). Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:35 ` Linus Torvalds @ 2004-08-26 17:20 ` Jamie Lokier 2004-08-26 17:29 ` Linus Torvalds 2004-08-27 9:18 ` Giuliano Pochini 1 sibling, 1 reply; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 17:20 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, Hans Reiser, Alex Zarochentsev, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Linus Torvalds wrote: > And it would be perfectly ok for O_DIRECTORY to open such a file, as long > as it opens the directory branch, not the special device. What about: cd /dev/into_directory_branch/ run_setuid_program -> calls pwd pwd opens("."), (".."), ("../..") etc. -> the setuid program thus ends up opening a device or fifo, when it does pwd's path walk. Yes it could use the getcwd syscall, but some programs do their own path walk. > I advocated (long ago) something like this for /dev handling, just because > I think it would make sense to have > > /dev/hda <- special file > /dev/hda/part1 <- partition 1 (aka /dev/hda1) > > which just seems like a very obvious and intuitive interface to me. Of > course, we have so much legacy in /dev that there's no real point to doing > this, but it's still an appealing approach, I think. It also fits the container idea very well: /dev/hda/part1/ <- the filesystem inside partition 1 That's not a contrivance, it's what I'd expect to happen if /dev/hda is an ordinary file containing a disk image: image.bin <- hard disk image (a regular file) image.bin/part1 <- partition 1 image.bin/part1/ <- the filesystem inside partition 1 That's assuming there's a format handler which recognises that image. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:20 ` Jamie Lokier @ 2004-08-26 17:29 ` Linus Torvalds 2004-08-26 17:54 ` viro ` (2 more replies) 0 siblings, 3 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-08-26 17:29 UTC (permalink / raw) To: Jamie Lokier Cc: Christoph Hellwig, Hans Reiser, Alex Zarochentsev, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 26 Aug 2004, Jamie Lokier wrote: > > run_setuid_program > > -> calls pwd > pwd opens("."), (".."), ("../..") etc. Ehh.. Not only does pwd not do that.. (hint: there's a getcwd() system call) > -> the setuid program thus ends up opening a device or fifo, > when it does pwd's path walk. Yes it could use the getcwd > syscall, but some programs do their own path walk. .. but even if it did that, it should use O_DIRECTORY when it did so. If it doesn't, it's broken. So no, it would _not_ open the device or fifo when it did so. The fact is, anything that expects to open a directory should already be opening it with O_DIRECTORY. That said, ".." and "." are special already inside the kernel, and it migth be worth making them automatically imply O_DIRECTORY, since nothing else makes sense anyway. That would fix the case where somebody uses ".." _without_ using O_DIRECTORY. > It also fits the container idea very well: > > /dev/hda/part1/ <- the filesystem inside partition 1 I don't think you can do that. The kernel has no idea how to mount the filesystem. If it's already mounted somewhere else, that's a different issue. Although it might be mounted in several places (as a bind mount) with different writability, I guess, so even then it might be "interesting". Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:29 ` Linus Torvalds @ 2004-08-26 17:54 ` viro 2004-08-26 18:16 ` Jamie Lokier 2004-08-30 13:25 ` Alex Zarochentsev 2 siblings, 0 replies; 1392+ messages in thread From: viro @ 2004-08-26 17:54 UTC (permalink / raw) To: Linus Torvalds Cc: Jamie Lokier, Christoph Hellwig, Hans Reiser, Alex Zarochentsev, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 10:29:45AM -0700, Linus Torvalds wrote: > > /dev/hda/part1/ <- the filesystem inside partition 1 > > I don't think you can do that. The kernel has no idea how to mount the > filesystem. ... and what's more, fundamental thing for mount is fs type, what to do with device(s) (if any) and any other arguments is the filesystem's business. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:29 ` Linus Torvalds 2004-08-26 17:54 ` viro @ 2004-08-26 18:16 ` Jamie Lokier 2004-08-26 18:27 ` Linus Torvalds 2004-08-30 13:25 ` Alex Zarochentsev 2 siblings, 1 reply; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 18:16 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, Hans Reiser, Alex Zarochentsev, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Linus Torvalds wrote: > > -> the setuid program thus ends up opening a device or fifo, > > when it does pwd's path walk. Yes it could use the getcwd > > syscall, but some programs do their own path walk. > > .. but even if it did that, it should use O_DIRECTORY when it did so. If > it doesn't, it's broken. Didn't someone just say that O_DIRECTORY will succeed on a device, precisely because opendir() is supposed to succeed on the device? I'm not sure. Let me refresh my memory. Linus Torvalds wrote: > And it would be perfectly ok for O_DIRECTORY to open such a file, as long > as it opens the directory branch, not the special device. Oh, I see. You're right. ".." sees the path of directory branches. > > It also fits the container idea very well: > > > > /dev/hda/part1/ <- the filesystem inside partition 1 > > I don't think you can do that. The kernel has no idea how to mount the > filesystem. It is not the kernel which decides. The filesystem containing /dev/hda/part1 opens "the directory branch". If the filesystem is suitably configured, it's exactly the same as opening a .zip or ELF file: it asks userspace for help. Userspace helps by either mounting it, or creating a container virtual directory -- in exactly the same way it helps with .zip and ELF files by creating a container virtual directory for those. It would mount it if the kernel supports the fs, and create a virtual directory if the kernel doesn't support the fs and userspace does. This is exactly the same as when you cd into a filesystem image: the helper mounts (this time with a lookback file mount) if the kernel supports the image fs, otherwise the helper may be able to decode it in userspace. By the way, these mechanisms could allow some of the old filesystems that nobody uses any more to be removed from the kernel altogether and kept in an archive of userspace-only supported filesystems. What do you think of that idea? > If it's already mounted somewhere else, that's a different issue. > Although it might be mounted in several places (as a bind mount) with > different writability, I guess, so even then it might be "interesting". The obvious implementation has the userspace helper just mounting it, end of story. If the mount command fails, it fails. Much like autofs. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:16 ` Jamie Lokier @ 2004-08-26 18:27 ` Linus Torvalds 2004-08-26 19:40 ` Jamie Lokier 0 siblings, 1 reply; 1392+ messages in thread From: Linus Torvalds @ 2004-08-26 18:27 UTC (permalink / raw) To: Jamie Lokier Cc: Christoph Hellwig, Hans Reiser, Alex Zarochentsev, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, 26 Aug 2004, Jamie Lokier wrote: > > > .. but even if it did that, it should use O_DIRECTORY when it did so. If > > it doesn't, it's broken. > > Didn't someone just say that O_DIRECTORY will succeed on a device, > precisely because opendir() is supposed to succeed on the device? It will succeed on the _name_ of the device, but it won't open the device node. It would open the _directory_ node associated with that name. Think of it this way: a pathname always points to one "container". That container is either a directory or a single node - and O_DIRECTORY ends up being the thing that chooses between them. > > I don't think you can do that. The kernel has no idea how to mount the > > filesystem. > > It is not the kernel which decides. The filesystem containing > /dev/hda/part1 opens "the directory branch". But that filesystem cannot know what the _other_ filesystem configurations are. And that's what you'd have to have to mount. > The obvious implementation has the userspace helper just mounting it, > end of story. If the mount command fails, it fails. Much like autofs. Yes, that would work, but it's of questionable use. If you want autofs, then just _use_ autofs. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:27 ` Linus Torvalds @ 2004-08-26 19:40 ` Jamie Lokier 0 siblings, 0 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 19:40 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, Hans Reiser, Alex Zarochentsev, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List Linus Torvalds wrote: > > It is not the kernel which decides. The filesystem containing > > /dev/hda/part1 opens "the directory branch". > > But that filesystem cannot know what the _other_ filesystem configurations > are. And that's what you'd have to have to mount. At which point, userspace comes in. Who knows what userspace chooses. > > The obvious implementation has the userspace helper just mounting it, > > end of story. If the mount command fails, it fails. Much like autofs. > > Yes, that would work, but it's of questionable use. If you want autofs, > then just _use_ autofs. I don't mean to imply it's useful behaviour: only that it seems to happen by default if you don't do anything special to prevent it, if you see what I mean. It does seem useful to have _files_ mounted automagically (using loopback devices): for example, cd into a .iso file and it would be nice to see the contents. At this point, deciding on the preferred -EBUSY or not behaviour becomes relevant, as does deciding whether the .iso in this example is viewed using the kernel's isofs code, or using a userspace helper in the same way as we'd use one to browse a .zip file. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:29 ` Linus Torvalds 2004-08-26 17:54 ` viro 2004-08-26 18:16 ` Jamie Lokier @ 2004-08-30 13:25 ` Alex Zarochentsev 2 siblings, 0 replies; 1392+ messages in thread From: Alex Zarochentsev @ 2004-08-30 13:25 UTC (permalink / raw) To: Linus Torvalds Cc: Jamie Lokier, Christoph Hellwig, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, ReiserFS List On Thu, Aug 26, 2004 at 10:29:45AM -0700, Linus Torvalds wrote: > > > On Thu, 26 Aug 2004, Jamie Lokier wrote: > > > > run_setuid_program > > > > -> calls pwd > > pwd opens("."), (".."), ("../..") etc. > > Ehh.. Not only does pwd not do that.. > > (hint: there's a getcwd() system call) > > > -> the setuid program thus ends up opening a device or fifo, > > when it does pwd's path walk. Yes it could use the getcwd > > syscall, but some programs do their own path walk. > > .. but even if it did that, it should use O_DIRECTORY when it did so. If > it doesn't, it's broken. > > So no, it would _not_ open the device or fifo when it did so. > > The fact is, anything that expects to open a directory should already be > opening it with O_DIRECTORY. reiser4 files-as-dirs do not depend on that. A file behaves as directory if one calls i_op->lookup() or f_op->readdir(). I am not sure that O_DIRECTORY should be a switch. > That said, ".." and "." are special already inside the kernel, and it > migth be worth making them automatically imply O_DIRECTORY, since nothing > else makes sense anyway. That would fix the case where somebody uses ".." > _without_ using O_DIRECTORY. > > > It also fits the container idea very well: > > > > /dev/hda/part1/ <- the filesystem inside partition 1 > > I don't think you can do that. The kernel has no idea how to mount the > filesystem. > > If it's already mounted somewhere else, that's a different issue. > Although it might be mounted in several places (as a bind mount) with > different writability, I guess, so even then it might be "interesting". > > Linus -- Alex. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:35 ` Linus Torvalds 2004-08-26 17:20 ` Jamie Lokier @ 2004-08-27 9:18 ` Giuliano Pochini 2004-08-30 14:10 ` Alex Zarochentsev 1 sibling, 1 reply; 1392+ messages in thread From: Giuliano Pochini @ 2004-08-27 9:18 UTC (permalink / raw) To: Linus Torvalds Cc: ReiserFS List, linux-kernel, linux-fsdevel, Hans Reiser, Jamie Lokier, Christoph Hellwig On 26-Aug-2004 Linus Torvalds wrote: > I advocated (long ago) something like this for /dev handling, just because > I think it would make sense to have > > /dev/hda <- special file > /dev/hda/part1 <- partition 1 (aka /dev/hda1) This breaks the r4 semantics if I understood it correctly. Because /partN are not simply associated to the file, they are part of the file. ie. when I modify /dev/hda I also change /dev/hda/partN and vice-versa. I don't see any pratical problem, though. > Still, I really do like the idea of merging the notion of file and > directory into one notion of "container". I absolutely _detest_ files with > internal structure that tools have to know about (ie I hate seeing all > those embedded formats that I can't use "grep" on - MIME being one case). > I'd much rather see a "group of files" and a "file with a grouping of > information". You're actually looking for a database with a legacy fs-like interface :) -- Giuliano. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 9:18 ` Giuliano Pochini @ 2004-08-30 14:10 ` Alex Zarochentsev 0 siblings, 0 replies; 1392+ messages in thread From: Alex Zarochentsev @ 2004-08-30 14:10 UTC (permalink / raw) To: Giuliano Pochini Cc: Linus Torvalds, ReiserFS List, linux-kernel, linux-fsdevel, Hans Reiser, Jamie Lokier, Christoph Hellwig On Fri, Aug 27, 2004 at 11:18:55AM +0200, Giuliano Pochini wrote: > > On 26-Aug-2004 Linus Torvalds wrote: > > > I advocated (long ago) something like this for /dev handling, just because > > I think it would make sense to have > > > > /dev/hda <- special file > > /dev/hda/part1 <- partition 1 (aka /dev/hda1) > > This breaks the r4 semantics if I understood it correctly. Reiser4 currently links /metas/ directory under each reiser4 fs object. Reiser4 "silently" allows regular file to be a directory for that. That is only an example how an fs object may link its internal hierarchical structure to the fs tree. /dev/hda/-as-a-dir is another example, but, I am afraid, it is not a reiser4 business to look into the devices. Devfs or its successor may do it. > Because > /partN are not simply associated to the file, they are part of the > file. ie. when I modify /dev/hda I also change /dev/hda/partN and > vice-versa. I don't see any pratical problem, though. > > > > Still, I really do like the idea of merging the notion of file and > > directory into one notion of "container". I absolutely _detest_ files with > > internal structure that tools have to know about (ie I hate seeing all > > those embedded formats that I can't use "grep" on - MIME being one case). > > I'd much rather see a "group of files" and a "file with a grouping of > > information". > > You're actually looking for a database with a legacy fs-like interface :) > > > -- > Giuliano. -- Alex. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:13 ` Christoph Hellwig 2004-08-26 16:35 ` Linus Torvalds @ 2004-08-29 18:43 ` Alex Zarochentsev 1 sibling, 0 replies; 1392+ messages in thread From: Alex Zarochentsev @ 2004-08-29 18:43 UTC (permalink / raw) To: Christoph Hellwig, Jamie Lokier, Hans Reiser, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Thu, Aug 26, 2004 at 06:13:03PM +0200, Christoph Hellwig wrote: > On Thu, Aug 26, 2004 at 05:06:38PM +0100, Jamie Lokier wrote: > > Christoph Hellwig wrote: > > > > There's bound to be some security issue, but I'm not sure what you're > > > > getting at with /tmp. What sort of sort of security problem arises > > > > with a world-writeable directory such as /tmp, that cannot arise with > > > > the standard fs semantics? > > > > > > Actually you are right on that issue because it would open the > > > device/fifo as directory and not device/fifo (in fact I'd had to look at > > > the code again to see whether they actually do this only for files or > > > also for special files) > > > > Are you saying that with reiser4, you can open a device or fifo with > > O_DIRECTORY? > > That's what I thought, but as far as I can follow the code this is not > actually true. All reiser4 inodes have i_ops->lookup != NULL, so open(..., O_DIRECTORY) would succeed on them (thanks Nikita for reminding me that). It may be better to pass the control of that to ->i_op->permission() through explicit MAY_LOOKUP flag. It can be possible to eliminate the race by some more complex logic (strict ownership check, for example) in ->permission(). -- Alex. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 19:53 ` Hans Reiser 2004-08-25 20:06 ` Matthew Wilcox 2004-08-25 20:08 ` Christoph Hellwig @ 2004-08-25 20:20 ` Jeremy Allison 2004-08-26 8:42 ` Hans Reiser 2004-08-25 20:22 ` Chris Mason ` (2 subsequent siblings) 5 siblings, 1 reply; 1392+ messages in thread From: Jeremy Allison @ 2004-08-25 20:20 UTC (permalink / raw) To: Hans Reiser Cc: Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Wed, Aug 25, 2004 at 12:53:28PM -0700, Hans Reiser wrote: > > You ignored everything I said during the discussion of xattrs about how > there is no need to have attributes when you can just have files and > directories, and that xattrs reflected a complete ignorance of name > space design principles. When I said we should just add some nice > optional features to files and directories so that they can do > everything that attributes can do if they are used that way, you just > didn't get it. You instead went for the quick ugly hack called xattrs. > You then got that ugly hack done first, because quick hacks are, well, > quick. I then went about doing it the right way for Reiser4, and got > DARPA to fund doing it. I was never silent about it. I don't want to comment on any of the technical issues about VFS etc. as I would be completely out of my depth, however I do want to say 2 things. Firstly, this is a feature that Samba users have been needing for many years to maintain compatibility with NTFS and Windows clients. Microsoft no longer sell any servers or clients without support for multiple data streams per file, and their latest XP SP2 code *does* use this feature. Whatever the kernel issues I'm really glad that Hans and Namesys have created something we can use to match this functionality - soon we will need it in order to be able to exist in a Microsoft client-dominated world. My second point is the following. Hans - did you *really* have to reinvent the wheel w.r.t userspace API calls ? Did you look at this work (done in 2001 for Solaris) ? http://bama.ua.edu/cgi-bin/man-cgi?fsattr+5 http://bama.ua.edu/cgi-bin/man-cgi?attropen+3C http://bama.ua.edu/cgi-bin/man-cgi?openat+2 I'm complaining here as someone who will have to write portable code to try and work on all these "files with streams" systems. Jeremy. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:20 ` Jeremy Allison @ 2004-08-26 8:42 ` Hans Reiser 2004-08-26 13:27 ` Rik van Riel 0 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 8:42 UTC (permalink / raw) To: Jeremy Allison Cc: Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Jeremy Allison wrote: >On Wed, Aug 25, 2004 at 12:53:28PM -0700, Hans Reiser wrote: > > >>You ignored everything I said during the discussion of xattrs about how >>there is no need to have attributes when you can just have files and >>directories, and that xattrs reflected a complete ignorance of name >>space design principles. When I said we should just add some nice >>optional features to files and directories so that they can do >>everything that attributes can do if they are used that way, you just >>didn't get it. You instead went for the quick ugly hack called xattrs. >>You then got that ugly hack done first, because quick hacks are, well, >>quick. I then went about doing it the right way for Reiser4, and got >>DARPA to fund doing it. I was never silent about it. >> >> > >I don't want to comment on any of the technical issues about VFS etc. as >I would be completely out of my depth, however I do want to say 2 things. Firstly, >this is a feature that Samba users have been needing for many years to maintain >compatibility with NTFS and Windows clients. Microsoft no longer sell any servers >or clients without support for multiple data streams per file, and their latest >XP SP2 code *does* use this feature. Whatever the kernel issues I'm really glad >that Hans and Namesys have created something we can use to match this >functionality - soon we will need it in order to be able to exist in >a Microsoft client-dominated world. > > I agree that your work is important without agreeing that MS client domination will last.;-) It is indeed my desire to give you every single feature you need to emulate MS streams within files, but doing it using directories that are files. I would like to support you in emulating windows faster than windows. >My second point is the following. Hans - did you *really* have to reinvent >the wheel w.r.t userspace API calls ? Did you look at this work (done in 2001 >for Solaris) ? > > I interviewed for the file system architect job at Sun in, I think, 1999, and they offered me the job conditional on my giving up on my Linux work. (After much trying and failing to convince them that it would be okay for me to work on Linux also, I declined the job, much to my fiscal loss and work satisfaction.) They do not do a pure job of implementing attributes in the file namespace though. There are far more distinctions between files and attributes than are necessary that are described in these man pages below, and those distinctions cause a loss of closure. I can say more on that if asked. >http://bama.ua.edu/cgi-bin/man-cgi?fsattr+5 >http://bama.ua.edu/cgi-bin/man-cgi?attropen+3C >http://bama.ua.edu/cgi-bin/man-cgi?openat+2 > >I'm complaining here as someone who will have to write portable code >to try and work on all these "files with streams" systems. > >Jeremy. > > > > ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:42 ` Hans Reiser @ 2004-08-26 13:27 ` Rik van Riel 2004-08-26 13:56 ` Markus Törnqvist 2004-08-26 23:53 ` Hans Reiser 0 siblings, 2 replies; 1392+ messages in thread From: Rik van Riel @ 2004-08-26 13:27 UTC (permalink / raw) To: Hans Reiser Cc: Jeremy Allison, Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Thu, 26 Aug 2004, Hans Reiser wrote: > I agree that your work is important without agreeing that MS client > domination will last.;-) It is indeed my desire to give you every > single feature you need to emulate MS streams within files, but doing it > using directories that are files. I would like to support you in > emulating windows faster than windows. 1) how do you back up and restore files with streams inside ? 2) how do standard unix utilities handle them ? -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 13:27 ` Rik van Riel @ 2004-08-26 13:56 ` Markus Törnqvist 2004-08-26 19:58 ` Paul Jackson 2004-08-26 23:53 ` Hans Reiser 1 sibling, 1 reply; 1392+ messages in thread From: Markus Törnqvist @ 2004-08-26 13:56 UTC (permalink / raw) To: Rik van Riel Cc: Hans Reiser, Jeremy Allison, Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Thu, Aug 26, 2004 at 09:27:52AM -0400, Rik van Riel wrote: > >1) how do you back up and restore files with streams inside ? With a program that supports files with streams inside. >2) how do standard unix utilities handle them ? Most likely they don't ;) That is, until they are fixed or replaced. I've heard of people who want xattrs to be backed up so they use star, not gnu tar, already. -- mjt ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 13:56 ` Markus Törnqvist @ 2004-08-26 19:58 ` Paul Jackson 0 siblings, 0 replies; 1392+ messages in thread From: Paul Jackson @ 2004-08-26 19:58 UTC (permalink / raw) To: Markus Törnqvist Cc: riel, reiser, jra, hch, akpm, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Markus wrote: > >2) how do standard unix utilities handle them ? > > Most likely they don't ;) That is, until they are fixed or replaced. Not good, in my view. -- I won't rest till it's the best ... Programmer, Linux Scalability Paul Jackson <pj@sgi.com> 1.650.933.1373 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 13:27 ` Rik van Riel 2004-08-26 13:56 ` Markus Törnqvist @ 2004-08-26 23:53 ` Hans Reiser 1 sibling, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 23:53 UTC (permalink / raw) To: Rik van Riel Cc: Jeremy Allison, Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Rik van Riel wrote: >On Thu, 26 Aug 2004, Hans Reiser wrote: > > > >>I agree that your work is important without agreeing that MS client >>domination will last.;-) It is indeed my desire to give you every >>single feature you need to emulate MS streams within files, but doing it >>using directories that are files. I would like to support you in >>emulating windows faster than windows. >> >> > >1) how do you back up and restore files with streams inside ? > >2) how do standard unix utilities handle them ? > > > To repeat, I think it would be nice to implement a filename/pseudos/backup method for all the plugins. Guys, we just have the beginnings in place. One plugin method at a time it will all fall into place. What we have now is useful now. The more methods come into existence, the more compelling it becomes --- standard network economic theory applies here. Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 19:53 ` Hans Reiser ` (2 preceding siblings ...) 2004-08-25 20:20 ` Jeremy Allison @ 2004-08-25 20:22 ` Chris Mason 2004-08-26 8:42 ` Hans Reiser 2004-08-25 20:23 ` Chris Friesen 2004-08-25 22:28 ` Andrew Morton 5 siblings, 1 reply; 1392+ messages in thread From: Chris Mason @ 2004-08-25 20:22 UTC (permalink / raw) To: Hans Reiser Cc: Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Wed, 2004-08-25 at 15:53, Hans Reiser wrote: > I had not intended to respond to this because I have nothing positive to > say, but Andrew said I needed to respond and suggested I should copy > Linus. Sigh. > > Dear Christoph, > > Let me see if I can summarize what you and your contingent are saying, > and if I misconstrue anything, let me know.;-) > Just for fun why don't we look at the way things are today: 1) reiser4 has semantics that do belong at the VFS level. They weren't implemented at the VFS level for a variety of reasons, none of which really matter right now. 2) new kernel patches that fragment the application developers between apis are a bad thing. There does need to be one interface here, and it is in Hans' best interest to unify his work by working with people to introduce new kernel wide apis. This starts with exactly what Christoph described in writing a short summary of how you want things to work today. Since we can't resist, we'll also go ahead and rehash all the old flame wars over this, but try to include some new ideas about where you want to see the reiser4 interfaces in 6 months as well. -chris ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 20:22 ` Chris Mason @ 2004-08-26 8:42 ` Hans Reiser 2004-08-26 9:36 ` Christoph Hellwig 0 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 8:42 UTC (permalink / raw) To: Chris Mason Cc: Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Chris Mason wrote: > > >This starts with exactly what Christoph described in writing a short >summary of how you want things to work today. Since we can't resist, >we'll also go ahead and rehash all the old flame wars over this, but try >to include some new ideas about where you want to see the reiser4 >interfaces in 6 months as well. > >-chris > > > > > > > Did Christoph read the www.namesys.com URL that has for several years been hiding from him? ;-) Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:42 ` Hans Reiser @ 2004-08-26 9:36 ` Christoph Hellwig 0 siblings, 0 replies; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-26 9:36 UTC (permalink / raw) To: Hans Reiser Cc: Chris Mason, Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List On Thu, Aug 26, 2004 at 01:42:50AM -0700, Hans Reiser wrote: > Did Christoph read the www.namesys.com URL that has for several years > been hiding from him? ;-) Yes, and I'm sick of your marketing bullshit Hans. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 19:53 ` Hans Reiser ` (3 preceding siblings ...) 2004-08-25 20:22 ` Chris Mason @ 2004-08-25 20:23 ` Chris Friesen 2004-08-25 22:28 ` Andrew Morton 5 siblings, 0 replies; 1392+ messages in thread From: Chris Friesen @ 2004-08-25 20:23 UTC (permalink / raw) To: Hans Reiser Cc: Christoph Hellwig, akpm, linux-fsdevel, linux-kernel, Alexander Lyamin aka FLX, Linus Torvalds, ReiserFS List Hans Reiser wrote: > Making files into directories caused only two applications out of the > entire OS to notice the change, and that was because of a bug in what > error code we returned that we are going to fix. You think that was a > disaster; I think it was a triumph. > Since we have such a performance lead, Namesys is about to change its > focus from the storage layer to semantics, look at > www.namesys.com/whitepaper.html for details. Semantic enhancements are > the important stuff, and finally Namesys is where we have all the > storage layer prerequisites done right, and the real work can begin. Just curious about your comments on Jamie Lokier's suggestions for enabling files-as-directories semantics without breaking existing apps. Chris ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 19:53 ` Hans Reiser ` (4 preceding siblings ...) 2004-08-25 20:23 ` Chris Friesen @ 2004-08-25 22:28 ` Andrew Morton 2004-08-25 22:51 ` Spam 2004-08-26 8:31 ` Hans Reiser 5 siblings, 2 replies; 1392+ messages in thread From: Andrew Morton @ 2004-08-25 22:28 UTC (permalink / raw) To: Hans Reiser Cc: hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Hans Reiser <reiser@namesys.com> wrote: > > I had not intended to respond to this because I have nothing positive to > say, but Andrew said I needed to respond and suggested I should copy > Linus. Yes, but I didn't say "flame Christoph and ignore the issues" ;) There are lots of little things to do with implementation, coding style, module exports, deadlocks, what code goes where, etc. These are all normal daily kernel business and we should set them aside for now and concentrate on the bigger issues. And as I see it, there are two big issues: a) reiser4 extends the Linux API in ways which POSIX/Unix/etc do not anticipate and b) it does this within the context of just a single filesystem. I see three possible responses: a) accept the reiser4-only extensions as-is (possibly with post-review modifications, of course) or b) accept the reiser4-only extensions with a view to turning them into kernel-wide extensions at some time in the future, so all filesystems will offer the extensions (as much as poss) or c) reject the extensions. My own order of preference is b) c) a). The fact that one filesystem will offer features which other filesystems do not and cannot offer makes me queasy for some reason. b) means that at some time in the future we need to hoist the reiser4 extensions (at a conceptual level) (and probably with restrictions) up into the VFS. This will involve much thought, argument and work. To get us started on this route it would really help me (and, probably, others) if you could describe what these API extensions are in a very simple way, without referring to incomprehehsible web pages, and without using terms which non-reiser4 people don't understand. It would be best if each extension was addressed in a separate email thread. We also need to discuss what a reiser4 "module" is, what its capabilities are, and what licensing implications they have. Then, we can look at each one and say "yup, that makes sense - we want Linux to do that" and we can also think about how we would implement it at the VFS level. If we follow the above route I believe we can make progress in a technical direction and not get deadlocked on personal/political stuff. Now, an alternative to all the above is to just merge reiser4 as-is, after addressing all the lower-level coding issues. And see what happens. That may be a thing which Linus wishes to do. I'm easy. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 22:28 ` Andrew Morton @ 2004-08-25 22:51 ` Spam 2004-08-25 22:51 ` Christoph Hellwig ` (2 more replies) 2004-08-26 8:31 ` Hans Reiser 1 sibling, 3 replies; 1392+ messages in thread From: Spam @ 2004-08-25 22:51 UTC (permalink / raw) To: Andrew Morton Cc: Hans Reiser, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list > And as I see it, there are two big issues: > a) reiser4 extends the Linux API in ways which POSIX/Unix/etc do not > anticipate and > b) it does this within the context of just a single filesystem. > I see three possible responses: > a) accept the reiser4-only extensions as-is (possibly with post-review > modifications, of course) or > b) accept the reiser4-only extensions with a view to turning them into > kernel-wide extensions at some time in the future, so all filesystems > will offer the extensions (as much as poss) or > c) reject the extensions. > My own order of preference is b) c) a). The fact that one filesystem will > offer features which other filesystems do not and cannot offer makes me > queasy for some reason. This last sentence makes me wonder. Where is Linux heading? The idea that a FS cannot contain features that no other FS has is very scary. I am all for uniformity, but not at the expense of shutting down advanced progress that Linux is so badly needing. This talk about old UNIX seems like people want to still live in the 70'ies and not look forward. Please wake up! ~S ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 22:51 ` Spam @ 2004-08-25 22:51 ` Christoph Hellwig 2004-08-25 22:59 ` Linus Torvalds 2004-08-26 8:44 ` Hans Reiser 2 siblings, 0 replies; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-25 22:51 UTC (permalink / raw) To: Spam Cc: Andrew Morton, Hans Reiser, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list > This last sentence makes me wonder. Where is Linux heading? The idea > that a FS cannot contain features that no other FS has is very > scary. > > I am all for uniformity, but not at the expense of shutting down > advanced progress that Linux is so badly needing. > > This talk about old UNIX seems like people want to still live in the > 70'ies and not look forward. Please wake up! Just because semantics are at the VFS layer doesn't mean every filesystem has to implement them. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 22:51 ` Spam 2004-08-25 22:51 ` Christoph Hellwig @ 2004-08-25 22:59 ` Linus Torvalds 2004-08-25 23:19 ` Spam 2004-08-26 8:44 ` Hans Reiser 2 siblings, 1 reply; 1392+ messages in thread From: Linus Torvalds @ 2004-08-25 22:59 UTC (permalink / raw) To: Spam Cc: Andrew Morton, Hans Reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Spam wrote: > > > My own order of preference is b) c) a). The fact that one filesystem will > > offer features which other filesystems do not and cannot offer makes me > > queasy for some reason. > > This last sentence makes me wonder. Where is Linux heading? The idea > that a FS cannot contain features that no other FS has is very > scary. That's not what Andrew said or meant. Note the "cannot offer". As in "there is no way to offer them even if the filesystem could support it otherwise". We have tons of filesystems that do things other filesystems cannot do. Most filesystems support writing to a file - despite the fact that some filesystems (iso9600 being an obvious one) cannot. The infrastructure is there in the VFS layer, and it becomes a _choice_ for the filesystem whether it offers certain capabilities. So look at what Andrew said, again: his top choice would be (b). Let's see what that was again, shall we? > b) accept the reiser4-only extensions with a view to turning them into > kernel-wide extensions at some time in the future, so all filesystems > will offer the extensions (as much as poss) or In other words, if reiserfs does something special, we should make standard interfaces for doing that special thing, so that everybody can do it without stepping on other peoples toes. That doesn't mean that we'd _force_ everybody to do it. The same way we don't force iso9660 to write to a CD-ROM. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 22:59 ` Linus Torvalds @ 2004-08-25 23:19 ` Spam 2004-08-25 23:32 ` Andrew Morton 0 siblings, 1 reply; 1392+ messages in thread From: Spam @ 2004-08-25 23:19 UTC (permalink / raw) To: Linus Torvalds Cc: Andrew Morton, Hans Reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > On Thu, 26 Aug 2004, Spam wrote: >> >> > My own order of preference is b) c) a). The fact that one filesystem will >> > offer features which other filesystems do not and cannot offer makes me >> > queasy for some reason. >> >> This last sentence makes me wonder. Where is Linux heading? The idea >> that a FS cannot contain features that no other FS has is very >> scary. > That's not what Andrew said or meant. > Note the "cannot offer". As in "there is no way to offer them even if the > filesystem could support it otherwise". > We have tons of filesystems that do things other filesystems cannot do. > Most filesystems support writing to a file - despite the fact that some > filesystems (iso9600 being an obvious one) cannot. The infrastructure is > there in the VFS layer, and it becomes a _choice_ for the filesystem > whether it offers certain capabilities. > So look at what Andrew said, again: his top choice would be (b). Let's see > what that was again, shall we? >> b) accept the reiser4-only extensions with a view to turning them into >> kernel-wide extensions at some time in the future, so all filesystems >> will offer the extensions (as much as poss) or > In other words, if reiserfs does something special, we should make > standard interfaces for doing that special thing, so that everybody can > do it without stepping on other peoples toes. Agreed that would be the best. But how much time and effort will it be, and how much of the original ideas would be lost on the way to implement them in the VFS? Especially with new and very advanced FS's like Reiser4. Isn't the line between the actual file system and the virtual one very hair thin? Where would the separation lay in Reiser4? > That doesn't mean that we'd _force_ everybody to do it. The same way we > don't force iso9660 to write to a CD-ROM. > Linus I got caught in the moment of flame war. My appologies. ~S ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 23:19 ` Spam @ 2004-08-25 23:32 ` Andrew Morton 2004-08-25 23:37 ` Jeremy Allison 2004-08-26 9:03 ` Anton Altaparmakov 0 siblings, 2 replies; 1392+ messages in thread From: Andrew Morton @ 2004-08-25 23:32 UTC (permalink / raw) To: Spam Cc: torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Spam <spam@tnonline.net> wrote: > > > In other words, if reiserfs does something special, we should make > > standard interfaces for doing that special thing, so that everybody can > > do it without stepping on other peoples toes. > > Agreed that would be the best. But how much time and effort will it > be Zero. We can add these new features tomorrow, as reiser4-only features, with a plan in hand to generalise them later. -->>__if__<<-- we think these are features which Linux should offer. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 23:32 ` Andrew Morton @ 2004-08-25 23:37 ` Jeremy Allison 2004-08-25 23:46 ` Wichert Akkerman 2004-08-26 9:03 ` Anton Altaparmakov 1 sibling, 1 reply; 1392+ messages in thread From: Jeremy Allison @ 2004-08-25 23:37 UTC (permalink / raw) To: Andrew Morton Cc: Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Wed, Aug 25, 2004 at 04:32:25PM -0700, Andrew Morton wrote: > > We can add these new features tomorrow, as reiser4-only features, with a > plan in hand to generalise them later. > > -->>__if__<<-- we think these are features which Linux should offer. Multiple-data-stream files are something we should offer, definately (IMHO). I don't care how we do it, but I know it's something we need as application developers. Jeremy. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 23:37 ` Jeremy Allison @ 2004-08-25 23:46 ` Wichert Akkerman 2004-08-26 0:42 ` Nicholas Miell 2004-08-26 9:40 ` Spam 0 siblings, 2 replies; 1392+ messages in thread From: Wichert Akkerman @ 2004-08-25 23:46 UTC (permalink / raw) To: Jeremy Allison Cc: Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Previously Jeremy Allison wrote: > Multiple-data-stream files are something we should offer, definately (IMHO). > I don't care how we do it, but I know it's something we need as application > developers. Aside from samba, is there any other application that has a use for them? Wichert. -- Wichert Akkerman <wichert@wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 23:46 ` Wichert Akkerman @ 2004-08-26 0:42 ` Nicholas Miell 2004-08-26 1:03 ` Jamie Lokier ` (2 more replies) 2004-08-26 9:40 ` Spam 1 sibling, 3 replies; 1392+ messages in thread From: Nicholas Miell @ 2004-08-26 0:42 UTC (permalink / raw) To: Wichert Akkerman Cc: Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Wed, 2004-08-25 at 16:46, Wichert Akkerman wrote: > Previously Jeremy Allison wrote: > > Multiple-data-stream files are something we should offer, definately (IMHO). > > I don't care how we do it, but I know it's something we need as application > > developers. > > Aside from samba, is there any other application that has a use for > them? > Anything that currently stores a file's metadata in another file really wants this right now. Things like image thumbnails, document summaries, digital signatures, etc. As to how to do it, I think the Solaris interface is reasonably decent. The overview is at http://docs.sun.com/db/doc/816-0220/6m6nkorp9?a=view (An important detail for those who want to access their multiple-data-streams from non-MDS aware apps is the runat shell command, which basically does a chdir into the specified file's attribute directory and then runs a command. i.e. 'runat ~/blah ls' will list the ~/blah's attributes.) The only real problem I have with their design is the calling them attributes and using "at" everywhere. "Attributes", because it will get confused with the current Linux xattr implementation (which is still useful for things that actually are file attributes, like security labels, ACLs, weird attributess that FAT/NTFS/whatever have, etc.). I don't like "at" because the API changes don't have anything to do with the actual attributes. It's a general set of changes to allow paths relative to a fd instead of the cwd, and doesn't really have anything specifically to do with attributes (with the exception of the O_XATTR flag). Replace "at" with "rel" and O_XATTR with O_FORK or O_MULTI or something, and it's all good. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 0:42 ` Nicholas Miell @ 2004-08-26 1:03 ` Jamie Lokier 2004-08-26 1:26 ` Nicholas Miell ` (2 more replies) 2004-08-26 1:22 ` Jeremy Allison 2004-08-26 4:44 ` Matt Mackall 2 siblings, 3 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 1:03 UTC (permalink / raw) To: Nicholas Miell Cc: Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Nicholas Miell wrote: > Anything that currently stores a file's metadata in another file really > wants this right now. Things like image thumbnails, document summaries, > digital signatures, etc. Additionally, all of those things you describe should be deleted if the file is modified -- to indicate that they're no longer valid and should be regenerated if needed. Whereas there are some other kinds of metadata which should not be deleted if the file is modified. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 1:03 ` Jamie Lokier @ 2004-08-26 1:26 ` Nicholas Miell 2004-08-26 1:53 ` Jamie Lokier 2004-08-26 8:41 ` Hans Reiser 2004-08-26 19:06 ` Giuseppe Bilotta 2 siblings, 1 reply; 1392+ messages in thread From: Nicholas Miell @ 2004-08-26 1:26 UTC (permalink / raw) To: Jamie Lokier Cc: Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Wed, 2004-08-25 at 18:03, Jamie Lokier wrote: > Nicholas Miell wrote: > > Anything that currently stores a file's metadata in another file really > > wants this right now. Things like image thumbnails, document summaries, > > digital signatures, etc. > > Additionally, all of those things you describe should be deleted if > the file is modified -- to indicate that they're no longer valid and > should be regenerated if needed. > > Whereas there are some other kinds of metadata which should not be > deleted if the file is modified. > > -- Jamie Presumably the app which uses the metadata will be smart enough to compare the st_mtime of the MDS/stream/attribute/whatever (can we choose a name for these things now?) to the st_mtime of the file and do the right thing. thumbnail - regenerate it summary - keep it, it's relatively independent of the file's exact contents signature - always verify it etc. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 1:26 ` Nicholas Miell @ 2004-08-26 1:53 ` Jamie Lokier 2004-08-26 2:02 ` Nicholas Miell 0 siblings, 1 reply; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 1:53 UTC (permalink / raw) To: Nicholas Miell Cc: Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Nicholas Miell wrote: > > Additionally, all of those things you describe should be deleted if > > the file is modified -- to indicate that they're no longer valid and > > should be regenerated if needed. > > > > Whereas there are some other kinds of metadata which should not be > > deleted if the file is modified. > > > > -- Jamie > > Presumably the app which uses the metadata will be smart enough to > compare the st_mtime of the MDS/stream/attribute/whatever (can we choose > a name for these things now?) to the st_mtime of the file and do the > right thing. [This is straying off the topic of files-as-directories]. st_mtime tests are weak. They break sometimes, and are not suitable for strong data models such as transparently caching generated data from a file's contents. They're especially breakable if you change a file and then read it within a second. Sometimes, more than a second. As has been raised before, nanosecond timestamps (a) don't solve this unless they're stored in the filesystem; (b) even then they will fail one day, on a sufficiently fast box; (c) don't work when there are changes to wall clock time. They're fine where you don't care if the generated data is wrong sometimes. If you want the _equivalent_ behaviour to opening and parsing the file, but fast, then they're no good. Modification serial numbers would work, though. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 1:53 ` Jamie Lokier @ 2004-08-26 2:02 ` Nicholas Miell 0 siblings, 0 replies; 1392+ messages in thread From: Nicholas Miell @ 2004-08-26 2:02 UTC (permalink / raw) To: Jamie Lokier Cc: Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Wed, 2004-08-25 at 18:53, Jamie Lokier wrote: > Nicholas Miell wrote: > > > Additionally, all of those things you describe should be deleted if > > > the file is modified -- to indicate that they're no longer valid and > > > should be regenerated if needed. > > > > > > Whereas there are some other kinds of metadata which should not be > > > deleted if the file is modified. > > > > > > -- Jamie > > > > Presumably the app which uses the metadata will be smart enough to > > compare the st_mtime of the MDS/stream/attribute/whatever (can we choose > > a name for these things now?) to the st_mtime of the file and do the > > right thing. > > [This is straying off the topic of files-as-directories]. > > st_mtime tests are weak. They break sometimes, and are not suitable > for strong data models such as transparently caching generated data > from a file's contents. > > They're especially breakable if you change a file and then read it > within a second. Sometimes, more than a second. > > As has been raised before, nanosecond timestamps (a) don't solve this > unless they're stored in the filesystem; (b) even then they will fail > one day, on a sufficiently fast box; (c) don't work when there are > changes to wall clock time. > > They're fine where you don't care if the generated data is wrong sometimes. > > If you want the _equivalent_ behaviour to opening and parsing the > file, but fast, then they're no good. > > Modification serial numbers would work, though. > > -- Jamie Remember that the Solaris attribute model is just another filesystem tree rooted in a hidden directory associated with a regular file. If you can come up with semantics that work in general for all files, that's fine. inodes that delete themselves when other inodes are changed creep me out. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 1:03 ` Jamie Lokier 2004-08-26 1:26 ` Nicholas Miell @ 2004-08-26 8:41 ` Hans Reiser 2004-08-26 19:06 ` Giuseppe Bilotta 2 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 8:41 UTC (permalink / raw) To: Jamie Lokier Cc: Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Jamie Lokier wrote: >Nicholas Miell wrote: > > >>Anything that currently stores a file's metadata in another file really >>wants this right now. Things like image thumbnails, document summaries, >>digital signatures, etc. >> >> > >Additionally, all of those things you describe should be deleted if >the file is modified -- to indicate that they're no longer valid and >should be regenerated if needed. > >Whereas there are some other kinds of metadata which should not be >deleted if the file is modified. > >-- Jamie > > > > Yes, I agree. Actually we plan to have a whole link taxonomy, and one expected feature is that some links don't count towards the refcount needed to keep an object in existence (For instance, the links between key words and text documents, you don't want to have to explicitly unlink every keyword in order to delete a document). ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 1:03 ` Jamie Lokier 2004-08-26 1:26 ` Nicholas Miell 2004-08-26 8:41 ` Hans Reiser @ 2004-08-26 19:06 ` Giuseppe Bilotta 2 siblings, 0 replies; 1392+ messages in thread From: Giuseppe Bilotta @ 2004-08-26 19:06 UTC (permalink / raw) To: linux-kernel; +Cc: reiserfs-list, linux-fsdevel, reiserfs-list, linux-kernel Jamie Lokier wrote: > Nicholas Miell wrote: > > Anything that currently stores a file's metadata in another file really > > wants this right now. Things like image thumbnails, document summaries, > > digital signatures, etc. > > Additionally, all of those things you describe should be deleted if > the file is modified -- to indicate that they're no longer valid and > should be regenerated if needed. In principle, not all of them. For example, a document summary for a text document or a long (textual) description of a video clip might remain the same when the user is only working on the finishing details. Maybe the metadata needs an attribute to determine how 'immutable' it should be wrt changes on the file? (Can you spell meta-meta-data <g>) -- Giuseppe "Oblomov" Bilotta Can't you see It all makes perfect sense Expressed in dollar and cents Pounds shillings and pence (Roger Waters) ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 0:42 ` Nicholas Miell 2004-08-26 1:03 ` Jamie Lokier @ 2004-08-26 1:22 ` Jeremy Allison 2004-08-26 11:01 ` Wichert Akkerman 2004-08-26 4:44 ` Matt Mackall 2 siblings, 1 reply; 1392+ messages in thread From: Jeremy Allison @ 2004-08-26 1:22 UTC (permalink / raw) To: Nicholas Miell Cc: Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Wed, Aug 25, 2004 at 05:42:21PM -0700, Nicholas Miell wrote: > > Anything that currently stores a file's metadata in another file really > wants this right now. Things like image thumbnails, document summaries, > digital signatures, etc. Which is exactly how Windows apps have started to use streams. > As to how to do it, I think the Solaris interface is reasonably decent. > The overview is at http://docs.sun.com/db/doc/816-0220/6m6nkorp9?a=view I agree. This is an interface Samba can live with I think. I was thinking of implementing it anyway, just so I could piss off the Linux kernel developers by saying "oh if you need proper Windows semantics on Samba you have to use an advanced OS like Solaris, Linux doesn't cut it" :-) :-). > The only real problem I have with their design is the calling them > attributes and using "at" everywhere. Yep - they're different from xattrs. The easiest way to remember this is that file streams are seekable and get a fd, xattrs aren't and don't :-). Jeremy. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 1:22 ` Jeremy Allison @ 2004-08-26 11:01 ` Wichert Akkerman [not found] ` <c7ef32a70408260942c8efa54@mail.gmail.com> 0 siblings, 1 reply; 1392+ messages in thread From: Wichert Akkerman @ 2004-08-26 11:01 UTC (permalink / raw) To: Jeremy Allison Cc: Nicholas Miell, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Previously Jeremy Allison wrote: > Which is exactly how Windows apps have started to use streams. Interesting I can not find a way using the .NET framework, which is supposed to be The Future, to use streams. Wichert. -- Wichert Akkerman <wichert@wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. ^ permalink raw reply [flat|nested] 1392+ messages in thread
[parent not found: <c7ef32a70408260942c8efa54@mail.gmail.com>]
* Re: silent semantic changes with reiser4 [not found] ` <c7ef32a70408260942c8efa54@mail.gmail.com> @ 2004-08-26 16:44 ` Tierra 0 siblings, 0 replies; 1392+ messages in thread From: Tierra @ 2004-08-26 16:44 UTC (permalink / raw) To: linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004 13:01:41 +0200, Wichert Akkerman <wichert@wiggy.net> wrote: > Previously Jeremy Allison wrote: > > Which is exactly how Windows apps have started to use streams. > > Interesting I can not find a way using the .NET framework, which is > supposed to be The Future, to use streams. That's because you don't even need .NET... http://support.microsoft.com/default.aspx?scid=http://support.microsoft.com:80/support/kb/articles/q105/7/63.asp&NoWebContent=1&NoWebContent=1 http://tinyurl.com/45mfa Bryan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 0:42 ` Nicholas Miell 2004-08-26 1:03 ` Jamie Lokier 2004-08-26 1:22 ` Jeremy Allison @ 2004-08-26 4:44 ` Matt Mackall 2004-08-26 5:09 ` Nicholas Miell ` (3 more replies) 2 siblings, 4 replies; 1392+ messages in thread From: Matt Mackall @ 2004-08-26 4:44 UTC (permalink / raw) To: Nicholas Miell Cc: Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Wed, Aug 25, 2004 at 05:42:21PM -0700, Nicholas Miell wrote: > On Wed, 2004-08-25 at 16:46, Wichert Akkerman wrote: > > Previously Jeremy Allison wrote: > > > Multiple-data-stream files are something we should offer, definately (IMHO). > > > I don't care how we do it, but I know it's something we need as application > > > developers. > > > > Aside from samba, is there any other application that has a use for > > them? > > > > Anything that currently stores a file's metadata in another file really > wants this right now. Things like image thumbnails, document summaries, > digital signatures, etc. That is _highly_ debatable. I would much rather have my cp and grep and cat and tar and such continue to work than have to rewrite every tool because we've thrown the file-is-a-stream-of-bytes concept out the window. Never mind that I've got thumbnails, document summaries, and digital signatures already. While the number of annoying properties of files with forks is practically endless, the biggest has got to be utter lack of portability. How do you stick the thing in an attachment or on an ftp site? Well you can't because it's NOT A FILE. A file is a stream of bytes. -- Mathematics is the supreme nostalgia of our time. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 4:44 ` Matt Mackall @ 2004-08-26 5:09 ` Nicholas Miell 2004-08-26 5:17 ` James Morris ` (3 more replies) 2004-08-26 5:23 ` Jamie Lokier ` (2 subsequent siblings) 3 siblings, 4 replies; 1392+ messages in thread From: Nicholas Miell @ 2004-08-26 5:09 UTC (permalink / raw) To: Matt Mackall Cc: Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Wed, 2004-08-25 at 21:44, Matt Mackall wrote: > On Wed, Aug 25, 2004 at 05:42:21PM -0700, Nicholas Miell wrote: > > On Wed, 2004-08-25 at 16:46, Wichert Akkerman wrote: > > > Previously Jeremy Allison wrote: > > > > Multiple-data-stream files are something we should offer, definately (IMHO). > > > > I don't care how we do it, but I know it's something we need as application > > > > developers. > > > > > > Aside from samba, is there any other application that has a use for > > > them? > > > > > > > Anything that currently stores a file's metadata in another file really > > wants this right now. Things like image thumbnails, document summaries, > > digital signatures, etc. > > That is _highly_ debatable. I would much rather have my cp and grep > and cat and tar and such continue to work than have to rewrite every > tool because we've thrown the file-is-a-stream-of-bytes concept out > the window. Never mind that I've got thumbnails, document summaries, > and digital signatures already. > > While the number of annoying properties of files with forks is > practically endless, the biggest has got to be utter lack of > portability. How do you stick the thing in an attachment or on an ftp > site? Well you can't because it's NOT A FILE. > > A file is a stream of bytes. "OMG! It breaks tar and email!!!" argument doesn't fly. Things break all the time and are fixed. It's called progress. cp, grep, cat, and tar will continue to work just fine on files with multiple streams. tar and cp will lose the extra streams until somebody fixes them, but they lose ACLs and xattrs right now, and I don't hear anybody suggesting that ACLs and xattrs be removed from the kernel because of this. Fixing programs that do recursive filesystem traversals is a matter of a glibc patch to nftw(3) and a modification to their option processing. Holding back a useful feature because you don't want to upgrade coreutils is just plain dumb. (BTW, for email, multipart/parallel is a start, but a specific multipart content type for multi-stream file attachments would probably be more appropriate.) ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 5:09 ` Nicholas Miell @ 2004-08-26 5:17 ` James Morris 2004-08-26 5:32 ` Matt Mackall ` (2 subsequent siblings) 3 siblings, 0 replies; 1392+ messages in thread From: James Morris @ 2004-08-26 5:17 UTC (permalink / raw) To: Nicholas Miell Cc: Matt Mackall, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Wed, 25 Aug 2004, Nicholas Miell wrote: > (BTW, for email, multipart/parallel is a start, but a specific multipart > content type for multi-stream file attachments would probably be more > appropriate.) I don't understand why something like this is so important that it needs underlying VFS support. Multipart messages seem to have survived fine up until now. Where is the justification? - James -- James Morris <jmorris@redhat.com> ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 5:09 ` Nicholas Miell 2004-08-26 5:17 ` James Morris @ 2004-08-26 5:32 ` Matt Mackall 2004-08-26 7:34 ` Denis Vlasenko ` (4 more replies) 2004-08-26 6:53 ` Paul Jackson 2004-08-26 9:54 ` Spam 3 siblings, 5 replies; 1392+ messages in thread From: Matt Mackall @ 2004-08-26 5:32 UTC (permalink / raw) To: Nicholas Miell Cc: Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Wed, Aug 25, 2004 at 10:09:08PM -0700, Nicholas Miell wrote: > On Wed, 2004-08-25 at 21:44, Matt Mackall wrote: > > On Wed, Aug 25, 2004 at 05:42:21PM -0700, Nicholas Miell wrote: > > > On Wed, 2004-08-25 at 16:46, Wichert Akkerman wrote: > > > > Previously Jeremy Allison wrote: > > > > > Multiple-data-stream files are something we should offer, definately (IMHO). > > > > > I don't care how we do it, but I know it's something we need as application > > > > > developers. > > > > > > > > Aside from samba, is there any other application that has a use for > > > > them? > > > > > > > > > > Anything that currently stores a file's metadata in another file really > > > wants this right now. Things like image thumbnails, document summaries, > > > digital signatures, etc. > > > > That is _highly_ debatable. I would much rather have my cp and grep > > and cat and tar and such continue to work than have to rewrite every > > tool because we've thrown the file-is-a-stream-of-bytes concept out > > the window. Never mind that I've got thumbnails, document summaries, > > and digital signatures already. > > > > While the number of annoying properties of files with forks is > > practically endless, the biggest has got to be utter lack of > > portability. How do you stick the thing in an attachment or on an ftp > > site? Well you can't because it's NOT A FILE. > > > > A file is a stream of bytes. > > "OMG! It breaks tar and email!!!" argument doesn't fly. Things break all > the time and are fixed. It's called progress. What it breaks is the concept of a file. In ways that are ill-defined, not portable, hard to work with, and needlessly complex. Along the way, it breaks every single application that ever thought it knew what a file was. Progress? No, this has been done before. Various dead operating systems have done it or similar and regretted it. Most recently MacOS, which jumped through major hurdles to begin purging themselves of resource forks when they went to OS X. They're still there, but heavily deprecated. > cp, grep, cat, and tar will continue to work just fine on files with > multiple streams. Find some silly person with an iBook and open a shell on OS X. Use cp to copy a file with a resource fork. Oh look, the Finder has no idea what the new file is, even though it looks exactly identical in the shell. Isn't that _wonderful_? Now try cat < a > b on a file with a fork. How is that ever going to work? I like cat < a > b. You can keep your progress. -- Mathematics is the supreme nostalgia of our time. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 5:32 ` Matt Mackall @ 2004-08-26 7:34 ` Denis Vlasenko 2004-08-26 11:42 ` Rik van Riel 2004-08-26 7:53 ` Markus Törnqvist ` (3 subsequent siblings) 4 siblings, 1 reply; 1392+ messages in thread From: Denis Vlasenko @ 2004-08-26 7:34 UTC (permalink / raw) To: Matt Mackall, Nicholas Miell Cc: Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > > > A file is a stream of bytes. > > > > "OMG! It breaks tar and email!!!" argument doesn't fly. Things break all > > the time and are fixed. It's called progress. > > What it breaks is the concept of a file. In ways that are ill-defined, > not portable, hard to work with, and needlessly complex. Along the > way, it breaks every single application that ever thought it knew what > a file was. > > Progress? No, this has been done before. Various dead operating > systems have done it or similar and regretted it. Most recently MacOS, > which jumped through major hurdles to begin purging themselves of > resource forks when they went to OS X. They're still there, but > heavily deprecated. > > > cp, grep, cat, and tar will continue to work just fine on files with > > multiple streams. > > Find some silly person with an iBook and open a shell on OS X. Use cp > to copy a file with a resource fork. Oh look, the Finder has no idea > what the new file is, even though it looks exactly identical in the > shell. Isn't that _wonderful_? Now try cat < a > b on a file with a > fork. How is that ever going to work? > > I like cat < a > b. You can keep your progress. cat <a >b does not preserve following file properties even on standard UNIX filesystems: name,owner,group,permissions. -- vda ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 7:34 ` Denis Vlasenko @ 2004-08-26 11:42 ` Rik van Riel 2004-08-26 12:00 ` Christophe Saout ` (3 more replies) 0 siblings, 4 replies; 1392+ messages in thread From: Rik van Riel @ 2004-08-26 11:42 UTC (permalink / raw) To: Denis Vlasenko Cc: Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Denis Vlasenko wrote: > > I like cat < a > b. You can keep your progress. > > cat <a >b does not preserve following file properties even on standard > UNIX filesystems: name,owner,group,permissions. Losing permissions is one thing. Annoying, mostly. However, actual losing file data during such a copy is nothing short of a disaster, IMHO. In my opinion we shouldn't merge file-as-a-directory semantics into the kernel until we figure out how to fix the backup/restore problem and keep standard unix utilities work. -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:42 ` Rik van Riel @ 2004-08-26 12:00 ` Christophe Saout 2004-08-28 11:00 ` Yury Umanets 2004-08-28 11:12 ` Yury Umanets 2004-08-26 12:12 ` Jamie Lokier ` (2 subsequent siblings) 3 siblings, 2 replies; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 12:00 UTC (permalink / raw) To: Rik van Riel Cc: Denis Vlasenko, Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 3556 bytes --] Am Donnerstag, den 26.08.2004, 07:42 -0400 schrieb Rik van Riel: > > > I like cat < a > b. You can keep your progress. > > > > cat <a >b does not preserve following file properties even on standard > > UNIX filesystems: name,owner,group,permissions. > > Losing permissions is one thing. Annoying, mostly. > > However, actual losing file data during such a copy is > nothing short of a disaster, IMHO. > > In my opinion we shouldn't merge file-as-a-directory > semantics into the kernel until we figure out how to > fix the backup/restore problem and keep standard unix > utilities work. Well, again, what about xattrs and ACLs...? Actually, reiser4 doesn't currently implement storage of arbitrary user data under a file. It's just some sysfs-like information that can be retrieved, some properties can be changed. If you copy a file the worst thing that can happen right now is that you lose the information whether that file should be encrypted or compressed. You don't lose data. In case some people are wondering, this is what you can find in the reiser4 metas: leto:/home/chtephan/.muttrc/metas > la insgesamt 1 dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 . -rwx------ 1 chtephan users 290 12. Jan 2004 .. -r--r--r-- 1 chtephan users 0 26. Aug 13:52 bmap -rw-r--r-- 1 chtephan users 0 26. Aug 13:52 gid -r--r--r-- 1 chtephan users 0 26. Aug 13:52 items -r--r--r-- 1 chtephan users 0 26. Aug 13:52 key -r--r--r-- 1 chtephan users 0 26. Aug 13:52 locality --w------- 1 chtephan users 0 26. Aug 13:52 new -r--r--r-- 1 chtephan users 0 26. Aug 13:52 nlink -r--r--r-- 1 chtephan users 0 26. Aug 13:52 oid dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 plugin -r--r--r-- 1 chtephan users 0 26. Aug 13:52 pseudo -r--r--r-- 1 chtephan users 0 26. Aug 13:52 readdir -rw-r--r-- 1 chtephan users 0 26. Aug 13:52 rwx -r--r--r-- 1 chtephan users 0 26. Aug 13:52 size -rw-r--r-- 1 chtephan users 0 26. Aug 13:52 uid Some of them are reiser4 specific things (items, key, locality, oid) which can only be queried anyway. The rest is what you can also get using other VFS calls. leto:/home/chtephan/.muttrc/metas/plugin > la insgesamt 0 dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 . dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 .. -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 compression -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 crypto -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 digest -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 dir -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 dir_item -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 fibration -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 file -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 formatting -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 hash -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 perm -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 sd Here you can change reiser4 specific things. Here you can change some properties how the file is stored in the reiser4 tree. For example you can activate compression, encryption or authentication or set some hints to optimize speed. You can't create or remove these pseudo-files, just like in sysfs. I don't know, but if someone wants to store user data I think you would have to implement a directory "userdata" or something where you can store it. Not all of the information does make a lot sense inside a modified tar. Who is interested in the bmap information of the file or some other read-only information? [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 12:00 ` Christophe Saout @ 2004-08-28 11:00 ` Yury Umanets 2004-08-28 11:12 ` Yury Umanets 1 sibling, 0 replies; 1392+ messages in thread From: Yury Umanets @ 2004-08-28 11:00 UTC (permalink / raw) To: Christophe Saout Cc: Rik van Riel, Denis Vlasenko, Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Christophe Saout wrote: >Am Donnerstag, den 26.08.2004, 07:42 -0400 schrieb Rik van Riel: > > > >>>>I like cat < a > b. You can keep your progress. >>>> >>>> >>>cat <a >b does not preserve following file properties even on standard >>>UNIX filesystems: name,owner,group,permissions. >>> >>> >>Losing permissions is one thing. Annoying, mostly. >> >>However, actual losing file data during such a copy is >>nothing short of a disaster, IMHO. >> >>In my opinion we shouldn't merge file-as-a-directory >>semantics into the kernel until we figure out how to >>fix the backup/restore problem and keep standard unix >>utilities work. >> >> > >Well, again, what about xattrs and ACLs...? > >Actually, reiser4 doesn't currently implement storage of arbitrary user >data under a file. > >It's just some sysfs-like information that can be retrieved, some >properties can be changed. If you copy a file the worst thing that can >happen right now is that you lose the information whether that file >should be encrypted or compressed. You don't lose data. > >In case some people are wondering, this is what you can find in the >reiser4 metas: > >leto:/home/chtephan/.muttrc/metas > la >insgesamt 1 >dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 . >-rwx------ 1 chtephan users 290 12. Jan 2004 .. >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 bmap >-rw-r--r-- 1 chtephan users 0 26. Aug 13:52 gid >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 items >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 key >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 locality >--w------- 1 chtephan users 0 26. Aug 13:52 new >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 nlink >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 oid >dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 plugin >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 pseudo >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 readdir >-rw-r--r-- 1 chtephan users 0 26. Aug 13:52 rwx >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 size >-rw-r--r-- 1 chtephan users 0 26. Aug 13:52 uid > >Some of them are reiser4 specific things (items, key, locality, oid) >which can only be queried anyway. > >The rest is what you can also get using other VFS calls. > >leto:/home/chtephan/.muttrc/metas/plugin > la >insgesamt 0 >dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 . >dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 .. >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 compression >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 crypto >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 digest >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 dir >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 dir_item >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 fibration >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 file >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 formatting >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 hash >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 perm >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 sd > >Here you can change reiser4 specific things. Here you can change some >properties how the file is stored in the reiser4 tree. For example you >can activate compression, encryption or authentication or set some hints >to optimize speed. > > I suppose that you are talking about the prinsiple ability to change these things. Because in real life it will also need some kind of convertion from one look (say with old plugin) to another one. Just like tail conversion rebuilds file's body after tail policy plugin gets changed. -- umka ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 12:00 ` Christophe Saout 2004-08-28 11:00 ` Yury Umanets @ 2004-08-28 11:12 ` Yury Umanets 1 sibling, 0 replies; 1392+ messages in thread From: Yury Umanets @ 2004-08-28 11:12 UTC (permalink / raw) To: Christophe Saout Cc: Rik van Riel, Denis Vlasenko, Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Christophe Saout wrote: >Am Donnerstag, den 26.08.2004, 07:42 -0400 schrieb Rik van Riel: > > > >>>>I like cat < a > b. You can keep your progress. >>>> >>>> >>>cat <a >b does not preserve following file properties even on standard >>>UNIX filesystems: name,owner,group,permissions. >>> >>> >>Losing permissions is one thing. Annoying, mostly. >> >>However, actual losing file data during such a copy is >>nothing short of a disaster, IMHO. >> >>In my opinion we shouldn't merge file-as-a-directory >>semantics into the kernel until we figure out how to >>fix the backup/restore problem and keep standard unix >>utilities work. >> >> > >Well, again, what about xattrs and ACLs...? > >Actually, reiser4 doesn't currently implement storage of arbitrary user >data under a file. > >It's just some sysfs-like information that can be retrieved, some >properties can be changed. If you copy a file the worst thing that can >happen right now is that you lose the information whether that file >should be encrypted or compressed. You don't lose data. > >In case some people are wondering, this is what you can find in the >reiser4 metas: > >leto:/home/chtephan/.muttrc/metas > la >insgesamt 1 >dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 . >-rwx------ 1 chtephan users 290 12. Jan 2004 .. >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 bmap >-rw-r--r-- 1 chtephan users 0 26. Aug 13:52 gid >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 items >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 key >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 locality >--w------- 1 chtephan users 0 26. Aug 13:52 new >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 nlink >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 oid >dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 plugin >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 pseudo >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 readdir >-rw-r--r-- 1 chtephan users 0 26. Aug 13:52 rwx >-r--r--r-- 1 chtephan users 0 26. Aug 13:52 size >-rw-r--r-- 1 chtephan users 0 26. Aug 13:52 uid > >Some of them are reiser4 specific things (items, key, locality, oid) >which can only be queried anyway. > >The rest is what you can also get using other VFS calls. > >leto:/home/chtephan/.muttrc/metas/plugin > la >insgesamt 0 >dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 . >dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 .. >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 compression >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 crypto >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 digest >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 dir >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 dir_item >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 fibration >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 file >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 formatting >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 hash >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 perm >-rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 sd > >Here you can change reiser4 specific things. Here you can change some >properties how the file is stored in the reiser4 tree. For example you >can activate compression, encryption or authentication or set some hints >to optimize speed. > > I suppose that you are talking about the principle ability to change these things. Because in real life it will also need some kind of conversion from one look (say with old plugin) to another one. Just like tail conversion rebuilds file's body after tail policy plugin gets changed. -- umka ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:42 ` Rik van Riel 2004-08-26 12:00 ` Christophe Saout @ 2004-08-26 12:12 ` Jamie Lokier 2004-08-26 12:25 ` Wichert Akkerman ` (2 more replies) 2004-08-26 20:35 ` Lee Revell 2004-08-27 0:02 ` Hans Reiser 3 siblings, 3 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 12:12 UTC (permalink / raw) To: Rik van Riel Cc: Denis Vlasenko, Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Rik van Riel wrote: > > cat <a >b does not preserve following file properties even on standard > > UNIX filesystems: name,owner,group,permissions. > > Losing permissions is one thing. Annoying, mostly. I find losing mtime annoying. That's why I don't use Mozilla's download manager (and Navigator's before it) because they presumed that mtime isn't worth copying. ncftp and wget do the right thing, but it's annoying that I can't use the fancy tools and get the metadata _I_ want transferred by them. > However, actual losing file data during such a copy is > nothing short of a disaster, IMHO. Note that file-as-directory doesn't imply that you can store just anything into those directories. Is it a problem to decree that "file data MUST NOT be stored in a metadata directory; only non-essential metadata and data computed from the file data may be stored"? > In my opinion we shouldn't merge file-as-a-directory > semantics into the kernel until we figure out how to > fix the backup/restore problem and keep standard unix > utilities work. Same as xattrs, for data that should be stored. But, unlike xattrs, while permissions and C-source-encoding should be stored, some of this metadata should _not_ be stored when making an archive and perhaps not when making a backup either. Personally I wouldn't want an archive containing a file to save and restore recomputable metadata like cached, extracted-on-demand MP3 id tags and email headers-as-metadata. I'd rather they were recalculated when needed, implying a stronger guarantee about their relationship to the data from which they're extracted - and implying that a user program might not be allowed to set them to inconsistent values. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 12:12 ` Jamie Lokier @ 2004-08-26 12:25 ` Wichert Akkerman 2004-09-03 4:36 ` Grzegorz Jaśkiewicz 2004-09-03 4:40 ` Grzegorz Jaśkiewicz 2 siblings, 0 replies; 1392+ messages in thread From: Wichert Akkerman @ 2004-08-26 12:25 UTC (permalink / raw) To: Jamie Lokier Cc: Rik van Riel, Denis Vlasenko, Matt Mackall, Nicholas Miell, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Previously Jamie Lokier wrote: > I find losing mtime annoying. That's why I don't use Mozilla's > download manager (and Navigator's before it) because they presumed > that mtime isn't worth copying. It's free software, you could fix that :) > Is it a problem to decree that "file data MUST NOT be stored in a > metadata directory; only non-essential metadata and data computed from > the file data may be stored"? That would put us right back at the samba-and-other-apps-want-streams point in the discussion, but it certainly would not be a bad starting point. From there we could always add streams later on if the API is right. Wichert. -- Wichert Akkerman <wichert@wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 12:12 ` Jamie Lokier 2004-08-26 12:25 ` Wichert Akkerman @ 2004-09-03 4:36 ` Grzegorz Jaśkiewicz 2004-09-03 8:01 ` Frank van Maarseveen 2004-09-03 17:12 ` Horst von Brand 2004-09-03 4:40 ` Grzegorz Jaśkiewicz 2 siblings, 2 replies; 1392+ messages in thread From: Grzegorz Jaśkiewicz @ 2004-09-03 4:36 UTC (permalink / raw) To: Jamie Lokier Cc: Rik van Riel, Denis Vlasenko, Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Jamie Lokier wrote: >Note that file-as-directory doesn't imply that you can store just >anything into those directories. > >Is it a problem to decree that "file data MUST NOT be stored in a >metadata directory; only non-essential metadata and data computed from >the file data may be stored"? > > That's exactly what folks seem to lost in this debate. We (developers) want to have files as dirs to: * be able to extract/modify file part on fly. For instance, simple .iso file, without need to mount it over loop (btw, if someone is going to redesign VFS, can we handle that kinda case too please?), .tar, or unzipped (kernel can handle zips) "streams". File is a stream, why shouldn't I have a chance to grab a stream out of archive/image than ? * be able to store metadata, that doesn't matter on copy, and even should not be copied sometimes. Things like thumbnails, metainformation (used for search). All of these extracted from acctual file (either by VFS 'plugins', or by userspace application), and saved there so that it can be used later on. So we save some time on extracting it. -- GJ ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-03 4:36 ` Grzegorz Jaśkiewicz @ 2004-09-03 8:01 ` Frank van Maarseveen 2004-09-03 17:12 ` Horst von Brand 1 sibling, 0 replies; 1392+ messages in thread From: Frank van Maarseveen @ 2004-09-03 8:01 UTC (permalink / raw) To: Grzegorz Ja??kiewicz Cc: Jamie Lokier, Rik van Riel, Denis Vlasenko, Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, Sep 03, 2004 at 06:36:41AM +0200, Grzegorz Ja??kiewicz wrote: > Jamie Lokier wrote: > > >Is it a problem to decree that "file data MUST NOT be stored in a > >metadata directory; only non-essential metadata and data computed from > >the file data may be stored"? > > That's exactly what folks seem to lost in this debate. And that's not even the most important part: How would you educate all the developers _not_ on this list: application developers? I think this is just a disaster waiting for us to occur. > * be able to extract/modify file part on fly. > * be able to store metadata, that doesn't matter on copy those may be orthogonal. -- Frank ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-03 4:36 ` Grzegorz Jaśkiewicz 2004-09-03 8:01 ` Frank van Maarseveen @ 2004-09-03 17:12 ` Horst von Brand 1 sibling, 0 replies; 1392+ messages in thread From: Horst von Brand @ 2004-09-03 17:12 UTC (permalink / raw) To: Grzegorz Jaśkiewicz Cc: Jamie Lokier, Rik van Riel, Denis Vlasenko, Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list =?UTF-8?B?R3J6ZWdvcnogSmHFm2tpZXdpY3o=?= <gj@pointblue.com.pl> said: > Jamie Lokier wrote: > >Note that file-as-directory doesn't imply that you can store just > >anything into those directories. > >Is it a problem to decree that "file data MUST NOT be stored in a > >metadata directory; only non-essential metadata and data computed from > >the file data may be stored"? > That's exactly what folks seem to lost in this debate. > We (developers) want to have files as dirs to: > * be able to extract/modify file part on fly. For instance, simple .iso > file, without need to mount it over loop (btw, if someone is going to > redesign VFS, can we handle that kinda case too please?), .tar, or > unzipped (kernel can handle zips) "streams". File is a stream, why > shouldn't I have a chance to grab a stream out of archive/image than ? ISO9660 is designed for read-only, i.e., modifying something in the middle is probably as hard to do as just unpack/modify/repack. Same for tar, cpio, and compressed versions of same. No gain, for a major hassle implementing this. > * be able to store metadata, that doesn't matter on copy, and even > should not be copied sometimes. Things like thumbnails, metainformation > (used for search). All of these extracted from acctual file (either by > VFS 'plugins', or by userspace application), and saved there so that it > can be used later on. So we save some time on extracting it. You save iff the resources spent updating this stuff is less than recomputing on the fly each time. Also, placing random stuff in the kernel is _not_ "zero cost", it is _very_ expensive (kernel memory is always resident, kernel code is _extremely_ critical, once it is in the kernel it is very hard to change interfaces). Besides, this kind of "metadata" will most probably be different for each user, and a user should also be able to associate it to files she has no write right on. I.e., keep it in a per-user directory, managed by whatever GUI tools she uses. What you (developer) wish for and what kernel hackers are able to deliver aren't the same. As it stands this is a badly-thought-through proposal, with very complex ramifications that few here are aware of (see Al Viro's and Linus' posts about locking; it breaks the fundamental principle that something is a file or a directory, so this would break many applications), and no clean solutions as of now. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 12:12 ` Jamie Lokier 2004-08-26 12:25 ` Wichert Akkerman 2004-09-03 4:36 ` Grzegorz Jaśkiewicz @ 2004-09-03 4:40 ` Grzegorz Jaśkiewicz 2 siblings, 0 replies; 1392+ messages in thread From: Grzegorz Jaśkiewicz @ 2004-09-03 4:40 UTC (permalink / raw) To: Jamie Lokier Cc: Rik van Riel, Denis Vlasenko, Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Oh, and if I may notice something more. Just my opinion. It's going to be 'samba uses only' thing, as long as it's not in VFS. xattrs are not so much needed as file streams there. So argument that xattrs exists already in VFS, and are not used, so why do we need streams there, and that it wouldn't change anything is just silly. Btw, reiserfs4 on my laptop is providing very nice base for KDE source tree (XFS sucked so much, I had to replace it). Saving me some time on recompilation, and cvs up. Thanks. -- GJ ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:42 ` Rik van Riel 2004-08-26 12:00 ` Christophe Saout 2004-08-26 12:12 ` Jamie Lokier @ 2004-08-26 20:35 ` Lee Revell 2004-08-27 0:02 ` Hans Reiser 3 siblings, 0 replies; 1392+ messages in thread From: Lee Revell @ 2004-08-26 20:35 UTC (permalink / raw) To: Rik van Riel Cc: Denis Vlasenko, Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 2004-08-26 at 07:42, Rik van Riel wrote: > On Thu, 26 Aug 2004, Denis Vlasenko wrote: > > > > I like cat < a > b. You can keep your progress. > > > > cat <a >b does not preserve following file properties even on standard > > UNIX filesystems: name,owner,group,permissions. > > Losing permissions is one thing. Annoying, mostly. > cat <a >b does preserve permissions - the permissions of b. I always considered this a feature. Lee ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:42 ` Rik van Riel ` (2 preceding siblings ...) 2004-08-26 20:35 ` Lee Revell @ 2004-08-27 0:02 ` Hans Reiser 3 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-27 0:02 UTC (permalink / raw) To: Rik van Riel Cc: Denis Vlasenko, Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Rik van Riel wrote: > > >In my opinion we shouldn't merge file-as-a-directory >semantics into the kernel until we figure out how to >fix the backup/restore problem and keep standard unix >utilities work. > > > One incremental piece at a time our solution will fall into place. Pieces should go in when they are individually ready. Warning users that some pieces might be changed out from under them up until when 2.8 ships is ok with me though. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 5:32 ` Matt Mackall 2004-08-26 7:34 ` Denis Vlasenko @ 2004-08-26 7:53 ` Markus Törnqvist 2004-08-26 16:32 ` Jan Harkes 2004-09-03 4:43 ` Grzegorz Jaśkiewicz 2004-08-26 8:31 ` Helge Hafting ` (2 subsequent siblings) 4 siblings, 2 replies; 1392+ messages in thread From: Markus Törnqvist @ 2004-08-26 7:53 UTC (permalink / raw) To: Matt Mackall Cc: Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 12:32:00AM -0500, Matt Mackall wrote: > >What it breaks is the concept of a file. In ways that are ill-defined, >not portable, hard to work with, and needlessly complex. Along the >way, it breaks every single application that ever thought it knew what >a file was. It breaks the concept of a file. In ways that offer more versatility, challenge the imagination to make even better progress and keeps Linux competing with competitors who are implementing this stuff as we speak. I for one would truly welcome the coming of thumbnails and descriptions in picture files, because I have a real-life project going on where that would be extremely handy to have in the actual file. Were I any richer, I'd pay Namesys to have this work for me :) >Find some silly person with an iBook and open a shell on OS X. Use cp >to copy a file with a resource fork. Oh look, the Finder has no idea >what the new file is, even though it looks exactly identical in the >shell. Isn't that _wonderful_? Now try cat < a > b on a file with a >fork. How is that ever going to work? Then I guess OS X ships a broken implementation of cp, yes? On the cat example, what if cat < a > b simply copies the "main stream" and not the metadata, as a feature. The key being, "as a feature" The metadata streams could get file descriptors of their own OR another program, streamcat or something, could be written to compensate. >I like cat < a > b. You can keep your progress. With all due respect, I hope not too many people agree with you :) -- mjt ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 7:53 ` Markus Törnqvist @ 2004-08-26 16:32 ` Jan Harkes 2004-08-26 16:48 ` Linus Torvalds 2004-09-03 4:43 ` Grzegorz Jaśkiewicz 1 sibling, 1 reply; 1392+ messages in thread From: Jan Harkes @ 2004-08-26 16:32 UTC (permalink / raw) To: Markus Törnqvist Cc: Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 10:53:48AM +0300, Markus Törnqvist wrote: > On Thu, Aug 26, 2004 at 12:32:00AM -0500, Matt Mackall wrote: > >What it breaks is the concept of a file. In ways that are ill-defined, > >not portable, hard to work with, and needlessly complex. Along the > >way, it breaks every single application that ever thought it knew what > >a file was. > > It breaks the concept of a file. In ways that offer more versatility, > challenge the imagination to make even better progress and keeps > Linux competing with competitors who are implementing this stuff > as we speak. Sigh, it really isn't all that groundbreaking. The only special feature is the fact that it introduces directory hardlinks. Just turn the phrase 'file-as-dir' around and you might see the point. As far as userspace is concerned the implementation would behave 99% identical if we provided a 'dir-as-file' semantics. i.e. opening a directory without 'O_DIRECTORY' would return a filehandle to a default file within the directory. Something like open("/path/to/dir", O_RDONLY) would be translated to open('/path/to/dir/_contents', O_RDONLY). So I could simply replace all my files with directories and move the original file contents into a '_contents' file. Applications can still open them by the same name, although the attributes would look a bit different. Poof, we have the same file-as-dir concept, but now we play fair with the existing VFS and don't introduced races/deadlocks and other issues. Also existing backup tools will be able to back up and restore these 'streams'. (btw. the same could be implemented completely in userspace, for instance in glibc. Whenever an open call gets an EISDIR error, simply retry the open, but this time with /_contents appended). > I for one would truly welcome the coming of thumbnails and descriptions > in picture files, because I have a real-life project going on where > that would be extremely handy to have in the actual file. Ehhh, that already exists and doesn't require meta-data streams, http://www.exif.org/ > >Find some silly person with an iBook and open a shell on OS X. Use cp > >to copy a file with a resource fork. Oh look, the Finder has no idea > >what the new file is, even though it looks exactly identical in the > >shell. Isn't that _wonderful_? Now try cat < a > b on a file with a > >fork. How is that ever going to work? > > Then I guess OS X ships a broken implementation of cp, yes? No cp is working just fine, there is no copy syscall in the VFS, so any file system specific features such meta data streams will have to be handled by the application. So at some point 'cp' will have to be taught about macos resource forks, posix acls, xattrs, afs acls, reiserfs file-as-dir, freebsd attributes, samba/ntfs streams. And possibly even learn the semantical differences between these so that it can copy metadata between the various file systems types. Jan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:32 ` Jan Harkes @ 2004-08-26 16:48 ` Linus Torvalds 2004-08-26 18:07 ` Jan Harkes ` (2 more replies) 0 siblings, 3 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-08-26 16:48 UTC (permalink / raw) To: Jan Harkes Cc: Markus Törnqvist, Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Jan Harkes wrote: > > (btw. the same could be implemented completely in userspace, for > instance in glibc. Whenever an open call gets an EISDIR error, simply > retry the open, but this time with /_contents appended). Yes, and no - just to make it obvious before people jump on this, a lot of things can be prototyped in user space with things like this, but once you have to deal with races and mixed tool environments, user space suddenly doesn't work so well any more. I think Jan understands this distinction, I just wanted to make sure everybody else is aware of the _one_ thing that kernel land does well: - safely synchronize globally visible data structures That's quite fundamental. 99% of what a kernel does is exactly that. TCP would be in user space too, if it wasn't for _exactly_ this issue. A lot of people think that kernels are about hardware access, and yes, that's the other 99% of the picture (I see the _big_ picture, remember?), but the "safe access to common data" is really very fundamental. The kernel is literally the thing that makes sure that you don't have - and _cannot_ have - user programs that confuse each other by modifying data unsynchronized. For example, a filesystem is really nothing but a way to access a disk in a controlled manner - it's not so much about hardware access, as it is about maintaining a coherent view of how some shared data (disk or whatever) is maintained. Same goes for caches. We could cache things in user space, but if you want to _share_ your caches (so that you don't have to re-load them for every new application), you need some entity that manages those shared data structures in a secure manner. In other words, you need the kernel. The same goes for something like a "container file". Whether you see it as "dir-as-file" or "file-as-dir" (and I agree with Jan that the two are totally equivalent), the point of having the capability in the kernel is not that the operations cannot be done in user space - the point is that they cannot be done in user space _safely_. The kernel is kind of the thing that guarantees that everybody follows the rules. Imagine the security problems if a set-uid program were to (unwittingly) depend on a user-space library that implements what Jan's prototype library would do? Races galore, since a user-space implementation wouldn't have _any_ way to do tests like the above atomically. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:48 ` Linus Torvalds @ 2004-08-26 18:07 ` Jan Harkes 2004-08-31 3:39 ` Tom Vier 2004-09-10 19:22 ` Timothy Miller 2 siblings, 0 replies; 1392+ messages in thread From: Jan Harkes @ 2004-08-26 18:07 UTC (permalink / raw) To: Linus Torvalds Cc: Andrew Morton, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list, viro I trimmed/adjusted the CC list somewhat in the hope of getting a more productive discussion going. On Thu, Aug 26, 2004 at 09:48:04AM -0700, Linus Torvalds wrote: > On Thu, 26 Aug 2004, Jan Harkes wrote: > > > > (btw. the same could be implemented completely in userspace, for > > instance in glibc. Whenever an open call gets an EISDIR error, simply > > retry the open, but this time with /_contents appended). > > Yes, and no - just to make it obvious before people jump on this, a lot of > things can be prototyped in user space with things like this, but once you > have to deal with races and mixed tool environments, user space suddenly > doesn't work so well any more. Agreed. > The same goes for something like a "container file". Whether you see it as > "dir-as-file" or "file-as-dir" (and I agree with Jan that the two are > totally equivalent), the point of having the capability in the kernel is Only to some point. I was mostly trying to make the point that similar functionality is already achievable within the current VFS framework. However, files and directories still have some semantical differences, and these are probably the points that we would have to address to get Hans's efforts work well in cooperation with the Linux VFS. We cannot atomically unlink a dir-as-file, but can unlink a file-as-dir. In this case, the dcache seems to have no problems unlinking/unhashing a non-empty directory, so I'm assuming that the unlink problem is mostly enforced to avoid unreachable, but still allocated files in a file system. (I'm sure that Viro can probably come up with other reasons why unlinking a non-empty directory can be evil). Also, we cannot create hardlinks to a directory (-as-file), but I guess reiserfs4 allows hardlinks to files (-as-directory). Luckily as far as hardlinked directories are concerned the file-as-dir implementation should be a relatively benign subset. I assume that we are not allowed to rename or link arbitrary objects from the regular fs into it, Coda returns EXDEV in such situations. As a result there will be no cycles and this simple premise should get rid of at least a couple of the possible deadlock scenarios that I've seen mentioned. Jan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:48 ` Linus Torvalds 2004-08-26 18:07 ` Jan Harkes @ 2004-08-31 3:39 ` Tom Vier 2004-08-31 4:08 ` Linus Torvalds 2004-09-10 19:22 ` Timothy Miller 2 siblings, 1 reply; 1392+ messages in thread From: Tom Vier @ 2004-08-31 3:39 UTC (permalink / raw) To: Linus Torvalds; +Cc: linux-kernel On Thu, Aug 26, 2004 at 09:48:04AM -0700, Linus Torvalds wrote: > - safely synchronize globally visible data structures > That's quite fundamental. 99% of what a kernel does is exactly that. TCP > would be in user space too, if it wasn't for _exactly_ this issue. A lot What about microkernels? They do tcp in userspace. So did winsock, iirc. As long as a trusted process keeps data such as free ports, what's the problem? Perhaps a process could even simply hand off proto:port handles and let a library do the rest, but i know little about exokernels. (There's probably some catches.) -- Tom Vier <tmv@comcast.net> DSA Key ID 0x15741ECE ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 3:39 ` Tom Vier @ 2004-08-31 4:08 ` Linus Torvalds 2004-08-31 4:38 ` Jeff Garzik ` (3 more replies) 0 siblings, 4 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-08-31 4:08 UTC (permalink / raw) To: Tom Vier; +Cc: linux-kernel On Mon, 30 Aug 2004, Tom Vier wrote: > On Thu, Aug 26, 2004 at 09:48:04AM -0700, Linus Torvalds wrote: > > - safely synchronize globally visible data structures > > That's quite fundamental. 99% of what a kernel does is exactly that. TCP > > would be in user space too, if it wasn't for _exactly_ this issue. A lot > > What about microkernels? They do tcp in userspace. No they don't. They do TCP in a separate address space from user space, that just also happens to be separate from the "microkernel address space". So a microkernel will have _more_ address spaces, and they won't be "user space". They'll be "server deamon space" or something. Now, that's also why they tend to have performance problems - because you need to copy the data between different address spaces, and switch the CPU context etc around. Not user space. They may be "ring 3" from a CPU standpoint, but they aren't user space from a _user_ standpoint - it's still very much a separate address space, with domain protection. > So did winsock, iirc. Now that is a different case. Things like the PalmOS TCP stack (and, I believe, Winsock) are true "user space" TCP stacks, in that they really do live as libraries in the same address space as the user app. It sucks. Exactly because now the data structures are _not_ protected, and they are _not_ shared. So your library basically ends up being a "single client" library, with no protection between clients (or no sharing: you can have "protected" single-stream TCP, but then you won't share the TCP state that needs to be shared like listen queues etc). This works in an environment like Palm or Win-3.0, which are really just single-client _anyway_, without any protection. But notice how Windows started doing TCP in the kernel, and notice how you can actually use it as a server these days? In short: you _need_ to have a separate address space (either kernel, or "TCP server" or whatever) if you want to have reliable, secure and generally usable TCP. > As long as a trusted process keeps data such as free ports, what's the > problem? None - because it's not user space any more. Well, performance might still suck, of course. And it does. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 4:08 ` Linus Torvalds @ 2004-08-31 4:38 ` Jeff Garzik 2004-08-31 5:30 ` Userspace file systems & MKs (Re: silent semantic changes with reiser4) Bill Huey 2004-08-31 19:01 ` silent semantic changes with reiser4 Ryan Breen 2004-08-31 10:57 ` Alan Cox ` (2 subsequent siblings) 3 siblings, 2 replies; 1392+ messages in thread From: Jeff Garzik @ 2004-08-31 4:38 UTC (permalink / raw) To: Linus Torvalds; +Cc: Tom Vier, linux-kernel Linus Torvalds wrote: > > On Mon, 30 Aug 2004, Tom Vier wrote: > > >>On Thu, Aug 26, 2004 at 09:48:04AM -0700, Linus Torvalds wrote: >> >>> - safely synchronize globally visible data structures >>>That's quite fundamental. 99% of what a kernel does is exactly that. TCP >>>would be in user space too, if it wasn't for _exactly_ this issue. A lot >> >>What about microkernels? They do tcp in userspace. > > > No they don't. They do TCP in a separate address space from user space, > that just also happens to be separate from the "microkernel address > space". > > So a microkernel will have _more_ address spaces, and they won't be "user > space". They'll be "server deamon space" or something. Now, that's also > why they tend to have performance problems - because you need to copy the > data between different address spaces, and switch the CPU context etc > around. Man, this thread came a long way. I peek in and see a pet topic... _My_ own definition of microkernel, which differs from traditional CS, is where I would love to see Linux go: move as much as humanly possible to userspace, such that, the kernel only contains pagecache operations, driver fast paths, and security checks. Move slow paths, including ACPI probing, PCI bus walking, driver init/reset paths, some of the ioctl handling, to userspace. Be willing to accept extra context switches as a cost in all but the fast paths. What you have left after you move all the slow paths to userspace is my version of a microkernel. The kernel is still monolithic and a single address space, but a lot smaller. Now this (a) is likely just a pipe dream and (b) will increase complexity because each driver will _require_ a userspace component, but hey... you gotta have goals in life. Jeff ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Userspace file systems & MKs (Re: silent semantic changes with reiser4) 2004-08-31 4:38 ` Jeff Garzik @ 2004-08-31 5:30 ` Bill Huey 2004-08-31 5:56 ` Jeff Garzik 2004-08-31 9:01 ` Tonnerre 2004-08-31 19:01 ` silent semantic changes with reiser4 Ryan Breen 1 sibling, 2 replies; 1392+ messages in thread From: Bill Huey @ 2004-08-31 5:30 UTC (permalink / raw) To: Jeff Garzik; +Cc: Linus Torvalds, Tom Vier, linux-kernel On Tue, Aug 31, 2004 at 12:38:14AM -0400, Jeff Garzik wrote: > _My_ own definition of microkernel, which differs from traditional CS, > is where I would love to see Linux go: move as much as humanly possible > to userspace, such that, the kernel only contains pagecache operations, > driver fast paths, and security checks. Move slow paths, including ACPI > probing, PCI bus walking, driver init/reset paths, some of the ioctl > handling, to userspace. Be willing to accept extra context switches as > a cost in all but the fast paths. http://www.dragonflybsd.org DragonFly BSD, the only remotely functional open source BSD project on this planet, has plans in place to push certain kernel components like their VFS layer into userspace for easier debugging, testing and other things that go with developing file systems easily. If they back it with something like C++ for doing constructor style type conversion on top of overloaded operators to back VFS data structure translation, could easily import stuff like most Linux file systems without major restructuring, say, if they had their translation library written. In this case, userspace kernel systems have some serious programming advantages over traditional kernels. They're also pushing an async syscall model to support a non-1:1 threading system for userspace unlike what Linux has done with futexes. It'll allow for them to plug in various userspace schedulers at will for whatever scheduling policy you want dynamically, even while the program is running. This isn't mentioning their single system image clustering goals that have abstracted various kernel systems like their VFS stuff to a messaging model which should get them proccess migration as an intrinsic part of their OS design. They've made a lot of progress over the last year and it's largely because of Matt Dillon's ability to create a non-overly political group that creates trust. He's a true leader, not some random loud mouth(s) unlike other BSD efforts. It's much like the Linux community in this regard with Linus. bill ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4) 2004-08-31 5:30 ` Userspace file systems & MKs (Re: silent semantic changes with reiser4) Bill Huey @ 2004-08-31 5:56 ` Jeff Garzik 2004-08-31 6:50 ` Bill Huey [not found] ` <20040831155613.2b25df1e.diegocg@teleline.es> 2004-08-31 9:01 ` Tonnerre 1 sibling, 2 replies; 1392+ messages in thread From: Jeff Garzik @ 2004-08-31 5:56 UTC (permalink / raw) To: Bill Huey (hui); +Cc: Linus Torvalds, Tom Vier, linux-kernel Bill Huey (hui) wrote: > DragonFly BSD, the only remotely functional open source BSD project on this > planet, has plans in place to push certain kernel components like their VFS > layer into userspace for easier debugging, testing and other things that go That violates Jeff's First Rule, put the fast path stuff in the kernel. > with developing file systems easily. If they back it with something like C++ > for doing constructor style type conversion on top of overloaded operators > to back VFS data structure translation, could easily import stuff like most > Linux file systems without major restructuring, say, if they had their > translation library written. In this case, userspace kernel systems have > some serious programming advantages over traditional kernels. Sounds like security would be a pain in the ass :) > They're also pushing an async syscall model to support a non-1:1 threading > system for userspace unlike what Linux has done with futexes. It'll allow messaging passing is also known as "really slow C function calls" My PCI bus passes messages all the time. A message is in the eye of the beholder. Jeff ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4) 2004-08-31 5:56 ` Jeff Garzik @ 2004-08-31 6:50 ` Bill Huey [not found] ` <20040831155613.2b25df1e.diegocg@teleline.es> 1 sibling, 0 replies; 1392+ messages in thread From: Bill Huey @ 2004-08-31 6:50 UTC (permalink / raw) To: Jeff Garzik; +Cc: Bill Huey (hui), Linus Torvalds, Tom Vier, linux-kernel On Tue, Aug 31, 2004 at 01:56:45AM -0400, Jeff Garzik wrote: > Bill Huey (hui) wrote: > >DragonFly BSD, the only remotely functional open source BSD project on this > >planet, has plans in place to push certain kernel components like their VFS > >layer into userspace for easier debugging, testing and other things that go > > That violates Jeff's First Rule, put the fast path stuff in the kernel. It's not for native file system, but one that are coming around. I kind of misspoke when I said that the VFS layer is going to be pushed into userspace. It's not. Access to the VFS layer is going to be exported to userspace. The system is interesting because of its potential programming flexibility. > >with developing file systems easily. If they back it with something like > >C++ > >for doing constructor style type conversion on top of overloaded operators > >to back VFS data structure translation, could easily import stuff like most > >Linux file systems without major restructuring, say, if they had their > >translation library written. In this case, userspace kernel systems have > >some serious programming advantages over traditional kernels. > > Sounds like security would be a pain in the ass :) Don't know. But if you can get away from the typical C programming errors, then it could be a win. > >They're also pushing an async syscall model to support a non-1:1 threading > >system for userspace unlike what Linux has done with futexes. It'll allow > > messaging passing is also known as "really slow C function calls" > > My PCI bus passes messages all the time. A message is in the eye of the > beholder. Message passing is used as a method of concurrency in that system outside of the typical use of function calles. It makes access to kernel facilities, remote or local, transparent. They also don't use a mutex for their locking, but something called a token which is per-CPU kind of locking primitive. It's definitely an interesting system with a lot of potential. bill ^ permalink raw reply [flat|nested] 1392+ messages in thread
[parent not found: <20040831155613.2b25df1e.diegocg@teleline.es>]
* Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4) [not found] ` <20040831155613.2b25df1e.diegocg@teleline.es> @ 2004-08-31 20:52 ` Bill Huey 2004-09-02 17:20 ` Diego Calleja 2004-08-31 21:23 ` Bill Huey 1 sibling, 1 reply; 1392+ messages in thread From: Bill Huey @ 2004-08-31 20:52 UTC (permalink / raw) To: Diego Calleja; +Cc: Jeff Garzik, bhuey, torvalds, tmv, linux-kernel On Tue, Aug 31, 2004 at 03:56:13PM +0200, Diego Calleja wrote: > El Tue, 31 Aug 2004 01:56:45 -0400 Jeff Garzik <jgarzik@pobox.com> escribi?: > > messaging passing is also known as "really slow C function calls" > > > > My PCI bus passes messages all the time. A message is in the eye of the > > beholder. > > There're some numbers for that: > > http://lists.freebsd.org/pipermail/freebsd-hackers/2003-August/002426.html As you can see the numbers are very fast for a general purpose system like that. Add that with their XIO framework for data propagation and you have a really amazingly fast system that combines both aggressive design and optimized low level facilities for both remote and local commuication. Matt's one of the very few open source engineers on this planet that can do both well. bill ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4) 2004-08-31 20:52 ` Bill Huey @ 2004-09-02 17:20 ` Diego Calleja 2004-09-02 21:37 ` Bill Huey 0 siblings, 1 reply; 1392+ messages in thread From: Diego Calleja @ 2004-09-02 17:20 UTC (permalink / raw) To: Bill Huey; +Cc: jgarzik, bhuey, torvalds, tmv, linux-kernel El Tue, 31 Aug 2004 13:52:11 -0700 Bill Huey (hui) <bhuey@lnxw.com> escribió: > As you can see the numbers are very fast for a general purpose system > like that. Add that with their XIO framework for data propagation and For the syscall case, I remember that they would be able to "batch" syscalls, so this framework could have even better performance in some cases. The good thing is that they didn't do it just for the sake of doing it (like microkernel people) but to achieve their goals (SSI, etc) ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4) 2004-09-02 17:20 ` Diego Calleja @ 2004-09-02 21:37 ` Bill Huey 0 siblings, 0 replies; 1392+ messages in thread From: Bill Huey @ 2004-09-02 21:37 UTC (permalink / raw) To: Diego Calleja; +Cc: Bill Huey, jgarzik, torvalds, tmv, linux-kernel On Thu, Sep 02, 2004 at 07:20:58PM +0200, Diego Calleja wrote: > El Tue, 31 Aug 2004 13:52:11 -0700 Bill Huey (hui) <bhuey@lnxw.com> escribi?: > > > As you can see the numbers are very fast for a general purpose system > > like that. Add that with their XIO framework for data propagation and > > For the syscall case, I remember that they would be able to "batch" syscalls, > so this framework could have even better performance in some cases. > The good thing is that they didn't do it just for the sake of doing it > (like microkernel people) but to achieve their goals (SSI, etc) For what they're doing, they have minimize the overhead cross boundaries as much as possible for usage in both user and kernel spaces. If dfBSD achieves what they intend to do, they'll have on of the best SSI systems on the planet and it'll be able to use the VFS layer in ways that other systems can't even touch with iSCSI drivers, etc... They've got process suspend and restore already in order to support serialization across machines. Having it VFS driven, VM objects, etc... makes the concept of migration a function of the VFS layer. It's quite clever and I'm sure that it's driven by Matt's experience with distributed databases, specifically Backplane. All locking in his system is CPU local, no mutexes of any sort, so migration as a function of messaging, scheduling and other things are done in per-CPU chunks. It's quite clever make handling in per-process resources chunks. http://twiki.im.ufba.br/pub/MAT154/WebHome/duality78.pdf Talks about part of this and how "procedural" is isomorphic to "message-passing" concurrency system. One can be transformed to another. It's a project that's move much faster in the area of SMP than FreeBSD 5.x (Solaris style SMP) and really the only BSD project on this planet that has the talent and social chemistry to make it happen. bill ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4) [not found] ` <20040831155613.2b25df1e.diegocg@teleline.es> 2004-08-31 20:52 ` Bill Huey @ 2004-08-31 21:23 ` Bill Huey 1 sibling, 0 replies; 1392+ messages in thread From: Bill Huey @ 2004-08-31 21:23 UTC (permalink / raw) To: Diego Calleja; +Cc: Jeff Garzik, bhuey, torvalds, tmv, linux-kernel On Tue, Aug 31, 2004 at 03:56:13PM +0200, Diego Calleja wrote: > There're some numbers for that: > > http://lists.freebsd.org/pipermail/freebsd-hackers/2003-August/002426.html Link about XIO and things hijacked from FreeBSD-current. http://leaf.dragonflybsd.org/mailarchive/kernel/2004-03/msg00469.html bill ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4) 2004-08-31 5:30 ` Userspace file systems & MKs (Re: silent semantic changes with reiser4) Bill Huey 2004-08-31 5:56 ` Jeff Garzik @ 2004-08-31 9:01 ` Tonnerre 2004-08-31 11:15 ` Userspace file systems Luca Ferroni 2004-08-31 20:53 ` Userspace file systems & MKs (Re: silent semantic changes with reiser4) Bill Huey 1 sibling, 2 replies; 1392+ messages in thread From: Tonnerre @ 2004-08-31 9:01 UTC (permalink / raw) To: Bill Huey; +Cc: Jeff Garzik, Linus Torvalds, Tom Vier, linux-kernel [-- Attachment #1: Type: text/plain, Size: 841 bytes --] Salut, On Mon, Aug 30, 2004 at 10:30:55PM -0700, Bill Huey wrote: > DragonFly BSD, the only remotely functional open source BSD project on this > planet, has plans in place to push certain kernel components like their VFS > layer into userspace for easier debugging, testing and other things that go > with developing file systems easily. If they back it with something like C++ > for doing constructor style type conversion on top of overloaded operators > to back VFS data structure translation, could easily import stuff like most > Linux file systems without major restructuring, say, if they had their > translation library written. In this case, userspace kernel systems have > some serious programming advantages over traditional kernels. Uh, what about enhancing lufs code in this regard and porting all fs'es to it? Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: Userspace file systems 2004-08-31 9:01 ` Tonnerre @ 2004-08-31 11:15 ` Luca Ferroni 2004-08-31 20:53 ` Userspace file systems & MKs (Re: silent semantic changes with reiser4) Bill Huey 1 sibling, 0 replies; 1392+ messages in thread From: Luca Ferroni @ 2004-08-31 11:15 UTC (permalink / raw) To: Tonnerre; +Cc: linux-kernel, miklos Il Tue, 31 Aug 2004 11:01:58 +0200, Tonnerre <tonnerre@thundrix.ch> ha scritto: > Uh, what about enhancing lufs code in this regard and porting all fs'es to it? > I think that porting all file systems to LUFS is a mad idea :), but providing a facility in the kernel for user-space file system implementations could be great. I am working on PackageFS (http://packagefs.sourceforge.net) which is an user-space file system to manage packages: it relies on LUFS, but I plan to port it soon to FUSE ( http://sourceforge.net/projects/avf) that, at present, is maintained by Miklos Szeredi and works with latest kernels. PackageFS relies also on existent package managers, it will be transparent to all package formats. I. e.: you can install a package by copying, and see package informations by a simple cat. Now it works only in Debian distro, and it is far from complete (it is 0.09 version :) ). So, what do you think about including a kernel module to make a user able to develop, install and use an user-space file system ? Maybe Miklos Szeredi will write more details about FUSE which is the interesting part for kernel inclusion. Bye Luca -- ----------------------------------------------- They'll never stole us our.... FREEDOM!!! Luca Ferroni ICQ #317977679 www.cs.unibo.it/~fferroni/ ----------------------------------------------- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4) 2004-08-31 9:01 ` Tonnerre 2004-08-31 11:15 ` Userspace file systems Luca Ferroni @ 2004-08-31 20:53 ` Bill Huey 1 sibling, 0 replies; 1392+ messages in thread From: Bill Huey @ 2004-08-31 20:53 UTC (permalink / raw) To: Tonnerre; +Cc: Bill Huey, Jeff Garzik, Linus Torvalds, Tom Vier, linux-kernel On Tue, Aug 31, 2004 at 11:01:58AM +0200, Tonnerre wrote: > Salut, > > On Mon, Aug 30, 2004 at 10:30:55PM -0700, Bill Huey wrote: ... > > translation library written. In this case, userspace kernel systems have > > some serious programming advantages over traditional kernels. > > Uh, what about enhancing lufs code in this regard and porting all fs'es to it? I wasn't aware that your project has gotten this far. I'll tell Hiten about this to see if some of this can be used in their project. Thanks bill ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 4:38 ` Jeff Garzik 2004-08-31 5:30 ` Userspace file systems & MKs (Re: silent semantic changes with reiser4) Bill Huey @ 2004-08-31 19:01 ` Ryan Breen 2004-08-31 19:12 ` Jeff Garzik 2004-09-01 15:37 ` Clay Haapala 1 sibling, 2 replies; 1392+ messages in thread From: Ryan Breen @ 2004-08-31 19:01 UTC (permalink / raw) To: Jeff Garzik, linux-kernel On Tue, 31 Aug 2004 00:38:14 -0400, Jeff Garzik <jgarzik@pobox.com> wrote: > > Man, this thread came a long way. > You said it -- from Reiser to microkernel. If we can only figure out a way to get a BitKeeper discussion going, we'll have the Grand Unified Flamewar. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 19:01 ` silent semantic changes with reiser4 Ryan Breen @ 2004-08-31 19:12 ` Jeff Garzik 2004-09-01 15:37 ` Clay Haapala 1 sibling, 0 replies; 1392+ messages in thread From: Jeff Garzik @ 2004-08-31 19:12 UTC (permalink / raw) To: Ryan Breen; +Cc: linux-kernel Ryan Breen wrote: > On Tue, 31 Aug 2004 00:38:14 -0400, Jeff Garzik <jgarzik@pobox.com> wrote: > >>Man, this thread came a long way. >> > > > You said it -- from Reiser to microkernel. If we can only figure out > a way to get a BitKeeper discussion going, we'll have the Grand > Unified Flamewar. If you can find a link between the reiserfs thread and the webcam driver thread, you've gone all the way from filesystems to Nazis, thus invoking Godwin's Law ;-) Jeff [1] http://en.wikipedia.org/wiki/Godwin's_law ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 19:01 ` silent semantic changes with reiser4 Ryan Breen 2004-08-31 19:12 ` Jeff Garzik @ 2004-09-01 15:37 ` Clay Haapala 2004-09-03 12:50 ` Yury Umanets 1 sibling, 1 reply; 1392+ messages in thread From: Clay Haapala @ 2004-09-01 15:37 UTC (permalink / raw) To: Ryan Breen; +Cc: Jeff Garzik, linux-kernel On Tue, 31 Aug 2004, Ryan Breen verbalised: > On Tue, 31 Aug 2004 00:38:14 -0400, Jeff Garzik <jgarzik@pobox.com> wrote: >> >> Man, this thread came a long way. >> > > You said it -- from Reiser to microkernel. If we can only figure out > a way to get a BitKeeper discussion going, we'll have the Grand > Unified Flamewar. OK, I'll byte: consider file versioning of an SCM implemented as a ReiserFS 4 plug-in, as in: diff -u ultrastore.c/delta/3.2.3 ultrastore.c/delta/3.2.4 (Brrr! Just had a VMS flash-back shiver.) -- Clay Haapala (chaapala@cisco.com) Cisco Systems Storage +1 763-398-1056 6450 Wedgwood Rd, Suite 130 Maple Grove MN 55311 PGP: C89240AD The mysterious blob seen in satellite photos off the east coast of Florida during the passing of Hurricane Charley has now been positively identified as a large cloud of Democrat absentee-ballot applications and voter lists. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 15:37 ` Clay Haapala @ 2004-09-03 12:50 ` Yury Umanets 0 siblings, 0 replies; 1392+ messages in thread From: Yury Umanets @ 2004-09-03 12:50 UTC (permalink / raw) To: linux-kernel Clay Haapala wrote: >On Tue, 31 Aug 2004, Ryan Breen verbalised: > > >>On Tue, 31 Aug 2004 00:38:14 -0400, Jeff Garzik <jgarzik@pobox.com> wrote: >> >> >>>Man, this thread came a long way. >>> >>> >>> >>You said it -- from Reiser to microkernel. If we can only figure out >>a way to get a BitKeeper discussion going, we'll have the Grand >>Unified Flamewar. >> >> > >OK, I'll byte: consider file versioning of an SCM implemented as a >ReiserFS 4 plug-in, as in: > >diff -u ultrastore.c/delta/3.2.3 ultrastore.c/delta/3.2.4 > >(Brrr! Just had a VMS flash-back shiver.) > > hehe ;) Seems to develop this extention NAMESYS should stop use BitKeeper. Or should buy it. -- umka ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 4:08 ` Linus Torvalds 2004-08-31 4:38 ` Jeff Garzik @ 2004-08-31 10:57 ` Alan Cox 2004-08-31 17:15 ` Linus Torvalds 2004-08-31 19:45 ` Toon van der Pas 2004-09-02 22:35 ` Tom Vier 3 siblings, 1 reply; 1392+ messages in thread From: Alan Cox @ 2004-08-31 10:57 UTC (permalink / raw) To: Linus Torvalds; +Cc: Tom Vier, Linux Kernel Mailing List On Maw, 2004-08-31 at 05:08, Linus Torvalds wrote: > > What about microkernels? They do tcp in userspace. > > No they don't. They do TCP in a separate address space from user space, > that just also happens to be separate from the "microkernel address > space". Several do TCP in user space. The only thing you need in kernel for TCP/IP is enough decode to decide who gets the packet. Even some non microkernel embedded OS's do this in order to keep kernel size down. Alan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 10:57 ` Alan Cox @ 2004-08-31 17:15 ` Linus Torvalds 2004-08-31 21:13 ` Frank van Maarseveen 0 siblings, 1 reply; 1392+ messages in thread From: Linus Torvalds @ 2004-08-31 17:15 UTC (permalink / raw) To: Alan Cox; +Cc: Tom Vier, Linux Kernel Mailing List On Tue, 31 Aug 2004, Alan Cox wrote: > > Several do TCP in user space. The only thing you need in kernel for > TCP/IP is enough decode to decide who gets the packet. Only thing? I don't think so. You also want to make sure that regular users cannot send "impossible" packets. Think about the old "ping of death" kind of thing, where a normal mis-behaving (and I'm not saying intentionally so: it might be a small bug that just overwrites some data) program should _not_ be able to cause problems on the network. Admins absolutely _hate_ that. They will ban an OS if it sends out packets that cause troublem. You should remember that - we used to do strange things on the net (long long time ago), and we brought down servers by mistake, and nobody ever considered it a server bug: it was a Linux bug that it wouldn't do the right thing. Things like not sending FIN-packets when a program suddenly goes away is NOT acceptable behaviour! Neither is it acceptable behaviour to allow user programs to make up their own packets. > Even some non microkernel embedded OS's do this in order to keep kernel > size down. ..and I'm not disagreeing that it doesn't happen. I explicitly mentioned PalmOS, I bet it happens in other cases too. But I'd strongly argue that it's a bug, not a feature. It's a bug that people tend to accept in a "single-client" environment. NOTE! This is totally ignoring the fact that you can't be called "UNIX" any more. You _need_ to have sequence numbers etc be shared between multiple programs that all write to the stream. Again, that _does_ mean that you have another protection domain (aka "kernel" or "TCP deamon") that keeps track of the sequence number. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 17:15 ` Linus Torvalds @ 2004-08-31 21:13 ` Frank van Maarseveen 2004-08-31 21:25 ` Linus Torvalds 0 siblings, 1 reply; 1392+ messages in thread From: Frank van Maarseveen @ 2004-08-31 21:13 UTC (permalink / raw) To: Linus Torvalds; +Cc: Alan Cox, Tom Vier, Linux Kernel Mailing List On Tue, Aug 31, 2004 at 10:15:25AM -0700, Linus Torvalds wrote: > > Admins absolutely _hate_ that. They will ban an OS if it sends out packets > that cause troublem. You should remember that - we used to do strange > things on the net (long long time ago), and we brought down servers by > mistake, and nobody ever considered it a server bug: it was a Linux bug > that it wouldn't do the right thing. > > Things like not sending FIN-packets when a program suddenly goes away is > NOT acceptable behaviour! Neither is it acceptable behaviour to allow user > programs to make up their own packets. The user/kernel distinction is not always (heck, maybe almost never) present in the embedded world (a large world). The notion of a "regular user" does not apply at all in such a case. This is not a bug but merely the state of technology. It will slowly go away I think because embedded software becomes more complex and hardware becomes cheaper. To implement multiple TCP clients _and_ the TCP/IP stack in one space is perfectly possible and it's actually done in practice. It has advantages (speed, when done properly) and disadvantages (complexity/bug-prone). > > NOTE! This is totally ignoring the fact that you can't be called "UNIX" > any more. You _need_ to have sequence numbers etc be shared between > multiple programs that all write to the stream. Again, that _does_ mean > that you have another protection domain (aka "kernel" or "TCP deamon") > that keeps track of the sequence number. There is nothing in the networking or UNIX standards that prescibe another protection domain for this. Would be insane to leave that out in a hosted environment but it _can_ be done without. -- Frank ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 21:13 ` Frank van Maarseveen @ 2004-08-31 21:25 ` Linus Torvalds 0 siblings, 0 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-08-31 21:25 UTC (permalink / raw) To: Frank van Maarseveen; +Cc: Alan Cox, Tom Vier, Linux Kernel Mailing List On Tue, 31 Aug 2004, Frank van Maarseveen wrote: > > There is nothing in the networking or UNIX standards that prescibe another > protection domain for this. Would be insane to leave that out in a hosted > environment but it _can_ be done without. My point is that TCP _does_ have a lot of state that needs to be handled in a safe manner by a proper operating system. The fact that there are OS's out there that are crap doesn't change that matter. There are lots of embedded OS's out there that still do multitasking in a purely cooperative way. I don't think it's a valid model for anything but toys. Same goes for putting TCP in user space. It's doable, but it's not an "OS". It's a program loader. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 4:08 ` Linus Torvalds 2004-08-31 4:38 ` Jeff Garzik 2004-08-31 10:57 ` Alan Cox @ 2004-08-31 19:45 ` Toon van der Pas 2004-09-02 22:35 ` Tom Vier 3 siblings, 0 replies; 1392+ messages in thread From: Toon van der Pas @ 2004-08-31 19:45 UTC (permalink / raw) To: linux-kernel On Mon, Aug 30, 2004 at 09:08:07PM -0700, Linus Torvalds wrote: > > > On Mon, 30 Aug 2004, Tom Vier wrote: > > > On Thu, Aug 26, 2004 at 09:48:04AM -0700, Linus Torvalds wrote: > > > - safely synchronize globally visible data structures > > > That's quite fundamental. 99% of what a kernel does is exactly > > > that. TCP would be in user space too, if it wasn't for > > > _exactly_ this issue. A lot > > > > What about microkernels? They do tcp in userspace. > > No they don't. They do TCP in a separate address space from user > space, that just also happens to be separate from the "microkernel > address space". The HP NonStop Server (formerly called Tandem) has an MPP architecture and a message passing OS (the NonStop Kernel). On this OS the TCP/IP stack is implemented as a library against which the application software is linked. The ethernet controllers/drivers maintain a routing table for routing incoming traffic to the right process. So the TCP/IP stack is implemented in userspace on the NonStop Server, I would say. Regards, Toon. -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 4:08 ` Linus Torvalds ` (2 preceding siblings ...) 2004-08-31 19:45 ` Toon van der Pas @ 2004-09-02 22:35 ` Tom Vier 2004-09-02 22:53 ` Linus Torvalds 3 siblings, 1 reply; 1392+ messages in thread From: Tom Vier @ 2004-09-02 22:35 UTC (permalink / raw) To: Linus Torvalds; +Cc: linux-kernel On Mon, Aug 30, 2004 at 09:08:07PM -0700, Linus Torvalds wrote: > > What about microkernels? They do tcp in userspace. > > No they don't. They do TCP in a separate address space from user space, > that just also happens to be separate from the "microkernel address > space". Well, I'd call that userspace (if it has a pid and it's own addr space, like every other proc). > So a microkernel will have _more_ address spaces, and they won't be "user > space". They'll be "server deamon space" or something. Now, that's also > why they tend to have performance problems - because you need to copy the > data between different address spaces, and switch the CPU context etc > around. Just out of curiousity, what do you think of L4? I don't remember the numbers, but it wasn't much slower than linux (iirc), even on x86. I think k42 has msg passing close to the speed of a syscall. (Bulk data could use shared mem, maybe.) Anyway, it would be neat to have a tcp/ip daemon. You could run experimental code w/o fear of it clobbering other things. Somethings, of course, a microkernel can't help (if the rootfs's fs daemon dies, you're screwed). A little bit more containment from bad kernel code would be nice. (BTW, for those who don't know, mach optimized away msg passing and used a single addr space, once code was "trusted". You could choose speed over protection. That wasn't a hard choice since Mach msg passing sucks. Mach gave microkernels a bad name.) > Not user space. They may be "ring 3" from a CPU standpoint, but they > aren't user space from a _user_ standpoint - it's still very much a > separate address space, with domain protection. How are they different from regular user procs, other then being trusted to manage certain resources? > In short: you _need_ to have a separate address space (either kernel, or > "TCP server" or whatever) if you want to have reliable, secure and > generally usable TCP. Exokernels are another topic. 8) > > As long as a trusted process keeps data such as free ports, what's the > > problem? > > None - because it's not user space any more. Yes it is, it's a user process. > Well, performance might still suck, of course. And it does. -- Tom Vier <tmv@comcast.net> DSA Key ID 0x15741ECE ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 22:35 ` Tom Vier @ 2004-09-02 22:53 ` Linus Torvalds 0 siblings, 0 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-09-02 22:53 UTC (permalink / raw) To: Tom Vier; +Cc: linux-kernel On Thu, 2 Sep 2004, Tom Vier wrote: > > > Not user space. They may be "ring 3" from a CPU standpoint, but they > > aren't user space from a _user_ standpoint - it's still very much a > > separate address space, with domain protection. > > How are they different from regular user procs, other then being trusted to > manage certain resources? Ehh, they are separate the same way "inetd" is separate. It's not a _user_ proc, it's a system proc. The user can't actually do anything about it. In many ways UNIX _is_ a microkernel. It does nonessential stuff in "user space". Anything that is critical for performance or the working of the machine is in kernel space. The big difference between UNIX and what people _call_ "microkernels" is that UNIX has a very functional and sane partitioning of what is a critical thing. But from a kernel _protection_ angle, the only part that is important is that the services be in some protected domain. That was what started this discussion: 99% of what the kernel does is protecting shared data. Whether it does so by passing it on to some trusted third party or not is an implementation issue, and is totally pointless from a user standpoint, since the user won't see it anyway. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:48 ` Linus Torvalds 2004-08-26 18:07 ` Jan Harkes 2004-08-31 3:39 ` Tom Vier @ 2004-09-10 19:22 ` Timothy Miller 2004-09-10 22:18 ` Elladan 2004-09-12 20:02 ` Pavel Machek 2 siblings, 2 replies; 1392+ messages in thread From: Timothy Miller @ 2004-09-10 19:22 UTC (permalink / raw) To: Linus Torvalds Cc: Jan Harkes, Markus Törnqvist, Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Linus Torvalds wrote: > > The same goes for something like a "container file". Whether you see it as > "dir-as-file" or "file-as-dir" (and I agree with Jan that the two are > totally equivalent), the point of having the capability in the kernel is > not that the operations cannot be done in user space - the point is that > they cannot be done in user space _safely_. The kernel is kind of the > thing that guarantees that everybody follows the rules. > Well, it CAN be done safely if every client has to go through the kernel which does all the appropriate locks/semaphors and then passes the request down to the daemon. (Isn't NFS implemented this way?) This is very micro-kernel-ish, but it's a reasonable idea to do it this way in cases where things are non-critical. Say there's a way to cd into a tgz file to look around. If the access methods through the kernel get routed back to a user-space process (which probably does some amount of caching in memory and on disk of uncompressed bits of the archive), it could be a bit slower than if it were all in-kernel. The thing is that the processing time in the daemon is probably quite high compared to the overhead of the context switches, so it doesn't cost much. (And if it CAN be done in userspace without being horribly convoluted or unsafe, then it SHOULD be done in userspace.) Besides, even if it were a LOT slower to access a tgz file without extracting it first, I would STILL think it was wonderful AND use it a LOT. Heh... now, the next thing is to be able to do some sort of union-mount like thing where the tgz is really read-only but if you WRITE to it, it gets stored somewhere else. This way, you can compile a kernel tree without extracting it first. Taking this a bit further, the union-mount could be done away with if the daemon were to lazily rebuild the tgz file from its cache when the tgz is modified. That is, writes have little or no penalty because they just get stored in the daemon's cache, and the daemon rebuilds the archive in the background. The VFS layer would have to recognize that a particular file has been accessed in such a way that it must go through the daemon for ANY access to it, which means that if someone tries to access the original archive as a file, requests have to go through the daemon. This makes it coherent, although with an expected performance penalty. One issue to consider is that of shutdowns, both intentional and crashes. This is a non-issue with an underlying journaling file system. As long as the tgz daemon comes up early enough, it would see its cache (which is on-disk anyway), recognize which archive it belongs to, and continue where it left off. This way, the daemon doesn't have to go through a long process of rebuilding every archive you've ever modified on the disk on shut-down. If, however, you needed to access the archive without the daemon, that could be a problem (but what you'd get is probably the archive before it was modified). Also, some sort of sync command would be needed to cause the daemon to rebuild all archives immediately. I really dig the idea of being able to access archives without extracting them first, regardless of any performance penalty. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-10 19:22 ` Timothy Miller @ 2004-09-10 22:18 ` Elladan 2004-09-12 20:11 ` Pavel Machek 2004-09-12 20:02 ` Pavel Machek 1 sibling, 1 reply; 1392+ messages in thread From: Elladan @ 2004-09-10 22:18 UTC (permalink / raw) To: Timothy Miller; +Cc: linux-fsdevel, linux-kernel On Fri, Sep 10, 2004 at 03:22:59PM -0400, Timothy Miller wrote: > > Linus Torvalds wrote: > > >The same goes for something like a "container file". Whether you see it as > >"dir-as-file" or "file-as-dir" (and I agree with Jan that the two are > >totally equivalent), the point of having the capability in the kernel is > >not that the operations cannot be done in user space - the point is that > >they cannot be done in user space _safely_. The kernel is kind of the > >thing that guarantees that everybody follows the rules. > > Well, it CAN be done safely if every client has to go through the kernel > which does all the appropriate locks/semaphors and then passes the > request down to the daemon. (Isn't NFS implemented this way?) > > This is very micro-kernel-ish, but it's a reasonable idea to do it this > way in cases where things are non-critical. > > Say there's a way to cd into a tgz file to look around. If the access > methods through the kernel get routed back to a user-space process > (which probably does some amount of caching in memory and on disk of > uncompressed bits of the archive), it could be a bit slower than if it > were all in-kernel. The thing is that the processing time in the daemon > is probably quite high compared to the overhead of the context switches, > so it doesn't cost much. (And if it CAN be done in userspace without > being horribly convoluted or unsafe, then it SHOULD be done in > userspace.) Besides, even if it were a LOT slower to access a tgz file > without extracting it first, I would STILL think it was wonderful AND > use it a LOT. This is only safe for some limited definitions of safety. Leaving aside the obvious complexities of making sure the user space daemon doesn't just do something crazy, you have a number of problems: * Which user does the daemon run as? If it runs as root, it needs to enforce strict security requirements in terms of VFS operations coming in, and also, it has all the problems of a SUID application running on an arbitrary user file. I don't know about you, but I don't trust joebob's tarfsd to be suid when running on some script kiddie tarball. If it runs as the user making the request, then what happens when two users try to open the same file at the same time. Do they see two different views of the file? Is one of them locked out? How can they possibly synchronize if the views are writable? * If the daemon runs as a user, then what happens if you try to run a suid program inside of the view? What if root tries to walk into a tar file? The conservative view is that the kernel would need to always return ENOENT for any process with elevated capabilities, but even this is not safe. * Consider what happens if foo.tar/blah_blah is automatically bound to enter a tarfsd view of foo.tar. What happens if I point the web server at foo.tar/blah? The web server runs as httpd or something, so presumably httpd ends up running some sort of tarfsd view on the file. But if the tarball was made by a malicious person, presumably it can obtain httpd user access now by exploiting a bug in tarfsd. * Even if you assume the view is read-only and the kernel coerces all permissions and ownership and such, there's the possibility of tarfsd presenting unexpected syscall results - weird error codes, short reads, file data changing underneath mmap, etc. that user applications don't expect and may be susceptible to. * Besides all these security issues, if this scheme is writable it has all the resource problems that loopback network filesystems suffer. What if the kernel is short on memory and tries to flush dirty buffers to reclaim it. If those buffers are running against the user FS daemon, then that daemon must wake up to clear dirty buffers. If it tries to allocate memory, deadlock in the kernel. Probably the security problems can be solved to some degree by being very paranoid in the kernel (at an associated loss in utility), and the resource issues can be solved by restricting dirty buffers for loopback mounts (at an associated loss in performance), but it's hardly simple. -J ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-10 22:18 ` Elladan @ 2004-09-12 20:11 ` Pavel Machek 0 siblings, 0 replies; 1392+ messages in thread From: Pavel Machek @ 2004-09-12 20:11 UTC (permalink / raw) To: Elladan; +Cc: Timothy Miller, linux-fsdevel, linux-kernel Hi! I'm talking about uservfs.sf.net, aka podfuk, which implements tarfs (and more). > Leaving aside the obvious complexities of making sure the user space > daemon doesn't just do something crazy, you have a number of problems: > > * Which user does the daemon run as? If it runs as root, it needs to > enforce strict security requirements in terms of VFS operations coming > in, and also, it has all the problems of a SUID application running on > an arbitrary user file. I don't know about you, but I don't trust > joebob's tarfsd to be suid when running on some script kiddie tarball. It needs to run as root. If there's bug in tar, that script kiddie is going to get your users anyway. Audit the code. > * If the daemon runs as a user, then what happens if you try to run a It does not. > * Consider what happens if foo.tar/blah_blah is automatically bound to > enter a tarfsd view of foo.tar. What happens if I point the web > server at foo.tar/blah? The web server runs as httpd or something, so > presumably httpd ends up running some sort of tarfsd view on the > file. But if the tarball was made by a malicious person, presumably > it can obtain httpd user access now by exploiting a bug in tarfsd. If there's a bug in tar, I can do a lot of interesting things. Audit tar. > * Even if you assume the view is read-only and the kernel coerces all > permissions and ownership and such, there's the possibility of tarfsd > presenting unexpected syscall results - weird error codes, short > reads, file data changing underneath mmap, etc. that user applications > don't expect and may be susceptible to. FUD. Fix tarfsd if its problem. > * Besides all these security issues, if this scheme is writable it has There are no security issues. Yes, uservfs is security-sensitive code. > all the resource problems that loopback network filesystems suffer. > What if the kernel is short on memory and tries to flush dirty buffers > to reclaim it. If those buffers are running against the user FS > daemon, then that daemon must wake up to clear dirty buffers. If it > tries to allocate memory, deadlock in the kernel. Coda solves this: all writable things are file-backed on file in /tmp => bad performance if you only need piece of very large file, but no deadlocks. > Probably the security problems can be solved to some degree by being > very paranoid in the kernel (at an associated loss in utility), and the > resource issues can be solved by restricting dirty buffers for loopback > mounts (at an associated loss in performance), but it's hardly simple. See sources. It is <1000 lines of code. -- 64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-10 19:22 ` Timothy Miller 2004-09-10 22:18 ` Elladan @ 2004-09-12 20:02 ` Pavel Machek 1 sibling, 0 replies; 1392+ messages in thread From: Pavel Machek @ 2004-09-12 20:02 UTC (permalink / raw) To: Timothy Miller Cc: Linus Torvalds, Jan Harkes, Markus Törnqvist, Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Hi! > Say there's a way to cd into a tgz file to look around. If the > access methods through the kernel get routed back to a user-space > process (which probably does some amount of caching in memory and on > disk of uncompressed bits of the archive), it could be a bit slower > than if it were all in-kernel. The thing is that the processing time Exactly. See uservfs.sf.net. It is using coda hooks into the kernel. > be done in userspace.) Besides, even if it were a LOT slower to > access a tgz file without extracting it first, I would STILL think it > was wonderful AND use it a LOT. Go ahead :-). -- 64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 7:53 ` Markus Törnqvist 2004-08-26 16:32 ` Jan Harkes @ 2004-09-03 4:43 ` Grzegorz Jaśkiewicz 2004-09-03 5:05 ` viro 2004-09-05 11:10 ` Tonnerre 1 sibling, 2 replies; 1392+ messages in thread From: Grzegorz Jaśkiewicz @ 2004-09-03 4:43 UTC (permalink / raw) To: Markus Törnqvist Cc: Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Markus Törnqvist wrote: >On Thu, Aug 26, 2004 at 12:32:00AM -0500, Matt Mackall wrote: > > >>Find some silly person with an iBook and open a shell on OS X. Use cp >>to copy a file with a resource fork. Oh look, the Finder has no idea >>what the new file is, even though it looks exactly identical in the >>shell. Isn't that _wonderful_? Now try cat < a > b on a file with a >>fork. How is that ever going to work? >> >> > >Then I guess OS X ships a broken implementation of cp, yes? > > > Nope, GUI handles it perfectly. it's maybe 0.1% of users of MacOS that acctually care about cp being broken. -- GJ ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-03 4:43 ` Grzegorz Jaśkiewicz @ 2004-09-03 5:05 ` viro 2004-09-03 5:40 ` Grzegorz Jaśkiewicz 2004-09-05 11:10 ` Tonnerre 1 sibling, 1 reply; 1392+ messages in thread From: viro @ 2004-09-03 5:05 UTC (permalink / raw) To: Grzegorz Ja??kiewicz Cc: Markus Törnqvist, Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, Sep 03, 2004 at 06:43:21AM +0200, Grzegorz Ja??kiewicz wrote: > >Then I guess OS X ships a broken implementation of cp, yes? > > > Nope, GUI handles it perfectly. it's maybe 0.1% of users of MacOS that > acctually care about cp being broken. Gotta love the Mac logics: "Is $FOO broken?" - "Nope, $BAR works. Very few care about $FOO being broken". ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-03 5:05 ` viro @ 2004-09-03 5:40 ` Grzegorz Jaśkiewicz 0 siblings, 0 replies; 1392+ messages in thread From: Grzegorz Jaśkiewicz @ 2004-09-03 5:40 UTC (permalink / raw) To: viro Cc: Markus Törnqvist, Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list viro@parcelfarce.linux.theplanet.co.uk wrote: >On Fri, Sep 03, 2004 at 06:43:21AM +0200, Grzegorz Ja??kiewicz wrote: > > >>>Then I guess OS X ships a broken implementation of cp, yes? >>> >>> >>> >>Nope, GUI handles it perfectly. it's maybe 0.1% of users of MacOS that >>acctually care about cp being broken. >> >> > >Gotta love the Mac logics: "Is $FOO broken?" - "Nope, $BAR works. Very few >care about $FOO being broken". > > I don't represent Mac logic ;) IMO it's wrong, but it's still a fact. -- GJ ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-03 4:43 ` Grzegorz Jaśkiewicz 2004-09-03 5:05 ` viro @ 2004-09-05 11:10 ` Tonnerre 1 sibling, 0 replies; 1392+ messages in thread From: Tonnerre @ 2004-09-05 11:10 UTC (permalink / raw) To: Grzegorz Ja??kiewicz Cc: Markus T??rnqvist, Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1109 bytes --] Salut, On Fri, Sep 03, 2004 at 06:43:21AM +0200, Grzegorz Ja??kiewicz wrote: > >Then I guess OS X ships a broken implementation of cp, yes? > > Nope, GUI handles it perfectly. it's maybe 0.1% of users of MacOS that > acctually care about cp being broken. Actually, there is a (non-GNU) free implementation of the fileutils on MacOS/X that treats and copies the .DS_Store entries (or say is metadata aware). And even though I might agree on that the principle of having per-directory database files is somewhat hacky, it appears to work like a charm (As long as userland programs do care about it). On Linux we have a solution that actually works just the same and is a lot cooler. It's called POSIX extended attributes... Tonnerre PS. If you want to talk further on the subject of how MacOS/X does it and how I would do it using extended attributes, discuss it in private mail with me, since MacOS/X isn't really Linux. And no, I don't want a single daemon do that, but I want it to be a standard implementation that libc and fileutils support. [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 5:32 ` Matt Mackall 2004-08-26 7:34 ` Denis Vlasenko 2004-08-26 7:53 ` Markus Törnqvist @ 2004-08-26 8:31 ` Helge Hafting 2004-08-26 18:17 ` Horst von Brand 2004-09-02 23:05 ` Tom Vier 2004-08-26 9:59 ` Spam 2004-08-26 13:05 ` Joseph Fannin 4 siblings, 2 replies; 1392+ messages in thread From: Helge Hafting @ 2004-08-26 8:31 UTC (permalink / raw) To: Matt Mackall Cc: Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Matt Mackall wrote: [...] >What it breaks is the concept of a file. In ways that are ill-defined, >not portable, hard to work with, and needlessly complex. Along the >way, it breaks every single application that ever thought it knew what >a file was. > >Progress? No, this has been done before. Various dead operating >systems have done it or similar and regretted it. Most recently MacOS, >which jumped through major hurdles to begin purging themselves of >resource forks when they went to OS X. They're still there, but >heavily deprecated. > > > >>cp, grep, cat, and tar will continue to work just fine on files with >>multiple streams. >> >> > >Find some silly person with an iBook and open a shell on OS X. Use cp >to copy a file with a resource fork. Oh look, the Finder has no idea >what the new file is, even though it looks exactly identical in the >shell. Isn't that _wonderful_? > It is what I'd expect. Now, use cp -R to copy the file _with its directory_, and see if that fares better. If not - bad implementation of fs and/or cp. The way I see file-as -directory is that _file_ operations (like the reads issued by cat) only work on the file part. You want the directory part? Use directory operations such as those "cp -R" use. >Now try cat < a > b on a file with a >fork. How is that ever going to work? > > It is going to copy the _file_ part, of course. I wouldn't expect anything else - "cat" doesn't deal with directories. This also raise the question of when to use file-as-directory. A usage where you need everything to follow the file, even when using "cat" calls out for an application that puts everything into one file. Directory-as-file is the wrong tool for that job, so don't worry about such problems. Sticking thumbnails in a file-as-dir is another story though. Move the file (with "mv a b", not "cat a>b;rm a") moves the image file _and_ the thumbnail. No time wasted on regenerating thumbnails, no disk space or cleanup time wasted on dangling thumbs. Use the image file for its intended purposes (view it, mail it off, serve it on the web, edit it, print it) and the thumbnail doesn't get in the way because it isn't embedded in the file format. Embedding it in the file might work for jpeg which support generic embedded data, it surely won't work for every image format out there. This is my idea of how file-as-directory should work. Helge Hafting Helge Hafting ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:31 ` Helge Hafting @ 2004-08-26 18:17 ` Horst von Brand 2004-08-28 16:49 ` Helge Hafting 2004-09-02 23:05 ` Tom Vier 1 sibling, 1 reply; 1392+ messages in thread From: Horst von Brand @ 2004-08-26 18:17 UTC (permalink / raw) To: Helge Hafting Cc: Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Helge Hafting <helge.hafting@hist.no> said: > Matt Mackall wrote: [...] > >Find some silly person with an iBook and open a shell on OS X. Use cp > >to copy a file with a resource fork. Oh look, the Finder has no idea > >what the new file is, even though it looks exactly identical in the > >shell. Isn't that _wonderful_? > It is what I'd expect. Great. That means that all the tools of the trade stop working. Sounds like show-killer feature to me. > Now, use cp -R to copy the file > _with its directory_, Either it is a file or a directory. Make up your mind. If you have no clear distinction, you'll only get messed up. Badly. > and see if that fares better. If not - bad > implementation of fs and/or cp. The way I see file-as -directory > is that _file_ operations (like the reads issued by cat) only > work on the file part. You want the directory part? Use > directory operations such as those "cp -R" use. Excuse me, I must grab my sickness bag here. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:17 ` Horst von Brand @ 2004-08-28 16:49 ` Helge Hafting 0 siblings, 0 replies; 1392+ messages in thread From: Helge Hafting @ 2004-08-28 16:49 UTC (permalink / raw) To: Horst von Brand Cc: Helge Hafting, Matt Mackall, Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 02:17:03PM -0400, Horst von Brand wrote: > Helge Hafting <helge.hafting@hist.no> said: > > Matt Mackall wrote: > > [...] > > > >Find some silly person with an iBook and open a shell on OS X. Use cp > > >to copy a file with a resource fork. Oh look, the Finder has no idea > > >what the new file is, even though it looks exactly identical in the > > >shell. Isn't that _wonderful_? > > > It is what I'd expect. > > Great. That means that all the tools of the trade stop working. Sounds like > show-killer feature to me. > All wrong. Feel free to dislike the concept all you want, but don't provide false arguments. I am sure you can think up some _good_ ones, if you feel so strongly about this. _No_ tool "stop working" with such a fs. Of course the new feature (file-as-directory) may not be supported - which is a problem only for files that use the new feature. Stick a lot of ordinary files and directories in such a fs, and the tools works as before with no breakage. People obviously won't deploy new features before they are useable - the fs have to come first. Then the tools, and widespread use only after that. This is similiar to other extensions like ACLs which many tools don't understand. They're still useable though. Of course file-as-directory can have pilot uses like the often mentioned thumbnails - because it doesn't matter if those aren't backed up or get lost at random during the experimental phase. > > Now, use cp -R to copy the file > > _with its directory_, > > Either it is a file or a directory. Make up your mind. Wrong again. The file-as-directory concept means that a filename refers to something that is a file, or a directory, or both. That's the point of it. Being both isn't possible with current filesystems (except reiser4) of course, but file-as-directory is a proposed _extension_. > If you have no clear > distinction, you'll only get messed up. Badly. > The distinction is clear enough - three options instead of two. > Excuse me, I must grab my sickness bag here. Oh, very impressive. Please take a look at what we're talking about before attacking. Helge Hafting ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:31 ` Helge Hafting 2004-08-26 18:17 ` Horst von Brand @ 2004-09-02 23:05 ` Tom Vier 2004-09-03 8:07 ` Helge Hafting 1 sibling, 1 reply; 1392+ messages in thread From: Tom Vier @ 2004-09-02 23:05 UTC (permalink / raw) To: linux-kernel What's wrong with ~/.thumbcache or a daemon that manages system wide cache? I have no problem if a plugin wants to parse a file and split it inside the file-as-dir. However, multiple file streams are a cross-platform nighmare. Anyone who's used mac os knows this. -- Tom Vier <tmv@comcast.net> DSA Key ID 0x15741ECE ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 23:05 ` Tom Vier @ 2004-09-03 8:07 ` Helge Hafting 2004-09-03 8:19 ` Jan Dittmer 2004-09-05 11:13 ` Tonnerre 0 siblings, 2 replies; 1392+ messages in thread From: Helge Hafting @ 2004-09-03 8:07 UTC (permalink / raw) To: Tom Vier; +Cc: linux-kernel Tom Vier wrote: >What's wrong with ~/.thumbcache or a daemon that manages system wide cache? > > > Moving a file doen't move the associated thumbnail, and then you notice something is missing, or don't find the file, or have to wait for regeneration when the app notices a file without a tumb. That could take some time if you moved a directory full of postscript files, for example. Helge Hafting ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-03 8:07 ` Helge Hafting @ 2004-09-03 8:19 ` Jan Dittmer 2004-09-03 9:54 ` Erik Hensema 2004-09-05 11:13 ` Tonnerre 1 sibling, 1 reply; 1392+ messages in thread From: Jan Dittmer @ 2004-09-03 8:19 UTC (permalink / raw) To: Helge Hafting; +Cc: Tom Vier, linux-kernel Zitat von Helge Hafting <helge.hafting@hist.no>: > Tom Vier wrote: > > >What's wrong with ~/.thumbcache or a daemon that manages system wide > cache? > > > > > > > Moving a file doen't move the associated thumbnail, and then you > notice something is missing, or don't find the file, or have to wait > for regeneration when the app notices a file without a tumb. > That could take some time if you moved a directory full of postscript > files, for example. Use hash + filename in ~/.thumbcache and be smart when trying to find a thumbnail. That really can all be done in userspace. Jan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-03 8:19 ` Jan Dittmer @ 2004-09-03 9:54 ` Erik Hensema 0 siblings, 0 replies; 1392+ messages in thread From: Erik Hensema @ 2004-09-03 9:54 UTC (permalink / raw) To: linux-kernel Jan Dittmer (jdittmer@ppp0.net) wrote: > Zitat von Helge Hafting <helge.hafting@hist.no>: >> Tom Vier wrote: >> >> >What's wrong with ~/.thumbcache or a daemon that manages system wide >> cache? >> > >> > >> > >> Moving a file doen't move the associated thumbnail, and then you >> notice something is missing, or don't find the file, or have to wait >> for regeneration when the app notices a file without a tumb. >> That could take some time if you moved a directory full of postscript >> files, for example. > > Use hash + filename in ~/.thumbcache and be smart when trying to find a > thumbnail. That really can all be done in userspace. Yes, it can be done. It's slow, ineffecient and dangerous (rename a file, lose the thumbnail... so you'd have to do some garbage collection on the thumbnail cache). Thumbnails are actually an *excelent* example of a multistream file. Renaming/moving a file leaves the image<->thumbnail link intact. Removing the image removes the thumbnail. Copying the file to another multistream-file capable machine also copies the thumbnail, saving you the time to regenerate the thumbnail. When the target machine doesn't understand a multistream file, you simply lose the thumbnail, no harm done. -- Erik Hensema <erik@hensema.net> ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-03 8:07 ` Helge Hafting 2004-09-03 8:19 ` Jan Dittmer @ 2004-09-05 11:13 ` Tonnerre 2004-09-08 9:13 ` Helge Hafting 1 sibling, 1 reply; 1392+ messages in thread From: Tonnerre @ 2004-09-05 11:13 UTC (permalink / raw) To: Helge Hafting; +Cc: Tom Vier, linux-kernel [-- Attachment #1: Type: text/plain, Size: 467 bytes --] Salut, On Fri, Sep 03, 2004 at 10:07:00AM +0200, Helge Hafting wrote: > Moving a file doen't move the associated thumbnail, and then you > notice something is missing, or don't find the file, or have to wait > for regeneration when the app notices a file without a tumb. > That could take some time if you moved a directory full of postscript > files, for example. That's why the userland file utilities must be aware of the feature.. Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-05 11:13 ` Tonnerre @ 2004-09-08 9:13 ` Helge Hafting 0 siblings, 0 replies; 1392+ messages in thread From: Helge Hafting @ 2004-09-08 9:13 UTC (permalink / raw) To: Tonnerre; +Cc: Tom Vier, linux-kernel Tonnerre wrote: >Salut, > >On Fri, Sep 03, 2004 at 10:07:00AM +0200, Helge Hafting wrote: > > >>Moving a file doen't move the associated thumbnail, and then you >>notice something is missing, or don't find the file, or have to wait >>for regeneration when the app notices a file without a tumb. >>That could take some time if you moved a directory full of postscript >>files, for example. >> >> > >That's why the userland file utilities must be aware of the feature.. > > That won't ever happen. There are many ways of implementing thumbnails, and people don't want to slow down the operation of "mv" by having it search for _all_ the known thumbnails schemes. (Yes - several may be in use simultaneosly because the user uses several file managers!) The waste of time is even more interesting for the many directories that doesn't have thumbs at all. A file-as-dir implementation of thumbnails avoids this particular problem. There are other uses, so I hope people won't see file-as-dir as a mechanism for "file attributes" _only_. Helge Hafting ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 5:32 ` Matt Mackall ` (2 preceding siblings ...) 2004-08-26 8:31 ` Helge Hafting @ 2004-08-26 9:59 ` Spam 2004-08-26 13:05 ` Joseph Fannin 4 siblings, 0 replies; 1392+ messages in thread From: Spam @ 2004-08-26 9:59 UTC (permalink / raw) To: Matt Mackall Cc: Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list >> cp, grep, cat, and tar will continue to work just fine on files with >> multiple streams. > Find some silly person with an iBook and open a shell on OS X. Use cp > to copy a file with a resource fork. Oh look, the Finder has no idea > what the new file is, even though it looks exactly identical in the > shell. Isn't that _wonderful_? Now try cat < a > b on a file with a > fork. How is that ever going to work? It should work on the main file, the extra streams and meta data won't be copied. Just as it is now with special attributes etc. Cat reads the content of a file stream, not the file container. > I like cat < a > b. You can keep your progress. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 5:32 ` Matt Mackall ` (3 preceding siblings ...) 2004-08-26 9:59 ` Spam @ 2004-08-26 13:05 ` Joseph Fannin 4 siblings, 0 replies; 1392+ messages in thread From: Joseph Fannin @ 2004-08-26 13:05 UTC (permalink / raw) To: Matt Mackall Cc: Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 2248 bytes --] On Thu, Aug 26, 2004 at 12:32:00AM -0500, Matt Mackall wrote: > What it breaks is the concept of a file. In ways that are ill-defined, > not portable, hard to work with, and needlessly complex. Along the > way, it breaks every single application that ever thought it knew what > a file was. > > Progress? No, this has been done before. Various dead operating > systems have done it or similar and regretted it. Most recently MacOS, > which jumped through major hurdles to begin purging themselves of > resource forks when they went to OS X. They're still there, but > heavily deprecated. > > > cp, grep, cat, and tar will continue to work just fine on files with > > multiple streams. > > Find some silly person with an iBook and open a shell on OS X. Use cp > to copy a file with a resource fork. Oh look, the Finder has no idea > what the new file is, even though it looks exactly identical in the > shell. Isn't that _wonderful_? Now try cat < a > b on a file with a > fork. How is that ever going to work? I've been known to be silly now and then, for sure. Apple says this about the next version of OS X, due next year: "Tiger provides a standard, Darwin-level API for managing resource forks, filesystem metadata, security information, properties and other attributes in a consistent, cross-platform manner. For example, common UNIX utilities such as cp, tar and rsync can properly handle HFS+ resource forks." That's marketing copy, but I think it's worth noting. It seems to be that resource forks aren't deprecated across the board, just... changed. Apple has found a way to make this work, at least partially. I don't know if it's a good way, but there it is. I'd like to be able to keep my Macintosh stuff on my Linux fileserver, without losing the resource forks. That would require that the network fs support it of course, and whether I could use tar to back up my linux box would depend on how streams might be implemented there. > I like cat < a > b. You can keep your progress. Can't we have both? -- Joseph Fannin jhf@rivenstone.net "Bull in pure form is rare; there is usually some contamination by data." -- William Graves Perry Jr. [-- Attachment #2: Type: application/pgp-signature, Size: 186 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 5:09 ` Nicholas Miell 2004-08-26 5:17 ` James Morris 2004-08-26 5:32 ` Matt Mackall @ 2004-08-26 6:53 ` Paul Jackson 2004-08-26 9:54 ` Spam 3 siblings, 0 replies; 1392+ messages in thread From: Paul Jackson @ 2004-08-26 6:53 UTC (permalink / raw) To: Nicholas Miell Cc: mpm, wichert, jra, akpm, spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > "OMG! It breaks tar and email!!!" argument doesn't fly. Things break all > the time and are fixed. It's called progress. Yes - we break things all the time. But we take notice, when we are breaking things, of how long standing and deeply embedded they are. The deeper the roots, the more respect we show it, and the harder it will be to change. Heaping scorn on someone who is reluctant to change something as deeply embedded as "a file is a byte stream" does not further the discussion. -- I won't rest till it's the best ... Programmer, Linux Scalability Paul Jackson <pj@sgi.com> 1.650.933.1373 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 5:09 ` Nicholas Miell ` (2 preceding siblings ...) 2004-08-26 6:53 ` Paul Jackson @ 2004-08-26 9:54 ` Spam 3 siblings, 0 replies; 1392+ messages in thread From: Spam @ 2004-08-26 9:54 UTC (permalink / raw) To: Nicholas Miell Cc: Matt Mackall, Wichert Akkerman, Jeremy Allison, Andrew Morton, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > cp, grep, cat, and tar will continue to work just fine on files with > multiple streams. > tar and cp will lose the extra streams until somebody fixes them, but > they lose ACLs and xattrs right now, and I don't hear anybody suggesting > that ACLs and xattrs be removed from the kernel because of this. I thought that the underlaying FS/VFS would make sure the extra streams would be preserved. Normal applications that doesn't have support for them should not destroy the extra streams (and meta data,attributes etc). Of course tar is different, but any probgram that uses the normal OS API to copy, write and read files should not affect the extra info. ~S > Fixing programs that do recursive filesystem traversals is a matter of a > glibc patch to nftw(3) and a modification to their option processing. > Holding back a useful feature because you don't want to upgrade > coreutils is just plain dumb. > (BTW, for email, multipart/parallel is a start, but a specific multipart > content type for multi-stream file attachments would probably be more > appropriate.) ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 4:44 ` Matt Mackall 2004-08-26 5:09 ` Nicholas Miell @ 2004-08-26 5:23 ` Jamie Lokier 2004-08-26 8:16 ` Helge Hafting 2004-08-26 9:51 ` Spam 3 siblings, 0 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 5:23 UTC (permalink / raw) To: Matt Mackall Cc: Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Matt Mackall wrote: > > Anything that currently stores a file's metadata in another file really > > wants this right now. Things like image thumbnails, document summaries, > > digital signatures, etc. > > That is _highly_ debatable. I would much rather have my cp and grep > and cat and tar and such continue to work than have to rewrite every > tool because we've thrown the file-is-a-stream-of-bytes concept out > the window. Never mind that I've got thumbnails, document summaries, > and digital signatures already. > > While the number of annoying properties of files with forks is > practically endless, the biggest has got to be utter lack of > portability. How do you stick the thing in an attachment or on an ftp > site? Well you can't because it's NOT A FILE. > > A file is a stream of bytes. I couldn't agree more. Metadata which is easily lost is a terrible place to put a "document summary". However, it's not a bad place to put thumbnails which are generated from the file contents, if they can be regenerated after transporting. It's slightly better than a ".thumbnails" directory because the latter won't follow renames and things like that. It's a very good place to put metadata which is semantically bound to the file in special ways not intended for transport, such as security attributes, auto-invalidated digests of the file's contents, an auto-unpacked view of an archive, a virtual tree of an XML or other structured file, or an auto-generated textual representation of a binary file. The key concept is "not intended for transport". Generally that means permissions metadata, and things which are re-generated from the file's contents on demand.* It's a very bad place to put an abstract of a text document, or the names of authors, or the MIME content type, or the character encoding of the document, or the name of the shell to run the file as a script, unless these things are also deducible from the file's contents. However, sometimes the content doesn't give any clue, and then something like the character encoding of a text file is usefully stored in the metadata simply because that's better than nothing. Of course the facility can be abused. What does Windows XP do? Does it only store the appropriate kind of metadata in the alternate streams? Even Windows XP users expect files transfered over HTTP or WebDAV to work, don't they? -- Jamie * - By my definition, modification time arguably shouldn't be metadata. Personally I like downloads to be stamped with their source's modification time, which is why I don't use Mozilla's download manager but cut-and-paste URLs to "wget -N" instead. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 4:44 ` Matt Mackall 2004-08-26 5:09 ` Nicholas Miell 2004-08-26 5:23 ` Jamie Lokier @ 2004-08-26 8:16 ` Helge Hafting 2004-08-26 9:51 ` Spam 3 siblings, 0 replies; 1392+ messages in thread From: Helge Hafting @ 2004-08-26 8:16 UTC (permalink / raw) To: Matt Mackall Cc: Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, Spam, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Matt Mackall wrote: >On Wed, Aug 25, 2004 at 05:42:21PM -0700, Nicholas Miell wrote: > > >>On Wed, 2004-08-25 at 16:46, Wichert Akkerman wrote: >> >> >>>Previously Jeremy Allison wrote: >>> >>> >>>>Multiple-data-stream files are something we should offer, definately (IMHO). >>>>I don't care how we do it, but I know it's something we need as application >>>>developers. >>>> >>>> >>>Aside from samba, is there any other application that has a use for >>>them? >>> >>> >>> >>Anything that currently stores a file's metadata in another file really >>wants this right now. Things like image thumbnails, document summaries, >>digital signatures, etc. >> >> > >That is _highly_ debatable. I would much rather have my cp and grep >and cat and tar and such continue to work than have to rewrite every >tool because we've thrown the file-is-a-stream-of-bytes concept out >the window. Never mind that I've got thumbnails, document summaries, >and digital signatures already. > > Utilities that works on files only, such as cat, should keep working. No problem there. If you cat a file that also is a directory, then the file contents is all you get - by design. Utilities that _only_ traverse the directory tree, such as find, should keep working too. Perhaps with a very minor update so they don't mistake a file-directory for a file only. I.e. find must recurse into anything that support directory semantics. Something that both recurse and operate on files (cp -a, tar, grep, ... will need minor updating. They already knows how to handle files and directories, now they will need an update so they're open for objects that are both. I.e. let grep scan the file as usual, then recurse into its directory part in the usual way too. Performance problems should be avoided by not supporting directory operations on files with no directory content, which I believe will be many of them. The "file-as-directory" thing will not be that useful before the tools gets these relatively simple updates. Till then it'll be a toy, which shouldn't stop it from getting into the VFS. Updating the tools will be a task for the file-as-dir fans. (I don't know wether reiser4 does things this way - it is certainly they way I would want file-as-directory though.) >While the number of annoying properties of files with forks is >practically endless, the biggest has got to be utter lack of >portability. How do you stick the thing in an attachment or on an ftp >site? Well you can't because it's NOT A FILE. > > Stick the file in an attachment and you get the file only. No problem, it is designed that way. An app that really wants everything in a single file should use a file structured for that, not file-as-dir. File-as-dir attach stuff to a file in a more loose way. If you want to attach the directory contents too, do what you usually do when you want to mail someone a directory tree. You can't stick a directory in an attachment because it is not a file. So you either attach every file in the tree, or use tar. In this case, an updated tar. The ftp server shouldn't be a problem. It supports both files and directories already. It may need a minor update in order to not mistake directory for file or vice-versa when someone request an operation. ftp> get filename #Will get you the contents of the file part only - by design. ftp> cd filename #Will change into the directory (if the file indeed provides one.) >A file is a stream of bytes. > > Sure. And a file with a directory support both directory and file operations. You can get the stream of bytes as usual - that's the file part. Or you can cd into the directory as usual. There isn't much overlap between file operations and directory operations, so there is little conflict the way I see it. Merely letting the tools know that being a file no longer rule out the possibilities of directory operations. Getting the VFS right is another matter of course, but I don't worry about userland tools. Helge Hafting ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 4:44 ` Matt Mackall ` (2 preceding siblings ...) 2004-08-26 8:16 ` Helge Hafting @ 2004-08-26 9:51 ` Spam 3 siblings, 0 replies; 1392+ messages in thread From: Spam @ 2004-08-26 9:51 UTC (permalink / raw) To: Matt Mackall Cc: Nicholas Miell, Wichert Akkerman, Jeremy Allison, Andrew Morton, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > On Wed, Aug 25, 2004 at 05:42:21PM -0700, Nicholas Miell wrote: >> On Wed, 2004-08-25 at 16:46, Wichert Akkerman wrote: >> > Previously Jeremy Allison wrote: >> > > Multiple-data-stream files are something we should offer, definately (IMHO). >> > > I don't care how we do it, but I know it's something we need as application >> > > developers. >> > >> > Aside from samba, is there any other application that has a use for >> > them? >> > >> >> Anything that currently stores a file's metadata in another file really >> wants this right now. Things like image thumbnails, document summaries, >> digital signatures, etc. > That is _highly_ debatable. I would much rather have my cp and grep > and cat and tar and such continue to work than have to rewrite every > tool because we've thrown the file-is-a-stream-of-bytes concept out > the window. Never mind that I've got thumbnails, document summaries, > and digital signatures already. In Windows, the extra file streams are not lost or removed if you use a program that doesn't support them. They are only lost if you move the file to a file system that doesn't support the streams. Even RAR support the NTFS file streams. > While the number of annoying properties of files with forks is > practically endless, the biggest has got to be utter lack of > portability. How do you stick the thing in an attachment or on an ftp > site? Well you can't because it's NOT A FILE. > A file is a stream of bytes. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 23:46 ` Wichert Akkerman 2004-08-26 0:42 ` Nicholas Miell @ 2004-08-26 9:40 ` Spam 2004-08-26 9:49 ` Andrew Morton 1 sibling, 1 reply; 1392+ messages in thread From: Spam @ 2004-08-26 9:40 UTC (permalink / raw) To: Wichert Akkerman Cc: Jeremy Allison, Andrew Morton, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > Previously Jeremy Allison wrote: >> Multiple-data-stream files are something we should offer, definately (IMHO). >> I don't care how we do it, but I know it's something we need as application >> developers. > Aside from samba, is there any other application that has a use for > them? Yes, for example documents, image files etc. The multiple data streams can contain thumbnails, info about who is editing the file (useful for networked files) etc. Could be used for version handling and much more. Also, just because we do not see all the benefits right now, doesn't mean there won't be any. ~S > Wichert. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 9:40 ` Spam @ 2004-08-26 9:49 ` Andrew Morton 2004-08-26 10:20 ` Spam ` (2 more replies) 0 siblings, 3 replies; 1392+ messages in thread From: Andrew Morton @ 2004-08-26 9:49 UTC (permalink / raw) To: Spam Cc: wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Spam <spam@tnonline.net> wrote: > > Yes, for example documents, image files etc. The multiple data > streams can contain thumbnails, info about who is editing the file > (useful for networked files) etc. Could be used for version handling > and much more. All of which can be handled in userspace library code. What compelling reason is there for doing this in the kernel? ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 9:49 ` Andrew Morton @ 2004-08-26 10:20 ` Spam 2004-08-26 10:24 ` Andrew Morton ` (2 more replies) 2004-08-26 13:36 ` Rik van Riel 2004-08-26 17:32 ` Jeremy Allison 2 siblings, 3 replies; 1392+ messages in thread From: Spam @ 2004-08-26 10:20 UTC (permalink / raw) To: Andrew Morton Cc: wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > Spam <spam@tnonline.net> wrote: >> >> Yes, for example documents, image files etc. The multiple data >> streams can contain thumbnails, info about who is editing the file >> (useful for networked files) etc. Could be used for version handling >> and much more. > All of which can be handled in userspace library code. > What compelling reason is there for doing this in the kernel? Because having user space tools and code will make it not work with everything. Keeping stuff in the kernel should make the new features transparent to the applications. Applications that support the new features will benefit, all others will continue to work without destroying data. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:20 ` Spam @ 2004-08-26 10:24 ` Andrew Morton 2004-08-26 10:51 ` Spam 2004-08-26 10:54 ` Wichert Akkerman 2004-08-26 11:16 ` Christer Weinigel 2 siblings, 1 reply; 1392+ messages in thread From: Andrew Morton @ 2004-08-26 10:24 UTC (permalink / raw) To: Spam Cc: wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Spam <spam@tnonline.net> wrote: > > > > > Spam <spam@tnonline.net> wrote: > >> > >> Yes, for example documents, image files etc. The multiple data > >> streams can contain thumbnails, info about who is editing the file > >> (useful for networked files) etc. Could be used for version handling > >> and much more. > > > All of which can be handled in userspace library code. > > > What compelling reason is there for doing this in the kernel? > > > Because having user space tools and code will make it not work with > everything. Keeping stuff in the kernel should make the new features > transparent to the applications. > > Applications that support the new features will benefit, all others > will continue to work without destroying data. Sorry, but that all sounds a bit fluffy. Please provide some examples. (Generally, getting all of userspace to agree on a particular library is socially hard [*], but I don't see that as a reason for putting the functionality into the kernel) [*] Example: where's the library to manipulate /etc/whatever.conf? [**] [**] yes, I know about gconf. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:24 ` Andrew Morton @ 2004-08-26 10:51 ` Spam 2004-08-26 10:55 ` Andrew Morton ` (2 more replies) 0 siblings, 3 replies; 1392+ messages in thread From: Spam @ 2004-08-26 10:51 UTC (permalink / raw) To: Andrew Morton Cc: wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > Spam <spam@tnonline.net> wrote: >> >> >> >> > Spam <spam@tnonline.net> wrote: >> >> >> >> Yes, for example documents, image files etc. The multiple data >> >> streams can contain thumbnails, info about who is editing the file >> >> (useful for networked files) etc. Could be used for version handling >> >> and much more. >> >> > All of which can be handled in userspace library code. >> >> > What compelling reason is there for doing this in the kernel? >> >> >> Because having user space tools and code will make it not work with >> everything. Keeping stuff in the kernel should make the new features >> transparent to the applications. >> >> Applications that support the new features will benefit, all others >> will continue to work without destroying data. > Sorry, but that all sounds a bit fluffy. Please provide some examples. We already had the examples with cp and mv. Both should continue to work and the files will still be copied. The same with Konqueror and Nautilus. Files and their meta-files/streams/attributes will be retained as long as applications are using the OS API. However, if things are to implemented in user-space, then old applications will not work correctly and there is risk that all the extra information will be lost or corrupted. You said it would be socially hard. I think it would be very much close to impossible to get it right. Imagine that Gnome and Nautilus would implement support for these. I doubt that cp, mv, KDE, mc, app-xyz would implement this anytime soon and in the meantime the data is at risk. If we want things to work from the start, including with old applications then these features should be at below-application level. If it is in VFS or FS doesn't matter to me, as long as it doesn't hinder or delay progress to long. > (Generally, getting all of userspace to agree on a particular library is > socially hard [*], but I don't see that as a reason for putting the > functionality into the kernel) > [*] Example: where's the library to manipulate /etc/whatever.conf? [**] > [**] yes, I know about gconf. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:51 ` Spam @ 2004-08-26 10:55 ` Andrew Morton 2004-08-26 11:15 ` Spam ` (3 more replies) 2004-08-26 11:11 ` Bernd Petrovitsch 2004-08-26 23:59 ` Hans Reiser 2 siblings, 4 replies; 1392+ messages in thread From: Andrew Morton @ 2004-08-26 10:55 UTC (permalink / raw) To: Spam Cc: wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Spam <spam@tnonline.net> wrote: > > > > > Spam <spam@tnonline.net> wrote: > >> > >> > >> > >> > Spam <spam@tnonline.net> wrote: > >> >> > >> >> Yes, for example documents, image files etc. The multiple data > >> >> streams can contain thumbnails, info about who is editing the file > >> >> (useful for networked files) etc. Could be used for version handling > >> >> and much more. > >> > >> > All of which can be handled in userspace library code. > >> > >> > What compelling reason is there for doing this in the kernel? > >> > >> > >> Because having user space tools and code will make it not work with > >> everything. Keeping stuff in the kernel should make the new features > >> transparent to the applications. > >> > >> Applications that support the new features will benefit, all others > >> will continue to work without destroying data. > > > Sorry, but that all sounds a bit fluffy. Please provide some examples. > > We already had the examples with cp and mv. Both should continue to > work and the files will still be copied. The same with Konqueror and > Nautilus. Files and their meta-files/streams/attributes will be > retained as long as applications are using the OS API. > > However, if things are to implemented in user-space, then old > applications will not work correctly and there is risk that all the > extra information will be lost or corrupted. > > You said it would be socially hard. I think it would be very much > close to impossible to get it right. Imagine that Gnome and Nautilus > would implement support for these. I doubt that cp, mv, KDE, mc, > app-xyz would implement this anytime soon and in the meantime the > data is at risk. > No. All of the applications which you initially identified can be implemented by putting the various bits of data into a single file and getting applications to agree on the format of that file. For example, some image file formats already support embedded metadata, do they not? ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:55 ` Andrew Morton @ 2004-08-26 11:15 ` Spam 2004-08-26 11:20 ` Andrew Morton ` (3 more replies) 2004-08-26 11:48 ` Jamie Lokier ` (2 subsequent siblings) 3 siblings, 4 replies; 1392+ messages in thread From: Spam @ 2004-08-26 11:15 UTC (permalink / raw) To: Andrew Morton Cc: wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > Spam <spam@tnonline.net> wrote: >> >> >> >> > Spam <spam@tnonline.net> wrote: >> >> >> >> >> >> >> >> > Spam <spam@tnonline.net> wrote: >> >> >> >> >> >> Yes, for example documents, image files etc. The multiple data >> >> >> streams can contain thumbnails, info about who is editing the file >> >> >> (useful for networked files) etc. Could be used for version handling >> >> >> and much more. >> >> >> >> > All of which can be handled in userspace library code. >> >> >> >> > What compelling reason is there for doing this in the kernel? >> >> >> >> >> >> Because having user space tools and code will make it not work with >> >> everything. Keeping stuff in the kernel should make the new features >> >> transparent to the applications. >> >> >> >> Applications that support the new features will benefit, all others >> >> will continue to work without destroying data. >> >> > Sorry, but that all sounds a bit fluffy. Please provide some examples. >> >> We already had the examples with cp and mv. Both should continue to >> work and the files will still be copied. The same with Konqueror and >> Nautilus. Files and their meta-files/streams/attributes will be >> retained as long as applications are using the OS API. >> >> However, if things are to implemented in user-space, then old >> applications will not work correctly and there is risk that all the >> extra information will be lost or corrupted. >> >> You said it would be socially hard. I think it would be very much >> close to impossible to get it right. Imagine that Gnome and Nautilus >> would implement support for these. I doubt that cp, mv, KDE, mc, >> app-xyz would implement this anytime soon and in the meantime the >> data is at risk. >> > No. All of the applications which you initially identified can be > implemented by putting the various bits of data into a single file and > getting applications to agree on the format of that file. > For example, some image file formats already support embedded metadata, do > they not? Yes, JPEG, TIFF and PNG files for example. But, if you modify any of these with an application that doesn't support the extensions then you will loose them. Also, you are thinking _very_ narrowly now. There are thousands of file formats. Implementing the support for meta-data/ streams/ attributes in the kernel will make it possible to use this generically for all files. I use this in Windows quite much. I put information description fields for lots of different files. These descriptions are then searchable etc. I could use the command prompt to copy the files and the descriptions would still be there. Secondly, do you expect file managers like Nautilus and Konqueror to support every piece of file format on the planet so they could read information directly from the documents? ~S ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:15 ` Spam @ 2004-08-26 11:20 ` Andrew Morton 2004-08-26 11:29 ` Wichert Akkerman 2004-08-28 0:24 ` Horst von Brand 2004-08-26 11:27 ` Wichert Akkerman ` (2 subsequent siblings) 3 siblings, 2 replies; 1392+ messages in thread From: Andrew Morton @ 2004-08-26 11:20 UTC (permalink / raw) To: Spam Cc: wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Spam <spam@tnonline.net> wrote: > > Secondly, do you expect file managers like Nautilus and Konqueror to > support every piece of file format on the planet so they could read > information directly from the documents? Sure. You're proposing that we implement a single, golden multi-stream file format in the kernel. We could just as well do that in libmultistreamfileformat.so. But I'll grant that one cannot go adding new metadata to, say, C files this way. I don't know how useful such a thing is though. Remember that my main point is that there's a lack of coordination in userspace. Hell, there's none. Putting it in-kernel forces that coordination, and may be the way to go, but it's pretty sad. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:20 ` Andrew Morton @ 2004-08-26 11:29 ` Wichert Akkerman 2004-09-02 23:18 ` Tom Vier 2004-09-09 16:46 ` Timothy Miller 2004-08-28 0:24 ` Horst von Brand 1 sibling, 2 replies; 1392+ messages in thread From: Wichert Akkerman @ 2004-08-26 11:29 UTC (permalink / raw) To: Andrew Morton Cc: Spam, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Previously Andrew Morton wrote: > But I'll grant that one cannot go adding new metadata to, say, C files this > way. I don't know how useful such a thing is though. That is actually one of the few places where a bit of metadata would be very useful. Right now there is no way to indicate in what encoding a source is written: ascii, utf-8, ucs16, etc. are all possible. But a compiler or interpreter has no good way to figure that out. Wichert. -- Wichert Akkerman <wichert@wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:29 ` Wichert Akkerman @ 2004-09-02 23:18 ` Tom Vier 2004-09-09 16:46 ` Timothy Miller 1 sibling, 0 replies; 1392+ messages in thread From: Tom Vier @ 2004-09-02 23:18 UTC (permalink / raw) To: Wichert Akkerman; +Cc: linux-kernel On Thu, Aug 26, 2004 at 01:29:38PM +0200, Wichert Akkerman wrote: > That is actually one of the few places where a bit of metadata would be > very useful. Right now there is no way to indicate in what encoding a > source is written: ascii, utf-8, ucs16, etc. are all possible. But a > compiler or interpreter has no good way to figure that out. Make it a cmd line option. The makefile can contain info on what files are what and invoke the right options. -- Tom Vier <tmv@comcast.net> DSA Key ID 0x15741ECE ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:29 ` Wichert Akkerman 2004-09-02 23:18 ` Tom Vier @ 2004-09-09 16:46 ` Timothy Miller 1 sibling, 0 replies; 1392+ messages in thread From: Timothy Miller @ 2004-09-09 16:46 UTC (permalink / raw) To: Wichert Akkerman Cc: Andrew Morton, Spam, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Wichert Akkerman wrote: > Previously Andrew Morton wrote: > >>But I'll grant that one cannot go adding new metadata to, say, C files this >>way. I don't know how useful such a thing is though. > > > That is actually one of the few places where a bit of metadata would be > very useful. Right now there is no way to indicate in what encoding a > source is written: ascii, utf-8, ucs16, etc. are all possible. But a > compiler or interpreter has no good way to figure that out. > [NOTE: I am 5000 messages behind. Please forgive any redundancy.] This reminds me of a paper someone wrote on how HFS(+) stored the file type (actually, application that knows how to use the file) as metadata, separate from the filename. He was lamenting the fact that the Mac was being 'corrupted' by the PC's broken philosophy of including as part of the filename something which should not be. He also mentioned that Windows' feature of hiding the extension doesn't cut it. One benefit, I recall, was that you can't change the association accidentally when changing the filename. Another thing was that file name and file type are not semantically related, so they shouldn't be squished together. I don't remember this well enough, so I can't argue the point, but having the file type as metadata separate from the filename has SOME amount of elegant appeal to me. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:20 ` Andrew Morton 2004-08-26 11:29 ` Wichert Akkerman @ 2004-08-28 0:24 ` Horst von Brand 1 sibling, 0 replies; 1392+ messages in thread From: Horst von Brand @ 2004-08-28 0:24 UTC (permalink / raw) To: Andrew Morton Cc: Spam, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Andrew Morton <akpm@osdl.org> said: > Spam <spam@tnonline.net> wrote: > > > > Secondly, do you expect file managers like Nautilus and Konqueror to > > support every piece of file format on the planet so they could read > > information directly from the documents? > > Sure. You're proposing that we implement a single, golden multi-stream file > format in the kernel. Which won't be able to cater for all needs, I'm afraid. Just look at /etc, and contemplate why there are so many different configuration file formats. Sure, some are just random coder preferences set in stone, but others do make sense being different. The same will happen with file formats. Heck, it is already that way, with file formats for C source code, ELF shared libraries, various graphics and audio and video formats, ... > We could just as well do that in libmultistreamfileformat.so. Right. > But I'll grant that one cannot go adding new metadata to, say, C files this > way. I don't know how useful such a thing is though. You can never know. Perhaps annotations in swahili? Alternative versions for different architectures? In object format(s)? Copyright info? Changelog? > Remember that my main point is that there's a lack of coordination in > userspace. Hell, there's none. Putting it in-kernel forces that > coordination, and may be the way to go, but it's pretty sad. I think that forcing coordination is wrong. For example, I do value having ar/tar/cpio/zip doing essentially the same thing (and being able to define and use my own way of doing it if it pleases me), and I'd strongly oppose the kernel forcing one way of doing it down my throat. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:15 ` Spam 2004-08-26 11:20 ` Andrew Morton @ 2004-08-26 11:27 ` Wichert Akkerman 2004-08-26 13:39 ` Rik van Riel 2004-08-27 23:58 ` Horst von Brand 3 siblings, 0 replies; 1392+ messages in thread From: Wichert Akkerman @ 2004-08-26 11:27 UTC (permalink / raw) To: Spam Cc: Andrew Morton, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Previously Spam wrote: > Also, you are thinking _very_ narrowly now. There are thousands of > file formats. Implementing the support for meta-data/ streams/ > attributes in the kernel will make it possible to use this > generically for all files. And how many different formats will be used for thumbnails? How many different names to indicate a summary for a document? Having the ability to add attributes standardizes a way to access that data, but not the format and naming of those attributes. It could be a step closer to a more flexible way of handling storage, but you have to realize that a lot (if not most) of the work still needs to be done in userspace. Wichert. -- Wichert Akkerman <wichert@wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:15 ` Spam 2004-08-26 11:20 ` Andrew Morton 2004-08-26 11:27 ` Wichert Akkerman @ 2004-08-26 13:39 ` Rik van Riel 2004-08-26 13:59 ` Spam 2004-08-27 23:58 ` Horst von Brand 3 siblings, 1 reply; 1392+ messages in thread From: Rik van Riel @ 2004-08-26 13:39 UTC (permalink / raw) To: Spam Cc: Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Spam wrote: > Yes, JPEG, TIFF and PNG files for example. But, if you modify any of > these with an application that doesn't support the extensions then > you will loose them. OK, so we've got a choice. Either we will lose the extensions when modifying the file with an unaware program, or you lose the extensions when copying (or restoring from backup) using an unaware program. Personally I'd prefer to keep the file intact when not modifying it... -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 13:39 ` Rik van Riel @ 2004-08-26 13:59 ` Spam 2004-08-26 14:39 ` Felipe Alfaro Solana 0 siblings, 1 reply; 1392+ messages in thread From: Spam @ 2004-08-26 13:59 UTC (permalink / raw) To: Rik van Riel Cc: Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > On Thu, 26 Aug 2004, Spam wrote: >> Yes, JPEG, TIFF and PNG files for example. But, if you modify any of >> these with an application that doesn't support the extensions then >> you will loose them. > OK, so we've got a choice. > Either we will lose the extensions when modifying the file > with an unaware program, or you lose the extensions when > copying (or restoring from backup) using an unaware program. > Personally I'd prefer to keep the file intact when not > modifying it... Backup to me is a special case, not simply a copy of files, but also retaining all the extra data, info, attributes, etc, that comes with the file. Enabling support to have all of this extra stuff below the application level it will be possible to retain everything even though applications do not support them. Backup tools, however, must know about these new features to be able to backup them. And so will every tool that directly accesses the filesystem instead of using the OS API for it. And from I learned in an earlier message, tools like cp actually work on the fs level and themselves move the data to the new file instead of letting the OS handle it. This seem to me as it could be dangerous and could also prevent any kind of enhancements to the FS unless every tool like cp is patched. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 13:59 ` Spam @ 2004-08-26 14:39 ` Felipe Alfaro Solana 0 siblings, 0 replies; 1392+ messages in thread From: Felipe Alfaro Solana @ 2004-08-26 14:39 UTC (permalink / raw) To: Spam Cc: Rik van Riel, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thursday 26 August 2004 15:59, Spam wrote: > Backup to me is a special case, not simply a copy of files, but also > retaining all the extra data, info, attributes, etc, that comes with > the file. > > Enabling support to have all of this extra stuff below the > application level it will be possible to retain everything even > though applications do not support them. > > Backup tools, however, must know about these new features to be able > to backup them. And so will every tool that directly accesses the > filesystem instead of using the OS API for it. Thus, this kind of metadata API must be made accessible through the VFS layer, if we want to make it FS-independent (not tied to Reiser4). If the real FS does not support metadata operations, just ignore them (much like in ISO9660 ignoring write operations). > And from I learned in an earlier message, tools like cp actually > work on the fs level and themselves move the data to the new file > instead of letting the OS handle it. This seem to me as it could > be dangerous and could also prevent any kind of enhancements to the > FS unless every tool like cp is patched. I still don't understand why a copy_file() syscall is still missing. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:15 ` Spam ` (2 preceding siblings ...) 2004-08-26 13:39 ` Rik van Riel @ 2004-08-27 23:58 ` Horst von Brand 2004-08-29 15:17 ` Markus Törnqvist ` (2 more replies) 3 siblings, 3 replies; 1392+ messages in thread From: Horst von Brand @ 2004-08-27 23:58 UTC (permalink / raw) To: Spam Cc: Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Spam <spam@tnonline.net> said: > Andrew Morton <akpm@osdl.org> said: [...] > > For example, some image file formats already support embedded metadata, do > > they not? > Yes, JPEG, TIFF and PNG files for example. But, if you modify any of > these with an application that doesn't support the extensions then > you will loose them. As you will each time you muck around with your files-are-directories. Nothing new there. > Also, you are thinking _very_ narrowly now. There are thousands of > file formats. Implementing the support for meta-data/ streams/ > attributes in the kernel will make it possible to use this > generically for all files. So the _kernel_ has to know about thousands of formats, just in case it some blue day it comes across a strange file? Better leave that to the applications. You will _not_ be able to define a single format for extra data about the file, there will be differing extra data for different users (do you suggest a root-only file with special writable pouches for "graphical icon for the file" for each user on the system?!), so the idea in itself is doomed from the start. > I use this in Windows quite much. Then use it and be happy. No need to screw up Linux for that. > I put information description > fields for lots of different files. These descriptions are then > searchable etc. I could use the command prompt to copy the files and > the descriptions would still be there. The descriptions might make sense to _you_, _now_. No guarantee they make any sense (or are in the least useful) for other users on your system, and I might want them in arabic or some such. The descriptions might make no sense to you in a couple of years. > Secondly, do you expect file managers like Nautilus and Konqueror to > support every piece of file format on the planet so they could read > information directly from the documents? That's their (self-selected) job, yes. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 23:58 ` Horst von Brand @ 2004-08-29 15:17 ` Markus Törnqvist 2004-08-29 16:18 ` Spam 2004-08-29 19:10 ` Tonnerre 2 siblings, 0 replies; 1392+ messages in thread From: Markus Törnqvist @ 2004-08-29 15:17 UTC (permalink / raw) To: Horst von Brand Cc: Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, Aug 27, 2004 at 07:58:40PM -0400, Horst von Brand wrote: > >So the _kernel_ has to know about thousands of formats, just in case it >some blue day it comes across a strange file? Better leave that to the >applications. Or come up with a way to export some interface to userspace. Or make a standard description format in something like XML that can be used for arbitrary files and have userspace interface with that metadata stream that is the XML. This is, however, dangerously close to what we have now and does. not. work, so the only added bonus we might have is the fact that it's endorsed by the kernel. I don't like this idea except as a last fallback solution... >> I use this in Windows quite much. >Then use it and be happy. No need to screw up Linux for that. Linux does not need to be screwed up by that. It can be done right. >The descriptions might make sense to _you_, _now_. No guarantee they make >any sense (or are in the least useful) for other users on your system, and They also make perfect sense to me, and I would like this to have had this yesterday. or last year ;) >I might want them in arabic or some such. The descriptions might make no >sense to you in a couple of years. So what do you do if you use someone else's system and he doesn't have Arabic man pages but you want them in Arabic? And a description that doesn't make sense in many years is badly formed. >> Secondly, do you expect file managers like Nautilus and Konqueror to >> support every piece of file format on the planet so they could read >> information directly from the documents? >That's their (self-selected) job, yes. Sure, but they store it differently in different places. Having them in the actual file makes it universal. I think the biggest issue is that you are arguing against this out of some principle. There is a certain want for these new features among the users, there are people who are willing to implement them, there is Linus who seems to agree and accept. What do you, and people who agree with you, lose if this gets implemented? Drive space when you have the slightly-grown kernel sources downloaded? -- mjt ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 23:58 ` Horst von Brand 2004-08-29 15:17 ` Markus Törnqvist @ 2004-08-29 16:18 ` Spam 2004-08-29 19:10 ` Tonnerre 2 siblings, 0 replies; 1392+ messages in thread From: Spam @ 2004-08-29 16:18 UTC (permalink / raw) To: Horst von Brand Cc: Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > Spam <spam@tnonline.net> said: >> Andrew Morton <akpm@osdl.org> said: > [...] >> > For example, some image file formats already support embedded metadata, do >> > they not? >> Yes, JPEG, TIFF and PNG files for example. But, if you modify any of >> these with an application that doesn't support the extensions then >> you will loose them. > As you will each time you muck around with your files-are-directories. > Nothing new there. >> Also, you are thinking _very_ narrowly now. There are thousands of >> file formats. Implementing the support for meta-data/ streams/ >> attributes in the kernel will make it possible to use this >> generically for all files. > So the _kernel_ has to know about thousands of formats, just in case it > some blue day it comes across a strange file? Better leave that to the > applications. No, not at all. The only think the kernel would need to support is the actual meta files and streams - not the contents of the streams. That is up to the applications to know and use. The idea with plugins to reiser4 would allow someone to create a plugin that would export the thumbnails from TIFF and JPEG images as separate streams. > You will _not_ be able to define a single format for extra data about the > file, there will be differing extra data for different users (do you > suggest a root-only file with special writable pouches for "graphical icon > for the file" for each user on the system?!), so the idea in itself is > doomed from the start. We do not need to define a single format. You are locking yourself to the contents of streams and meta files. If someone likes the idea of a plugin that would allow for icons then that may be it. It would be up to application developers to know about this and use it. Perhaps they could agree on a standard even. >> I use this in Windows quite much. > Then use it and be happy. No need to screw up Linux for that. This wouldn't screw anything up besides allowing for extending the functionality. There is noone that forces you to enable and instll plugins, use extra attributes or meta info. All of this always was optional (as I see it). >> I put information description >> fields for lots of different files. These descriptions are then >> searchable etc. I could use the command prompt to copy the files and >> the descriptions would still be there. > The descriptions might make sense to _you_, _now_. No guarantee they make > any sense (or are in the least useful) for other users on your system, and > I might want them in arabic or some such. The descriptions might make no > sense to you in a couple of years. So? The same may be for your own word documents etc. It doesn't mean I do not find it useful now, or that anyone else might not. Shall we not include features because "someone" on the same system may not use it? Seems very backward to me. If I include a filestream, description.xml, then that wouldn't affect anyone. I would know how to read the file, and that would be enough. >> Secondly, do you expect file managers like Nautilus and Konqueror to >> support every piece of file format on the planet so they could read >> information directly from the documents? > That's their (self-selected) job, yes. Yes and no. I used them as example but there are lots of other programs that can access files. Midnight commander, cp, tar, email programs and lots of others. besides, I think it would be easier for them to 1) write plugins and 2) just use normal file access method to provide the extra data to the users. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 23:58 ` Horst von Brand 2004-08-29 15:17 ` Markus Törnqvist 2004-08-29 16:18 ` Spam @ 2004-08-29 19:10 ` Tonnerre 2004-08-30 0:39 ` Alexander G. M. Smith 2 siblings, 1 reply; 1392+ messages in thread From: Tonnerre @ 2004-08-29 19:10 UTC (permalink / raw) To: Horst von Brand Cc: Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 889 bytes --] Salut, On Fri, Aug 27, 2004 at 07:58:40PM -0400, Horst von Brand wrote: > > Also, you are thinking _very_ narrowly now. There are thousands of > > file formats. Implementing the support for meta-data/ streams/ > > attributes in the kernel will make it possible to use this > > generically for all files. > > So the _kernel_ has to know about thousands of formats, just in case it > some blue day it comes across a strange file? Better leave that to the > applications. I'd do that quite differently: Collect metadata about files in extended attributes (which are rather universally useable), and do it from an userspace daemon/application/file manager plugin which uses dynamically loadable plugins for the different file types differentiated by libmagic. This has nothing to do in the kernel. Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 19:10 ` Tonnerre @ 2004-08-30 0:39 ` Alexander G. M. Smith 2004-08-30 18:39 ` Spam 2004-08-31 8:15 ` Tonnerre 0 siblings, 2 replies; 1392+ messages in thread From: Alexander G. M. Smith @ 2004-08-30 0:39 UTC (permalink / raw) To: Tonnerre Cc: spam, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list, vonbrand Tonnerre wrote on Sun, 29 Aug 2004 21:10:44 +0200: > On Fri, Aug 27, 2004 at 07:58:40PM -0400, Horst von Brand wrote: > > So the _kernel_ has to know about thousands of formats, just in case it > > some blue day it comes across a strange file? Better leave that to the > > applications. > > I'd do that quite differently: Collect metadata about files in > extended attributes (which are rather universally useable), and do it > from an userspace daemon/application/file manager plugin which uses > dynamically loadable plugins for the different file types > differentiated by libmagic. > > This has nothing to do in the kernel. I mostly agree, like I was saying earlier, file types are needed! The kernel doesn't have to know about all of them, just some of them. It should be possible to attach a file type to everything so you know what kind of thing it is, not just block or character device or file, but something like a MIME type. If space is a concern, add to the inode a unique 8 byte character code for each popular type and a global translation table to get back the MIME string. The kernel (includes file systems and plug-ins) would have to know some standard file types and how to compare them for indexing purposes. For example, an integer attribute like the year a music album was released would be stored as a 4 byte attribute "file" with a type like "primitive/int32", which tells the kernel to use 32 bit integer number comparison for indexing order. Another type the kernel could be aware of are the file types used for security attributes. A third kind it might need to know about are the computed attributes, which get set whenever related attributes they are computed from change. For any unknown types, the kernel just stores the data and doesn't do any extra processing. Optionally the kernel could also maintain the global list of all file types and their properties (such as which ones are indexed, which are computed), though that could also be done in userland if you aren't doing indexing or computed attributes or other fancy operations. The determination of the types would be in userland (smart programs would write the type when they created the file). Like you said, a daemon could classify unknown files in the background (BeOS does this). Most of the uses of the file type would also be in userland, such as deciding which program can handle a given file, or what icon to display, or to hint that a file/directory/attribute thing should be interpreted as an old style directory to the user, or to mark a file thing as being worth backing up (don't need to back up the computed ones). - Alex ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-30 0:39 ` Alexander G. M. Smith @ 2004-08-30 18:39 ` Spam 2004-08-31 8:15 ` Tonnerre 1 sibling, 0 replies; 1392+ messages in thread From: Spam @ 2004-08-30 18:39 UTC (permalink / raw) To: Alexander G. M. Smith Cc: Tonnerre, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list, vonbrand > Tonnerre wrote on Sun, 29 Aug 2004 21:10:44 +0200: >> On Fri, Aug 27, 2004 at 07:58:40PM -0400, Horst von Brand wrote: >> > So the _kernel_ has to know about thousands of formats, just in case it >> > some blue day it comes across a strange file? Better leave that to the >> > applications. >> >> I'd do that quite differently: Collect metadata about files in >> extended attributes (which are rather universally useable), and do it >> from an userspace daemon/application/file manager plugin which uses >> dynamically loadable plugins for the different file types >> differentiated by libmagic. >> >> This has nothing to do in the kernel. > I mostly agree, like I was saying earlier, file types are needed! > The kernel doesn't have to know about all of them, just some of > them. It should be possible to attach a file type to everything so > you know what kind of thing it is, not just block or character > device or file, but something like a MIME type. If space is a > concern, add to the inode a unique 8 byte character code for each > popular type and a global translation table to get back the MIME > string. The kernel doesn't have anything to know about normal files.It is up to applications to determine and know the contents. Isn't it? Exception would be of course special meta files that define attributes and permissions for the filesystem. > The kernel (includes file systems and plug-ins) would have to > know some standard file types and how to compare them for indexing > purposes. For example, an integer attribute like the year a music > album was released would be stored as a 4 byte attribute "file" with > a type like "primitive/int32", which tells the kernel to use 32 bit > integer number comparison for indexing order. Another type the > kernel could be aware of are the file types used for security > attributes. A third kind it might need to know about are the > computed attributes, which get set whenever related attributes they > are computed from change. For any unknown types, the kernel just > stores the data and doesn't do any extra processing. > Optionally the kernel could also maintain the global list of all > file types and their properties (such as which ones are indexed, > which are computed), though that could also be done in userland if > you aren't doing indexing or computed attributes or other fancy > operations. > The determination of the types would be in userland (smart > programs would write the type when they created the file). Like you > said, a daemon could classify unknown files in the background (BeOS > does this). Most of the uses of the file type would also be in > userland, such as deciding which program can handle a given file, or > what icon to display, or to hint that a file/directory/attribute > thing should be interpreted as an old style directory to the user, > or to mark a file thing as being worth backing up (don't need to > back up the computed ones). > - Alex ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-30 0:39 ` Alexander G. M. Smith 2004-08-30 18:39 ` Spam @ 2004-08-31 8:15 ` Tonnerre 2004-09-01 20:16 ` Jamie Lokier 1 sibling, 1 reply; 1392+ messages in thread From: Tonnerre @ 2004-08-31 8:15 UTC (permalink / raw) To: Alexander G. M. Smith Cc: spam, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list, vonbrand [-- Attachment #1: Type: text/plain, Size: 1351 bytes --] Salut, On Sun, Aug 29, 2004 at 08:39:25PM -0400, Alexander G. M. Smith wrote: > I mostly agree, like I was saying earlier, file types are needed! > The kernel doesn't have to know about all of them, just some of > them. It should be possible to attach a file type to everything so > you know what kind of thing it is, not just block or character > device or file, but something like a MIME type. Then store the value computed by libmagic into an extended attribute, if you like. I still don't see why the kernel should even care. It shouldn't read things from files anyway, as a kernel shall be bootable on a busybox system (and *Step) as well, so we don't rely on the fs layout in any way. > Optionally the kernel could also maintain the global list of all > file types and their properties (such as which ones are indexed, > which are computed), though that could also be done in userland if > you aren't doing indexing or computed attributes or other fancy > operations. Quel horreur! Do it in userland, really. If I get the time, I'll write you a small daemon based on libmagic which stores the file attributes in xattrs, or if they're not supported, in some MacOS/Xish per-directory files. Even a file manager ("finder") can do that, there's not even the need for a daemon. Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 8:15 ` Tonnerre @ 2004-09-01 20:16 ` Jamie Lokier [not found] ` <1094070506.28509.24.camel@shaggy.austin.ibm.com> ` (2 more replies) 0 siblings, 3 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-09-01 20:16 UTC (permalink / raw) To: Tonnerre Cc: Alexander G. M. Smith, spam, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list, vonbrand Tonnerre wrote: > Quel horreur! > Do it in userland, really. I'm amazed that after all this discussion where all the realistic implementations are done in userland with kernel support for calling out to it, there are people who think the kernel is supposed to decode MP3 files or whatever. Nobody is advocating that! > If I get the time, I'll write you a small daemon based on libmagic > which stores the file attributes in xattrs, or if they're not > supported, in some MacOS/Xish per-directory files. Even a file manager > ("finder") can do that, there's not even the need for a daemon. How are you going to do the part where the xattr changes when the file is modified? (For example, if I edit an HTML file which is encoded in iso-8859-1, change it to utf-8 and indicate that in a META element, and save it under the same name, the full content-type should change from "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".) I don't see how you can do that without kernel support. Don't say dnotify or inotify, because neither would work. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
[parent not found: <1094070506.28509.24.camel@shaggy.austin.ibm.com>]
* Re: silent semantic changes with reiser4 [not found] ` <1094070506.28509.24.camel@shaggy.austin.ibm.com> @ 2004-09-02 5:57 ` Jamie Lokier 0 siblings, 0 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-09-02 5:57 UTC (permalink / raw) To: Dave Kleikamp Cc: Tonnerre, Alexander G. M. Smith, spam, Andrew Morton, wichert, jra, Linus Torvalds, reiser, Christoph Hellwig, fsdevel, linux-kernel, flx, reiserfs-list, vonbrand Tonnerre wrote: > > > I'll write you a small daemon based on libmagic which stores the > > > file attributes in xattrs, or if they're not supported, in some > > > MacOS/Xish per-directory files. Even a file manager ("finder") can > > > do that, there's not even the need for a daemon. Jamie Lokier wrote: > > (For example, if I edit an HTML file which is encoded in iso-8859-1, > > change it to utf-8 and indicate that in a META element, and save it > > under the same name, the full content-type should change from > > "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".) > > > > I don't see how you can do that without kernel support. Dave Kleikamp wrote: > Your html editor should do that. My html editor is "vi". If I am supposed to manually set the content-type attribute after exiting vi, doesn't that rather invalidate the idea of a "small daemon based on libmagic" which sets it for me? That was the sole point of my statement. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:16 ` Jamie Lokier [not found] ` <1094070506.28509.24.camel@shaggy.austin.ibm.com> @ 2004-09-02 9:50 ` Alan Cox 2004-09-02 20:25 ` Jamie Lokier 2004-09-02 18:16 ` Christer Weinigel 2 siblings, 1 reply; 1392+ messages in thread From: Alan Cox @ 2004-09-02 9:50 UTC (permalink / raw) To: Jamie Lokier Cc: Tonnerre, Alexander G. M. Smith, spam, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, Linux Kernel Mailing List, flx, reiserfs-list, vonbrand On Mer, 2004-09-01 at 21:16, Jamie Lokier wrote: > (For example, if I edit an HTML file which is encoded in iso-8859-1, > change it to utf-8 and indicate that in a META element, and save it > under the same name, the full content-type should change from > "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".) > > I don't see how you can do that without kernel support. > > Don't say dnotify or inotify, because neither would work. inotify done right is useful here as well as in a lot of other desktop cases where dnotify doesn't really scale. Its enough to let me - Find the new file - Virus scan it - Classify its possible type heirarchies - Index it Alan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 9:50 ` Alan Cox @ 2004-09-02 20:25 ` Jamie Lokier 0 siblings, 0 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-09-02 20:25 UTC (permalink / raw) To: Alan Cox Cc: Tonnerre, Alexander G. M. Smith, spam, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, Linux Kernel Mailing List, flx, reiserfs-list, vonbrand Alan Cox wrote: > On Mer, 2004-09-01 at 21:16, Jamie Lokier wrote: > > (For example, if I edit an HTML file which is encoded in iso-8859-1, > > change it to utf-8 and indicate that in a META element, and save it > > under the same name, the full content-type should change from > > "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".) > > > > I don't see how you can do that without kernel support. > > > > Don't say dnotify or inotify, because neither would work. > > inotify done right is useful here as well as in a lot of other desktop > cases where dnotify doesn't really scale. Its enough to let me > > - Find the new file > - Virus scan it > - Classify its possible type heirarchies > - Index it Indeed it does, but it fails for the example I was commenting on to which you replied.. 1. The file /var/www/site/index.html's written in vi. 2. "The daemon" (that's what I objected to) receives inotify. blocks waiting for scheduler, however... 3. Seeing that vi is now finished, I phone person on other side of world and say the updated file is now available. 4. Person fetches http://site/index.html ...oh! They receive the wrong content-type (bollocks!) 5. Eventually after paging & scheduling deems it appropriate, "The daemon" gets to run, looks at file, updates content-type. In other words I was criticising Tonnerre's idea of a daemon which updates the xattrs by looking at file contents... by definition, a daemon runs in the background, and the whole point of it updating a content-type xattr was obviously so that programs like httpd could just _use_ that xattr. That doesn't work, even if that daemon uses inotify. Basically, anytime where you want an ordering guarantee that something will be recalculated in the interval after a file is modified and before the calculated result is used, you have to be very careful about exactly which code is using dnotify or inotify to achieve that. Single threaded programs can use it easily, but a multi-threaded file server has to be very careful about ordering if it's to avoid glitches. It is exactly a real problem I have faced in a multi-threaded HTTP server which uses dnotify to detect changes to prerequisite files and also changes to the path walked and permissions on it, and thus invalidate cached generated pages, thus giving strong cache guarantees. Performance is great, if you ignore the race conditions... (One of the more popular small web servers, lighttpd (PHP people seem to like it), also uses dnotify. Apparently it provides a big performance boost, just by avoiding stat() calls. So I'm not alone in using dnotify for such things.) -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:16 ` Jamie Lokier [not found] ` <1094070506.28509.24.camel@shaggy.austin.ibm.com> 2004-09-02 9:50 ` Alan Cox @ 2004-09-02 18:16 ` Christer Weinigel 2004-09-02 19:58 ` Jamie Lokier 2 siblings, 1 reply; 1392+ messages in thread From: Christer Weinigel @ 2004-09-02 18:16 UTC (permalink / raw) To: Jamie Lokier Cc: Tonnerre, Alexander G. M. Smith, spam, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list, vonbrand Jamie Lokier <jamie@shareable.org> writes: > (For example, if I edit an HTML file which is encoded in iso-8859-1, > change it to utf-8 and indicate that in a META element, and save it > under the same name, the full content-type should change from > "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".) > > I don't see how you can do that without kernel support. charset_cache = dbm(os.getenv('HOME') + '/.charset_cache') def get_charset(path): file_mtime = get_mtime(path) cache_tuple = dbm.get(path, (None, None)) if file_mtime != cache_tuple[0]: cache_tuple = (file_mtime, figure_out_charset(path)) dbm.put(path, cache_tuple) return cache_tuple[1] This code is guaranteed to always give you an up to date charset for a file, provided that the mtime is guaranteed to change every time the file changes. If the file can change during the call to get_charset you'll have to lock the file while you are working with it or handle it some other way (comparing mtime before and after and retrying if they differ), but that is true even if you do it all in the kernel. This has the added advantage that the cache is updated lazily, so the charset is never calculated unless someone really needs it. If the disk gets full, it is time to shrink the cache and preferably stale entries are evicted first. If that's not good enough, a notifier such as dnotify or inotify can be used to invalidate the cache immediately after the file has changed. If this gets popular, the cache management can be moved to a daemon (naturally with all the security aspects in mind). > Don't say dnotify or inotify, because neither would work. Why not? The approach above works on any filesystem, even without dnotify or inotify but will be more efficient with them. /Christer -- "Just how much can I get away with and still go to heaven?" Freelance consultant specializing in device driver programming for Linux Christer Weinigel <christer@weinigel.se> http://www.weinigel.se ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 18:16 ` Christer Weinigel @ 2004-09-02 19:58 ` Jamie Lokier 0 siblings, 0 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-09-02 19:58 UTC (permalink / raw) To: Christer Weinigel Cc: Tonnerre, Alexander G. M. Smith, spam, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list, vonbrand Christer Weinigel wrote: > Jamie Lokier <jamie@shareable.org> writes: > > > (For example, if I edit an HTML file which is encoded in iso-8859-1, > > change it to utf-8 and indicate that in a META element, and save it > > under the same name, the full content-type should change from > > "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".) > > > > I don't see how you can do that without kernel support. > > [code which runs in the program which needs to read the content-type] > > This code is guaranteed to always give you an up to date charset for a > file, provided that the mtime is guaranteed to change every time the > file changes. ...which it isn't, but ignoring that. Yes we all know that code will work, turning a blind eye to occasional mtime failures. What I objected to isn't that, but Tonerre's idea of a daemon (i.e. in the background, that's what daemons do) updating xattrs. Your code will work. Tonerre's has a serious race condition. > > Don't say dnotify or inotify, because neither would work. > > Why not? The approach above works on any filesystem, even without > dnotify or inotify but will be more efficient with them. The above approach is in fact better with dnotify/inotify, when they area available, because then it doesn't have mtime problems. My objection is that dnotify & inotify don't fix the problems of a "daemon" (i.e. in the background) updating an xattr - because the daemon might not run scheduled between the write and when the file needs to be used. That said, yours has practical problems too. The intended purpose (*) of storing content-type in an xattr was so that programs which need to read content without understanding it may just do so (e.g. mail agents, httpd), and are distinct from the programs which compute and set it (e.g. programs which understand specific content formats). (*) I don't necessarily agree with that purpose. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:55 ` Andrew Morton 2004-08-26 11:15 ` Spam @ 2004-08-26 11:48 ` Jamie Lokier 2004-08-26 16:07 ` James Bruce 2004-08-26 23:52 ` Hans Reiser 3 siblings, 0 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 11:48 UTC (permalink / raw) To: Andrew Morton Cc: Spam, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Andrew Morton wrote: > No. All of the applications which you initially identified can be > implemented by putting the various bits of data into a single file and > getting applications to agree on the format of that file. > > For example, some image file formats already support embedded metadata, do > they not? Same for MP3s and their metadata tags. _That_ kind of data should be part of the contents of the file: it is intended to be transported. It is useful to auto-extract it from the file's data stream, using format-specific programs, and offer that metadata in separate files where it is easy to use e.g. by shell scripts and indexing programs, which can then work independently of file formats. That kind of metadata should be _derived_ from the file's data stream, and ideally the extraction should be automatic, with userspace assistance. It may be useful for the filesystem to record what's extracted, for fast access and indexing. The same applies to thumbnails. The only difference is that these may take longer to compute. But they're the same thing: something calculated from the file's data stream, which may (or may not) be stored in the filesystem so it doesn't have to be recomputed each time. It might even be worth expiring the metadata from disk if it hasn't been used for a while and can be recomputed. These are the same as looking into archives or structured files. The only difference is that the unpacked views might not be stored on the filesystem (but they might be, if the filesystem knows how to cache views for faster future access).[*] Those are all examples of data derived from the data stream of a flat file. They are useful applications, and all work fine with FTP, cp, cat <a >b, and email attachments, _no application changes required_. There's another kind of metadata which isn't intended for transport. That would be things like the classic attributes: permissions, timestamps. It would also include things like author information when the file format cannot carry it. -- Jamie [*] - A fancier variant, which is hard without good fs support, it writing to structured files through a view. I.e. modifying files inside a .tar.gz, or (this is useful) inside an {MS,Open,Gnome,K}Office container file or *SQL database table. To do this efficiently, the file's main data stream is regenerated on demand from the view's data. As I said, it requires good, special fs support to do it efficiently (fancy caching). With appropriate userspace libraries you can make apps work with container files on all other OSes (just like .tar.gz and *Office containers at the moment) but are significantly faster to work with on Linux (no need to serialise and (de-)compress except on demand) and more visible to the standard unix scripting tools. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:55 ` Andrew Morton 2004-08-26 11:15 ` Spam 2004-08-26 11:48 ` Jamie Lokier @ 2004-08-26 16:07 ` James Bruce 2004-08-26 16:58 ` Jamie Lokier 2004-08-26 23:52 ` Hans Reiser 3 siblings, 1 reply; 1392+ messages in thread From: James Bruce @ 2004-08-26 16:07 UTC (permalink / raw) To: Andrew Morton Cc: Spam, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Andrew Morton wrote: >No. All of the applications which you initially identified can be >implemented by putting the various bits of data into a single file and >getting applications to agree on the format of that file. > > So in order to avoid breaking backup and file utilities, we'd instead break every application that reads files? That way surely lies madness :) This /could/ possibly work with lots of cooperation from glibc, but we'd also need a way to insert and delete bytes in a file... moving 1MB of attributes/streams just to change a 20 character author attribute seems a bit silly. In addition, there are little things, such as how adding a thumbnail image shouldn't change the modification time of the original picture IMO, but it does if you embed it in the file. If it should be in userspace, we could take the OS-X approach of using directories for everything, and the "data" that would be in a hybrid directory-file is a specially named file under that directory. Programs for instance are actually directories, with the elf file underneath it. When you click on the "directory-program" in the gui it runs the associated elf file rather than opening the directory. I think that approach is promising, but so few unix programs have any indirection for file access it'd be hell to teach them all how it works. Of course with emminent breakage of some sort, maybe now is the time... - Jim Bruce ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:07 ` James Bruce @ 2004-08-26 16:58 ` Jamie Lokier 2004-08-26 21:34 ` Felipe Alfaro Solana 0 siblings, 1 reply; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 16:58 UTC (permalink / raw) To: James Bruce Cc: Andrew Morton, Spam, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list James Bruce wrote: > moving 1MB of attributes/streams just to change a 20 character > author attribute seems a bit silly. That's what we do now. Try editing an MP3's id tags. > If it should be in userspace, we could take the OS-X approach of using > directories for everything, and the "data" that would be in a hybrid > directory-file is a specially named file under that directory. Programs > for instance are actually directories, with the elf file underneath it. > When you click on the "directory-program" in the gui it runs the > associated elf file rather than opening the directory. I think that > approach is promising, but so few unix programs have any indirection for > file access it'd be hell to teach them all how it works. The interfaces we've been talking about are very close to that. The indirection is automatic: instead of a document (or program) looking like a directory, it'll look like a file, and reading (or executing) the file will do the indirection, with kernel control (and so consistency and performance) via an appropriate flat<->structured relation helper. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:58 ` Jamie Lokier @ 2004-08-26 21:34 ` Felipe Alfaro Solana 0 siblings, 0 replies; 1392+ messages in thread From: Felipe Alfaro Solana @ 2004-08-26 21:34 UTC (permalink / raw) To: Jamie Lokier Cc: James Bruce, Andrew Morton, Spam, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thursday 26 August 2004 18:58, Jamie Lokier wrote: > James Bruce wrote: > > moving 1MB of attributes/streams just to change a 20 character > > author attribute seems a bit silly. > > That's what we do now. Try editing an MP3's id tags. What's the problem? Let's say a file has three streams/forks/whatever: the unnamed one, and two additional named. The filesystem can treat these three streams like three independent (but related) files on disk so, if the author attribute belongs to the first named stream, only the first named stream (and its corresponding file on the backend) would need to be modified. No need to move MBs of data around. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:55 ` Andrew Morton ` (2 preceding siblings ...) 2004-08-26 16:07 ` James Bruce @ 2004-08-26 23:52 ` Hans Reiser 3 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 23:52 UTC (permalink / raw) To: Andrew Morton Cc: Spam, wichert, jra, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Andrew Morton wrote: > >No. All of the applications which you initially identified can be >implemented by putting the various bits of data into a single file and >getting applications to agree on the format of that file. > >For example, some image file formats already support embedded metadata, do >they not? > >- > > and that leads to vasts numbers of file formats, all of which require special code to parse them, and apps that don't duplicate that code can't access the data, and it is a software engineering mistake. Uniform interfaces greatly reduce the cost of an OS and increase its expressive power. Closure is the most important and least understood principle of OS design. By contrast, suppose everything was stored in files and directories. Every app can afford the coding cost to learn about files and directories. Data can freely bounce from one object to another in the OS because the APIs for the objects are all the same. That's increased expressive power. The number of connections between objects determines the expressive power of the OS, not the number of objects in it. Unified namespaces do a lot for an OS. Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:51 ` Spam 2004-08-26 10:55 ` Andrew Morton @ 2004-08-26 11:11 ` Bernd Petrovitsch 2004-08-26 23:59 ` Hans Reiser 2 siblings, 0 replies; 1392+ messages in thread From: Bernd Petrovitsch @ 2004-08-26 11:11 UTC (permalink / raw) To: Spam; +Cc: Andrew Morton, jra, hch, linux-fsdevel, linux-kernel [ shortened Cc: of reiser- and other folks since it is no longer reiser-specific - sorry, if it was too much ] [ fixed missing attribution ] On Thu, 2004-08-26 at 12:51 +0200, Spam wrote: > On Thu, 2004-08-26 at 03:24 -0700, Andrew Morton wrote: > > Spam <spam@tnonline.net> wrote: [...] > >> Applications that support the new features will benefit, all others > >> will continue to work without destroying data. > > > Sorry, but that all sounds a bit fluffy. Please provide some examples. It is too fluffy. > We already had the examples with cp and mv. Both should continue to > work and the files will still be copied. The same with Konqueror and ... after they are patched to support streams. As stated each stream has an own fd, so - from the user-space perspective - one has to open *all* streams of this file, read them and write them into the destination. > Nautilus. Files and their meta-files/streams/attributes will be > retained as long as applications are using the OS API. The OS-API is *one* file descriptor where you can read, write, mmap, ... on. Therefore stream selection must happen with open() since open() returns a fd (which uniquely identify a stream). Voila. And actually this makes transition actually possible: Old apps simply ignore streams (yes, there will be a default stream, which is used id nothing else is specified) and throw them away (think of e.g. patch which opens a new file, writes into it, unlink's the old one and mv's the new one on the old place) until they are modified to support them. Bernd -- Firmix Software GmbH http://www.firmix.at/ mobil: +43 664 4416156 fax: +43 1 7890849-55 Embedded Linux Development and Services ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:51 ` Spam 2004-08-26 10:55 ` Andrew Morton 2004-08-26 11:11 ` Bernd Petrovitsch @ 2004-08-26 23:59 ` Hans Reiser 2 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 23:59 UTC (permalink / raw) To: Spam Cc: Andrew Morton, wichert, jra, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list What is important is that it all be one packaged solution from the same maintainer. That is necessary for coherency of solution. There is also benefit to it being one coherent body of code, but this is harder to articulate. It is always hard to articulate why functionality belongs in the same program rather than split into multiple programs. Filesystems should provide primitives for ordering, structuring, and naming objects. It is hard to say why we should do all of that, but on some level I just know it is our task. Hans Spam wrote: > > > You said it would be socially hard. I think it would be very much > close to impossible to get it right. Imagine that Gnome and Nautilus > would implement support for these. I doubt that cp, mv, KDE, mc, > app-xyz would implement this anytime soon and in the meantime the > data is at risk. > > > > > > ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:20 ` Spam 2004-08-26 10:24 ` Andrew Morton @ 2004-08-26 10:54 ` Wichert Akkerman 2004-08-26 23:48 ` Hans Reiser 2004-08-26 11:16 ` Christer Weinigel 2 siblings, 1 reply; 1392+ messages in thread From: Wichert Akkerman @ 2004-08-26 10:54 UTC (permalink / raw) To: Spam Cc: Andrew Morton, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Previously Spam wrote: > Because having user space tools and code will make it not work with > everything. Keeping stuff in the kernel should make the new features > transparent to the applications. But having it in the kernel has the same problem. If you read the Solaris documentation you will see that a bunch of utilties had to get a new commandline option to be able to access the metadata and a special utility was added for other applications. If you look at windows you will see that you need to use a filename like <realfilename>:<streamname>:$DATA (which obviously does not work for single-character filenames). Ignoring samba for a bit which just needs streams to stay compatible with windows I see few reasons for using streams: * files are more complex these days and tend to include multiple different things: images with thumbnails and exif data, 'office' documents containing both text and images * standard way to add common metadata to a file which can be used for searching tools (author, copyright, keywords, etc.) But both can already be done in userland (modern image formats can store thumbnails and exif data internally, applications use tar or zip-like files for documents, etc.). The metadata part is a lot more complicated as well since the behaviour of attributes might need to be complex: if I change an image using an application that is not stream-aware, what should happen to its thumbnail? The only common benefits I can see are standardisation and optimization: instead of every file format or application defining a way to specify metadata for a file you get a common API defined by the OS (but you'll still need to standardize on attribute names and formats, so plenty of room that will still not help), and instead of parsing different files or XML streams you can directly access a bit of metadata. So far I'm not convinced that streams are worth the effort. Not that my opinion is all that relevant here, but still :) Wichert. -- Wichert Akkerman <wichert@wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:54 ` Wichert Akkerman @ 2004-08-26 23:48 ` Hans Reiser 0 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 23:48 UTC (permalink / raw) To: Wichert Akkerman Cc: Spam, Andrew Morton, jra, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Wichert Akkerman wrote: >Previously Spam wrote: > > >> Because having user space tools and code will make it not work with >> everything. Keeping stuff in the kernel should make the new features >> transparent to the applications. >> >> > >But having it in the kernel has the same problem. If you read the >Solaris documentation you will see that a bunch of utilties had to >get a new commandline option to be able to access the metadata and >a special utility was added for other applications. > They did it wrong. > If you look >at windows you will see that you need to use a filename like ><realfilename>:<streamname>:$DATA (which obviously does not work for >single-character filenames). > > again, they did it wrong. See Rob Pike's "The Hideous Name" for why that is wrong. >Ignoring samba for a bit which just needs streams to stay compatible >with windows I see few reasons for using streams: > >* files are more complex these days and tend to include multiple > different things: images with thumbnails and exif data, 'office' > documents containing both text and images > >* standard way to add common metadata to a file which can be used for > searching tools (author, copyright, keywords, etc.) > >But both can already be done in userland (modern image formats can store >thumbnails and exif data internally, applications use tar or zip-like >files for documents, etc.). The metadata part is a lot more complicated >as well since the behaviour of attributes might need to be complex: >if I change an image using an application that is not stream-aware, what >should happen to its thumbnail? > >The only common benefits I can see are standardisation and optimization: >instead of every file format or application defining a way to specify >metadata for a file you get a common API defined by the OS (but you'll >still need to standardize on attribute names and formats, so plenty of >room that will still not help), and instead of parsing different files >or XML streams you can directly access a bit of metadata. > >So far I'm not convinced that streams are worth the effort. Not that my >opinion is all that relevant here, but still :) > >Wichert. > > > ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:20 ` Spam 2004-08-26 10:24 ` Andrew Morton 2004-08-26 10:54 ` Wichert Akkerman @ 2004-08-26 11:16 ` Christer Weinigel 2004-08-26 11:35 ` Spam ` (2 more replies) 2 siblings, 3 replies; 1392+ messages in thread From: Christer Weinigel @ 2004-08-26 11:16 UTC (permalink / raw) To: Spam Cc: Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Spam <spam@tnonline.net> writes: > Keeping stuff in the kernel should make the new features > transparent to the applications. No, it will make just one special case, rename within the same filesystem, work. (Well, two special cases, rm will also delete the other forks). Unless we add a new copy(2) syscall (which would be nice for other reasons) all applicatons that copy files will fail to copy the streams. So no working cp command, no nautilus nor konqueror without changes to the application. And if you have to change the applications anyway, isn't it much better to agree on a shared library in userspace that everyone uses? Which has the added bonus that it can be made to work on top of Linux, Windows, Ultrix and VMS? /Christer -- "Just how much can I get away with and still go to heaven?" Freelance consultant specializing in device driver programming for Linux Christer Weinigel <christer@weinigel.se> http://www.weinigel.se ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:16 ` Christer Weinigel @ 2004-08-26 11:35 ` Spam 2004-08-26 11:33 ` Wichert Akkerman 2004-08-26 11:58 ` Jamie Lokier 2004-08-26 14:00 ` Denis Vlasenko 2 siblings, 1 reply; 1392+ messages in thread From: Spam @ 2004-08-26 11:35 UTC (permalink / raw) To: Christer Weinigel Cc: Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > Spam <spam@tnonline.net> writes: >> Keeping stuff in the kernel should make the new features >> transparent to the applications. > No, it will make just one special case, rename within the same > filesystem, work. (Well, two special cases, rm will also delete > the other forks). > Unless we add a new copy(2) syscall (which would be nice for other > reasons) all applicatons that copy files will fail to copy the > streams. How so? The whole idea is that the underlaying OS that handles the copying should also know to copy everything, otherwise you can implement everything into applications and just skip the whole filesystem part. Of course, it won't work if applications themselves are actually transferring the data and create the new files directly onto the filesystem. This just seem so wrong. ~S > So no working cp command, no nautilus nor konqueror without > changes to the application. And if you have to change the > applications anyway, isn't it much better to agree on a shared library > in userspace that everyone uses? Which has the added bonus that it > can be made to work on top of Linux, Windows, Ultrix and VMS? > /Christer ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:35 ` Spam @ 2004-08-26 11:33 ` Wichert Akkerman 2004-08-26 11:43 ` Spam ` (2 more replies) 0 siblings, 3 replies; 1392+ messages in thread From: Wichert Akkerman @ 2004-08-26 11:33 UTC (permalink / raw) To: Spam Cc: Christer Weinigel, Andrew Morton, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Previously Spam wrote: > How so? The whole idea is that the underlaying OS that handles the > copying should also know to copy everything, otherwise you can > implement everything into applications and just skip the whole > filesystem part. UNIX doesn't have a copy systemcall, applications copy the data manually. Wichert. -- Wichert Akkerman <wichert@wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:33 ` Wichert Akkerman @ 2004-08-26 11:43 ` Spam [not found] ` <1093592467.18603.6.camel@tara.firmix.at> 2004-08-26 23:56 ` Hans Reiser 2004-08-27 0:00 ` Stelian Ionescu 2 siblings, 1 reply; 1392+ messages in thread From: Spam @ 2004-08-26 11:43 UTC (permalink / raw) To: Wichert Akkerman Cc: Christer Weinigel, Andrew Morton, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > Previously Spam wrote: >> How so? The whole idea is that the underlaying OS that handles the >> copying should also know to copy everything, otherwise you can >> implement everything into applications and just skip the whole >> filesystem part. > UNIX doesn't have a copy systemcall, applications copy the data > manually. Oh, this is very unfortunate and should be a bigger issue to fix. > Wichert. ^ permalink raw reply [flat|nested] 1392+ messages in thread
[parent not found: <1093592467.18603.6.camel@tara.firmix.at>]
* Re: silent semantic changes with reiser4 [not found] ` <1093592467.18603.6.camel@tara.firmix.at> @ 2004-08-27 8:05 ` Hans Reiser 2004-08-27 8:51 ` Christoph Hellwig 2004-08-27 8:49 ` Christoph Hellwig 1 sibling, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-27 8:05 UTC (permalink / raw) To: Bernd Petrovitsch Cc: Spam, Wichert Akkerman, Christer Weinigel, Andrew Morton, jra, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Bernd Petrovitsch wrote: >On Thu, 2004-08-26 at 13:43 +0200, Spam wrote: > > >>>Previously Spam wrote: >>> >>> >>>> How so? The whole idea is that the underlaying OS that handles the >>>> copying should also know to copy everything, otherwise you can >>>> implement everything into applications and just skip the whole >>>> filesystem part. >>>> >>>> >>>UNIX doesn't have a copy systemcall, applications copy the data >>>manually. >>> >>> >> Oh, this is very unfortunate and should be a bigger issue to fix. >> >> > >Then you have to rewrite POSIX und SuSv3. > > Bernd > > Spam is right. Posix is a standard, not a vision, and the future is always a vision not a standard. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 8:05 ` Hans Reiser @ 2004-08-27 8:51 ` Christoph Hellwig 0 siblings, 0 replies; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-27 8:51 UTC (permalink / raw) To: Hans Reiser Cc: Bernd Petrovitsch, Spam, Wichert Akkerman, Christer Weinigel, Andrew Morton, jra, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, Aug 27, 2004 at 01:05:02AM -0700, Hans Reiser wrote: > Spam is right. Posix is a standard, not a vision, and the future is > always a vision not a standard. An old friend of mine always said "if someone has visions he needs to stop somking mushrooms.." ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 [not found] ` <1093592467.18603.6.camel@tara.firmix.at> 2004-08-27 8:05 ` Hans Reiser @ 2004-08-27 8:49 ` Christoph Hellwig 2004-08-27 9:03 ` Bernd Petrovitsch 1 sibling, 1 reply; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-27 8:49 UTC (permalink / raw) To: Bernd Petrovitsch Cc: Spam, Wichert Akkerman, Christer Weinigel, Andrew Morton, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, Aug 27, 2004 at 09:41:07AM +0200, Bernd Petrovitsch wrote: > > > UNIX doesn't have a copy systemcall, applications copy the data > > > manually. > > > > Oh, this is very unfortunate and should be a bigger issue to fix. > > Then you have to rewrite POSIX und SuSv3. They don't say 'you must now have a copy syscall'. Having one that's actually used by system tools would be a great optimization for many network or distributed filesystems. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 8:49 ` Christoph Hellwig @ 2004-08-27 9:03 ` Bernd Petrovitsch 0 siblings, 0 replies; 1392+ messages in thread From: Bernd Petrovitsch @ 2004-08-27 9:03 UTC (permalink / raw) To: Christoph Hellwig Cc: Spam, Wichert Akkerman, Christer Weinigel, Andrew Morton, jra, torvalds, reiser, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, 2004-08-27 at 10:49 +0200, Christoph Hellwig wrote: > On Fri, Aug 27, 2004 at 09:41:07AM +0200, Bernd Petrovitsch wrote: > > > > UNIX doesn't have a copy systemcall, applications copy the data > > > > manually. > > > > > > Oh, this is very unfortunate and should be a bigger issue to fix. > > > > Then you have to rewrite POSIX und SuSv3. > > They don't say 'you must now have a copy syscall'. Having one that's > actually used by system tools would be a great optimization for many > network or distributed filesystems. ACK. But actually spam apparently assumes that the kernel - userspace interface is at the wrong abstraction level. And this is conceptually bound IMHO to POSIX. And it does not solve the problem, that the read() and write() sys-calls will not vanish. Even if a copy() sys-call exists. Bernd -- Firmix Software GmbH http://www.firmix.at/ mobil: +43 664 4416156 fax: +43 1 7890849-55 Embedded Linux Development and Services ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:33 ` Wichert Akkerman 2004-08-26 11:43 ` Spam @ 2004-08-26 23:56 ` Hans Reiser 2004-08-27 1:05 ` Andreas Dilger 2004-08-27 0:00 ` Stelian Ionescu 2 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 23:56 UTC (permalink / raw) To: Wichert Akkerman Cc: Spam, Christer Weinigel, Andrew Morton, jra, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Wichert Akkerman wrote: >Previously Spam wrote: > > >> How so? The whole idea is that the underlaying OS that handles the >> copying should also know to copy everything, otherwise you can >> implement everything into applications and just skip the whole >> filesystem part. >> >> > >UNIX doesn't have a copy systemcall, applications copy the data >manually. > >Wichert. > > > See sys_reiser4()..... ;-) you can go "A<-B". I have hopes of getting drive vendors to support us in doing single disk copies without even leaving the disk drive..... ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 23:56 ` Hans Reiser @ 2004-08-27 1:05 ` Andreas Dilger 0 siblings, 0 replies; 1392+ messages in thread From: Andreas Dilger @ 2004-08-27 1:05 UTC (permalink / raw) To: Hans Reiser Cc: Wichert Akkerman, Spam, Christer Weinigel, Andrew Morton, jra, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1124 bytes --] On Aug 26, 2004 16:56 -0700, Hans Reiser wrote: > Wichert Akkerman wrote: > >UNIX doesn't have a copy systemcall, applications copy the data > >manually. > > See sys_reiser4()..... ;-) you can go "A<-B". I have hopes of getting > drive vendors to support us in doing single disk copies without even > leaving the disk drive..... Yes, we've wanted to have a "copy" syscall for a while now also, so that Lustre (or other network FS for that matter) can do remote copies of large files without having to go through the VM of the client. The work on sendfile() in 2.6 is a step in that direction, I eagerly await the day when "cp" will try to do a sys_sendfile() call first, before doing the read+write looping it does now. Consider for a network filesystem we are limited to 1/2 of the client network bandwidth (say 50-75MB/s) for copying instead of the server-local or multiple server-server bandwidth for copies (400-600MB/s or more). Cheers, Andreas -- Andreas Dilger http://sourceforge.net/projects/ext2resize/ http://members.shaw.ca/adilger/ http://members.shaw.ca/golinux/ [-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:33 ` Wichert Akkerman 2004-08-26 11:43 ` Spam 2004-08-26 23:56 ` Hans Reiser @ 2004-08-27 0:00 ` Stelian Ionescu 2 siblings, 0 replies; 1392+ messages in thread From: Stelian Ionescu @ 2004-08-27 0:00 UTC (permalink / raw) To: Wichert Akkerman Cc: Andrew Morton, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 01:33:32PM +0200, Wichert Akkerman wrote: >Previously Spam wrote: >> How so? The whole idea is that the underlaying OS that handles the >> copying should also know to copy everything, otherwise you can >> implement everything into applications and just skip the whole >> filesystem part. > >UNIX doesn't have a copy systemcall, applications copy the data >manually. couldn't sendfile(2) be used for that ? perhaps a flag parameter could be added to sendfile(2) which specifies whether to copy _all_ metadata if both fd's correspond to files, something like either O_METADATA or O_NOMETADATA -- Stelian Ionescu aka fe[nl]ix Quidquid latine dictum sit, altum viditur. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:16 ` Christer Weinigel 2004-08-26 11:35 ` Spam @ 2004-08-26 11:58 ` Jamie Lokier 2004-08-26 14:00 ` Denis Vlasenko 2 siblings, 0 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 11:58 UTC (permalink / raw) To: Christer Weinigel Cc: Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Christer Weinigel wrote: > all applicatons that copy files will fail to copy the streams. So > no working cp command, no nautilus nor konqueror without changes to > the application. No true. A lot of the fancy stuff depends on metadata which is merely another view of the serialised contents in the flat file. Nautilus & Konqueror thumbnails are an example of that (I gave a few more in other mails). cp works fine with that, and so does ftp. The metadata is just recalculated when it's needed from the target file. > And if you have to change the applications anyway, isn't it much > better to agree on a shared library in userspace that everyone uses? > Which has the added bonus that it can be made to work on top of > Linux, Windows, Ultrix and VMS? Ideally yes, a shared library _or_ at least an agreed representation. However, with filesystem support you can improve performance by avoiding unneed serialisation (write to your huge *Office presentation, read it from another program, no need to wait for the slow tar+compress stage yet it's _as if_ those were done), retain computed metadata with coherency guarantees (e.g. real time search indexes, crypto digests and such), and let the filesystem decide when best to prune computed metadata or convert representations. All of these things can work with a combination of userspace plugins (not the same as reiser4 plugins), and filesystem support. Without the filesystem support, you can still use the userspace plugins -- so apps would still be portable -- but you don't get the extras mentioned above. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 11:16 ` Christer Weinigel 2004-08-26 11:35 ` Spam 2004-08-26 11:58 ` Jamie Lokier @ 2004-08-26 14:00 ` Denis Vlasenko 2004-08-26 14:12 ` Rik van Riel 2004-08-26 14:42 ` Felipe Alfaro Solana 2 siblings, 2 replies; 1392+ messages in thread From: Denis Vlasenko @ 2004-08-26 14:00 UTC (permalink / raw) To: Christer Weinigel, Spam Cc: Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thursday 26 August 2004 14:16, Christer Weinigel wrote: > Spam <spam@tnonline.net> writes: > > Keeping stuff in the kernel should make the new features > > transparent to the applications. > > No, it will make just one special case, rename within the same > filesystem, work. (Well, two special cases, rm will also delete > the other forks). In traditional UNIX filesystems, copy loses some info too (owner,group,mode bits). cp and tar was taught about this and they now can preserve that info. However, I do not think that hacking on cp, tar, etc everytime we need to add something to our filesystems (xattrs, ACLs, whatever) is a terribly attractive idea. I think reiserfs is going to handle that by giving uniform access to such ancillary data. It will be visible as file/meta/* filelets ('small files') or something like that and can be saved/restored by e.g. tar *without* needing to know what data is stored there, and in what format. I think Hans is not planning turning old "file is a stream of bytes" into eight-stream octopus. One stream will remain as a 'main' one, which contains actual data. Others will keep metadata, etc... Note that even today's files aren't simple "streams of bytes" because they also have names,mode bits etc. -- vda ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 14:00 ` Denis Vlasenko @ 2004-08-26 14:12 ` Rik van Riel 2004-08-26 14:24 ` Christophe Saout 2004-08-26 15:19 ` Denis Vlasenko 2004-08-26 14:42 ` Felipe Alfaro Solana 1 sibling, 2 replies; 1392+ messages in thread From: Rik van Riel @ 2004-08-26 14:12 UTC (permalink / raw) To: Denis Vlasenko Cc: Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Denis Vlasenko wrote: > I think Hans is not planning turning old "file is a stream of bytes" > into eight-stream octopus. One stream will remain as a 'main' one, > which contains actual data. Others will keep metadata, etc... This is exactly what the Samba people want, though. Office suites can store a document with embedded images and spread sheets "easily" by putting the text, the images and spread sheets all in different file streams. -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 14:12 ` Rik van Riel @ 2004-08-26 14:24 ` Christophe Saout 2004-08-26 15:44 ` Jamie Lokier 2004-08-26 15:19 ` Denis Vlasenko 1 sibling, 1 reply; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 14:24 UTC (permalink / raw) To: Rik van Riel Cc: Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1259 bytes --] Am Donnerstag, den 26.08.2004, 10:12 -0400 schrieb Rik van Riel: > > I think Hans is not planning turning old "file is a stream of bytes" > > into eight-stream octopus. One stream will remain as a 'main' one, > > which contains actual data. Others will keep metadata, etc... > > This is exactly what the Samba people want, though. > > Office suites can store a document with embedded images > and spread sheets "easily" by putting the text, the > images and spread sheets all in different file streams. Ouch. I think Hans' idea is (I don't know if it is a good idea nor if it is doable, but at least it sounds interesting) to have special compound files where you can do something like this: cp text.txt test.compound/test.txt cp image.jpg test.compound/image.jpg And if you read test.compound (the main stream) you get a special format that contains all the components. You can copy that single stream of bytes to another (reiser4) fs and then access test.compound/test.txt again. The only thing I'm worrying about with this approach is what happens if someone tries to simultaneously open test.compound and test.compound/test.txt. Hans has an example somewhere where he does something like that with /etc/passwd. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 14:24 ` Christophe Saout @ 2004-08-26 15:44 ` Jamie Lokier 2004-08-26 15:54 ` Rik van Riel 0 siblings, 1 reply; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 15:44 UTC (permalink / raw) To: Christophe Saout Cc: Rik van Riel, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Christophe Saout wrote: > I think Hans' idea is (I don't know if it is a good idea nor if it is > doable, but at least it sounds interesting) to have special compound > files where you can do something like this: > > cp text.txt test.compound/test.txt > cp image.jpg test.compound/image.jpg > > And if you read test.compound (the main stream) you get a special format > that contains all the components. You can copy that single stream of > bytes to another (reiser4) fs and then access test.compound/test.txt > again. (To Rik especially), this is the design which more or less satisfies lots of different goals at once. I wrote: > Properly implemented metadata can: > > (1) operate in both modes simultaneously; > (2) work with unaware applications; > (3) provide performance enhancements to aware applications; > (4) provide storage enhancements to both; > (5) provide useful features that work with standard unmodified unix tools, (1) is by design. You can copy these "files" safely using cp or cat or ftp. You can email them. They're flat files. That's (2). You can poke around inside them. Applications which do poke around inside get performance gains, because they don't have to serialise the flat format every time they "load" or "save" a file. For example, if you have a huge OpenOffice document containing lots of sub-documents and pictures. We can imagine a version of OpenOffice which just opens the sub-documents, and writes the ones which are changed after editing. Also, another program such as a viewer, post-processor or search indexing program only needs to read the parts which have changed; it doesn't have to decode the whole thing after a change. That's (3). We already know how useful this is with big mailboxes, hence people switching to maildir. Now imagine the attachments in each message are also viewable, cat'able etc., and that the filesystem stores each message compressed automatically, or in aggregate (your choice) and that you can use both mbox and maildir tools on the same mailbox. That's (4). You can ls, cp, grep, less, vi, gqview etc. the pieces of files, and every supported file format needs just one tool to provide the interface to its contents. That's (5). While I'm here I might as well add: (6) improve support for real time search indexing. The "view coherency" mechanism which is needed for efficient userspace VFS support has just the right properties to keep informing an indexer of changes. They could even enforce operation orderings if we cared about _perfect_ real-timeness, although we probably don't. (7) yet another way of doing distros. Why not just install .tgz or .zip files of entire binary packages, and let the filesystem unpack the bits that are needed when they're needed. Some platforms (i.e. Java) do this already with .jar files and it's quite popular. This would extend it in a versatile way. Rik, I hope I've explained how those 5 properties are satisfied by Hans Reiser's brilliant design (ahem, great minds think alike, and remember Linus likes files-as-directories too :). But if I haven't please do pick holes. > The only thing I'm worrying about with this approach is what happens if > someone tries to simultaneously open test.compound and > test.compound/test.txt. It is possible to implement it so that every change to one file immediately affects the other in the appropriate way, _as if_ the other file is regenerated every time before reading it. This is what I mean by coherent views. It can be done even when userspace is doing the work, and the obvious mechanisms just happen to have the property of being useful in other ways (like a better dnotify, but I digress...). They're just an ordinary cache coherency protocols. However, if you are writing a few bytes to test.compound, then the coherent view in test.compound/test.txt is probably a corrupt or non-existent file, because test.compound's intermediate state is a truncated and/or corrupt archive file, until you've written some more. That doesn't look nice, but it's exactly the same problem as you get when you're writing to a file that another program is reading. If you do that on a live http server, for example editing a .html file, very occasionally someone will see a truncated file. Even if your editor uses renames to update atomically, sometimes you need to update more than one file atomically. It can be done, but rarely is. Is there a solution to this? The obvious one is when you're writing to test.compound, don't read from test.compound/test.txt. That's _no different_ from the advice which says when you're saving a file from OpenOffice, don't load it into a document viewer during the time you're writing it, and don't serve it. The other advice is to save to a different file and call rename(). Again, you get pretty much the same results as now: it atomically changes, and currently open references to the old contents will continue to reference the old contents. There's another solution: transactions. reiser4 offers some kind of transaction facility, doesn't it? This is a fine example of when might want to declare that a sequence of writes needs to be a single transaction, visible only when the whole sequence is complete. Perfect! I don't know reiser4's interface to that, but it would make a lot of sense to have a simple shell tool called "atomic", to wrap a sequence of operations. That would be useful for writing files _and_ useful for those pesky "update several files on the web server at once" moments. (Yes I know you can build a hard link tree and rename the root directory, although even that doesn't always work). -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 15:44 ` Jamie Lokier @ 2004-08-26 15:54 ` Rik van Riel 2004-08-26 16:04 ` Christophe Saout ` (2 more replies) 0 siblings, 3 replies; 1392+ messages in thread From: Rik van Riel @ 2004-08-26 15:54 UTC (permalink / raw) To: Jamie Lokier Cc: Christophe Saout, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Jamie Lokier wrote: > Christophe Saout wrote: > > And if you read test.compound (the main stream) you get a special format > > that contains all the components. You can copy that single stream of > > bytes to another (reiser4) fs and then access test.compound/test.txt > > again. > > (To Rik especially), this is the design which more or less satisfies > lots of different goals at once. And if an unaware application reads the compound file and then writes it out again, does the filesystem interpret the contents and create the other streams ? Unless I overlook something (please tell me what), the scheme just proposed requires filesystems to look at the content of files that is being written out, in order to make the streams work. -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 15:54 ` Rik van Riel @ 2004-08-26 16:04 ` Christophe Saout 2004-08-26 16:07 ` Christoph Hellwig ` (2 more replies) 2004-08-26 16:53 ` Jamie Lokier 2004-08-26 17:05 ` Diego Calleja 2 siblings, 3 replies; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 16:04 UTC (permalink / raw) To: Rik van Riel Cc: Jamie Lokier, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1291 bytes --] Am Donnerstag, den 26.08.2004, 11:54 -0400 schrieb Rik van Riel: > > > And if you read test.compound (the main stream) you get a special format > > > that contains all the components. You can copy that single stream of > > > bytes to another (reiser4) fs and then access test.compound/test.txt > > > again. > > > > (To Rik especially), this is the design which more or less satisfies > > lots of different goals at once. > > And if an unaware application reads the compound file > and then writes it out again, does the filesystem > interpret the contents and create the other streams ? > > Unless I overlook something (please tell me what), the > scheme just proposed requires filesystems to look at > the content of files that is being written out, in > order to make the streams work. Yes, the main compound stream unaware applications would see would just be a writable view of its contents. Probably not trivial to implement but doable. It should be a simple format. Something simliar to tar. The worst thing that can happen is people start writing plugins for every existing compound format out there. It should be the other way around, agree on a simple compound format and encourage applications to use this one if the want to use this advantage. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:04 ` Christophe Saout @ 2004-08-26 16:07 ` Christoph Hellwig 2004-08-26 16:44 ` Jamie Lokier 2004-08-26 18:40 ` David Masover 2004-08-26 21:05 ` David Lang 2 siblings, 1 reply; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-26 16:07 UTC (permalink / raw) To: Christophe Saout Cc: Rik van Riel, Jamie Lokier, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 06:04:42PM +0200, Christophe Saout wrote: > It should be a simple format. Something simliar to tar. The worst thing > that can happen is people start writing plugins for every existing > compound format out there. It should be the other way around, agree on a > simple compound format and encourage applications to use this one if the > want to use this advantage. or placing this into a userspace helper that the kernel can invoke transparently. If you're doing funny thing like that for your swap device deadlocks are the last of your worries :) ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:07 ` Christoph Hellwig @ 2004-08-26 16:44 ` Jamie Lokier 0 siblings, 0 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 16:44 UTC (permalink / raw) To: Christoph Hellwig, Christophe Saout, Rik van Riel, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, linux-fsdevel, linux-kernel, flx, reiserfs-list Christoph Hellwig wrote: > or placing this into a userspace helper that the kernel can invoke > transparently. Yes, exactly this for most file formats. Ideally, with _some_ user control over the handlers for their own files a bit like Hurd. Not just archive files: metadata pulled from audio files, images, distro packages, etc. In other words much the same as Gnome and KDE's things, but with more emphasis on metadata extractors and efficient on-disk cache interaction, and a sane page-cache coherency interface (possibly FUSE). Metadata extraction is useful for search and indexing purposes. (That, and a way to hook change notifications up with indexers -- again comes with any proper coherency interface). For new applications using a container, it's tempting to want one format which is optimised for this. But really many applications want to write XML or compressed XML, or something else that isn't especially efficient but is easy to work with on many OSes. We should support applications whatever format they need to work with, if providing an interior view as a directory is useful for them. (This implies applications can introduce their own formats for files they need to operate on.) By the way, lazy updating of containers is an even bigger advantage if you have to use XML as the flat format, because the savings from not serialising when you don't need to are more pronounced. For some applications the "content" isn't structured in a way which maps naturally to a plain archive file. Hans' /etc/passwd is a simple example: at least one of the "flat" representations, when there are enough entries, probably ought to be a DB file for faster lookup. (Though, it would be great if /etc/passwd could automagically get fast lookup due to an automagic underlying representation, while still appearing to be a flat file when you cat it). Also, imagine "cd" into a MySQL table file. It might just work. By the way, we don't need containers to waste huge amounts of space like the Microsoft Word "fast save" documents did. Because our containers are created on demand, there is no performance advantage in them having unused holes, unlike the case were Word was saving changes periodically. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:04 ` Christophe Saout 2004-08-26 16:07 ` Christoph Hellwig @ 2004-08-26 18:40 ` David Masover 2004-08-26 21:17 ` David Lang 2004-08-26 21:05 ` David Lang 2 siblings, 1 reply; 1392+ messages in thread From: David Masover @ 2004-08-26 18:40 UTC (permalink / raw) To: Christophe Saout Cc: Rik van Riel, Jamie Lokier, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Christophe Saout wrote: | Yes, the main compound stream unaware applications would see would just | be a writable view of its contents. Probably not trivial to implement | but doable. Not the main stream, I think, but we do need a way to make "eight-stream octopus" files portable. But there needs to be a copy(2) syscall, and apps need to start supporting it. After all, copy-on-write was planned for Reiser4, now 4.1*. And don't anyone dare suggest "userland library". | It should be a simple format. Something simliar to tar. The worst thing Simple, yes. Similar to tar, no. How about a serializer plugin? foo.mp3/serialize (say) should return some single stream which contains all of foo.mp3. It could work for directories as well, recursively, at least on the same filesystem -- you'd back up by doing cp /serialize /backup/20040826 This has two potential problems. One is security -- I want to be able to serialize foo.mp3 without being root, but root doesn't want me to be able to serialize /etc and thus get access to shadow. The second problem is that in order to actually be used for backup, we need snapshots, which are not done yet*. Currently, it'd have to lock the file/directory (recursively), which you don't want to do to entire live filesystems... So, no massive backups of live systems yet, but definitely useful for sending around mp3s and such. | that can happen is people start writing plugins for every existing | compound format out there. It should be the other way around, agree on a | simple compound format and encourage applications to use this one if the | want to use this advantage. For the format, I vote for whatever format we're using at the storage layer. It'd be a lot faster that way, both in execution time (I think) and in development time (I'm sure). Plus, imagine the restore: cat /backup/20040826 > /dev/hda5 resizefs_reiser4 /dev/hda5 * Disclaimer -- all this is "I think, last I checked". -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQS4uhngHNmZLgCUhAQKTaw//WCHt0Umgaf0GaMUpEr0PLhiXhFTYQqKV MKgpVt6SjP7lC9k3RbuCnPCLyOJGb9lIqo+Wd+B3b4TcQAQd4OYsSI6WiMPQjxXY OdNnB7/xYkkd/ULw7/7IGcokyc4FRDjhTyOMe4KscQS/6uhdX1dCriCCzVq7iELx u3HLulunHv1bwHbnZ6CyEZdQEr3j8XUu62+IXPTc6hWnfRRCvskV8Y4qyyVd8wFY Kp44TxbJJSzDeWmuI/YYkxVJsPmSoxRLxmw9/B8D72KGNg1SYLnM4ueIc5IV8RH0 kG/o8tzwTRxE/eqfcHP+UBt6yXZR/i84DDogAmJFKP73g9X+U+J5o/kONOBQODhY qY5LUwQu4UjlzXttMxC4NmZPESdUw85DgC/GhTQlsBMYBcmQHmpLAK67qt83+Kg0 1QqGPqJm4HL4weHLNknJ0DukLNKH7Bw5CVg335IwruaoiKUG7HDgthUhA319FCab VvYa+bEfeBEw9RorSE/gWM9Hg73luM7eoSzKc9zHgx++cACTi8SBsTFWqI2FU6DX UyWG5sNPybOiczpPsFu5S5FJBBNM3fLjkfeYgfqxUIjxGTNeUn/g/9I5ShUrCF1o T+3nd1jFxfoehYGkfGB2YLpdeNVBv4EdKK16j39+iducfJrXnRlL1yjdo/Kte2uz tTQZw3uTnhU= =7bBF -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:40 ` David Masover @ 2004-08-26 21:17 ` David Lang 0 siblings, 0 replies; 1392+ messages in thread From: David Lang @ 2004-08-26 21:17 UTC (permalink / raw) To: David Masover Cc: Christophe Saout, Rik van Riel, Jamie Lokier, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, David Masover wrote: > Christophe Saout wrote: > > | Yes, the main compound stream unaware applications would see would just > | be a writable view of its contents. Probably not trivial to implement > | but doable. > > Not the main stream, I think, but we do need a way to make "eight-stream > octopus" files portable. But there needs to be a copy(2) syscall, and > apps need to start supporting it. After all, copy-on-write was planned > for Reiser4, now 4.1*. And don't anyone dare suggest "userland library". > > | It should be a simple format. Something simliar to tar. The worst thing > > Simple, yes. Similar to tar, no. > > How about a serializer plugin? foo.mp3/serialize (say) should return > some single stream which contains all of foo.mp3. It could work for > directories as well, recursively, at least on the same filesystem -- > you'd back up by doing > > cp /serialize /backup/20040826 > > This has two potential problems. One is security -- I want to be able > to serialize foo.mp3 without being root, but root doesn't want me to be > able to serialize /etc and thus get access to shadow. > > The second problem is that in order to actually be used for backup, we > need snapshots, which are not done yet*. Currently, it'd have to lock > the file/directory (recursively), which you don't want to do to entire > live filesystems... So, no massive backups of live systems yet, but > definitely useful for sending around mp3s and such. > > | that can happen is people start writing plugins for every existing > | compound format out there. It should be the other way around, agree on a > | simple compound format and encourage applications to use this one if the > | want to use this advantage. > > For the format, I vote for whatever format we're using at the storage > layer. It'd be a lot faster that way, both in execution time (I think) > and in development time (I'm sure). Plus, imagine the restore: > > cat /backup/20040826 > /dev/hda5 > resizefs_reiser4 /dev/hda5 > one of the virtual metatags should be something about the method to serialize the data if you are viewing a tarfile as a directory the VFS/Filesystem shoud know that the link between the whole thing and the contents is tar, and it should be possible for userspace programs to find this out by reading teh appropriate thing inside the virtual directory (and eventually change it as well, which would trigger regeneration of the base object, or at least noting that the current version is wrong and needs to be regenerated) David Lang -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:04 ` Christophe Saout 2004-08-26 16:07 ` Christoph Hellwig 2004-08-26 18:40 ` David Masover @ 2004-08-26 21:05 ` David Lang 2004-08-26 21:19 ` Jamie Lokier 2004-08-26 22:30 ` Felipe Alfaro Solana 2 siblings, 2 replies; 1392+ messages in thread From: David Lang @ 2004-08-26 21:05 UTC (permalink / raw) To: Christophe Saout Cc: Rik van Riel, Jamie Lokier, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Christophe Saout wrote: > Am Donnerstag, den 26.08.2004, 11:54 -0400 schrieb Rik van Riel: > >>>> And if you read test.compound (the main stream) you get a special format >>>> that contains all the components. You can copy that single stream of >>>> bytes to another (reiser4) fs and then access test.compound/test.txt >>>> again. >>> >>> (To Rik especially), this is the design which more or less satisfies >>> lots of different goals at once. >> >> And if an unaware application reads the compound file >> and then writes it out again, does the filesystem >> interpret the contents and create the other streams ? >> >> Unless I overlook something (please tell me what), the >> scheme just proposed requires filesystems to look at >> the content of files that is being written out, in >> order to make the streams work. > > Yes, the main compound stream unaware applications would see would just > be a writable view of its contents. Probably not trivial to implement > but doable. > > It should be a simple format. Something simliar to tar. The worst thing > that can happen is people start writing plugins for every existing > compound format out there. It should be the other way around, agree on a > simple compound format and encourage applications to use this one if the > want to use this advantage. > I also don't see why the VFS/Filesystem can't decide that (for example) this tar.gz is so active that instead of storing it as a tar.gz and providing a virtual directory of the contents that it instead stores the directory of the contents and makes the tar.gz virtual (regenerating it as needed or as extra system resources are available) implementation wise I see headaches in doing this, but conceptually this is just an optimization that could take place in the future if we fine that it's needed. David Lang -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 21:05 ` David Lang @ 2004-08-26 21:19 ` Jamie Lokier 2004-08-26 21:31 ` Linus Torvalds 2004-08-26 22:30 ` Felipe Alfaro Solana 1 sibling, 1 reply; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 21:19 UTC (permalink / raw) To: David Lang Cc: Christophe Saout, Rik van Riel, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list David Lang wrote: > I also don't see why the VFS/Filesystem can't decide that (for example) > this tar.gz is so active that instead of storing it as a tar.gz and > providing a virtual directory of the contents that it instead stores the > directory of the contents and makes the tar.gz virtual (regenerating it as > needed or as extra system resources are available) Absolutely. It could keep both views, if they're both actively being used. Or more than both, if there are more. (You could think of compression being an alternate view, and both compressed and uncompressed may as well remain on disk if there's space and it's being actively accessed. > implementation wise I see headaches in doing this, but conceptually this > is just an optimization that could take place in the future if we fine > that it's needed. untarring a big source tree takes ages. I wouldn't like to do it after every reboot, if there was a .tar.bz2 tree I looked at often. That's why I have things like glibc untarred in my home directory. With good cacheing, I could cd into the .tar.bz2 files and _effectively_ have the performance of untarred source trees for the ones I look at often on my disk -- automatically cleaned if the space if needed for something else, too. It would be quite nice. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 21:19 ` Jamie Lokier @ 2004-08-26 21:31 ` Linus Torvalds 0 siblings, 0 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-08-26 21:31 UTC (permalink / raw) To: Jamie Lokier Cc: David Lang, Christophe Saout, Rik van Riel, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Jamie Lokier wrote: > > With good cacheing, I could cd into the .tar.bz2 files and > _effectively_ have the performance of untarred source trees for the > ones I look at often on my disk -- automatically cleaned if the space > if needed for something else, too. It would be quite nice. What you're talking about is just a caching filesystem. Then you can have any deamon do whatever it wants to fill it - whether from tar-files or over the network or anything else. I think you should ask David Howells about it.. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 21:05 ` David Lang 2004-08-26 21:19 ` Jamie Lokier @ 2004-08-26 22:30 ` Felipe Alfaro Solana 2004-08-26 22:32 ` Chris Dukes 2004-08-26 22:52 ` David Lang 1 sibling, 2 replies; 1392+ messages in thread From: Felipe Alfaro Solana @ 2004-08-26 22:30 UTC (permalink / raw) To: David Lang Cc: Christophe Saout, Rik van Riel, Jamie Lokier, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thursday 26 August 2004 23:05, David Lang wrote: > I also don't see why the VFS/Filesystem can't decide that (for example) > this tar.gz is so active that instead of storing it as a tar.gz and > providing a virtual directory of the contents that it instead stores the > directory of the contents and makes the tar.gz virtual (regenerating it as > needed or as extra system resources are available) Because that would mean the kernel should "talk" the tar format, which is, IMHO, a Bad Idea (TM). Maybe the kernel could notify a user-space daemon to perform this task, instead. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 22:30 ` Felipe Alfaro Solana @ 2004-08-26 22:32 ` Chris Dukes 2004-08-26 22:52 ` David Lang 1 sibling, 0 replies; 1392+ messages in thread From: Chris Dukes @ 2004-08-26 22:32 UTC (permalink / raw) To: Felipe Alfaro Solana Cc: David Lang, Christophe Saout, Rik van Riel, Jamie Lokier, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, Aug 27, 2004 at 12:30:19AM +0200, Felipe Alfaro Solana wrote: > On Thursday 26 August 2004 23:05, David Lang wrote: > > > I also don't see why the VFS/Filesystem can't decide that (for example) > > this tar.gz is so active that instead of storing it as a tar.gz and > > providing a virtual directory of the contents that it instead stores the > > directory of the contents and makes the tar.gz virtual (regenerating it as > > needed or as extra system resources are available) > > Because that would mean the kernel should "talk" the tar format, which is, > IMHO, a Bad Idea (TM). Maybe the kernel could notify a user-space daemon to > perform this task, instead. Wasn't HURD supposed to do this for us? -- Chris Dukes Warning: Do not use the reflow toaster oven to prepare foods after it has been used for solder paste reflow. http://www.stencilsunlimited.com/stencil_article_page5.htm ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 22:30 ` Felipe Alfaro Solana 2004-08-26 22:32 ` Chris Dukes @ 2004-08-26 22:52 ` David Lang 1 sibling, 0 replies; 1392+ messages in thread From: David Lang @ 2004-08-26 22:52 UTC (permalink / raw) To: Felipe Alfaro Solana Cc: Christophe Saout, Rik van Riel, Jamie Lokier, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, 27 Aug 2004, Felipe Alfaro Solana wrote: > On Thursday 26 August 2004 23:05, David Lang wrote: > >> I also don't see why the VFS/Filesystem can't decide that (for example) >> this tar.gz is so active that instead of storing it as a tar.gz and >> providing a virtual directory of the contents that it instead stores the >> directory of the contents and makes the tar.gz virtual (regenerating it as >> needed or as extra system resources are available) > > Because that would mean the kernel should "talk" the tar format, which is, > IMHO, a Bad Idea (TM). Maybe the kernel could notify a user-space daemon to > perform this task, instead. > the kernel will definantly need the ability to use user-space code to do the transformations from one version to the other (if nothing else think of the thumbnail version of images, we don't want the image manipulation code in the kernel and we definantly want this sort of option available) the interesting issue is going to be defining the kernel->user-space interface for doing the extractions. David Lang -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 15:54 ` Rik van Riel 2004-08-26 16:04 ` Christophe Saout @ 2004-08-26 16:53 ` Jamie Lokier 2004-08-26 19:32 ` John Stoffel 2004-08-30 17:37 ` Alex Zarochentsev 2004-08-26 17:05 ` Diego Calleja 2 siblings, 2 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 16:53 UTC (permalink / raw) To: Rik van Riel Cc: Christophe Saout, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Rik van Riel wrote: > And if an unaware application reads the compound file > and then writes it out again, does the filesystem > interpret the contents and create the other streams ? Yes, exactly that. The streams are created on demand of course, and by userspace helpers when that's appropriate which I suspect it almost always is. > Unless I overlook something (please tell me what), the > scheme just proposed requires filesystems to look at > the content of files that is being written out, in > order to make the streams work. Yes. Hence the idea of coherent views between two files: writing to one affects the content of the other, although the calcalation is only done on demand (or when the fs wants to migrate the representation -- for example, creating the flat container prior to deleting the regeneratable pieces in order to save space). I haven't seen anything from Namesys that says they'll do that. I have the impression the streams are just generated in memory on the fly, not stored on disk with a cacheing policy, but that's just an impression. (We've all seen the Namesys white papers, they're not _that_ revealing). :) I'm just pointing out how to do it _right_. I think it will turn out like this eventually, either next year or some time over the next decade after some iterations. Inevitable. Mark my words, etc. :) -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:53 ` Jamie Lokier @ 2004-08-26 19:32 ` John Stoffel 2004-08-26 19:51 ` Denis Vlasenko 2004-08-26 20:13 ` Jamie Lokier 2004-08-30 17:37 ` Alex Zarochentsev 1 sibling, 2 replies; 1392+ messages in thread From: John Stoffel @ 2004-08-26 19:32 UTC (permalink / raw) To: Jamie Lokier Cc: Rik van Riel, Christophe Saout, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Jamie> Rik van Riel wrote: >> And if an unaware application reads the compound file >> and then writes it out again, does the filesystem >> interpret the contents and create the other streams ? Jamie> Yes, exactly that. The streams are created on demand of Jamie> course, and by userspace helpers when that's appropriate which Jamie> I suspect it almost always is. So how would a program that converts between a JPEG file (with exif data) and a PNG work, such as ImageMagick? Are we proposing to teach the VFS (or worse yet each filesystem) how to do this? I've been following this discussion a bit and I'm not sure that I've actually seen any concrete examples of where this is a *good* thing to have. People talk about only having to modify 20 bytes at a time instead of reading and writing 1mb of data. Isn't that what mmap() does? Now I can sorta understand the idea that having a directory look like a file is neat, and certainly simplifies some aspects, but I think that going all the way down to the logical conclusion here is a bit silly. To use the principle of blowing things up to make them very large or very small, what happens if I decide that the best idea is to make all files just be directories which contain single byte files? Isn't that the logical extreme here? So my 1mb JPEG file is not just some image data and header info in multiple files, but it's really just 1 million (ok 1024 * 1024) individual files that the VFS knows how to put together. Seems like the logical extreme. Oh wait, maybe we should be exposing a single file per bit instead! >From my point of view, there lies madness. As Rik pointed out, how do backup and restore tools work with this stuff? Most people could care less about how their data is organized, but they certainly care when they can't restore it from backups. I'd really like to see a concrete example from Hans or other proponents about why this makes things easier/faster/better to do. Mostly, I've just seen "proof by vigorous handwaving" that it's a good thing. In alot of ways, I think people are going in the wrong direction, you want to excapsulate and hide the details more, not expose them. That's what a good API does, it hides the details while giving you a rich set of semantics to manage your data. God knows I'm not smart enough or driven enough to actually come up with my own ideas, but I certainly haven't seen anyone else (even Linus) come up with an earth shattering arguement to say why this is the right move to make. As Linus says, most of the OS's job is to mediate access to objects/data. Why do we want to expose such low level data then? John John Stoffel - Senior Unix Systems Administrator - Lucent Technologies stoffel@lucent.com - http://www.lucent.com - 978-952-7548 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 19:32 ` John Stoffel @ 2004-08-26 19:51 ` Denis Vlasenko 2004-08-26 20:13 ` Jamie Lokier 1 sibling, 0 replies; 1392+ messages in thread From: Denis Vlasenko @ 2004-08-26 19:51 UTC (permalink / raw) To: John Stoffel, Jamie Lokier Cc: Rik van Riel, Christophe Saout, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thursday 26 August 2004 22:32, John Stoffel wrote: > Jamie> Rik van Riel wrote: > >> And if an unaware application reads the compound file > >> and then writes it out again, does the filesystem > >> interpret the contents and create the other streams ? > > Jamie> Yes, exactly that. The streams are created on demand of > Jamie> course, and by userspace helpers when that's appropriate which > Jamie> I suspect it almost always is. > > So how would a program that converts between a JPEG file (with exif > data) and a PNG work, such as ImageMagick? Are we proposing to teach > the VFS (or worse yet each filesystem) how to do this? > > I've been following this discussion a bit and I'm not sure that I've > actually seen any concrete examples of where this is a *good* thing to > have. People talk about only having to modify 20 bytes at a time > instead of reading and writing 1mb of data. Isn't that what mmap() > does? > > Now I can sorta understand the idea that having a directory look like > a file is neat, and certainly simplifies some aspects, but I think > that going all the way down to the logical conclusion here is a bit > silly. > > To use the principle of blowing things up to make them very large or > very small, what happens if I decide that the best idea is to make all > files just be directories which contain single byte files? Isn't that > the logical extreme here? So my 1mb JPEG file is not just some image > data and header info in multiple files, but it's really just 1 > million (ok 1024 * 1024) individual files that the VFS knows how to > put together. Seems like the logical extreme. Oh wait, maybe we > should be exposing a single file per bit instead! It is doable. It doesn't mean we shall do this. I think some reiser4 supporters try to do too much too soon. It is more sensible to do it in small steps. I think we can start small and make all file metadata to be accessible via file/meta/{uid,gid,mode,mtime,atime,...}. Normal tools will be unable to see file/meta/ because file is not a directory and file/ is not a directory (i.e. open(O_DIRECTORY) will fail on them). Hardlinking to file/meta/* is not allowed. However, file/. _is_ a directory. We can make a tar-like tool which can do its work and save/restore metadata along with data by just tarring/untarring file/meta/*. Notice how this 'new tar' does not need to know about exact kind of metadata supported by fs. You can add ACLs to the fs, and this 'new tar' will be able to save/restore ACLs. Without any modufications. Contrast this with existing tar which has hardcoded knowlendge about uid,gid,mode,etc... -- vda ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 19:32 ` John Stoffel 2004-08-26 19:51 ` Denis Vlasenko @ 2004-08-26 20:13 ` Jamie Lokier 1 sibling, 0 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 20:13 UTC (permalink / raw) To: John Stoffel Cc: Rik van Riel, Christophe Saout, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list John Stoffel wrote: > Jamie> Yes, exactly that. The streams are created on demand of > Jamie> course, and by userspace helpers when that's appropriate which > Jamie> I suspect it almost always is. > > So how would a program that converts between a JPEG file (with exif > data) and a PNG work, such as ImageMagick? Are we proposing to teach > the VFS (or worse yet each filesystem) how to do this? No. (1) That's what "userspace helpers" are for. (2) What does image format conversion have to do with viewing files in their component parts? (3) I suspect someone will write a plugin that does indeed convert virtually every known image format to a common format (probably PNG and something "raw") at some point. Why not? It's just a small script (it would run ImageMagick!). > I've been following this discussion a bit and I'm not sure that I've > actually seen any concrete examples of where this is a *good* thing to > have. People talk about only having to modify 20 bytes at a time > instead of reading and writing 1mb of data. Isn't that what mmap() > does? Sure, if you think mmap() is an easy substitute for "parse the author & license tags from this [unknown format] audio, video or font file". > Now I can sorta understand the idea that having a directory look like > a file is neat, and certainly simplifies some aspects, but I think > that going all the way down to the logical conclusion here is a bit > silly. > > To use the principle of blowing things up to make them very large or > very small, what happens if I decide that the best idea is to make all > files just be directories which contain single byte files? Isn't that > the logical extreme here? So my 1mb JPEG file is not just some image > data and header info in multiple files, but it's really just 1 > million (ok 1024 * 1024) individual files that the VFS knows how to > put together. Seems like the logical extreme. Oh wait, maybe we > should be exposing a single file per bit instead! If you decide to do that, you are welcome to write the userspace helper which creates that view in your directory. Why not? It's very silly, but it's no sillier then writing a program to read your 1mb JPEG and write it out one file per bit, which you're welcome to write right now. > >From my point of view, there lies madness. As Rik pointed out, how do > backup and restore tools work with this stuff? Most people could care > less about how their data is organized, but they certainly care when > they can't restore it from backups. Generated views are something which should _not_ be backed up and restored. That's not what they are for. Auxiliary metadata, such as author info and signatures which cannot be stored in the main file for some reason -- that should be backed up. Permissions, that should be backed up. But not views which are computed from the main file. You don't need to back them up, and they don't need to take any real space. They're virtual, just like an infinitely deep directory on a web server can be virtual. You can make those very silly too, if you want to. (Some help from the filesystem with storing temporarily cached values is fine, for performance, but we shouldn't pretend that generated views are anything other than virtual). > I'd really like to see a concrete example from Hans or other > proponents about why this makes things easier/faster/better to do. > Mostly, I've just seen "proof by vigorous handwaving" that it's a good > thing. > > In alot of ways, I think people are going in the wrong direction, you > want to excapsulate and hide the details more, not expose them. > That's what a good API does, it hides the details while giving you a > rich set of semantics to manage your data. Would you like to propose an alternate API? I was under the impression that we were discussing one, right here in this thread. It's called POSIX with an extension where you can cd into a file. It has a rich set of semantics to manage your data, and works with a lot of familiar programs. A better one would be welcome, if you have an idea. > God knows I'm not smart enough or driven enough to actually come up > with my own ideas, but I certainly haven't seen anyone else (even > Linus) come up with an earth shattering arguement to say why this is > the right move to make. Come up with a better one otherwise it's right ;) > As Linus says, most of the OS's job is to mediate access to > objects/data. Why do we want to expose such low level data then? Eh? In this case the OS is mediating by providing a uniform interface between programs that access the data and programs that handle file formats. What is this "low level" data you speak of? I would say something like the "designer" of a font, or the "from address" and "subject" of an email are very high level abstractions, and that's among the sorts of things which are to be exposed by these interfaces. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:53 ` Jamie Lokier 2004-08-26 19:32 ` John Stoffel @ 2004-08-30 17:37 ` Alex Zarochentsev 2004-08-30 18:51 ` Hans Reiser 1 sibling, 1 reply; 1392+ messages in thread From: Alex Zarochentsev @ 2004-08-30 17:37 UTC (permalink / raw) To: Jamie Lokier Cc: Rik van Riel, Christophe Saout, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 05:53:51PM +0100, Jamie Lokier wrote: > Rik van Riel wrote: > > And if an unaware application reads the compound file > > and then writes it out again, does the filesystem > > interpret the contents and create the other streams ? > > Yes, exactly that. The streams are created on demand of course, and > by userspace helpers when that's appropriate which I suspect it almost > always is. > > > Unless I overlook something (please tell me what), the > > scheme just proposed requires filesystems to look at > > the content of files that is being written out, in > > order to make the streams work. > > Yes. Hence the idea of coherent views between two files: writing to > one affects the content of the other, although the calcalation is only > done on demand (or when the fs wants to migrate the representation -- > for example, creating the flat container prior to deleting the > regeneratable pieces in order to save space). > > I haven't seen anything from Namesys that says they'll do that. I > have the impression the streams are just generated in memory on the > fly, not stored on disk with a cacheing policy, but that's just an > impression. (We've all seen the Namesys white papers, they're not > _that_ revealing). :) would it be a tautology if I say that the streams implementation depends on (reiser4 now, may be vfs later) plugins? All proposals including user-level helpers, shadow caching/indexing may be implemented, why not? > I'm just pointing out how to do it _right_. I think it will turn out > like this eventually, either next year or some time over the next > decade after some iterations. Inevitable. Mark my words, etc. :) > > -- Jamie -- Alex. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-30 17:37 ` Alex Zarochentsev @ 2004-08-30 18:51 ` Hans Reiser 0 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-30 18:51 UTC (permalink / raw) To: Alex Zarochentsev Cc: Jamie Lokier, Rik van Riel, Christophe Saout, Denis Vlasenko, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Alex Zarochentsev wrote: >On Thu, Aug 26, 2004 at 05:53:51PM +0100, Jamie Lokier wrote: > > >>Rik van Riel wrote: >> >> >>>And if an unaware application reads the compound file >>>and then writes it out again, does the filesystem >>>interpret the contents and create the other streams ? >>> >>> >>Yes, exactly that. The streams are created on demand of course, and >>by userspace helpers when that's appropriate which I suspect it almost >>always is. >> >> >> >>>Unless I overlook something (please tell me what), the >>>scheme just proposed requires filesystems to look at >>>the content of files that is being written out, in >>>order to make the streams work. >>> >>> >>Yes. Hence the idea of coherent views between two files: writing to >>one affects the content of the other, although the calcalation is only >>done on demand (or when the fs wants to migrate the representation -- >>for example, creating the flat container prior to deleting the >>regeneratable pieces in order to save space). >> >>I haven't seen anything from Namesys that says they'll do that. >> We do that, err, it is our design to do that. Hmm, the plugins to do that seem to all be in the next to implement list and not on the implemented already list..... However, with respect to streams, those are just files in a file-directory object in our scheme. If you want them unified, well, then you cat compound-document/pseudos/glued, or you link compound-document/pseudos/glued to compound-document/pseudos/body, where body is our term for the default stream, and then you just cat compound-document If you have the impression I am being vaporwarish, well, you are right.... glued is not yet on the already implemented list. Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 15:54 ` Rik van Riel 2004-08-26 16:04 ` Christophe Saout 2004-08-26 16:53 ` Jamie Lokier @ 2004-08-26 17:05 ` Diego Calleja 2004-08-26 17:16 ` Rik van Riel 2 siblings, 1 reply; 1392+ messages in thread From: Diego Calleja @ 2004-08-26 17:05 UTC (permalink / raw) To: Rik van Riel Cc: jamie, christophe, vda, christer, spam, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list El Thu, 26 Aug 2004 11:54:51 -0400 (EDT) Rik van Riel <riel@redhat.com> escribió: > And if an unaware application reads the compound file > and then writes it out again, does the filesystem > interpret the contents and create the other streams ? If the old utils doesn't support streams there's no way to make them create streams. There's not a real solution for that except fixing the old tools. Why instead of being "compound" you just make test.compound to behave as a real directory? ie: # echo "foo" > streamfile # echo "fake metadata" > streamfile/somemetadata # cat streamfile foo # cat streamfile/somemetadata fake metadata Now the old backup utils could see "streamfile" as a directory, ie: old apps could see that "streamfile" is a directory and then make their backups with the following contents: streamfile/somemetadata streamfile/defaultstream The defaultstream would be what its name means. There could be more default streams like "streamfile/compoundstream" All this looks like reinventing the file/directory concept wheel. Instead of adding support for streams and "use files as directories", why not orientate it to "use directories as files? Streams could very well be provided by directories containing files, with the addon of being able to cat a directory (which would cat the output of a designed file inside every directory like "defaultstream", then there could be other "special files" like "compoundstream", although the idea of "fixed name- special files" doesn't sounds very cool) All this could be very well provided done userspace, but then couldn't userspace also supply the whole VFS functionality and still it lives in the kernel...? ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:05 ` Diego Calleja @ 2004-08-26 17:16 ` Rik van Riel 2004-08-26 17:20 ` Linus Torvalds 2004-08-26 17:40 ` Diego Calleja 0 siblings, 2 replies; 1392+ messages in thread From: Rik van Riel @ 2004-08-26 17:16 UTC (permalink / raw) To: Diego Calleja Cc: jamie, christophe, vda, christer, spam, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Diego Calleja wrote: > All this looks like reinventing the file/directory concept wheel. > Instead of adding support for streams and "use files as directories", > why not orientate it to "use directories as files? Streams could very > well be provided by directories containing files, So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ? This is getting surreal. -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:16 ` Rik van Riel @ 2004-08-26 17:20 ` Linus Torvalds 2004-08-26 17:27 ` Jamie Lokier ` (2 more replies) 2004-08-26 17:40 ` Diego Calleja 1 sibling, 3 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-08-26 17:20 UTC (permalink / raw) To: Rik van Riel Cc: Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Rik van Riel wrote: > > On Thu, 26 Aug 2004, Diego Calleja wrote: > > > All this looks like reinventing the file/directory concept wheel. > > Instead of adding support for streams and "use files as directories", > > why not orientate it to "use directories as files? Streams could very > > well be provided by directories containing files, > > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ? No no. The stream you get from a directory is totally _independent_ of the contents of the directory. Anything else would be pointless. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:20 ` Linus Torvalds @ 2004-08-26 17:27 ` Jamie Lokier 2004-08-26 17:32 ` Linus Torvalds 2004-08-26 17:57 ` Rik van Riel 2004-08-27 8:08 ` Hans Reiser 2 siblings, 1 reply; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 17:27 UTC (permalink / raw) To: Linus Torvalds Cc: Rik van Riel, Diego Calleja, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Linus Torvalds wrote: > > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ? > > No no. The stream you get from a directory is totally _independent_ of the > contents of the directory. Anything else would be pointless. Surely it depends on the directory? The stream from a "classic directory", i.e. something made with mkdir(), mode drwxr-xr-x filled with files, might well be a reproducible representation of its contents. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:27 ` Jamie Lokier @ 2004-08-26 17:32 ` Linus Torvalds 2004-08-26 19:09 ` Jamie Lokier 0 siblings, 1 reply; 1392+ messages in thread From: Linus Torvalds @ 2004-08-26 17:32 UTC (permalink / raw) To: Jamie Lokier Cc: Rik van Riel, Diego Calleja, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Jamie Lokier wrote: > Linus Torvalds wrote: > > > > No no. The stream you get from a directory is totally _independent_ of the > > contents of the directory. Anything else would be pointless. > > Surely it depends on the directory? Well, you _could_ make it do a "tar", but what's the point? That really doesn't add any value as far as user space is concerned. Think of it this way: if the directory and the "default stream" associated with it aren't independent, then user space already has the information, and the stream is pointless. In fact, the stream is _worse_ than pointless, because it hides the fact that there is no independent information. See my point? Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:32 ` Linus Torvalds @ 2004-08-26 19:09 ` Jamie Lokier 0 siblings, 0 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 19:09 UTC (permalink / raw) To: Linus Torvalds Cc: Rik van Riel, Diego Calleja, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Linus Torvalds wrote: > Well, you _could_ make it do a "tar", but what's the point? That really > doesn't add any value as far as user space is concerned. > > Think of it this way: if the directory and the "default stream" associated > with it aren't independent, then user space already has the information, > and the stream is pointless. In fact, the stream is _worse_ than > pointless, because it hides the fact that there is no independent > information. > > See my point? I see your point, but I agree 50%. (1) One of the uses of "streams" is to have extra information which accompanies a file, and is separate from the flat file's information. For example, metadata like "this C source file is in UTF-8", "the author is Jamie (nowhere to put it in the data file)" and "Jamie's signature". (2) Filesystem settings like "compress this", and perhaps "permissions". (3) Filesystem information like "this file has been compressed by 59%, uses 5 1k blocks". (4) Another use, which is more akin to containers, is to _present_ the information of the flat file in a different way. For example, the id tags and comments of any media file, the component parts of any media file, license (of course :), and expanded or decoded views such as inside an archive file, filesystem image, or other structured storage file (.doc :). Those alternate views are useful just because they can be accessed using ordinary unix tools, no matter what the obscure codec format provided it's supported. With a bit of effort a standard presentation for certain kinds of data may arise. (5) There is also data which is _calculated_ from the flat file or its extra metadata, and which the kernel guarantees is either consistent with the file, not set at all, or at least is invalidated if the file is changed. It may be calculated on demand through the kernel interface, or applications may simply have to calculate it themselves when they don't have an up to date value. For example MD5 or SHA1 digests of the flat file, or of its parts. The key thing is that these are always up to date, not dependent on weak tests like mtime. Partial checksums for speeding up rsync and other delta algorithms. Computed thumbnails, MP3 average power levels, character encoding extracted from parsing old HTML META tags. Anything really that can be calculated and usefully cached. This is similar to (4), but I'm emphasising things that may take a while to compute, may be cached with the filesystem itself helping to decide when to evict, and writing them won't affect the data they're calculated from, if writing is allowed at all. The only really important of these are MD5 or SHA1 digests, or some other way of tracking modifications like a strong serial number. All other computed cached data can be maintained by applications elsewhere, if they know when the data has been modified since the last time. (Though there are advantages to making the data disappear if the file is deleted, or move if it's renamed). The thing with these different categories is that we want different behaviour when serialising the data for a backup or to copy it. We probably always (1) to be copied. (2) is questionable: it's like unix file permissions, you might want to copy it; you might not. (3) Shouldn't be copied or stored except for special reasons. It's a bit like /proc: it's not meant to be stored on your backup CD and it's definitely not meant to be written during a restore. :) (4) Should never be copied if you're copying the flat file, because it's exactly the same information, just accessed differently. You might want to copy it by itself for some reason though, and you certainly would like to work with it from scripts and applications. These streams are special in another way: they might themselves decompose into further streams. Just think of a AVI file containing an OGG file containing a FLAC containing metadata and data streams, or something like that. They happen. (5) shouldn't be copied if you're transmitting a file, as a matter of data integrity among other things. You might want it on the backup, then again maybe only some of the values. Probably never the crypto digests. To summarise: most data inside a directory-as-file should _not_ usually be copied and transported. A little of it, (1), should be copied if the serialised form will have it, and some of the other metadata should be copied under special circumstances. I've suggested that we could simply not have (1), and recommend that only reproducible, non-essential data appears inside the directory. Everything essential would be part of the serialised, flat file. However, you seem to like (1), metadata which should be serialised and copied around, and which is not part of the flat file. Note that (1) will only be copied around on systems which handle "extra forks" anyway -- things like Samba and MacOS compatibility. FTP and HTTP obviously won't copy the metadata of (1), although HTTP WebDAV might allow it to be visible as properties. All that suggests a common structure for the directory inside a "container" -- one where the metadata to be copied by default is clearly distinguished from the metadata which should never by copied, or which is cached, or which is a view. Possibly it should be distinguished by name, although permission bits or (not sure I like this) meta-metadata could serve the same role without being name dependent. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:20 ` Linus Torvalds 2004-08-26 17:27 ` Jamie Lokier @ 2004-08-26 17:57 ` Rik van Riel 2004-08-26 18:15 ` Linus Torvalds 2004-08-26 18:28 ` Denis Vlasenko 2004-08-27 8:08 ` Hans Reiser 2 siblings, 2 replies; 1392+ messages in thread From: Rik van Riel @ 2004-08-26 17:57 UTC (permalink / raw) To: Linus Torvalds Cc: Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Linus Torvalds wrote: > > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ? > > No no. The stream you get from a directory is totally _independent_ of the > contents of the directory. Anything else would be pointless. It's a relief to know that nobody's taking my humorous suggestion seriously, but now we still have the "standard Unix tools can't manipulate files" problem... -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:57 ` Rik van Riel @ 2004-08-26 18:15 ` Linus Torvalds 2004-08-26 18:41 ` Rik van Riel ` (2 more replies) 2004-08-26 18:28 ` Denis Vlasenko 1 sibling, 3 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-08-26 18:15 UTC (permalink / raw) To: Rik van Riel Cc: Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Rik van Riel wrote: > > It's a relief to know that nobody's taking my humorous > suggestion seriously, but now we still have the "standard > Unix tools can't manipulate files" problem... I disagree. They can manipulate the files a whole lot better than they can manipulate xattr's. For example, you _could_ probably (but hey, maybe "tar" tries to strip slashes off the end of filenames, so this might not work due to silly reasons like that) back up a compound file with tar cvf file.tar file file/ although unpacking it would require that tar be taught about the thing. And you definitely could write a script to do the thing, ie even with an unmodified tar you could do tar cvf file-archive.tar file cd file tar cvf ../attribute-archive.tar . which is a hell of a lot better than what you can do with the fsattr interfaces and unmodified legacy applications. So one of the advantages of "dir-as-file/file-as-dir" is exactly that you _can_ manipulate the data with legacy tools. Sure, things that traverse a directory tree might need some (likely fairly trivial) modifications if they really want to take advantage of the subfiles, but that's still likely to be _much_ less of an issue than with fsattr's that have a totally different model entirely. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:15 ` Linus Torvalds @ 2004-08-26 18:41 ` Rik van Riel 2004-08-26 18:55 ` Linus Torvalds 2004-08-26 20:44 ` Martin J. Bligh [not found] ` <412E33D6.5020500@slaphack.com> 2 siblings, 1 reply; 1392+ messages in thread From: Rik van Riel @ 2004-08-26 18:41 UTC (permalink / raw) To: Linus Torvalds Cc: Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Linus Torvalds wrote: > For example, you _could_ probably (but hey, maybe "tar" tries to strip > slashes off the end of filenames, so this might not work due to silly > reasons like that) back up a compound file with > > tar cvf file.tar file file/ So you'd have both a file and a directory that just happen to have the same name ? How would this work in the dcache? -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:41 ` Rik van Riel @ 2004-08-26 18:55 ` Linus Torvalds 2004-08-26 18:59 ` Rik van Riel 2004-08-27 8:46 ` Herbert Poetzl 0 siblings, 2 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-08-26 18:55 UTC (permalink / raw) To: Rik van Riel Cc: Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Rik van Riel wrote: > > So you'd have both a file and a directory that just happen > to have the same name ? How would this work in the dcache? There would be only one entry in the dcache. The lookup will select whether it opens the file or the directory based on O_DIRECTORY (and usage, of course - if it's in the middle of a path, it obviously needs to be opened as a directory regardless). That's not the problem. The problem from a dcache standpoint ends up being when the file has a link, and you have two paths to the same sub-file through two different ways: .. create file 'x' with named stream 'y' ... ln x z ls -l x/y z/y /* it's the same attribute!! */ but this is actually exactly the same thing that we already have with mounts, ie it is equivalent (from a dentry standpoint) to .. create directory 'x' with file 'y' .. mkdir z mount --bind x z ls -l x/y z/y /* It's the same file!! */ so none of this is really anything "new" from a dcache standpoint. Except for all the details, of course ;) Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:55 ` Linus Torvalds @ 2004-08-26 18:59 ` Rik van Riel 2004-08-26 19:14 ` Christophe Saout ` (2 more replies) 2004-08-27 8:46 ` Herbert Poetzl 1 sibling, 3 replies; 1392+ messages in thread From: Rik van Riel @ 2004-08-26 18:59 UTC (permalink / raw) To: Linus Torvalds Cc: Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Linus Torvalds wrote: > On Thu, 26 Aug 2004, Rik van Riel wrote: > > > > So you'd have both a file and a directory that just happen > > to have the same name ? How would this work in the dcache? > > There would be only one entry in the dcache. The lookup will select > whether it opens the file or the directory based on O_DIRECTORY (and > usage, of course - if it's in the middle of a path, it obviously needs to > be opened as a directory regardless). Hmmm, I just straced "cp /bin/bash /tmp". One line stood out as a potential problem: open("/tmp/bash", O_WRONLY|O_CREAT|O_LARGEFILE, 0100755) = 4 What do we do with O_CREAT ? Do we always allow both a directory and a file to be created with the same name ? Does this create a new class of "symlink attack" style security holes ? -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:59 ` Rik van Riel @ 2004-08-26 19:14 ` Christophe Saout 2004-08-26 19:23 ` Linus Torvalds 2004-08-26 19:29 ` Jamie Lokier 2 siblings, 0 replies; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 19:14 UTC (permalink / raw) To: Rik van Riel Cc: Linus Torvalds, Diego Calleja, jamie, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 673 bytes --] Am Donnerstag, den 26.08.2004, 14:59 -0400 schrieb Rik van Riel: > open("/tmp/bash", O_WRONLY|O_CREAT|O_LARGEFILE, 0100755) = 4 > > What do we do with O_CREAT ? > > Do we always allow both a directory and a file to be created with > the same name ? I would say that the directory under a file is implicit. There are still the pseudo-files a filesystem might want to provide or the VFS if there should be an agreement to generally provide file/pseudo/uid or possibly some security hooks. Some filesystem might want to provide those pseudo files but doesn't have the technical possibility to store non-pseudeo files from the user under such a directory. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:59 ` Rik van Riel 2004-08-26 19:14 ` Christophe Saout @ 2004-08-26 19:23 ` Linus Torvalds 2004-08-26 19:44 ` Rik van Riel ` (2 more replies) 2004-08-26 19:29 ` Jamie Lokier 2 siblings, 3 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-08-26 19:23 UTC (permalink / raw) To: Rik van Riel Cc: Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Rik van Riel wrote: > > Hmmm, I just straced "cp /bin/bash /tmp". > One line stood out as a potential problem: > > open("/tmp/bash", O_WRONLY|O_CREAT|O_LARGEFILE, 0100755) = 4 > > What do we do with O_CREAT ? > > Do we always allow both a directory and a file to be created with > the same name ? Either I am confused, or you are. To me, a filesystem that allows this thing doesn't really _have_ the concept of "directory vs file". It's just a "filesystem object", and it can act as _both_ a directory and a file. So when you create "/tmp/bash" - assuming /tmp supports the file-as-dir semantics at all, you're just creating a new inode - the same you always have. When you write to that inode, it writes to the default stream. There's no special cases here. Now, after you have your regular /tmp/bash, you can then start adding named streams to it, ie you can do open("/tmp/bash/icon", O_WRONLY|O_CREAT|O_LARGEFILE, 0755); and that will create the "icon" named stream. See? So "/tmp/bash" is _not_ two different things. It is _one_ entity, that contains both a standard data stream (the "file" part) _and_ pointers to other named streams (the "directory" part). Hey, think of it as a wave-particle duality. Both "modes" exist at the same time, and cannot be separated from each other. Which one you see depends entirely on your "experiment", ie how you open the file. > Does this create a new class of "symlink attack" style security > holes ? I don't believe so. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 19:23 ` Linus Torvalds @ 2004-08-26 19:44 ` Rik van Riel 2004-08-26 19:47 ` Christophe Saout 2004-08-26 20:10 ` Rik van Riel 2004-08-27 8:06 ` Hans Reiser 2 siblings, 1 reply; 1392+ messages in thread From: Rik van Riel @ 2004-08-26 19:44 UTC (permalink / raw) To: Linus Torvalds Cc: Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Linus Torvalds wrote: > So "/tmp/bash" is _not_ two different things. It is _one_ entity, that > contains both a standard data stream (the "file" part) _and_ pointers to > other named streams (the "directory" part). OK, that makes sense. > Hey, think of it as a wave-particle duality. Both "modes" exist at the > same time, and cannot be separated from each other. Which one you see > depends entirely on your "experiment", ie how you open the file. Guess I'm scared again now. We need to make sure that backup programs don't fall victim to the uncertainty principle ;) -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 19:44 ` Rik van Riel @ 2004-08-26 19:47 ` Christophe Saout 0 siblings, 0 replies; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 19:47 UTC (permalink / raw) To: Rik van Riel Cc: Linus Torvalds, Diego Calleja, jamie, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 526 bytes --] Am Donnerstag, den 26.08.2004, 15:44 -0400 schrieb Rik van Riel: > > Hey, think of it as a wave-particle duality. Both "modes" exist at the > > same time, and cannot be separated from each other. Which one you see > > depends entirely on your "experiment", ie how you open the file. > > Guess I'm scared again now. We need to make sure that > backup programs don't fall victim to the uncertainty > principle ;) At least you can be sure that the backup will go into a defined state as soon as you observe it. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 19:23 ` Linus Torvalds 2004-08-26 19:44 ` Rik van Riel @ 2004-08-26 20:10 ` Rik van Riel 2004-08-26 20:17 ` Christophe Saout ` (5 more replies) 2004-08-27 8:06 ` Hans Reiser 2 siblings, 6 replies; 1392+ messages in thread From: Rik van Riel @ 2004-08-26 20:10 UTC (permalink / raw) To: Linus Torvalds Cc: Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Linus Torvalds wrote: > So "/tmp/bash" is _not_ two different things. It is _one_ entity, that > contains both a standard data stream (the "file" part) _and_ pointers to > other named streams (the "directory" part). Thinking about it some more, how would file managers and file chosers handle this situation ? Currently the user browses the directory tree and when the user clicks on something, one of the following happens: 1) if it is a directory, the file manager/choser changes into that directory 2) if it is a file, the file is opened Now how do we present things to users ? How will users know when an object can only be chdired into, or only be opened ? For objects that do both, how does the user choose ? Do we really want to have a file paradigm that's different from the other OSes out there ? What happens when users want to transfer data from Linux to another system ? -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:10 ` Rik van Riel @ 2004-08-26 20:17 ` Christophe Saout 2004-08-26 20:21 ` Jamie Lokier 2004-08-26 20:35 ` Stephen Wille Padnos ` (4 subsequent siblings) 5 siblings, 1 reply; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 20:17 UTC (permalink / raw) To: Rik van Riel Cc: Linus Torvalds, Diego Calleja, jamie, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1317 bytes --] Am Donnerstag, den 26.08.2004, 16:10 -0400 schrieb Rik van Riel: > > So "/tmp/bash" is _not_ two different things. It is _one_ entity, that > > contains both a standard data stream (the "file" part) _and_ pointers to > > other named streams (the "directory" part). > > Thinking about it some more, how would file managers and > file chosers handle this situation ? I would say they don't. That's where metadata and pseudo-files sit. It's up to the applications to do something useful with the data. > Do we really want to have a file paradigm that's different > from the other OSes out there ? Some other OSes have similar things. > What happens when users want to transfer data from Linux > to another system ? No one should store really important data inside file/ that makes the file completely useless. I just love these postscript fonts on the Mac. The actual main file stream is 0 bytes in size. The actual font is inside the resource fork. When I want to convert it the Mac user needs to pack it into a compound file using a special tool and then I can decode it using another tool to access the PFB file. Crap. It's useful for metadata like keywords, ACLs and these things. If you can pack it using an aware backup program, fine. If you don't, you still have the data. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:17 ` Christophe Saout @ 2004-08-26 20:21 ` Jamie Lokier 0 siblings, 0 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 20:21 UTC (permalink / raw) To: Christophe Saout Cc: Rik van Riel, Linus Torvalds, Diego Calleja, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Christophe Saout wrote: > I just love these postscript fonts on the Mac. The actual main file > stream is 0 bytes in size. The actual font is inside the resource > fork. When I want to convert it the Mac user needs to pack it into a > compound file using a special tool and then I can decode it using > another tool to access the PFB file. Crap. I thought it was a cunning way to stop me from copying those nice fonts onto my Linux box. The original DRM. Thanks for the tip :) -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:10 ` Rik van Riel 2004-08-26 20:17 ` Christophe Saout @ 2004-08-26 20:35 ` Stephen Wille Padnos 2004-08-31 11:01 ` Claus Färber 2004-08-26 20:40 ` Linus Torvalds ` (3 subsequent siblings) 5 siblings, 1 reply; 1392+ messages in thread From: Stephen Wille Padnos @ 2004-08-26 20:35 UTC (permalink / raw) To: Rik van Riel Cc: Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Rik van Riel wrote: >On Thu, 26 Aug 2004, Linus Torvalds wrote: > > >>So "/tmp/bash" is _not_ two different things. It is _one_ entity, that >>contains both a standard data stream (the "file" part) _and_ pointers to >>other named streams (the "directory" part). >> >> >Thinking about it some more, how would file managers and >file chosers handle this situation ? > >Currently the user browses the directory tree and when >the user clicks on something, one of the following >happens: > >1) if it is a directory, the file manager/choser changes > into that directory > > How does the file manager / chooser decide whether you're trying to move into a directory, or the meta-data-directory for a directory? It's not just files that should have metadata - directories need* them too. Making it possible to see attributes as a directory under a file is great, but you'd still need an O_META flag for accessing directory metadata (since there are already files under a directory). >2) if it is a file, the file is opened > >Now how do we present things to users ? > >How will users know when an object can only be chdired >into, or only be opened ? > >For objects that do both, how does the user choose ? > >Do we really want to have a file paradigm that's different >from the other OSes out there ? > > MacOS does, Be did (sort of). I'm not sure it would be the end of the world, as long as the data can be extracted. >What happens when users want to transfer data from Linux >to another system ? > > That would depend on the other system. Data is easy, metadata is hard. It would be possible to create an XML schema for metadata, and if requested (O_EVERYTHING?), the file data is returned with all metadata in XML tags. (not advocating this, just an idea :) - Steve * I say need in the same way as one *needs* to upgrade their 2GHz computer - it would be nice :) ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:35 ` Stephen Wille Padnos @ 2004-08-31 11:01 ` Claus Färber 2004-09-01 15:13 ` Alan Cox 0 siblings, 1 reply; 1392+ messages in thread From: Claus Färber @ 2004-08-31 11:01 UTC (permalink / raw) To: linux-kernel; +Cc: reiserfs-list, linux-fsdevel Stephen Wille Padnos <spadnos@sover.net> schrieb/wrote: > How does the file manager / chooser decide whether you're trying to > move into a directory, or the meta-data-directory for a directory? > It's not just files that should have metadata - directories need* them > too. Making it possible to see attributes as a directory under a file > is great, but you'd still need an O_META flag for accessing directory > metadata (since there are already files under a directory). A simple convention that meta data files start with, say ".$", would be enough. Claus -- http://www.faerber.muc.de ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 11:01 ` Claus Färber @ 2004-09-01 15:13 ` Alan Cox 2004-09-03 13:23 ` Claus Färber 0 siblings, 1 reply; 1392+ messages in thread From: Alan Cox @ 2004-09-01 15:13 UTC (permalink / raw) To: Claus Färber; +Cc: Linux Kernel Mailing List, reiserfs-list, linux-fsdevel On Maw, 2004-08-31 at 12:01, Claus Färber wrote: > A simple convention that meta data files start with, say ".$", would be > enough. POSIX/SuS don't permit this. The only "free" namespace is that starting "//" (and not as some desktops seem to think foo://). Remember always send GUI desktop users files called http://read.me .. its fun 8) The // doesn't really help because you don't want extensions at the path top. Another interesting question btw with substreams of a file is what the semantics of fchown, fstat, fchmod, fchdir etc are, or of mounting a file over a substream. Alan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 15:13 ` Alan Cox @ 2004-09-03 13:23 ` Claus Färber 0 siblings, 0 replies; 1392+ messages in thread From: Claus Färber @ 2004-09-03 13:23 UTC (permalink / raw) To: linux-kernel; +Cc: reiserfs-list, linux-fsdevel Alan Cox <alan@lxorguk.ukuu.org.uk> schrieb/wrote: > On Maw, 2004-08-31 at 12:01, Claus Färber wrote: >> A simple convention that meta data files start with, say ".$", would >> be enough. > POSIX/SuS don't permit this. The only "free" namespace is that starting > "//" (and not as some desktops seem to think foo://). Remember always > send GUI desktop users files called http://read.me .. its fun 8) Well, the problem is that one does want the metadata to be in the ordinary namespace so that they are accessible with POSIX tools. (The current version of) POSIX does not have a "unified" namespace so a unified namespace can't be POSIX. I wonder if it makes sense to have an environment variable (maybe POSIXLY_CORRECT) to switch between POSIX and the unified namespace model. Of course, this would mean that all of this is handled in userspace (e.g. libc), whereas the kernel and filesystems export more "traditional" (e.g. with openat) interfaces. Accessing metadata as ordinary files would then work with other filesystems, too. Claus -- http://www.faerber.muc.de ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:10 ` Rik van Riel 2004-08-26 20:17 ` Christophe Saout 2004-08-26 20:35 ` Stephen Wille Padnos @ 2004-08-26 20:40 ` Linus Torvalds 2004-08-26 22:53 ` Hans Reiser 2004-08-26 20:48 ` Jamie Lokier ` (2 subsequent siblings) 5 siblings, 1 reply; 1392+ messages in thread From: Linus Torvalds @ 2004-08-26 20:40 UTC (permalink / raw) To: Rik van Riel Cc: Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Rik van Riel wrote: > > Thinking about it some more, how would file managers and file chosers > handle this situation ? The same way they already handle it on other platforms that support it? By taking advantage of it.. People literally use this for icon and preview information, so some of the stuff shows up very much in file managers. And I'd assume that a normal double-click would just open the main file, and you can right-click for management information, including opening the file. If you want an entity that acts as a directory, you create a directory. Directories don't go away - you still have S_ISDIR() and S_ISREG(), and they still return information. A file that has associated information is still a _file_, and people should treat it that way, it's just that it also has a list of named sub-streams. You can open it as a directory, but the stat information clearly says it is a file, and the "directory" view is very much associated with _that_ file. I definitely don't think you want the file manager to act as if a named stream is exactly the same as a stand-alone file. They have all the same operations, but there's no question that there are differences too. Especially on a conceptual level - but for most filesystems there are likely real technical differences too. For example, it's likely that most filesystems would _not_ allow linking of a named stream anywhere else. And you might not be able to change the permissions or date on the named stream either, since it may or may not have a separate date/permission thing from the container. So don't believe that just because the named streams are _named_ like real files, that they suddenly have any existence beyond the container that they are part of. There may be other limitations too - again depending on how the filesystem actually implements named streams. It might not support more than one level of naming, for example - so you might not be able to create a directory structure within the named streams, for example. In short: the fact that the VFS layer exposes the _capability_ to see the named streams as a full POSIX filesystem of its own does _not_ mean that the low-level filesystem necessarily allows the full possible semantics. So you shouldn't design your apps that know about named streams to think they are normal directories. The directory thing is just a very powerful naming scheme, and one that fits into the existing UNIX model. > Do we really want to have a file paradigm that's different > from the other OSes out there ? Different from what other OSes? Last I looked, Windows, Solaris, and OSX all supported named streams. What other OS's are out there that you care about? In other words, this is _not_ a different paradigm from what others do. The discussion is whether we want to implement it at all, and more importantly about syntactic issues (ie we clearly already implement extended attributes, just with a much more limited syntactic power). We don't have to go all the way. Solaris has "openat()", which is kind of a half-way there - not really directly available in the same namespace, but at least the result is available as a real file interface (as opposed to the Linux "xattr" interfaces that are _totally_ special-cased system calls). In other words - the paradigm is already there. It's just that currently it's pretty much unusable under Linux, because it requires so much specialized knowledge that it's not worth it to modify existing apps. And the interfaces are really very limited, so even if you _do_ end up using the specialized Linux interfaces, you can't actually do a lot of what you might want to do. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:40 ` Linus Torvalds @ 2004-08-26 22:53 ` Hans Reiser 2004-08-26 23:27 ` Linus Torvalds 0 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 22:53 UTC (permalink / raw) To: Linus Torvalds Cc: Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Linus Torvalds wrote: > > >For example, it's likely that most filesystems would _not_ allow linking >of a named stream anywhere else. > Why? It is just a file that inherits its stat data and is referenced by a concatenation method of its parent directory. The link will also inherit its stat data from the original parent directory. > And you might not be able to change the >permissions or date on the named stream either, since it may or may not >have a separate date/permission thing from the container. > > You should be able to change the permission and data, but when you change it, you change it for the whole container. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 22:53 ` Hans Reiser @ 2004-08-26 23:27 ` Linus Torvalds 2004-08-26 23:47 ` Hans Reiser 0 siblings, 1 reply; 1392+ messages in thread From: Linus Torvalds @ 2004-08-26 23:27 UTC (permalink / raw) To: Hans Reiser Cc: Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Hans Reiser wrote: > > Linus Torvalds wrote: > > > >For example, it's likely that most filesystems would _not_ allow linking > >of a named stream anywhere else. > > Why? It is just a file that inherits its stat data and is referenced by > a concatenation method of its parent directory. The link will also > inherit its stat data from the original parent directory. In reiser4, maybe. Not in general. Look at what other filesystems do for attributes.. > > And you might not be able to change the > >permissions or date on the named stream either, since it may or may not > >have a separate date/permission thing from the container. > > You should be able to change the permission and data, but when you > change it, you change it for the whole container. Why? I'd much rather disallow it, than allow it and then have "nonlocal" effects. I can also see a filesystem validly having _extra_ permissions (ie apart from the container) on it too. Not all the world is reiser4. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 23:27 ` Linus Torvalds @ 2004-08-26 23:47 ` Hans Reiser [not found] ` <20040827020350.GF21964@parcelfarce.linux.theplanet.co.uk> 0 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 23:47 UTC (permalink / raw) To: Linus Torvalds Cc: Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Linus Torvalds wrote: > > > >>>And you might not be able to change the >>>permissions or date on the named stream either, since it may or may not >>>have a separate date/permission thing from the container. >>> >>> >>You should be able to change the permission and data, >> ^data^date >> but when you >>change it, you change it for the whole container. >> >> > >Why? I'd much rather disallow it, than allow it and then have "nonlocal" >effects. > > Sometimes you want the nonlocal effects and sometimes you don't, and by decomposing streams into smaller primitives we can let users choose as they want. ^ permalink raw reply [flat|nested] 1392+ messages in thread
[parent not found: <20040827020350.GF21964@parcelfarce.linux.theplanet.co.uk>]
* Re: silent semantic changes with reiser4 [not found] ` <20040827020350.GF21964@parcelfarce.linux.theplanet.co.uk> @ 2004-08-27 8:10 ` Hans Reiser 0 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-27 8:10 UTC (permalink / raw) To: viro Cc: Linus Torvalds, Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list viro@parcelfarce.linux.theplanet.co.uk wrote: >On Thu, Aug 26, 2004 at 04:47:39PM -0700, Hans Reiser wrote: > > > >>Sometimes you want the nonlocal effects and sometimes you don't, and by >>decomposing streams into smaller primitives we can let users choose as >>they want. >> >> > >Right. Now, would you kindly post the detailed technical analysis of your >wonderful design that handles that stuff safely? With proof of correctness, >please. Since we are expected to take your code and use it as "Uber" >replacement for the existing one, it surely would not be too much to expect, >would it? > >And no, "we have a nice presentation somewhere on namesys" does not >qualify. > >I apologize for the obvious posting in a thread already full of noise, >but I would like to make *very* sure that lack of ripping you another >one does *not* mutate into "no objections from viro" in a lovely thread >on SlashDot in a week or so. > > > > Over the next 6-18 months, one piece at a time, it will all fall into place. Reiser4 as it is shipping today, consists of close to the minimal functionality necessary to replace reiser3, albeit done according to our plugin model, and with vastly higher performance. The full solution is not yet written, wait for it. I understand why you think deadlocks are everything, but avoiding them is only a small, albeit necessary, piece of the task. Thanks though for pointing out issues with unlinking files that have been cd'd to. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:10 ` Rik van Riel ` (2 preceding siblings ...) 2004-08-26 20:40 ` Linus Torvalds @ 2004-08-26 20:48 ` Jamie Lokier 2004-08-26 20:52 ` Jeremy Allison 2004-08-26 22:20 ` Felipe Alfaro Solana 2004-08-29 0:04 ` Horst von Brand 5 siblings, 1 reply; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 20:48 UTC (permalink / raw) To: Rik van Riel Cc: Linus Torvalds, Diego Calleja, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Rik van Riel wrote: > For objects that do both, how does the user choose ? > > Do we really want to have a file paradigm that's different > from the other OSes out there ? What does MacOS X do? Someone said that documents are directories in it; it must know how to handle that. What does Windows do when you click on a .zip file and WinZIP is installed? It opens the .zip file and lets you explore inside. When you click on a .doc file, though, it opens a viewer or editor -- you don't get the option to look inside. > What happens when users want to transfer data from Linux > to another system ? This is why I favour storing all essential metadata (about the file's content) inside the file's contents, the primary stream. We have another problem: what happens when users want to transfer data from Windows (with secondary streams) and MacOS (with resource forks)? -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:48 ` Jamie Lokier @ 2004-08-26 20:52 ` Jeremy Allison 2004-08-27 9:19 ` Christoph Hellwig 0 siblings, 1 reply; 1392+ messages in thread From: Jeremy Allison @ 2004-08-26 20:52 UTC (permalink / raw) To: Jamie Lokier Cc: Rik van Riel, Linus Torvalds, Diego Calleja, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 09:48:41PM +0100, Jamie Lokier wrote: > > This is why I favour storing all essential metadata (about the file's > content) inside the file's contents, the primary stream. > > We have another problem: what happens when users want to transfer data > from Windows (with secondary streams) and MacOS (with resource forks)? I can tell you that - right now we (Samba) has to drop all but the primary stream, because the Linux filesystems don't have data stream support. Oh look, we're losing user data (that's a *bad* thing :-). Jeremy. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:52 ` Jeremy Allison @ 2004-08-27 9:19 ` Christoph Hellwig 2004-08-28 3:56 ` Jeremy Allison 0 siblings, 1 reply; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-27 9:19 UTC (permalink / raw) To: Jeremy Allison Cc: Jamie Lokier, Rik van Riel, Linus Torvalds, Diego Calleja, christophe, vda, christer, spam, akpm, wichert, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 01:52:18PM -0700, Jeremy Allison wrote: > On Thu, Aug 26, 2004 at 09:48:41PM +0100, Jamie Lokier wrote: > > > > This is why I favour storing all essential metadata (about the file's > > content) inside the file's contents, the primary stream. > > > > We have another problem: what happens when users want to transfer data > > from Windows (with secondary streams) and MacOS (with resource forks)? > > I can tell you that - right now we (Samba) has to drop all but the primary stream, > because the Linux filesystems don't have data stream support. Oh look, > we're losing user data (that's a *bad* thing :-). Maybe you should learn from netatalk. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 9:19 ` Christoph Hellwig @ 2004-08-28 3:56 ` Jeremy Allison 2004-08-28 19:18 ` Brad Boyer 0 siblings, 1 reply; 1392+ messages in thread From: Jeremy Allison @ 2004-08-28 3:56 UTC (permalink / raw) To: Christoph Hellwig, Jeremy Allison, Jamie Lokier, Rik van Riel, Linus Torvalds, Diego Calleja, christophe, vda, christer, spam, akpm, wichert, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, Aug 27, 2004 at 10:19:56AM +0100, Christoph Hellwig wrote: > > Maybe you should learn from netatalk. I am well aware of what netatalk does. However netatalk isn't as widely used as Samba. Things they can get away with would cause our user community to flay us alive. We need a proper solution, not a nasty hack. That's like me telling you to "learn from *BSD". You have different user constituencies, you have to serve yours, I have to serve mine. Jeremy. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 3:56 ` Jeremy Allison @ 2004-08-28 19:18 ` Brad Boyer 0 siblings, 0 replies; 1392+ messages in thread From: Brad Boyer @ 2004-08-28 19:18 UTC (permalink / raw) To: Jeremy Allison Cc: Christoph Hellwig, Jamie Lokier, Rik van Riel, Linus Torvalds, Diego Calleja, christophe, vda, christer, spam, akpm, wichert, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, Aug 27, 2004 at 08:56:29PM -0700, Jeremy Allison wrote: > I am well aware of what netatalk does. However netatalk > isn't as widely used as Samba. Things they can get away > with would cause our user community to flay us alive. > > We need a proper solution, not a nasty hack. > > That's like me telling you to "learn from *BSD". You have > different user constituencies, you have to serve yours, > I have to serve mine. While I do think that there are some hacks in netatalk, the hacks there are not nearly as bad as the hacks that made it into the kernel to support exporting a native Mac filesystem. Take a look at the fork code in the hfs filesystem from 2.4 or before for exporting to netatalk. It's disgusting, and mostly because there is no easy way to export the resource fork and extra metadata that is essential to the client. The 2.6 code doesn't have that anymore, but I don't think it actually works with netatalk either. Perhaps someone can correct me on that, but I don't see how it could work. I would hope that something comes out of this whole discussion that will allow a much better interface between the hfs/hfsplus code in the kernel and netatalk. Brad Boyer flar@allandria.com ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:10 ` Rik van Riel ` (3 preceding siblings ...) 2004-08-26 20:48 ` Jamie Lokier @ 2004-08-26 22:20 ` Felipe Alfaro Solana 2004-08-29 0:04 ` Horst von Brand 5 siblings, 0 replies; 1392+ messages in thread From: Felipe Alfaro Solana @ 2004-08-26 22:20 UTC (permalink / raw) To: Rik van Riel Cc: Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thursday 26 August 2004 22:10, Rik van Riel wrote: > Do we really want to have a file paradigm that's different > from the other OSes out there ? > > What happens when users want to transfer data from Linux > to another system ? I think it depends on the transport being used: some transports could allow for metadata (i.e. a MIME-compatible transport) while some others don't. Of course, the remote side needs to also support metadata (i,e. using a MIME-compatible transport against a remote host that doesn't understand metadata is certainly impossible). For example, with FTP transports, I guess the only possible option is to transfer the unnamed/default stream. NFS should allow metadata, but this probably needs some kind of extensions to the NFS protocol. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:10 ` Rik van Riel ` (4 preceding siblings ...) 2004-08-26 22:20 ` Felipe Alfaro Solana @ 2004-08-29 0:04 ` Horst von Brand 2004-09-01 22:45 ` Jamie Lokier 5 siblings, 1 reply; 1392+ messages in thread From: Horst von Brand @ 2004-08-29 0:04 UTC (permalink / raw) To: Rik van Riel Cc: Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Rik van Riel <riel@redhat.com> said: > On Thu, 26 Aug 2004, Linus Torvalds wrote: > > So "/tmp/bash" is _not_ two different things. It is _one_ entity, that > > contains both a standard data stream (the "file" part) _and_ pointers to > > other named streams (the "directory" part). > Thinking about it some more, how would file managers and > file chosers handle this situation ? > > Currently the user browses the directory tree and when > the user clicks on something, one of the following > happens: > > 1) if it is a directory, the file manager/choser changes > into that directory > > 2) if it is a file, the file is opened And now you have a mess. Is it "really" a file, or a directory? Why not just keep both well apart, and stay happy? I just fail to see what could be gained by having directories that really aren't, and files that aren't either. Use a directory if you want one, use a file elsewhere. > Now how do we present things to users ? > > How will users know when an object can only be chdired > into, or only be opened ? Easy: It is a directory, or it isn't. > For objects that do both, how does the user choose ? Don't give silly choices. > Do we really want to have a file paradigm that's different > from the other OSes out there ? I vote "no". There are/have been OSes with weird "files", none of them survived to get anywhere as popular as Unix and "file == stream of bytes". Even with much simpler variants like files as sequences of records. For a good reason: The Unix way is simple, and extremely flexible, as my proggie can define at its own whim how to handle what's inside. If a single stream isn't enough, we have directories. No need to innovate there. > What happens when users want to transfer data from Linux > to another system ? Or between Linux systems with different kernels that happen to implement different views/metadata on files. Please do remember devfs: It sounded like a cool idea, got into the kernel just to be thrown out later because nobody used it. Much heat was generated, nothing of permanent value. This looks the same: A very vocal tiny minority is clamoring for something completely non-Unix for totally bogus reasons. What happened to "code talks, bullshit walks"? There is _no_ code (== real-world, user programs that can't be done efficiently enough without this), so this nonsense should just be thrown out, and everybody go back to real hacking. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 0:04 ` Horst von Brand @ 2004-09-01 22:45 ` Jamie Lokier 2004-09-02 5:23 ` Olaf Hering 2004-09-03 8:22 ` Greg KH 0 siblings, 2 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-09-01 22:45 UTC (permalink / raw) To: Horst von Brand; +Cc: linux-fsdevel, linux-kernel, reiserfs-list Horst von Brand wrote: > Please do remember devfs: It sounded like a cool idea, got into the kernel > just to be thrown out later because nobody used it. Are you kidding? Lots of distros and many embedded Linuxes use devfs. And udev _still_ doesn't create device nodes properly. (Hint: I have to run two modprobe commands before pppd works. I have to run modprobe before openvpn or bochs work.) > What happened to "code talks, bullshit walks"? devfs is a fine example of why code isn't enough. With devfs the code came first, the >1 year of strategic bullshit politics from the "it's not traditional unix" crowd came later, then it went in, then lots of people used it, then it was replaced by something which still doesn't work as well as 2.4 does with or without devfs, and people are still using it despite it's faults. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 22:45 ` Jamie Lokier @ 2004-09-02 5:23 ` Olaf Hering 2004-09-03 8:22 ` Greg KH 1 sibling, 0 replies; 1392+ messages in thread From: Olaf Hering @ 2004-09-02 5:23 UTC (permalink / raw) To: Jamie Lokier; +Cc: Horst von Brand, linux-fsdevel, linux-kernel, reiserfs-list On Wed, Sep 01, Jamie Lokier wrote: > And udev _still_ doesn't create device nodes properly. (Hint: I have > to run two modprobe commands before pppd works. I have to run > modprobe before openvpn or bochs work.) Just because it is supposed to work that way. -- USB is for mice, FireWire is for men! sUse lINUX ag, nÜRNBERG ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 22:45 ` Jamie Lokier 2004-09-02 5:23 ` Olaf Hering @ 2004-09-03 8:22 ` Greg KH 2004-09-03 8:32 ` Grzegorz Jaśkiewicz 1 sibling, 1 reply; 1392+ messages in thread From: Greg KH @ 2004-09-03 8:22 UTC (permalink / raw) To: Jamie Lokier; +Cc: Horst von Brand, linux-fsdevel, linux-kernel, reiserfs-list On Wed, Sep 01, 2004 at 11:45:13PM +0100, Jamie Lokier wrote: > Horst von Brand wrote: > > What happened to "code talks, bullshit walks"? > > devfs is a fine example of why code isn't enough. With devfs the code > came first, the >1 year of strategic bullshit politics from the "it's > not traditional unix" crowd came later, then it went in, then lots of > people used it, then it was replaced by something which still doesn't > work as well as 2.4 does with or without devfs, and people are still > using it despite it's faults. What is udev's faults that have an issue with? Yes, we don't do module autoloading when opening a device node, but that's well known, documented, and the way the kernel has evolved to anyway. thanks, greg k-h ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-03 8:22 ` Greg KH @ 2004-09-03 8:32 ` Grzegorz Jaśkiewicz 2004-09-03 8:54 ` Helge Hafting 0 siblings, 1 reply; 1392+ messages in thread From: Grzegorz Jaśkiewicz @ 2004-09-03 8:32 UTC (permalink / raw) To: Greg KH Cc: Jamie Lokier, Horst von Brand, linux-fsdevel, linux-kernel, reiserfs-list On Fri, 3 Sep 2004 10:22:57 +0200, Greg KH <greg@kroah.com> wrote: > On Wed, Sep 01, 2004 at 11:45:13PM +0100, Jamie Lokier wrote: > > Horst von Brand wrote: > > > What happened to "code talks, bullshit walks"? > > > > devfs is a fine example of why code isn't enough. With devfs the code > > came first, the >1 year of strategic bullshit politics from the "it's > > not traditional unix" crowd came later, then it went in, then lots of > > people used it, then it was replaced by something which still doesn't > > work as well as 2.4 does with or without devfs, and people are still > > using it despite it's faults. > > What is udev's faults that have an issue with? > > Yes, we don't do module autoloading when opening a device node, but > that's well known, documented, and the way the kernel has evolved to > anyway. devfs was very natural, and simple solution. But to have it right, it would have to be the only /dev filesystem. But no, we like choices, so we have chaos. Udev is just another thing adding to that chaos. Someone was numbering things that are good in BSD design, in that thread. One of those things was going for devfs. No cheap solutions. One fs for /dev. And it works great. Sorry for bit of trolling. -- GJ ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-03 8:32 ` Grzegorz Jaśkiewicz @ 2004-09-03 8:54 ` Helge Hafting 2004-09-03 9:14 ` Grzegorz Jaśkiewicz 0 siblings, 1 reply; 1392+ messages in thread From: Helge Hafting @ 2004-09-03 8:54 UTC (permalink / raw) To: Grzegorz Jaśkiewicz Cc: Greg KH, Jamie Lokier, Horst von Brand, linux-fsdevel, linux-kernel, reiserfs-list Grzegorz Jaśkiewicz wrote: > >devfs was very natural, and simple solution. But to have it right, it >would have to be the only /dev filesystem. >But no, we like choices, so we have chaos. >Udev is just another thing adding to that chaos. > >Someone was numbering things that are good in BSD design, in that >thread. One of those things was going for devfs. No cheap solutions. >One fs for /dev. And it works great. > >Sorry for bit of trolling. > > Devfs was a ver good idea. The implementation of it was a problem, and after some time nobody maintained it. No surprise it had to go. Now udev+tmpfs can do the same job, and more. Helge Hafting ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-03 8:54 ` Helge Hafting @ 2004-09-03 9:14 ` Grzegorz Jaśkiewicz 2004-09-03 9:58 ` Miquel van Smoorenburg ` (2 more replies) 0 siblings, 3 replies; 1392+ messages in thread From: Grzegorz Jaśkiewicz @ 2004-09-03 9:14 UTC (permalink / raw) To: Helge Hafting Cc: Greg KH, Jamie Lokier, Horst von Brand, linux-fsdevel, linux-kernel, reiserfs-list On Fri, 03 Sep 2004 10:54:26 +0200, Helge Hafting <helge.hafting@hist.no> wrote: > Grzegorz Jaśkiewicz wrote: > > > > >devfs was very natural, and simple solution. But to have it right, it > >would have to be the only /dev filesystem. > >But no, we like choices, so we have chaos. > >Udev is just another thing adding to that chaos. > > > >Someone was numbering things that are good in BSD design, in that > >thread. One of those things was going for devfs. No cheap solutions. > >One fs for /dev. And it works great. > > > >Sorry for bit of trolling. > > > > > Devfs was a ver good idea. The implementation of it > was a problem, and after some time nobody maintained it. > No surprise it had to go. Now udev+tmpfs can do the same > job, and more. udef is a one big mistake, having need for userspace tool to use FS is at least silly. I can understeand need for some things in kernel to have userspace daemon. But FS is out of question the least one. I am supprised noone wanted to maintain devfs. Maybe because people didn't want to go to devfs only. But still to have classic /dev. It's also silly, because person writing driver needs to choose between, or implement all. That's more than bad. Once I have loads of time, and no work in KDE, I can take over devfs happily :-) -- GJ ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-03 9:14 ` Grzegorz Jaśkiewicz @ 2004-09-03 9:58 ` Miquel van Smoorenburg 2004-09-03 15:18 ` Grzegorz Jaśkiewicz 2004-09-03 10:43 ` Greg KH 2004-09-08 9:09 ` Helge Hafting 2 siblings, 1 reply; 1392+ messages in thread From: Miquel van Smoorenburg @ 2004-09-03 9:58 UTC (permalink / raw) To: linux-kernel In article <2f4958ff04090302141bc222e5@mail.gmail.com>, Grzegorz JaÅ kiewicz <gryzman@gmail.com> wrote: >I am supprised noone wanted to maintain devfs. Maybe because people >didn't want to go to devfs only. It's because disk was spelled with a 'c' (disc). Mike. -- "In times of universal deceit, telling the truth becomes a revolutionary act." -- George Orwell. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-03 9:58 ` Miquel van Smoorenburg @ 2004-09-03 15:18 ` Grzegorz Jaśkiewicz 2004-09-03 15:29 ` Alan Cox 0 siblings, 1 reply; 1392+ messages in thread From: Grzegorz Jaśkiewicz @ 2004-09-03 15:18 UTC (permalink / raw) To: Miquel van Smoorenburg; +Cc: linux-kernel Miquel van Smoorenburg wrote: >In article <2f4958ff04090302141bc222e5@mail.gmail.com>, >Grzegorz JaÅ kiewicz <gryzman@gmail.com> wrote: > > >>I am supprised noone wanted to maintain devfs. Maybe because people >>didn't want to go to devfs only. >> >> > >It's because disk was spelled with a 'c' (disc). > > That suppose to be a joke ? -- GJ ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-03 15:18 ` Grzegorz Jaśkiewicz @ 2004-09-03 15:29 ` Alan Cox 2004-09-03 16:41 ` Grzegorz Jaśkiewicz 0 siblings, 1 reply; 1392+ messages in thread From: Alan Cox @ 2004-09-03 15:29 UTC (permalink / raw) To: Grzegorz Jaśkiewicz Cc: Miquel van Smoorenburg, Linux Kernel Mailing List On Gwe, 2004-09-03 at 16:18, Grzegorz Jaśkiewicz wrote: > >It's because disk was spelled with a 'c' (disc). > That suppose to be a joke ? The international joke symbol (8-)) is only mandatory in en_US locales. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-03 15:29 ` Alan Cox @ 2004-09-03 16:41 ` Grzegorz Jaśkiewicz 2004-09-03 23:46 ` Paul Jackson 0 siblings, 1 reply; 1392+ messages in thread From: Grzegorz Jaśkiewicz @ 2004-09-03 16:41 UTC (permalink / raw) To: Alan Cox; +Cc: Miquel van Smoorenburg, Linux Kernel Mailing List Alan Cox wrote: >On Gwe, 2004-09-03 at 16:18, Grzegorz Jaśkiewicz wrote: > > >>>It's because disk was spelled with a 'c' (disc). >>> >>> >>That suppose to be a joke ? >> >> > >The international joke symbol (8-)) is only mandatory in en_US locales. > > > Ok, Planty of times I did asked Brits and Americans if there's any difference. Neither did say that either disc or disk is more correct. And I am using both all the time, noone has any objections ( I live in UK for 2.5years so far ). So.... me is confused :-0 -- GJ ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-03 16:41 ` Grzegorz Jaśkiewicz @ 2004-09-03 23:46 ` Paul Jackson 0 siblings, 0 replies; 1392+ messages in thread From: Paul Jackson @ 2004-09-03 23:46 UTC (permalink / raw) To: Grzegorz Ja__kiewicz; +Cc: alan, miquels, linux-kernel > Ok, Planty of times I did asked Brits and Americans if there's any > difference. Neither did say that either disc or disk is more correct. >From http://en.wikipedia.org/wiki/Disk_or_disc Disk or Disc? The divergence in spelling is due in part to the way in which the words originated. Disk came into the English language in the mid-17th century, and was modelled on words such as whisk; disc arose some time later, and was based on the original Latin root discus. In the 19th century, disc became the conventional spelling for audio recordings made on a flat plate, such as the gramophone record; this usage gave rise to the modern term disc jockey. Early BBC technicians differentiated between disks (in-house transcription records) and discs (the colloquial term for commercial gramophone records, or what the BBC dubbed CGRs). By the 20th century, the c-spelling was more popular in British English, while the k-spelling was preferred in American English. In the 1940s, when the American company IBM pioneered the first hard disk storage devices, the k-spelling was used. In 1979 the European company Philips, along with Sony, developed the compact disc medium; here, the c-spelling was chosen, possibly because of the predominating British spelling, or because the compact disc was seen as a successor to the analogue disc record. Whatever their heritage, in computer jargon today it is common for the k-spelling to refer mainly to magnetic storage devices, while the c-spelling is customary for optical media such as the compact disc and similar technologies. Even in the computing field, however, the terms are used inconsistently; software documentation often uses the k-spelling exclusively. -- I won't rest till it's the best ... Programmer, Linux Scalability Paul Jackson <pj@sgi.com> 1.650.933.1373 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-03 9:14 ` Grzegorz Jaśkiewicz 2004-09-03 9:58 ` Miquel van Smoorenburg @ 2004-09-03 10:43 ` Greg KH 2004-09-08 9:09 ` Helge Hafting 2 siblings, 0 replies; 1392+ messages in thread From: Greg KH @ 2004-09-03 10:43 UTC (permalink / raw) To: Grzegorz Ja??kiewicz Cc: Helge Hafting, Jamie Lokier, Horst von Brand, linux-fsdevel, linux-kernel, reiserfs-list On Fri, Sep 03, 2004 at 11:14:51AM +0200, Grzegorz Ja??kiewicz wrote: > On Fri, 03 Sep 2004 10:54:26 +0200, Helge Hafting <helge.hafting@hist.no> wrote: > > Grzegorz Ja??kiewicz wrote: > > > > > > > >devfs was very natural, and simple solution. But to have it right, it > > >would have to be the only /dev filesystem. > > >But no, we like choices, so we have chaos. > > >Udev is just another thing adding to that chaos. > > > > > >Someone was numbering things that are good in BSD design, in that > > >thread. One of those things was going for devfs. No cheap solutions. > > >One fs for /dev. And it works great. > > > > > >Sorry for bit of trolling. > > > > > > > > Devfs was a ver good idea. The implementation of it > > was a problem, and after some time nobody maintained it. > > No surprise it had to go. Now udev+tmpfs can do the same > > job, and more. > > udef is a one big mistake, having need for userspace tool to use FS is > at least silly. I have never heard of a program called "udef". Any pointers to it? :) If it's such a big mistake, then don't use it. No one is forcing you to. > I can understeand need for some things in kernel to have userspace > daemon. But FS is out of question the least one. It's not a daemon (although to make things a bit easier on itself, it does provide one by default, but some distros don't use it.) It just creates device nodes when the kernel finds a new device, with a name that you pick. Which is something that an in-kernel devfs can not do. > I am supprised noone wanted to maintain devfs. Maybe because people > didn't want to go to devfs only. But still to have classic /dev. It's > also silly, because person writing driver needs to choose between, or > implement all. That's more than bad. Once I have loads of time, and no > work in KDE, I can take over devfs happily :-) Well, until you do that, baseless criticism and trolling like this will be ignored. greg k-h ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-03 9:14 ` Grzegorz Jaśkiewicz 2004-09-03 9:58 ` Miquel van Smoorenburg 2004-09-03 10:43 ` Greg KH @ 2004-09-08 9:09 ` Helge Hafting 2004-09-08 15:25 ` Tonnerre 2 siblings, 1 reply; 1392+ messages in thread From: Helge Hafting @ 2004-09-08 9:09 UTC (permalink / raw) To: Grzegorz Jaśkiewicz Cc: Greg KH, Jamie Lokier, Horst von Brand, linux-fsdevel, linux-kernel, reiserfs-list Grzegorz Jaśkiewicz wrote: >On Fri, 03 Sep 2004 10:54:26 +0200, Helge Hafting <helge.hafting@hist.no> wrote: > > >>Grzegorz Jaśkiewicz wrote: >> >> >> >>>devfs was very natural, and simple solution. But to have it right, it >>>would have to be the only /dev filesystem. >>>But no, we like choices, so we have chaos. >>>Udev is just another thing adding to that chaos. >>> >>>Someone was numbering things that are good in BSD design, in that >>>thread. One of those things was going for devfs. No cheap solutions. >>>One fs for /dev. And it works great. >>> >>>Sorry for bit of trolling. >>> >>> >>> >>> >>Devfs was a ver good idea. The implementation of it >>was a problem, and after some time nobody maintained it. >>No surprise it had to go. Now udev+tmpfs can do the same >>job, and more. >> >> > >udef is a one big mistake, having need for userspace tool to use FS is >at least silly. > > Well, devfs had devfsd - a userspace tool . . . >I can understeand need for some things in kernel to have userspace >daemon. But FS is out of question the least one. > >I am supprised noone wanted to maintain devfs. > I believe it had soemthing to do with the design - in order to fix it you had to rewrite it almost from scratch. People work on whatever they want to, and devfs wasn't it. >Maybe because people >didn't want to go to devfs only. But still to have classic /dev. It's >also silly, because person writing driver needs to choose between, or >implement all. That's more than bad. Once I have loads of time, and no >work in KDE, I can take over devfs happily :-) > > Go ahead! Perhaps you get it right. Then you'll have to convince users of udev (or plain old /dev) that your way is better. Helge Hafting ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-08 9:09 ` Helge Hafting @ 2004-09-08 15:25 ` Tonnerre 0 siblings, 0 replies; 1392+ messages in thread From: Tonnerre @ 2004-09-08 15:25 UTC (permalink / raw) To: Helge Hafting Cc: Grzegorz Ja??kiewicz, Greg KH, Jamie Lokier, Horst von Brand, linux-fsdevel, linux-kernel, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 657 bytes --] Salut, On Wed, Sep 08, 2004 at 11:09:06AM +0200, Helge Hafting wrote: > >udef is a one big mistake, having need for userspace tool to use FS is > >at least silly. > > Well, devfs had devfsd - a userspace tool . . . devfsd was not a requirement, it was only a tool for people who wanted their old device names for devfs devices. I was once doing a miniature environment for a package installation system which used pure devfs (i.e. no old dev names), and it was doing super well without devfs. sysinstall was the only process running, and I could install Linux on any writeable device without using devfsd. Tonnerre [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 19:23 ` Linus Torvalds 2004-08-26 19:44 ` Rik van Riel 2004-08-26 20:10 ` Rik van Riel @ 2004-08-27 8:06 ` Hans Reiser 2 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-27 8:06 UTC (permalink / raw) To: Linus Torvalds Cc: Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Linus Torvalds wrote: > > >To me, a filesystem that allows this thing doesn't really _have_ the >concept of "directory vs file". It's just a "filesystem object", and it >can act as _both_ a directory and a file. > > > I agree. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:59 ` Rik van Riel 2004-08-26 19:14 ` Christophe Saout 2004-08-26 19:23 ` Linus Torvalds @ 2004-08-26 19:29 ` Jamie Lokier 2004-08-26 19:42 ` Christophe Saout 2 siblings, 1 reply; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 19:29 UTC (permalink / raw) To: Rik van Riel Cc: Linus Torvalds, Diego Calleja, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Rik van Riel wrote: > What do we do with O_CREAT ? > > Do we always allow both a directory and a file to be created with > the same name ? I haven't analysed it thoroughly. I suspect it's fine as long as: (1) O_CREAT creates something with "file-like attributes", meaning stat() says it's a regular file. (2) File-like means it can be unlinked, linked and renamed, even if someone has something inside it open. Nothing that can be created inside it will prevent it from being unlinked (unlike a directory-like object, where rmdir() will refuse if it's not empty). File-like also means that programs like "cp from to_my_new_file" won't do anything so silly as to write _inside_, the way that "cp from to_my_new_file/from" would. Basic utilities will need to be checked to make sure they don't try appending "/" as their method of deciding if the target object is a directory and should be entered. > Does this create a new class of "symlink attack" style security > holes ? Yes, but they don't need O_CREAT. An adversary creates a symlink to metadata inside your file. You write to it: it has interesting effects that weren't anticipated, such as either modifying another (virtual) file, or altering permissions or other parameters which writing doesn't normally do. This is very difficult to prevent. We are creating a way for scripts and classic unix tools to have easy access to fancy attributes which may affect the security: things like being able to change a file's permissions just by writing to the appropriate path, or reading characteristics of files that shouldn't be visible. In Hans Reiser's example of expanded /etc/passwd, atimes and mtimes of individual passwd entries is security information that perhaps shouldn't be exposed. In a way, these holes are similar to /proc, in that just writing has side effects which might not be acceptable. The solution is the same as for /proc (I hope): make sure the read permissions on all metadata inside a directory branch are restricted to the permissions of the file branch, and write permissions even more restricted at least by default. For the paranoid, one way is to make new files "not readable or executable as directories". It's possible for the file and directory branches to have different permission bits, though I could see that getting a bit annoying. Other possibilities abound. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 19:29 ` Jamie Lokier @ 2004-08-26 19:42 ` Christophe Saout 0 siblings, 0 replies; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 19:42 UTC (permalink / raw) To: Jamie Lokier Cc: Rik van Riel, Linus Torvalds, Diego Calleja, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 2037 bytes --] Am Donnerstag, den 26.08.2004, 20:29 +0100 schrieb Jamie Lokier: > (1) O_CREAT creates something with "file-like > attributes", meaning stat() says it's a regular file. > > (2) File-like means it can be unlinked, linked and renamed, even if > someone has something inside it open. Nothing that can be created > inside it will prevent it from being unlinked (unlike a > directory-like object, where rmdir() will refuse if it's not empty). I would say hat the file and all of its directory gets invalidated and deleted as soon as the last opener of the file's main stream or some of its directory content disappears. How would that be? > > Does this create a new class of "symlink attack" style security > > holes ? > > Yes, but they don't need O_CREAT. > > An adversary creates a symlink to metadata inside your file. You > write to it: it has interesting effects that weren't anticipated, such > as either modifying another (virtual) file, or altering permissions or > other parameters which writing doesn't normally do. Hmm. > This is very difficult to prevent. Right. But to change some properties you usually need to be the owner of that file anyway, or root. > In Hans Reiser's example of expanded /etc/passwd, atimes > and mtimes of individual passwd entries is security information that > perhaps shouldn't be exposed. Only assuming mtime/ctime/atime/rwx/uid/gid of file contents can actually be independant of those from the main stream. I would say that the rights (read and write) for non-pseudo sub-files should the same as the file, uid and gid 644 (only owner can change these), something like that. mtime, atime and ctime always show the same as the file itself. Hmm. > The solution is the same as for /proc (I hope): make sure the read > permissions on all metadata inside a directory branch are restricted > to the permissions of the file branch, and write permissions even more > restricted at least by default. That's what I was thinking. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:55 ` Linus Torvalds 2004-08-26 18:59 ` Rik van Riel @ 2004-08-27 8:46 ` Herbert Poetzl 1 sibling, 0 replies; 1392+ messages in thread From: Herbert Poetzl @ 2004-08-27 8:46 UTC (permalink / raw) To: Linus Torvalds Cc: Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 11:55:07AM -0700, Linus Torvalds wrote: > > > On Thu, 26 Aug 2004, Rik van Riel wrote: > > > > So you'd have both a file and a directory that just happen > > to have the same name ? How would this work in the dcache? > > There would be only one entry in the dcache. The lookup will select > whether it opens the file or the directory based on O_DIRECTORY (and > usage, of course - if it's in the middle of a path, it obviously needs to > be opened as a directory regardless). hmm, that might be interesting for applications like apache which allow to append arguments by 'building' a 'virtual' path ... http://www.kernel.org/list.cgi/some/more/args where the web server actually decides where the script ends and the arguments start by checking the filesystem ... > That's not the problem. The problem from a dcache standpoint ends up being > when the file has a link, and you have two paths to the same sub-file > through two different ways: > > .. create file 'x' with named stream 'y' ... > ln x z > ls -l x/y z/y /* it's the same attribute!! */ > > but this is actually exactly the same thing that we already have with > mounts, ie it is equivalent (from a dentry standpoint) to > > .. create directory 'x' with file 'y' .. > mkdir z > mount --bind x z > ls -l x/y z/y /* It's the same file!! */ > > so none of this is really anything "new" from a dcache standpoint. > > Except for all the details, of course ;) if the file is removed, but some attributes are 'locked' by access, will the 'other' attributes remain visible or disappear ... leaving a partial 'view' of the contents? or did I get the idea completely wrong? TIA, Herbert > Linus > - > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:15 ` Linus Torvalds 2004-08-26 18:41 ` Rik van Riel @ 2004-08-26 20:44 ` Martin J. Bligh 2004-08-26 20:54 ` Linus Torvalds [not found] ` <412E33D6.5020500@slaphack.com> 2 siblings, 1 reply; 1392+ messages in thread From: Martin J. Bligh @ 2004-08-26 20:44 UTC (permalink / raw) To: Linus Torvalds, Rik van Riel Cc: Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list --On Thursday, August 26, 2004 11:15:19 -0700 Linus Torvalds <torvalds@osdl.org> wrote: > On Thu, 26 Aug 2004, Rik van Riel wrote: >> >> It's a relief to know that nobody's taking my humorous >> suggestion seriously, but now we still have the "standard >> Unix tools can't manipulate files" problem... > > I disagree. They can manipulate the files a whole lot better than they can > manipulate xattr's. > > For example, you _could_ probably (but hey, maybe "tar" tries to strip > slashes off the end of filenames, so this might not work due to silly > reasons like that) back up a compound file with > > tar cvf file.tar file file/ > > although unpacking it would require that tar be taught about the thing. > And you definitely could write a script to do the thing, ie even with an > unmodified tar you could do > > tar cvf file-archive.tar file > cd file > tar cvf ../attribute-archive.tar . > > which is a hell of a lot better than what you can do with the fsattr > interfaces and unmodified legacy applications. > > So one of the advantages of "dir-as-file/file-as-dir" is exactly that you > _can_ manipulate the data with legacy tools. Sure, things that traverse a > directory tree might need some (likely fairly trivial) modifications if > they really want to take advantage of the subfiles, but that's still > likely to be _much_ less of an issue than with fsattr's that have a > totally different model entirely. What would "test -d" and "test -f" return on these magic beasties? I can't think of any combinations that wouldn't confuse the crap out of userspace. M. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:44 ` Martin J. Bligh @ 2004-08-26 20:54 ` Linus Torvalds 2004-08-26 21:00 ` Martin J. Bligh ` (2 more replies) 0 siblings, 3 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-08-26 20:54 UTC (permalink / raw) To: Martin J. Bligh Cc: Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Martin J. Bligh wrote: > > What would "test -d" and "test -f" return on these magic beasties? I can't > think of any combinations that wouldn't confuse the crap out of userspace. "It's a feature". The S_ISDIR/S_ISREG tests show real information: it shows not only user intent ("you should consider this a file, even if it has attributes"), but also whether it is a directory or a container. And there's a real technical difference there: the streams contained within a file are bound to that file. The files contained within a directory are _independent_ of that directory. Big difference. HUGE difference. So it's not confusing. If it tests as a file, you think of it as a file. It may have attributes aka named streams associated with it, and you may be able to open those attributes by treating the file as a directory, but that doesn't really change the fact that it's a file. The _big_ difference is that when you can make the compound object _look_ like a directory, that means that you can now manage the attributes with standard tools. They are still attributes, though. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:54 ` Linus Torvalds @ 2004-08-26 21:00 ` Martin J. Bligh 2004-08-26 21:09 ` Christophe Saout ` (2 more replies) 2004-08-26 21:29 ` Chris Wright 2004-08-27 8:19 ` Anton Altaparmakov 2 siblings, 3 replies; 1392+ messages in thread From: Martin J. Bligh @ 2004-08-26 21:00 UTC (permalink / raw) To: Linus Torvalds Cc: Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > On Thu, 26 Aug 2004, Martin J. Bligh wrote: >> >> What would "test -d" and "test -f" return on these magic beasties? I can't >> think of any combinations that wouldn't confuse the crap out of userspace. > > "It's a feature". > > The S_ISDIR/S_ISREG tests show real information: it shows not only user > intent ("you should consider this a file, even if it has attributes"), but > also whether it is a directory or a container. > > And there's a real technical difference there: the streams contained > within a file are bound to that file. The files contained within a > directory are _independent_ of that directory. Big difference. HUGE > difference. > > So it's not confusing. If it tests as a file, you think of it as a file. > It may have attributes aka named streams associated with it, and you may > be able to open those attributes by treating the file as a directory, but > that doesn't really change the fact that it's a file. > > The _big_ difference is that when you can make the compound object _look_ > like a directory, that means that you can now manage the attributes with > standard tools. They are still attributes, though. I think what you're saying is that they'd both return positive, right? Unfortunately I think lots of code in userspace assumes files / directories are mutually exclusive conditions, and acts accordingly ... eg: if [ -f file ] do file stuff else do directory stuff fi. Now I'm not saying they're not broken ... but seems like a dangerous thing to change in the midst of a stable kernel series. M. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 21:00 ` Martin J. Bligh @ 2004-08-26 21:09 ` Christophe Saout 2004-08-26 21:13 ` Linus Torvalds 2004-08-26 22:44 ` Miquel van Smoorenburg 2 siblings, 0 replies; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 21:09 UTC (permalink / raw) To: Martin J. Bligh Cc: Linus Torvalds, Rik van Riel, Diego Calleja, jamie, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1459 bytes --] Am Donnerstag, den 26.08.2004, 14:00 -0700 schrieb Martin J. Bligh: > > "It's a feature". > > > > The S_ISDIR/S_ISREG tests show real information: it shows not only user > > intent ("you should consider this a file, even if it has attributes"), but > > also whether it is a directory or a container. > > > > And there's a real technical difference there: the streams contained > > within a file are bound to that file. The files contained within a > > directory are _independent_ of that directory. Big difference. HUGE > > difference. > > > > So it's not confusing. If it tests as a file, you think of it as a file. > > It may have attributes aka named streams associated with it, and you may > > be able to open those attributes by treating the file as a directory, but > > that doesn't really change the fact that it's a file. > > > > The _big_ difference is that when you can make the compound object _look_ > > like a directory, that means that you can now manage the attributes with > > standard tools. They are still attributes, though. > > I think what you're saying is that they'd both return positive, right? No. A file is still a file and S_ISREG will thus return false. The application should just try to open it as a directory if it wants to access the assiociated embedded attribute files. If the open fails, well, it knows that the filesystem (or Linux version, whatever) doesn't support this feature. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 21:00 ` Martin J. Bligh 2004-08-26 21:09 ` Christophe Saout @ 2004-08-26 21:13 ` Linus Torvalds 2004-08-26 21:24 ` Jamie Lokier 2004-08-27 9:36 ` David Greaves 2004-08-26 22:44 ` Miquel van Smoorenburg 2 siblings, 2 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-08-26 21:13 UTC (permalink / raw) To: Martin J. Bligh Cc: Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Martin J. Bligh wrote: > > I think what you're saying is that they'd both return positive, right? No. I'd say that a file would look like a file, even if it has attributes. It wouldn't show as a directory at all - unless you start looking at attributes. Because it really _is_ a file, and it's "directory aspect" is really nothing but a way to make its named streams visible. So you really should consider it a perfectly regular file, and so only S_ISREG() will return true, and S_ISDIR() will return false. Think of it this way: when you add a named attribute to current files using the xattr interfaces, do you start thinking of the file as a directory? No. Even though it actually now is a starting point for finding more information. It's just that thanks to it's directory aspect, people and apps that _care_ about the attributes suddenly can trivially access them. You can access them in shell scripts, you can access them in programs, you can access them in perl. With no special knowledge necessary. Remember: in the file-as-directory model, we're always talking about just _one_ object. That one object just happens to have namd streams or attributes or whatever you want to call them associated with it, and you can _see_ them through the normal directory interfaces. But that doesn't really make the object a "directory". It's still a file. In other words, the "directory" part is just a _view_ into the file. A view that potentially exposes a lot _more_ of the file, but we're still talking about the same file. In contrast, a S_IFDIR-like _directory_ is something else entirely. When you view the things in that, you aren't looking at data "inside" the directory. You're looking at somethign totally independent. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 21:13 ` Linus Torvalds @ 2004-08-26 21:24 ` Jamie Lokier 2004-08-27 9:36 ` David Greaves 1 sibling, 0 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 21:24 UTC (permalink / raw) To: Linus Torvalds Cc: Martin J. Bligh, Rik van Riel, Diego Calleja, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Linus Torvalds wrote: > In other words, the "directory" part is just a _view_ into the file. A > view that potentially exposes a lot _more_ of the file, but we're still > talking about the same file. > > In contrast, a S_IFDIR-like _directory_ is something else entirely. When > you view the things in that, you aren't looking at data "inside" the > directory. You're looking at somethign totally independent. One of the constraints is that you can create any name in an S_IFDIR-like directory, but you probably can't create any name "in" an S_IFREG-like file, because the names have meaning -- except in designated subtrees of the view intended for holding arbitrary names. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 21:13 ` Linus Torvalds 2004-08-26 21:24 ` Jamie Lokier @ 2004-08-27 9:36 ` David Greaves 1 sibling, 0 replies; 1392+ messages in thread From: David Greaves @ 2004-08-27 9:36 UTC (permalink / raw) To: Linus Torvalds Cc: Martin J. Bligh, Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Linus Torvalds wrote: >On Thu, 26 Aug 2004, Martin J. Bligh wrote: > > >>I think what you're saying is that they'd both return positive, right? >> >> > >No. I'd say that a file would look like a file, even if it has attributes. > >It wouldn't show as a directory at all - unless you start looking at >attributes. Because it really _is_ a file, and it's "directory aspect" is >really nothing but a way to make its named streams visible. > >So you really should consider it a perfectly regular file, and so only >S_ISREG() will return true, and S_ISDIR() will return false. > > > ie, logically: S_ISREG() no longer precludes O_DIRECTORY from succeeding conversley O_DIRECTORY succeeding no longer implies S_ISDIR() However, apps making those assumptions will not notice new behaviour. ? David ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 21:00 ` Martin J. Bligh 2004-08-26 21:09 ` Christophe Saout 2004-08-26 21:13 ` Linus Torvalds @ 2004-08-26 22:44 ` Miquel van Smoorenburg 2 siblings, 0 replies; 1392+ messages in thread From: Miquel van Smoorenburg @ 2004-08-26 22:44 UTC (permalink / raw) To: linux-kernel In article <57730000.1093554054@flay>, Martin J. Bligh <mbligh@aracnet.com> wrote: >> On Thu, 26 Aug 2004, Martin J. Bligh wrote: >>> >>> What would "test -d" and "test -f" return on these magic beasties? I can't >>> think of any combinations that wouldn't confuse the crap out of userspace. >> >I think what you're saying is that they'd both return positive, right? test -f file -> true test -f file/ -> false test -d file -> false test -d file/ -> true Mike. -- "In times of universal deceit, telling the truth becomes a revolutionary act." -- George Orwell. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:54 ` Linus Torvalds 2004-08-26 21:00 ` Martin J. Bligh @ 2004-08-26 21:29 ` Chris Wright 2004-08-27 8:19 ` Anton Altaparmakov 2 siblings, 0 replies; 1392+ messages in thread From: Chris Wright @ 2004-08-26 21:29 UTC (permalink / raw) To: Linus Torvalds Cc: Martin J. Bligh, Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list * Linus Torvalds (torvalds@osdl.org) wrote: > The _big_ difference is that when you can make the compound object _look_ > like a directory, that means that you can now manage the attributes with > standard tools. They are still attributes, though. For things that actually use such attributes to determine file access, do the attributes get attributes? thanks, -chris -- Linux Security Modules http://lsm.immunix.org http://lsm.bkbits.net ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:54 ` Linus Torvalds 2004-08-26 21:00 ` Martin J. Bligh 2004-08-26 21:29 ` Chris Wright @ 2004-08-27 8:19 ` Anton Altaparmakov 2 siblings, 0 replies; 1392+ messages in thread From: Anton Altaparmakov @ 2004-08-27 8:19 UTC (permalink / raw) To: Linus Torvalds Cc: Martin J. Bligh, Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, Andrew Morton, wichert, jra, reiser, hch, linux-fsdevel, lkml, flx, reiserfs-list On Thu, 2004-08-26 at 21:54, Linus Torvalds wrote: > The S_ISDIR/S_ISREG tests show real information: it shows not only user > intent ("you should consider this a file, even if it has attributes"), but > also whether it is a directory or a container. > > And there's a real technical difference there: the streams contained > within a file are bound to that file. The files contained within a > directory are _independent_ of that directory. Big difference. HUGE > difference. > > So it's not confusing. If it tests as a file, you think of it as a file. > It may have attributes aka named streams associated with it, and you may > be able to open those attributes by treating the file as a directory, but > that doesn't really change the fact that it's a file. What about the attributes/named streams of a directory though? If you open it as a directory, you would get the files inside the directory. So how do you get at the attributes and named streams of the directory itself using this interface? Best regards, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net WWW: http://linux-ntfs.sf.net/, http://www-stu.christs.cam.ac.uk/~aia21/ ^ permalink raw reply [flat|nested] 1392+ messages in thread
[parent not found: <412E33D6.5020500@slaphack.com>]
* Re: silent semantic changes with reiser4 [not found] ` <412E33D6.5020500@slaphack.com> @ 2004-08-27 8:06 ` Hans Reiser 2004-08-27 14:12 ` Rik van Riel 2004-08-27 14:45 ` Rik van Riel 0 siblings, 2 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-27 8:06 UTC (permalink / raw) To: David Masover Cc: Linus Torvalds, Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Why are you guys even considering going to any pain at all to distort semantics for the sake of backup? tar is easy, we'll fix it and send in a patch. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 8:06 ` Hans Reiser @ 2004-08-27 14:12 ` Rik van Riel 2004-08-27 18:27 ` Hans Reiser 2004-08-27 20:56 ` Christian Mayrhuber 2004-08-27 14:45 ` Rik van Riel 1 sibling, 2 replies; 1392+ messages in thread From: Rik van Riel @ 2004-08-27 14:12 UTC (permalink / raw) To: Hans Reiser Cc: David Masover, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, 27 Aug 2004, Hans Reiser wrote: > Why are you guys even considering going to any pain at all to distort > semantics for the sake of backup? tar is easy, we'll fix it and send in > a patch. Because not everybody uses tar. Quite a few people use a network backup system, while others use duplicity, RPM uses cpio internally and big companies tend to use proprietary network backup suites. Breaking people's setup is something to worry about. -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 14:12 ` Rik van Riel @ 2004-08-27 18:27 ` Hans Reiser 2004-08-27 18:44 ` Lee Revell 2004-08-27 19:13 ` Rik van Riel 2004-08-27 20:56 ` Christian Mayrhuber 1 sibling, 2 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-27 18:27 UTC (permalink / raw) To: Rik van Riel Cc: David Masover, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Rik van Riel wrote: >On Fri, 27 Aug 2004, Hans Reiser wrote: > > > >>Why are you guys even considering going to any pain at all to distort >>semantics for the sake of backup? tar is easy, we'll fix it and send in >>a patch. >> >> > >Because not everybody uses tar. Quite a few people use a >network backup system, while others use duplicity, RPM uses >cpio internally and big companies tend to use proprietary >network backup suites. > >Breaking people's setup is something to worry about. > > > It is the tail that should not wag the dog. New semantics are going to break backups other than dd. We need a LOT of new semantics if we don't want to be inferior to Apple and MS. We should implement backup plugin methods though, so that we only have to break things once.... ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 18:27 ` Hans Reiser @ 2004-08-27 18:44 ` Lee Revell 2004-08-27 19:13 ` Rik van Riel 1 sibling, 0 replies; 1392+ messages in thread From: Lee Revell @ 2004-08-27 18:44 UTC (permalink / raw) To: Hans Reiser Cc: Rik van Riel, David Masover, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, Andrew Morton, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, 2004-08-27 at 14:27, Hans Reiser wrote: > Rik van Riel wrote: > >On Fri, 27 Aug 2004, Hans Reiser wrote: > > > >>Why are you guys even considering going to any pain at all to distort > >>semantics for the sake of backup? tar is easy, we'll fix it and send in > >>a patch. > > > >Because not everybody uses tar. Quite a few people use a > >network backup system, while others use duplicity, RPM uses > >cpio internally and big companies tend to use proprietary > >network backup suites. > > > >Breaking people's setup is something to worry about. > > > It is the tail that should not wag the dog. New semantics are going to > break backups other than dd. We need a LOT of new semantics if we > don't want to be inferior to Apple and MS. > > We should implement backup plugin methods though, so that we only have > to break things once.... Every Linux shop I know of uses amanda anyway, which will not be affected by these changes, as it uses dump/restore. restore will need minor changes, I think dump might not have to be changed at all. If your backup system can't easily handle these changes, that's a pretty good indication that it's poorly designed. Lee ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 18:27 ` Hans Reiser 2004-08-27 18:44 ` Lee Revell @ 2004-08-27 19:13 ` Rik van Riel 2004-08-28 6:10 ` Hans Reiser 1 sibling, 1 reply; 1392+ messages in thread From: Rik van Riel @ 2004-08-27 19:13 UTC (permalink / raw) To: Hans Reiser Cc: David Masover, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, 27 Aug 2004, Hans Reiser wrote: > It is the tail that should not wag the dog. That's no reason for dismissing technical questions with vigorous handwaving. > New semantics are going to break backups other than dd. We need a LOT > of new semantics if we don't want to be inferior to Apple and MS. Technical objections need to be addressed, otherwise Linux will end up inferior to Apple and MS. Whatever choice we end up making, it should be something that's both workable and elegant. It would be really nice if all the technical issues were worked out on linux-kernel before the files-as-directories stuff gets merged. There are quite a few technical questions in the thread that haven't been answered yet... -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 19:13 ` Rik van Riel @ 2004-08-28 6:10 ` Hans Reiser 0 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-28 6:10 UTC (permalink / raw) To: Rik van Riel Cc: David Masover, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Rik van Riel wrote: > >It would be really nice if all the technical issues were >worked out on linux-kernel before the files-as-directories >stuff gets merged. There are quite a few technical questions >in the thread that haven't been answered yet... > > > Well, did I answer the technical question with the filename/pseudos/backup method answer in your opinion? ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 14:12 ` Rik van Riel 2004-08-27 18:27 ` Hans Reiser @ 2004-08-27 20:56 ` Christian Mayrhuber 1 sibling, 0 replies; 1392+ messages in thread From: Christian Mayrhuber @ 2004-08-27 20:56 UTC (permalink / raw) To: reiserfs-list; +Cc: linux-fsdevel, linux-kernel On Friday 27 August 2004 16:12, Rik van Riel wrote: > Because not everybody uses tar. Quite a few people use a > network backup system, while others use duplicity, RPM uses > cpio internally and big companies tend to use proprietary > network backup suites. Big backup systems support scripts that can be run prior to backup and post restore. You aready have to use star to do a metadata backup the acl's/ea into some tgz, which can be unpacked after restore. > Breaking people's setup is something to worry about. Yeah, maybe. But as usual on UN*X like systems most admins are able to come up with a quick shell/perl script to solve their particular problem. -- lg, Chris ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 8:06 ` Hans Reiser 2004-08-27 14:12 ` Rik van Riel @ 2004-08-27 14:45 ` Rik van Riel 2004-08-27 15:15 ` Spam 2004-08-27 18:28 ` Hans Reiser 1 sibling, 2 replies; 1392+ messages in thread From: Rik van Riel @ 2004-08-27 14:45 UTC (permalink / raw) To: Hans Reiser Cc: David Masover, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, 27 Aug 2004, Hans Reiser wrote: > Why are you guys even considering going to any pain at all to distort > semantics for the sake of backup? tar is easy, we'll fix it and send in > a patch. It's not as easy as you make it out, and not just because there are a few dozen backup programs that need fixing. The problem is more fundamental than that. Some of the file streams proposed need to be backed up, while others are alternative presentations of the file, which should not be backed up. Currently I see no way to distinguish between the stuff that should be backed up and the stuff that shouldn't. That problem needs to be resolved before we can even start thinking about fixing archivers... -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 14:45 ` Rik van Riel @ 2004-08-27 15:15 ` Spam [not found] ` <87llg0mnl0.fsf@uhoreg.ca> [not found] ` <20040827154156.GA31757@mail.shareable.org> 2004-08-27 18:28 ` Hans Reiser 1 sibling, 2 replies; 1392+ messages in thread From: Spam @ 2004-08-27 15:15 UTC (permalink / raw) To: Rik van Riel Cc: Hans Reiser, David Masover, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > On Fri, 27 Aug 2004, Hans Reiser wrote: >> Why are you guys even considering going to any pain at all to distort >> semantics for the sake of backup? tar is easy, we'll fix it and send in >> a patch. > It's not as easy as you make it out, and not just because > there are a few dozen backup programs that need fixing. > The problem is more fundamental than that. Some of the > file streams proposed need to be backed up, while others > are alternative presentations of the file, which should > not be backed up. No, not really. This is a user decision and should be options in the backup software. I don't think it is up to the kernel, filesystem, or the OS in general to decide what information the user want to retain or not. > Currently I see no way to distinguish between the stuff > that should be backed up and the stuff that shouldn't. > That problem needs to be resolved before we can even start > thinking about fixing archivers... The archivers should, as I said, allow the user to choose. It shouldn't be automatic. Default, should IMO be to store everything. ~S ^ permalink raw reply [flat|nested] 1392+ messages in thread
[parent not found: <87llg0mnl0.fsf@uhoreg.ca>]
* Re: silent semantic changes with reiser4 [not found] ` <87llg0mnl0.fsf@uhoreg.ca> @ 2004-08-28 0:02 ` Spam 0 siblings, 0 replies; 1392+ messages in thread From: Spam @ 2004-08-28 0:02 UTC (permalink / raw) To: Hubert Chan; +Cc: reiserfs-list, linux-fsdevel, linux-kernel >>>>>> "Spam" == Spam <spam@tnonline.net> writes: >> Rik van Riel wrote: >>> On Fri, 27 Aug 2004, Hans Reiser wrote: >>>> Why are you guys even considering going to any pain at all to >>>> distort semantics for the sake of backup? tar is easy, we'll fix it >>>> and send in a patch. >>> It's not as easy as you make it out, and not just because there are a >>> few dozen backup programs that need fixing. >>> The problem is more fundamental than that. Some of the file streams >>> proposed need to be backed up, while others are alternative >>> presentations of the file, which should not be backed up. Spam>> No, not really. This is a user decision and should be options in Spam>> the backup software. I don't think it is up to the kernel, Spam>> filesystem, or the OS in general to decide what information the Spam>> user want to retain or not. > Why not just define an attribute named something like "do-not-backup"? > Then whatever program that generates the thumbnail can automatically add > the do-not-backup bit, and the backup software knows to ignore it. > (Obviously, that bit should apply recursively down the subtree.) This is about the same idea as the archive flag in DOS/Windows environments. It didn't really work to well IMO. If meta files are definable by users/application then the backup system could create its own meta files with the specific information it needs. ~S ^ permalink raw reply [flat|nested] 1392+ messages in thread
[parent not found: <20040827154156.GA31757@mail.shareable.org>]
* Re: silent semantic changes with reiser4 [not found] ` <20040827154156.GA31757@mail.shareable.org> @ 2004-08-28 0:19 ` Spam 2004-08-28 1:59 ` Rik van Riel 0 siblings, 1 reply; 1392+ messages in thread From: Spam @ 2004-08-28 0:19 UTC (permalink / raw) To: Jamie Lokier Cc: Rik van Riel, Hans Reiser, David Masover, Linus Torvalds, Diego Calleja, christophe, vda, christer, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > Spam wrote: >> > The problem is more fundamental than that. Some of the >> > file streams proposed need to be backed up, while others >> > are alternative presentations of the file, which should >> > not be backed up. >> >> No, not really. This is a user decision and should be options in the >> backup software. I don't think it is up to the kernel, filesystem, >> or the OS in general to decide what information the user want to >> retain or not. > It is helpful for the OS, or a naming convention, to indicate what > _is information_ though. > It makes no sense to backup two or more copies of the _same > information_, and it makes even less sense to try to restore them as > it'll either be slow, fail (you can't always write to alternative > presentations), or cause unwanted side effects. > Just like when you backup a dynamic web site. You store the files > which the server is using. You don't use "wget" to store the > generated pages, that's not a useful backup and you can't restore from it. I do not agree. Everything can be considered information, even if it is derived from an already existing file. If the user wants to not backup certain things then that is an option to tell the backup program. >> > Currently I see no way to distinguish between the stuff >> > that should be backed up and the stuff that shouldn't. >> >> > That problem needs to be resolved before we can even start >> > thinking about fixing archivers... >> >> The archivers should, as I said, allow the user to choose. It >> shouldn't be automatic. Default, should IMO be to store everything. > Don't try to store different views of the same thing. > When you try to restore, you _won't_ necessarily get back what you stored. No, it would be restored as it were before with the meta information intact. > Whereas if you follow the OS's advice, and skip virtual files, then > backup and restore will recreate the filesystem, which is what you want. > _That's_ storing everything. It's what you want from a backup. It would be storing everything but the virtual files. > -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 0:19 ` Spam @ 2004-08-28 1:59 ` Rik van Riel 2004-08-28 2:05 ` Linus Torvalds 0 siblings, 1 reply; 1392+ messages in thread From: Rik van Riel @ 2004-08-28 1:59 UTC (permalink / raw) To: Spam Cc: Jamie Lokier, Hans Reiser, David Masover, Linus Torvalds, Diego Calleja, christophe, vda, christer, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Sat, 28 Aug 2004, Spam wrote: > > Whereas if you follow the OS's advice, and skip virtual files, then > > backup and restore will recreate the filesystem, which is what you want. > > > _That's_ storing everything. It's what you want from a backup. > > It would be storing everything but the virtual files. Thing is, there is no way to distinguish between what are virtual files and what are actual streams hidden inside a file. You don't know what should and shouldn't be backed up... Hans says there must be a way, somehow, but I haven't seen him tell this mailing list what exactly it is ;) -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 1:59 ` Rik van Riel @ 2004-08-28 2:05 ` Linus Torvalds 2004-08-28 9:53 ` Hans Reiser 2004-08-28 17:05 ` Helge Hafting 0 siblings, 2 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-08-28 2:05 UTC (permalink / raw) To: Rik van Riel Cc: Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro On Fri, 27 Aug 2004, Rik van Riel wrote: > > Thing is, there is no way to distinguish between what are > virtual files and what are actual streams hidden inside a > file. You don't know what should and shouldn't be backed > up... I think that lack of distinguishing poiwer is more serious for directories. The more I think I think about it, the more I wonder whether Solaris did things right - having a special operation to "cross the boundary". I suspect Solaris did it that way because it's a hell of a lot easier to do it like that, but regardless, it would solve the issue of real directories having both real children _and_ the "extra streams". Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 2:05 ` Linus Torvalds @ 2004-08-28 9:53 ` Hans Reiser 2004-08-28 11:21 ` Spam 2004-08-28 17:05 ` Helge Hafting 1 sibling, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-28 9:53 UTC (permalink / raw) To: Linus Torvalds Cc: Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro Just have a special name instead of a special boundary, or, better, have a filename/pseudos/backup method that outputs everything needed to backup the object "filename". Hans Linus Torvalds wrote: >On Fri, 27 Aug 2004, Rik van Riel wrote: > > >>Thing is, there is no way to distinguish between what are >>virtual files and what are actual streams hidden inside a >>file. You don't know what should and shouldn't be backed >>up... >> >> > >I think that lack of distinguishing poiwer is more serious for >directories. The more I think I think about it, the more I wonder whether >Solaris did things right - having a special operation to "cross the >boundary". > >I suspect Solaris did it that way because it's a hell of a lot easier to >do it like that, but regardless, it would solve the issue of real >directories having both real children _and_ the "extra streams". > > Linus > > > > ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 9:53 ` Hans Reiser @ 2004-08-28 11:21 ` Spam 0 siblings, 0 replies; 1392+ messages in thread From: Spam @ 2004-08-28 11:21 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, Rik van Riel, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro I am thinking that is there was a proper API for accessing the filesystem then this problem wouldn't arise because things could be done behind the curtains inside the API, instead of having all the tools to be rewritten to know. Think of FAT32, for example, the new filenames will be retained even if you use old tools on them. It is only if you mount it as a FAT16 volume they will be lost. Even if this is fixed for Reiser4 now. Next time someone wants to make changes like this we have the exact same problem yet again. > Just have a special name instead of a special boundary, or, better, have > a filename/pseudos/backup method that outputs everything needed to > backup the object "filename". > Hans > Linus Torvalds wrote: >>On Fri, 27 Aug 2004, Rik van Riel wrote: >> >> >>>Thing is, there is no way to distinguish between what are >>>virtual files and what are actual streams hidden inside a >>>file. You don't know what should and shouldn't be backed >>>up... >>> >>> >> >>I think that lack of distinguishing poiwer is more serious for >>directories. The more I think I think about it, the more I wonder whether >>Solaris did things right - having a special operation to "cross the >>boundary". >> >>I suspect Solaris did it that way because it's a hell of a lot easier to >>do it like that, but regardless, it would solve the issue of real >>directories having both real children _and_ the "extra streams". >> >> Linus >> >> >> >> ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 2:05 ` Linus Torvalds 2004-08-28 9:53 ` Hans Reiser @ 2004-08-28 17:05 ` Helge Hafting 2004-08-28 18:09 ` Linus Torvalds 1 sibling, 1 reply; 1392+ messages in thread From: Helge Hafting @ 2004-08-28 17:05 UTC (permalink / raw) To: Linus Torvalds Cc: Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro On Fri, Aug 27, 2004 at 07:05:35PM -0700, Linus Torvalds wrote: > > > On Fri, 27 Aug 2004, Rik van Riel wrote: > > > > Thing is, there is no way to distinguish between what are > > virtual files and what are actual streams hidden inside a > > file. You don't know what should and shouldn't be backed > > up... > > I think that lack of distinguishing poiwer is more serious for > directories. The more I think I think about it, the more I wonder whether > Solaris did things right - having a special operation to "cross the > boundary". > > I suspect Solaris did it that way because it's a hell of a lot easier to > do it like that, but regardless, it would solve the issue of real > directories having both real children _and_ the "extra streams". There are many ways of doing this. Several extra streams to a directory that aren't ordinary files in the directory? It seems to me that we can get a lot of nice functionality in a simpler way: Instead of thinking about a number of streams attached to something that is either an ordinary file or directory, just say that the only change will be that a directory may have a _single_ file stream in addition to being a plain directory. The conceptual change is smaller. Still you achieve multiple streams, the "main" stream in such a dir+file is the single stream attached to the directory. All the "extra streams" are files inside the (normal) directory. * This way, the extra streams aren't "special" in any way and should work fine with existing tools. * Serving such a filesystem to clients that use multiple streams will allow any number of streams per file for them * Moving the directory+file thing will move both the directory and the special stream, and all the extra streams follow because they're �part of the directory as usual�. If the VFS is to be extended in order to support file-as-directory (or vice versa) then hopefully it can be done in a simple way. Helge Hafting ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 17:05 ` Helge Hafting @ 2004-08-28 18:09 ` Linus Torvalds 2004-08-28 18:20 ` Rik van Riel ` (4 more replies) 0 siblings, 5 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-08-28 18:09 UTC (permalink / raw) To: Helge Hafting Cc: Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro On Sat, 28 Aug 2004, Helge Hafting wrote: > > > I think that lack of distinguishing power is more serious for > > directories. The more I think I think about it, the more I wonder whether > > Solaris did things right - having a special operation to "cross the > > boundary". > > > > I suspect Solaris did it that way because it's a hell of a lot easier to > > do it like that, but regardless, it would solve the issue of real > > directories having both real children _and_ the "extra streams". > > There are many ways of doing this. Several extra streams to a directory > that aren't ordinary files in the directory? Well.. Yes. We already have "." and "..", which are "special extra streams" in a sense. However, people expect them, and know to ignore them. The same wouldn't be true of new naming. > It seems to me that we can get a lot of nice functionality in a simpler way: > Instead of thinking about a number of streams attached to something > that is either an ordinary file or directory, just say that the only > change will be that a directory may have a _single_ file stream in > addition to being a plain directory. That doesn't really help us. What would the name be, and how could you avoid clashes? > If the VFS is to be extended in order to support file-as-directory (or > vice versa) then hopefully it can be done in a simple way. I'm pretty confident that we can extend the VFS layer to support named streams (see the technical discussion with Al, rather than the flames in this thread). I also clearly believe that it is worth it, but I'm starting to wonder if we should have a special open flag to make people select the stream. If you look at the Solaris interface, the _nice_ part about "openat()" is that you can do something like file = open(filename, O_RDONLY); if (file < 0) return -ENOENT; icon = openat(file, "icon", O_RDONLY | O_XATTR); if (icon < 0) icon = default_icon_file; .. and it will work regardless of whether "filename" is a directory or a regular file, if I've understood correctly. Now, I think that makes sense for several reasons: - single case - race-free (think "stat()" vs "fstat()" races). - I think we want to do "openat()" regardless of whether we ever support extended attributes or not ("openat()" is nice for doing "namei()" in user space even in the absense of any attributes or named streams). So what we can do is - implement openat() regardless, and expect to do the Solaris thing for it if we ever do streams. - _also_ support the "implied named attributes" for regular files, so that you don't have to use "openat()" to access them. Comments? Does anybody hate "openat()" for any reason (regardless of attributes)? We can easily support it, we'd just need to pass in the file to use as part of the "nameidata" thing or add an argument (it would also possibly be cleaner if we made "fs->pwd" be a "struct file"). Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 18:09 ` Linus Torvalds @ 2004-08-28 18:20 ` Rik van Riel 2004-08-28 18:29 ` viro ` (3 subsequent siblings) 4 siblings, 0 replies; 1392+ messages in thread From: Rik van Riel @ 2004-08-28 18:20 UTC (permalink / raw) To: Linus Torvalds Cc: Helge Hafting, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro On Sat, 28 Aug 2004, Linus Torvalds wrote: > Well.. Yes. We already have "." and "..", which are "special extra > streams" in a sense. However, people expect them, and know to ignore them. > The same wouldn't be true of new naming. If the streams that don't contain data weren't real files, but instead fifos, backup programs would ignore them fine. Tar seems to ignore special nodes ... -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 18:09 ` Linus Torvalds 2004-08-28 18:20 ` Rik van Riel @ 2004-08-28 18:29 ` viro 2004-08-28 18:44 ` Linus Torvalds 2004-08-28 22:29 ` Hans Reiser ` (2 subsequent siblings) 4 siblings, 1 reply; 1392+ messages in thread From: viro @ 2004-08-28 18:29 UTC (permalink / raw) To: Linus Torvalds Cc: Helge Hafting, Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list On Sat, Aug 28, 2004 at 11:09:38AM -0700, Linus Torvalds wrote: > Comments? Does anybody hate "openat()" for any reason (regardless of > attributes)? We can easily support it, we'd just need to pass in the file > to use as part of the "nameidata" thing or add an argument (it would also > possibly be cleaner if we made "fs->pwd" be a "struct file"). What would your openat() produce? Normal struct file? Then what's going to be its vfsmount/dentry and what will they be attached to? ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 18:29 ` viro @ 2004-08-28 18:44 ` Linus Torvalds 2004-08-28 18:56 ` viro 0 siblings, 1 reply; 1392+ messages in thread From: Linus Torvalds @ 2004-08-28 18:44 UTC (permalink / raw) To: viro Cc: Helge Hafting, Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list On Sat, 28 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > > What would your openat() produce? Normal struct file? Then what's going > to be its vfsmount/dentry and what will they be attached to? Normal file descriptor, exactly like "open()". And it's going to have all the same vfsmount/dentry thing it would have if you looked it up the whole way. I don't understand your question.. Ignore the O_XATTR thing for a while, and assume it's just a convenient combination of "fchdir + open" (plus "fchdir back" of course, but that's beside the point - openat() doesn't ever really change cwd). Going back to O_XATTR: that would end up doing the "special vfsmount" magic at the beginning of the lookup. If the dentry you started with wasn't marked D_HYBRID, it would just return -ENOTDIR. So we could do openat() _without_ any of the lookup_mnt() etc special cases. This interface is independent of whether we want to expose the attributes through a normal lookup - we can do either, both, or neither as we choose. The nice thing about "openat()" is - people can definitely find uses for it even without attributes. - it's "portable". Well, at least somebody else does the same thing, which is nice for user-space developers. You don't use a Linux-only interface, you use a Linux/Solaris one, which makes a lot of people a lot more happy. Remember, portability has always been very important to Linux, and Linux-only features while nice are certainly not as nice as features you can also find in other places. NIH is a disease. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 18:44 ` Linus Torvalds @ 2004-08-28 18:56 ` viro 2004-08-28 19:16 ` Linus Torvalds 2004-08-29 0:46 ` Paul Jakma 0 siblings, 2 replies; 1392+ messages in thread From: viro @ 2004-08-28 18:56 UTC (permalink / raw) To: Linus Torvalds Cc: Helge Hafting, Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list On Sat, Aug 28, 2004 at 11:44:52AM -0700, Linus Torvalds wrote: > Going back to O_XATTR: that would end up doing the "special vfsmount" > magic at the beginning of the lookup. If the dentry you started with > wasn't marked D_HYBRID, it would just return -ENOTDIR. OK, let me restate the question - what do we get from pwd if we do fchdir() to such beast? ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 18:56 ` viro @ 2004-08-28 19:16 ` Linus Torvalds 2004-08-28 19:41 ` Andries Brouwer 2004-08-28 19:45 ` viro 2004-08-29 0:46 ` Paul Jakma 1 sibling, 2 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-08-28 19:16 UTC (permalink / raw) To: viro Cc: Helge Hafting, Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list On Sat, 28 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > > On Sat, Aug 28, 2004 at 11:44:52AM -0700, Linus Torvalds wrote: > > Going back to O_XATTR: that would end up doing the "special vfsmount" > > magic at the beginning of the lookup. If the dentry you started with > > wasn't marked D_HYBRID, it would just return -ENOTDIR. > > OK, let me restate the question - what do we get from pwd if we do > fchdir() to such beast? We'll see the "attribute path". We could (if we want to) mark the point where we walked into attribute space somehow (since we can see it by just looking at the vfsmount: d_root/d_mountpoint being the same), but even if we don't, we'll get a sane-looking path. (It's not just "getcwd()", it's /proc interfaces too, for opening files, and we already have the notion of magic markers like "(deleted)" to show human-readable information). The question about what to do at the "attribute point" (and there may actually be several, if the filesystem supports attributes on attribute files) likely depends on whether we support the previously discussed "lookup()" magic for attributes. So if we do support it, I think we should just make the attribute point look exactly like a normal directory, since that path would work as-is. If we don't support it (and at directory points), we might want to just consider it a special root. NOTE! Anybody who tries to use the "getcwd()" string as a real path is already broken - you have pathname permissions that may not make it possible to look up the path you see. So we have multiple options: - support "file/attribute" lookup: show the path as-is, so you'd show it as "/path/to/file/attribute" - alternatively, even if you _do_ support the normal lookup, show it with a double slash (which will still be a valid path), just as a visual clue: "/path/to/file//attribute" - for directories, or if we do _not_ support the extended lookup format, we could show it the same way we show deleted files, something like this: "/path/to/file/attribute (attr)" - using "http notation" for non-standard-namespaces (we already do this for sockets and pipes, for example) "attr:attribute@/path/to/file" pick your poison. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 19:16 ` Linus Torvalds @ 2004-08-28 19:41 ` Andries Brouwer 2004-08-28 19:46 ` Linus Torvalds 2004-08-28 19:45 ` viro 1 sibling, 1 reply; 1392+ messages in thread From: Andries Brouwer @ 2004-08-28 19:41 UTC (permalink / raw) To: Linus Torvalds Cc: viro, Helge Hafting, Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list On Sat, Aug 28, 2004 at 12:16:47PM -0700, Linus Torvalds wrote: If I see it correctly, you want to group a file and some ancillary files together. The Unix way would be to make a directory and put them all there: xterm/xterm xterm/xterm.icon But you are unsatisfied and want xterm xterm/xterm.icon As long as we agree that the latter really means the former, there are no problems in finding out what should happen. The conclusion is, that a directory carries an additional bit that says "if I am opened as a regular file then use the file of the same name inside". Now there is no attribute space, just a shorthand. Andries ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 19:41 ` Andries Brouwer @ 2004-08-28 19:46 ` Linus Torvalds 2004-08-29 13:56 ` Andries Brouwer 0 siblings, 1 reply; 1392+ messages in thread From: Linus Torvalds @ 2004-08-28 19:46 UTC (permalink / raw) To: Andries Brouwer Cc: viro, Helge Hafting, Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list On Sat, 28 Aug 2004, Andries Brouwer wrote: > > If I see it correctly, you want to group a file and some > ancillary files together. > > The Unix way would be to make a directory and put them all there: > xterm/xterm > xterm/xterm.icon > > But you are unsatisfied and want > xterm > xterm/xterm.icon > > As long as we agree that the latter really means the former, > there are no problems in finding out what should happen. > > The conclusion is, that a directory carries an additional bit > that says "if I am opened as a regular file then use the file > of the same name inside". > > Now there is no attribute space, just a shorthand. It's more than a shorthand, though. _Much_ more. There's the small issue of atomicity and locking. There's the small issue of hardlinks. Both of those are why it would need special support. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 19:46 ` Linus Torvalds @ 2004-08-29 13:56 ` Andries Brouwer 0 siblings, 0 replies; 1392+ messages in thread From: Andries Brouwer @ 2004-08-29 13:56 UTC (permalink / raw) To: Linus Torvalds Cc: Andries Brouwer, viro, Helge Hafting, Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list On Sat, Aug 28, 2004 at 12:46:10PM -0700, Linus Torvalds wrote: > > Now there is no attribute space, just a shorthand. > > It's more than a shorthand, though. _Much_ more. > ... > Both of those are why it would need special support. Yes - support, I do not argue against that. But I argue against an attribute space. This xterm.icon was xterm.icons - with both large and small icons. And these icons have copyrights. Etc. There is just an arbitrary tree below this xterm filesystem object. The files below are in no way different from the files above. Unix says: everything is a file. Now we get: everything is a directory. And some directories have a file attached. Andries ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 19:16 ` Linus Torvalds 2004-08-28 19:41 ` Andries Brouwer @ 2004-08-28 19:45 ` viro 2004-08-29 5:12 ` Linus Torvalds 1 sibling, 1 reply; 1392+ messages in thread From: viro @ 2004-08-28 19:45 UTC (permalink / raw) To: Linus Torvalds Cc: Helge Hafting, Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list On Sat, Aug 28, 2004 at 12:16:47PM -0700, Linus Torvalds wrote: > So if we do support it, I think we should just make the attribute point > look exactly like a normal directory, since that path would work as-is. If > we don't support it (and at directory points), we might want to just > consider it a special root. > > NOTE! Anybody who tries to use the "getcwd()" string as a real path is > already broken - you have pathname permissions that may not make it > possible to look up the path you see. OK, forget getcwd(). What does lookup of .. do from that point? *Especially* for stuff you've got from regular files. That's the decision that needs to be made. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 19:45 ` viro @ 2004-08-29 5:12 ` Linus Torvalds 2004-08-29 5:44 ` Jeremy Allison 0 siblings, 1 reply; 1392+ messages in thread From: Linus Torvalds @ 2004-08-29 5:12 UTC (permalink / raw) To: viro Cc: Helge Hafting, Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list On Sat, 28 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > > OK, forget getcwd(). What does lookup of .. do from that point? *Especially* > for stuff you've got from regular files. That's the decision that needs to > be made. I think that will decide on whether we expose attributes through the normal namespace or not. If we do expose them in the normal namespace, then ".." should work the way the namespace looks: if you do ".." on the "attribute directory" of a file, you get the directory that the file was in. Ie an old-style user-space "getcwd()" would give the right path (well, an old-style user-space getcwd() would probably refuse the file on the base that it is S_IFREG, but ignoring that..) If we _don't_ expose it in the normal namespace, we should should either just error out (logically you'd get the file itself, but I really don't wan tto have ".." return a non-directory, because _that_ really might confuse things), or you'd just return the same directory (ie it would be a "local root" in the namespace you got moved to). So let's try to be self-consistent with how we expose it in the normal namespace. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 5:12 ` Linus Torvalds @ 2004-08-29 5:44 ` Jeremy Allison 0 siblings, 0 replies; 1392+ messages in thread From: Jeremy Allison @ 2004-08-29 5:44 UTC (permalink / raw) To: Linus Torvalds Cc: viro, Helge Hafting, Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list On Sat, Aug 28, 2004 at 10:12:39PM -0700, Linus Torvalds wrote: > > If we do expose them in the normal namespace, then ".." should work the > way the namespace looks: if you do ".." on the "attribute directory" of a > file, you get the directory that the file was in. Ie an old-style > user-space "getcwd()" would give the right path (well, an old-style > user-space getcwd() would probably refuse the file on the base that it is > S_IFREG, but ignoring that..) Errr, why don't we check what Solaris does in these cases and see if that makes sense. They do seem to have made some progress along this path and it wouldn't make sense to create new semantics just for the sake of it. Jeremy. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 18:56 ` viro 2004-08-28 19:16 ` Linus Torvalds @ 2004-08-29 0:46 ` Paul Jakma 1 sibling, 0 replies; 1392+ messages in thread From: Paul Jakma @ 2004-08-29 0:46 UTC (permalink / raw) To: viro Cc: Linus Torvalds, Helge Hafting, Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list On Sat, 28 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > OK, let me restate the question - what do we get from pwd if we do > fchdir() to such beast? On solaris: ENOTDIR from getcwd(), with 'pwd' executed inside a shell started with runat. .. entry exists, but doesnt lead anywhere, cd .. returning ENOTDIR too. regards, -- Paul Jakma paul@clubi.ie paul@jakma.org Key ID: 64A2FF6A Fortune: If you didn't get caught, did you really do it? ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 18:09 ` Linus Torvalds 2004-08-28 18:20 ` Rik van Riel 2004-08-28 18:29 ` viro @ 2004-08-28 22:29 ` Hans Reiser 2004-08-29 0:40 ` The Necessity of File Types (was silent semantic changes with reiser4) Alexander G. M. Smith ` (2 more replies) 2004-08-28 23:47 ` Jeremy Allison 2004-08-30 13:56 ` Helge Hafting 4 siblings, 3 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-28 22:29 UTC (permalink / raw) To: Linus Torvalds Cc: Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro I object to openat()..... My reason is that the things that distinguish between objects should be the names, not the choice of system call. The reason for this is that it improves closure and namespace unification to do so, because it allows all the objects to be accessed within the same namespace. Yes, it can be useful to allow a namespace to exclude some objects, but that exclusion should not be mandated. If you want to exclude, you should cd or chroot to /proc/nopseudos and find a view of the filesystem that excludes metas, or mount with -nopseuodos. Do you see why I say this? I can say a lot more about the damage of fragmenting namespaces into multiple apis.... Why look at xattrs....;-) Hans Linus Torvalds wrote: > > >I'm pretty confident that we can extend the VFS layer to support named >streams (see the technical discussion with Al, rather than the flames in >this thread). I also clearly believe that it is worth it, but I'm starting >to wonder if we should have a special open flag to make people select the >stream. > >If you look at the Solaris interface, the _nice_ part about "openat()" is >that you can do something like > > file = open(filename, O_RDONLY); > if (file < 0) > return -ENOENT; > icon = openat(file, "icon", O_RDONLY | O_XATTR); > if (icon < 0) > icon = default_icon_file; > .. > >and it will work regardless of whether "filename" is a directory or a >regular file, if I've understood correctly. > >Now, I think that makes sense for several reasons: > - single case > - race-free (think "stat()" vs "fstat()" races). > - I think we want to do "openat()" regardless of whether we ever > support extended attributes or not ("openat()" is nice for doing > "namei()" in user space even in the absense of any attributes or > named streams). > >So what we can do is > - implement openat() regardless, and expect to do the Solaris thing for > it if we ever do streams. > - _also_ support the "implied named attributes" for regular files, so > that you don't have to use "openat()" to access them. > >Comments? Does anybody hate "openat()" for any reason (regardless of >attributes)? We can easily support it, we'd just need to pass in the file >to use as part of the "nameidata" thing or add an argument (it would also >possibly be cleaner if we made "fs->pwd" be a "struct file"). > > Linus >- >To unsubscribe from this list: send the line "unsubscribe linux-kernel" in >the body of a message to majordomo@vger.kernel.org >More majordomo info at http://vger.kernel.org/majordomo-info.html >Please read the FAQ at http://www.tux.org/lkml/ > > > > ^ permalink raw reply [flat|nested] 1392+ messages in thread
* The Necessity of File Types (was silent semantic changes with reiser4) 2004-08-28 22:29 ` Hans Reiser @ 2004-08-29 0:40 ` Alexander G. M. Smith 2004-08-31 0:42 ` David Masover 2004-08-29 5:01 ` silent semantic changes with reiser4 Linus Torvalds 2004-08-30 15:00 ` John Stoffel 2 siblings, 1 reply; 1392+ messages in thread From: Alexander G. M. Smith @ 2004-08-29 0:40 UTC (permalink / raw) To: Hans Reiser; +Cc: linux-fsdevel, linux-kernel, reiserfs-list, torvalds Linus Torvalds wrote: > I'm pretty confident that we can extend the VFS layer to support > named streams [...] Hans Reiser wrote on Sat, 28 Aug 2004 15:29:33 -0700: > I object to openat()..... > > My reason is that the things that distinguish between objects should be > the names, not the choice of system call. The reason for this is that > it improves closure and namespace unification to do so, because it > allows all the objects to be accessed within the same namespace. I have to agree with Hans there. Just have one main kind of file system object. No separate stream system, other than as an optional alternative way of viewing things. Treat it as a file or a directory or as an attribute as the context implies. Have just one open call to open it, another to enumerate its children, and another to read and write data to it. Paths to file system objects, no matter what their purpose, are all just plain paths (you can get fancy later with sub-name-spaces and name space boundaries). But how do you tell what kind of thing it is? How do you tell if it is important enough to back up? How do you tell that a thing is intended to be treated as a directory containing other things rather than as a stand-alone object? File types are needed for that! In BeOS we had a MIME type database and all important files were given an attribute of BEOS:TYPE which contained their MIME string. A system-wide database associated applications that handled each type, and also listed the standard attributes for each MIME type (such as Subject/From/To/... for "text/e-mail" files). It even included some extra info about which attributes should be shown to the user and which were editable by the user in the desktop directory window GUI. By the way, the database was just a tree of files (one for each MIME type using the same structure as MIME names) with attributes attached to them containing the relevant meta-meta-data. Unclassified files (ones with no type, such as those generated by "cp" and other non-attribute aware programs) were treated as application/octet-stream. There was also a background task that would try to identify them when the computer was idle, or if you tried to open them in a file requestor. It used the file extension and also looked for characteristic signature bytes to pick the type. Unfortunately BeOS had fragmented the type system by having a separate system of 4 character codes to identify the type of attributes (since attributes had their own API separate from files). That would tell you the attribute was an INT32 or C-String or Date. I'd like to see one unified type system for the unified file/directory/attribute thing (has anyone named those multipurpose file system objects yet or can I call them Fildirutes?). That means a global list of types and an attribute attached to all files needing a type. That standard child file (used as an attribute) would be named something like "..meta.mimetype". The global list would contain information about whether that type of thing needed to be backed up, what its standard computed attributes would be, and all sorts of common properties that all files of that type need. Plus I'd add the primitive types, so "primitive/int32" and "primitive/date64" would be possible types for things which are simple attributes. If you're worried about extra files, those file types could be stored as a unique ID (I recommend 8 character codes, using semi-readable text, for portability reasons) in each file inode. The type would only appear as a file to users of the file system. Another need for the type system is the need to index attributes. Someone has to tell the OS that "MAIL:date-received" is a 64 bit number of microseconds since the start of time, so that the OS knows how to sort the index for that attribute. Jan Harkes wrote on Sat, 28 Aug 2004 12:11:14 -0400: > From what I saw in one part of the discussion, it allows for infinite > depth recursion (file/metas/metas/metas/...). Some applications are > going to have a lot of fun with that. Some of the type information can be faked out as appearing as files. So looking for the MIME type of a MIME type file (thing/..meta.mimetype/..meta.mimetype) would pretend to be a file that contained "text/mime-string", but not actually represent anything on disk. Instead stock answers for the really primitive data types would be generated by the file system, recursively if needed. The recursion should be there. Just like it is for "." and "..", which "ls -R" for some reason doesn't traverse :-) By the way, I disagree on Hans with using a directory for the meta data, I'd prefer to just have "..meta." as the name prefix of the metadata things. Otherwise there's an extra directory level in there, making ..meta/ different from other directories (making browsing it a bit awkward since it doesn't have metadata of its own - like window coordinates for displaying the directory). - Alex ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: The Necessity of File Types (was silent semantic changes with reiser4) 2004-08-29 0:40 ` The Necessity of File Types (was silent semantic changes with reiser4) Alexander G. M. Smith @ 2004-08-31 0:42 ` David Masover 0 siblings, 0 replies; 1392+ messages in thread From: David Masover @ 2004-08-31 0:42 UTC (permalink / raw) To: Alexander G. M. Smith Cc: Hans Reiser, linux-fsdevel, linux-kernel, reiserfs-list, torvalds -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Alexander G. M. Smith wrote: [...] | By the way, I disagree on Hans with using a directory for the meta data, I'd prefer to just have "..meta." as the name prefix of the metadata things. Otherwise there's an extra directory level in there, making ..meta/ different from other directories (making browsing it a bit awkward since it doesn't have metadata of its own - like window coordinates for displaying the directory). You'd rather do "cat ..meta.pseudo" than "ls ..metas/"? And we already have weird directories. /proc doesn't do things that /tmp does. I also doubt proc will ever store that kind of metadata. I don't like either "metas" or "..metas", but you'll have to look in the archives for why. I am NOT starting that again. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTPJW3gHNmZLgCUhAQKuEw/+PjrDfSlZvuTUwiOSa/kU05qjTKkIp05F nBkrSIUKSUfoYN3gILucIGOjjJ5qBRW3wLIwWyO+raOAzHFTdylvR1rEWP9h/JUG Psz3X8RE8cubiuNwjyJawI87cPX8HW1xQxgkiiquZN2L5XbvHsFsbwAJV1tJLnrM 2lL6IBUXcxDIh7pdB0/z+4t945ElT3Rl09CLP/MfP4GNvt/UWjhbeHNWdeLq93C+ w0irn6VIsm+1+GKqEhpYLLMj9V4cbBY6m7XMuBzH+gEyu5exAMqGzPLGDiG/I6y/ rXI6IF94i2p4sLMv0LmtEvQsV37LFIKRBrWVScV5oiQ93LPVLG6pYmAi6RLsHN03 QazMmcRwjVvrTAf0KS21KXaE0qYbvgoQpteUEkPZLRkczSmiry2d3e6gBQlCV7xj 9Hzkm9HxR0BjxmRaMVLEWZ86t08OHkl0Ab186DiHUC5RlL2FPIvdyYthXo9wwtI/ XlA03Rvc3kl6DVRO8UM/+fa8v5MDn58ZMI2QgdPrU0V2Q9bGv/woByszXaDA+IVJ ZiHQ8IgShYlkA279DffOrZnDPnZMOWa7iRYetX5PYPoAyfkrmFKofT8va+dDL7pf HA5crUR5ws1EZZ8lUfrtXXnsBeEn9PS0lUHvJY7zZ4tJfZ65Fa+IiDpK5p+1tJZi ylCBNEidK/8= =CiZ9 -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 22:29 ` Hans Reiser 2004-08-29 0:40 ` The Necessity of File Types (was silent semantic changes with reiser4) Alexander G. M. Smith @ 2004-08-29 5:01 ` Linus Torvalds 2004-08-29 9:36 ` Hans Reiser ` (2 more replies) 2004-08-30 15:00 ` John Stoffel 2 siblings, 3 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-08-29 5:01 UTC (permalink / raw) To: Hans Reiser Cc: Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro On Sat, 28 Aug 2004, Hans Reiser wrote: > > I object to openat()..... Sound slike you object to O_XATTRS, not openat() itself. Realize that openat() works independently of any special streams, it's fundamentally a "look up name starting from this file" (rather than "starting from root" or "starting from cwd"). Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 5:01 ` silent semantic changes with reiser4 Linus Torvalds @ 2004-08-29 9:36 ` Hans Reiser 2004-08-29 10:44 ` CaT 2004-08-29 17:57 ` Linus Torvalds 2004-08-29 15:12 ` Alexander Lyamin 2004-08-29 16:20 ` viro 2 siblings, 2 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-29 9:36 UTC (permalink / raw) To: Linus Torvalds Cc: Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro Linus Torvalds wrote: >On Sat, 28 Aug 2004, Hans Reiser wrote: > > >>I object to openat()..... >> >> > >Sound slike you object to O_XATTRS, not openat() itself. > >Realize that openat() works independently of any special streams, it's >fundamentally a "look up name starting from this file" (rather than >"starting from root" or "starting from cwd"). > > Linus > > > > well, isn't that namespace fragmentation by definition? If you can't go cat filenameA/metas/permissions > filenameB/metas/permissions find / -exec cat {}/permissions \; | grep 4777 | wc -l then aren't you missing the whole point of this exercise which is to allow the whole OS to be better integrated into a more unified namespace so that data can leap from one tool to another, and one container to another, with the greatest of ease? If cat cannot access the metadata without changing the code of cat, then all the elegance goes poof. It completely baffles me what disabling filenameA/.. does for us. Why add asymmetries? Ease of implementation is no excuse for such asymmetry. Tomorrow I am going to send a little essay I wrote this evening on these metafiles. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 9:36 ` Hans Reiser @ 2004-08-29 10:44 ` CaT [not found] ` <87fz65wt82.fsf@uhoreg.ca> 2004-08-29 17:57 ` Linus Torvalds 1 sibling, 1 reply; 1392+ messages in thread From: CaT @ 2004-08-29 10:44 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro On Sun, Aug 29, 2004 at 02:36:50AM -0700, Hans Reiser wrote: > Linus Torvalds wrote: > >On Sat, 28 Aug 2004, Hans Reiser wrote: > >>I object to openat()..... > > > >Sound slike you object to O_XATTRS, not openat() itself. > > > >Realize that openat() works independently of any special streams, it's > >fundamentally a "look up name starting from this file" (rather than > >"starting from root" or "starting from cwd"). > > well, isn't that namespace fragmentation by definition? If you can't go > > cat filenameA/metas/permissions > filenameB/metas/permissions > > find / -exec cat {}/permissions \; | grep 4777 | wc -l My apologies if this question has been raised (and answered) elsewhere. I tried to keep up with the flamefest/discussion but it wasn't always possible. How would you do it for directories without making certain names illegal? ie taking the above you'd do: cat dirA/metas/permissions > dirB/metas/permissions But how does that distinguish an access to the metas for dirA and dirB from an access to a file called permissions within the metas directories of dirA and dirB? I think what Linus is trying to do, as far as naming, is create a consistant access scheme that works for all types of objects in the fs without making certain names illegal (and with plugins, those names are not even a fixed thing I believe). Feel free to correct me if I'm wrong, ofcourse. :) -- Red herrings strewn hither and yon. ^ permalink raw reply [flat|nested] 1392+ messages in thread
[parent not found: <87fz65wt82.fsf@uhoreg.ca>]
* Re: silent semantic changes with reiser4 [not found] ` <87fz65wt82.fsf@uhoreg.ca> @ 2004-08-30 6:31 ` Markus Törnqvist 0 siblings, 0 replies; 1392+ messages in thread From: Markus Törnqvist @ 2004-08-30 6:31 UTC (permalink / raw) To: Hubert Chan; +Cc: reiserfs-list, linux-fsdevel, linux-kernel On Sun, Aug 29, 2004 at 05:05:17PM -0400, Hubert Chan wrote: >Reiserfs list, and I don't think there was much consensus that came out It's like Gentoo users and patch sets ;) >of it. Currently for Reiser4, AFAIK, the "metas" name is a compile-time >option that you can change by changing a #define, if you're worried >about name clashes. http://mjt.nysv.org/reiser/reiser4.metas.patch -- mjt ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 9:36 ` Hans Reiser 2004-08-29 10:44 ` CaT @ 2004-08-29 17:57 ` Linus Torvalds 2004-08-29 19:53 ` Hans Reiser 1 sibling, 1 reply; 1392+ messages in thread From: Linus Torvalds @ 2004-08-29 17:57 UTC (permalink / raw) To: Hans Reiser Cc: Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro On Sun, 29 Aug 2004, Hans Reiser wrote: > > > >Realize that openat() works independently of any special streams, it's > >fundamentally a "look up name starting from this file" (rather than > >"starting from root" or "starting from cwd"). > > well, isn't that namespace fragmentation by definition? No. There's no difference between fd = open("/usr/bin/yes", O_RDWR); and dirfd = open("/usr/bin", O_RDONLY | O_DIRECTORY); fd = openat(dirfd, "yes", O_RDWR); apart from error checking and permissions.. No new namespace. Only O_XATTR implies a "namespace change". Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 17:57 ` Linus Torvalds @ 2004-08-29 19:53 ` Hans Reiser 2004-08-29 20:14 ` Nicholas Miell 2004-08-29 21:01 ` Linus Torvalds 0 siblings, 2 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-29 19:53 UTC (permalink / raw) To: Linus Torvalds Cc: Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro Linus Torvalds wrote: >On Sun, 29 Aug 2004, Hans Reiser wrote: > > >>>Realize that openat() works independently of any special streams, it's >>>fundamentally a "look up name starting from this file" (rather than >>>"starting from root" or "starting from cwd"). >>> >>> >>well, isn't that namespace fragmentation by definition? >> >> > >No. > >There's no difference between > > fd = open("/usr/bin/yes", O_RDWR); > >and > > dirfd = open("/usr/bin", O_RDONLY | O_DIRECTORY); > fd = openat(dirfd, "yes", O_RDWR); > > The difference is that cat uses open() not openat(). With your model, can I do: cat filenameA/metas/permissions > filenameB/metas/permissions find / -exec cat {}/permissions \; | grep 4777 | wc -l If yes, then we are talking past each other somehow rather than disagreeing. If metafiles can be opened with both open and openat() in your model, then we are discussing some small detail. I think the answer is no though, in which case you are missing the point of the new design. Is the answer no? ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 19:53 ` Hans Reiser @ 2004-08-29 20:14 ` Nicholas Miell 2004-08-29 21:01 ` Linus Torvalds 1 sibling, 0 replies; 1392+ messages in thread From: Nicholas Miell @ 2004-08-29 20:14 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro On Sun, 2004-08-29 at 12:53, Hans Reiser wrote: > With your model, can I do: > > cat filenameA/metas/permissions > filenameB/metas/permissions > runat filenameA "cat permissions" | runat filenameB "cat > permissions" > find / -exec cat {}/permissions \; | grep 4777 | wc -l find / -exec runat {} "cat permissions" \; | grep 4777 | wc -l Although, whether or not the kernel exposes file permissions as an attribute named permissions is up to the filesystem. (And the wrong thing to do, in my opinion, but that's irrelevant.) > If yes, then we are talking past each other somehow rather than > disagreeing. If metafiles can be opened with both open and openat() in > your model, then we are discussing some small detail. It can be opened by both, but in order to do it via open(2), you need to fchdir(2) to the attribute directory as returned by openat(2) and then use a relative pathname. > I think the answer is no though, in which case you are missing the point > of the new design. Is the answer no? -- Nicholas Miell <nmiell@comcast.net> ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 19:53 ` Hans Reiser 2004-08-29 20:14 ` Nicholas Miell @ 2004-08-29 21:01 ` Linus Torvalds 1 sibling, 0 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-08-29 21:01 UTC (permalink / raw) To: Hans Reiser Cc: Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro On Sun, 29 Aug 2004, Hans Reiser wrote: > > The difference is that cat uses open() not openat(). Hans. Step back a bit. Stop this idiocy about thinking that this is all about you, and all about your reiser4 "metas" etc. This is about VFS interfaces, and about being generic and _useful_. Quite frankly, there is absolutely _zero_ point to make attributes just because reiser4 can implement them. Zero. Nada. Zilch. It just won't matter, because people are NOT GOING TO SWITCH TO REISER4 IN MASS. So in order to make this useful, it has to be: - stable. This means that if Al Viro asks about locking and aliasing issues, you don't ignore it, you ask "how high?" - portable. Preferably across operating systems, but at the very _least_ across filesystems. See the NFSv4 thing, for example. - useful to the people who would actually use it. Right now that's samba, and not a whole lot else. For everybody else, the pain of not having _all_ major filesystems support it is probably so insurmountable that it's just not going to happen. Do you get it? If you can't get your mind out of your little current reiser4 issues, don't even bother talking about it. Look through the thread on "possible design issues for hybrids" for me actually talking with Al on how to handle problem #1. That's one important thing, and in fact, that one is important whether we expose it to a normal "open()" call or not, since all the same aliasing issues exist _regardless_ of interfaces. "openat()" happens to be _one_ solution to #2 and #3. You haven't given any constructive input to _any_ of the problems, and you keep on raising totally irrelevant issues. Face the music, Hans. If you can't solve those three problems, whatever you do simply DOES NOT MATTER. In particular, in the two last emails to you, I tried to show that "openat()" is _independent_ of streams/attributes. That it has semantics that have absolutely _nothing_ to do with metas. And in BOTH CASES you could not wrap your brain around that fact, and kept on harping about meta's. There are bigger issues here. I'm perfectly happy to _also_ expose the streams as regular filenames, but I've tried to explain that that doesn't work for directories, for example, so YOU CANNOT -JUST- talk about the filename-only thing. You do need to have a different namespace. Full stop. What's so hard to understand about that? So stop ignoring the issues. Stop ignoring the fact that not everything revolves around how you happened to implement something. Get with the program. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 5:01 ` silent semantic changes with reiser4 Linus Torvalds 2004-08-29 9:36 ` Hans Reiser @ 2004-08-29 15:12 ` Alexander Lyamin 2004-08-29 16:20 ` viro 2 siblings, 0 replies; 1392+ messages in thread From: Alexander Lyamin @ 2004-08-29 15:12 UTC (permalink / raw) To: Linus Torvalds Cc: Hans Reiser, Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro Sat, Aug 28, 2004 at 10:01:23PM -0700, Linus Torvalds wrote: > > > On Sat, 28 Aug 2004, Hans Reiser wrote: > > > > I object to openat()..... > > Sound slike you object to O_XATTRS, not openat() itself. xattr deserve to be objected. quickiehackie design practice. -- "the liberation loophole will make it clear.." lex lyamin ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 5:01 ` silent semantic changes with reiser4 Linus Torvalds 2004-08-29 9:36 ` Hans Reiser 2004-08-29 15:12 ` Alexander Lyamin @ 2004-08-29 16:20 ` viro 2 siblings, 0 replies; 1392+ messages in thread From: viro @ 2004-08-29 16:20 UTC (permalink / raw) To: Linus Torvalds Cc: Hans Reiser, Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list On Sat, Aug 28, 2004 at 10:01:23PM -0700, Linus Torvalds wrote: > > > On Sat, 28 Aug 2004, Hans Reiser wrote: > > > > I object to openat()..... > > Sound slike you object to O_XATTRS, not openat() itself. > > Realize that openat() works independently of any special streams, it's > fundamentally a "look up name starting from this file" (rather than > "starting from root" or "starting from cwd"). Linus, openat() is trivial to implement in userland *IF* we are talking about stuff already in namespace. Proof: int openat(int fd, char *path, int flags, mode_t mode) { int len = snprintf(NULL, 0, "/proc/self/%d/%s", fd, path); char name[len + 1]; sprintf(name, "/proc/self/%d/%s", fd, path); return open(name, flags, mode); } 7 lines. Replace VLA with alloca() or malloc() in pre-C99 environment. In other words, there's nothing magical about syscall itself - compatibility with Solaris is not an issue here. And basic functionality (i.e. what you've described above) can be trivially achieved with no extra primitives at all. Now, if you want to use that puppy as explicit indicator of magic steps taken, fine, but that again says nothing about the primitives we want to express the magic steps in question - reproducing openat() as one of the libc functions won't be hard anyway. So I have to agree with Hans on that one - I believe that as a fundamental primitive it's wrong choice. Convenient to have in libc - sure, why not? ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 22:29 ` Hans Reiser 2004-08-29 0:40 ` The Necessity of File Types (was silent semantic changes with reiser4) Alexander G. M. Smith 2004-08-29 5:01 ` silent semantic changes with reiser4 Linus Torvalds @ 2004-08-30 15:00 ` John Stoffel 2 siblings, 0 replies; 1392+ messages in thread From: John Stoffel @ 2004-08-30 15:00 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro Hans> My reason is that the things that distinguish between objects Hans> should be the names, not the choice of system call. The reason Hans> for this is that it improves closure and namespace unification Hans> to do so, because it allows all the objects to be accessed Hans> within the same namespace. So who decides what the names mean in a specific context? How do we get consistency across applications? In the Apple and MicroSoft cases, it's a single company which sets the standards for the names of the streams and what they mean. I have no knowledge of how Sun is doing this. I don't see us setting up a single naming authority and enforcement mechanism. I went and read the Pike paper on namespaces which Hans quotes much earlier in this thread, and I think it isn't applicable to this thread. In the paper Pike compares the MVS, VMS, DOS and Unix name spaces for disk layouts. It's an interesting paper, though a bit dated since it's 20 years old. The concepts are good, but the examples aren't any more. How many Unix systems today have /n/faces/... to link email addresses to digitized images of people faces? The big point of this paper though was to make the syntax simple, and the semantics clear and unambiguous. For example, they point out how the "Cedar" file system added in VMS style version numbers to filenames with the syntax "<name>!<version>" and they pointedly ask what does "/usr/rob!3/bin/cat-v mean? Which I think just points out that the syntax of streams (metas) needs be carefully thought out, and how that syntax implies various semantics. Lots of people keep asking for ACLs, but time and time again people point out that while the Unix permission model isn't as flexible, it is _simpler_ and easier to keep track of. As an example of complexity, just keeping track of our /etc/sudoers file with all the users, hosts and commands that can be defined is a huge hassle. You pretty much need to have a 'lint' style program to do the consistency checking. Hans> Yes, it can be useful to allow a namespace to exclude some Hans> objects, but that exclusion should not be mandated. Conversely, should a namespace *require* that objects always be there? What happens when I mount an NFS filesystem onto a Resierfs4 tree? How does the user/apps determine when the semantics of the underlying namespace have changed? Hans> If you want to exclude, you should cd or chroot to Hans> /proc/nopseudos and find a view of the filesystem that excludes Hans> metas, or mount with -nopseuodos. Hans> Do you see why I say this? I can say a lot more about the Hans> damage of fragmenting namespaces into multiple apis.... Why look Hans> at xattrs....;-) I think one of the advantages of the xattrs is that the core namespace doesn't change, nor do the semantics. They are layered on top and can be explicitly looked at. A real life problem of namespaces is Network Appliance Filers. If you have a filer with both NFS and CIFS exports, it will nicely manage the CIFS ACLs in harmony (for some level you determine) with the NFS and Unix permissions. But to backup that data, and to store that information, you need to use either their 'dump' tool or NDMP. If you NFS mount the filesystem to your backup host and then use a simple tar to back it up, you lose the information since that window you use to look into the namespace is restricted. It's a gotcha. Until we can (or maybe computing in general) come up with a new namespace paradym which we can all agree to, this is going to be a big issue. John ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 18:09 ` Linus Torvalds ` (2 preceding siblings ...) 2004-08-28 22:29 ` Hans Reiser @ 2004-08-28 23:47 ` Jeremy Allison 2004-08-30 13:56 ` Helge Hafting 4 siblings, 0 replies; 1392+ messages in thread From: Jeremy Allison @ 2004-08-28 23:47 UTC (permalink / raw) To: Linus Torvalds Cc: Helge Hafting, Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro On Sat, Aug 28, 2004 at 11:09:38AM -0700, Linus Torvalds wrote: > > If you look at the Solaris interface, the _nice_ part about "openat()" is > that you can do something like > > file = open(filename, O_RDONLY); > if (file < 0) > return -ENOENT; > icon = openat(file, "icon", O_RDONLY | O_XATTR); > if (icon < 0) > icon = default_icon_file; > .. > > and it will work regardless of whether "filename" is a directory or a > regular file, if I've understood correctly. > > Now, I think that makes sense for several reasons: > - single case > - race-free (think "stat()" vs "fstat()" races). > - I think we want to do "openat()" regardless of whether we ever > support extended attributes or not ("openat()" is nice for doing > "namei()" in user space even in the absense of any attributes or > named streams). > > So what we can do is > - implement openat() regardless, and expect to do the Solaris thing for > it if we ever do streams. Hurrah ! Definately the best way to go as far as Samba is concerned. Makes our portability easier. Now the NT interface allows an open on a named data stream without an open on the containing file (ie. you can do CreateFile("filename::STREAMNAME",....) without doing CreateFile("filename",..." first, but I personally think those semantics are a nasty hack. It makes much more sense (IMHO) to ensure that you already have an open fd on a file before allowing access to the named streams inside, Windows be damned. If we have some semantic issues mapping from the Windows semantics to Solaris/Linux semantics then so be it, it's a Samba problem so to speak :-). Thanks, Jeremy. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 18:09 ` Linus Torvalds ` (3 preceding siblings ...) 2004-08-28 23:47 ` Jeremy Allison @ 2004-08-30 13:56 ` Helge Hafting 4 siblings, 0 replies; 1392+ messages in thread From: Helge Hafting @ 2004-08-30 13:56 UTC (permalink / raw) To: Linus Torvalds Cc: Helge Hafting, Rik van Riel, Spam, Jamie Lokier, Hans Reiser, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list, Al Viro Linus Torvalds wrote: >On Sat, 28 Aug 2004, Helge Hafting wrote: > > >>>I think that lack of distinguishing power is more serious for >>>directories. The more I think I think about it, the more I wonder whether >>>Solaris did things right - having a special operation to "cross the >>>boundary". >>> >>>I suspect Solaris did it that way because it's a hell of a lot easier to >>>do it like that, but regardless, it would solve the issue of real >>>directories having both real children _and_ the "extra streams". >>> >>> >>There are many ways of doing this. Several extra streams to a directory >>that aren't ordinary files in the directory? >> >> > >Well.. Yes. We already have "." and "..", which are "special extra >streams" in a sense. However, people expect them, and know to ignore them. >The same wouldn't be true of new naming. > > > >>It seems to me that we can get a lot of nice functionality in a simpler way: >>Instead of thinking about a number of streams attached to something >>that is either an ordinary file or directory, just say that the only >>change will be that a directory may have a _single_ file stream in >>addition to being a plain directory. >> >> > >That doesn't really help us. What would the name be, and how could you >avoid clashes? > > The name for the file stream and the directory would be the same, distinguished by how they're used. I.e. fopen("filedirname", "rw") gets the stream, while chdir("filedirname") changes into the directory. fopen("filedirname/substream", "rw"); opens some stream inside the directory - as usual. My idea is that the "extra streams" aren't special at all, they are simply files in a directory and will therefore work with any normal tool. cd'ing into such a directory won't be special either - because it is a plain directory. A collection of named streams is usually a bunch of files in a directory - nothing new there. Someone serving files for clients using other os'es may attach special meaning to them, but I don't see the need for a special meaning under linux. So the only new thing is that the directory have a stream of its own. Then there must be a way to turn a ordinary file into such a directory - people may then implement thumbnails, summaries and so on as substreams. Where the substreams really are plain files in this slightly special directory. Perhaps mkdir with an existing filename could succeed, creating a mixed file/directory, if the underlying fs support it. > > >>If the VFS is to be extended in order to support file-as-directory (or >>vice versa) then hopefully it can be done in a simple way. >> >> > >I'm pretty confident that we can extend the VFS layer to support named >streams (see the technical discussion with Al, rather than the flames in >this thread). I also clearly believe that it is worth it, but I'm starting >to wonder if we should have a special open flag to make people select the >stream. > >If you look at the Solaris interface, the _nice_ part about "openat()" is >that you can do something like > > file = open(filename, O_RDONLY); > if (file < 0) > return -ENOENT; > icon = openat(file, "icon", O_RDONLY | O_XATTR); > if (icon < 0) > icon = default_icon_file; > .. > >and it will work regardless of whether "filename" is a directory or a >regular file, if I've understood correctly. > > An alternative: support open("filename/icon", O_CREAT); and have this turn "filename" into file-as-directory if it wasn't so already. Then it works both with files and directories. A new flag might be needed if old programs somehow depend on the stuff above failing for plain files. >Now, I think that makes sense for several reasons: > - single case > - race-free (think "stat()" vs "fstat()" races). > - I think we want to do "openat()" regardless of whether we ever > support extended attributes or not ("openat()" is nice for doing > "namei()" in user space even in the absense of any attributes or > named streams). > >So what we can do is > - implement openat() regardless, and expect to do the Solaris thing for > it if we ever do streams. > - _also_ support the "implied named attributes" for regular files, so > that you don't have to use "openat()" to access them. > >Comments? Does anybody hate "openat()" for any reason (regardless of >attributes)? We can easily support it, we'd just need to pass in the file >to use as part of the "nameidata" thing or add an argument (it would also >possibly be cleaner if we made "fs->pwd" be a "struct file"). > > > No problem with openat() Helge Hafting ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 14:45 ` Rik van Riel 2004-08-27 15:15 ` Spam @ 2004-08-27 18:28 ` Hans Reiser 2004-08-28 6:08 ` Paul Jackson 2004-08-29 15:00 ` Alexander Lyamin 1 sibling, 2 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-27 18:28 UTC (permalink / raw) To: Rik van Riel Cc: David Masover, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Rik van Riel wrote: > > >Currently I see no way to distinguish between the stuff >that should be backed up and the stuff that shouldn't. > > > We create filename/pseudos/backup, and that tells the archiver what to do..... ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 18:28 ` Hans Reiser @ 2004-08-28 6:08 ` Paul Jackson 2004-08-29 15:02 ` Alexander Lyamin 2004-08-30 17:43 ` Paul Stewart 2004-08-29 15:00 ` Alexander Lyamin 1 sibling, 2 replies; 1392+ messages in thread From: Paul Jackson @ 2004-08-28 6:08 UTC (permalink / raw) To: Hans Reiser Cc: riel, ninja, torvalds, diegocg, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Hans wrote: > We create filename/pseudos/backup, and that tells the archiver what to > do..... Instead of exposing the old semantics under a new interface, why not expose the new semantics under a new interface. There exist plenty of programs that know the old Unix semantics. There don't exist many working programs that use the new semantics that you're adding. I raise again the example of how Windows adapted to long filenames. Old DOS and FAT programs, including my Unix backups of today, see a 8.3 name space. Only code that knows the new magic sees the long names. If given the choice of breaking much old, existing stuff, or some new, mostly not yet existing stuff, does not it make more sense to break what mostly doesn't exist yet? One possible way to do this, of no doubt many: * Stealing a corner of the existing filename space for some magic names with the new semantics. * A new option on open(2), hence opendir(3), that lights up these magic names. * Doing any of the classic pathname calls with such a new magic name exposes the new semantics - such calls as: access execve mkdir mknod mount readlink rename rmdir stat truncate unlink This means essentially constructing a map between old and new, such that changes made in either view are sane and visible from the other view. -- I won't rest till it's the best ... Programmer, Linux Scalability Paul Jackson <pj@sgi.com> 1.650.933.1373 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 6:08 ` Paul Jackson @ 2004-08-29 15:02 ` Alexander Lyamin 2004-08-29 18:28 ` Hans Reiser 2004-08-30 17:43 ` Paul Stewart 1 sibling, 1 reply; 1392+ messages in thread From: Alexander Lyamin @ 2004-08-29 15:02 UTC (permalink / raw) To: Paul Jackson Cc: Hans Reiser, riel, ninja, torvalds, diegocg, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Fri, Aug 27, 2004 at 11:08:57PM -0700, Paul Jackson wrote: > Hans wrote: > > We create filename/pseudos/backup, and that tells the archiver what to > > do..... > > Instead of exposing the old semantics under a new interface, why not > expose the new semantics under a new interface. yeah. that could work. > There exist plenty of programs that know the old Unix semantics. There > don't exist many working programs that use the new semantics that you're > adding. > > I raise again the example of how Windows adapted to long filenames. Old > DOS and FAT programs, including my Unix backups of today, see a 8.3 name > space. Only code that knows the new magic sees the long names. > > If given the choice of breaking much old, existing stuff, or some new, > mostly not yet existing stuff, does not it make more sense to break what > mostly doesn't exist yet? > > One possible way to do this, of no doubt many: > > * Stealing a corner of the existing filename space for > some magic names with the new semantics. > > * A new option on open(2), hence opendir(3), that lights up > these magic names. > > * Doing any of the classic pathname calls with such a > new magic name exposes the new semantics - such calls > as: > access execve mkdir mknod mount readlink > rename rmdir stat truncate unlink > > This means essentially constructing a map between old and new, > such that changes made in either view are sane and visible > from the other view. It would be intresting to hear comments from Hans Reiser on proposals stated above... -- "the liberation loophole will make it clear.." lex lyamin ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 15:02 ` Alexander Lyamin @ 2004-08-29 18:28 ` Hans Reiser 2004-08-29 18:36 ` viro 0 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-29 18:28 UTC (permalink / raw) To: flx Cc: Paul Jackson, riel, ninja, torvalds, diegocg, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Alexander Lyamin wrote: >Fri, Aug 27, 2004 at 11:08:57PM -0700, Paul Jackson wrote: > > >> >>One possible way to do this, of no doubt many: >> >> * Stealing a corner of the existing filename space for >> some magic names with the new semantics. >> >> * A new option on open(2), hence opendir(3), that lights up >> these magic names. >> >> * Doing any of the classic pathname calls with such a >> new magic name exposes the new semantics - such calls >> as: >> access execve mkdir mknod mount readlink >> rename rmdir stat truncate unlink >> >>This means essentially constructing a map between old and new, >>such that changes made in either view are sane and visible >>from the other view. >> >> > >It would be intresting to hear comments from Hans Reiser on proposals stated above... > > > > just use a view, and skip the options on the system calls. if you cd to /nometas/your_home_directory_path you don't see the metafiles. Why is a view better than a syscall flag? Because it lets the user choose what he wants without recompiling to do it. This kind of a view requires no coding because you can just mount the root filesystem two ways, one with the -nopseudos mount option, and one without it. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 18:28 ` Hans Reiser @ 2004-08-29 18:36 ` viro 2004-08-29 18:57 ` viro 0 siblings, 1 reply; 1392+ messages in thread From: viro @ 2004-08-29 18:36 UTC (permalink / raw) To: Hans Reiser Cc: flx, Paul Jackson, riel, ninja, torvalds, diegocg, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Sun, Aug 29, 2004 at 11:28:42AM -0700, Hans Reiser wrote: > just use a view, and skip the options on the system calls. if you cd to > /nometas/your_home_directory_path you don't see the metafiles. Why is a > view better than a syscall flag? Because it lets the user choose what > he wants without recompiling to do it. This kind of a view requires no > coding because you can just mount the root filesystem two ways, one with > the -nopseudos mount option, and one without it. *What*? OK, now I want detailed explanation of the reasons why that doesn't create cache coherency problems. Do you have an analysis of locking in the entire thing? ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 18:36 ` viro @ 2004-08-29 18:57 ` viro 2004-08-29 20:06 ` Hans Reiser 0 siblings, 1 reply; 1392+ messages in thread From: viro @ 2004-08-29 18:57 UTC (permalink / raw) To: Hans Reiser Cc: flx, Paul Jackson, riel, ninja, torvalds, diegocg, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Sun, Aug 29, 2004 at 07:36:29PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote: > > he wants without recompiling to do it. This kind of a view requires no > > coding because you can just mount the root filesystem two ways, one with > > the -nopseudos mount option, and one without it. > > *What*? > > OK, now I want detailed explanation of the reasons why that doesn't create > cache coherency problems. > > Do you have an analysis of locking in the entire thing? And I am very, very serious about that - we are talking about very nasty minefield and design choices in that area have fundamental impact on the entire layer, wherever it is located. It's *NOT* something that you can leave until later and hope it somehow falls into place - it can be merged in steps, but you MUST know the goal on that level. To rearchitect later might be possible (even though you will a hell of a time avoiding plugins breakage), but it will be *hard*. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 18:57 ` viro @ 2004-08-29 20:06 ` Hans Reiser 2004-08-29 21:27 ` viro 0 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-29 20:06 UTC (permalink / raw) To: viro Cc: flx, Paul Jackson, riel, ninja, torvalds, diegocg, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list viro@parcelfarce.linux.theplanet.co.uk wrote: >On Sun, Aug 29, 2004 at 07:36:29PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote: > > >>>he wants without recompiling to do it. This kind of a view requires no >>>coding because you can just mount the root filesystem two ways, one with >>>the -nopseudos mount option, and one without it. >>> >>> >>*What*? >> >>OK, now I want detailed explanation of the reasons why that doesn't create >>cache coherency problems. >> >>Do you have an analysis of locking in the entire thing? >> >> > >And I am very, very serious about that - we are talking about very nasty >minefield and design choices in that area have fundamental impact on the >entire layer, wherever it is located. > >It's *NOT* something that you can leave until later and hope it somehow >falls into place - it can be merged in steps, but you MUST know the goal >on that level. To rearchitect later might be possible (even though you >will a hell of a time avoiding plugins breakage), but it will be *hard*. > > > > How about if you educate me on the problems you see for a bit before I respond? I think it might help us move into a constructive discussion. You are more experienced with your code than I am.... but I firmly believe that having different views of the same filesystem in which some files are visible in one view but not in another is technically feasible for some VFS, though I'd like to learn about the problems you see for doing it in the current VFS. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 20:06 ` Hans Reiser @ 2004-08-29 21:27 ` viro 2004-08-29 21:50 ` Linus Torvalds ` (2 more replies) 0 siblings, 3 replies; 1392+ messages in thread From: viro @ 2004-08-29 21:27 UTC (permalink / raw) To: Hans Reiser Cc: flx, Paul Jackson, riel, ninja, torvalds, diegocg, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Sun, Aug 29, 2004 at 01:06:41PM -0700, Hans Reiser wrote: > How about if you educate me on the problems you see for a bit before I > respond? I think it might help us move into a constructive discussion. A slew of cache coherency issues (and memory consumption on top of that). Rigth now we have a signle dentry tree for all fs instances, no matter how many times it and its subtrees are visible in the tree. Which, obviously, avoids all that crap. As soon as we start trying to have multiple trees over the same fs, we are in for a *lot* of fun. The basic question is how do you propagate the changes from one tree to another and what do you do when change to "invisible" entry happens via a tree where it is visible. Unfortunately, all obvious solutions either hit the pathname resolution and hit it *hard* (both in scalability and in price of uncontended case) or create a mess for stuff like NFS silly-rename semantics, VFAT aliases, etc. Saying that some unspecified data structures might work doesn't help. obviously - any variant will have tradeoffs of its own and has to be discussed individually. fsdevel is there for exactly that sort of stuff, so if you have any specific proposals you want to discuss, you are more than welcome. Again, right now all cache coherency issues are sidestepped by having a single instance of dentry tree per superblock. Each mount/binding is represented by vfsmount and _refers_ to a (sub)tree of dentry tree of that fs - dentries and inodes are the same (ditto for private data structures owned by filesystem, obviously). So we have a forest of dentry trees (one per filesystem) and forest of vfsmount trees (one tree per namespace). Each vfsmount corresponds to a chunk of namespace - think of the full user-visible tree cut into pieces by mountpoints. For every piece we know a) what fs it's from (->mnt_sb) b) what subtree of that fs it corresponds (->mnt_root) c) what piece it's attached to (->mnt_parent) d) where in that piece it's attached (->mnt_mountpoint) e) where to find its siblings and children f) which namespace (== tree in vfsmount forest) it's from Point in a namespace is determined by pair (vfsmount, dentry) - which piece we are looking at and where in that piece we are. Pathname lookups operate on such pairs - as long as we do not cross into another piece we just step from dentry to dentry, when we cross mountpoint towards root, we flip to (mnt_parent, mnt_mountpoint) first, when we cross mountpoint into mounted filesystem, we do hash lookup (lookup_mnt()) by our pair, find vfsmount with such mnt_parent and mnt_mountpoint and step into (vfsmount, vfsmount->mnt_root). Lifetime of vfsmounts is controlled by a simple refcount. Having vfsmount attached to tree contributes +1, so does having it pinned down by lookup/ chdir/chroot/opened file. When the last vfsmount goes away, filesystem is shut down (basically, that's what happens on umount). Binding simply creates a vfsmount pointing to target and attaches it at source. Cloning a namespace copies the vfsmount tree of parent process' namespace and flips vfsmounts of our root and cwd into the corresponding nodes in that copy. Normally, fork()/clone() just increments namespace refcount instead of cloning it, so all children share namespace with parent. When all processes in a namespace are gone its refcount goes to zero. At that point we dissolve all attachments in its vfsmount tree (which drops a reference to each vfsmount and can in turn lead to fs shutdowns). Lazy umount does similar operation with specified subtree - it detaches all pieces mounted anywhere in that subtree and drops reference to each. Ones that are not busy will be shut down immediately, ones that are will go away as soon as they stop being busy. vfsmounts can have flags of their own - one of the pending projects is to allow individual vfsmounts to be read-only. Noexec, nosuid and nodev are already per-mountpoint. Tree of vfsmounts is protected by a) vfsmount spinlock (protects hash lookups, mostly) b) per-namespace semaphore (that one is obviously blocking) c) in cases when we can go from 0 to 1 vfsmounts with given dentry as a mountpoint, we also hold ->d_inode->i_sem on the mountpoint-to-be (that closes races between rmdir/mount, rename/mount, etc.) Dentry tree is messier - these days we have dcache_lock and RCU stuff. If you can help getting the documentation out of these guys, you've got a *lot* of thanks from a lot of people. As it is, it's RTFS country. I can answer specific questions there, but I won't even try to produce readable manual covering the entire thing. Directory operations have exclusion based on ->i_sem - see Documentation/filesystems/directory-locking for description and proof of correctness. For operations that can destroy an object (unlink/rmdir/overwriting rename) we try to unhash the victim dentry first, so the filesystems that can't handle unlinked-but-busy stuff can detect such attempt by seeing the victim still hashed _and_ can be sure that if it's not hashed, nobody will come and see it (lookup would have to go to filesystem code, since we have the sucker unhashed and we have exclusion there). If operation is unsuccessful, we rehash the victim. Files involved: fs/dcache.c fs/super.c fs/inode.c fs/namespace.c include/linux/dcache.h include/linux/fs.h include/linux/mount.h If you need more details on any specific area (or gaps in the above) - just ask. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 21:27 ` viro @ 2004-08-29 21:50 ` Linus Torvalds [not found] ` <Pine.LNX.4.60.0408300009001.10533@alpha.polcom.net> 2004-08-29 23:17 ` Trond Myklebust 2004-08-30 2:18 ` Hans Reiser 2004-08-30 7:42 ` Hans Reiser 2 siblings, 2 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-08-29 21:50 UTC (permalink / raw) To: viro Cc: Hans Reiser, flx, Paul Jackson, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Sun, 29 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > > A slew of cache coherency issues (and memory consumption on top of that). > Rigth now we have a signle dentry tree for all fs instances, no matter > how many times it and its subtrees are visible in the tree. Which, > obviously, avoids all that crap. As soon as we start trying to have > multiple trees over the same fs, we are in for a *lot* of fun. Al, I think you should make the argument a bit more specific, because I doubt a lot of people understand just what the problems are with aliased names. Just a few examples of the problems involved will illuminate things very well, I think. People who haven't been intimate with the name caches probably simply don't understand why you worry so much. I'll start out with some trivial examples, just to let Hans and others get an idea about what the issues are. I think examples of problems are often better ways to explain them than the abstract issues themselves. Aliases: let's say that you have filename "a" hard-linked to filename "b", and you have a directory structure of streams under there. So you have a/file1 a/dir1/file2 a/dir2/file3 and (through the hard-link with "b") you have aliases of all these same names available as "b/file1", "b/dir1/file2" etc). Now, imagine that you have two processes doing mv a/dir1 a/dir2/newdir and mv b/dir2 b/dir1/newdir at the same time. Both of them MUST NOT SUCCEED, for pretty obvious reasons (you'd have moved two directories within each other, and now neither would be accessible any more). How do you handle locking for this situation? Another interesting case is what happens when you have looked up and cache the filename "a/file1" and then another process does "rm b/file1". How do you update the _other_ cached copy, since they had two different names, but _both_ names went away at the same time. Also again, how do you handle locking? The general VFS layer has a lot of rules, and avoids these problems by simply never having aliases between two directories. If the same directory shows up multiple times (which can happen with bind mounts), they have the exact same dentry for the directory, it's just found through two different vfsmount instances. That's why vfsmounts exist - they allow the same name cache entry to show up in different places at the same time. So when we do a bind mount, and the same directory shows up under two different names "a" and "b", and we do a "rm b/file1", it _automatically_ disappears from "a/file1" too, simply by virtue of "a" and "b" literally being the same dentry. No aliasing ever happens, and this makes coherency and locking much easier (which is not to say that they are trivial, but they are pretty damn clear in comparison to the alternatives). What Al (and others) worries about is that the reiser4 name handling has _none_ of these issues figured out and protected against. You can protect against them by taking very heavy locks (you can trivially protect against all races by taking one large lock around any operation), but the fact is, that is just not an option for high-performance name lookup. These aliasing/locking rules need to be global and wll-though-out. Not just fix the two examples above, but be shown to be safe _in_general_. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
[parent not found: <Pine.LNX.4.60.0408300009001.10533@alpha.polcom.net>]
* Re: silent semantic changes with reiser4 [not found] ` <Pine.LNX.4.60.0408300009001.10533@alpha.polcom.net> @ 2004-08-29 22:37 ` Linus Torvalds 2004-08-29 23:28 ` viro ` (2 more replies) 0 siblings, 3 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-08-29 22:37 UTC (permalink / raw) To: Grzegorz Kulewski; +Cc: Kernel Mailing List [ Linux-kernel cc'd, because I don't think the question is stupid, and I can't even fully answer the kNFSd thing other than point to it as a problem. ] On Mon, 30 Aug 2004, Grzegorz Kulewski wrote: > > Sorry if my qestion is stupid, but why can't we deal with (hard)links to > directories in (nearly) same way we deal with bind mounts (= making > exactly one object representing target and only referencing to it)? On a VFS level we could, these days, I think. But realize that bind mounts and the vfsmounts are pretty recent things. We don't have any filesystems that support the notion, though, and we don't have any interfaces for the filesystem to tell us about it right now. The VFS layer could try to figure it out on its own from aliasing information, so the latter may be a non-issue, but the former is why nobody does it. And even if Linux _these days_ could handle hardlinked directories, the fact is that they would cause slightly more memory usage (due to the vfsmounts), and that nobody else can handle such filesystems - including older versions of Linux. So nobody would likely use the feature (not to mention that nobody is even really asking for it ;). And the lack of filesystem support is not theoretical. It's not easy to just retrofit directory hardlinks on a UNIX filesystem. The ".." entry actually exists on _disk_ on traditional unix filesystems, and with hardlinks on directories, that's a real problem. A hardlinked directory has multiple parents. Also, while the VFS layer no longer cares (to it, ".." is purely virtual, and it never uses it), the NFS export routines still do actually want to get the on-disk parent. A filesystem that can't do that may be unable to be exported with full semantics (ie you might get ESTALE errors after server reboots, although you'd have to ask somebody with more kNFSd knowledge than me on exactly why that is the case ;) Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 22:37 ` Linus Torvalds @ 2004-08-29 23:28 ` viro 2004-08-30 1:43 ` Neil Brown 2004-09-02 10:51 ` Stephan von Krawczynski 2 siblings, 0 replies; 1392+ messages in thread From: viro @ 2004-08-29 23:28 UTC (permalink / raw) To: Linus Torvalds; +Cc: Grzegorz Kulewski, Kernel Mailing List On Sun, Aug 29, 2004 at 03:37:16PM -0700, Linus Torvalds wrote: > > Sorry if my qestion is stupid, but why can't we deal with (hard)links to > > directories in (nearly) same way we deal with bind mounts (= making > > exactly one object representing target and only referencing to it)? > > On a VFS level we could, these days, I think. But realize that bind mounts > and the vfsmounts are pretty recent things. Bindings won't replace hardlinks. a) lifetime rules and keeping stuff busy b) who's bound on top? Note that for real hardlinks to directories (not just "directory on top of file" hybrids) it's a serious question c) for real hardlinks we would want at least rename() working ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 22:37 ` Linus Torvalds 2004-08-29 23:28 ` viro @ 2004-08-30 1:43 ` Neil Brown 2004-09-02 10:51 ` Stephan von Krawczynski 2 siblings, 0 replies; 1392+ messages in thread From: Neil Brown @ 2004-08-30 1:43 UTC (permalink / raw) To: Linus Torvalds; +Cc: Grzegorz Kulewski, Kernel Mailing List On Sunday August 29, torvalds@osdl.org wrote: > > Also, while the VFS layer no longer cares (to it, ".." is purely virtual, > and it never uses it), the NFS export routines still do actually want to > get the on-disk parent. A filesystem that can't do that may be unable to > be exported with full semantics (ie you might get ESTALE errors after > server reboots, although you'd have to ask somebody with more kNFSd > knowledge than me on exactly why that is the case ;) > The VFS requires all directories to have full paths from the filesystem root in the dcache. This is needed for loop detection when renaming directories. suppose and NFS client: (UPPERCASE words refer to file handles) Says "lookup ROOT 'a'" and gets A (file handle for /a) Says "lookup A 'b'" and gets B (file handle for /a/b) waits for /a and /a/b to be flushed from the dcache Says "RENAME ROOT 'a' to B 'z'" (i.e. asks for /a to be moved to /a/b/c) how can the VFS detect that that is not allowed? It needs to know that B is a subdirectory of /a. The complete path cannot fit in filehandle, so to needs to be computed by repeated lookup of "..", which must be on disk. Even without that, storing ".." on disk makes lots of sense for fsck-like tools. NeilBrown ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 22:37 ` Linus Torvalds 2004-08-29 23:28 ` viro 2004-08-30 1:43 ` Neil Brown @ 2004-09-02 10:51 ` Stephan von Krawczynski 2004-09-02 11:26 ` Frank van Maarseveen 2 siblings, 1 reply; 1392+ messages in thread From: Stephan von Krawczynski @ 2004-09-02 10:51 UTC (permalink / raw) To: Linus Torvalds; +Cc: kangur, linux-kernel On Sun, 29 Aug 2004 15:37:16 -0700 (PDT) Linus Torvalds <torvalds@osdl.org> wrote: > [...] > And even if Linux _these days_ could handle hardlinked directories, the > fact is that they would cause slightly more memory usage (due to the > vfsmounts), and that nobody else can handle such filesystems - including > older versions of Linux. So nobody would likely use the feature (not to > mention that nobody is even really asking for it ;). Huh? Me about a year ago ;-) Been in fact pretty much boo'd for it :-) I therefore declare as this years hot issue: How to use more than 32 GIDs on nfs? Frank van Maarseveens' patch being available for years I guess, but with 2.6 supporting lots of GIDs becoming very actual... :-))) Sorry for being a bit off-topic but shouldn't we first solve the obvious (broken) and simple issues around fs? Regards, Stephan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 10:51 ` Stephan von Krawczynski @ 2004-09-02 11:26 ` Frank van Maarseveen 2004-09-02 12:58 ` Stephan von Krawczynski 0 siblings, 1 reply; 1392+ messages in thread From: Frank van Maarseveen @ 2004-09-02 11:26 UTC (permalink / raw) To: Stephan von Krawczynski; +Cc: Linus Torvalds, kangur, linux-kernel On Thu, Sep 02, 2004 at 12:51:56PM +0200, Stephan von Krawczynski wrote: > > I therefore declare as this years hot issue: > How to use more than 32 GIDs on nfs? Frank van Maarseveens' patch being ^^ The limit for NFS is 16. > available for years I guess, but with 2.6 supporting lots of GIDs becoming very > actual... thank you for reminding me that I still need to port it to 2.6. The patch came into existence around 2.2.17 IIRC. -- Frank ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 11:26 ` Frank van Maarseveen @ 2004-09-02 12:58 ` Stephan von Krawczynski 0 siblings, 0 replies; 1392+ messages in thread From: Stephan von Krawczynski @ 2004-09-02 12:58 UTC (permalink / raw) To: Frank van Maarseveen; +Cc: torvalds, kangur, linux-kernel On Thu, 2 Sep 2004 13:26:23 +0200 Frank van Maarseveen <frankvm@xs4all.nl> wrote: > On Thu, Sep 02, 2004 at 12:51:56PM +0200, Stephan von Krawczynski wrote: > > > > I therefore declare as this years hot issue: > > How to use more than 32 GIDs on nfs? Frank van Maarseveens' patch being > ^^ > The limit for NFS is 16. Yes, of course. Sorry for that exaggeration ;-) > > available for years I guess, but with 2.6 supporting lots of GIDs becoming > > very actual... > > thank you for reminding me that I still need to port it to 2.6. The patch > came into existence around 2.2.17 IIRC. Oh it will be very welcome once your done, the only way to circumvent the situation currently is to give world-rights, which is of course _bad_. Thanks in advance for this patch, Stephan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 21:50 ` Linus Torvalds [not found] ` <Pine.LNX.4.60.0408300009001.10533@alpha.polcom.net> @ 2004-08-29 23:17 ` Trond Myklebust 2004-08-29 23:25 ` viro ` (2 more replies) 1 sibling, 3 replies; 1392+ messages in thread From: Trond Myklebust @ 2004-08-29 23:17 UTC (permalink / raw) To: Linus Torvalds Cc: Alexander Viro, Hans Reiser, flx, Paul Jackson, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, Andrew Morton, wichert, jra, hch, Linux Filesystem Development, linux-kernel, flx, reiserfs-list På su , 29/08/2004 klokka 17:50, skreiv Linus Torvalds: > The general VFS layer has a lot of rules, and avoids these problems by > simply never having aliases between two directories. If the same directory > shows up multiple times (which can happen with bind mounts), they have the > exact same dentry for the directory, it's just found through two different > vfsmount instances. That's why vfsmounts exist - they allow the same name > cache entry to show up in different places at the same time. So could you explain what is stopping us from reducing the whole problem to the bind mount problem? IOW have "a/" be a directory that acts as if it is dynamically bind mounted on top of the file "a". Is it just the fantasy of supporting hard-links across "stream boundaries" (as in "touch a b; ln b a/b; ln a b/a")? I'm pretty sure nobody wants to have to add cyclic graph detection to their filesystems anyway. 8-) What other issues would need to be addressed? Cheers, Trond ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 23:17 ` Trond Myklebust @ 2004-08-29 23:25 ` viro 2004-08-29 23:54 ` Linus Torvalds 2004-08-30 0:07 ` Alexander G. M. Smith 2 siblings, 0 replies; 1392+ messages in thread From: viro @ 2004-08-29 23:25 UTC (permalink / raw) To: Trond Myklebust Cc: Linus Torvalds, Hans Reiser, flx, Paul Jackson, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, Andrew Morton, wichert, jra, hch, Linux Filesystem Development, linux-kernel, flx, reiserfs-list On Sun, Aug 29, 2004 at 07:17:10PM -0400, Trond Myklebust wrote: > På su , 29/08/2004 klokka 17:50, skreiv Linus Torvalds: > > The general VFS layer has a lot of rules, and avoids these problems by > > simply never having aliases between two directories. If the same directory > > shows up multiple times (which can happen with bind mounts), they have the > > exact same dentry for the directory, it's just found through two different > > vfsmount instances. That's why vfsmounts exist - they allow the same name > > cache entry to show up in different places at the same time. > > So could you explain what is stopping us from reducing the whole problem > to the bind mount problem? IOW have "a/" be a directory that acts as if > it is dynamically bind mounted on top of the file "a". > > > Is it just the fantasy of supporting hard-links across "stream > boundaries" (as in "touch a b; ln b a/b; ln a b/a")? I'm pretty sure > nobody wants to have to add cyclic graph detection to their filesystems > anyway. 8-) > > What other issues would need to be addressed? Go upthread and read. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 23:17 ` Trond Myklebust 2004-08-29 23:25 ` viro @ 2004-08-29 23:54 ` Linus Torvalds 2004-08-30 1:42 ` Trond Myklebust 2004-08-30 0:07 ` Alexander G. M. Smith 2 siblings, 1 reply; 1392+ messages in thread From: Linus Torvalds @ 2004-08-29 23:54 UTC (permalink / raw) To: Trond Myklebust Cc: Alexander Viro, Hans Reiser, flx, Paul Jackson, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, Andrew Morton, wichert, jra, hch, Linux Filesystem Development, linux-kernel, flx, reiserfs-list On Sun, 29 Aug 2004, Trond Myklebust wrote: > > So could you explain what is stopping us from reducing the whole problem > to the bind mount problem? IOW have "a/" be a directory that acts as if > it is dynamically bind mounted on top of the file "a". Hey, I suggested people do exactly that. See the other thread between me and Al Viro on exact implementation issues - even some code snippets ;) The problem really ends up being directories with attributes (where we can't just overmount the existing directory). That's where "openat()" helps us. (The other problem is the purely _practical_ problem of reiser4 going behind the VFS layer, and thus _not_ getting the aliasing and locking right, but that's an implementation issue in my book, and nothing really fundamental). > Is it just the fantasy of supporting hard-links across "stream > boundaries" (as in "touch a b; ln b a/b; ln a b/a")? I'm pretty sure > nobody wants to have to add cyclic graph detection to their filesystems > anyway. 8-) It's easy enough to do the graph detection at the VFS layer, exactly because of the density of the dentry graph. (Of course, nfs-exporting a filesystem breaks that density, thanks to the "lookup by fh" stuff, so we might not allow it for an NFS client). I suspect most filesystems wouldn't allow hard-links across "stream boundaries" in the first place. I _suspect_ that most stream implementations would bind the attributes more tightly than that to the file that owns them. reiser4 migth be the only one that might ever support something like that. > What other issues would need to be addressed? - Whether we want it in the first place - whether we need the "separate namespace" thing that O_XATTR and openat() brings us (they certainly solve the problem, but maybe it is solvable another way too) - how to actually test this out in practice (ie getting reiser4 to do the proper thing wrt the VFS layer, but preferably _also_ having another filesystem like NFSv4 or cifs that actually uses this and shows what the problems are). - whether it makes any sense at all unless we also make at least a few other filesystems support it, so that people start using it as an "expected feature" rather than a "works only on a couple of machines". And probably others. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 23:54 ` Linus Torvalds @ 2004-08-30 1:42 ` Trond Myklebust 2004-08-30 2:31 ` Linus Torvalds 0 siblings, 1 reply; 1392+ messages in thread From: Trond Myklebust @ 2004-08-30 1:42 UTC (permalink / raw) To: Linus Torvalds Cc: Alexander Viro, Hans Reiser, flx, Paul Jackson, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, Andrew Morton, wichert, jra, hch, Linux Filesystem Development, linux-kernel, flx, reiserfs-list På su , 29/08/2004 klokka 19:54, skreiv Linus Torvalds: > The problem really ends up being directories with attributes (where we > can't just overmount the existing directory). That's where "openat()" > helps us. Well, yes there has to be a distinction between a true bind mount which actually covers the file or directory, and something like the stream "bind mount" which doesn't. The stream "bind mount" is just there to allow you to root the attributes in a single tree. It can be made functionally entirely equivalent to the openat(), but uses pathname semantics (e.g., "//") to denote the attribute fork instead of an extra function call. > > Is it just the fantasy of supporting hard-links across "stream > > boundaries" (as in "touch a b; ln b a/b; ln a b/a")? I'm pretty sure > > nobody wants to have to add cyclic graph detection to their filesystems > > anyway. 8-) > > It's easy enough to do the graph detection at the VFS layer, exactly > because of the density of the dentry graph. Don't you end up having to lock the entire paths b/c/d and a/e/f in order to prevent "ln a b/c/d/a; ln b a/e/f/b"? > > What other issues would need to be addressed? > - how to actually test this out in practice (ie getting reiser4 to do the > proper thing wrt the VFS layer, but preferably _also_ having another > filesystem like NFSv4 or cifs that actually uses this and shows what > the problems are). As I said, NFSv4 can be made ready pretty quickly: Bruce is already finishing up the xattr implementation. > - whether it makes any sense at all unless we also make at least a few > other filesystems support it, so that people start using it as an > "expected feature" rather than a "works only on a couple of machines". NTFS? ;-) Cheers, Trond ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-30 1:42 ` Trond Myklebust @ 2004-08-30 2:31 ` Linus Torvalds 2004-08-30 3:01 ` viro ` (3 more replies) 0 siblings, 4 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-08-30 2:31 UTC (permalink / raw) To: Trond Myklebust Cc: Alexander Viro, Hans Reiser, flx, Paul Jackson, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, Andrew Morton, wichert, jra, hch, Linux Filesystem Development, linux-kernel, flx, reiserfs-list On Sun, 29 Aug 2004, Trond Myklebust wrote: > > Well, yes there has to be a distinction between a true bind mount which > actually covers the file or directory, and something like the stream > "bind mount" which doesn't. > > The stream "bind mount" is just there to allow you to root the > attributes in a single tree. It can be made functionally entirely > equivalent to the openat(), but uses pathname semantics (e.g., "//") to > denote the attribute fork instead of an extra function call. Using '//' would be nice, but would break real apps. If I remember correctly, POSIX specifies that '//' can be special at the _beginning_ of a path, but in the middle, it has to act like a single '/'. And that's not just theory - it's quite common for programs to just concatenate a directory name (which may or may not end with a slash) with another path-name that starts with a slash. So you _will_ see existing scripts and programs using things like "/usr/include//sys/type.h", and they'd break if "//" would switch from "regular namespace" to "attribute namespace". So I don't see any way to extend pathname semantics to distinguish between "directory contents" and "directory attribute stream". > > It's easy enough to do the graph detection at the VFS layer, exactly > > because of the density of the dentry graph. > > Don't you end up having to lock the entire paths b/c/d and a/e/f in > order to prevent "ln a b/c/d/a; ln b a/e/f/b"? That's not the problem - since it's in memory, we can just get the dcache lock, and do it locked for t least local filesystems. However, being prodded by Andries, I think I'm wrong _anyway_. Since the dcache is only "dense" down one path to the root, and doesn't contain all the alternate ways of getting to a particular directory, I came to the conclusion that the VFS layer can't actually do cyclic detection after all... So together with the fact that nobody really _wants_ hardlinks to directories, I think the right answer is "no". It's not a problem as long as the attributes streams are always tied to the file/directory they are attributes of - then the "directory link" is really just a file link, and can't cause any cycles. > > - how to actually test this out in practice (ie getting reiser4 to do the > > proper thing wrt the VFS layer, but preferably _also_ having another > > filesystem like NFSv4 or cifs that actually uses this and shows what > > the problems are). > > As I said, NFSv4 can be made ready pretty quickly: Bruce is already > finishing up the xattr implementation. Do we have any servers that implement it? I think NFSv4 might be a good test-case if so. > > - whether it makes any sense at all unless we also make at least a few > > other filesystems support it, so that people start using it as an > > "expected feature" rather than a "works only on a couple of machines". > > NTFS? ;-) Hey, I see the smiley, but I'd still like to point out that not many people use it under Linux, and while I think writing to it might be stable these days, I don't believe named streams are necessarily going to materialize all that quickly.. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-30 2:31 ` Linus Torvalds @ 2004-08-30 3:01 ` viro 2004-08-30 3:55 ` Linus Torvalds 2004-08-30 3:12 ` Trond Myklebust ` (2 subsequent siblings) 3 siblings, 1 reply; 1392+ messages in thread From: viro @ 2004-08-30 3:01 UTC (permalink / raw) To: Linus Torvalds Cc: Trond Myklebust, Hans Reiser, flx, Paul Jackson, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, Andrew Morton, wichert, jra, hch, Linux Filesystem Development, linux-kernel, flx, reiserfs-list On Sun, Aug 29, 2004 at 07:31:49PM -0700, Linus Torvalds wrote: > And that's not just theory - it's quite common for programs to just > concatenate a directory name (which may or may not end with a slash) with > another path-name that starts with a slash. So you _will_ see existing > scripts and programs using things like "/usr/include//sys/type.h", and > they'd break if "//" would switch from "regular namespace" to "attribute > namespace". > > So I don't see any way to extend pathname semantics to distinguish between > "directory contents" and "directory attribute stream". I do, actually. There might be a way and it is kinda-sorta similar to your openat() variant, but lives in normal namespace. No, I'm not too fond of that, but since we are discussing weird variants anyway... a) associated directory tree of object is not automounted on top of it. Instead of that, we always do detached vfsmount (and do it on demand - see below) b) we have a bunch of pseudo-symlinks in /proc/<pid>/fd/ - same kind as what we already have there, but instead of (file->f_vfsmount, file->f_dentry) they lead to associated vfsmount (allocated if needed). Once we get a reference to such guy, we can * do further lookups * chdir there and poke around * hell, we can even bind it someplace (that will require slight change in attach_mnt() logics, but it's not hard) and get it permanently mounted Since it's not attached anywhere, normal GC logics works just fine. And yes, they are usable from scripts, etc. - exec 42<foo/bar/baz cat /proc/self/fd/#42/whatever/crap/you/want and enjoy. > However, being prodded by Andries, I think I'm wrong _anyway_. Since the > dcache is only "dense" down one path to the root, and doesn't contain all > the alternate ways of getting to a particular directory, I came to the > conclusion that the VFS layer can't actually do cyclic detection after > all... <blinks> <rereads a bunch of earlier postings> Oh, _that_'s what you meant... No, we definitely have no chance in hell to catch loops, dcache or not. It costs too much - we need to examine a *lot* of nodes to do that and all of them would have to be read at some point. We either need entire fs tree locked in core or we might have to reread it on every rename(). The former will kill us on memory use, the latter - on amount of IO. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-30 3:01 ` viro @ 2004-08-30 3:55 ` Linus Torvalds 2004-08-30 4:46 ` viro 0 siblings, 1 reply; 1392+ messages in thread From: Linus Torvalds @ 2004-08-30 3:55 UTC (permalink / raw) To: viro Cc: Trond Myklebust, Hans Reiser, flx, Paul Jackson, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, Andrew Morton, wichert, jra, hch, Linux Filesystem Development, linux-kernel, flx, reiserfs-list On Mon, 30 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > > > > So I don't see any way to extend pathname semantics to distinguish between > > "directory contents" and "directory attribute stream". > > I do, actually. There might be a way and it is kinda-sorta similar to > your openat() variant, but lives in normal namespace. No, I'm not too > fond of that, but since we are discussing weird variants anyway... > > a) associated directory tree of object is not automounted on top > of it. Instead of that, we always do detached vfsmount (and do it on demand - > see below) > b) we have a bunch of pseudo-symlinks in /proc/<pid>/fd/ - same > kind as what we already have there, but instead of (file->f_vfsmount, > file->f_dentry) they lead to associated vfsmount (allocated if needed). > Once we get a reference to such guy, we can > * do further lookups > * chdir there and poke around > * hell, we can even bind it someplace (that will require slight change > in attach_mnt() logics, but it's not hard) and get it permanently mounted Well, the above _is_ the same as "openat()", really. It's just using a filesystem starting point to emulate a new system call. Same thing conceptually. You could do pretty much any system call as a filesystem action if you wanted to ;) I don't disagree with doing so - as a way to expose the new system call to scripts. But I don't think you're being entirely intellectually honest if you think this suddendly makes it be "one namespace". It's still a secondary namespace rooted in an entry in the normal ones - exactly like "openat()". For a non-script, a native "openat()" interface would be more efficient and less confusing, and conceptually no different from yours. No reason we couldn't have both, since they are 100% equivalent and would share the same code anyway... Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-30 3:55 ` Linus Torvalds @ 2004-08-30 4:46 ` viro 2004-08-30 6:33 ` viro 0 siblings, 1 reply; 1392+ messages in thread From: viro @ 2004-08-30 4:46 UTC (permalink / raw) To: Linus Torvalds Cc: Trond Myklebust, Hans Reiser, flx, Paul Jackson, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, Andrew Morton, wichert, jra, hch, Linux Filesystem Development, linux-kernel, flx, reiserfs-list On Sun, Aug 29, 2004 at 08:55:02PM -0700, Linus Torvalds wrote: > Well, the above _is_ the same as "openat()", really. It's just using a > filesystem starting point to emulate a new system call. Same thing > conceptually. You could do pretty much any system call as a filesystem > action if you wanted to ;) Umm... Yes and no - open() is not the only thing you can do there. You can emulate some of the other stuff with open() + fchdir() + syscall, but... Note that it also gives you access to other tasks' files (provided that tasks are yours, so you can get to their /proc/.../fd). > I don't disagree with doing so - as a way to expose the new system call to > scripts. But I don't think you're being entirely intellectually honest if > you think this suddendly makes it be "one namespace". It's still a > secondary namespace rooted in an entry in the normal ones - exactly like > "openat()". Well - it *does* expose these objects to all normal syscalls. E.g. you can unlink() a component in there without fd2 = open(".", ...); fd3 = openat(fd, ".", ....) fchdir(fd3); unlink("blah"); fchdir(fd2); and similar horrors (now have fun adding locking for multithreaded process, etc.). In that sense it is, indeed, the same namespace. > For a non-script, a native "openat()" interface would be more efficient > and less confusing, and conceptually no different from yours. No reason > we couldn't have both, since they are 100% equivalent and would share the > same code anyway... I'm not sure that openat() is the right interface for e.g. fileservers. And no, I'm not saying that above is suitable for them - IMO neither variant is what they need. The thing is, fileserver almost certainly wants to create and remove objects. And you either end up with new syscalls for doing *that* relative to opened fd or you do tons of fchdir(), which makes benefits of openat() dubious in the best case. Arguments about O_NOFOLLOW on the intermediate stages are bullshit, IMNSHO - if they want to make some parts of tree inaccessible, they should simply mkdir /tmp/FOAD; chmod 0 /tmp/FOAD; mount --bind /tmp/FOAD <blocked path> in the namespace their daemon is running in. And forget all that crap about filtering pathnames and blocking symlinks on intermediate stages (the latter is obviously worthless without the former since one can simply substitute the symlink body in the pathname). ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-30 4:46 ` viro @ 2004-08-30 6:33 ` viro 0 siblings, 0 replies; 1392+ messages in thread From: viro @ 2004-08-30 6:33 UTC (permalink / raw) To: Linus Torvalds Cc: Trond Myklebust, Hans Reiser, flx, Paul Jackson, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, Andrew Morton, wichert, jra, hch, Linux Filesystem Development, linux-kernel, flx, reiserfs-list On Mon, Aug 30, 2004 at 05:46:37AM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote: > Arguments about O_NOFOLLOW on the intermediate stages are bullshit, IMNSHO - > if they want to make some parts of tree inaccessible, they should simply > mkdir /tmp/FOAD; chmod 0 /tmp/FOAD; mount --bind /tmp/FOAD <blocked path> > in the namespace their daemon is running in. And forget all that crap > about filtering pathnames and blocking symlinks on intermediate stages > (the latter is obviously worthless without the former since one can simply > substitute the symlink body in the pathname). Ehh... After looking at that for a while... No, it's not that simple and removing the stuff that way won't do what these guys want, at least not without something else. Frankly, what I've seen worries me a lot - it looks like there is a missing primitive here that would be saner than this sort of filtering. It appears that most of this stuff would be covered by a fast way to tell if the resulting object belongs to given subtree. That could be arranged (not without some changes, but doable), but I'm not sure that it's enough to cover the stuff they are really trying to do. It does look like an interesting problem and current solutions certainly suck. And I very much doubt that "do a lookup if it doesn't run into anything that could be too tricky for our pathname-based checks, otherwise let's do it step-by-step from userland" is the right approach here. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-30 2:31 ` Linus Torvalds 2004-08-30 3:01 ` viro @ 2004-08-30 3:12 ` Trond Myklebust 2004-08-30 5:42 ` Brad Boyer 2004-09-07 20:08 ` Tom Vier 3 siblings, 0 replies; 1392+ messages in thread From: Trond Myklebust @ 2004-08-30 3:12 UTC (permalink / raw) To: Linus Torvalds Cc: Alexander Viro, Hans Reiser, flx, Paul Jackson, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, Andrew Morton, wichert, jra, hch, Linux Filesystem Development, linux-kernel, flx, reiserfs-list På su , 29/08/2004 klokka 22:31, skreiv Linus Torvalds: > So together with the fact that nobody really _wants_ hardlinks to > directories, I think the right answer is "no". It's not a problem as > long as the attributes streams are always tied to the file/directory > they are attributes of - then the "directory link" is really just a > file link, and can't cause any cycles. Right. > Do we have any servers that implement it? I think NFSv4 might be a good > test-case if so. I believe Solaris Express already has an implementation, so I should have at least one server implementation available in the basement. So probably does Hummingbird, since their NFSv4 server runs on Windows. CITI is working on the Linux code, but I'm not sure how far they've come on the server side. I'll ask Bruce in the morning... Cheers, Trond ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-30 2:31 ` Linus Torvalds 2004-08-30 3:01 ` viro 2004-08-30 3:12 ` Trond Myklebust @ 2004-08-30 5:42 ` Brad Boyer 2004-09-07 20:08 ` Tom Vier 3 siblings, 0 replies; 1392+ messages in thread From: Brad Boyer @ 2004-08-30 5:42 UTC (permalink / raw) To: Linus Torvalds Cc: Trond Myklebust, Alexander Viro, Hans Reiser, flx, Paul Jackson, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, Andrew Morton, wichert, jra, hch, Linux Filesystem Development, linux-kernel, flx, reiserfs-list On Sun, Aug 29, 2004 at 07:31:49PM -0700, Linus Torvalds wrote: > On Sun, 29 Aug 2004, Trond Myklebust wrote: > > > - how to actually test this out in practice (ie getting reiser4 to do the > > > proper thing wrt the VFS layer, but preferably _also_ having another > > > filesystem like NFSv4 or cifs that actually uses this and shows what > > > the problems are). > > > > As I said, NFSv4 can be made ready pretty quickly: Bruce is already > > finishing up the xattr implementation. > > Do we have any servers that implement it? I think NFSv4 might be a good > test-case if so. I know the mention of it in the past has brought up ridicule, but netatalk requires some level of multi-stream support in the export to the client, and currently just emulates it on the server side by hiding the extra files in various directories. It should be possible to modify it to use named streams/attributes for the extra data. If that happens, it would be nice if hfs and hfsplus could export the data so it could be used. The view exported by netatalk does match the internal structure of hfs, after all. (For those of you that aren't familiar with the Macintosh, a Mac style filesystem has two peer streams, data and resource. They are each a seekable bytestream, and are actually at the same level inside the filesystem. In addition, there is a large set of fixed length attributes stored, such as the file type info and icon position. In addition, hfsplus also supports full named streams and arbitrary metadata, tho that isn't used much.) Brad Boyer flar@allandria.com ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-30 2:31 ` Linus Torvalds ` (2 preceding siblings ...) 2004-08-30 5:42 ` Brad Boyer @ 2004-09-07 20:08 ` Tom Vier 3 siblings, 0 replies; 1392+ messages in thread From: Tom Vier @ 2004-09-07 20:08 UTC (permalink / raw) To: linux-kernel On Sun, Aug 29, 2004 at 07:31:49PM -0700, Linus Torvalds wrote: > Using '//' would be nice, but would break real apps. If I remember > correctly, POSIX specifies that '//' can be special at the _beginning_ of > a path, but in the middle, it has to act like a single '/'. FWIW, QNX uses // for it's network fs (nfs?). //0/ means current node, //1/ first node, //2/ second, etc. I didn't even know posix said anything about // before. -- Tom Vier <tmv@comcast.net> DSA Key ID 0x15741ECE ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 23:17 ` Trond Myklebust 2004-08-29 23:25 ` viro 2004-08-29 23:54 ` Linus Torvalds @ 2004-08-30 0:07 ` Alexander G. M. Smith 2 siblings, 0 replies; 1392+ messages in thread From: Alexander G. M. Smith @ 2004-08-30 0:07 UTC (permalink / raw) To: Trond Myklebust Cc: viro, reiser, flx, pj, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list, torvalds Trond Myklebust wrote on Sun, 29 Aug 2004 19:17:10 -0400: > Is it just the fantasy of supporting hard-links across "stream > boundaries" (as in "touch a b; ln b a/b; ln a b/a")? I'm pretty sure > nobody wants to have to add cyclic graph detection to their filesystems > anyway. 8-) Been there, done that. It works in a rough form in an experimental RAM file system for BeOS. But deleting files requires a graph traversal to see if it is isolating just one item or a whole bunch of them. It also implies keeping track of multiple parent directories for everything. It does make it useful for the user - you can have files and directories in multiple places, so you don't have to decide on one classification for a file (or directory). One other downside is a lot of locking complexity (or not if you don't need fine grained locking) to guarantee atomicity of the delete. For user convenience, the low level delete function returns an error code if there is more than one item cut off from the root, but it could just as well delete all of them (sort of like rmdir giving an error if the directory isn't empty rather than just deleting everything). - Alex ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 21:27 ` viro 2004-08-29 21:50 ` Linus Torvalds @ 2004-08-30 2:18 ` Hans Reiser 2004-08-30 7:42 ` Hans Reiser 2 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-30 2:18 UTC (permalink / raw) To: viro Cc: flx, Paul Jackson, riel, ninja, torvalds, diegocg, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list You are assuming that the hiding has to be embodied by the dentry cache. That sounds hard. Let's not do that; I am too lazy. We can put it at a layer above that. Let us have a "mask" concept. For the purposes of this discussion, the mask can be made totally trivial and hardcoded, with none of the expressive power of my new DoD project, and can probably be coded in a few hours to exclude the metas. For my new DoD project we are creating a configurable, scalable, sophisticated "mask" abstraction, and you must successfully pass through the mask before you can ask if the filesystem contains X/Y/Z. (For the darpa project with the general purpose masks with specifications and all that we will take 6 months, not a few hours.) The rest shall wait until after I get a bite to eat, ok? Hans viro@parcelfarce.linux.theplanet.co.uk wrote: >On Sun, Aug 29, 2004 at 01:06:41PM -0700, Hans Reiser wrote: > > > >>How about if you educate me on the problems you see for a bit before I >>respond? I think it might help us move into a constructive discussion. >> >> > >A slew of cache coherency issues (and memory consumption on top of that). >Rigth now we have a signle dentry tree for all fs instances, no matter >how many times it and its subtrees are visible in the tree. Which, >obviously, avoids all that crap. As soon as we start trying to have >multiple trees over the same fs, we are in for a *lot* of fun. > > > ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 21:27 ` viro 2004-08-29 21:50 ` Linus Torvalds 2004-08-30 2:18 ` Hans Reiser @ 2004-08-30 7:42 ` Hans Reiser 2004-08-30 13:17 ` Alexander G. M. Smith 2 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-30 7:42 UTC (permalink / raw) To: viro Cc: flx, Paul Jackson, riel, ninja, torvalds, diegocg, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list I think we have to either: 1) Link only to the file and not the directory in the file-directory duality. This means we change the semantics of hard link so that it only links to the data and the standard metadata, and the optional attributes/streams/files-in-directory are not seen by the second link. You can think of this as meaning that hard links only connect to the file and not the directory in the file-directory wave-particle duality or, 2) we should ask Alexander Smith to help with applying the graph traversal cycle detection code that he wrote. I can go either way contentedly for now. 2) is the right long term solution. 1) is probably the right short term solution. Ok, Linus and Viro, now I see why it was hard. Being able to effectively connect to compound documents only with symlinks is a bit distasteful, but it is quite livable, and I very much hope you decide it is better than fragmenting the namespace. Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-30 7:42 ` Hans Reiser @ 2004-08-30 13:17 ` Alexander G. M. Smith 0 siblings, 0 replies; 1392+ messages in thread From: Alexander G. M. Smith @ 2004-08-30 13:17 UTC (permalink / raw) To: Hans Reiser Cc: flx, pj, riel, ninja, torvalds, diegocg, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list, viro Hans Reiser wrote on Mon, 30 Aug 2004 00:42:13 -0700: > or, 2) we should ask Alexander Smith to help with applying the graph > traversal cycle detection code that he wrote. I put my foot into it, didn't I? It's not too bad, one trick to simplify things is to have the concept of a main parent and step parents. The file system name space graph is kept structured so that by always going upward through the main parent, you're guaranteed to reach the root. It doesn't save anything on traversal when deleting files (you still have to visit all the relevant nodes to check for cycles), but allows you to replace hard links with a new kind of dynamic symbolic link. The hard link pretends to be a symbolic link presenting the "true path" to its target, which changes as the target moves around (so it's not as reliable as real hard link). Old software doesn't see the cycles in the graph, just follows the single true path to any particular object like it currently does, and sees a bunch of symbolic links to everything else. New software can see the full parent list (not just a single "..") and the actual hard links. > Ok, Linus and Viro, now I see why it was hard. Being able to effectively > connect to compound documents only with symlinks is a bit distasteful, > but it is quite livable, and I very much hope you decide it is better > than fragmenting the namespace. Still, having hard links to attributes or directories of attributes is useful. For example, if several contact manager entries reference the same person, it would be nice to have that represented as a hard link to the attribute directory tree describing that person. That's assuming you implement contacts (name, address, etc) as a bunch of attributes in a directory. - Alex ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 6:08 ` Paul Jackson 2004-08-29 15:02 ` Alexander Lyamin @ 2004-08-30 17:43 ` Paul Stewart 2004-08-30 17:58 ` Linus Torvalds 1 sibling, 1 reply; 1392+ messages in thread From: Paul Stewart @ 2004-08-30 17:43 UTC (permalink / raw) To: Paul Jackson Cc: Hans Reiser, riel, ninja, torvalds, diegocg, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On 2004.08.27 23:08, Paul Jackson wrote: > Hans wrote: > > We create filename/pseudos/backup, and that tells the archiver what > > to do..... > > Instead of exposing the old semantics under a new interface, why not > expose the new semantics under a new interface. Here's another take on the same theme. To see attrs on files, one can either use a newly developed application which can use special new syscalls/flags on syscalls a Paul Jackson recommends. However from an old shell or application one can also open the attribute node on /home/myself/foo.txt by checking out /attr/home/myself/foo.txt/, which points to the "as directory" node on the filesystem that foo.txt points to. The strange part of this idea is that the /attr filesystem wouldn't be conventionally browsable. An opendir() on any path would return the attributes, not the subdirectory contents, even if the target was a conventional directory. File based operations (open(), unlink(), etc) would operate on the "as directory" node of the dirname() of the file argument. The /attr filesystem would be completely virtual, honoring the permissions, crossing filesystems, and inheriting the root directory of the calling process. Just a thought from a Linux fs layman -- certainly one of the Reiser unwashed masses. -- Paul ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-30 17:43 ` Paul Stewart @ 2004-08-30 17:58 ` Linus Torvalds 0 siblings, 0 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-08-30 17:58 UTC (permalink / raw) To: Paul Stewart Cc: Paul Jackson, Hans Reiser, riel, ninja, diegocg, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Mon, 30 Aug 2004, Paul Stewart wrote: > > Here's another take on the same theme. To see attrs on files, one can > either use a newly developed application which can use special new > syscalls/flags on syscalls a Paul Jackson recommends. However from an > old shell or application one can also open the attribute node on > /home/myself/foo.txt by checking out /attr/home/myself/foo.txt/, which > points to the "as directory" node on the filesystem that foo.txt points > to. This is the same idea as Al Viro's /proc/self/fd/#42/attr issue, except yours has two fundamental problem: races and ambiguities. If you open a filename in some "secondary" tree (be it /proc or //attr or whatever) based on the filename in the primary one, you have two issues that you need to work out: - how do you handle a name change in the primary tree at the same time as lookup - how do you handle the ambiguity of //attr/usr/bin/emacs/icon (is that the "icon" attribute on "/usr/bin/emacs", or is it perhaps the "emacs/icon" attribute on "/usr/bin"). The ambiguity can be handled by saying that attributes only have one component (ie only the _last_ component of a lookup is the attribute name). But the race between primary tree and secondary tree cannot be handled in a normal name-space. What Al did was to avoid both by "fixing" the attribute lookup point with another open - _exactly_ the same way "openat()" handles it. So Al's naming convention avoids both the ambiguity and the primary tree name change races by first opening the primary tree file, and then explicitly using that file as the "anchor" in the secondary tree. He did it in /proc, where we obviously already do export an open fd as an anchor-point. > The strange part of this idea is that the /attr filesystem wouldn't be > conventionally browsable. That may make it non-intuitive to use, but that's not the real problem. See above. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 18:28 ` Hans Reiser 2004-08-28 6:08 ` Paul Jackson @ 2004-08-29 15:00 ` Alexander Lyamin 2004-08-31 0:46 ` David Masover 1 sibling, 1 reply; 1392+ messages in thread From: Alexander Lyamin @ 2004-08-29 15:00 UTC (permalink / raw) To: Hans Reiser Cc: Rik van Riel, David Masover, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Fri, Aug 27, 2004 at 11:28:51AM -0700, Hans Reiser wrote: > Rik van Riel wrote: > > > > > > >Currently I see no way to distinguish between the stuff > >that should be backed up and the stuff that shouldn't. > > > > > > > We create filename/pseudos/backup, and that tells the archiver what to > do..... its ugly idea. representations should be distinguished from sources. and its reasonably to put this distinguisher somewhere in namespace. but filename/pseudos/backup is plain ugly. -- "the liberation loophole will make it clear.." lex lyamin ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 15:00 ` Alexander Lyamin @ 2004-08-31 0:46 ` David Masover 2004-08-31 6:47 ` Hans Reiser 0 siblings, 1 reply; 1392+ messages in thread From: David Masover @ 2004-08-31 0:46 UTC (permalink / raw) To: flx Cc: Hans Reiser, Rik van Riel, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Alexander Lyamin wrote: | Fri, Aug 27, 2004 at 11:28:51AM -0700, Hans Reiser wrote: | |>Rik van Riel wrote: |> |> |>> |>>Currently I see no way to distinguish between the stuff |>>that should be backed up and the stuff that shouldn't. |>> |>> |>> |> |>We create filename/pseudos/backup, and that tells the archiver what to |>do..... | | | its ugly idea. | | representations should be distinguished from sources. and its reasonably | to put this distinguisher somewhere in namespace. but filename/pseudos/backup | is plain ugly. How about filename/metas/serialized, or filename/metas/fschunk? And I did mention before about how it'd be nice for it to be an fs chunk... Imagine 5000 tiny little files that, by the time reiser's done with them, fit inside 5 blocks or so. It's insanely cheaper to copy the five blocks than to tar them up. It also means that we don't have to worry about supporting tar on other platforms -- if they don't have reiser4, they need a separate converter. If apps are going to use reiser4 metadata for anything significant, they probably would link against some sort of userland implementation of reiser4 as a Windows library, say, to access the files elsewhere. That's extreme, but you get the idea. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTPKc3gHNmZLgCUhAQJ5xw//T+kLDCHlfQEaGM9XAeFDZU9f4m4PWjet n2xODVIxaT7WmnZ4nQ3rRTiJjsXvRBcPdWUTFRZl6NbLqa/K6ESpUJAZsZynRhXd NLJ9gEpeqcUrXBLrmTD5AbzENOLdqDIo2qQ+hde68H5vo+naa5xtTyrMiHNHQWbf 9P45WKXYt36madg4KjAhZcYB+T843fc/ItjLQBTwxBNOykBcZ82UUNwGeGbiqczm uXftCD6BNtKsXnJxu2jVc8CK93lOw4SIf7BY09PFXgK2XTpAsKzOwwqpBtdxSzqD lkz+nGEeYZk6GRA3N1OTbPJnboee7WHZzMyvXDMBsExfR6puV1SYA2NWhj9uGDmS nYNZwdN+RimiouD1D9DD+oCQe/Gr66THicex5FM37cMBdaDf0jGblvpDdQRfkhcX 2KGLCt1ax620ia2945/+/+6iQqM7+oTQsVNziYjVGQ4gCNRzZh8SBtM0yUZuecNO H84fWYLICGos+Qk7FyCuQOquTSVRWKadkiDaOSre2hFk+S9VmDWkghJSNGrnqoKi pFYFrUzL+NXPHs6G8b8EKAaQQt+Z8iHE6xgdc+yS56Pdrq8apRzeQv2zhgZaRq9J kCRgvYo40xFMX4B+EqexxIiwc6nPUD5dFbt0OnJMEk4pduFq64lMVdoQA8Qp4X8a 0feuVuaWHr0= =ymeH -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 0:46 ` David Masover @ 2004-08-31 6:47 ` Hans Reiser 2004-09-01 2:04 ` David Masover 0 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-31 6:47 UTC (permalink / raw) To: David Masover Cc: flx, Rik van Riel, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list David Masover wrote: > > > And I did mention before about how it'd be nice for it to be an fs > chunk... Please define fs chunk. My mind has an open door.;-) > > Imagine 5000 tiny little files that, by the time reiser's done with > them, fit inside 5 blocks or so. It's insanely cheaper to copy the five > blocks than to tar them up. It also means that we don't have to worry > about supporting tar on other platforms -- if they don't have reiser4, > they need a separate converter. If apps are going to use reiser4 > metadata for anything significant, they probably would link against some > sort of userland implementation of reiser4 as a Windows library, say, to > access the files elsewhere. > > That's extreme, but you get the idea. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-31 6:47 ` Hans Reiser @ 2004-09-01 2:04 ` David Masover 0 siblings, 0 replies; 1392+ messages in thread From: David Masover @ 2004-09-01 2:04 UTC (permalink / raw) To: Hans Reiser Cc: flx, Rik van Riel, Linus Torvalds, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hans Reiser wrote: | David Masover wrote: | |> |> |> And I did mention before about how it'd be nice for it to be an fs |> chunk... | | | Please define fs chunk. My mind has an open door.;-) I think this is sort of like dump/restore, but for a subset of the tree. ~ So it's faster for all the reasons dump/restore are faster than tar for a whole filesystem. But I don't know all that much about either dump/restore or reiser4 format plugins. I'm defining an "fs chunk" as a complete, read-only reiser4 filesystem of the data in question. As a snapshot. I'm imagining it being faster to create and ultimately smaller and more intelligently packed than a tarball. What if you could get a blocklist for a directory? Of course some things would have to be changed to completely divorce it from the parent filesystem and have it still be usable. I'm imagining that literally grabbing a chunk of the filesystem -- using its own storage layer -- would be faster than tarring. Think like if directories had blocklists as well as files. It would also be smaller - -- see below. And it completely eliminates the question of "what may I never back up" -- the fs already knows that. Without having to be told by plugins. If the fs stores it to disk, the backup stores it, unless it's part of the "cache" proposal. This is a rough idea, and has issues. For instance, how do we know what's cache and what's not? If you have foo.gz/ugz cached and someone writes to foo.gz/ugz, I think that should be written to the cache and the original foo.gz should be removed -- regenerated if necessary from foo.gz/ugz. But the reverse is true -- write to foo.gz and foo.gz/ugz should go away, to be regenerated if necessary. Only one should go into the backup, though. I think foo.gz goes into the backup and is regenerated from foo.gz/ugz if necessary. More generally, when you have two redundant streams, backup the smaller one. |> |> Imagine 5000 tiny little files that, by the time reiser's done with |> them, fit inside 5 blocks or so. It's insanely cheaper to copy the five |> blocks than to tar them up. It also means that we don't have to worry |> about supporting tar on other platforms -- if they don't have reiser4, |> they need a separate converter. If apps are going to use reiser4 |> metadata for anything significant, they probably would link against some |> sort of userland implementation of reiser4 as a Windows library, say, to |> access the files elsewhere. |> |> That's extreme, but you get the idea. | | P.S. Sorry for the dupe, Hans, accidently hit "reply" instead of "reply-to-all"... -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTUuS3gHNmZLgCUhAQKWCQ/+LTNqHvndRAMqzdAJ0rugZl30bqqPmrFs vNqSwEx09RUxN6caOd+C4F7oVj/as0AJrNqyN5YR6/t6BKGxh+xEDnKdKiIRR5qp /TJmsMSkh54MSqaZZPL5EB6/DjWscBN7zKCaaCN6f8xB6KU2BCOK5pNSHDvYhISA uyD8AXrsOLRGC82dNtmh68KdrG+7VcTZMohulXNnosLLRJSuKeYEphB0i5HrvXQn nZ010yb6LnmKtNKD+w7bFxt3ul+lQo8gti7zBJqb7mV4pa1MnQNhGZRgvey/Pq/l ahXrrABauUFNwM4PfFWUoY9zjOdCufQpWev/mNGuoo9BS2ZiRg8LXoUpf2bOvi6I yOTMRb37w6+w37Jv8B0vYPehW3bmuJpb+jDyZ5AYS0GygJmm3bEIFZYupYt2pU/v 7EBIzLszCEnqPbm0glGdWT5jyYiZGucvDK+W4qQ3ymEqaBIdGIti4G56mQ7bTQer DPVT4dv38OE/PEtKb6xLQUaos7hBg+sWM5+mT9B+7kX/cYDhtdBkZpYqf10jAg34 4ufl2NS48CxqCcml7RayPVAuKT1ZN9sDcfXVYnPw33R94/lziZnfJrIInbnf9MZK +P9v6WlFi6+9GZjwqFK9ZCJBoTCKRq1lGiy5flTDf1APAgpmU6CS/0Da4s1QiLLI YRoeX8Nv8Us= =WFSh -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:57 ` Rik van Riel 2004-08-26 18:15 ` Linus Torvalds @ 2004-08-26 18:28 ` Denis Vlasenko 2004-08-26 18:46 ` Linus Torvalds 2004-08-26 19:36 ` silent semantic changes with reiser4 Jonathan Abbey 1 sibling, 2 replies; 1392+ messages in thread From: Denis Vlasenko @ 2004-08-26 18:28 UTC (permalink / raw) To: Rik van Riel, Linus Torvalds Cc: Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thursday 26 August 2004 20:57, Rik van Riel wrote: > On Thu, 26 Aug 2004, Linus Torvalds wrote: > > > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ? > > > > No no. The stream you get from a directory is totally _independent_ of > > the contents of the directory. Anything else would be pointless. > > It's a relief to know that nobody's taking my humorous > suggestion seriously, but now we still have the "standard > Unix tools can't manipulate files" problem... Is it possible to sufficiently hide "dirs inside files" so that old tools will be unable to see them? I just checked: ls -d /foo does lstat64("/foo", ...) ls -d /foo/ does lstat64("/foo", ...) but ls -d /foo/. does lstat64("/foo/.", ...) Will it work out if "dir inside file" will only be visible when referred as "file/."? -- vda ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:28 ` Denis Vlasenko @ 2004-08-26 18:46 ` Linus Torvalds 2004-08-26 19:13 ` viro 2004-08-26 19:36 ` silent semantic changes with reiser4 Jonathan Abbey 1 sibling, 1 reply; 1392+ messages in thread From: Linus Torvalds @ 2004-08-26 18:46 UTC (permalink / raw) To: Denis Vlasenko Cc: Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Denis Vlasenko wrote: > > Is it possible to sufficiently hide "dirs inside files" > so that old tools will be unable to see them? Certainly possible. > I just checked: > > ls -d /foo does lstat64("/foo", ...) > ls -d /foo/ does lstat64("/foo", ...) > but > ls -d /foo/. does lstat64("/foo/.", ...) > > Will it work out if "dir inside file" will only be visible when referred as "file/."? That would likely be the _easiest_ approach for the kernel, and does solve the problem with some apps knowingly removing the trailing '/'. Note that we could try this out with existing filesystems with very minimal changes: - make directory bind mounts work on top of files ("graft_tree()") - make open_namei() and friend _not_ do the mount-point following for the last component if it's a non-directory. - probably some trivial fixups I haven't thought about. There might be some places that use "S_ISDIR()" to check for whether something can be looked up, but the main path walking already just checks whether there is a ".lookup" operation or not. This would already allow people to "try out" how different applications would react to a file that can show up both as a directory and a file. The patch might end up being less than 25 lines or so, the difficulty is in finding all the right places. Al, anything I missed? (And yes, it's a quick hack, but it's a quick hack that would probably mimic a good part of what we would have to do internally in the VFS layer to support this notion anyway). Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:46 ` Linus Torvalds @ 2004-08-26 19:13 ` viro 2004-08-26 20:32 ` viro 0 siblings, 1 reply; 1392+ messages in thread From: viro @ 2004-08-26 19:13 UTC (permalink / raw) To: Linus Torvalds Cc: Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [reposted in thread] On Thu, Aug 26, 2004 at 11:46:33AM -0700, Linus Torvalds wrote: > Note that we could try this out with existing filesystems with very > minimal changes: > > - make directory bind mounts work on top of files ("graft_tree()") > - make open_namei() and friend _not_ do the mount-point following for the > last component if it's a non-directory. > - probably some trivial fixups I haven't thought about. There might be > some places that use "S_ISDIR()" to check for whether something can be > looked up, but the main path walking already just checks whether there > is a ".lookup" operation or not. > > This would already allow people to "try out" how different applications > would react to a file that can show up both as a directory and a file. The > patch might end up being less than 25 lines or so, the difficulty is in > finding all the right places. The real issue is what to do with unlink() et.al. on these guys. Note that "unlink is OK if all we have there is a bunch of directory mounts" won't work well - we have no good way to check that condition. Even funnier one is what we do if we have directory mounted there *and* have something mounted on stuff in that directory. Yes, that's one of the probable directions for such stuff, but there's a lot of fun semantics questions and answers to them will matter a lot. Hey, if we lose the "can't unlink/rmdir/rename over something that is a mountpoint in other life" - I'm happy and we can get a lot of much more interesting stuff to work. It will take some work (e.g. making sure we can find all vfsmounts over given mountpoint and sorting out the locking issues, which won't be trivial), but the main obstacle in that direction is not in architecture - it's in SuS and tradition; as the matter of fact, our life would be much easier if we stopped trying to give -EBUSY here and just dissolved all subtrees mounted on anything that has that dentry. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 19:13 ` viro @ 2004-08-26 20:32 ` viro 2004-08-26 20:36 ` Christophe Saout 2004-08-26 20:47 ` Linus Torvalds 0 siblings, 2 replies; 1392+ messages in thread From: viro @ 2004-08-26 20:32 UTC (permalink / raw) To: Linus Torvalds Cc: Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 08:13:23PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote: > Hey, if we lose the "can't unlink/rmdir/rename over something that is > a mountpoint in other life" - I'm happy and we can get a lot of much > more interesting stuff to work. It will take some work (e.g. making > sure we can find all vfsmounts over given mountpoint and sorting out > the locking issues, which won't be trivial), but the main obstacle in > that direction is not in architecture - it's in SuS and tradition; as > the matter of fact, our life would be much easier if we stopped trying > to give -EBUSY here and just dissolved all subtrees mounted on anything > that has that dentry. Argh... OK, now I remember why I went for -EBUSY for unlink() (we obviously are not bound by SuS on that one). Consider the following scenario: * local file foo got something else bound on it for a while * we are tight on space - time to clean up * oh, look - contents of foo is junk * rm foo * ... oh, fuck, there goes the underlying file. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:32 ` viro @ 2004-08-26 20:36 ` Christophe Saout 2004-08-26 20:47 ` Linus Torvalds 1 sibling, 0 replies; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 20:36 UTC (permalink / raw) To: viro Cc: Linus Torvalds, Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 619 bytes --] Am Donnerstag, den 26.08.2004, 21:32 +0100 schrieb : > Argh... OK, now I remember why I went for -EBUSY for unlink() (we obviously > are not bound by SuS on that one). Consider the following scenario: > * local file foo got something else bound on it for a while > * we are tight on space - time to clean up > * oh, look - contents of foo is junk > * rm foo > * ... oh, fuck, there goes the underlying file. Right. Good thinking. You can't delete a directory while there regular files in it either. Another question: /mnt/test is a mountpoint. Why can I rename `mnt' but can't rename `test'? [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:32 ` viro 2004-08-26 20:36 ` Christophe Saout @ 2004-08-26 20:47 ` Linus Torvalds 2004-08-26 21:28 ` [some sanity for a change] possible design issues for hybrids viro 1 sibling, 1 reply; 1392+ messages in thread From: Linus Torvalds @ 2004-08-26 20:47 UTC (permalink / raw) To: viro Cc: Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > > Argh... OK, now I remember why I went for -EBUSY for unlink() (we obviously > are not bound by SuS on that one). Consider the following scenario: > * local file foo got something else bound on it for a while > * we are tight on space - time to clean up > * oh, look - contents of foo is junk > * rm foo > * ... oh, fuck, there goes the underlying file. Hey, that's a valid reason for doing -EBUSY for normal bind-mounts, but it actually _is_ what we want for an "implied-by-way-of-container-mount". After all, when you do a "rm foo", you do mean "remove the container foo". I replied to your earlier off-list mail in private, so let's re-iterate for the list: the easiest way to handle this is to just have a "mount option", and have "MNT_ALLOWUNLINK" that gets set for containers, and that users could possibly choose to set for regular mounts too (as a mount option) if they really want to (and if we want them to). So there's no reason we'd have to drop existing mount behaviour only because we also have special files that look like mountpoints. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* [some sanity for a change] possible design issues for hybrids 2004-08-26 20:47 ` Linus Torvalds @ 2004-08-26 21:28 ` viro 2004-08-26 22:04 ` Linus Torvalds 0 siblings, 1 reply; 1392+ messages in thread From: viro @ 2004-08-26 21:28 UTC (permalink / raw) To: Linus Torvalds Cc: Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [subject changed and *please* let's keep the wankfest out of that branch; we are talking about possible ways to handle hybrids, *NOT* their desirability, effect on DARPA funding, size of KDE developers' genitals experienced by Aunt Tillie, etc.] On Thu, Aug 26, 2004 at 01:47:37PM -0700, Linus Torvalds wrote: > Hey, that's a valid reason for doing -EBUSY for normal bind-mounts, but it > actually _is_ what we want for an "implied-by-way-of-container-mount". > After all, when you do a "rm foo", you do mean "remove the container foo". > > I replied to your earlier off-list mail in private, so let's re-iterate > for the list: the easiest way to handle this is to just have a "mount > option", and have "MNT_ALLOWUNLINK" that gets set for containers, and that > users could possibly choose to set for regular mounts too (as a mount > option) if they really want to (and if we want them to). > > So there's no reason we'd have to drop existing mount behaviour only > because we also have special files that look like mountpoints. All right, let's see where that would take us. 1) we would need to find all vfsmounts over given dentry. Probably a cyclic list (we want to check if there are normal mounts/bindings among those and we want to dissolve them if there's none). 2) we would need to do something about locking, since mount trees in other guys' namespaces are protected by semaphores of their own. 3) what do we do on umount(2)? We can get a bunch of vfsmounts hanging off it. MNT_DETACH will have no problems, but normal umount() is a different story. Note that it's not just hybrid-related problem - implementing the mount traps will cause the same kind of trouble, 4) OK, we have those hybrids and want to create vfsmounts when crossing a mountpoint. When do they go away, anyway? When we don't reference them anymore? Right now "attached to mount tree" == "+1 to refcount" and detaching happens explicitly - outside of the "dropping the final reference" path. Might become a locking issue. 5) Creation of these vfsmounts: fs should somehow tell us whether it wants one or not (at the very least, we should stop *somewhere*). Can we use the same dentry/inode? I'm not sure and I really doubt that we'd like that. 6) if it's a method, where should it live, *especially* if we want them on device nodes. Note that inode_operations belongs to underlying fs, so it's not particulary good place for device case. 7) automount folks want partially shared mount trees (well, mirrored, actually). The basic idea is that while namespace boundary is a trust boundary, we might want to be able to say "I trust this guy to handle that subtree under /home/stuff/foobar/mounts". It's the same situation as with shared mappings (I want separate address space, but I'm willing to share that chunk of memory with other process), except that we want it to be allowed to become asymmetric (shared r/o mapping with somebody else having it r/w). That stuff (and mount traps) is the next pending major work on the mount trees. We probably want hybrids work to go with it, since they affect the same data structures and need more or less the same changes. [And yes, this is the open season on design discussions for shared subtrees - automount folks are welcome to join] 8) what should happen when something is mounted on top of directory-over-file? How do we treat such beasts? What are the implications? 9) how do we recognize such mountpoints in the path lookups? It *is* a hot path, so we should be careful in that area; the impact will be felt by everything in the system. 10) how do we deal with directories, anyway? Mixing "attributes" with normal directory contents is going to be fun, what with lseek() insanity. That's not an issue for hybrids, but it is one for anybody who wants any sort of common metadata exported that way. Note that it's really important for fs writers - having two different pieces of code to export the same information (for directories and non-directories resp.) is going to become a prime breeding ground for bugs. 11) if we go for your "here's stuff that belongs in device node viewed as directory", how would that play with fs metadata exporters? Again, due to the insanity of lseek() on directories it's *very* hard to deal with unions, when parts of directory come from different chunks of code. That's it for starters. Technical answers/questions/comments are welcome. Generic masturbation => over there in the parent thread, please. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: [some sanity for a change] possible design issues for hybrids 2004-08-26 21:28 ` [some sanity for a change] possible design issues for hybrids viro @ 2004-08-26 22:04 ` Linus Torvalds 2004-08-26 22:36 ` viro 2004-08-27 8:56 ` Anton Altaparmakov 0 siblings, 2 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-08-26 22:04 UTC (permalink / raw) To: viro Cc: Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [ This is quite possibly just impossible and buggy, but here's my implementation notes. You asked for them. ] On Thu, 26 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > > All right, let's see where that would take us. > > 1) we would need to find all vfsmounts over given dentry. Probably a cyclic > list (we want to check if there are normal mounts/bindings among those and > we want to dissolve them if there's none). Not per-inode? dentries are a bitmore memory-constrained than inodes, and we only need this for filesystems that want to support it, so we wouldn't need to put this information in each dentry. Since the vfsmounts have back-pointers to the dentry they are mounted on, you can still do a "per-dentry" traversal, by just doign the inode list and checking the dentry pointer. No? Alternatively, we could just put the list on an external hash-chain entirely, and hash off the dentry. It depends on how often we end up needing it. Or we could just put it in the dentry itself. I'd hate to make it any bigger than it already is, but maybe it doesn't matter that much. > 2) we would need to do something about locking, since mount trees in other > guys' namespaces are protected by semaphores of their own. Ok, I'll admit that I don't know how to handle namespaces. These things should just go into a global namespace, and I was kind of assuming it would happen automatically in "lookup_mnt()" or something like that. A special case in lookup_mnt which says something like "if you didn't find a vfsmount, we create a new one for you". It should be reasonably easy to create new ones on-the-fly, since we'd have all the information (the parent vfsmount comes stated, and the vfsmount we create would point to the same things that the "base" one would). > 3) what do we do on umount(2)? We can get a bunch of vfsmounts hanging off > it. MNT_DETACH will have no problems, but normal umount() is a different > story. Note that it's not just hybrid-related problem - implementing the > mount traps will cause the same kind of trouble, Don't allow umount. It's not something the user can unmount - the mount is "implied" in the file. > 4) OK, we have those hybrids and want to create vfsmounts when crossing a > mountpoint. When do they go away, anyway? When we don't reference them > anymore? Right now "attached to mount tree" == "+1 to refcount" and detaching > happens explicitly - outside of the "dropping the final reference" path. > Might become a locking issue. Ahh. Umm.. Yes. I think this might be the real problem. Unless I seriously clossed something over when I blathered about the "create the vfsmount on the fly" thing above ;) > 5) Creation of these vfsmounts: fs should somehow tell us whether it wants > one or not (at the very least, we should stop *somewhere*). Can we use > the same dentry/inode? I'm not sure and I really doubt that we'd like that. Why not? When doing the ->lookup() operation, the filesystem would create the vfsmount and bind it to the current vfsmount. That guarantees that it has a vfsmount, and will mean that it will show up positive with the "d_mountpoint()" query, which in turn will cause us to do the "lookup_mnt()". Which in turn will create the other vfsmounts as needed, if you have multiple namespaces. So I _think_ creation is easy. Getting rid of the dang things migth be harder. > 6) if it's a method, where should it live, *especially* if we want them on > device nodes. Note that inode_operations belongs to underlying fs, so it's > not particulary good place for device case. Why not just let the existing .lookup method initialize the mount-point thing? After that, it's all in the VFS layer (I'd hate to have filesystems mess around with vfsmounts - they'll just get it wrong). > 7) automount folks want partially shared mount trees (well, mirrored, > actually). I don't think you can get partial sharing on one of these puppies. You'd always have one vfsmount per namespace (well, lazily created, so maybe in practice you'd see a lot fewer). > 8) what should happen when something is mounted on top of directory-over-file? > How do we treat such beasts? What are the implications? Allow file-on-file mounts - it will just totally hide the thing (in that namespace, at least). But don't allow the dir-on-file thing (that we already don't allow). > 9) how do we recognize such mountpoints in the path lookups? It *is* a > hot path, so we should be careful in that area; the impact will be felt > by everything in the system. I don't think you'll have any special cases. Same d_mountpount(), same lookup_mnt(). > 10) how do we deal with directories, anyway? Mixing "attributes" with > normal directory contents is going to be fun, what with lseek() insanity. You couldn't get at the attributes that way anyway, so I think the point is moot. The "real" directory always takes over. Crazy people could try to just use the regular "xattrs" interfaces if they really want attributes on directories. You wouldn't ever be able to use the "easy" one. > 11) if we go for your "here's stuff that belongs in device node viewed > as directory", how would that play with fs metadata exporters? Again, > due to the insanity of lseek() on directories it's *very* hard to deal > with unions, when parts of directory come from different chunks of code. Don't go there. See above. Directories would be just plain directories, you could never see their metadata. Same goes for at least symlinks, and possibly other filetypes too (ie at least initially, a block or character special device will just take over the whole "file_operations", which includes "readdir", so it's actually hard to have the filesystem do anything about those). Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: [some sanity for a change] possible design issues for hybrids 2004-08-26 22:04 ` Linus Torvalds @ 2004-08-26 22:36 ` viro 2004-08-26 22:45 ` Linus Torvalds 2004-08-27 8:56 ` Anton Altaparmakov 1 sibling, 1 reply; 1392+ messages in thread From: viro @ 2004-08-26 22:36 UTC (permalink / raw) To: Linus Torvalds Cc: Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 03:04:21PM -0700, Linus Torvalds wrote: > > 2) we would need to do something about locking, since mount trees in other > > guys' namespaces are protected by semaphores of their own. > > Ok, I'll admit that I don't know how to handle namespaces. These things > should just go into a global namespace, and I was kind of assuming it > would happen automatically in "lookup_mnt()" or something like that. A > special case in lookup_mnt which says something like "if you didn't find a > vfsmount, we create a new one for you". > > It should be reasonably easy to create new ones on-the-fly, since we'd > have all the information (the parent vfsmount comes stated, and the > vfsmount we create would point to the same things that the "base" one > would). Erm... What do we do upon unlink()? I'm killing a file, fs it's in is mounted in a dozen of places (no namespaces, just chroot jails, whatever). We need to find all vfsmounts to be killed by that. And BTW that's an argument against anchoring that list in inode - unlink() on foo should not screw bar/... even if bar and foo are links to the same file. So we'll need to check for dentry match anyway. > > 3) what do we do on umount(2)? We can get a bunch of vfsmounts hanging off > > it. MNT_DETACH will have no problems, but normal umount() is a different > > story. Note that it's not just hybrid-related problem - implementing the > > mount traps will cause the same kind of trouble, > > Don't allow umount. It's not something the user can unmount - the mount is > "implied" in the file. See below. > > 4) OK, we have those hybrids and want to create vfsmounts when crossing a > > mountpoint. When do they go away, anyway? When we don't reference them > > anymore? Right now "attached to mount tree" == "+1 to refcount" and detaching > > happens explicitly - outside of the "dropping the final reference" path. > > Might become a locking issue. > > Ahh. Umm.. Yes. I think this might be the real problem. Unless I seriously > clossed something over when I blathered about the "create the vfsmount on > the fly" thing above ;) > > 5) Creation of these vfsmounts: fs should somehow tell us whether it wants > > one or not (at the very least, we should stop *somewhere*). Can we use > > the same dentry/inode? I'm not sure and I really doubt that we'd like that. > > Why not? When doing the ->lookup() operation, the filesystem would create > the vfsmount and bind it to the current vfsmount. That guarantees that it > has a vfsmount, and will mean that it will show up positive with the > "d_mountpoint()" query, which in turn will cause us to do the > "lookup_mnt()". Several paragraphs below you are saying that you don't like fs messing with vfsmounts. Use of ->lookup() would mean that we should not only create and attach vfsmounts from within fs code, but would actually have to make ->lookup() return vfsmount+dentry, AFAICS. > > 6) if it's a method, where should it live, *especially* if we want them on > > device nodes. Note that inode_operations belongs to underlying fs, so it's > > not particulary good place for device case. > > Why not just let the existing .lookup method initialize the mount-point > thing? After that, it's all in the VFS layer (I'd hate to have filesystems > mess around with vfsmounts - they'll just get it wrong). > Allow file-on-file mounts - it will just totally hide the thing (in that > namespace, at least). But don't allow the dir-on-file thing (that we > already don't allow). Err... What about dir-on-dir-that-is-on-file? I.e. mount on foo/. when foo is a file? > > 9) how do we recognize such mountpoints in the path lookups? It *is* a > > hot path, so we should be careful in that area; the impact will be felt > > by everything in the system. > I don't think you'll have any special cases. Same d_mountpount(), same > lookup_mnt(). See above on use ->lookup() ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: [some sanity for a change] possible design issues for hybrids 2004-08-26 22:36 ` viro @ 2004-08-26 22:45 ` Linus Torvalds 2004-08-26 22:53 ` viro 0 siblings, 1 reply; 1392+ messages in thread From: Linus Torvalds @ 2004-08-26 22:45 UTC (permalink / raw) To: viro Cc: Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > > > > It should be reasonably easy to create new ones on-the-fly, since we'd > > have all the information (the parent vfsmount comes stated, and the > > vfsmount we create would point to the same things that the "base" one > > would). > > Erm... What do we do upon unlink()? I'm killing a file, fs it's in is > mounted in a dozen of places (no namespaces, just chroot jails, whatever). > We need to find all vfsmounts to be killed by that. But that should be trivial: that's what the per-inode vfsmount list was (your first question in the last email). > And BTW that's an argument against anchoring that list in inode - unlink() > on foo should not screw bar/... even if bar and foo are links to the same > file. So we'll need to check for dentry match anyway. And again - I talked about this in the previous email. Even if you anchor the list in "struct inode", or you do it with a totally external hash-list, you'll always have the "vfsmount->mnt_mountpoint" pointer to point to the dentry. So you can just iterate over the list, and cherry-pick the ones that point to the dentry you are removing. > > > > 3) what do we do on umount(2)? We can get a bunch of vfsmounts hanging off > > > it. MNT_DETACH will have no problems, but normal umount() is a different > > > story. Note that it's not just hybrid-related problem - implementing the > > > mount traps will cause the same kind of trouble, > > > > Don't allow umount. It's not something the user can unmount - the mount is > > "implied" in the file. > > See below. > > > > 4) OK, we have those hybrids and want to create vfsmounts when crossing a > > > mountpoint. When do they go away, anyway? When we don't reference them > > > anymore? Right now "attached to mount tree" == "+1 to refcount" and detaching > > > happens explicitly - outside of the "dropping the final reference" path. > > > Might become a locking issue. > > > > Ahh. Umm.. Yes. I think this might be the real problem. Unless I seriously > > clossed something over when I blathered about the "create the vfsmount on > > the fly" thing above ;) > > > > 5) Creation of these vfsmounts: fs should somehow tell us whether it wants > > > one or not (at the very least, we should stop *somewhere*). Can we use > > > the same dentry/inode? I'm not sure and I really doubt that we'd like that. > > > > Why not? When doing the ->lookup() operation, the filesystem would create > > the vfsmount and bind it to the current vfsmount. That guarantees that it > > has a vfsmount, and will mean that it will show up positive with the > > "d_mountpoint()" query, which in turn will cause us to do the > > "lookup_mnt()". > > Several paragraphs below you are saying that you don't like fs messing with > vfsmounts. Use of ->lookup() would mean that we should not only create > and attach vfsmounts from within fs code, but would actually have to make > ->lookup() return vfsmount+dentry, AFAICS. No, lookup would just return the dentry, but the dentry would already be filled in with the mount-point information. And you can do that with a simple vfs helper function, ie the filesystem itself would just need to do pseudo_mount(dentry, inode); thing - which just fills in dentry->d_mountpoint with a new vfsmount thing. It would allocate a new root dentry (for the pseudo-mount) and a new vfsmount, and make dentry->d_mountpoint point to it. IOW, the filesystem itself would never mess around with d_mountpoint itself. > Err... What about dir-on-dir-that-is-on-file? I.e. mount on foo/. when foo > is a file? Hmm.. We might as well allow it, I suspect. It's not like it should hurt. We'd end up following the mount-chain twice, but we already have that issue with multi-mount cases.. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: [some sanity for a change] possible design issues for hybrids 2004-08-26 22:45 ` Linus Torvalds @ 2004-08-26 22:53 ` viro 2004-08-26 23:24 ` Linus Torvalds 0 siblings, 1 reply; 1392+ messages in thread From: viro @ 2004-08-26 22:53 UTC (permalink / raw) To: Linus Torvalds Cc: Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 03:45:09PM -0700, Linus Torvalds wrote: > No, lookup would just return the dentry, but the dentry would already be > filled in with the mount-point information. > > And you can do that with a simple vfs helper function, ie the filesystem > itself would just need to do > > pseudo_mount(dentry, inode); > > thing - which just fills in dentry->d_mountpoint with a new vfsmount > thing. It would allocate a new root dentry (for the pseudo-mount) and a > new vfsmount, and make dentry->d_mountpoint point to it. What dentry->d_mountpoint? No such thing... Note that we can't get vfsmount by dentry - that's the point of having these guys in the first place. So I'm not sure what you are trying to do here - dentry + inode is definitely not enough to attach any vfsmounts anywhere. That's not about namespaces - same fs mounted in several places will give the same problem - one dentry, many vfsmounts. And we obviously *can't* have one vfsmount for all of them - if the same fs is mounted on /foo and /bar, we will have the same dentry for /foo/splat and /bar/splat. So what should we get for /foo/splat/. and /bar/splat/.? Same dentry *and* same vfsmount? I'd expect .. from the former to give /foo and from the latter - /bar... ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: [some sanity for a change] possible design issues for hybrids 2004-08-26 22:53 ` viro @ 2004-08-26 23:24 ` Linus Torvalds 2004-08-26 23:40 ` viro 0 siblings, 1 reply; 1392+ messages in thread From: Linus Torvalds @ 2004-08-26 23:24 UTC (permalink / raw) To: viro Cc: Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > > What dentry->d_mountpoint? No such thing... Sorry - set "dentry->d_mounted++" + "add vfsmount/dentry to hashes". Yes, it's not a direct list off the dentry, but it effectively is the same thing. So basically: the "d_mounted++" just makes sure we get into "lookup_mnt()". That's where we will usually find the actual mount thing. And that's also where the special case comes in: if we _don't_ find the mount thing there, that's where we need to create it. That will only happen if somebody looks it up using another namespace, though, so it should be rare. And when it does happen, we can just create a new vfsmount - we have all the information there (we'll have to walk the per-inode-or-whatever vfsmount list to find all the information to populate the thing with, of course. But we need that list _anyway_, so it should be a fairly straightforward special case). Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: [some sanity for a change] possible design issues for hybrids 2004-08-26 23:24 ` Linus Torvalds @ 2004-08-26 23:40 ` viro 2004-08-26 23:57 ` Linus Torvalds 0 siblings, 1 reply; 1392+ messages in thread From: viro @ 2004-08-26 23:40 UTC (permalink / raw) To: Linus Torvalds Cc: Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 04:24:51PM -0700, Linus Torvalds wrote: > So basically: the "d_mounted++" just makes sure we get into > "lookup_mnt()". That's where we will usually find the actual mount thing. > > And that's also where the special case comes in: if we _don't_ find the > mount thing there, that's where we need to create it. That will only > happen if somebody looks it up using another namespace, though, so it > should be rare. No. Trivial example: mount --bind /foo /bar mount /dev/sda1 /bar/baz do lookup for /foo/baz. No namespaces involved, no vfsmounts found, d_mounted positive and we certainly do *not* want anything to be created at that point. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: [some sanity for a change] possible design issues for hybrids 2004-08-26 23:40 ` viro @ 2004-08-26 23:57 ` Linus Torvalds 2004-08-27 1:01 ` viro 0 siblings, 1 reply; 1392+ messages in thread From: Linus Torvalds @ 2004-08-26 23:57 UTC (permalink / raw) To: viro Cc: Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, 27 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > On Thu, Aug 26, 2004 at 04:24:51PM -0700, Linus Torvalds wrote: > > So basically: the "d_mounted++" just makes sure we get into > > "lookup_mnt()". That's where we will usually find the actual mount thing. > > > > And that's also where the special case comes in: if we _don't_ find the > > mount thing there, that's where we need to create it. That will only > > happen if somebody looks it up using another namespace, though, so it > > should be rare. > > No. Trivial example: > > mount --bind /foo /bar > mount /dev/sda1 /bar/baz > > do lookup for /foo/baz. No namespaces involved, no vfsmounts found, d_mounted > positive and we certainly do *not* want anything to be created at that point. Right. We obviously need to mark the dentry somehow, and only do the "create vfsmount" special case in this special case. If we didn't do that, then it wouldn't be a special case, now would it? So clearly lookup_mnt() needs to check the dentry in the failure case. The marking could be in any of three places - mark the dentry itself by just using a dentry flag ("DCACHE_AUTOVFSMNT") or by having a dentry operation for this. - mark the inode itself (same logic as dentry) - look up the first vfsmount (on the inode list), and look if that one is of the automatic type. Clearly we should not _always_ create a vfsmount, that would just break the existign logic. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: [some sanity for a change] possible design issues for hybrids 2004-08-26 23:57 ` Linus Torvalds @ 2004-08-27 1:01 ` viro 0 siblings, 0 replies; 1392+ messages in thread From: viro @ 2004-08-27 1:01 UTC (permalink / raw) To: Linus Torvalds Cc: Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 04:57:01PM -0700, Linus Torvalds wrote: > Right. We obviously need to mark the dentry somehow, and only do the > "create vfsmount" special case in this special case. If we didn't do that, > then it wouldn't be a special case, now would it? > > So clearly lookup_mnt() needs to check the dentry in the failure case. The > marking could be in any of three places > - mark the dentry itself by just using a dentry flag ("DCACHE_AUTOVFSMNT") > or by having a dentry operation for this. > - mark the inode itself (same logic as dentry) > - look up the first vfsmount (on the inode list), and look if that one is > of the automatic type. > > Clearly we should not _always_ create a vfsmount, that would just break > the existign logic. Hmm... IOW, you are suggesting to use normal trigger for lookup_mnt() and then have extra check in lookup_mnt() failure exit. That will probably work (and I'd prefer to mark dentry here). Freeing these guys will not be fun. The final reference could be given up under very different locking conditions - it's certainly a blocking operation (it can trigger final umount, among other things), but having it trigger removal from mount tree(s) can get ugly. We might be able to tweak that, but it will probably need require major surgery in several places. Right now we assume that vfsmount lock *and* per-namespace semaphore are held when detaching from the tree (attaching what could be the first at given dentry --- same + i_sem on its ->i_sem). We also assume that vfsmount is detached before refcount reaches zero. The thing is, that final refcount can be dropped while holding namespace semaphore. It's not fatal (we'll need to hold onto these guys until dropping that semaphore or add an analog of mntput() for places under namespace sem), but it can get messy. Situation with unlink(): we need to (a) check that all vfsmounts over given point are "hybrid" ones (locking is not a problem at that point) and (b) after successful operation (unlink() and rename() alike, AFAICS) we want to go and kill everything under the victim dentry. *That* can get very ugly, since fs could be mounted in a lot of places and/or in a lot of namespaces. Note that we also need to do in VFS what NFS tries to do in revalidation - if dentry is invalidated, everything under it should be detached and dropped. So it's not something new - we do need something that would be called outside of any superblock/namespace/vfsmount locks (both unlink() and revalidation qualify) and would kill the vfsmounts that become unreachable. Locking rules for attaching might need adjustment (the logics with ->i_sem is that we want !d_mountpoint() stay true around call of ->unlink(), ->rmdir() and ->rename(); so cloning or replacing doesn't need that protection. Now it might become interesting - we'll need to be careful in pivot_root(2) and possibly some other places). I'll see what can be done (it'll definitely take careful looking at the code), but there is a chance of things getting very ugly. BTW, shared subtrees *are* compatible with that puppy - there's a lot of corner cases to look at, but AFAICS they are all doable. Lifetime rules can get ugly in some of them, but that's on par with the situation without sharing. One thing that looks like a bad interface: we get forcible "use same dentry for file and directory" with that design. That can lead to a big can of worms - without a large audit I wouldn't bet a dime on ability to get that right (and same applies to reiser4 code as-is, for the same reasons). No idea how bad it will turn out to be - right now all I can say is that we might run into trouble and that we almost certainly will get extra "be careful not to do <list of things>" rules out of that. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: [some sanity for a change] possible design issues for hybrids 2004-08-26 22:04 ` Linus Torvalds 2004-08-26 22:36 ` viro @ 2004-08-27 8:56 ` Anton Altaparmakov 2004-08-27 9:00 ` Christoph Hellwig 1 sibling, 1 reply; 1392+ messages in thread From: Anton Altaparmakov @ 2004-08-27 8:56 UTC (permalink / raw) To: Linus Torvalds Cc: viro, Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, Andrew Morton, wichert, jra, reiser, hch, linux-fsdevel, lkml, flx, reiserfs-list On Thu, 2004-08-26 at 23:04, Linus Torvalds wrote: > [ This is quite possibly just impossible and buggy, but here's my > implementation notes. You asked for them. ] > On Thu, 26 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote: > > All right, let's see where that would take us. > > 3) what do we do on umount(2)? We can get a bunch of vfsmounts hanging off > > it. MNT_DETACH will have no problems, but normal umount() is a different > > story. Note that it's not just hybrid-related problem - implementing the > > mount traps will cause the same kind of trouble, > > Don't allow umount. It's not something the user can unmount - the mount is > "implied" in the file. > > > 4) OK, we have those hybrids and want to create vfsmounts when crossing a > > mountpoint. When do they go away, anyway? When we don't reference them > > anymore? Right now "attached to mount tree" == "+1 to refcount" and detaching > > happens explicitly - outside of the "dropping the final reference" path. > > Might become a locking issue. > > Ahh. Umm.. Yes. I think this might be the real problem. Unless I seriously > clossed something over when I blathered about the "create the vfsmount on > the fly" thing above ;) If it is created on the fly, it should be "easy" to destroy on the fly using time-based expiry, i.e. a kernel daemon going over all of those beasts every X seconds (X = 5 perhaps?) and doing something like: for (each vfsmount) { lock_vfsmount(vfsmount); if (MOUNT_IS_BUSY(vfsmount)) { unlock_vfsmount(vfsmount); continue; } if (current_time() < (vfsmount->last_used_time + vfsmount->expire_after)) { unlock_vfsmount(vfsmount); continue; } destroy_locked_vfsmount(vfsmount); } Wouldn't that work? > > 10) how do we deal with directories, anyway? Mixing "attributes" with > > normal directory contents is going to be fun, what with lseek() insanity. > > You couldn't get at the attributes that way anyway, so I think the point > is moot. The "real" directory always takes over. > > Crazy people could try to just use the regular "xattrs" interfaces if they > really want attributes on directories. You wouldn't ever be able to use > the "easy" one. But that defeats the whole point of the hybrid objects! We might as well just keep the xattrs interface and throw away the new one if we will have to keep the old one anyway so that we can do named streams/attributes on directories. Windows (and other OS?) certainly do allow them and do use them on directories as well as files... > > 11) if we go for your "here's stuff that belongs in device node viewed > > as directory", how would that play with fs metadata exporters? Again, > > due to the insanity of lseek() on directories it's *very* hard to deal > > with unions, when parts of directory come from different chunks of code. > > Don't go there. See above. Directories would be just plain directories, > you could never see their metadata. Same goes for at least symlinks, and > possibly other filetypes too (ie at least initially, a block or character > special device will just take over the whole "file_operations", which > includes "readdir", so it's actually hard to have the filesystem do > anything about those). Ah, but at least in other OS (Windows is the one I know about) directories _also_ have the same semantics as files with respect to named streams/attributes. Best regards, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net WWW: http://linux-ntfs.sf.net/, http://www-stu.christs.cam.ac.uk/~aia21/ ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: [some sanity for a change] possible design issues for hybrids 2004-08-27 8:56 ` Anton Altaparmakov @ 2004-08-27 9:00 ` Christoph Hellwig 0 siblings, 0 replies; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-27 9:00 UTC (permalink / raw) To: Anton Altaparmakov Cc: Linus Torvalds, viro, Denis Vlasenko, Rik van Riel, Diego Calleja, jamie, christophe, christer, spam, Andrew Morton, wichert, jra, reiser, hch, linux-fsdevel, lkml, flx, reiserfs-list On Fri, Aug 27, 2004 at 09:56:39AM +0100, Anton Altaparmakov wrote: > If it is created on the fly, it should be "easy" to destroy on the fly > using time-based expiry, i.e. a kernel daemon going over all of those > beasts every X seconds (X = 5 perhaps?) and doing something like: > > for (each vfsmount) { > lock_vfsmount(vfsmount); > if (MOUNT_IS_BUSY(vfsmount)) { > unlock_vfsmount(vfsmount); > continue; > } > if (current_time() < (vfsmount->last_used_time + > vfsmount->expire_after)) { > unlock_vfsmount(vfsmount); > continue; > } > destroy_locked_vfsmount(vfsmount); > } > > Wouldn't that work? That would work for a low number of them. But with Hans' "visions" we'd have a damn lot of them at which point this isn't really scalable. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:28 ` Denis Vlasenko 2004-08-26 18:46 ` Linus Torvalds @ 2004-08-26 19:36 ` Jonathan Abbey 2004-08-26 20:16 ` Jamie Lokier 1 sibling, 1 reply; 1392+ messages in thread From: Jonathan Abbey @ 2004-08-26 19:36 UTC (permalink / raw) To: Denis Vlasenko Cc: Rik van Riel, Linus Torvalds, Diego Calleja, jamie, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 930 bytes --] On Thu, Aug 26, 2004 at 09:28:41PM +0300, Denis Vlasenko wrote: | | Is it possible to sufficiently hide "dirs inside files" | so that old tools will be unable to see them? | | I just checked: | | ls -d /foo does lstat64("/foo", ...) | ls -d /foo/ does lstat64("/foo", ...) | but | ls -d /foo/. does lstat64("/foo/.", ...) | | Will it work out if "dir inside file" will only be visible when referred as "file/."? I'm used to using ls symlink/. to get ls to show me the directory on the far side of a symbolic link. That's a pretty analagous case to the one we're discussing here, I think? Jon -- ------------------------------------------------------------------------------- Jonathan Abbey jonabbey@arlut.utexas.edu Applied Research Laboratories The University of Texas at Austin GPG Key: 71767586 at keyserver pgp.mit.edu, http://www.ganymeta.org/workkey.gpg [-- Attachment #2: Type: application/pgp-signature, Size: 185 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 19:36 ` silent semantic changes with reiser4 Jonathan Abbey @ 2004-08-26 20:16 ` Jamie Lokier 2004-08-26 20:25 ` Christophe Saout 0 siblings, 1 reply; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 20:16 UTC (permalink / raw) To: Jonathan Abbey Cc: Denis Vlasenko, Rik van Riel, Linus Torvalds, Diego Calleja, christophe, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Jonathan Abbey wrote: > | Will it work out if "dir inside file" will only be visible when > referred as "file/."? > > I'm used to using ls symlink/. to get ls to show me the directory on > the far side of a symbolic link. That's a pretty analagous case to > the one we're discussing here, I think? By the way, do symlinks have metadata? Where do you find it? :) -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:16 ` Jamie Lokier @ 2004-08-26 20:25 ` Christophe Saout 2004-08-26 21:45 ` Nikita Danilov 0 siblings, 1 reply; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 20:25 UTC (permalink / raw) To: Jamie Lokier Cc: Jonathan Abbey, Denis Vlasenko, Rik van Riel, Linus Torvalds, Diego Calleja, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 733 bytes --] Am Donnerstag, den 26.08.2004, 21:16 +0100 schrieb Jamie Lokier: > > | Will it work out if "dir inside file" will only be visible when > > referred as "file/."? > > > > I'm used to using ls symlink/. to get ls to show me the directory on > > the far side of a symbolic link. That's a pretty analagous case to > > the one we're discussing here, I think? > > By the way, do symlinks have metadata? Where do you find it? :) Oops. :) At least in reiser4 they don't have, or at least you can't access them. ln -s foo bar; cd bar/metas shows me the content of foo/metas. symlinks are special anyway. Their rights are 777. The only thing they can have is an owner. No chance to do a symlink/metas/uid then. Hmm. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:25 ` Christophe Saout @ 2004-08-26 21:45 ` Nikita Danilov 2004-08-26 21:48 ` Christophe Saout 0 siblings, 1 reply; 1392+ messages in thread From: Nikita Danilov @ 2004-08-26 21:45 UTC (permalink / raw) To: Christophe Saout Cc: Jamie Lokier, Jonathan Abbey, Denis Vlasenko, Rik van Riel, Linus Torvalds, Diego Calleja, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Christophe Saout writes: > Am Donnerstag, den 26.08.2004, 21:16 +0100 schrieb Jamie Lokier: > > > > | Will it work out if "dir inside file" will only be visible when > > > referred as "file/."? > > > > > > I'm used to using ls symlink/. to get ls to show me the directory on > > > the far side of a symbolic link. That's a pretty analagous case to > > > the one we're discussing here, I think? > > > > By the way, do symlinks have metadata? Where do you find it? :) > > Oops. :) > > At least in reiser4 they don't have, or at least you can't access them. They do. > ln -s foo bar; cd bar/metas shows me the content of foo/metas. That's because lookup for "bar" performs symlink resolution. > > symlinks are special anyway. Their rights are 777. The only thing they Symlink is represented by normal inode in UNIX. It has full set of attributes. It can even be hard-linked. > can have is an owner. No chance to do a symlink/metas/uid then. Hmm. > Nikita. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 21:45 ` Nikita Danilov @ 2004-08-26 21:48 ` Christophe Saout 2004-08-26 22:17 ` Emil Larsson 2004-08-26 22:21 ` Nikita Danilov 0 siblings, 2 replies; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 21:48 UTC (permalink / raw) To: Nikita Danilov Cc: Jamie Lokier, Jonathan Abbey, Denis Vlasenko, Rik van Riel, Linus Torvalds, Diego Calleja, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 453 bytes --] Am Freitag, den 27.08.2004, 01:45 +0400 schrieb Nikita Danilov: > > At least in reiser4 they don't have, or at least you can't access them. > > They do. > > > ln -s foo bar; cd bar/metas shows me the content of foo/metas. > > That's because lookup for "bar" performs symlink resolution. So I can't access them and it is pointless. ;-) BTW, I can do a cd metas/metas/metas/metas/plugin/metas... I don't think this makes sense. :) [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 21:48 ` Christophe Saout @ 2004-08-26 22:17 ` Emil Larsson 2004-08-26 22:21 ` Nikita Danilov 1 sibling, 0 replies; 1392+ messages in thread From: Emil Larsson @ 2004-08-26 22:17 UTC (permalink / raw) To: Christophe Saout Cc: Nikita Danilov, Jamie Lokier, Jonathan Abbey, Denis Vlasenko, Rik van Riel, Linus Torvalds, Diego Calleja, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Christophe Saout wrote: >Am Freitag, den 27.08.2004, 01:45 +0400 schrieb Nikita Danilov: > > > >> > At least in reiser4 they don't have, or at least you can't access them. >> >>They do. >> >> > ln -s foo bar; cd bar/metas shows me the content of foo/metas. >> >>That's because lookup for "bar" performs symlink resolution. >> >> > >So I can't access them and it is pointless. ;-) > >BTW, I can do a cd metas/metas/metas/metas/plugin/metas... I don't think >this makes sense. :) > > > I prefer this one: hepburn:/# mount /dev/sda3 on / type reiser4 (rw) hepburn:/# touch apa hepburn:/# chmod +x apa hepburn:/# cd apa hepburn:/apa# ls ls: reading directory .: Not a directory So there is already file-as-dir with legacy "ls" - and it works just like people seem to want it to do with legacy apps. Not at all. /Emil ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 21:48 ` Christophe Saout 2004-08-26 22:17 ` Emil Larsson @ 2004-08-26 22:21 ` Nikita Danilov 2004-08-26 22:29 ` Christophe Saout 2004-08-26 23:35 ` Hans Reiser 1 sibling, 2 replies; 1392+ messages in thread From: Nikita Danilov @ 2004-08-26 22:21 UTC (permalink / raw) To: Christophe Saout Cc: Jamie Lokier, Jonathan Abbey, Denis Vlasenko, Rik van Riel, Linus Torvalds, Diego Calleja, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Christophe Saout writes: > Am Freitag, den 27.08.2004, 01:45 +0400 schrieb Nikita Danilov: > > > > At least in reiser4 they don't have, or at least you can't access them. > > > > They do. > > > > > ln -s foo bar; cd bar/metas shows me the content of foo/metas. > > > > That's because lookup for "bar" performs symlink resolution. > > So I can't access them and it is pointless. ;-) > > BTW, I can do a cd metas/metas/metas/metas/plugin/metas... I don't think > this makes sense. :) Why? foo/metas is a file system object just like foo. It has owner, permission bits, so access to its meta-data should be provided, and uniform way to provide access to the file system object meta-data is to have these little magic files inside metas directory, which is a file system object just like metas. It has owner^@^@^@^@*** - Lisp stack overflow. RESET Nikita. > ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 22:21 ` Nikita Danilov @ 2004-08-26 22:29 ` Christophe Saout 2004-08-26 23:35 ` Hans Reiser 1 sibling, 0 replies; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 22:29 UTC (permalink / raw) To: Nikita Danilov Cc: Jamie Lokier, Jonathan Abbey, Denis Vlasenko, Rik van Riel, Linus Torvalds, Diego Calleja, christer, spam, akpm, wichert, jra, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1323 bytes --] Am Freitag, den 27.08.2004, 02:21 +0400 schrieb Nikita Danilov: > > BTW, I can do a cd metas/metas/metas/metas/plugin/metas... I don't think > > this makes sense. :) > > Why? foo/metas is a file system object just like foo. It has owner, > permission bits, so access to its meta-data should be provided, and > uniform way to provide access to the file system object meta-data is to > have these little magic files inside metas directory, which is a file > system object just like metas. It has owner^@^@^@^@*** - Lisp stack > overflow. RESET Yes, of course. But who's going to need that? The meta files are there to describe regular unix files. They're just exported using a file system name space. I don't think we need metadata to describe metadata. Assuming someone adds a directory inside a file where user can add some attributes/streams/resource forks. As Linus said, they are just attributes, even if they seem to behave like normal files, they're not, they belong to the real file. Adding attributes to attributes to attributes? That would be like adding ACLS to ACLS. Hello the madness. ;-) There really should be a point where you can't recurse further. Assume there is a backup application that always tries to open every dentry it finds as directory and then recurse. Oops. :-) [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 22:21 ` Nikita Danilov 2004-08-26 22:29 ` Christophe Saout @ 2004-08-26 23:35 ` Hans Reiser 2004-08-27 2:00 ` Stephen Wille Padnos 1 sibling, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 23:35 UTC (permalink / raw) To: Nikita Danilov Cc: Christophe Saout, Jamie Lokier, Jonathan Abbey, Denis Vlasenko, Rik van Riel, Linus Torvalds, Diego Calleja, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Nikita Danilov wrote: >Christophe Saout writes: > > Am Freitag, den 27.08.2004, 01:45 +0400 schrieb Nikita Danilov: > > > > > > At least in reiser4 they don't have, or at least you can't access them. > > > > > > They do. > > > > > > > ln -s foo bar; cd bar/metas shows me the content of foo/metas. > > > > > > That's because lookup for "bar" performs symlink resolution. > > > > So I can't access them and it is pointless. ;-) > > > > BTW, I can do a cd metas/metas/metas/metas/plugin/metas... I don't think > > this makes sense. :) > >Why? foo/metas is a file system object just like foo. It has owner, >permission bits, so access to its meta-data should be provided, and >uniform way to provide access to the file system object meta-data is to >have these little magic files inside metas directory, which is a file >system object just like metas. It has owner^@^@^@^@*** - Lisp stack >overflow. RESET > >Nikita. > > > >- >To unsubscribe from this list: send the line "unsubscribe linux-kernel" in >the body of a message to majordomo@vger.kernel.org >More majordomo info at http://vger.kernel.org/majordomo-info.html >Please read the FAQ at http://www.tux.org/lkml/ > > > > I think Christophe is a bit right here. While in general having meta-meta objects makes sense, in this particular instance, I don't see the functional need for it. Can you supply an example of where it would be useful? ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 23:35 ` Hans Reiser @ 2004-08-27 2:00 ` Stephen Wille Padnos 0 siblings, 0 replies; 1392+ messages in thread From: Stephen Wille Padnos @ 2004-08-27 2:00 UTC (permalink / raw) To: Hans Reiser Cc: Nikita Danilov, Christophe Saout, Jamie Lokier, Jonathan Abbey, Denis Vlasenko, Rik van Riel, Linus Torvalds, Diego Calleja, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Hans Reiser wrote: > Nikita Danilov wrote: > >> Christophe Saout writes: >> > Am Freitag, den 27.08.2004, 01:45 +0400 schrieb Nikita Danilov: >> > > > > At least in reiser4 they don't have, or at least you can't >> access them. >> > > > > They do. >> > > > > > ln -s foo bar; cd bar/metas shows me the content of >> foo/metas. >> > > > > That's because lookup for "bar" performs symlink resolution. >> > > So I can't access them and it is pointless. ;-) >> > > BTW, I can do a cd metas/metas/metas/metas/plugin/metas... I >> don't think >> > this makes sense. :) >> >> Why? foo/metas is a file system object just like foo. It has owner, >> permission bits, so access to its meta-data should be provided, and >> uniform way to provide access to the file system object meta-data is to >> have these little magic files inside metas directory, which is a file >> system object just like metas. It has owner^@^@^@^@*** - Lisp stack >> overflow. RESET >> >> Nikita. >> >> > - >> To unsubscribe from this list: send the line "unsubscribe >> linux-kernel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> Please read the FAQ at http://www.tux.org/lkml/ >> >> >> >> > I think Christophe is a bit right here. While in general having > meta-meta objects makes sense, in this particular instance, I don't > see the functional need for it. Can you supply an example of where it > would be useful? One example is to supply hints about what to do with the attribute if the file is copied / moved. Several types of attributes have been mentioned: 1) Those that are essentially a cached object, which can be re-derived from the source file (e.g. thumbnail image) 2) Those that become invalid if the source file changes (e.g. hash value) 3) Those that should be maintained on file copy (e.g. author, copyright, etc) 4) Those that may not want to be copied with the file (e.g. security keys or the like) etc. This could be accomplished without meta-meta information (like permissions), but that wouldn't be very extensible, would it :) - Steve ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:20 ` Linus Torvalds 2004-08-26 17:27 ` Jamie Lokier 2004-08-26 17:57 ` Rik van Riel @ 2004-08-27 8:08 ` Hans Reiser 2 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-27 8:08 UTC (permalink / raw) To: Linus Torvalds Cc: Rik van Riel, Diego Calleja, jamie, christophe, vda, christer, spam, akpm, wichert, jra, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Linus Torvalds wrote: > > > >No no. The stream you get from a directory is totally _independent_ of the >contents of the directory. > but the user can link filename/metas/body to filename/metas/tar, if we implement a tar plugin, if it happens that the user wants it to be totally dependent. > Anything else would be pointless. > > Linus >- >To unsubscribe from this list: send the line "unsubscribe linux-kernel" in >the body of a message to majordomo@vger.kernel.org >More majordomo info at http://vger.kernel.org/majordomo-info.html >Please read the FAQ at http://www.tux.org/lkml/ > > > > ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:16 ` Rik van Riel 2004-08-26 17:20 ` Linus Torvalds @ 2004-08-26 17:40 ` Diego Calleja 2004-08-26 17:55 ` Christophe Saout 2004-08-26 17:59 ` Rik van Riel 1 sibling, 2 replies; 1392+ messages in thread From: Diego Calleja @ 2004-08-26 17:40 UTC (permalink / raw) To: Rik van Riel Cc: jamie, christophe, vda, christer, spam, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list El Thu, 26 Aug 2004 13:16:22 -0400 (EDT) Rik van Riel <riel@redhat.com> escribió: > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ? /bin could be separated (like linus said) but cat /bin/.compound could do it. This is the /etc/passwd Hans' example, I think: "If /new_syntax_access_path/big_directory_of_small_files/.glued is a plugin that aggregates every file in big_directory_of_small_files with a delimiter separating every file within the aggregation, then one can simply type emacs /new_syntax_access_path/big_directory_of_small_files/.glued, and the filesystem has done all the work emacs needs to be effective at this. Not a line of emacs needs to be changed." ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:40 ` Diego Calleja @ 2004-08-26 17:55 ` Christophe Saout 2004-08-26 17:59 ` Rik van Riel 1 sibling, 0 replies; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 17:55 UTC (permalink / raw) To: Diego Calleja Cc: Rik van Riel, jamie, vda, christer, spam, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 07:40:10PM +0200, Diego Calleja wrote: > > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ? > > /bin could be separated (like linus said) but cat /bin/.compound could do > it. This is the /etc/passwd Hans' example, I think: Yes, but what about locking? If we have compound files with individually accessible components a lock on the compound files could lock its components at the same time. But what with /usr/.compound? Is it read-only? What if you truncate it? Does it atomically delete the whole /usr directory tree? What about reading? It has to go through all these independent files, what if they change while you are reading the compound file? Does it lock everything? Point-in-time snapshot? If reiser4 can't do this. ;-) I don't say that it's impossible to do but I think it's a lot harder than doing it the other way round. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:40 ` Diego Calleja 2004-08-26 17:55 ` Christophe Saout @ 2004-08-26 17:59 ` Rik van Riel 2004-08-26 18:20 ` Jamie Lokier 1 sibling, 1 reply; 1392+ messages in thread From: Rik van Riel @ 2004-08-26 17:59 UTC (permalink / raw) To: Diego Calleja Cc: jamie, christophe, vda, christer, spam, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Diego Calleja wrote: > El Thu, 26 Aug 2004 13:16:22 -0400 (EDT) Rik van Riel <riel@redhat.com> escribió: > > > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ? > > /bin could be separated (like linus said) but cat /bin/.compound could > do it. This is the /etc/passwd Hans' example, I think: Arghhhh. I wrote it down to ridicule the idea and now people are taking it seriously ;( It should be obvious enough that anything depending on the kernel parsing file contents will lead to problems. -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:59 ` Rik van Riel @ 2004-08-26 18:20 ` Jamie Lokier 2004-08-27 8:06 ` Hans Reiser 0 siblings, 1 reply; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 18:20 UTC (permalink / raw) To: Rik van Riel Cc: Diego Calleja, christophe, vda, christer, spam, akpm, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Rik van Riel wrote: > > /bin could be separated (like linus said) but cat /bin/.compound could > > do it. This is the /etc/passwd Hans' example, I think: > > Arghhhh. I wrote it down to ridicule the idea and now people > are taking it seriously ;( > > It should be obvious enough that anything depending on the > kernel parsing file contents will lead to problems. This is one case where the kernel _isn't_ parsing file contents, but I agree it's ridiculous :) -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:20 ` Jamie Lokier @ 2004-08-27 8:06 ` Hans Reiser 0 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-27 8:06 UTC (permalink / raw) To: Jamie Lokier Cc: Rik van Riel, Diego Calleja, christophe, vda, christer, spam, akpm, wichert, jra, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Jamie Lokier wrote: >Rik van Riel wrote: > > >>>/bin could be separated (like linus said) but cat /bin/.compound could >>>do it. This is the /etc/passwd Hans' example, I think: >>> >>> >>Arghhhh. I wrote it down to ridicule the idea and now people >>are taking it seriously ;( >> >>It should be obvious enough that anything depending on the >>kernel parsing file contents will lead to problems. >> >> > >This is one case where the kernel _isn't_ parsing file contents, >but I agree it's ridiculous :) > >-- Jamie > > Umm, no, when you write the glued /etc/passwd file it parses the delimiters in it to determine what subfile to put the parts of /etc/passwd into. Inheritance with delimiters is the best way to make emacs effective for editing lots of small files, Stallman was right and deserves credit for it. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 14:12 ` Rik van Riel 2004-08-26 14:24 ` Christophe Saout @ 2004-08-26 15:19 ` Denis Vlasenko 2004-08-29 14:30 ` Alan Cox 1 sibling, 1 reply; 1392+ messages in thread From: Denis Vlasenko @ 2004-08-26 15:19 UTC (permalink / raw) To: Rik van Riel Cc: Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thursday 26 August 2004 17:12, Rik van Riel wrote: > On Thu, 26 Aug 2004, Denis Vlasenko wrote: > > I think Hans is not planning turning old "file is a stream of bytes" > > into eight-stream octopus. One stream will remain as a 'main' one, > > which contains actual data. Others will keep metadata, etc... > > This is exactly what the Samba people want, though. > > Office suites can store a document with embedded images > and spread sheets "easily" by putting the text, the > images and spread sheets all in different file streams. Kinda far reaching. It's hard to assess whether that is good or bad idea. Can we start small, like putting metadata (e.g. ACLs) into these streams? -- vda ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 15:19 ` Denis Vlasenko @ 2004-08-29 14:30 ` Alan Cox 2004-08-29 17:42 ` Jeremy Allison ` (2 more replies) 0 siblings, 3 replies; 1392+ messages in thread From: Alan Cox @ 2004-08-29 14:30 UTC (permalink / raw) To: Denis Vlasenko Cc: Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, Linux Kernel Mailing List, flx, reiserfs-list On Iau, 2004-08-26 at 16:19, Denis Vlasenko wrote: > > Office suites can store a document with embedded images > > and spread sheets "easily" by putting the text, the > > images and spread sheets all in different file streams. > > Kinda far reaching. It's hard to assess whether that is good or bad idea. > Can we start small, like putting metadata (e.g. ACLs) into these streams? Openoffice does this in user space and the user space vfs code desktops use can handle zips so this "just works" already including over NFS, unlike a kernel proposed method. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 14:30 ` Alan Cox @ 2004-08-29 17:42 ` Jeremy Allison 2004-08-29 18:41 ` Trond Myklebust 2004-09-01 20:25 ` Jamie Lokier 2 siblings, 0 replies; 1392+ messages in thread From: Jeremy Allison @ 2004-08-29 17:42 UTC (permalink / raw) To: Alan Cox Cc: Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, Linux Kernel Mailing List, flx, reiserfs-list On Sun, Aug 29, 2004 at 03:30:03PM +0100, Alan Cox wrote: > > Openoffice does this in user space and the user space vfs code desktops > use can handle zips so this "just works" already including over NFS, > unlike a kernel proposed method. Hurrah for OpenOffice. Now all you need to do is to persuade Microsoft to store Word files in the same format...... Jeremy. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 14:30 ` Alan Cox 2004-08-29 17:42 ` Jeremy Allison @ 2004-08-29 18:41 ` Trond Myklebust 2004-08-29 19:38 ` Jeremy Allison 2004-09-01 20:25 ` Jamie Lokier 2 siblings, 1 reply; 1392+ messages in thread From: Trond Myklebust @ 2004-08-29 18:41 UTC (permalink / raw) To: Alan Cox Cc: Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, jra, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list På su , 29/08/2004 klokka 10:30, skreiv Alan Cox: > Openoffice does this in user space and the user space vfs code desktops > use can handle zips so this "just works" already including over NFS, > unlike a kernel proposed method. NFS is not the roadblock here. Sure v2 and v3 support would be impossible to do, but since this is all 2.6 work anyway, people have the option of using NFSv4, which has full support for streams... NFSv4 has an OPENATTR call which acts on files to return a filehandle that works fine with both READDIR and LOOKUP, so if a VFS interface for streams existed, we could code up full support tomorrow. As it is, we're having to shoehorn this into the xattr interface. 8-( Cheers, Trond ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 18:41 ` Trond Myklebust @ 2004-08-29 19:38 ` Jeremy Allison 2004-09-01 20:19 ` Jamie Lokier 0 siblings, 1 reply; 1392+ messages in thread From: Jeremy Allison @ 2004-08-29 19:38 UTC (permalink / raw) To: Trond Myklebust Cc: Alan Cox, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, jra, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list On Sun, Aug 29, 2004 at 02:41:04PM -0400, Trond Myklebust wrote: > > NFSv4 has an OPENATTR call which acts on files to return a filehandle > that works fine with both READDIR and LOOKUP, so if a VFS interface for > streams existed, we could code up full support tomorrow. > As it is, we're having to shoehorn this into the xattr interface. 8-( Yeah, that's what I'm really trying to avoid for Samba. We're currently using the xattr interface for DOS attributes, and soon for NT ACL support, but it doesn't really fit as a streams interface (no lseek() support) although someone in HP coded up a test interface to NT streams that map into xattrs (most of the streams in a Word file for instance are quite small) - but I really don't want to have to do this :-). Jeremy. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 19:38 ` Jeremy Allison @ 2004-09-01 20:19 ` Jamie Lokier 2004-09-01 20:26 ` Jeremy Allison 0 siblings, 1 reply; 1392+ messages in thread From: Jamie Lokier @ 2004-09-01 20:19 UTC (permalink / raw) To: Jeremy Allison Cc: Trond Myklebust, Alan Cox, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list Jeremy Allison wrote: > (most of the streams in a Word file for instance are quite small) Streams in a Word file? Are you saying that when I copy a .doc file onto my Linux box and off, I lose part of a Word document? -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:19 ` Jamie Lokier @ 2004-09-01 20:26 ` Jeremy Allison 2004-09-01 20:31 ` Jamie Lokier 2004-09-02 9:48 ` Alan Cox 0 siblings, 2 replies; 1392+ messages in thread From: Jeremy Allison @ 2004-09-01 20:26 UTC (permalink / raw) To: Jamie Lokier Cc: Jeremy Allison, Trond Myklebust, Alan Cox, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list On Wed, Sep 01, 2004 at 09:19:45PM +0100, Jamie Lokier wrote: > Jeremy Allison wrote: > > (most of the streams in a Word file for instance are quite small) > > Streams in a Word file? Yep. > Are you saying that when I copy a .doc file onto my Linux box and off, > I lose part of a Word document? Right now no, because when Samba refuses the stream open, Word falls back into a "tar"-like mode where it linearises the streams into the data (it's a legacy mode for storing data on a FAT drive, not an NTFS drive). However, the problem is that no currently supported Microsoft OS doesn't have streams-capable NTFS support. This means that in a future MS-Office revision, this backwards support may be broken by accident or by design (less likely, Microsoft really don't do that kind of thing without very high level requests :-) and no testers at Microsoft will notice (because they only test against MS servers). Jeremy. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:26 ` Jeremy Allison @ 2004-09-01 20:31 ` Jamie Lokier 2004-09-01 20:35 ` Jeremy Allison 2004-09-01 21:09 ` David Lang 2004-09-02 9:48 ` Alan Cox 1 sibling, 2 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-09-01 20:31 UTC (permalink / raw) To: Jeremy Allison Cc: Trond Myklebust, Alan Cox, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list Jeremy Allison wrote: > > Streams in a Word file? > > Yep. > > > Are you saying that when I copy a .doc file onto my Linux box and off, > > I lose part of a Word document? > > Right now no, because when Samba refuses the stream open, Word falls > back into a "tar"-like mode where it linearises the streams into the > data (it's a legacy mode for storing data on a FAT drive, not an NTFS > drive). However, the problem is that no currently supported Microsoft > OS doesn't have streams-capable NTFS support. I meant when I copy not using Samba. For example, I copy the .doc file in Windows NT to an FTP server. Does the FTP operation magically linearise the .doc streams on demand? Or does FTP lose part of the Word document? -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:31 ` Jamie Lokier @ 2004-09-01 20:35 ` Jeremy Allison 2004-09-01 20:47 ` Jamie Lokier 2004-09-01 21:09 ` David Lang 1 sibling, 1 reply; 1392+ messages in thread From: Jeremy Allison @ 2004-09-01 20:35 UTC (permalink / raw) To: Jamie Lokier Cc: Jeremy Allison, Trond Myklebust, Alan Cox, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list On Wed, Sep 01, 2004 at 09:31:01PM +0100, Jamie Lokier wrote: > > I meant when I copy not using Samba. For example, I copy the .doc > file in Windows NT to an FTP server. > > Does the FTP operation magically linearise the .doc streams on demand? > > Or does FTP lose part of the Word document? Good question. It depends if the Microsoft ftp client is streams-aware, and understands the Microsoft OLE structured storage format and will do the linearisation on demand or not. I must confess I haven't tested this, as I don't ever run Windows other than on vmware sessions for Samba testing these days :-). Probably a non-Microsoft ftp client would lose part of the word doc. Jeremy. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:35 ` Jeremy Allison @ 2004-09-01 20:47 ` Jamie Lokier 2004-09-01 20:51 ` Jeremy Allison 0 siblings, 1 reply; 1392+ messages in thread From: Jamie Lokier @ 2004-09-01 20:47 UTC (permalink / raw) To: Jeremy Allison Cc: Trond Myklebust, Alan Cox, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list Jeremy Allison wrote: > > I meant when I copy not using Samba. For example, I copy the .doc > > file in Windows NT to an FTP server. > > > > Does the FTP operation magically linearise the .doc streams on demand? > > Or does FTP lose part of the Word document? > > Good question. It depends if the Microsoft ftp client is streams-aware, > and understands the Microsoft OLE structured storage format and will do > the linearisation on demand or not. I must confess I haven't tested this, > as I don't ever run Windows other than on vmware sessions for Samba testing > these days :-). > > Probably a non-Microsoft ftp client would lose part of the word doc. So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd will _all_ lose part of a Word document when they handle it on a Window box? Ouch! The only sensible implementation I can imagine would be if the OS linearised multi-stream Word documents into the non-stream format automatically for all programs which don't know about streams. Which is of course what I would like to implement for Linux... - Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:47 ` Jamie Lokier @ 2004-09-01 20:51 ` Jeremy Allison 2004-09-01 21:35 ` Lee Revell ` (2 more replies) 0 siblings, 3 replies; 1392+ messages in thread From: Jeremy Allison @ 2004-09-01 20:51 UTC (permalink / raw) To: Jamie Lokier Cc: Jeremy Allison, Trond Myklebust, Alan Cox, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list On Wed, Sep 01, 2004 at 09:47:46PM +0100, Jamie Lokier wrote: > Jeremy Allison wrote: > > > I meant when I copy not using Samba. For example, I copy the .doc > > > file in Windows NT to an FTP server. > > > > > > Does the FTP operation magically linearise the .doc streams on demand? > > > Or does FTP lose part of the Word document? > > > > Good question. It depends if the Microsoft ftp client is streams-aware, > > and understands the Microsoft OLE structured storage format and will do > > the linearisation on demand or not. I must confess I haven't tested this, > > as I don't ever run Windows other than on vmware sessions for Samba testing > > these days :-). > > > > Probably a non-Microsoft ftp client would lose part of the word doc. > > So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd > will _all_ lose part of a Word document when they handle it on a > Window box? > > Ouch! Yep. It's the meta data that Word stores in streams that will get lost. Jeremy. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:51 ` Jeremy Allison @ 2004-09-01 21:35 ` Lee Revell 2004-09-01 23:22 ` Oliver Hunt 2004-09-02 12:54 ` Theodore Ts'o 2 siblings, 0 replies; 1392+ messages in thread From: Lee Revell @ 2004-09-01 21:35 UTC (permalink / raw) To: Jeremy Allison Cc: Jamie Lokier, Trond Myklebust, Alan Cox, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list On Wed, 2004-09-01 at 16:51, Jeremy Allison wrote: > On Wed, Sep 01, 2004 at 09:47:46PM +0100, Jamie Lokier wrote: > > Jeremy Allison wrote: > > > > I meant when I copy not using Samba. For example, I copy the .doc > > > > file in Windows NT to an FTP server. > > > > > > > > Does the FTP operation magically linearise the .doc streams on demand? > > > > Or does FTP lose part of the Word document? > > > > > > Good question. It depends if the Microsoft ftp client is streams-aware, > > > and understands the Microsoft OLE structured storage format and will do > > > the linearisation on demand or not. I must confess I haven't tested this, > > > as I don't ever run Windows other than on vmware sessions for Samba testing > > > these days :-). > > > > > > Probably a non-Microsoft ftp client would lose part of the word doc. > > > > So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd > > will _all_ lose part of a Word document when they handle it on a > > Window box? > > > > Ouch! > > Yep. It's the meta data that Word stores in streams that will get lost. > This is shocking. When was this behavior introduced? Lee ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:51 ` Jeremy Allison 2004-09-01 21:35 ` Lee Revell @ 2004-09-01 23:22 ` Oliver Hunt 2004-09-07 11:48 ` Herbert Poetzl 2004-09-02 12:54 ` Theodore Ts'o 2 siblings, 1 reply; 1392+ messages in thread From: Oliver Hunt @ 2004-09-01 23:22 UTC (permalink / raw) To: Jeremy Allison; +Cc: Linux Kernel Mailing List, reiserfs-list The loss of forks in the file is exxactly the problem you used to have when transferring native Mac files to a PC... This meant in order to transfer files to different filesystem you often needed to tar/zip/whatever them first. Bare in mind this would let us do the whole MacOS thing of putting an entire application(plus plugins, etc) inside one "file"... --Oliver On Wed, 1 Sep 2004 13:51:40 -0700, Jeremy Allison <jra@samba.org> wrote: > On Wed, Sep 01, 2004 at 09:47:46PM +0100, Jamie Lokier wrote: > > Jeremy Allison wrote: > > > > I meant when I copy not using Samba. For example, I copy the .doc > > > > file in Windows NT to an FTP server. > > > > > > > > Does the FTP operation magically linearise the .doc streams on demand? > > > > Or does FTP lose part of the Word document? > > > > > > Good question. It depends if the Microsoft ftp client is streams-aware, > > > and understands the Microsoft OLE structured storage format and will do > > > the linearisation on demand or not. I must confess I haven't tested this, > > > as I don't ever run Windows other than on vmware sessions for Samba testing > > > these days :-). > > > > > > Probably a non-Microsoft ftp client would lose part of the word doc. > > > > So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd > > will _all_ lose part of a Word document when they handle it on a > > Window box? > > > > Ouch! > > Yep. It's the meta data that Word stores in streams that will get lost. > > > > Jeremy. > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 23:22 ` Oliver Hunt @ 2004-09-07 11:48 ` Herbert Poetzl 0 siblings, 0 replies; 1392+ messages in thread From: Herbert Poetzl @ 2004-09-07 11:48 UTC (permalink / raw) To: Oliver Hunt; +Cc: Jeremy Allison, Linux Kernel Mailing List, reiserfs-list On Thu, Sep 02, 2004 at 11:22:41AM +1200, Oliver Hunt wrote: > The loss of forks in the file is exxactly the problem you used to have > when transferring native Mac files to a PC... > > This meant in order to transfer files to different filesystem you > often needed to tar/zip/whatever them first. > > Bare in mind this would let us do the whole MacOS thing of putting an > entire application(plus plugins, etc) inside one "file"... hum, well aside from the fact that recent MacOS (X) does not put applications in one "file", it does simply store related stuff inside a directory which is presented as one unit by the GUI ... something I consider useful and which allows to 'move' the application around without rendering them useless ... best, Herbert > --Oliver > > On Wed, 1 Sep 2004 13:51:40 -0700, Jeremy Allison <jra@samba.org> wrote: > > On Wed, Sep 01, 2004 at 09:47:46PM +0100, Jamie Lokier wrote: > > > Jeremy Allison wrote: > > > > > I meant when I copy not using Samba. For example, I copy the .doc > > > > > file in Windows NT to an FTP server. > > > > > > > > > > Does the FTP operation magically linearise the .doc streams on demand? > > > > > Or does FTP lose part of the Word document? > > > > > > > > Good question. It depends if the Microsoft ftp client is streams-aware, > > > > and understands the Microsoft OLE structured storage format and will do > > > > the linearisation on demand or not. I must confess I haven't tested this, > > > > as I don't ever run Windows other than on vmware sessions for Samba testing > > > > these days :-). > > > > > > > > Probably a non-Microsoft ftp client would lose part of the word doc. > > > > > > So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd > > > will _all_ lose part of a Word document when they handle it on a > > > Window box? > > > > > > Ouch! > > > > Yep. It's the meta data that Word stores in streams that will get lost. > > > > > > > > Jeremy. > > - > > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > Please read the FAQ at http://www.tux.org/lkml/ > > > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:51 ` Jeremy Allison 2004-09-01 21:35 ` Lee Revell 2004-09-01 23:22 ` Oliver Hunt @ 2004-09-02 12:54 ` Theodore Ts'o 2004-09-02 14:45 ` Stuart Young 2 siblings, 1 reply; 1392+ messages in thread From: Theodore Ts'o @ 2004-09-02 12:54 UTC (permalink / raw) To: Jeremy Allison Cc: Jamie Lokier, Trond Myklebust, Alan Cox, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list On Wed, Sep 01, 2004 at 01:51:40PM -0700, Jeremy Allison wrote: > > So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd > > will _all_ lose part of a Word document when they handle it on a > > Window box? > > > > Ouch! > > Yep. It's the meta data that Word stores in streams that will get lost. And this is why I believe that using streams in application is well, ill-advised. Indeed, one of my concerns with providing streams support is that application authors may make the mistake of using it, and we will be back to the bad old days (when MacOS made this mistake) where you will need to binhex files before you ftp them (and unbinhex them on the otherside) --- and if you forget, the resulting file will be useless. I understand why the Samba folks want this feature very badly; however, hopefully other projects will know enough *not* to use streams once they become available in Linux.... - Ted ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 12:54 ` Theodore Ts'o @ 2004-09-02 14:45 ` Stuart Young 2004-09-03 7:53 ` Helge Hafting 0 siblings, 1 reply; 1392+ messages in thread From: Stuart Young @ 2004-09-02 14:45 UTC (permalink / raw) To: linux-kernel Cc: Theodore Ts'o, Jeremy Allison, Jamie Lokier, Trond Myklebust, Alan Cox, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, flx, reiserfs-list On Thu, 2 Sep 2004 22:54, Theodore Ts'o wrote: > On Wed, Sep 01, 2004 at 01:51:40PM -0700, Jeremy Allison wrote: > > > So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd > > > will _all_ lose part of a Word document when they handle it on a > > > Window box? > > > > > > Ouch! > > > > Yep. It's the meta data that Word stores in streams that will get lost. > > And this is why I believe that using streams in application is well, > ill-advised. Indeed, one of my concerns with providing streams > support is that application authors may make the mistake of using it, > and we will be back to the bad old days (when MacOS made this mistake) > where you will need to binhex files before you ftp them (and unbinhex > them on the otherside) --- and if you forget, the resulting file will > be useless. At least currently (to my knowledge anyway) all stream support in Windows is data that is not important, and that can be either regenerated from filesystem metadata or (more usually) the main file stream itself. This sort of data is really where streams excel, by providing a way to access data that would otherwise take time/cpu to regenerate over and over, but that in itself is not indispensable. Good examples of this are indexes of data within a document, details of who owns/created/modified the document, common views or reformatting of the data, etc. With audio/video/graphics, you could store lower quality transforms of data (eg: stereo to mono, resolution reduction, thumbnails, etc) in the streams for a file. With a word document, it could be things like an index (assuming it's auto-generated from section headings). With a database, it could be the indexes, and a few views that are expensive time-wise to generate. All of these are easily regenerated from the original data stream, but takes a while. And if you've got the disk, why not use it? If streams were always to be considered volatile, then you could do all sorts of interesting things with them. Any disk cleanup mechanism you have could also reap old streams specifically if the disk gets below a certain amount free. This means that old streams that are hanging about don't end up wasting all your disk space. Of course, you'd want a way to disable this (for servers mainly), and streams would have to be considered volatile on more than just Linux as a platform for this to be useful. Note that I'm not particularly advocating streams here. I'm just pointing out 'how' it could be useful. It could be very easy to misuse streams and cause huge problems (as per Ted's comments), but it's always good to know the other side of the argument. -- Stuart Young (aka Cef) cef-lkml@optusnet.com.au is for LKML and related email only ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 14:45 ` Stuart Young @ 2004-09-03 7:53 ` Helge Hafting 2004-09-03 10:41 ` Stuart Young 0 siblings, 1 reply; 1392+ messages in thread From: Helge Hafting @ 2004-09-03 7:53 UTC (permalink / raw) To: Stuart Young Cc: linux-kernel, Theodore Ts'o, Jeremy Allison, Jamie Lokier, Trond Myklebust, Alan Cox, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, flx, reiserfs-list Stuart Young wrote: >On Thu, 2 Sep 2004 22:54, Theodore Ts'o wrote: > > >>On Wed, Sep 01, 2004 at 01:51:40PM -0700, Jeremy Allison wrote: >> >> >>>>So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd >>>>will _all_ lose part of a Word document when they handle it on a >>>>Window box? >>>> >>>>Ouch! >>>> >>>> >>>Yep. It's the meta data that Word stores in streams that will get lost. >>> >>> >>And this is why I believe that using streams in application is well, >>ill-advised. Indeed, one of my concerns with providing streams >>support is that application authors may make the mistake of using it, >>and we will be back to the bad old days (when MacOS made this mistake) >>where you will need to binhex files before you ftp them (and unbinhex >>them on the otherside) --- and if you forget, the resulting file will >>be useless. >> >> This is not a problem with multiple streams implemented right - as a directory. You don't stay away from directories just because you have to tar them in order to put them on ftp sites? ;-) Still, you're right that apps using streams jsut for the hell of it is bad. That sort of thing happens all the time when something new shows up, some people will use it without thinking. > >At least currently (to my knowledge anyway) all stream support in Windows is >data that is not important, and that can be either regenerated from >filesystem metadata or (more usually) the main file stream itself. > >This sort of data is really where streams excel, by providing a way to access >data that would otherwise take time/cpu to regenerate over and over, but that >in itself is not indispensable. > Streams as a cache. >Good examples of this are indexes of data >within a document, details of who owns/created/modified the document, common >views or reformatting of the data, etc. With audio/video/graphics, you could >store lower quality transforms of data (eg: stereo to mono, resolution >reduction, thumbnails, etc) in the streams for a file. With a word document, >it could be things like an index (assuming it's auto-generated from section >headings). With a database, it could be the indexes, and a few views that are >expensive time-wise to generate. All of these are easily regenerated from the >original data stream, but takes a while. And if you've got the disk, why not >use it? > > Actually, some if this is bad examples of using streams. Why can't that index be stored _in_ the document? After all, the word processor is the one who knows the document's internal format, how to generate the index, and how to use it. Well, this example is bad anyway as an index can be created so fast from a well structured document that there is little need for storing one. (Example - see how lyx keeps a live index in the "navigate" menu. . .) A document format may also contain fields specifying who made it, or even a log of who modified it and when. It seems to me that streams are more useful for stuff that the file's main application don't deal with itself. Such as attaching icons that follow the file around. >If streams were always to be considered volatile, then you could do all sorts >of interesting things with them. Any disk cleanup mechanism you have could >also reap old streams specifically if the disk gets below a certain amount >free. > That would limit streams to caching use _only_, disks fill occationally and we can't have _useful_ stuff disappearing at random. Helge Hafting ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-03 7:53 ` Helge Hafting @ 2004-09-03 10:41 ` Stuart Young 2004-09-03 19:17 ` Horst von Brand 0 siblings, 1 reply; 1392+ messages in thread From: Stuart Young @ 2004-09-03 10:41 UTC (permalink / raw) To: linux-kernel Cc: Helge Hafting, Theodore Ts'o, Jeremy Allison, Jamie Lokier, Trond Myklebust, Alan Cox, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, flx, reiserfs-list On Fri, 3 Sep 2004 17:53, Helge Hafting wrote: > Stuart Young wrote: > >On Thu, 2 Sep 2004 22:54, Theodore Ts'o wrote: > >>On Wed, Sep 01, 2004 at 01:51:40PM -0700, Jeremy Allison wrote: > >>>>So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd > >>>>will _all_ lose part of a Word document when they handle it on a > >>>>Window box? > >>>> > >>>>Ouch! > >>> > >>>Yep. It's the meta data that Word stores in streams that will get lost. > >> > >>And this is why I believe that using streams in application is well, > >>ill-advised. Indeed, one of my concerns with providing streams > >>support is that application authors may make the mistake of using it, > >>and we will be back to the bad old days (when MacOS made this mistake) > >>where you will need to binhex files before you ftp them (and unbinhex > >>them on the otherside) --- and if you forget, the resulting file will > >>be useless. > > This is not a problem with multiple streams implemented right - as a > directory. This works if you want (possibly editable) views of a single document. It doesn't really work that well for other things. Even then, something has to handle all the data translations and decoding, and that means you need to store all the reference information somewhere so that the app knows what to pass the original file to, so that it can be decoded for the user as they've requested. > You don't stay away from directories just because you have to tar > them in order to put them on ftp sites? ;-) No, but I know a lot of people who don't understand that with a lot of ftp servers that if you append .tar.gz to the directory name, you can fetch the directory. Sure it's all user education, but you have to remember that users don't generally want to be educated. They just want to get on with doing whatever they are doing. > Still, you're right that apps using streams jsut for the hell of it is bad. > That sort of thing happens all the time when something new shows up, > some people will use it without thinking. Hence why I was suggesting the idea of disposable data in streams. As long as people KNOW it's disposable, but useful to keep around as it cuts down the time needed to do stuff, then apps will start to pick up transporting streams properly. Least then (hopefully) no real information will get lost that is important. Once transporting streams becomes commonplace, then perhaps streams can be used for more useful things. > >At least currently (to my knowledge anyway) all stream support in Windows > > is data that is not important, and that can be either regenerated from > > filesystem metadata or (more usually) the main file stream itself. > > > >This sort of data is really where streams excel, by providing a way to > > access data that would otherwise take time/cpu to regenerate over and > > over, but that in itself is not indispensable. > > Streams as a cache. Almost exactly. > >Good examples of this are indexes of data > >within a document, details of who owns/created/modified the document, > > common views or reformatting of the data, etc. With audio/video/graphics, > > you could store lower quality transforms of data (eg: stereo to mono, > > resolution reduction, thumbnails, etc) in the streams for a file. With a > > word document, it could be things like an index (assuming it's > > auto-generated from section headings). With a database, it could be the > > indexes, and a few views that are expensive time-wise to generate. All of > > these are easily regenerated from the original data stream, but takes a > > while. And if you've got the disk, why not use it? > > Actually, some if this is bad examples of using streams. > Why can't that index be stored _in_ the document? > After all, the word processor is the one who knows the document's > internal format, how to generate the index, and how to use it. Well, > this example is bad anyway as an index can be created so fast from > a well structured document that there is little need for storing one. > (Example - see how lyx keeps a live index in the "navigate" menu. . .) The point of such information in my examples is that a stream can store information in a particular format (ie: an index) that is common to one indexing app/library. Such an index can be used by ANY app that knows the index format to search the document. This is almost exactly what MS will do (if they haven't done it already) with the File Indexing Service. As it's ONE library, then any new user app that creates data can add index creation by adding one library. And any app that wants to search these indexes would need only to add one library, not every library for every format that it wants to search. It's essentially an n^2 vs 2n problem. > A document format may also contain fields specifying who made it, or > even a log of who modified it and when. Exactly, but then you need to: 1. Read in the Document, and in some formats that are either compressed or that do not have a fixed format, this may mean you end up reading most of or even an entire file to get this data. This is expensive on CPU, disk cache, disk access and transfer times with large files, particularly if it's a slow data bus to the storage device. 2. Parse out the information you want, which means that you need to understand the format. With an index stream, in a fixed format, you only need one library that can handle the format for you. Without streams, you use lots of libraries. By using lots of libraries, you end up using lots of memory to hold them, and disk access time/transfer speed to load them. > It seems to me that streams are more useful for stuff that the file's > main application don't deal with itself. Such as attaching icons that > follow the file around. Icons themselves wouldn't be that good for multi-user environments. Attaching an icon type designator that then allows the app that wants to display an icon for the file to pull the appropriate icon from somewhere else (and therefore can be matched to whatever theme or specific icon the user wants) is probably a better idea. > >If streams were always to be considered volatile, then you could do all > > sorts of interesting things with them. Any disk cleanup mechanism you > > have could also reap old streams specifically if the disk gets below a > > certain amount free. > > That would limit streams to caching use _only_, disks fill occationally > and we can't have _useful_ stuff disappearing at random. If they transfer a file with streams on Windows using standard (Posix based) tools at the moment then they lose this information anyway. And application developers aren't all suddenly going to drop whatever it is they're doing to implement stream support as soon as it appears. Things take time, and till the majority of applications support it, it's useless to assume that the data is not going to dissapear because of a user using a non-updated tool. By limiting it to caching, at least in the short term, at least no one is going to (hopefully!) get bitten by data loss. Later (and I'm guessing a LOT later), who knows, streams could actually become useful for more than just caching. -- Stuart Young (aka Cef) cef-lkml@optusnet.com.au is for LKML and related email only ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-03 10:41 ` Stuart Young @ 2004-09-03 19:17 ` Horst von Brand 2004-09-04 5:03 ` Stuart Young 0 siblings, 1 reply; 1392+ messages in thread From: Horst von Brand @ 2004-09-03 19:17 UTC (permalink / raw) To: Stuart Young Cc: linux-kernel, Helge Hafting, Theodore Ts'o, Jeremy Allison, Jamie Lokier, Trond Myklebust, Alan Cox, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, flx, reiserfs-list Stuart Young <cef-lkml@optusnet.com.au> said: [...] > Hence why I was suggesting the idea of disposable data in streams. As > long as people KNOW it's disposable, but useful to keep around as it cuts > down the time needed to do stuff, then apps will start to pick up > transporting streams properly. Least then (hopefully) no real information > will get lost that is important. Once transporting streams becomes > commonplace, then perhaps streams can be used for more useful things. How will you prevent people putting the "real" data under some random stream, "just because it is a prettier name"? (Yes, I've seen Windows users exporting everything because they found the folder + hand icon prettier...). Short answer: You can't. And if you did, then it would be (another) hell to go through when you start using streams for "useful data". [...] > The point of such information in my examples is that a stream can store > information in a particular format (ie: an index) that is common to one > indexing app/library. Great. Now you just need to convince everybody and Aunt Tillie to use that same format. > Such an index can be used by ANY app that knows the > index format to search the document. This is almost exactly what MS will > do (if they haven't done it already) with the File Indexing Service. As > it's ONE library, then any new user app that creates data can add index > creation by adding one library. And any app that wants to search these > indexes would need only to add one library, not every library for every > format that it wants to search. It's essentially an n^2 vs 2n problem. Doable if you can just go and force a format/stream layout/application suite on each and every user. Won't happen in Linux (and I'm happy for that). -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-03 19:17 ` Horst von Brand @ 2004-09-04 5:03 ` Stuart Young 0 siblings, 0 replies; 1392+ messages in thread From: Stuart Young @ 2004-09-04 5:03 UTC (permalink / raw) To: linux-kernel Cc: Horst von Brand, Helge Hafting, Theodore Ts'o, Jeremy Allison, Jamie Lokier, Trond Myklebust, Alan Cox, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, flx, reiserfs-list On Sat, 4 Sep 2004 05:17, Horst von Brand wrote: > Stuart Young <cef-lkml@optusnet.com.au> said: > > Hence why I was suggesting the idea of disposable data in streams. As > > long as people KNOW it's disposable, but useful to keep around as it cuts > > down the time needed to do stuff, then apps will start to pick up > > transporting streams properly. Least then (hopefully) no real information > > will get lost that is important. Once transporting streams becomes > > commonplace, then perhaps streams can be used for more useful things. > > How will you prevent people putting the "real" data under some random > stream, "just because it is a prettier name"? This is a function of exporting something complex like file streams through a namespace like 'file-as-directory'. As soon as you start putting complex things like file streams through an 'easy to use' interface, you will get people using it in ways you don't expect. Sure, it has to be useful, and relatively easy, but you don't need to make it insanely so. And if it's all considered volatile, the only ones that NEED to know about it are system admins, power users and programmers, all of which have a much higher chance of understanding a complex object like a file stream. > (Yes, I've seen Windows users > exporting everything because they found the folder + hand icon > prettier...). Short answer: You can't. And if you did, then it would be > (another) hell to go through when you start using streams for "useful > data". Interesting you should say that. Pretty much every stream in Windows at the moment is considered to be volatile, as the information could be lost in a transfer to a networked machine or filesystem that is not stream enabled. Once Windows NT and Win98/ME are 'out of the picture', and FAT* filesystems vanish, this sort of behaviour could possibly be expected to change. Notice that it's a timeframe over about 5 yrs that they've been (almost silently) doing this already. They'll have much the same problem. Since I seriously doubt that streams are going to pop up in Linux overnight, with all the app support that this entails, we'll have the benefit of seeing just how the behemoth handles it, and be able to learn from their mistakes. Of note on this, the ability to create/manipulate/read/etc streams should almost certainly require it's own set of ACL's beyond blindly assuming that the standard permissions/ACL's can cover it as is. Document creators should be able to create/manipulate a stream, but a searching service does not necessarily need to do more than read it. You could even have the ability to access or write to a stream other than the default covered by a capability. > > The point of such information in my examples is that a stream can store > > information in a particular format (ie: an index) that is common to one > > indexing app/library. > > Great. Now you just need to convince everybody and Aunt Tillie to use that > same format. Welcome to standardisation. And in fact it's not so much necessary that everyone uses the same index format, as long as the same index format is used on one machine. In fact, it's not even that necessary, as each index format could use it's own stream to store it's version of the information. Yes it's duplication, but who isn't already used to that? (KDE/Gnome, vi/emacs, ext3/XFS/reiser/etc). You choose the format for the job, and by personal preference. If the apps don't support the format you want, you just lose whatever abilities the format gains you, but at least you don't lose the raw data. > > Such an index can be used by ANY app that knows the > > index format to search the document. This is almost exactly what MS will > > do (if they haven't done it already) with the File Indexing Service. As > > it's ONE library, then any new user app that creates data can add index > > creation by adding one library. And any app that wants to search these > > indexes would need only to add one library, not every library for every > > format that it wants to search. It's essentially an n^2 vs 2n problem. > > Doable if you can just go and force a format/stream layout/application > suite on each and every user. Won't happen in Linux (and I'm happy for > that). As above. My personal view of streams (if we MUST have it under Linux, and I'm in no way at all convinced of that fact) is that it should in many ways be like journaling or hash trees in ext3. They're useful functions, but they are not 'necessary' to the actual storage of data. Sure, they make it easy (and in some cases significantly easier) to recover or access data, but they are features that do not ultimately stop you using the core of the filesystem, it's main objective, which is storing data. -- Stuart Young (aka Cef) cef-lkml@optusnet.com.au is for LKML and related email only ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:31 ` Jamie Lokier 2004-09-01 20:35 ` Jeremy Allison @ 2004-09-01 21:09 ` David Lang 1 sibling, 0 replies; 1392+ messages in thread From: David Lang @ 2004-09-01 21:09 UTC (permalink / raw) To: Jamie Lokier Cc: Jeremy Allison, Trond Myklebust, Alan Cox, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list what happens when you ue one of the many software packages around that lets windows access a NFS server. when you copy the file to a NFS accessed drive does it loose part of the data? David Lang On Wed, 1 Sep 2004, Jamie Lokier wrote: > Date: Wed, 1 Sep 2004 21:31:01 +0100 > From: Jamie Lokier <jamie@shareable.org> > To: Jeremy Allison <jra@samba.org> > Cc: Trond Myklebust <trond.myklebust@fys.uio.no>, > Alan Cox <alan@lxorguk.ukuu.org.uk>, > Denis Vlasenko <vda@port.imtp.ilyichevsk.odessa.ua>, > Rik van Riel <riel@redhat.com>, Christer Weinigel <christer@weinigel.se>, > Spam <spam@tnonline.net>, Andrew Morton <akpm@osdl.org>, > wichert@wiggy.net, Linus Torvalds <torvalds@osdl.org>, reiser@namesys.com, > hch@lst.de, Linux Filesystem Development <linux-fsdevel@vger.kernel.org>, > Linux Kernel Mailing List <linux-kernel@vger.kernel.org>, flx@namesys.com, > reiserfs-list@namesys.com > Subject: Re: silent semantic changes with reiser4 > > Jeremy Allison wrote: >>> Streams in a Word file? >> >> Yep. >> >>> Are you saying that when I copy a .doc file onto my Linux box and off, >>> I lose part of a Word document? >> >> Right now no, because when Samba refuses the stream open, Word falls >> back into a "tar"-like mode where it linearises the streams into the >> data (it's a legacy mode for storing data on a FAT drive, not an NTFS >> drive). However, the problem is that no currently supported Microsoft >> OS doesn't have streams-capable NTFS support. > > I meant when I copy not using Samba. For example, I copy the .doc > file in Windows NT to an FTP server. > > Does the FTP operation magically linearise the .doc streams on demand? > > Or does FTP lose part of the Word document? > > -- Jamie > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-01 20:26 ` Jeremy Allison 2004-09-01 20:31 ` Jamie Lokier @ 2004-09-02 9:48 ` Alan Cox 2004-09-02 11:13 ` Oliver Neukum 2004-09-04 1:10 ` Brad Boyer 1 sibling, 2 replies; 1392+ messages in thread From: Alan Cox @ 2004-09-02 9:48 UTC (permalink / raw) To: Jeremy Allison Cc: Jamie Lokier, Trond Myklebust, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list On Mer, 2004-09-01 at 21:26, Jeremy Allison wrote: > Right now no, because when Samba refuses the stream open, Word falls > back into a "tar"-like mode where it linearises the streams into the > data (it's a legacy mode for storing data on a FAT drive, not an NTFS > drive). However, the problem is that no currently supported Microsoft > OS doesn't have streams-capable NTFS support. That implies that samba can do the same transform set without kernel help and you'd even get the advantages of being able to transfer the stuff around sanely afterwards. What I don't understand is the tie between Linux having such streams and Windows doing it for Samba to work. Netatalk has always handle this for Macintosh and portably. Presumably any Samba support would need to handle OS's without wacky files for portability too ? ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 9:48 ` Alan Cox @ 2004-09-02 11:13 ` Oliver Neukum 2004-09-02 11:01 ` Alan Cox 2004-09-04 1:10 ` Brad Boyer 1 sibling, 1 reply; 1392+ messages in thread From: Oliver Neukum @ 2004-09-02 11:13 UTC (permalink / raw) To: Alan Cox Cc: Jeremy Allison, Jamie Lokier, Trond Myklebust, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list Am Donnerstag, 2. September 2004 11:48 schrieb Alan Cox: > What I don't understand is the tie between Linux having such streams and > Windows doing it for Samba to work. Netatalk has always handle this for > Macintosh and portably. Presumably any Samba support would need to > handle OS's without wacky files for portability too ? Can you do an atomic rename of all streams without kernel support? Regards Oliver ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 11:13 ` Oliver Neukum @ 2004-09-02 11:01 ` Alan Cox 0 siblings, 0 replies; 1392+ messages in thread From: Alan Cox @ 2004-09-02 11:01 UTC (permalink / raw) To: Oliver Neukum Cc: Jeremy Allison, Jamie Lokier, Trond Myklebust, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list On Iau, 2004-09-02 at 12:13, Oliver Neukum wrote: > Am Donnerstag, 2. September 2004 11:48 schrieb Alan Cox: > > What I don't understand is the tie between Linux having such streams and > > Windows doing it for Samba to work. Netatalk has always handle this for > > Macintosh and portably. Presumably any Samba support would need to > > handle OS's without wacky files for portability too ? > > Can you do an atomic rename of all streams without kernel support? That depends how SAMBA chooses to handle the problem internally and how it chooses to store the data. The netatalk people have atomicity from the view of clients but not from the unix fs internal view. Alan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-02 9:48 ` Alan Cox 2004-09-02 11:13 ` Oliver Neukum @ 2004-09-04 1:10 ` Brad Boyer 2004-09-04 6:03 ` David Masover 1 sibling, 1 reply; 1392+ messages in thread From: Brad Boyer @ 2004-09-04 1:10 UTC (permalink / raw) To: Alan Cox Cc: Jeremy Allison, Jamie Lokier, Trond Myklebust, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list On Thu, Sep 02, 2004 at 10:48:46AM +0100, Alan Cox wrote: > What I don't understand is the tie between Linux having such streams and > Windows doing it for Samba to work. Netatalk has always handle this for > Macintosh and portably. Presumably any Samba support would need to > handle OS's without wacky files for portability too ? I'm not 100% sure on the samba side, but I think there is a pretty significant difference. On the Mac, the problem of copying forks and metadata onto non-Mac systems was recognized early on. There are several standard formats for serialized versions of this data. If you take the files that netatalk writes and copy them directly to a Mac separately, there are tools that can convert them back to the original format with all the data intact. I've never seen such a thing for NTFS named streams. Brad Boyer flar@allandria.com ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-09-04 1:10 ` Brad Boyer @ 2004-09-04 6:03 ` David Masover 0 siblings, 0 replies; 1392+ messages in thread From: David Masover @ 2004-09-04 6:03 UTC (permalink / raw) To: Brad Boyer Cc: Alan Cox, Jeremy Allison, Jamie Lokier, Trond Myklebust, Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, Linus Torvalds, reiser, hch, Linux Filesystem Development, Linux Kernel Mailing List, flx, reiserfs-list -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Brad Boyer wrote: | On Thu, Sep 02, 2004 at 10:48:46AM +0100, Alan Cox wrote: | |>What I don't understand is the tie between Linux having such streams and |>Windows doing it for Samba to work. Netatalk has always handle this for |>Macintosh and portably. Presumably any Samba support would need to |>handle OS's without wacky files for portability too ? | | | I'm not 100% sure on the samba side, but I think there is a pretty | significant difference. On the Mac, the problem of copying forks and | metadata onto non-Mac systems was recognized early on. There are several | standard formats for serialized versions of this data. If you take the | files that netatalk writes and copy them directly to a Mac separately, | there are tools that can convert them back to the original format with | all the data intact. I've never seen such a thing for NTFS named streams. I had an idea for how to solve this. Search the archives for "serialization" or "serialize". Basically, it involves creating a better interface to a better "dump". You do cat foo/serialize > foo.img and send foo.img over the network. Or you just tell mutt/scp/whatever to grab foo/serialize instead of foo. Can't be worse than a mac resource fork, and it looks more user-friendly to me. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQTlauHgHNmZLgCUhAQK7Iw//SAi7I84OIz4xHh5S7i1xc+JugPvELyh2 dvrVkETvQlvRcABZ0GYlCQQPPzL+QcTVnruMRAUZGrGCPJdTkxfae41tX5KDS8/4 OpPWEDki+FSJrv+Mn9Pbm3I5Bxlhu0+nuNOIS0HSULo+/IBV4f/ldv8TUKSbXeXJ YjAXJJECpWTlYwF8sTvM/ALHpo+6xEtJq5gQxoUFnw4Pio8eycalS9m2cDs/N6rq PYcOjb7pWjGEk+9qimmwIcX5LnBXl8L9OhXqMQoR3Of+blniIrEOtg/0WLrwzu62 rW+rBxDAfoDxIZZvquf/gyJ6stO8QzGeQqoxxTUXhbI5PUtD+qGO+tWT6+/EljHF qNri65JwB8PetbqdjWmsTO3V+FVSRy3hu4/vTNpnmwnR8H+yilWWDc7Tnsjql+dh 9j4ycfT5xDAA9dC6APOSV5AEgI9z8FntQZzOCPe1lyLD7Qwjdak+3Nw2I3yNS+Pl Xa5ijSm3IKJ+JqKpZKRzCRyXRQdeAj2kGaWQP4Ui3D/RwBEHtn7shwdJ9Hku6KkN 9ZyiTPqY5XGiyLrJWqRc9MraepGhuxxtYcq65A+vdJq2fxA5+XRF8UGbYc9KW5lF 1fYSXA2DBNJjeNEvZw8LWI54OrIMSlhj3dWLX8WkkXog2gWO7ly0E6OiA0odZV6M i9sDeRClqlM= =CFb4 -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 14:30 ` Alan Cox 2004-08-29 17:42 ` Jeremy Allison 2004-08-29 18:41 ` Trond Myklebust @ 2004-09-01 20:25 ` Jamie Lokier 2 siblings, 0 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-09-01 20:25 UTC (permalink / raw) To: Alan Cox Cc: Denis Vlasenko, Rik van Riel, Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, Linux Kernel Mailing List, flx, reiserfs-list Alan Cox wrote: > On Iau, 2004-08-26 at 16:19, Denis Vlasenko wrote: > > > Office suites can store a document with embedded images > > > and spread sheets "easily" by putting the text, the > > > images and spread sheets all in different file streams. > > > > Kinda far reaching. It's hard to assess whether that is good or bad idea. > > Can we start small, like putting metadata (e.g. ACLs) into these streams? > > Openoffice does this in user space and the user space vfs code desktops > use can handle zips so this "just works" already including over NFS, > unlike a kernel proposed method. OpenOffice is very sensible, and I agree that a method of handling document parts that _requires_ kernel support would be foolish. However, please see my message "The argument for fs assistance in handling archives" for how kernel support can improve performance of programs like OpenOffice on large compound documents, while preserving the sensible behaviour and remaining portable. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 14:00 ` Denis Vlasenko 2004-08-26 14:12 ` Rik van Riel @ 2004-08-26 14:42 ` Felipe Alfaro Solana 1 sibling, 0 replies; 1392+ messages in thread From: Felipe Alfaro Solana @ 2004-08-26 14:42 UTC (permalink / raw) To: Denis Vlasenko Cc: Christer Weinigel, Spam, Andrew Morton, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thursday 26 August 2004 16:00, Denis Vlasenko wrote: > > I think Hans is not planning turning old "file is a stream of bytes" > into eight-stream octopus. One stream will remain as a 'main' one, > which contains actual data. Others will keep metadata, etc... > > Note that even today's files aren't simple "streams of bytes" > because they also have names,mode bits etc. This is just exactly how NTFS does work: there's an unnamed stream (which is the one that the majority of userspace programs do manipulate) which contains the user data (the pixels from a PMG image, or text from a text file, etc.), and there are named streams which not all kind of applications do know how to handle. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 9:49 ` Andrew Morton 2004-08-26 10:20 ` Spam @ 2004-08-26 13:36 ` Rik van Riel 2004-08-26 13:59 ` Jamie Lokier 2004-08-26 23:54 ` Hans Reiser 2004-08-26 17:32 ` Jeremy Allison 2 siblings, 2 replies; 1392+ messages in thread From: Rik van Riel @ 2004-08-26 13:36 UTC (permalink / raw) To: Andrew Morton Cc: Spam, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Andrew Morton wrote: > All of which can be handled in userspace library code. > > What compelling reason is there for doing this in the kernel? There's a compelling reason to do it in userspace. If an unaware program copies or moves such a file with streams inside, it doesn't break the streams and aware programs will continue to see them. OTOH, if we had the streams in the kernel, unaware applications would continuously break the metadata and streams that the streams aware programs expect ! -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 13:36 ` Rik van Riel @ 2004-08-26 13:59 ` Jamie Lokier 2004-08-26 14:07 ` Rik van Riel 2004-08-26 23:54 ` Hans Reiser 1 sibling, 1 reply; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 13:59 UTC (permalink / raw) To: Rik van Riel Cc: Andrew Morton, Spam, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Rik van Riel wrote: > > All of which can be handled in userspace library code. > > > > What compelling reason is there for doing this in the kernel? > > There's a compelling reason to do it in userspace. If an > unaware program copies or moves such a file with streams > inside, it doesn't break the streams and aware programs will > continue to see them. > > OTOH, if we had the streams in the kernel, unaware applications > would continuously break the metadata and streams that the > streams aware programs expect ! You appear not to have read any of my mails on this topic. Properly implemented metadata can: (1) operate in both modes simultaneously; (2) work with unaware applications; (3) provide performance enhancements to aware applications; (4) provide storage enhancements to both; (5) provide useful features that work with standard unmodified unix tools, all at once. That includes program copies. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 13:59 ` Jamie Lokier @ 2004-08-26 14:07 ` Rik van Riel 0 siblings, 0 replies; 1392+ messages in thread From: Rik van Riel @ 2004-08-26 14:07 UTC (permalink / raw) To: Jamie Lokier Cc: Andrew Morton, Spam, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Jamie Lokier wrote: > Properly implemented metadata can: [snip world peace] > all at once. That includes program copies. Show me the design for such a properly implemented thing. None of the designs I've seen proposed so far are able to do what you claim. -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 13:36 ` Rik van Riel 2004-08-26 13:59 ` Jamie Lokier @ 2004-08-26 23:54 ` Hans Reiser 1 sibling, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 23:54 UTC (permalink / raw) To: Rik van Riel Cc: Andrew Morton, Spam, wichert, jra, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Rik van Riel wrote: >On Thu, 26 Aug 2004, Andrew Morton wrote: > > > >>All of which can be handled in userspace library code. >> >>What compelling reason is there for doing this in the kernel? >> >> > >There's a compelling reason to do it in userspace. If an >unaware program copies or moves such a file with streams >inside, it doesn't break the streams and aware programs will >continue to see them. > >OTOH, if we had the streams in the kernel, unaware applications >would continuously break the metadata and streams that the >streams aware programs expect ! > > > Well, first off, you don't want streams in the kernel, you want all the little pieces that can be composed together into a stream if you so choose. Streams are ugly, the pieces are all cool. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 9:49 ` Andrew Morton 2004-08-26 10:20 ` Spam 2004-08-26 13:36 ` Rik van Riel @ 2004-08-26 17:32 ` Jeremy Allison 2004-08-26 18:17 ` Rik van Riel 2004-08-27 8:13 ` Hans Reiser 2 siblings, 2 replies; 1392+ messages in thread From: Jeremy Allison @ 2004-08-26 17:32 UTC (permalink / raw) To: Andrew Morton Cc: Spam, wichert, jra, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, Aug 26, 2004 at 02:49:56AM -0700, Andrew Morton wrote: > Spam <spam@tnonline.net> wrote: > > > > Yes, for example documents, image files etc. The multiple data > > streams can contain thumbnails, info about who is editing the file > > (useful for networked files) etc. Could be used for version handling > > and much more. > > All of which can be handled in userspace library code. > > What compelling reason is there for doing this in the kernel? Because without kernel support there is no way someone can publish a new metadata type and have it automatically supported by all application data files (ie. most apps ignore it, and only apps that are aware of it can see it). Without kernel support you have to have all apps agree on a data format. And that's harder to do than getting linux kernel VFS engineers to agree on things :-). Jeremy. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:32 ` Jeremy Allison @ 2004-08-26 18:17 ` Rik van Riel 2004-08-26 22:03 ` Felipe Alfaro Solana [not found] ` <20040827052804.GD2736@localhost.localdomain> 2004-08-27 8:13 ` Hans Reiser 1 sibling, 2 replies; 1392+ messages in thread From: Rik van Riel @ 2004-08-26 18:17 UTC (permalink / raw) To: Jeremy Allison Cc: Andrew Morton, Spam, wichert, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thu, 26 Aug 2004, Jeremy Allison wrote: > Because without kernel support there is no way someone can > publish a new metadata type and have it automatically supported > by all application data files (ie. most apps ignore it, and only > apps that are aware of it can see it). So your backup software ignores it, and after a restore you've lost your new metadata ? -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:17 ` Rik van Riel @ 2004-08-26 22:03 ` Felipe Alfaro Solana [not found] ` <20040827052804.GD2736@localhost.localdomain> 1 sibling, 0 replies; 1392+ messages in thread From: Felipe Alfaro Solana @ 2004-08-26 22:03 UTC (permalink / raw) To: Rik van Riel Cc: Jeremy Allison, Andrew Morton, Spam, wichert, torvalds, reiser, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Thursday 26 August 2004 20:17, Rik van Riel wrote: > On Thu, 26 Aug 2004, Jeremy Allison wrote: > > Because without kernel support there is no way someone can > > publish a new metadata type and have it automatically supported > > by all application data files (ie. most apps ignore it, and only > > apps that are aware of it can see it). > > So your backup software ignores it, and after a restore you've > lost your new metadata ? I think there's no easy way to fix this, except fixing the backup software to support xattrs's/streams/forks/whatever. Even on Windows, it has happened this way with older apps that don't support named streams. ^ permalink raw reply [flat|nested] 1392+ messages in thread
[parent not found: <20040827052804.GD2736@localhost.localdomain>]
* Re: silent semantic changes with reiser4 [not found] ` <20040827052804.GD2736@localhost.localdomain> @ 2004-08-27 8:05 ` Hans Reiser 0 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-27 8:05 UTC (permalink / raw) To: Jeremy Allison Cc: Rik van Riel, Andrew Morton, Spam, wichert, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Well said. Jeremy Allison wrote: > Yes, but then you fix your backup software and everyone lives > >happily ever after :-)...... > >It's a better solution than changing *every single application >in existance* (the "userspace library" approach). > >IMHO of course. > >Jeremy. > > > > ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 17:32 ` Jeremy Allison 2004-08-26 18:17 ` Rik van Riel @ 2004-08-27 8:13 ` Hans Reiser 1 sibling, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-27 8:13 UTC (permalink / raw) To: Jeremy Allison Cc: Andrew Morton, Spam, wichert, torvalds, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list I think Jeremy is accurate in saying the below. Jeremy Allison wrote: >>What compelling reason is there for doing this in the kernel? >> >> > > >Because without kernel support there is no way someone can >publish a new metadata type and have it automatically supported >by all application data files (ie. most apps ignore it, and only >apps that are aware of it can see it). Without kernel support >you have to have all apps agree on a data format. And >that's harder to do than getting linux kernel VFS engineers >to agree on things :-). > >Jeremy. > > > > ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 23:32 ` Andrew Morton 2004-08-25 23:37 ` Jeremy Allison @ 2004-08-26 9:03 ` Anton Altaparmakov 2004-08-26 10:45 ` Christian Mayrhuber 1 sibling, 1 reply; 1392+ messages in thread From: Anton Altaparmakov @ 2004-08-26 9:03 UTC (permalink / raw) To: Andrew Morton Cc: Spam, Linus Torvalds, reiser, hch, linux-fsdevel, lkml, flx, reiserfs-list On Thu, 2004-08-26 at 00:32, Andrew Morton wrote: > Spam <spam@tnonline.net> wrote: > > > > > In other words, if reiserfs does something special, we should make > > > standard interfaces for doing that special thing, so that everybody can > > > do it without stepping on other peoples toes. > > > > Agreed that would be the best. But how much time and effort will it > > be > > Zero. > > We can add these new features tomorrow, as reiser4-only features, with a > plan in hand to generalise them later. > > -->>__if__<<-- we think these are features which Linux should offer. Please don't forget that if the reiser4 features are merged as they are now, then we will likely be stuck with the API reiser4 chooses. There will be tools that will rely on it springing up no doubt. Moving the reiser4 features to VFS later is fine and good, but what if the VFS doesn't want the same API for those features? Either we would have to allow reiser4 to continue providing the old API even though the VFS now provides a new, shiny API or we would have to break all existing API users on reiser4. Things like "I rebooted into the latest kernel and my computer failed to boot because essential app FOO failed to access the reiser4 API - Help!" spring to mind. Yes, I know I am painting a rather black picture here and I know you might well say "screw apps", its been done plenty of times in Linux kernel development before... Best regards, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net WWW: http://linux-ntfs.sf.net/, http://www-stu.christs.cam.ac.uk/~aia21/ ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 9:03 ` Anton Altaparmakov @ 2004-08-26 10:45 ` Christian Mayrhuber 2004-08-26 10:52 ` Christoph Hellwig 0 siblings, 1 reply; 1392+ messages in thread From: Christian Mayrhuber @ 2004-08-26 10:45 UTC (permalink / raw) To: reiserfs-list; +Cc: Anton Altaparmakov, linux-fsdevel, lkml On Thursday 26 August 2004 11:03, Anton Altaparmakov wrote: > Please don't forget that if the reiser4 features are merged as they are > now, then we will likely be stuck with the API reiser4 chooses. There > will be tools that will rely on it springing up no doubt. > > Moving the reiser4 features to VFS later is fine and good, but what if > the VFS doesn't want the same API for those features? Either we would > have to allow reiser4 to continue providing the old API even though the > VFS now provides a new, shiny API or we would have to break all existing > API users on reiser4. Things like "I rebooted into the latest kernel > and my computer failed to boot because essential app FOO failed to > access the reiser4 API - Help!" spring to mind. Andrew Morton wrote: >b) accept the reiser4-only extensions with a view to turning them into > kernel-wide extensions at some time in the future, so all filesystems > will offer the extensions (as much as poss) or If option b) is chosen Reiser4 can become a playground. There is the reiser4() syscall which you surely don't want to implement for other filesystems. Once there is some experience with this new fancy stuff the dust what is useful/insecure, etc. and what is not will settle and can be condensed into a vfs api. Apps like samba and user scripts will have to be adapted once this is the case, but this should not be to big a problem if this stuff is marked experimental. People which want something stable can continue to use xattrs and a magnitude of filesystems for now. -- lg, Chris ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:45 ` Christian Mayrhuber @ 2004-08-26 10:52 ` Christoph Hellwig 2004-08-26 12:43 ` Olivier Galibert 0 siblings, 1 reply; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-26 10:52 UTC (permalink / raw) To: Christian Mayrhuber Cc: reiserfs-list, Anton Altaparmakov, linux-fsdevel, lkml On Thu, Aug 26, 2004 at 12:45:47PM +0200, Christian Mayrhuber wrote: > There is the reiser4() syscall which you surely don't want to implement for > other filesystems. Actually a syscall that goes into filesystem code is the last thing we want. But it's not in the submission, so let's keep that flame^H^H^H^Hdiscussion for later. > Once there is some experience with this new fancy stuff the dust what > is useful/insecure, etc. and what is not will settle and can be condensed > into a vfs api. > Apps like samba and user scripts will have to be adapted once this is > the case, but this should not be to big a problem if this stuff is marked > experimental. > > People which want something stable can continue to use xattrs and a > magnitude of filesystems for now. Sure, no one stops you from playing around with new semantics. But please don't add them to the linux kernel stable series until we have semantics we a) want to stick to for a while and b) actually work. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 10:52 ` Christoph Hellwig @ 2004-08-26 12:43 ` Olivier Galibert 2004-08-26 12:44 ` Christoph Hellwig 2004-08-26 15:15 ` Adrian Bunk 0 siblings, 2 replies; 1392+ messages in thread From: Olivier Galibert @ 2004-08-26 12:43 UTC (permalink / raw) To: Christoph Hellwig, Christian Mayrhuber, reiserfs-list, Anton Altaparmakov, linux-fsdevel, lkml On Thu, Aug 26, 2004 at 11:52:29AM +0100, Christoph Hellwig wrote: > Sure, no one stops you from playing around with new semantics. But please > don't add them to the linux kernel stable series until we have semantics we > a) want to stick to for a while and b) actually work. He's not proposing to add it to 2.4, is he? OG. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 12:43 ` Olivier Galibert @ 2004-08-26 12:44 ` Christoph Hellwig 2004-08-26 12:49 ` Markus Törnqvist 2004-08-26 15:15 ` Adrian Bunk 1 sibling, 1 reply; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-26 12:44 UTC (permalink / raw) To: Olivier Galibert, Christoph Hellwig, Christian Mayrhuber, reiserfs-list, Anton Altaparmakov, linux-fsdevel, lkml On Thu, Aug 26, 2004 at 02:43:34PM +0200, Olivier Galibert wrote: > On Thu, Aug 26, 2004 at 11:52:29AM +0100, Christoph Hellwig wrote: > > Sure, no one stops you from playing around with new semantics. But please > > don't add them to the linux kernel stable series until we have semantics we > > a) want to stick to for a while and b) actually work. > > He's not proposing to add it to 2.4, is he? We're talking about 2.6 here. 2.4 is the obsolete kernel series these days. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 12:44 ` Christoph Hellwig @ 2004-08-26 12:49 ` Markus Törnqvist 2004-08-26 12:52 ` Christoph Hellwig 0 siblings, 1 reply; 1392+ messages in thread From: Markus Törnqvist @ 2004-08-26 12:49 UTC (permalink / raw) To: Christoph Hellwig, Olivier Galibert, Christian Mayrhuber, reiserfs-list, Anton Altaparmakov, linux-fsdevel, lkml On Thu, Aug 26, 2004 at 01:44:15PM +0100, Christoph Hellwig wrote: >On Thu, Aug 26, 2004 at 02:43:34PM +0200, Olivier Galibert wrote: >> He's not proposing to add it to 2.4, is he? >We're talking about 2.6 here. 2.4 is the obsolete kernel series these days. So run it in -mm or fork it to 2.7. 2.7 might not be a bad idea for trying such bleeding edge revolution stuff as this. -- mjt ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 12:49 ` Markus Törnqvist @ 2004-08-26 12:52 ` Christoph Hellwig 0 siblings, 0 replies; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-26 12:52 UTC (permalink / raw) To: Markus Törnqvist Cc: Olivier Galibert, Christian Mayrhuber, reiserfs-list, Anton Altaparmakov, linux-fsdevel, lkml On Thu, Aug 26, 2004 at 03:49:29PM +0300, Markus Törnqvist wrote: > 2.7 might not be a bad idea for trying such bleeding edge revolution > stuff as this. I already suggested to Linus that 2.7 is probably the right place to experiment with that. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 12:43 ` Olivier Galibert 2004-08-26 12:44 ` Christoph Hellwig @ 2004-08-26 15:15 ` Adrian Bunk 1 sibling, 0 replies; 1392+ messages in thread From: Adrian Bunk @ 2004-08-26 15:15 UTC (permalink / raw) To: Olivier Galibert, Christoph Hellwig, Christian Mayrhuber, reiserfs-list, Anton Altaparmakov, linux-fsdevel, lkml On Thu, Aug 26, 2004 at 02:43:34PM +0200, Olivier Galibert wrote: > On Thu, Aug 26, 2004 at 11:52:29AM +0100, Christoph Hellwig wrote: > > Sure, no one stops you from playing around with new semantics. But please > > don't add them to the linux kernel stable series until we have semantics we > > a) want to stick to for a while and b) actually work. > > He's not proposing to add it to 2.4, is he? 2.6 is a stable series... > OG. cu Adrian -- "Is there not promise of rain?" Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. "Only a promise," Lao Er said. Pearl S. Buck - Dragon Seed ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 22:51 ` Spam 2004-08-25 22:51 ` Christoph Hellwig 2004-08-25 22:59 ` Linus Torvalds @ 2004-08-26 8:44 ` Hans Reiser 2 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 8:44 UTC (permalink / raw) To: Spam Cc: Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Spam wrote: > >> The fact that one filesystem will >>offer features which other filesystems do not and cannot offer makes me >>queasy for some reason. >> >> > > This last sentence makes me wonder. Where is Linux heading? The idea > that a FS cannot contain features that no other FS has is very > scary. > > I am all for uniformity, but not at the expense of shutting down > advanced progress that Linux is so badly needing. > > This talk about old UNIX seems like people want to still live in the > 70'ies and not look forward. Please wake up! > > ~S > > > > > > well said. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-25 22:28 ` Andrew Morton 2004-08-25 22:51 ` Spam @ 2004-08-26 8:31 ` Hans Reiser 2004-08-26 8:45 ` Andrew Morton ` (5 more replies) 1 sibling, 6 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 8:31 UTC (permalink / raw) To: Andrew Morton Cc: hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Andrew Morton wrote: >Hans Reiser <reiser@namesys.com> wrote: > > >>I had not intended to respond to this because I have nothing positive to >>say, but Andrew said I needed to respond and suggested I should copy >>Linus. >> >> > >Yes, but I didn't say "flame Christoph and ignore the issues" ;) > > Oh....;-) >There are lots of little things to do with implementation, coding style, >module exports, deadlocks, what code goes where, etc. These are all normal >daily kernel business and we should set them aside for now and concentrate >on the bigger issues. > > Yes, you are right, but I am not sure Viro will go along with that..... ;-) >And as I see it, there are two big issues: > >a) reiser4 extends the Linux API in ways which POSIX/Unix/etc do not > anticipate and > >b) it does this within the context of just a single filesystem. > >I see three possible responses: > >a) accept the reiser4-only extensions as-is (possibly with post-review > modifications, of course) or > >b) accept the reiser4-only extensions with a view to turning them into > kernel-wide extensions at some time in the future, so all filesystems > will offer the extensions (as much as poss) or > >c) reject the extensions. > > >My own order of preference is b) c) a). > I don't object to b), though I think b) should wait for 2.7 and reiser4 should not. > The fact that one filesystem will >offer features which other filesystems do not and cannot offer makes me >queasy for some reason. > > Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem for Apple, and that means we need to put search engine and database functionality into the filesystem. It takes 11 years of serious research to build a clean storage layer able to handle doing that. Reiser4 has done that, finally. None of the other Linux filesystems have. The next major release of ReiserFS is going to be bursting with semantic enhancements, because the prerequisites for them are in place now. None of the other Linux filesystems have those prerequisites. They won't be able to keep up with the semantic enhancements. This metafiles and file-directories stuff is actually fairly trivial stuff. Look guys, in 1993 I anticipated the battle would be here, and I build the foundation for a defensive tower right at the spot MS and Apple are now maneuvering towards. Help me get the next level on the tower before they get here. It is one hell of a foundation, they won't be able to shake it, their trees are not as powerful. Don't move reiser4 into vfs, use reiser4 as the vfs. Don't write filesystems, write file plugins and disk format plugins and all the other kinds of plugins, and you won't be missing any expressive power that you really want.... Give Saveliev and I some credit. 10 years of hard work at an ivory tower nobody thought mattered. Now the battle leaves the browser and swings our way. Don't duplicate that infrastructure, use it. There is so much we could use help with if talented people like you chose to contribute. >b) means that at some time in the future we need to hoist the reiser4 >extensions (at a conceptual level) (and probably with restrictions) up into >the VFS. This will involve much thought, argument and work. > > >To get us started on this route it would really help me (and, probably, >others) if you could describe what these API extensions are in a very >simple way, without referring to incomprehehsible web pages, > what is not comprehensible....? > and without >using terms which non-reiser4 people don't understand. > > Well, I agree that there is value in defining things in more detail than we have. >It would be best if each extension was addressed in a separate email >thread. > >We also need to discuss what a reiser4 "module" is, what its capabilities >are, and what licensing implications they have. > >Then, we can look at each one and say "yup, that makes sense - we want >Linux to do that" and we can also think about how we would implement it at >the VFS level. > >If we follow the above route I believe we can make progress in a technical >direction and not get deadlocked on personal/political stuff. > > >Now, an alternative to all the above is to just merge reiser4 as-is, after >addressing all the lower-level coding issues. And see what happens. That >may be a thing which Linus wishes to do. I'm easy. > > > > ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:31 ` Hans Reiser @ 2004-08-26 8:45 ` Andrew Morton 2004-08-26 9:24 ` Hans Reiser 2004-08-26 12:18 ` Christophe Saout 2004-08-26 9:34 ` Christoph Hellwig ` (4 subsequent siblings) 5 siblings, 2 replies; 1392+ messages in thread From: Andrew Morton @ 2004-08-26 8:45 UTC (permalink / raw) To: Hans Reiser Cc: hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Hans Reiser <reiser@namesys.com> wrote: > > >To get us started on this route it would really help me (and, probably, > >others) if you could describe what these API extensions are in a very > >simple way, without referring to incomprehehsible web pages, > > > what is not comprehensible....? Pretty much anything at www.namesys.com. The amount of time which is needed to extract the technical info which one is looking for vastly exceeds a gnat-like attention span. As a starting point, please prepare a bullet-point list of userspace-visible changes which the filesystem introduces, or is planned to introduce. And describe the "plugin" system. Why does the filesystem need such a thing (other filesystems get their features via `patch -p1')? And what are the licensing implications of plugins? Are they derived works? Must they be GPL'ed? ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:45 ` Andrew Morton @ 2004-08-26 9:24 ` Hans Reiser 2004-08-26 19:26 ` Lee Revell 2004-08-26 12:18 ` Christophe Saout 1 sibling, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 9:24 UTC (permalink / raw) To: Andrew Morton Cc: hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Andrew Morton wrote: >Hans Reiser <reiser@namesys.com> wrote: > > >> >To get us started on this route it would really help me (and, probably, >> >others) if you could describe what these API extensions are in a very >> >simple way, without referring to incomprehehsible web pages, >> > >> what is not comprehensible....? >> >> > >Pretty much anything at www.namesys.com. The amount of time which is >needed to extract the technical info which one is looking for vastly >exceeds a gnat-like attention span. > >As a starting point, please prepare a bullet-point list of >userspace-visible changes which the filesystem introduces, or is planned to >introduce. > >And describe the "plugin" system. Why does the filesystem need such a >thing (other filesystems get their features via `patch -p1')? > > It takes 6 months or more to become competent to change a usual filesystem. Creating a new reiser4 plugin is a weekend programmer fun hack to do. Weekend programmers matter, because they tend to have clever ideas based on understanding a need they have. How many people can easily add new features to ext3 or reiserfs V3? Very few. What happens if you need a disk format change? Well, in V4, you can easily compose a plugin from plugin methods of other plugins, write a little piece of code with the one thing you want different, and add it in. Disk format changes, no big deal, add a new disk format plugin, or a new item plugin, or a new node plugin, etc., and you got your new format. There is a huge difference between code that is designed for reuse, and code that is not. That is the difference between V3 and V4. We were looking at our V3 balancing code, and it special cased each different kind of item (an item is a piece of something, which the balancing code chops objects into as it squeezes them into nodes). It looked like the complexity of the balancing code was going to be N squared, where N was the number of different kinds of items. So, we created item handlers, and wrote balancing code that could slice and dice and merge any item that implemented all of the operations required of an item handler. From there it grew, and we made everything pluggable. Adding features to the new code is far less than the time cost of adding features to the old code. I've seen Nikita complain that something would take 6 weeks to do (changing key assignment algorithms), and then it takes him 3 afternoons, and it was because of the plugins, because when we did something similar in V3 it took 3 man months. >And what are the licensing implications of plugins? Are they derived >works? > Yes. Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 9:24 ` Hans Reiser @ 2004-08-26 19:26 ` Lee Revell 2004-08-26 19:34 ` Christoph Hellwig ` (2 more replies) 0 siblings, 3 replies; 1392+ messages in thread From: Lee Revell @ 2004-08-26 19:26 UTC (permalink / raw) To: Hans Reiser Cc: Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, 2004-08-26 at 05:24, Hans Reiser wrote: > Andrew Morton wrote: > > > >And describe the "plugin" system. Why does the filesystem need such a > >thing (other filesystems get their features via `patch -p1')? > > > > > It takes 6 months or more to become competent to change a usual > filesystem. Creating a new reiser4 plugin is a weekend programmer fun > hack to do. Weekend programmers matter, because they tend to have > clever ideas based on understanding a need they have. How many people > can easily add new features to ext3 or reiserfs V3? Very few. > > What happens if you need a disk format change? > > Well, in V4, you can easily compose a plugin from plugin methods of > other plugins, write a little piece of code with the one thing you want > different, and add it in. Disk format changes, no big deal, add a new > disk format plugin, or a new item plugin, or a new node plugin, etc., > and you got your new format. > OK, real world example. My roommate has an AKAI MPC-2000, a very popular hardware sampler from the 90's. The disk format is known,there are a few utilities to edit the disks on a PC and extract the PCM samples, but there are no tools to mount it on a modern PC. Are you saying that, since I know the MPC disk format, I could write a reiser4 plugin to mount an MPC drive? If so, then Hans has an excellent point. Users do want this kind of thing, and it is worth having to fix tar et al. Lee ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 19:26 ` Lee Revell @ 2004-08-26 19:34 ` Christoph Hellwig 2004-08-26 19:45 ` Lee Revell 2004-08-26 20:30 ` Dmitry Baryshkov [not found] ` <1093548815.13881.10.camel@leto.cs.pocnet.net> 2 siblings, 1 reply; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-26 19:34 UTC (permalink / raw) To: Lee Revell Cc: Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, Aug 26, 2004 at 03:26:55PM -0400, Lee Revell wrote: > OK, real world example. My roommate has an AKAI MPC-2000, a very > popular hardware sampler from the 90's. The disk format is known,there > are a few utilities to edit the disks on a PC and extract the PCM > samples, but there are no tools to mount it on a modern PC. Are you > saying that, since I know the MPC disk format, I could write a reiser4 > plugin to mount an MPC drive? > > If so, then Hans has an excellent point. Users do want this kind of > thing, and it is worth having to fix tar et al. You don't need reiser4 for that, writing read-only linux filesystems is trivial as soon as you have a specification of the ondisk format. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 19:34 ` Christoph Hellwig @ 2004-08-26 19:45 ` Lee Revell 0 siblings, 0 replies; 1392+ messages in thread From: Lee Revell @ 2004-08-26 19:45 UTC (permalink / raw) To: Christoph Hellwig Cc: Hans Reiser, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, 2004-08-26 at 15:34, Christoph Hellwig wrote: > On Thu, Aug 26, 2004 at 03:26:55PM -0400, Lee Revell wrote: > > OK, real world example. My roommate has an AKAI MPC-2000, a very > > popular hardware sampler from the 90's. The disk format is known,there > > are a few utilities to edit the disks on a PC and extract the PCM > > samples, but there are no tools to mount it on a modern PC. Are you > > saying that, since I know the MPC disk format, I could write a reiser4 > > plugin to mount an MPC drive? > > > > If so, then Hans has an excellent point. Users do want this kind of > > thing, and it is worth having to fix tar et al. > > You don't need reiser4 for that, writing read-only linux filesystems is > trivial as soon as you have a specification of the ondisk format. > Of course I could just write an MPC filesystem driver. And, like Andrew said, a filesystem normally gets new features with 'patch -p1'. My question, which was answered by a previous post, was whether the same could be done in a reiser4 plugins. Lee ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 19:26 ` Lee Revell 2004-08-26 19:34 ` Christoph Hellwig @ 2004-08-26 20:30 ` Dmitry Baryshkov 2004-08-26 20:38 ` Christophe Saout [not found] ` <1093548815.13881.10.camel@leto.cs.pocnet.net> 2 siblings, 1 reply; 1392+ messages in thread From: Dmitry Baryshkov @ 2004-08-26 20:30 UTC (permalink / raw) To: Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Hello, On Thu, Aug 26, 2004 at 03:26:55PM -0400, Lee Revell wrote: > On Thu, 2004-08-26 at 05:24, Hans Reiser wrote: > > > > Well, in V4, you can easily compose a plugin from plugin methods of > > other plugins, write a little piece of code with the one thing you want > > different, and add it in. Disk format changes, no big deal, add a new > > disk format plugin, or a new item plugin, or a new node plugin, etc., > > and you got your new format. > > > > OK, real world example. My roommate has an AKAI MPC-2000, a very > popular hardware sampler from the 90's. The disk format is known,there > are a few utilities to edit the disks on a PC and extract the PCM > samples, but there are no tools to mount it on a modern PC. Are you > saying that, since I know the MPC disk format, I could write a reiser4 > plugin to mount an MPC drive? > Another example: Can ext2/etx3/reiserfsv3/xfs be implemented as reiser4 plugins? From Hans' words it seems so. If this is correct, then maybe reiser4 core should be updated to completely replace current VFS layer? Then it's a good point to create a branch (in old development model it would be 2.7, dunno for new :), replace VFS layer with reiser4 core, and rewrite all (or at least most used) FS as reiser4 plugins. Then everybody will be happy. But this looks too good to be true. Perhaps I misunderstood Hans' words aboud 'new disk format', did I? -- With best wishes Dmitry Baryshkov ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:30 ` Dmitry Baryshkov @ 2004-08-26 20:38 ` Christophe Saout 2004-08-26 20:49 ` Dmitry Baryshkov 2004-08-26 21:33 ` Nikita Danilov 0 siblings, 2 replies; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 20:38 UTC (permalink / raw) To: Dmitry Baryshkov Cc: Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 821 bytes --] Am Freitag, den 27.08.2004, 00:30 +0400 schrieb Dmitry Baryshkov: > Another example: Can ext2/etx3/reiserfsv3/xfs be implemented as reiser4 > plugins? From Hans' words it seems so. If this is correct, then maybe > reiser4 core should be updated to completely replace current VFS layer? > Then it's a good point to create a branch (in old development model it > would be 2.7, dunno for new :), replace VFS layer with reiser4 core, and > rewrite all (or at least most used) FS as reiser4 plugins. Then > everybody will be happy. > > But this looks too good to be true. Perhaps I misunderstood Hans' words > aboud 'new disk format', did I? No. You can change the format the reiser4 storage tree is stored in. As long as other filesystems don't use the same underlying storage tree this is not possible. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:38 ` Christophe Saout @ 2004-08-26 20:49 ` Dmitry Baryshkov 2004-08-26 21:33 ` Nikita Danilov 1 sibling, 0 replies; 1392+ messages in thread From: Dmitry Baryshkov @ 2004-08-26 20:49 UTC (permalink / raw) To: Christophe Saout Cc: Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Hello, On Thu, Aug 26, 2004 at 10:38:12PM +0200, Christophe Saout wrote: > > No. You can change the format the reiser4 storage tree is stored in. As > long as other filesystems don't use the same underlying storage tree > this is not possible. > Thank you for clarification. As I wrote, it's too good to be true. -- With best wishes Dmitry Baryshkov ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 20:38 ` Christophe Saout 2004-08-26 20:49 ` Dmitry Baryshkov @ 2004-08-26 21:33 ` Nikita Danilov 2004-08-26 21:46 ` Christophe Saout 1 sibling, 1 reply; 1392+ messages in thread From: Nikita Danilov @ 2004-08-26 21:33 UTC (permalink / raw) To: Christophe Saout Cc: Dmitry Baryshkov, Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Christophe Saout writes: > Am Freitag, den 27.08.2004, 00:30 +0400 schrieb Dmitry Baryshkov: > > > Another example: Can ext2/etx3/reiserfsv3/xfs be implemented as reiser4 > > plugins? From Hans' words it seems so. If this is correct, then maybe > > reiser4 core should be updated to completely replace current VFS layer? > > Then it's a good point to create a branch (in old development model it > > would be 2.7, dunno for new :), replace VFS layer with reiser4 core, and > > rewrite all (or at least most used) FS as reiser4 plugins. Then > > everybody will be happy. > > > > But this looks too good to be true. Perhaps I misunderstood Hans' words > > aboud 'new disk format', did I? > > No. You can change the format the reiser4 storage tree is stored in. As > long as other filesystems don't use the same underlying storage tree > this is not possible. Wrong, plugin is called just below entry point from the VFS to the file-system back-end. It can use reiser4 tree, or any storage layer it wants. Or none at all: think about pseudo-files like foo/metas/uid---they are also implemnted by plugins. Programmer dedicated enough can (besides programming Fortran in any language), write reiser4 plugin for any file-system. I will leave description of this activity to people with better command over English language. Nikita. > ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 21:33 ` Nikita Danilov @ 2004-08-26 21:46 ` Christophe Saout 2004-08-26 22:03 ` Nikita Danilov 0 siblings, 1 reply; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 21:46 UTC (permalink / raw) To: Nikita Danilov Cc: Dmitry Baryshkov, Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 594 bytes --] Am Freitag, den 27.08.2004, 01:33 +0400 schrieb Nikita Danilov: > Wrong, plugin is called just below entry point from the VFS to the > file-system back-end. It can use reiser4 tree, or any storage layer it > wants. Or none at all: think about pseudo-files like > foo/metas/uid---they are also implemnted by plugins. Yes. But which plugins are we talking about? What about fs/reiser4/plugin/node/ and fs/reiser4/plugin/disk_format/? Of course you can implement another filesystem inside the plugins but they wouldn't use fs/reiser4/*.c, so this would be rather stupid. Right? [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 21:46 ` Christophe Saout @ 2004-08-26 22:03 ` Nikita Danilov 2004-08-26 22:05 ` Christoph Hellwig 0 siblings, 1 reply; 1392+ messages in thread From: Nikita Danilov @ 2004-08-26 22:03 UTC (permalink / raw) To: Christophe Saout Cc: Dmitry Baryshkov, Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Christophe Saout writes: > Am Freitag, den 27.08.2004, 01:33 +0400 schrieb Nikita Danilov: > > > Wrong, plugin is called just below entry point from the VFS to the > > file-system back-end. It can use reiser4 tree, or any storage layer it > > wants. Or none at all: think about pseudo-files like > > foo/metas/uid---they are also implemnted by plugins. > > Yes. But which plugins are we talking about? > > What about fs/reiser4/plugin/node/ and fs/reiser4/plugin/disk_format/? > > Of course you can implement another filesystem inside the plugins but > they wouldn't use fs/reiser4/*.c, so this would be rather stupid. Right? > That was the message of my message. Nikita. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 22:03 ` Nikita Danilov @ 2004-08-26 22:05 ` Christoph Hellwig 2004-08-27 8:12 ` Hans Reiser 0 siblings, 1 reply; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-26 22:05 UTC (permalink / raw) To: Nikita Danilov Cc: Dmitry Baryshkov, Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Fri, Aug 27, 2004 at 02:03:36AM +0400, Nikita Danilov wrote: > > What about fs/reiser4/plugin/node/ and fs/reiser4/plugin/disk_format/? > > > > Of course you can implement another filesystem inside the plugins but > > they wouldn't use fs/reiser4/*.c, so this would be rather stupid. Right? > > > > That was the message of my message. And I think Christophe and me already agreed in this thread that these upper level plugin facility should go away before merge anyway. We made the mistake of not requesting removel of that one for XFS and now SGI blocks it's removal. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 22:05 ` Christoph Hellwig @ 2004-08-27 8:12 ` Hans Reiser 2004-08-27 8:57 ` Christoph Hellwig 0 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-27 8:12 UTC (permalink / raw) To: Christoph Hellwig Cc: Nikita Danilov, Dmitry Baryshkov, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Christoph Hellwig wrote: >On Fri, Aug 27, 2004 at 02:03:36AM +0400, Nikita Danilov wrote: > > >> > What about fs/reiser4/plugin/node/ and fs/reiser4/plugin/disk_format/? >> > >> > Of course you can implement another filesystem inside the plugins but >> > they wouldn't use fs/reiser4/*.c, so this would be rather stupid. Right? >> > >> >>That was the message of my message. >> >> > >And I think Christophe and me already agreed in this thread that these >upper level plugin facility should go away before merge anyway. We made >the mistake of not requesting removel of that one for XFS and now SGI >blocks it's removal. > >- >To unsubscribe from this list: send the line "unsubscribe linux-kernel" in >the body of a message to majordomo@vger.kernel.org >More majordomo info at http://vger.kernel.org/majordomo-info.html >Please read the FAQ at http://www.tux.org/lkml/ > > > > Your arrogance is misplaced for a man who has never done a significant piece of research. Christoph, we are doing work, you are in the way, get out of the way. Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 8:12 ` Hans Reiser @ 2004-08-27 8:57 ` Christoph Hellwig 0 siblings, 0 replies; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-27 8:57 UTC (permalink / raw) To: Hans Reiser Cc: Nikita Danilov, Dmitry Baryshkov, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Fri, Aug 27, 2004 at 01:12:44AM -0700, Hans Reiser wrote: > Your arrogance is misplaced for a man who has never done a significant > piece of research. > > Christoph, we are doing work, you are in the way, get out of the way. Could you please stop the namecalling. And if you want to check who has done significant work on Linux check the commit logs for yoour and for my shortname. ^ permalink raw reply [flat|nested] 1392+ messages in thread
[parent not found: <1093548815.13881.10.camel@leto.cs.pocnet.net>]
* Re: silent semantic changes with reiser4 [not found] ` <1093548815.13881.10.camel@leto.cs.pocnet.net> @ 2004-08-27 8:05 ` Hans Reiser 2004-08-27 9:24 ` Christoph Hellwig 0 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-27 8:05 UTC (permalink / raw) To: Christophe Saout Cc: Lee Revell, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Christophe Saout wrote: >Am Donnerstag, den 26.08.2004, 15:26 -0400 schrieb Lee Revell: > > > >>>Well, in V4, you can easily compose a plugin from plugin methods of >>>other plugins, write a little piece of code with the one thing you want >>>different, and add it in. Disk format changes, no big deal, add a new >>>disk format plugin, or a new item plugin, or a new node plugin, etc., >>>and you got your new format. >>> >>> >>OK, real world example. My roommate has an AKAI MPC-2000, a very >>popular hardware sampler from the 90's. The disk format is known,there >>are a few utilities to edit the disks on a PC and extract the PCM >>samples, but there are no tools to mount it on a modern PC. Are you >>saying that, since I know the MPC disk format, I could write a reiser4 >>plugin to mount an MPC drive? >> >> > >No, the underlying storage must be a reiser4-like tree. > Not necessarily. We just encourage it.... Reiser4 is a body of code that can be sliced and diced as you choose, and it is designed for easy slicing. >If you want to >mount an MPC drive, write an MPC filesystem. > > > However, this last sentence is probably sensible advice. A mount point is probably the right interface from reiser4 for what you want. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 8:05 ` Hans Reiser @ 2004-08-27 9:24 ` Christoph Hellwig 0 siblings, 0 replies; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-27 9:24 UTC (permalink / raw) To: Hans Reiser Cc: Christophe Saout, Lee Revell, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Fri, Aug 27, 2004 at 01:05:45AM -0700, Hans Reiser wrote: > Not necessarily. We just encourage it.... Reiser4 is a body of code > that can be sliced and diced as you choose, and it is designed for easy > slicing. So what about slicing it into a simple posix fs as a start so we can review that while still discussing semantic issues. Now that would be a refreshing thing after all that talking, no? ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:45 ` Andrew Morton 2004-08-26 9:24 ` Hans Reiser @ 2004-08-26 12:18 ` Christophe Saout 2004-08-26 12:49 ` Christoph Hellwig 2004-08-26 23:54 ` Hans Reiser 1 sibling, 2 replies; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 12:18 UTC (permalink / raw) To: Andrew Morton Cc: Hans Reiser, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 2289 bytes --] Am Donnerstag, den 26.08.2004, 01:45 -0700 schrieb Andrew Morton: > And describe the "plugin" system. Why does the filesystem need such a > thing (other filesystems get their features via `patch -p1')? Ok, let me try. (Hans, please don't shoot me if I don't get every detail right, I'm trying to simplify and translate) The reiser4 core is just a big and fast storage layer using a single tree. It is bound to a mount point and uses the page cache to manage its memory. The only thing it can do on its own is to flush dirty data to the disk when the VM wants to (memory pressure) or the VFS wants to (unmount, sync) or some "plugin" wants to. Additionally it's completely atomic (with respect to crashes, no isolation like in databases), comparable to data=journal but without the overhead of using a journal for everything. Up to this point there are no users of this "database" and it does not implement any of the VFS methods for a filesystem (except mount and unmount perhaps). All the functionality is provided to the plugins. You can insert, remove, lookup or modify key:object pairs (with an index into the object). The object is a sequence of units. For files a unit would be 1 byte and the index would be the byte offset. For directories the unit would be 1 entry and the index would be the filename. Now there are some plugins that define how the storage layout on the disk is (some kind of "backend" plugins). *And* there are plugins which are users of the "reiser4 client API" and implement the actual VFS methods. There's a UNIX directory plugin and a UNIX file plugin. Directories, inodes and file content are just key:object pairs and the plugins know how to operate on these. There's a new plugin in work that also implements UNIX file semantics but stores the data for that file encrypted and/or compressed. These plugins live between the VFS and the storage layer. Just like the filesystems live between the VFS and the block layer (Hans would say that filesystems are VFS plugins ;-)). > And what are the licensing implications of plugins? Are they derived > works? Must they be GPL'ed? I suppose yes, at least currently, since they can only be linked with reiser4, there's no module infrastructure. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 12:18 ` Christophe Saout @ 2004-08-26 12:49 ` Christoph Hellwig 2004-08-26 13:00 ` Christophe Saout 2004-08-26 23:54 ` Hans Reiser 1 sibling, 1 reply; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-26 12:49 UTC (permalink / raw) To: Christophe Saout Cc: Andrew Morton, Hans Reiser, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, Aug 26, 2004 at 02:18:49PM +0200, Christophe Saout wrote: > The reiser4 core is just a big and fast storage layer using a single > tree. It is bound to a mount point and uses the page cache to manage its > memory. The only thing it can do on its own is to flush dirty data to > the disk when the VM wants to (memory pressure) or the VFS wants to > (unmount, sync) or some "plugin" wants to. > Additionally it's completely atomic (with respect to crashes, no > isolation like in databases), comparable to data=journal but without the > overhead of using a journal for everything. > > Up to this point there are no users of this "database" and it does not > implement any of the VFS methods for a filesystem (except mount and > unmount perhaps). Now that part is interesting from the how to sell it to non-Linux users POV, but not for the linux kernel. > All the functionality is provided to the plugins. You can insert, > remove, lookup or modify key:object pairs (with an index into the > object). The object is a sequence of units. For files a unit would be 1 > byte and the index would be the byte offset. For directories the unit > would be 1 entry and the index would be the filename. > > Now there are some plugins that define how the storage layout on the > disk is (some kind of "backend" plugins). > > *And* there are plugins which are users of the "reiser4 client API" and > implement the actual VFS methods. Here comes the problem. All the access checking/race avoidance/loop creation avoiced, in short the posix+extension semantics are implemented in the Linux VFS layer. If you want to allow a second access method (e.g. Hans' pet syscall) you'd have to duplicate all VFS functioanlity inside reiser4. But this is absolutely not what we want, because we moved to this common as all previous version in the different filesystems were buggy, and in fact there are problems that are simply not solveable at the level at all (rename on directories comes to mind). So if you want to provide an additional API you'll have to go through the VFS to get it right - ergo a plugin architecture for upper plugins is worthless. Plugins for the disk-format level are an interesting and workable idea on the other hand. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 12:49 ` Christoph Hellwig @ 2004-08-26 13:00 ` Christophe Saout 2004-08-26 13:07 ` Christoph Hellwig 0 siblings, 1 reply; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 13:00 UTC (permalink / raw) To: Christoph Hellwig Cc: Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1732 bytes --] Am Donnerstag, den 26.08.2004, 14:49 +0200 schrieb Christoph Hellwig: > Now that part is interesting from the how to sell it to non-Linux users > POV, but not for the linux kernel. Why? The question was what these plugins are exactly. This is the answer. > > *And* there are plugins which are users of the "reiser4 client API" and > > implement the actual VFS methods. > > Here comes the problem. All the access checking/race avoidance/loop > creation avoiced, in short the posix+extension semantics are implemented > in the Linux VFS layer. If you want to allow a second access method > (e.g. Hans' pet syscall) you'd have to duplicate all VFS functioanlity > inside reiser4. Are you actually listening? If you implement a filesystem there's a place where you have to implement the Linux VFS methods. I'm talking about inode_operations and these things. This has nothing to do with doing anything outside the Linux VFS. And I'm not talking about these metas either. These really don't belong inside the filesystem. > So if you want to provide an additional API you'll have to go through > the VFS to get it right - ergo a plugin architecture for upper plugins > is worthless. See, you didn't listen. ;-) The reiser4 core doesn't know about inodes, directories or files. It's the glue code between the VFS and the storage layer that does. It's implemented as a plugin. This has nothing to do with semantic enhancements yet. These should be removed for now and made a 2.7 topic. Maybe Namesys could provide them as an additional patch from their webpage for those who need it. The plugins that would benefit from this (encrypted and compressed file storage) aren't ready yet anyway. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 13:00 ` Christophe Saout @ 2004-08-26 13:07 ` Christoph Hellwig 2004-08-26 13:17 ` reiser4 plugins (was: silent semantic changes with reiser4) Christophe Saout ` (2 more replies) 0 siblings, 3 replies; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-26 13:07 UTC (permalink / raw) To: Christophe Saout Cc: Christoph Hellwig, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, Aug 26, 2004 at 03:00:34PM +0200, Christophe Saout wrote: > Am Donnerstag, den 26.08.2004, 14:49 +0200 schrieb Christoph Hellwig: > > > Now that part is interesting from the how to sell it to non-Linux users > > POV, but not for the linux kernel. > > Why? The question was what these plugins are exactly. This is the > answer. I explained it below.. > > > *And* there are plugins which are users of the "reiser4 client API" and > > > implement the actual VFS methods. > > > > Here comes the problem. All the access checking/race avoidance/loop > > creation avoiced, in short the posix+extension semantics are implemented > > in the Linux VFS layer. If you want to allow a second access method > > (e.g. Hans' pet syscall) you'd have to duplicate all VFS functioanlity > > inside reiser4. > > Are you actually listening? If you implement a filesystem there's a > place where you have to implement the Linux VFS methods. I'm talking > about inode_operations and these things. This has nothing to do with > doing anything outside the Linux VFS. And I'm not talking about these > metas either. These really don't belong inside the filesystem. as I wrote in this mail this absolutely _does_ belong in the filesystem, it's a major part of it and isn't easily separatable. > > So if you want to provide an additional API you'll have to go through > > the VFS to get it right - ergo a plugin architecture for upper plugins > > is worthless. > > See, you didn't listen. ;-) > > The reiser4 core doesn't know about inodes, directories or files. It's > the glue code between the VFS and the storage layer that does. It's > implemented as a plugin. This has nothing to do with semantic > enhancements yet. These should be removed for now and made a 2.7 topic. Oh yes, it is. As soon as you use different access methods on an overlapping set of objects you see exactly the problems I described. If they don't overlap there's no point for the plugins to start with, you'll better turn the core into a library that can be used by different projects then. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 13:07 ` Christoph Hellwig @ 2004-08-26 13:17 ` Christophe Saout 2004-08-26 13:24 ` Christoph Hellwig 2004-08-26 13:27 ` silent semantic changes with reiser4 Christophe Saout 2004-08-26 14:44 ` Jamie Lokier 2 siblings, 1 reply; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 13:17 UTC (permalink / raw) To: Christoph Hellwig Cc: Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1907 bytes --] Am Donnerstag, den 26.08.2004, 15:07 +0200 schrieb Christoph Hellwig: > > Why? The question was what these plugins are exactly. This is the > > answer. > > I explained it below.. It's hard to talk to you if you're not even trying to understand what I'm trying to say. Andrew wanted to know what these plugins are. I tried to give an answer and then you come along to tell me that it's not relevant. Sure, Linux doesn't care about what the filesystem does internally but Andrew wanted to know that. > > Are you actually listening? If you implement a filesystem there's a > > place where you have to implement the Linux VFS methods. I'm talking > > about inode_operations and these things. This has nothing to do with > > doing anything outside the Linux VFS. And I'm not talking about these > > metas either. These really don't belong inside the filesystem. > > as I wrote in this mail this absolutely _does_ belong in the filesystem, > it's a major part of it and isn't easily separatable. Huh? Now you're completely confusing me. First you say that that file-as-a-directory is crap then you say that it does belong into the filesystem? I'll try again: inode_operations, etc... are implemented as a "reiser4 plugin". This means that reiser4 is usable as filesystem, you can store files in a directory hierarchy. There's absolute nothing special here. Absolutely nothing. And yes, reiser4 plugins are invisible from the VFS. It looks like a normal filesystem because it behaves like one. The "metas" I was referring in the second part are this file-as-a- directory thing. These don't belong in the filesystem and should therefore be removed. The latter doesn't necessarily have anything to do with plugins. The plugins are plugins for the storage layer, not for some semantic enhancement. Note that I changed the topic to avoid further confusion. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 13:17 ` reiser4 plugins (was: silent semantic changes with reiser4) Christophe Saout @ 2004-08-26 13:24 ` Christoph Hellwig 2004-08-26 13:35 ` Christophe Saout 2004-08-28 10:59 ` Alexander Lyamin 0 siblings, 2 replies; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-26 13:24 UTC (permalink / raw) To: Christophe Saout Cc: Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain; charset=unknown-8bit, Size: 2520 bytes --] On Thu, Aug 26, 2004 at 03:17:53PM +0200, Christophe Saout wrote: > It's hard to talk to you if you're not even trying to understand what > I'm trying to say. Andrew wanted to know what these plugins are. I tried > to give an answer and then you come along to tell me that it's not > relevant. Sure, Linux doesn't care about what the filesystem does > internally but Andrew wanted to know that. Huh? I don understand what reiser4 plugins are, and I tell you that a subset of these are _wrong_ for use in the kernel. > > > Are you actually listening? If you implement a filesystem there's a > > > place where you have to implement the Linux VFS methods. I'm talking > > > about inode_operations and these things. This has nothing to do with > > > doing anything outside the Linux VFS. And I'm not talking about these > > > metas either. These really don't belong inside the filesystem. > > > > as I wrote in this mail this absolutely _does_ belong in the filesystem, > > it's a major part of it and isn't easily separatable. > > Huh? > > Now you're completely confusing me. > > First you say that that file-as-a-directory is crap then you say that it > does belong into the filesystem? I think you're talking about something different then me, I'm not talking about the magic meta files but the VFS interface in general. This VFS interface is an integral part of ævery filesystem, and it doesn't make a whole lot to put it into a plugin. If you want your filesystem core portable it does to a certain extent make sense to abstract them out, but as someone who's worked on a few such 'portable' filesystems I can tell you that it doesn't work out as expected. > I'll try again: > > inode_operations, etc... are implemented as a "reiser4 plugin". This > means that reiser4 is usable as filesystem, you can store files in a > directory hierarchy. There's absolute nothing special here. Absolutely > nothing. And yes, reiser4 plugins are invisible from the VFS. It looks > like a normal filesystem because it behaves like one. Now kill the whole plugin mess and let them talk directly to another. My first mail explained to you why it doesn't make sense to have multiple such plugins to work on a common subset of data. > The latter doesn't necessarily have anything to do with plugins. > The plugins are plugins for the storage layer, not for some semantic > enhancement. Again plugins below the pagecache (if that's what you call the "storage layer") _do_ make sense. plugins at the semantic layer don't. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 13:24 ` Christoph Hellwig @ 2004-08-26 13:35 ` Christophe Saout 2004-08-26 13:40 ` Christoph Hellwig ` (2 more replies) 2004-08-28 10:59 ` Alexander Lyamin 1 sibling, 3 replies; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 13:35 UTC (permalink / raw) To: Christoph Hellwig Cc: Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1905 bytes --] Am Donnerstag, den 26.08.2004, 15:24 +0200 schrieb Christoph Hellwig: > > First you say that that file-as-a-directory is crap then you say that it > > does belong into the filesystem? > > I think you're talking about something different then me, I'm not > talking about the magic meta files but the VFS interface in general. > > This VFS interface is an integral part of very filesystem, and it > doesn't make a whole lot to put it into a plugin. Right. That's why these plugins are linked in uncoditionally. It doesn't work without them. Hence "plugins" is not a very good name. > If you want your > filesystem core portable it does to a certain extent make sense to > abstract them out, but as someone who's worked on a few such 'portable' > filesystems I can tell you that it doesn't work out as expected. That's your opinion. reiser4 seems to work very well. > Now kill the whole plugin mess and let them talk directly to another. This is how reiserfs v3 did it. And Hans says that it actually complicated things. > My first mail explained to you why it doesn't make sense to have > multiple such plugins to work on a common subset of data. What I understood is that you can select exactly one plugin that e.g. handles the file data. The default plugin is optimized for normal files, another one could implement transparent compression or encryption. Some of these plugins also give the storage layer hints how to group files together to optimized performance. Neither of these things mess with the VFS. > > The latter doesn't necessarily have anything to do with plugins. > > The plugins are plugins for the storage layer, not for some semantic > > enhancement. > > Again plugins below the pagecache (if that's what you call the "storage > layer") _do_ make sense. plugins at the semantic layer don't. Yes, that's what I was trying to say. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 13:35 ` Christophe Saout @ 2004-08-26 13:40 ` Christoph Hellwig 2004-08-26 13:58 ` Christophe Saout 2004-08-26 23:54 ` Hans Reiser 2004-08-26 13:52 ` reiser4 plugins (was: silent semantic changes with reiser4) Rik van Riel 2004-08-28 11:18 ` Alexander Lyamin 2 siblings, 2 replies; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-26 13:40 UTC (permalink / raw) To: Christophe Saout Cc: Christoph Hellwig, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, Aug 26, 2004 at 03:35:07PM +0200, Christophe Saout wrote: > > If you want your > > filesystem core portable it does to a certain extent make sense to > > abstract them out, but as someone who's worked on a few such 'portable' > > filesystems I can tell you that it doesn't work out as expected. > > That's your opinion. reiser4 seems to work very well. So how many OSes is it ported to currently? The problem is not that it doesn't work but that it's really hard to maintain. And remember that this maintaince overhead is not just for Hans but for everyone that does VFS-level work. Unless of course we get a blanko permission to break it as soon as fixing the mess becomes non-trivial. > What I understood is that you can select exactly one plugin that e.g. > handles the file data. The default plugin is optimized for normal files, > another one could implement transparent compression or encryption. Some > of these plugins also give the storage layer hints how to group files > together to optimized performance. Neither of these things mess with the > VFS. compression or encryption must sit below the pagecache to work nicely, and this hint things that usually sit at the pagecache level. But let's assume you have a valid use for different file_operations, why don't you simply add in different file_operations instead of adding another internal dispatch layer? ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 13:40 ` Christoph Hellwig @ 2004-08-26 13:58 ` Christophe Saout 2004-08-26 15:37 ` Christoph Hellwig 2004-08-26 23:55 ` reiser4 plugins Hans Reiser 2004-08-26 23:54 ` Hans Reiser 1 sibling, 2 replies; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 13:58 UTC (permalink / raw) To: Christoph Hellwig Cc: Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 2493 bytes --] Am Donnerstag, den 26.08.2004, 15:40 +0200 schrieb Christoph Hellwig: > > That's your opinion. reiser4 seems to work very well. > > So how many OSes is it ported to currently? The problem is not that it > doesn't work but that it's really hard to maintain. And remember that > this maintaince overhead is not just for Hans but for everyone that does > VFS-level work. Where does it add overhead? It's just the interface between the reiser4 storage layer and the rest abstracted out. Ok, there's the code that dispatches the VFS method calls to the plugin call but I would say that's about it. I didn't develop reiser4 so I don't know. /* ->mkdir() VFS method in reiser4 inode_operations */ static int reiser4_mkdir(struct inode *parent /* inode of parent * directory */ , struct dentry *dentry /* dentry of new object to * create */ , int mode /* new object's mode */ ) { reiser4_object_create_data data; reiser4_stat_inc_at(parent->i_sb, vfs_calls.mkdir); data.mode = S_IFDIR | mode; data.id = DIRECTORY_FILE_PLUGIN_ID; return invoke_create_method(parent, dentry, &data); } The invoke_create_method then just calls the create method of the plugin after setting up a context and lookup up the plugin. xfs_iops.c seems to do something similar. > Unless of course we get a blanko permission to break it > as soon as fixing the mess becomes non-trivial. Well, I think so. It's done relatively often. > > What I understood is that you can select exactly one plugin that e.g. > > handles the file data. The default plugin is optimized for normal files, > > another one could implement transparent compression or encryption. Some > > of these plugins also give the storage layer hints how to group files > > together to optimized performance. Neither of these things mess with the > > VFS. > > compression or encryption must sit below the pagecache to work nicely, > and this hint things that usually sit at the pagecache level. But let's > assume you have a valid use for different file_operations, why don't you > simply add in different file_operations instead of adding another > internal dispatch layer? I don't know, ask Hans. How could the VFS know it a filesystem wants to do something specific with a file that is completely transparent to the VFS? [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 13:58 ` Christophe Saout @ 2004-08-26 15:37 ` Christoph Hellwig 2004-08-26 15:48 ` Christophe Saout 2004-08-26 23:55 ` reiser4 plugins Hans Reiser 1 sibling, 1 reply; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-26 15:37 UTC (permalink / raw) To: Christophe Saout Cc: Christoph Hellwig, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list > > compression or encryption must sit below the pagecache to work nicely, > > and this hint things that usually sit at the pagecache level. But let's > > assume you have a valid use for different file_operations, why don't you > > simply add in different file_operations instead of adding another > > internal dispatch layer? > > I don't know, ask Hans. How could the VFS know it a filesystem wants to > do something specific with a file that is completely transparent to the > VFS? The VFS shouldn't, that the whole point. That's why it allows the filesystem to register different method tables for each object. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 15:37 ` Christoph Hellwig @ 2004-08-26 15:48 ` Christophe Saout 2004-08-26 16:06 ` Christoph Hellwig 0 siblings, 1 reply; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 15:48 UTC (permalink / raw) To: Christoph Hellwig Cc: Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1191 bytes --] Am Donnerstag, den 26.08.2004, 17:37 +0200 schrieb Christoph Hellwig: > > > compression or encryption must sit below the pagecache to work nicely, > > > and this hint things that usually sit at the pagecache level. But let's > > > assume you have a valid use for different file_operations, why don't you > > > simply add in different file_operations instead of adding another > > > internal dispatch layer? > > > > I don't know, ask Hans. How could the VFS know it a filesystem wants to > > do something specific with a file that is completely transparent to the > > VFS? > > The VFS shouldn't, that the whole point. That's why it allows the > filesystem to register different method tables for each object. Only the objects it can distinguish. > ops->file = reiser4_file_operations; > ops->symlink = reiser4_symlink_inode_operations; > ops->special = reiser4_special_inode_operations; > ops->dentry = reiser4_dentry_operations; > ops->as = reiser4_as_operations; How could reiser4 register other operations for files that should be stored encrypted or compressed? It's all under reiser4_file_operations. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 15:48 ` Christophe Saout @ 2004-08-26 16:06 ` Christoph Hellwig 2004-08-26 16:14 ` Christophe Saout 0 siblings, 1 reply; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-26 16:06 UTC (permalink / raw) To: Christophe Saout Cc: Christoph Hellwig, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, Aug 26, 2004 at 05:48:54PM +0200, Christophe Saout wrote: > > > I don't know, ask Hans. How could the VFS know it a filesystem wants to > > > do something specific with a file that is completely transparent to the > > > VFS? > > > > The VFS shouldn't, that the whole point. That's why it allows the > > filesystem to register different method tables for each object. > > Only the objects it can distinguish. Yes, every inode can have different operation vectors. Which is the smallest possible object the VFS knows about. > > ops->file = reiser4_file_operations; > > ops->symlink = reiser4_symlink_inode_operations; > > ops->special = reiser4_special_inode_operations; > > ops->dentry = reiser4_dentry_operations; > > ops->as = reiser4_as_operations; > > How could reiser4 register other operations for files that should be > stored encrypted or compressed? It's all under reiser4_file_operations. Again, your confusing upper and lower plugins. For things happening below the pagecache you could register different address_space operations which sometimes makes sense. But you want e.g. different inode_operations for directories vs symlinks vs files. Please read through some linux filesystem code, okay :) ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 16:06 ` Christoph Hellwig @ 2004-08-26 16:14 ` Christophe Saout 2004-08-26 16:25 ` Christoph Hellwig 0 siblings, 1 reply; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 16:14 UTC (permalink / raw) To: Christoph Hellwig Cc: Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 986 bytes --] Am Donnerstag, den 26.08.2004, 18:06 +0200 schrieb Christoph Hellwig: > Again, your confusing upper and lower plugins. For things happening > below the pagecache you could register different address_space > operations which sometimes makes sense. But you want e.g. different > inode_operations for directories vs symlinks vs files. Ok, I see your point. aops. Sorry. Looking at the code this could be done. The wrappers that dispatch the operations are really small and call the plugin that is registered with the inode of the mapping. Instead it could have directly set the corresponding operations. Right. The wrappers are doing a few things before calling the plugin. That could be done the other way round too. But that's more of an implementation issue and could still be changed. > Please read through some linux filesystem code, okay :) I'm partly familiar with that, I just either misunderstood you or wasn't really thinking (probably the latter...). [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 16:14 ` Christophe Saout @ 2004-08-26 16:25 ` Christoph Hellwig 2004-08-26 16:39 ` Christophe Saout 0 siblings, 1 reply; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-26 16:25 UTC (permalink / raw) To: Christophe Saout Cc: Christoph Hellwig, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, Aug 26, 2004 at 06:14:19PM +0200, Christophe Saout wrote: > Ok, I see your point. aops. Sorry. > Looking at the code this could be done. The wrappers that dispatch the > operations are really small and call the plugin that is registered with > the inode of the mapping. Instead it could have directly set the > corresponding operations. Right. The wrappers are doing a few things > before calling the plugin. That could be done the other way round too. > But that's more of an implementation issue and could still be changed. I agree that it's an implementation issue. But it's also a good proof for how Hans tries to ignore all the existing infrastructure for various reasons. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 16:25 ` Christoph Hellwig @ 2004-08-26 16:39 ` Christophe Saout 0 siblings, 0 replies; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 16:39 UTC (permalink / raw) To: Christoph Hellwig Cc: Christoph Hellwig, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 964 bytes --] Am Donnerstag, den 26.08.2004, 17:25 +0100 schrieb Christoph Hellwig: > > Looking at the code this could be done. The wrappers that dispatch the > > operations are really small and call the plugin that is registered with > > the inode of the mapping. Instead it could have directly set the > > corresponding operations. Right. The wrappers are doing a few things > > before calling the plugin. That could be done the other way round too. > > But that's more of an implementation issue and could still be changed. > > I agree that it's an implementation issue. But it's also a good proof > for how Hans tries to ignore all the existing infrastructure for various > reasons. I definitely agree with you on that one. It bugged me from the beginning because things like these are very likely to suffer from unnecessary problems that have been solved in the common code or might be solved when correctly done in the common code in the future. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2004-08-26 13:58 ` Christophe Saout 2004-08-26 15:37 ` Christoph Hellwig @ 2004-08-26 23:55 ` Hans Reiser 2004-08-27 12:04 ` Nikita Danilov 1 sibling, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 23:55 UTC (permalink / raw) To: Christoph Hellwig Cc: Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Christophe Saout wrote: > > >I don't know, ask Hans. How could the VFS know it a filesystem wants to >do something specific with a file that is completely transparent to the >VFS? > > > To know what method to use, you must determine the pluginid, and then find the method within that plugin for that vfs operation. As for overhead, well, who eats whose dust in the benchmarks....? ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2004-08-26 23:55 ` reiser4 plugins Hans Reiser @ 2004-08-27 12:04 ` Nikita Danilov 2004-08-27 18:15 ` Hans Reiser 0 siblings, 1 reply; 1392+ messages in thread From: Nikita Danilov @ 2004-08-27 12:04 UTC (permalink / raw) To: Hans Reiser Cc: Christoph Hellwig, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Hans Reiser writes: > Christophe Saout wrote: > > > > > > >I don't know, ask Hans. How could the VFS know it a filesystem wants to > >do something specific with a file that is completely transparent to the > >VFS? > > > > > > > To know what method to use, you must determine the pluginid, and then > find the method within that plugin for that vfs operation. > > As for overhead, well, who eats whose dust in the benchmarks....? Whoever sponsors the benchmark usually wins. Had you forgotten that mongo setup used by http://www.namesys.com/benchmarks.html was specially `tuned' to reach peak reiser4 performance? Remember why you decided to turn OVERWRITE and MODIFY phases off? People on #reiser4 report 90 _second_ stalls with reiser4 under high io loads (large atom is obviously being flushed and everyone waits on it...). In my opinion, it is such things that are of utmost importance for real reiser4 acceptance, not how to name `metas' sub-directory. Nikita. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2004-08-27 12:04 ` Nikita Danilov @ 2004-08-27 18:15 ` Hans Reiser 2004-08-27 18:55 ` Nikita Danilov 2004-08-27 22:29 ` Steve Bergman 0 siblings, 2 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-27 18:15 UTC (permalink / raw) To: Nikita Danilov Cc: Christoph Hellwig, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Nikita Danilov wrote: >Hans Reiser writes: > > Christophe Saout wrote: > > > > > > > > > > >I don't know, ask Hans. How could the VFS know it a filesystem wants to > > >do something specific with a file that is completely transparent to the > > >VFS? > > > > > > > > > > > To know what method to use, you must determine the pluginid, and then > > find the method within that plugin for that vfs operation. > > > > As for overhead, well, who eats whose dust in the benchmarks....? > >Whoever sponsors the benchmark usually wins. Had you forgotten that >mongo setup used by http://www.namesys.com/benchmarks.html was specially >`tuned' to reach peak reiser4 performance? Remember why you decided to >turn OVERWRITE and MODIFY phases off? People on #reiser4 report 90 >_second_ stalls with reiser4 under high io loads (large atom is >obviously being flushed and everyone waits on it...). In my opinion, it >is such things that are of utmost importance for real reiser4 >acceptance, not how to name `metas' sub-directory. > >Nikita. > > > > If you ask real users, they say that reiser4 is fast, and their experience matches our benchmark. You can criticize the benchmark if you want, but then you should run your own and publish it. There are still plenty of things needing fixing though, and you are right that they are more important than the pretext being attempted for keeping us out of the kernel. Probably you could supply them with many superior pretexts.;-) The 90 second stalls, those should be fixed by debugging copy on capture and turning it on, yes? You can hardly claim I failed to push for the completion of copy on capture.... and of course the reason it is not done is simply that there is too much to do.... I (sincerely) hope vs enjoys his vacation, the whole team is working too hard. I am doing 7 day weeks now to try to make payroll for us and keep darpa happy at the same time. fsync performance needs fixing badly, and Chris is right in his diagnosis of what we can do to fix it. Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2004-08-27 18:15 ` Hans Reiser @ 2004-08-27 18:55 ` Nikita Danilov 2004-08-28 9:53 ` Hans Reiser 2004-08-29 11:17 ` Alex Zarochentsev 2004-08-27 22:29 ` Steve Bergman 1 sibling, 2 replies; 1392+ messages in thread From: Nikita Danilov @ 2004-08-27 18:55 UTC (permalink / raw) To: Hans Reiser Cc: Christoph Hellwig, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Hans Reiser writes: > Nikita Danilov wrote: > > >Hans Reiser writes: > > > Christophe Saout wrote: > > > > > > > > > > > > > > >I don't know, ask Hans. How could the VFS know it a filesystem wants to > > > >do something specific with a file that is completely transparent to the > > > >VFS? > > > > > > > > > > > > > > > To know what method to use, you must determine the pluginid, and then > > > find the method within that plugin for that vfs operation. > > > > > > As for overhead, well, who eats whose dust in the benchmarks....? > > > >Whoever sponsors the benchmark usually wins. Had you forgotten that > >mongo setup used by http://www.namesys.com/benchmarks.html was specially > >`tuned' to reach peak reiser4 performance? Remember why you decided to > >turn OVERWRITE and MODIFY phases off? People on #reiser4 report 90 > >_second_ stalls with reiser4 under high io loads (large atom is > >obviously being flushed and everyone waits on it...). In my opinion, it > >is such things that are of utmost importance for real reiser4 > >acceptance, not how to name `metas' sub-directory. > > > >Nikita. > > > > > > > > > If you ask real users, they say that reiser4 is fast, and their > experience matches our benchmark. You can criticize the benchmark if They experience 90 second stalls. And please, do not tell me how fast reiser4 is, I spent a lot of time working with it, and I know very well when it's fast, and when it's deadly slow. > you want, but then you should run your own and publish it. I did, after which you told me to turn OVERWRITE and MODIFY phases off, beause performance was horrible. I not criticizing mongo benchmark per se. I think that it is fundamentally wrong to use results that were deliberatly manipulated to get best appearance to reiser4 (by omitting work-loads where it performs poorly) as an argument. It's not clear who will, according to your colorful expression, `eat dust' as a result of this. Or do you think that users never overwrite of modify files in real life? > The 90 second stalls, those should be fixed by debugging copy on capture > and turning it on, yes? You can hardly claim I failed to push for the No. I described so many times to you, why COC is ineffectual here. > Hans Nikita. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2004-08-27 18:55 ` Nikita Danilov @ 2004-08-28 9:53 ` Hans Reiser 2004-08-28 13:47 ` Nikita Danilov 2004-08-29 11:17 ` Alex Zarochentsev 1 sibling, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-28 9:53 UTC (permalink / raw) To: Nikita Danilov Cc: Christoph Hellwig, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list, Alexander Zarochentcev Nikita Danilov wrote: > > >Whoever sponsors the benchmark usually wins. Had you forgotten that > > >mongo setup used by http://www.namesys.com/benchmarks.html was specially > > >`tuned' to reach peak reiser4 performance? Remember why you decided to > > >turn OVERWRITE and MODIFY phases off? > What I should have done was what I did with fsync performance. With fsync performance I told people that we had not yet tuned for it, please wait a bit and we will tune for it, for now it sucks. Instead what I did was discuss with Zam at the time how it could be fixed, leave it off the website until Zam was given a chance to fix it, and then I managed to forget about it. After release one remembers what all the things that should have been fixed before release were, sigh. Zam and I both know what is needed to fix performance in these phases, and I just spoke with him on the phone. I imagine it is 2 weeks of work for him, but it could be up to 6 weeks. He will comment later. The fix should consist of the following: 1) tweak the relocation policies (zam will say more about this, as he is the maintainer of them) 2) optimize overwrite set so that in the modify phase we behave like a write twice journaling filesystem, which means implement a reserved for the journal only area that exists as long as plenty of disk space is free. 3) finish the repacker (but we won't do that this month....) The modify phase, which picks a random block to modify, is a worst case for a wandering log without a repacker. We could actually do very very well at that kind of activity with a repacker, probably better than a fixed journal, but for now we should try acting like a write twice journal for atoms that small. So, having realized my error thanks to the gracious kindness of how you pointed it out, we will modify the web site to say that we are not yet tuned for and currently suck at modifying random blocks in the middle of files, and appending small amounts to the end of them, and overwriting small files, but that we think we know what is needed to fix those things. I think your characterization of my reasons was unkind and also unfair. I will pass on saying similarly unkind things about you as you are mostly a good person. You never did seem to understand me. You didn't understand the reasons for my technical designs (when they worked it made no sense to you that they did), and you didn't understand me as a person. I wish you well in your new job, and thank you for the hard work you did for me. Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2004-08-28 9:53 ` Hans Reiser @ 2004-08-28 13:47 ` Nikita Danilov 2004-08-28 23:45 ` Hans Reiser 0 siblings, 1 reply; 1392+ messages in thread From: Nikita Danilov @ 2004-08-28 13:47 UTC (permalink / raw) To: Hans Reiser Cc: Christoph Hellwig, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list, Alexander Zarochentcev Hans Reiser writes: > Nikita Danilov wrote: > > > > >Whoever sponsors the benchmark usually wins. Had you forgotten that > > > >mongo setup used by http://www.namesys.com/benchmarks.html was specially > > > >`tuned' to reach peak reiser4 performance? Remember why you decided to > > > >turn OVERWRITE and MODIFY phases off? > > > What I should have done was what I did with fsync performance. With > fsync performance I told people that we had not yet tuned for it, please > wait a bit and we will tune for it, for now it sucks. Right, and instead of this you are now claiming that these `benchmarks' show superior reiser4 performance. Moreover, you are doing this aggressively and proposing other people to `eat the dust' over what happened to only exist due to your failing to remember something. As some other notable LKML poster put it `inform yourself before posting'. :-) > > Instead what I did was discuss with Zam at the time how it could be > fixed, leave it off the website until Zam was given a chance to fix it, It wasn't Zam. It was me to begin with. OVERWRITE and MODIFY phases were turned off after switching to large keys. > and then I managed to forget about it. After release one remembers > what all the things that should have been fixed before release were, sigh. > [...] > > I think your characterization of my reasons was unkind and also unfair. What fairness and kindness one expects after styling others as `puppies' in public, and commenting on their hair style in purportedly technical argument? [...] > > Hans Nikita. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2004-08-28 13:47 ` Nikita Danilov @ 2004-08-28 23:45 ` Hans Reiser 2004-08-29 9:35 ` Nikita Danilov 0 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-28 23:45 UTC (permalink / raw) To: Nikita Danilov Cc: Christoph Hellwig, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list, Alexander Zarochentcev Nikita Danilov wrote: > > > > Instead what I did was discuss with Zam at the time how it could be > > fixed, leave it off the website until Zam was given a chance to fix it, > >It wasn't Zam. It was me to begin with. OVERWRITE and MODIFY phases were >turned off after switching to large keys. > > > The online repacker is the best technical fix for these problems. Tight packing is generally more damaged in its layout by small changes than loose packing, and V4 is the ultimate tight packer. It is not clear to me that making V4 pack more loosely is necessarily a good idea. These phases disturb the original packing at flush time, and the other phases don't. One solution to try is packing less tightly. Maybe once every megabyte skip forward 1 megabyte, and once every 64 megabytes skip to a random disk location, when allocating large new atoms. I think I prefer using a repacker, but we should try it and see. The other solutions of the previous email should also help a lot. The issues of layout in this are similar to the ones improved by the fibration plugin. I remember talking with not just you but zam about how we could fix it, and there was too much in the queue of work, and everyone was complaining to me that we should be debugging not optimizing, and you were the only one who thought it was a big deal. I guess you still think it is a big issue and I still think things are good enough to use without it. I still think Zam understood the issues better than you did (allocation is his code not yours). There are some layout optimizations that a repacker can do best. Still, there are some rough spots in the current allocation policy, and we should look at it. Probably you will have reason to howl if we setup a benchmark which disturbs things with these phases, runs the repacker (I hope to have one in 6 months), and then measures our read performance compared to other filesystems without a repacker....;-) Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2004-08-28 23:45 ` Hans Reiser @ 2004-08-29 9:35 ` Nikita Danilov 0 siblings, 0 replies; 1392+ messages in thread From: Nikita Danilov @ 2004-08-29 9:35 UTC (permalink / raw) To: Hans Reiser Cc: Christoph Hellwig, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list, Alexander Zarochentcev Hans Reiser writes: [...] > > I remember talking with not just you but zam about how we could fix it, > and there was too much in the queue of work, and everyone was > complaining to me that we should be debugging not optimizing, and you > were the only one who thought it was a big deal. I guess you still You are trying to evade the point. And the point is not how performance in these phases (and their impact on other phases) can be improved, or what excuses did you have for not `doing' it, but the fact that after it was found that large keys behave badly, these phases were turned off, others were switched to lexicographical operation, _and_ resulting benchmark is used as a basis to call other people names. > think it is a big issue and I still think things are good enough to use > without it. I still think Zam understood the issues better than you did > (allocation is his code not yours). > > There are some layout optimizations that a repacker can do best. Still, > there are some rough spots in the current allocation policy, and we > should look at it. > > Probably you will have reason to howl if we setup a benchmark which Surely, as a `big dog' what other sounds can I produce? :) > disturbs things with these phases, runs the repacker (I hope to have one > in 6 months), and then measures our read performance compared to other > filesystems without a repacker....;-) That's ok with me, but it would make sense to remove unfair benchmarks from the site until then. > > Hans Nikita. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2004-08-27 18:55 ` Nikita Danilov 2004-08-28 9:53 ` Hans Reiser @ 2004-08-29 11:17 ` Alex Zarochentsev 1 sibling, 0 replies; 1392+ messages in thread From: Alex Zarochentsev @ 2004-08-29 11:17 UTC (permalink / raw) To: Nikita Danilov Cc: Hans Reiser, Christoph Hellwig, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Fri, Aug 27, 2004 at 10:55:50PM +0400, Nikita Danilov wrote: > Hans Reiser writes: > > Nikita Danilov wrote: > > > > >Hans Reiser writes: > > > > Christophe Saout wrote: > > > > > > > > > > > > > > > > > > >I don't know, ask Hans. How could the VFS know it a filesystem wants to > > > > >do something specific with a file that is completely transparent to the > > > > >VFS? > > > > > > > > > > > > > > > > > > > To know what method to use, you must determine the pluginid, and then > > > > find the method within that plugin for that vfs operation. > > > > > > > > As for overhead, well, who eats whose dust in the benchmarks....? > > > > > >Whoever sponsors the benchmark usually wins. Had you forgotten that > > >mongo setup used by http://www.namesys.com/benchmarks.html was specially > > >`tuned' to reach peak reiser4 performance? Remember why you decided to > > >turn OVERWRITE and MODIFY phases off? People on #reiser4 report 90 > > >_second_ stalls with reiser4 under high io loads (large atom is > > >obviously being flushed and everyone waits on it...). In my opinion, it > > >is such things that are of utmost importance for real reiser4 > > >acceptance, not how to name `metas' sub-directory. > > > > > >Nikita. > > > > > > > > > > > > > > If you ask real users, they say that reiser4 is fast, and their > > experience matches our benchmark. You can criticize the benchmark if > > They experience 90 second stalls. And please, do not tell me how fast > reiser4 is, I spent a lot of time working with it, and I know very well > when it's fast, and when it's deadly slow. > > > you want, but then you should run your own and publish it. > > I did, after which you told me to turn OVERWRITE and MODIFY phases off, > beause performance was horrible. Hmm, I think not the modify/overwrite phases performance were the problem. The modify and overwrite mongo phases fragment the filesystem too much, it had a bad effect visible in the read phase and even in delete phase. We measured/optimized read performance in assumption that fs is not fragmented, why we should measure bad effect of overwrite instead? other fs do not relocate already allocated blocks, negative effect of file set overwrite is zero for them. We could do the same for reiser4 by tuning flush algorithm and journal blocks allocation. If users want to see read and delete performance as they are, overwrite phase should be excluded, at least until we teach flush algorithm to not break good block allocation. If one wants to measure overwite performance, it is easy to enable overwrite phase in mongo, but read and delete speed will be affected (it was so when I tried to optimize delete). > I not criticizing mongo benchmark per se. I think that it is > fundamentally wrong to use results that were deliberatly manipulated to > get best appearance to reiser4 (by omitting work-loads where it performs > poorly) as an argument. It's not clear who will, according to your > colorful expression, `eat dust' as a result of this. > Or do you think > that users never overwrite of modify files in real life? not whole file set. I mean it is unusual to overwrite all files in the fs. > > The 90 second stalls, those should be fixed by debugging copy on capture > > and turning it on, yes? You can hardly claim I failed to push for the > > No. I described so many times to you, why COC is ineffectual here. There are other ideas :) but, it seems to me, if we will focus on benchmarks, we never complete plugins, metafiles and other useful things. What I want from reiser4 is a possiblity to encrypt and compress ~/Mail/ :). > > > Hans > > Nikita. -- Alex. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2004-08-27 18:15 ` Hans Reiser 2004-08-27 18:55 ` Nikita Danilov @ 2004-08-27 22:29 ` Steve Bergman 2004-08-28 6:54 ` Hans Reiser 2004-08-29 11:42 ` Alex Zarochentsev 1 sibling, 2 replies; 1392+ messages in thread From: Steve Bergman @ 2004-08-27 22:29 UTC (permalink / raw) To: Hans Reiser Cc: Nikita Danilov, Christoph Hellwig, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs On Fri, 2004-08-27 at 11:15 -0700, Hans Reiser wrote: > > > If you ask real users, they say that reiser4 is fast, and their > experience matches our benchmark. You can criticize the benchmark if > you want, but then you should run your own and publish it. > As a "real" desktop user who just converted all his partitions from ext3 to reiser4, I have not, as yet, noticed any startling performance increase. Being slightly slightly irked to hear that the benchmark numbers that have been paraded around on Slashdot and the internet in general, at ext3's expense, have had reiser4's "bad" results surgically extracted, I am running my own benchmarks to get the real story on reiser4/ext3 mongo performance on my, rather average, desktop hardware. I am using the latest Mongo on FC/rawhide and the 2.6.8.1-mm4 kernel. Unfortunately, I get an error from mongo.pl that "Done" is not a numeric argument at line 439. I have done this to fix it: --- mongo.pl 2004-08-27 17:07:01.681723313 -0500 +++ mongo_fixed.pl 2004-08-27 17:06:51.369306735 -0500 @@ -429,8 +429,8 @@ if ( -e ${ERR_FILE}) { &DIE ("\nEXITED WITH FAIL\n"); } - my $real = (split ' ', $time_output[0])[1]; - my $cpu = (split ' ', $time_output[2])[1]; + my $real = (split ' ', $time_output[1])[1]; + my $cpu = (split ' ', $time_output[3])[1]; unless ( $real =~ /\s*\d+/ && $cpu =~ /\s*\d+/) { LOG "@time_output"; What it gets me is the "real" line of the "time" output for "STAT REAL_TIME" and the "sys" line of the "time" output for "STAT CPU_TIME". i.e. only system time is counted. I believe this was the intent of the original code, but want to verify before continuing. Thanks, Steve Bergman ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2004-08-27 22:29 ` Steve Bergman @ 2004-08-28 6:54 ` Hans Reiser [not found] ` <1093687768.8097.25.camel@voyager.localdomain> 2004-08-29 11:42 ` Alex Zarochentsev 1 sibling, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-28 6:54 UTC (permalink / raw) To: Steve Bergman Cc: Nikita Danilov, Christoph Hellwig, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs, Alexander Zarochentcev Steve Bergman wrote: >On Fri, 2004-08-27 at 11:15 -0700, Hans Reiser wrote: > > >>If you ask real users, they say that reiser4 is fast, and their >>experience matches our benchmark. You can criticize the benchmark if >>you want, but then you should run your own and publish it. >> >> >> > > >As a "real" desktop user who just converted all his partitions from ext3 >to reiser4, I have not, as yet, noticed any startling performance >increase. Being slightly slightly irked to hear that the benchmark >numbers that have been paraded around on Slashdot and the internet in >general, at ext3's expense, have had reiser4's "bad" results surgically >extracted, I am running my own benchmarks to get the real story on >reiser4/ext3 mongo performance on my, rather average, desktop hardware. > >I am using the latest Mongo on FC/rawhide and the 2.6.8.1-mm4 kernel. > >Unfortunately, I get an error from mongo.pl that "Done" is not a numeric >argument at line 439. > >I have done this to fix it: > > >--- mongo.pl 2004-08-27 17:07:01.681723313 -0500 >+++ mongo_fixed.pl 2004-08-27 17:06:51.369306735 -0500 >@@ -429,8 +429,8 @@ > if ( -e ${ERR_FILE}) { > &DIE ("\nEXITED WITH FAIL\n"); > } >- my $real = (split ' ', $time_output[0])[1]; >- my $cpu = (split ' ', $time_output[2])[1]; >+ my $real = (split ' ', $time_output[1])[1]; >+ my $cpu = (split ' ', $time_output[3])[1]; > > unless ( $real =~ /\s*\d+/ && $cpu =~ /\s*\d+/) { > LOG "@time_output"; > > >What it gets me is the "real" line of the "time" output for "STAT >REAL_TIME" and the "sys" line of the "time" output for "STAT CPU_TIME". >i.e. only system time is counted. I believe this was the intent of the >original code, but want to verify before continuing. > >Thanks, >Steve Bergman > > > > > I didn't write this (more precisely, it only vaguely resembles what I wrote in 1996). Are you saying that it reports system time as real time? If yes, then it is an error, we need to go remove a bunch of numbers from our benchmarks, and thanks for finding it. Zam, please comment. Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
[parent not found: <1093687768.8097.25.camel@voyager.localdomain>]
* Re: reiser4 plugins [not found] ` <1093687768.8097.25.camel@voyager.localdomain> @ 2004-08-29 11:55 ` Alex Zarochentsev 0 siblings, 0 replies; 1392+ messages in thread From: Alex Zarochentsev @ 2004-08-29 11:55 UTC (permalink / raw) To: Steve Bergman; +Cc: Hans Reiser, Nikita Danilov, reiserfs, linux-kernel On Sat, Aug 28, 2004 at 05:09:28AM -0500, Steve Bergman wrote: > On Fri, 2004-08-27 at 23:54 -0700, Hans Reiser wrote: > > > > > I didn't write this (more precisely, it only vaguely resembles what I > > wrote in 1996). Are you saying that it reports system time as real > > time? If yes, then it is an error, we need to go remove a bunch of > > numbers from our benchmarks, and thanks for finding it. > > > > Zam, please comment. > > > > No. I'm saying that on my setup (kernel 2.6.8.1-mm4/perl 5.8.5/bash > 3.00) the first line returned from running the test is the: > > [1]+ Done ... > line which throws the array indexes off by 1 and I always get 0 for the > real time and, yes, the real time gets reported as the system time, I > think. Plus I get a warning about the fact that "Done" is not numeric. > This is obviously a problem with my particular setup. Yes. but that real/sys time parsing isn't reliable. > After bumping the indexes up by 1, I get the correct real time reported > as "STAT REAL_TIME". And the system time is reported as "STAT > CPU_TIME". > > "STAT CPU_UTIL", however is computed in a completely different way based > on numbers collected from /proc/stat. If I'm, reading > linux/Documentation/filesystems/proc.txt correctly, it is trying to > return the (system time) / (total time). The total time agrees with > "STAT REAL_TIME". However, the (system time) that it comes up with here > is always considerably higher than "STAT CPU_TIME". CPU_UTIL counts other background processes too. The foreground process may just wait when all work is done by pdflush. I think CPU_UTIL is more useful than CPU_TIME. > > User error is quite possible, as I am: > > 1. Just getting to know mongo. > 2. Not a perl guy. i don't like perl too much, but seems Perl is perfect for the things you just did (the fixes). > 3. There is obviously something mongo.pl doesn't like about my setup. > > -Steve Bergman > > P.S. In the 2 tests I've run, reiser4 is not doing all that badly > against ext3 in OVERWRITE and MODIFY, though ext3 does come in faster. > Reiser4 trails badly on APPEND however, ext3 (data=ordered, without > htree) being some 2.5 - 4 times faster. -- Alex. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2004-08-27 22:29 ` Steve Bergman 2004-08-28 6:54 ` Hans Reiser @ 2004-08-29 11:42 ` Alex Zarochentsev 1 sibling, 0 replies; 1392+ messages in thread From: Alex Zarochentsev @ 2004-08-29 11:42 UTC (permalink / raw) To: Steve Bergman Cc: Hans Reiser, Nikita Danilov, Christoph Hellwig, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs On Fri, Aug 27, 2004 at 05:29:07PM -0500, Steve Bergman wrote: > On Fri, 2004-08-27 at 11:15 -0700, Hans Reiser wrote: > > > > > If you ask real users, they say that reiser4 is fast, and their > > experience matches our benchmark. You can criticize the benchmark if > > you want, but then you should run your own and publish it. > > > > > As a "real" desktop user who just converted all his partitions from ext3 > to reiser4, I have not, as yet, noticed any startling performance > increase. Being slightly slightly irked to hear that the benchmark > numbers that have been paraded around on Slashdot and the internet in > general, at ext3's expense, have had reiser4's "bad" results surgically > extracted, I am running my own benchmarks to get the real story on > reiser4/ext3 mongo performance on my, rather average, desktop hardware. > > I am using the latest Mongo on FC/rawhide and the 2.6.8.1-mm4 kernel. > > Unfortunately, I get an error from mongo.pl that "Done" is not a numeric > argument at line 439. > > I have done this to fix it: > > > --- mongo.pl 2004-08-27 17:07:01.681723313 -0500 > +++ mongo_fixed.pl 2004-08-27 17:06:51.369306735 -0500 > @@ -429,8 +429,8 @@ > if ( -e ${ERR_FILE}) { > &DIE ("\nEXITED WITH FAIL\n"); > } > - my $real = (split ' ', $time_output[0])[1]; > - my $cpu = (split ' ', $time_output[2])[1]; > + my $real = (split ' ', $time_output[1])[1]; > + my $cpu = (split ' ', $time_output[3])[1]; > > unless ( $real =~ /\s*\d+/ && $cpu =~ /\s*\d+/) { > LOG "@time_output"; > > > What it gets me is the "real" line of the "time" output for "STAT > REAL_TIME" and the "sys" line of the "time" output for "STAT CPU_TIME". > i.e. only system time is counted. I believe this was the intent of the > original code, but want to verify before continuing. it works on our test servers which are SuSE9.0/9.1. I think there is a dependency on system utilities, "Done" is not printed by mongo.pl or any other program from the mongo package. It would be nice to tell us what linux distro you are using and what mongo phase fails. > Thanks, > Steve Bergman Thanks. -- Alex. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2004-08-26 13:40 ` Christoph Hellwig 2004-08-26 13:58 ` Christophe Saout @ 2004-08-26 23:54 ` Hans Reiser 1 sibling, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 23:54 UTC (permalink / raw) To: Christoph Hellwig Cc: Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Christoph Hellwig wrote: > > The problem is not that it >doesn't work but that it's really hard to maintain. > You really like to talk about what you know nothing of, yes? Reiser4 is FAR easier to maintain than V3. Ask anyone on reiserfs-dev. It was designed to reduce our software maintenance costs. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 13:35 ` Christophe Saout 2004-08-26 13:40 ` Christoph Hellwig @ 2004-08-26 13:52 ` Rik van Riel 2004-08-26 13:58 ` Chris Mason 2004-08-26 14:02 ` Christophe Saout 2004-08-28 11:18 ` Alexander Lyamin 2 siblings, 2 replies; 1392+ messages in thread From: Rik van Riel @ 2004-08-26 13:52 UTC (permalink / raw) To: Christophe Saout Cc: Christoph Hellwig, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, 26 Aug 2004, Christophe Saout wrote: > That's your opinion. reiser4 seems to work very well. Have you tried /bin/cp, or a backup+restore ? What happened to your file streams ? -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 13:52 ` reiser4 plugins (was: silent semantic changes with reiser4) Rik van Riel @ 2004-08-26 13:58 ` Chris Mason 2004-08-26 14:02 ` Christophe Saout 1 sibling, 0 replies; 1392+ messages in thread From: Chris Mason @ 2004-08-26 13:58 UTC (permalink / raw) To: Rik van Riel Cc: Christophe Saout, Christoph Hellwig, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, 2004-08-26 at 09:52, Rik van Riel wrote: > On Thu, 26 Aug 2004, Christophe Saout wrote: > > > That's your opinion. reiser4 seems to work very well. > > Have you tried /bin/cp, or a backup+restore ? > > What happened to your file streams ? Shrug, backup programs can be fixed. The fact that we couldn't backup 16GB files before O_LARGEFILE wasn't a reason not to do it. IMHO it's more important to ask the question 'is this important for linux' and then we can figure out how to make it work. -chris ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 13:52 ` reiser4 plugins (was: silent semantic changes with reiser4) Rik van Riel 2004-08-26 13:58 ` Chris Mason @ 2004-08-26 14:02 ` Christophe Saout 1 sibling, 0 replies; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 14:02 UTC (permalink / raw) To: Rik van Riel Cc: Christoph Hellwig, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 560 bytes --] Am Donnerstag, den 26.08.2004, 09:52 -0400 schrieb Rik van Riel: > > That's your opinion. reiser4 seems to work very well. > > Have you tried /bin/cp, or a backup+restore ? > > What happened to your file streams ? 1. I wasn't talking about reiser4 plugins, not about the file-as-a-directory thing or file streams 2. reiser4 doesn't currently have those 3. I don't really like file streams :-) But I'm using ACLs on a server and had to forward-port an rsync patch to make full backups using rsync possible. That's not really better. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 13:35 ` Christophe Saout 2004-08-26 13:40 ` Christoph Hellwig 2004-08-26 13:52 ` reiser4 plugins (was: silent semantic changes with reiser4) Rik van Riel @ 2004-08-28 11:18 ` Alexander Lyamin 2004-08-28 11:22 ` Christoph Hellwig 2004-08-28 16:11 ` Jan Harkes 2 siblings, 2 replies; 1392+ messages in thread From: Alexander Lyamin @ 2004-08-28 11:18 UTC (permalink / raw) To: Christophe Saout Cc: Christoph Hellwig, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Thu, Aug 26, 2004 at 03:35:07PM +0200, Christophe Saout wrote: > Am Donnerstag, den 26.08.2004, 15:24 +0200 schrieb Christoph Hellwig: > > > > First you say that that file-as-a-directory is crap then you say that it > > > does belong into the filesystem? > > > > I think you're talking about something different then me, I'm not > > talking about the magic meta files but the VFS interface in general. > > > > This VFS interface is an integral part of very filesystem, and it > > doesn't make a whole lot to put it into a plugin. > > Right. That's why these plugins are linked in uncoditionally. It doesn't > work without them. Hence "plugins" is not a very good name. its still plugins no matter what. they just emulate "conventional filesystem" behavior for VFS, but at some point you might want just to scrap VFS.. if you'd like to. And I honestly dont understand whats the other Christoph's worries are about. Its got perfomance. Its there. It can emulate "conventional filesystem" behaviour, for legacy apps. Thouse two that are currently crippled by metas, we are so happy arguing about, will get fixed fast. Thats a point where you could happily STOP and live with your happy "conventional filesystem emulation", yet enjoying perfomance aspects (if perfomance hurts you, do not compile reiserfs, just like one big-red distro does, nobody would not complain. except your hard-drive mechanics sick of seeks). But most people would not stop, and its good. Yes, I think it would be nice to have this infrastructure in VFS. Technically. But its not possible, cause of "committee clusterfuck". Socially. Stupidly. -- "the liberation loophole will make it clear.." lex lyamin ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 11:18 ` Alexander Lyamin @ 2004-08-28 11:22 ` Christoph Hellwig 2004-08-28 11:46 ` Alexander Lyamin 2004-08-28 16:11 ` Jan Harkes 1 sibling, 1 reply; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-28 11:22 UTC (permalink / raw) To: flx, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Sat, Aug 28, 2004 at 03:18:07PM +0400, Alexander Lyamin wrote: > Yes, I think it would be nice to have this infrastructure in VFS. Technically. > But its not possible, cause of "committee clusterfuck". Socially. Stupidly. Please explain your problems. I've not seen a single actually working proposal yet. Current reiser4 implementation apparently oopses and breaks guaranteed fs semantics, it's not going in in either reiser4 or the vfs. Al has started a thread to hash out working semantics, but there's not been a single namesys person involved. Similar all of you have absolutely ignore every single technical question or comment but started a flamefest consisting mostly of personal attacs and "mine is longer" politics. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 11:22 ` Christoph Hellwig @ 2004-08-28 11:46 ` Alexander Lyamin 2004-08-28 11:51 ` Christoph Hellwig 2004-08-28 11:58 ` Markus Törnqvist 0 siblings, 2 replies; 1392+ messages in thread From: Alexander Lyamin @ 2004-08-28 11:46 UTC (permalink / raw) To: Christoph Hellwig, flx, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Sat, Aug 28, 2004 at 01:22:55PM +0200, Christoph Hellwig wrote: > On Sat, Aug 28, 2004 at 03:18:07PM +0400, Alexander Lyamin wrote: > > Yes, I think it would be nice to have this infrastructure in VFS. Technically. > > But its not possible, cause of "committee clusterfuck". Socially. Stupidly. > > Please explain your problems. I've not seen a single actually working > proposal yet. Current reiser4 implementation apparently oopses and Not true. If true - send bugreports. Its work for me couple of months. there were few hiccups, but they got fixed quickly by zam@namesys.com. only ext2 partition is /boot cause of BIOS limitations. Yes, i use it with LVM2 and stuff... de facto its even more stable then reiserfs 3.6, which got issues with CONFIG_PREEMPT. > breaks guaranteed fs semantics, it's not going in in either reiser4 or > the vfs. A > > Al has started a thread to hash out working semantics, but there's not been > a single namesys person involved. Similar all of you have absolutely ignore Al message had a reply. "namesys persons" is Hans Reiser. Sufficient ? > every single technical question or comment but started a flamefest > consisting mostly of personal attacs and "mine is longer" politics. no need to comment on this. -- "the liberation loophole will make it clear.." lex lyamin ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 11:46 ` Alexander Lyamin @ 2004-08-28 11:51 ` Christoph Hellwig 2004-08-28 12:18 ` Alexander Lyamin 2004-08-28 11:58 ` Markus Törnqvist 1 sibling, 1 reply; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-28 11:51 UTC (permalink / raw) To: flx, Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Sat, Aug 28, 2004 at 03:46:28PM +0400, Alexander Lyamin wrote: > Not true. If true - send bugreports. > > Its work for me couple of months. there were few hiccups, but they got fixed > quickly by zam@namesys.com. only ext2 partition is /boot cause of BIOS > limitations. Yes, i use it with LVM2 and stuff... See the mails from Christophe in this thread. > > breaks guaranteed fs semantics, it's not going in in either reiser4 or > > the vfs. > A > > > > Al has started a thread to hash out working semantics, but there's not been > > a single namesys person involved. Similar all of you have absolutely ignore > Al message had a reply. > "namesys persons" is Hans Reiser. Sufficient ? http://marc.theaimsgroup.com/?t=109355582600002&r=1&w=2 I can't see Hans anywhere. And honestly Hans has been so out of touch with VFS internals that some person actually understanding the issue might be helpfull. That would probably whoever has taken over Nikita's position. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 11:51 ` Christoph Hellwig @ 2004-08-28 12:18 ` Alexander Lyamin 0 siblings, 0 replies; 1392+ messages in thread From: Alexander Lyamin @ 2004-08-28 12:18 UTC (permalink / raw) To: Christoph Hellwig, flx, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Sat, Aug 28, 2004 at 01:51:04PM +0200, Christoph Hellwig wrote: > On Sat, Aug 28, 2004 at 03:46:28PM +0400, Alexander Lyamin wrote: > > Its work for me couple of months. there were few hiccups, but they got fixed > > quickly by zam@namesys.com. only ext2 partition is /boot cause of BIOS > > limitations. Yes, i use it with LVM2 and stuff... > > See the mails from Christophe in this thread. seen. noted. > > > breaks guaranteed fs semantics, it's not going in in either reiser4 or > > > the vfs. > > A > > > > > > Al has started a thread to hash out working semantics, but there's not been > > > a single namesys person involved. Similar all of you have absolutely ignore > > Al message had a reply. > > "namesys persons" is Hans Reiser. Sufficient ? > > http://marc.theaimsgroup.com/?t=109355582600002&r=1&w=2 > > I can't see Hans anywhere. And honestly Hans has been so out of touch > with VFS internals that some person actually understanding the issue > might be helpfull. That would probably whoever has taken over Nikita's > position. I assume Vladimir Saveliev could play this out... -- "the liberation loophole will make it clear.." lex lyamin ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 11:46 ` Alexander Lyamin 2004-08-28 11:51 ` Christoph Hellwig @ 2004-08-28 11:58 ` Markus Törnqvist 1 sibling, 0 replies; 1392+ messages in thread From: Markus Törnqvist @ 2004-08-28 11:58 UTC (permalink / raw) To: flx, Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Sat, Aug 28, 2004 at 03:46:28PM +0400, Alexander Lyamin wrote: >Not true. If true - send bugreports. Hate to say it, but apparently symlinking out of metas oopses, but that should be trivial to fix? >> Al has started a thread to hash out working semantics, but there's not been >> a single namesys person involved. Similar all of you have absolutely ignore >Al message had a reply. >"namesys persons" is Hans Reiser. Sufficient ? Not to be disrespectful, but Hans has sometimes taken personal shots on the lists and that's not a productive approach. Not that he'd be the only one around to do so. Seems to me, as Christoph said, that some other Namesys guy than Hans should participate in the Al/Linus branch of the thread with ideas. Certainly the VFS may have to be extended right about now, so someone who knows the exact details of Reiser4 should offer ideas and I'm not so sure it's Hans. Looks like he had the ideas but not the implementation. There's also the risk that Reiser4's implementation's don't really bend into even a modified/extended VFS because of the plugins? That's definitely something that has to be approached properly... -- mjt ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 11:18 ` Alexander Lyamin 2004-08-28 11:22 ` Christoph Hellwig @ 2004-08-28 16:11 ` Jan Harkes 2004-08-28 16:26 ` Nikita Danilov 2004-08-28 19:41 ` Alexander Lyamin 1 sibling, 2 replies; 1392+ messages in thread From: Jan Harkes @ 2004-08-28 16:11 UTC (permalink / raw) To: flx, Christophe Saout, Christoph Hellwig, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Sat, Aug 28, 2004 at 03:18:07PM +0400, Alexander Lyamin wrote: > And I honestly dont understand whats the other Christoph's worries are about. I honestly don't understand what all this 'grassroot' campaining is all about. Very valid technical points have been made and mostly ignored. The whole 'it breaks backups' noise is just that, noise. - It breaks VFS locking rules. - It pretends to provide a uniform way to store streams or metadata. However it only does so files and not for directories, symlinks, fifos, unix domain sockets, or device nodes, which makes it less than uniform. - From what I saw in one part of the discussion, it allows for infinite depth recursion (file/metas/metas/metas/...). Some applications are going to have a lot of fun with that. And finally, - When reiserfs3 got merged, it introduced iget3 and read_inode2 in the VFS layer. Later on when I started to use them for Coda I almost immediately found serious consistency problems, resulting in the iget4_locked implementation in the 2.5 kernels. I don't think anyone ever fixed that race in reiser3. It should hit occasionally on concurrent lookups on SMP or preempt kernels. In 2.4, Coda needed a semaphore to prevent concurrency during the iget3 lookup until the new inode is actually initialized. > Its got perfomance. Its there. It can emulate "conventional > filesystem" behaviour, for legacy apps. So does ramfs. > Yes, I think it would be nice to have this infrastructure in VFS. > Technically. But its not possible, cause of "committee clusterfuck". > Socially. Stupidly. It will get there, just consider it constructive criticism. If we can help resolve or refute the technical issues, all the better. We might even end up with improvements or extensions to the VFS or MM making life easier for everyone. Jan ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 16:11 ` Jan Harkes @ 2004-08-28 16:26 ` Nikita Danilov 2004-08-28 19:41 ` Alexander Lyamin 1 sibling, 0 replies; 1392+ messages in thread From: Nikita Danilov @ 2004-08-28 16:26 UTC (permalink / raw) To: Jan Harkes Cc: flx, Christophe Saout, Christoph Hellwig, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Jan Harkes writes: > On Sat, Aug 28, 2004 at 03:18:07PM +0400, Alexander Lyamin wrote: > > And I honestly dont understand whats the other Christoph's worries are about. > [...] > - When reiserfs3 got merged, it introduced iget3 and read_inode2 in the > VFS layer. Later on when I started to use them for Coda I almost > immediately found serious consistency problems, resulting in the > iget4_locked implementation in the 2.5 kernels. > > I don't think anyone ever fixed that race in reiser3. It should hit Err... it was fixed. > Jan > Nikita. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 16:11 ` Jan Harkes 2004-08-28 16:26 ` Nikita Danilov @ 2004-08-28 19:41 ` Alexander Lyamin 1 sibling, 0 replies; 1392+ messages in thread From: Alexander Lyamin @ 2004-08-28 19:41 UTC (permalink / raw) To: flx, Christophe Saout, Christoph Hellwig, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Sat, Aug 28, 2004 at 12:11:14PM -0400, Jan Harkes wrote: > On Sat, Aug 28, 2004 at 03:18:07PM +0400, Alexander Lyamin wrote: > > It will get there, just consider it constructive criticism. If we can > help resolve or refute the technical issues, all the better. We might > even end up with improvements or extensions to the VFS or MM making life > easier for everyone. That would be nice. Some nice points too (metas/metas in just noice tough). -- "the liberation loophole will make it clear.." lex lyamin ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-26 13:24 ` Christoph Hellwig 2004-08-26 13:35 ` Christophe Saout @ 2004-08-28 10:59 ` Alexander Lyamin 2004-08-28 11:12 ` Christoph Hellwig ` (2 more replies) 1 sibling, 3 replies; 1392+ messages in thread From: Alexander Lyamin @ 2004-08-28 10:59 UTC (permalink / raw) To: Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Thu, Aug 26, 2004 at 03:24:39PM +0200, Christoph Hellwig wrote: > This VFS interface is an integral part of Фvery filesystem, and it VFS never was "an integral part" of ANY filesystem. my dog knows it. its just unified INTERFACE TO any filesystem (including reiser4). P.S. I imagine, how much flamed it would be if reiser4 made any intensive changes in linux VFS code... -- "the liberation loophole will make it clear.." lex lyamin ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 10:59 ` Alexander Lyamin @ 2004-08-28 11:12 ` Christoph Hellwig 2004-08-28 12:05 ` Alexander Lyamin 2004-08-28 11:14 ` reiser4 plugins (was: silent semantic changes with reiser4) Markus Törnqvist 2004-08-28 17:18 ` Linus Torvalds 2 siblings, 1 reply; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-28 11:12 UTC (permalink / raw) To: flx, Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Sat, Aug 28, 2004 at 02:59:29PM +0400, Alexander Lyamin wrote: > Thu, Aug 26, 2004 at 03:24:39PM +0200, Christoph Hellwig wrote: > > This VFS interface is an integral part of ??very filesystem, and it > > VFS never was "an integral part" of ANY filesystem. my dog knows it. > its just unified INTERFACE TO any filesystem (including reiser4). You's misquoting me. IF you quoted the whole context it'd be pretty sure that the part of the filesystem that intefaces with the VFS is meant. But one could even say VFS is integral part of a linux filesystem as it does most of the work a filesystem driver does in other operating systems. > P.S. I imagine, how much flamed it would be if reiser4 made any intensive > changes in linux VFS code... It really depends on how you sent them. If you had a big patch without explanations - sure. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 11:12 ` Christoph Hellwig @ 2004-08-28 12:05 ` Alexander Lyamin 2004-08-28 13:56 ` Christoph Hellwig 0 siblings, 1 reply; 1392+ messages in thread From: Alexander Lyamin @ 2004-08-28 12:05 UTC (permalink / raw) To: Christoph Hellwig, flx, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Sat, Aug 28, 2004 at 01:12:33PM +0200, Christoph Hellwig wrote: > On Sat, Aug 28, 2004 at 02:59:29PM +0400, Alexander Lyamin wrote: > > Thu, Aug 26, 2004 at 03:24:39PM +0200, Christoph Hellwig wrote: > > > This VFS interface is an integral part of ??very filesystem, and it > > > > VFS never was "an integral part" of ANY filesystem. my dog knows it. > > its just unified INTERFACE TO any filesystem (including reiser4). > > You's misquoting me. IF you quoted the whole context it'd be pretty > sure that the part of the filesystem that intefaces with the VFS is > meant. No. Its not me "misquoting", its just someone sound plain ?incoherrent?. Even if I overquoted reply with whole message, its still sound ?incohherent?. > > But one could even say VFS is integral part of a linux filesystem as > it does most of the work a filesystem driver does in other operating > systems. theres no "linux filesystem". there are "linux filesystems". thanks god. But I it would be really grate if you'll elaborate your sentence with example of VFS functionality (lack of it) on said "other operating systems" and if you'll define "most of work". > > > P.S. I imagine, how much flamed it would be if reiser4 made any intensive > > changes in linux VFS code... > > It really depends on how you sent them. If you had a big patch without > explanations - sure. It would work with small tweaks, but you just can take a look at reiser4 code and you'll understand that it just could not be chopped in "set of small patches" altough it could be documented better ofcourse, but its really well commented already. some times, some approaches to some problems just would not work. -- "the liberation loophole will make it clear.." lex lyamin ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 12:05 ` Alexander Lyamin @ 2004-08-28 13:56 ` Christoph Hellwig 2004-08-28 19:23 ` Alexander Lyamin 0 siblings, 1 reply; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-28 13:56 UTC (permalink / raw) To: flx, Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Sat, Aug 28, 2004 at 04:05:02PM +0400, Alexander Lyamin wrote: > > But one could even say VFS is integral part of a linux filesystem as > > it does most of the work a filesystem driver does in other operating > > systems. > > theres no "linux filesystem". there are "linux filesystems". > thanks god. a linux filesystem, not the linux filesystem, please read again. > But I it would be really grate if you'll elaborate your sentence with > example of VFS functionality (lack of it) on said "other operating systems" > and if you'll define "most of work". most trivial example is namespace locking, in *BSD, Windows, SVR4 and derivates it's done in the lowlevel filesystem. In plan9, Linux and soon DragonlyBSD it's done in the VFS. > > > P.S. I imagine, how much flamed it would be if reiser4 made any intensive > > > changes in linux VFS code... > > > > It really depends on how you sent them. If you had a big patch without > > explanations - sure. > It would work with small tweaks, but you just can take a look at reiser4 > code and you'll understand that it just could not be chopped in > "set of small patches" altough it could be documented better ofcourse, > but its really well commented already. > > some times, some approaches to some problems just would not work. You still haven't even bother explaining what you want to do. It's hard to argue against vague uncertainity. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 13:56 ` Christoph Hellwig @ 2004-08-28 19:23 ` Alexander Lyamin 2004-08-28 19:27 ` Alexander Lyamin 2004-08-28 22:36 ` reiser4 plugins Hans Reiser 0 siblings, 2 replies; 1392+ messages in thread From: Alexander Lyamin @ 2004-08-28 19:23 UTC (permalink / raw) To: Christoph Hellwig, flx, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Sat, Aug 28, 2004 at 03:56:55PM +0200, Christoph Hellwig wrote: > On Sat, Aug 28, 2004 at 04:05:02PM +0400, Alexander Lyamin wrote: > > > But one could even say VFS is integral part of a linux filesystem as > > > it does most of the work a filesystem driver does in other operating > > > systems. > > > > theres no "linux filesystem". there are "linux filesystems". > > thanks god. > > a linux filesystem, not the linux filesystem, please read again. > > > But I it would be really grate if you'll elaborate your sentence with > > example of VFS functionality (lack of it) on said "other operating systems" > > and if you'll define "most of work". > > most trivial example is namespace locking, in *BSD, Windows, SVR4 and > derivates it's done in the lowlevel filesystem. In plan9, Linux and > soon DragonlyBSD it's done in the VFS. ok. good examples. > > > > > P.S. I imagine, how much flamed it would be if reiser4 made any intensive > > > > changes in linux VFS code... > > > > > > It really depends on how you sent them. If you had a big patch without > > > explanations - sure. > > It would work with small tweaks, but you just can take a look at reiser4 > > code and you'll understand that it just could not be chopped in > > "set of small patches" altough it could be documented better ofcourse, > > but its really well commented already. > > > > some times, some approaches to some problems just would not work. > > You still haven't even bother explaining what you want to do. It's hard > to argue against vague uncertainity. o files as directories - no oppinion on that. o metafiles - AFAIK it was product of Nikita Danilov just playing and fooling. Probably solution that will satisfy you is to have "legacy (dumb?) mode" which is leaving all this fancy stuff out of sight. -- "the liberation loophole will make it clear.." lex lyamin ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 19:23 ` Alexander Lyamin @ 2004-08-28 19:27 ` Alexander Lyamin 2004-08-28 22:36 ` reiser4 plugins Hans Reiser 1 sibling, 0 replies; 1392+ messages in thread From: Alexander Lyamin @ 2004-08-28 19:27 UTC (permalink / raw) To: flx, Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Sat, Aug 28, 2004 at 11:23:50PM +0400, Alexander Lyamin wrote: > Sat, Aug 28, 2004 at 03:56:55PM +0200, Christoph Hellwig wrote: > > > > > > some times, some approaches to some problems just would not work. > > > > You still haven't even bother explaining what you want to do. It's hard > > to argue against vague uncertainity. > > o files as directories - no oppinion on that. > > o metafiles - AFAIK it was product of Nikita Danilov just playing and fooling. > > Probably solution that will satisfy you is to have "legacy (dumb?) mode" which > is leaving all this fancy stuff out of sight. and enabled compile time or mount option... -- "the liberation loophole will make it clear.." lex lyamin ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2004-08-28 19:23 ` Alexander Lyamin 2004-08-28 19:27 ` Alexander Lyamin @ 2004-08-28 22:36 ` Hans Reiser 1 sibling, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-28 22:36 UTC (permalink / raw) To: flx Cc: Christoph Hellwig, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Alexander Lyamin wrote: > > > o metafiles - AFAIK it was product of Nikita Danilov just playing and fooling. > > No, it was not, it was an idea I had for so long I cannot remember when, maybe 1984, and I proposed it to darpa, and then nikita got told about it probably from being told to read the web page on our darpa project, and he took on the task of coding it. Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 10:59 ` Alexander Lyamin 2004-08-28 11:12 ` Christoph Hellwig @ 2004-08-28 11:14 ` Markus Törnqvist 2004-08-28 17:18 ` Linus Torvalds 2 siblings, 0 replies; 1392+ messages in thread From: Markus Törnqvist @ 2004-08-28 11:14 UTC (permalink / raw) To: flx, Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Sat, Aug 28, 2004 at 02:59:29PM +0400, Alexander Lyamin wrote: >P.S. I imagine, how much flamed it would be if reiser4 made any intensive >changes in linux VFS code... Surely it would be flamefest galore, but thanks to Reiser4 there may be some VFS changes to that direction, maybe completely from Reiser4. Should keep some people at bay if the chances weren't so much replacing VFS than extending it. -- mjt ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 10:59 ` Alexander Lyamin 2004-08-28 11:12 ` Christoph Hellwig 2004-08-28 11:14 ` reiser4 plugins (was: silent semantic changes with reiser4) Markus Törnqvist @ 2004-08-28 17:18 ` Linus Torvalds 2004-08-28 19:03 ` Alexander Lyamin 2 siblings, 1 reply; 1392+ messages in thread From: Linus Torvalds @ 2004-08-28 17:18 UTC (permalink / raw) To: Alexander Lyamin Cc: Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, reiserfs-list On Sat, 28 Aug 2004, Alexander Lyamin wrote: > > VFS never was "an integral part" of ANY filesystem. my dog knows it. That's not really true. Name handling (dentry layer, mounting) is very much an integral part of the filesystem. Almost everything else in the VFS is "helper functions", ie a filesystem can choose to ignore buffer heads, page cache etc, but a filesystem really cannot ignore or override the VFS naming stuff. (Arguably the page cache isn't part of the VFS layer at all, it's really a memory management thing, although it's so intertwined with the VFS helper functions that you can't really draw the line). > its just unified INTERFACE TO any filesystem (including reiser4). True, but that's not the whole truth. It's way more than just an interface. It's a set of rules, and it's in many way the controlling party. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 17:18 ` Linus Torvalds @ 2004-08-28 19:03 ` Alexander Lyamin 2004-08-28 19:09 ` Christoph Hellwig 2004-08-28 19:26 ` reiser4 plugins (was: silent semantic changes with reiser4) Linus Torvalds 0 siblings, 2 replies; 1392+ messages in thread From: Alexander Lyamin @ 2004-08-28 19:03 UTC (permalink / raw) To: Linus Torvalds Cc: Alexander Lyamin, Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, reiserfs-list Sat, Aug 28, 2004 at 10:18:44AM -0700, Linus Torvalds wrote: > > > On Sat, 28 Aug 2004, Alexander Lyamin wrote: > > > > VFS never was "an integral part" of ANY filesystem. my dog knows it. > > That's not really true. Theres some truth. And its really arguably, but I think, I see your point. And taking on the situation with reiser4 with "smartmost VFS" approach, time should pass and with time and expirience on hands it will be evident what should go in VFS. Now its may be too early. Considering "amazing PR skills" of Hans Reiser it was the only viable way to get this changes in VFS. Cause, ironically, mr. Hellwig that currently demand it to be scrapped out or go in VFS would instakill Hans Reiser (i know many people would:) if he only touched holy cow of VFS for any reiser4 purpose. As I told before, some technically right things just do not work in this imperfect world. You have to count with social issues, and Hans Reiser wont, he just avoid them whenever its possible. > Name handling (dentry layer, mounting) is very much an integral part of > the filesystem. Almost everything else in the VFS is "helper functions", > ie a filesystem can choose to ignore buffer heads, page cache etc, but a > filesystem really cannot ignore or override the VFS naming stuff. > > (Arguably the page cache isn't part of the VFS layer at all, it's really a > memory management thing, although it's so intertwined with the VFS helper > functions that you can't really draw the line). > > > its just unified INTERFACE TO any filesystem (including reiser4). > > True, but that's not the whole truth. It's way more than just an > interface. It's a set of rules, and it's in many way the controlling > party. > > Linus "True, but thats not the whole truth." -- "the liberation loophole will make it clear.." lex lyamin ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 19:03 ` Alexander Lyamin @ 2004-08-28 19:09 ` Christoph Hellwig 2004-08-28 21:41 ` reiser4 plugins Hans Reiser 2004-08-28 19:26 ` reiser4 plugins (was: silent semantic changes with reiser4) Linus Torvalds 1 sibling, 1 reply; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-28 19:09 UTC (permalink / raw) To: flx, Linus Torvalds, Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, reiserfs-list On Sat, Aug 28, 2004 at 11:03:50PM +0400, Alexander Lyamin wrote: > And taking on the situation with reiser4 with "smartmost VFS" approach, > time should pass and with time and expirience on hands it will be evident what > should go in VFS. Now its may be too early. > > Considering "amazing PR skills" of Hans Reiser it was the only viable way > to get this changes in VFS. Cause, ironically, mr. Hellwig that currently > demand it to be scrapped out or go in VFS would instakill Hans Reiser > (i know many people would:) if he only touched holy cow of VFS for any > reiser4 purpose. Could you please stop talking such inproven crap? ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2004-08-28 19:09 ` Christoph Hellwig @ 2004-08-28 21:41 ` Hans Reiser 2004-08-30 16:02 ` Herbert Poetzl 0 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-28 21:41 UTC (permalink / raw) To: Linus Torvalds Cc: Christoph Hellwig, flx, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, reiserfs-list I think it is reasonable to make the -nopseudos (turns off the metafiles ) mount option mandatory, until the bugs are resolved. Our testing did not find these metafile/VFS bugs because of the reason for all our bugs, we screwed up. There is a distinct difference between some persons and I, which is that some think all of reiser4 should be excluded until metafiles are implemented by VFS some long time from now, and I, in that I merely think buggy optional features should be turned off until they are fixed. I, being renowned for my paranoia and asininity as I am, think these persons find it convenient as an excuse to keep us from competing, and I think that if we were slower there would be less hassle every time we try to get into the kernel. While reiser4 has some significant roughnesses remaining in its performance, I think the average user would find it performs better than other filesystems, and is stable enough for, say, a laptop, and I predict that by the time we have it stable enough for mission critical servers, all the roughness in various important corner cases will be gone. Persons benchmarking it with tarballs, please be sure to use tarballs created on reiser4, not ext2 tarballs, readdir order matters a lot for sorted directory filesystems. Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2004-08-28 21:41 ` reiser4 plugins Hans Reiser @ 2004-08-30 16:02 ` Herbert Poetzl 2004-08-30 18:55 ` Hans Reiser 0 siblings, 1 reply; 1392+ messages in thread From: Herbert Poetzl @ 2004-08-30 16:02 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, Christoph Hellwig, flx, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, reiserfs-list On Sat, Aug 28, 2004 at 02:41:12PM -0700, Hans Reiser wrote: > I think it is reasonable to make the -nopseudos (turns off the metafiles > ) mount option mandatory, until the bugs are resolved. > > Our testing did not find these metafile/VFS bugs because of the reason > for all our bugs, we screwed up. > > There is a distinct difference between some persons and I, which is that > some think all of reiser4 should be excluded until metafiles are > implemented by VFS some long time from now, and I, in that I merely > think buggy optional features should be turned off until they are > fixed. I, being renowned for my paranoia and asininity as I am, think > these persons find it convenient as an excuse to keep us from competing, > and I think that if we were slower there would be less hassle every time > we try to get into the kernel. > > While reiser4 has some significant roughnesses remaining in its > performance, I think the average user would find it performs better than > other filesystems, and is stable enough for, say, a laptop, and I > predict that by the time we have it stable enough for mission critical > servers, all the roughness in various important corner cases will be > gone. > > Persons benchmarking it with tarballs, please be sure to use tarballs > created on reiser4, not ext2 tarballs, readdir order matters a lot for > sorted directory filesystems. hmm, so probably we have to wait until all tar packagers moved to reiser4, so that the available tar files are 'sorted properly' ... best, Herbert > Hans > - > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2004-08-30 16:02 ` Herbert Poetzl @ 2004-08-30 18:55 ` Hans Reiser 0 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-30 18:55 UTC (permalink / raw) To: Herbert Poetzl Cc: Linus Torvalds, Christoph Hellwig, flx, Christophe Saout, Andrew Morton, linux-fsdevel, linux-kernel, flx, reiserfs-list Herbert Poetzl wrote: > >hmm, so probably we have to wait until all >tar packagers moved to reiser4, so that the >available tar files are 'sorted properly' ... > >best, >Herbert > > > Or wait for a repacker, which will probably happen sooner. Distros that use reiser4 will probably tar in reiser4 order. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins (was: silent semantic changes with reiser4) 2004-08-28 19:03 ` Alexander Lyamin 2004-08-28 19:09 ` Christoph Hellwig @ 2004-08-28 19:26 ` Linus Torvalds 1 sibling, 0 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-08-28 19:26 UTC (permalink / raw) To: Alexander Lyamin Cc: Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, reiserfs-list On Sat, 28 Aug 2004, Alexander Lyamin wrote: > > Considering "amazing PR skills" of Hans Reiser it was the only viable way > to get this changes in VFS. Cause, ironically, mr. Hellwig that currently > demand it to be scrapped out or go in VFS would instakill Hans Reiser > (i know many people would:) if he only touched holy cow of VFS for any > reiser4 purpose. Hey, you don't have to try to convince me - I haven't been arguing against the reiser4 approach. I think it's a prototyping scheme, and quite frankly I don't think you _can_ get the name lookup right without real VFS support (races, namespaces etc are just not something you can solve in the filesystem). But the fact that name lookup does seriously need VFS support means that I don't think we want to merge reiser4 as-is. No way. That doesn't mean that I think reiser4 was _wrong_. And I certainly agree that there are personality issues, and they are not all just Hans' ;) Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 13:07 ` Christoph Hellwig 2004-08-26 13:17 ` reiser4 plugins (was: silent semantic changes with reiser4) Christophe Saout @ 2004-08-26 13:27 ` Christophe Saout 2004-08-26 14:44 ` Jamie Lokier 2 siblings, 0 replies; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 13:27 UTC (permalink / raw) To: Christoph Hellwig Cc: Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1218 bytes --] Am Donnerstag, den 26.08.2004, 15:07 +0200 schrieb Christoph Hellwig: > > The reiser4 core doesn't know about inodes, directories or files. It's > > the glue code between the VFS and the storage layer that does. It's > > implemented as a plugin. This has nothing to do with semantic > > enhancements yet. These should be removed for now and made a 2.7 topic. > > Oh yes, it is. As soon as you use different access methods on an > overlapping set of objects you see exactly the problems I described. Right. That shouldn't happen. Since an object is usally tied to exactly one plugin this can't happen I think. At least as long the programmer doesn't mess anything up. > If they don't overlap there's no point for the plugins to start with, > you'll better turn the core into a library that can be used by different > projects then. I think that's what Hans is trying to suggest. The core is theoretically usable for anything. He made a filesystem of it. You can see reiser4 as a library and the UNIX directory and file plugins as the filesystem. That's why they are linked in uncoditionally because it doesn't make sense otherwise. It's mainly the naming that seems to confuse people. [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 13:07 ` Christoph Hellwig 2004-08-26 13:17 ` reiser4 plugins (was: silent semantic changes with reiser4) Christophe Saout 2004-08-26 13:27 ` silent semantic changes with reiser4 Christophe Saout @ 2004-08-26 14:44 ` Jamie Lokier 2004-08-26 16:03 ` Christoph Hellwig 2 siblings, 1 reply; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 14:44 UTC (permalink / raw) To: Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Christoph Hellwig and Christophe Saout, I think you two are miscommunicating a little. I've never looked at reiser4 code, but I think I understand what you are both saying, so I will explain. Christoph Hellwig wrote: > Here comes the problem. All the access checking/race avoidance/loop > creation avoiced, in short the posix+extension semantics are implemented > in the Linux VFS layer. If you want to allow a second access method > (e.g. Hans' pet syscall) you'd have to duplicate all VFS functioanlity > inside reiser4. That would be good. Even better would be if VFS implemented the best semantics that we agree is nice, after some experimenting with reiser4 lets us discover what works well. > > The reiser4 core doesn't know about inodes, directories or files. It's > > the glue code between the VFS and the storage layer that does. It's > > implemented as a plugin. This has nothing to do with semantic > > enhancements yet. These should be removed for now and made a 2.7 topic. > > Oh yes, it is. As soon as you use different access methods on an > overlapping set of objects you see exactly the problems I described. > > If they don't overlap there's no point for the plugins to start with, > you'll better turn the core into a library that can be used by different > projects then. Firstly, the core library _is_ used by different projects. That's the commercial aim of reiser4: to be used by projects other than just as Linux filesystem. For example it might be used by a database, or a portable application which needs to store structured data in a flat file (with random access). A good abstraction makes it more useful than "just" a filesystem. Secondly, the reiser4 design makes the "high-level plugin" interface a good clean abstraction. It's conceptually simple and versatile, so simplifies the code and makes it easier to try new ideas as well. I think the disagreement between you two comes down to the idea that a generally useful API abstraction to file-like objects really should be somewhere in the VFS, and not specific to a single filesystem. Unfortunately, the problem is that reiser4 is the only filesystem which is _technically capable_ of implementing that abstraction in a practical way, apparently. (I'm not sure if this is really true. reiser4's object model is not the same as paths and inodes, but the impedance mismatch doesn't seem huge.) Perhaps in future some other filesystems could do it. Then it would make sense for the API to be common between them. At the same time, some of the features which Hans Reiser would like to offer users -- especially things like metadata indexing -- don't seem to be very practical on top of the VFS abstraction. These are conflicting goals. Hence the argument. VFS is very POSIX, which rules out some interesting things like multiple views of an ordinary file (i.e. viewing a flat file and it's structured form together), coherent files (i.e. metadata which is calculated from the file contents and always guaranteed to match it), and real time indexing (i.e. "grep", "locate" and "local-google" but fancier, always up to date, and still fast). Those features can be implemented poorly on top of POSIX, and hence poorly on top of VFS. We do that today. It works, it's simple, but it sucks. Sometimes it goes wrong (index doesn't match files), it's slow (locate isn't real-time, and if grep kept content indexes it would still be too slow to verify them), and it's not really simple as you have to use special tools for every file format these days, each with their own unique interface. I think we can tidy that up a lot, still using the VFS, but with some extra hooks. But it's not clear what hooks those would be, and how the interesting features would be implemented by other POSIX-only filesystems. It'd be more useful: I want the fancy stuff to work over NFS to a bog standard server, or over an old ext2 filesystem that mustn't be upgraded, not just reiser4, even if I don't have the same performance and it has to create lots of articial "metadata" files to simulate the fancy stuff. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 14:44 ` Jamie Lokier @ 2004-08-26 16:03 ` Christoph Hellwig 2004-08-26 16:19 ` Jamie Lokier 0 siblings, 1 reply; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-26 16:03 UTC (permalink / raw) To: Jamie Lokier Cc: Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list > Firstly, the core library _is_ used by different projects. That's the > commercial aim of reiser4: to be used by projects other than just as > Linux filesystem. For example it might be used by a database, or a > portable application which needs to store structured data in a flat > file (with random access). A good abstraction makes it more useful > than "just" a filesystem. Which is okay as long as it still fits in the kernel. Remember that lots of projects had to adjust so far despite the commerical aims of their creators. > I think the disagreement between you two comes down to the idea that a > generally useful API abstraction to file-like objects really should be > somewhere in the VFS, and not specific to a single filesystem. IT should, and in fact it actually is most of the part. We can have differnt operation vectors per object, but reiser4 is the only filesystem in the tree not making use of that but provinding only a single one and then dispatching internally. Even XFS that carries around and horrible internal dispatching layer as IRIX legacy gets this right. > Unfortunately, the problem is that reiser4 is the only filesystem > which is _technically capable_ of implementing that abstraction in a > practical way, apparently. (I'm not sure if this is really true. > reiser4's object model is not the same as paths and inodes, but the > impedance mismatch doesn't seem huge.) Umm, no. In fact every filesystem does this. There's not too many objects with different namespace semantic: regular files and special files vs directories vs symlinks basically, but as soon as you go to the data patch you can have hundrets of different file_operations on a single filesystems (for special files). ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:03 ` Christoph Hellwig @ 2004-08-26 16:19 ` Jamie Lokier 2004-08-26 16:23 ` Christoph Hellwig 0 siblings, 1 reply; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 16:19 UTC (permalink / raw) To: Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Christoph Hellwig wrote: > > Unfortunately, the problem is that reiser4 is the only filesystem > > which is _technically capable_ of implementing that abstraction in a > > practical way, apparently. (I'm not sure if this is really true. > > reiser4's object model is not the same as paths and inodes, but the > > impedance mismatch doesn't seem huge.) > > Umm, no. In fact every filesystem does this. There's not too many > objects with different namespace semantic: regular files and special > files vs directories vs symlinks basically, but as soon as you go to the > data patch you can have hundrets of different file_operations on a > single filesystems (for special files). Remember that reiser4 allows you to operate on little pieces of data, glueing and rearranging them inside files (something like that). No other filesystem has that capability, and it's a data model which the fancy features (when they exist) will use. You can map those pieces to underlying directories and files and renames and unlinks, so that the fancy stuff works on other filesystems, but it would be a useless model because those other filesystems wouldn't be recognisably "ordinary" files any more. For reiser4 to expose that model through a VFS interface, and the fancy stuff to use it through the VFS interface, and for the fancy stuff to work (even imperfectly) on other filesystems which don't offer those operations, some kind of fall-back "store metadata and fragment rearrangements in auxiliary files with special names" layer would be requied. That's a big job. I think it's a good job to do (funding, anyone? :), and the right place for that (imperfect but useful) fall-back layer is userspace with perhaps minimal VFS support. Another reason to put the fall-back layer in userspace is that applications which depend on the fancy stuff can still be portable to other OSes, working, but a bit smoother, faster and more "integrated" on reiser4. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:19 ` Jamie Lokier @ 2004-08-26 16:23 ` Christoph Hellwig 2004-08-26 17:08 ` Jamie Lokier 0 siblings, 1 reply; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-26 16:23 UTC (permalink / raw) To: Jamie Lokier Cc: Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, Aug 26, 2004 at 05:19:11PM +0100, Jamie Lokier wrote: > Remember that reiser4 allows you to operate on little pieces of data, > glueing and rearranging them inside files (something like that). > > No other filesystem has that capability, and it's a data model which > the fancy features (when they exist) will use. > > You can map those pieces to underlying directories and files and > renames and unlinks, so that the fancy stuff works on other > filesystems, but it would be a useless model because those other > filesystems wouldn't be recognisably "ordinary" files any more. > > For reiser4 to expose that model through a VFS interface, and the > fancy stuff to use it through the VFS interface, and for the fancy > stuff to work (even imperfectly) on other filesystems which don't > offer those operations, some kind of fall-back "store metadata and > fragment rearrangements in auxiliary files with special names" layer > would be requied. That's a big job. I seem to miss those files in the current implementation. But again the big question is how will the interface look? If the smaller objects interact with normal files (link/rename, contained in normal directories), they will have to obey the same locking protocols. If not a filesystem could experiment with it of course. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:23 ` Christoph Hellwig @ 2004-08-26 17:08 ` Jamie Lokier 0 siblings, 0 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 17:08 UTC (permalink / raw) To: Christoph Hellwig, Christophe Saout, Andrew Morton, Hans Reiser, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Christoph Hellwig wrote: > If the smaller objects interact with normal files (link/rename, > contained in normal directories), they will have to obey the same > locking protocols. If not a filesystem could experiment with it of > course. I think the smaller objects will consist of special files, which are magic attributes like the ones you currently see in reiser4, and ordinary files (which might themselves be containers). Link/rename/locking etc. should work as usual on the ordinary smaller files, but probably aren't allowed at all on the special attribute files, except where there's an appropriate meaning. (For example locking certain attributes might have an appropriate meaning). It would make sense for the special attributes file to not identify as "regular files" (S_IFREG), but there aren't many mode bits to play with. They should also be harder to find than ordinary smaller files. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 12:18 ` Christophe Saout 2004-08-26 12:49 ` Christoph Hellwig @ 2004-08-26 23:54 ` Hans Reiser 1 sibling, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-26 23:54 UTC (permalink / raw) To: Christophe Saout Cc: Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Christophe Saout wrote: >Am Donnerstag, den 26.08.2004, 01:45 -0700 schrieb Andrew Morton: > > > >>And describe the "plugin" system. Why does the filesystem need such a >>thing (other filesystems get their features via `patch -p1')? >> >> > >Ok, let me try. (Hans, please don't shoot me if I don't get every detail >right, I'm trying to simplify and translate) > >The reiser4 core is just a big and fast storage layer using a single >tree. It is bound to a mount point and uses the page cache to manage its >memory. The only thing it can do on its own is to flush dirty data to >the disk when the VM wants to (memory pressure) or the VFS wants to >(unmount, sync) or some "plugin" wants to. >Additionally it's completely atomic (with respect to crashes, no >isolation like in databases), comparable to data=journal but without the >overhead of using a journal for everything. > >Up to this point there are no users of this "database" and it does not >implement any of the VFS methods for a filesystem (except mount and >unmount perhaps). > >All the functionality is provided to > ^to^via apologies for the correction but it might confuse some > the plugins. You can insert, >remove, lookup or modify key:object pairs (with an index into the >object). The object > ^object^item perform that substitution everywhere in this email > is a sequence of units. For files a unit would be 1 >byte and the index would be the byte offset. For directories the unit >would be 1 entry and the index would be the filename. > >Now there are some plugins that define how the storage layout on the >disk is (some kind of "backend" plugins). > >*And* there are plugins which are users of the "reiser4 client API" and >implement the actual VFS methods. > >There's a UNIX directory plugin and a UNIX file plugin. > >Directories, inodes and file content are just key:object pairs and the >plugins know how to operate on these. > >There's a new plugin in work that also implements UNIX file semantics >but stores the data for that file encrypted and/or compressed. > >These plugins live between the VFS and the storage layer. > >Just like the filesystems live between the VFS and the block layer (Hans >would say that filesystems are VFS plugins ;-)). > > > >>And what are the licensing implications of plugins? Are they derived >>works? Must they be GPL'ed? >> >> > >I suppose yes, at least currently, since they can only be linked with >reiser4, there's no module infrastructure. > > > ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:31 ` Hans Reiser 2004-08-26 8:45 ` Andrew Morton @ 2004-08-26 9:34 ` Christoph Hellwig 2004-08-26 9:37 ` Markus Törnqvist 2004-08-26 18:40 ` Thomas Cataldo 2004-08-26 16:32 ` Will Dyson ` (3 subsequent siblings) 5 siblings, 2 replies; 1392+ messages in thread From: Christoph Hellwig @ 2004-08-26 9:34 UTC (permalink / raw) To: Hans Reiser Cc: Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, Aug 26, 2004 at 01:31:34AM -0700, Hans Reiser wrote: > Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem > for Apple, and that means we need to put search engine and database Dou you know a nice thing? We (as in the Linux Community) don't have to compete with anyone. Sure, we're usually trying to be better than anyone else, but unlike companies under maret pressure we can wait until something is ready. Again, if you think you want your work in Linux cooperate with the Linux Developers, and as you have seen in this thread there's a pretty broad consensus that we want things working at the VFS (and actually working of course) ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 9:34 ` Christoph Hellwig @ 2004-08-26 9:37 ` Markus Törnqvist 2004-08-26 18:40 ` Thomas Cataldo 1 sibling, 0 replies; 1392+ messages in thread From: Markus Törnqvist @ 2004-08-26 9:37 UTC (permalink / raw) To: Christoph Hellwig, Hans Reiser, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, Aug 26, 2004 at 11:34:07AM +0200, Christoph Hellwig wrote: > >Dou you know a nice thing? We (as in the Linux Community) don't have to >compete with anyone. Sure, we're usually trying to be better than >anyone else, but unlike companies under maret pressure we can wait until >something is ready. It won't get ready unless it's made ready, yes? And what about the time when people will move away from Linux because Linux has nothing but crappy file systems to offer? Sure, it's not that we'd have to compete, but... >Again, if you think you want your work in Linux cooperate with the Linux >Developers, and as you have seen in this thread there's a pretty broad >consensus that we want things working at the VFS (and actually working >of course) So move it into VFS. Neext! ;) -- mjt ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 9:34 ` Christoph Hellwig 2004-08-26 9:37 ` Markus Törnqvist @ 2004-08-26 18:40 ` Thomas Cataldo 1 sibling, 0 replies; 1392+ messages in thread From: Thomas Cataldo @ 2004-08-26 18:40 UTC (permalink / raw) To: Christoph Hellwig Cc: Hans Reiser, Andrew Morton, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, 2004-08-26 at 11:34 +0200, Christoph Hellwig wrote: > On Thu, Aug 26, 2004 at 01:31:34AM -0700, Hans Reiser wrote: > > Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem > > for Apple, and that means we need to put search engine and database > > Dou you know a nice thing? We (as in the Linux Community) don't have to > compete with anyone. Sure, we're usually trying to be better than > anyone else, but unlike companies under maret pressure we can wait until > something is ready. By the way, this kind of features can be achieved easily in userspace with, for example, ext3, beagle (a userspace fulltext indexing daemon) and some (d|i)notify magic. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:31 ` Hans Reiser 2004-08-26 8:45 ` Andrew Morton 2004-08-26 9:34 ` Christoph Hellwig @ 2004-08-26 16:32 ` Will Dyson 2004-08-26 16:44 ` Christophe Saout 2004-08-27 8:08 ` Hans Reiser 2004-08-26 18:12 ` silent semantic changes with reiser4 Horst von Brand ` (2 subsequent siblings) 5 siblings, 2 replies; 1392+ messages in thread From: Will Dyson @ 2004-08-26 16:32 UTC (permalink / raw) To: Hans Reiser Cc: Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Hans Reiser wrote: > Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem > for Apple, and that means we need to put search engine and database > functionality into the filesystem. It takes 11 years of serious Hans, I'm very curious about your ideas on how to put search engine and database functionality into the filesystem. Although I've let the Befs filesystem driver largely stagnate over the past two years, from time to time I think about the problem of exporting to userspace the attribute indexes that BeFS keeps on disk. In the original BeOS, they solved the problem by having the filesystem driver itself take a text query string and parse it, returning a list of inodes that match. The whole business of parsing a query string in the kernel (let alone in the filesystem driver) has always seemed ugly to me. However, the best alternative I've come up with is to simply export the index data as a special file (perhaps in sysfs?) and have userspace responsible for searching the index. That would probably work, but it wouldn't help other filesystems that implement even a different index format, much less a different form of extra searchability. -- Will Dyson "Back off man, I'm a scientist!" -Dr. Peter Venkman ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:32 ` Will Dyson @ 2004-08-26 16:44 ` Christophe Saout 2004-08-26 17:22 ` Jamie Lokier 2004-08-27 8:08 ` Hans Reiser 1 sibling, 1 reply; 1392+ messages in thread From: Christophe Saout @ 2004-08-26 16:44 UTC (permalink / raw) To: Will Dyson Cc: Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 1007 bytes --] Am Donnerstag, den 26.08.2004, 12:32 -0400 schrieb Will Dyson: > > Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem > > for Apple, and that means we need to put search engine and database > > functionality into the filesystem. It takes 11 years of serious > > I'm very curious about your ideas on how to put search engine and > database functionality into the filesystem. I could imagine something like this. There are some special folders or files or whatever under every file where some userspace can put some metadata. A keyword perhaps. echo linux > file.bla/keywords/topic The filesystem might then automatically put these keywords into an index and then provide a search mechanism elsewhere where it could ask "find me all dentries with the keyword 'linux'." and it would return a list like locate does. Only that it's in realtime and also works when moving the file around (but not when copying with an unaware program for obvious reasons). [-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:44 ` Christophe Saout @ 2004-08-26 17:22 ` Jamie Lokier 0 siblings, 0 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-08-26 17:22 UTC (permalink / raw) To: Christophe Saout Cc: Will Dyson, Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Christophe Saout wrote: > echo linux > file.bla/keywords/topic > > The filesystem might then automatically put these keywords into an index > and then provide a search mechanism elsewhere where it could ask "find > me all dentries with the keyword 'linux'." and it would return a list > like locate does. Only that it's in realtime and also works when moving > the file around (but not when copying with an unaware program for > obvious reasons). It could work even when copying with an unaware program - provided the're a tool which knows to extract metadata from files where that hasn't been done already, or where the file's been modified. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 16:32 ` Will Dyson 2004-08-26 16:44 ` Christophe Saout @ 2004-08-27 8:08 ` Hans Reiser 2004-08-27 18:20 ` Will Dyson 1 sibling, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-27 8:08 UTC (permalink / raw) To: Will Dyson Cc: Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Will Dyson wrote: > > > In the original BeOS, they solved the problem by having the filesystem > driver itself take a text query string and parse it, returning a list > of inodes that match. The whole business of parsing a query string in > the kernel (let alone in the filesystem driver) has always seemed ugly > to me. Why? > However, the best alternative I've come up with is to simply export > the index data as a special file (perhaps in sysfs?) and have > userspace responsible for searching the index. That is the best implementation suggestion I've heard for splitting the filesystem into two parts, one in user space and one in kernel, but I still don't trust it to work well. > That would probably work, but it wouldn't help other filesystems that > implement even a different index format, much less a different form of > extra searchability. > After reiser6 is implemented, we'll have a better idea of what parts need what from what other parts. Until we get there, it needs to be kept together. Frankly, I suspect that naming is not going to split so easily, but when it is done we'll be able to know. The problems of putting too much functionality into the kernel are less than those of partitioning the filesystem code, and defining an inflexible API between the two parts. Kernel interfaces are fairly inflexible because there will always be some moron who sidesteps the fs library to program directly to the kernel interface, gets screwed when the interface changes, has the nerve to complain about it, and works for some powerful distro. Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 8:08 ` Hans Reiser @ 2004-08-27 18:20 ` Will Dyson 2004-08-27 18:38 ` David Ford ` (2 more replies) 0 siblings, 3 replies; 1392+ messages in thread From: Will Dyson @ 2004-08-27 18:20 UTC (permalink / raw) To: Hans Reiser Cc: Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Hans Reiser wrote: > Will Dyson wrote: >> >> In the original BeOS, they solved the problem by having the filesystem >> driver itself take a text query string and parse it, returning a list >> of inodes that match. The whole business of parsing a query string in >> the kernel (let alone in the filesystem driver) has always seemed ugly >> to me. > > Why? Hmm. Trying to explain aesthetic judgments is always fun, but I'll try. String parsing bloats the kernel with code that will be called rarely, and doing it inside the filesystem module makes for duplicate code if more than one filesystem does it. But a good common parser routine (or a kernel api that takes a pre-compiled parse tree) would reduce the bad taste. The real objection I have is that it puts the kernel in the position of doing more work than it has to for the system to operate correctly and efficiently. The user wants to know what files match a certain set of criteria. The filesystem provides special features which can greatly accelerate some searches. Does it make more sense to move the functionality of /usr/bin/find into the kernel, or to export the index information so that an enhanced version of find can make use of it? I don't think having find in the kernel would be worth the cost or complexity. However, now that I've had to think about it while writing the above, it occurs to me that if we wanted to offer something like the beos's live queries, doing it in the kernel might be the way to go. For the 99.9% of people who were not beos users back in the day, the "live query" was a way to register a query string with the kernel, so that you got notifications when the list of files matching the query changed. It was pretty cool, but this is really just blue-skying since I am not stepping up to write such a beast any time soon. >> However, the best alternative I've come up with is to simply export >> the index data as a special file (perhaps in sysfs?) and have >> userspace responsible for searching the index. > > That is the best implementation suggestion I've heard for splitting the > filesystem into two parts, one in user space and one in kernel, but I > still don't trust it to work well. Why not? And if it really splits the filesystem into two parts depends on which of the two following statements you agree with more: "Fast searching is a feature of the filesystem" "Maintaining indexes is a feature of the filesystem" PS. The file-as-directory attribute/stream/whatever model is a much better fit for befs's internals than the xattr api. If the kernel supported it, I would enable support for it in befs. -- Will Dyson "Back off man, I'm a scientist!" -Dr. Peter Venkman ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 18:20 ` Will Dyson @ 2004-08-27 18:38 ` David Ford 2004-08-28 6:14 ` Hans Reiser 2004-08-28 9:54 ` Hans Reiser 2004-08-28 15:18 ` Separating Indexing and Searching (was silent semantic changes with reiser4) Alexander G. M. Smith 2 siblings, 1 reply; 1392+ messages in thread From: David Ford @ 2004-08-27 18:38 UTC (permalink / raw) To: Will Dyson Cc: Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list [-- Attachment #1: Type: text/plain, Size: 2800 bytes --] Will Dyson wrote: > Hans Reiser wrote: > >> Will Dyson wrote: >> >>> >>> In the original BeOS, they solved the problem by having the >>> filesystem driver itself take a text query string and parse it, >>> returning a list of inodes that match. The whole business of parsing >>> a query string in the kernel (let alone in the filesystem driver) >>> has always seemed ugly to me. >> >> >> Why? > > > Hmm. Trying to explain aesthetic judgments is always fun, but I'll try. > > String parsing bloats the kernel with code that will be called rarely, > and doing it inside the filesystem module makes for duplicate code if > more than one filesystem does it. But a good common parser routine (or > a kernel api that takes a pre-compiled parse tree) would reduce the > bad taste. Like say.. printk() ? :) > The real objection I have is that it puts the kernel in the position > of doing more work than it has to for the system to operate correctly > and efficiently. The user wants to know what files match a certain set > of criteria. The filesystem provides special features which can > greatly accelerate some searches. Does it make more sense to move the > functionality of /usr/bin/find into the kernel, or to export the index > information so that an enhanced version of find can make use of it? I think it depends largely on where you see file systems going 5, 10, 20 years from now. Are they going to be decadent designs or will they be flush with meta data and other tidbits? Where does the heuristic line sit when it becomes better to put such a function into the kernel v.s. user land? > I don't think having find in the kernel would be worth the cost or > complexity. Probably not. But as I view it, file systems are slowly evolving into relational objects and the margin between what are files and directories versus content within said objects is shrinking. More and more, people are using a relational database to store content that once resided in singular files. Sure, it makes for a much fatter *FS system, but the user doesn't care too much, he simply enjoys the ability to manage, search, and cross reference his content much more easily. And then comes the "it's a user land thing" comment again. And again the response. When does moving this from user land into kernel land become a better choice. With the file systems we have today, it's really not much of a gain compared to the cost of the move. Consider Reiser4 now. More and more meta data is being used in regards to files. That margin is shrinking. BeOS broached it, Microsoft is building it. Just because they did it doesn't mean we have to do it nor that it is right. But it's food for thought. Especially as more and more tidbits get associated with files. -david [-- Attachment #2: david+challenge-response.vcf --] [-- Type: text/x-vcard, Size: 183 bytes --] begin:vcard fn:David Ford n:Ford;David email;internet:david@blue-labs.org title:Industrial Geek tel;home:Ask please tel;cell:(203) 650-3611 x-mozilla-html:TRUE version:2.1 end:vcard ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 18:38 ` David Ford @ 2004-08-28 6:14 ` Hans Reiser 0 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-28 6:14 UTC (permalink / raw) To: David Ford Cc: Will Dyson, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list David Ford wrote: > Will Dyson wrote: > >> Hans Reiser wrote: >> >>> Will Dyson wrote: >>> >> >> >> String parsing bloats the kernel with code that will be called >> rarely, and doing it inside the filesystem module makes for duplicate >> code if more than one filesystem does it. But a good common parser >> routine (or a kernel api that takes a pre-compiled parse tree) would >> reduce the bad taste. > > > > Like say.. printk() ? :) > or namei()....;-) Seriously, what we are doing is making namei() more powerful. That is all. Currently namei() is very simple. Evolution happens. Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 18:20 ` Will Dyson 2004-08-27 18:38 ` David Ford @ 2004-08-28 9:54 ` Hans Reiser 2004-08-31 3:14 ` Will Dyson 2004-08-28 15:18 ` Separating Indexing and Searching (was silent semantic changes with reiser4) Alexander G. M. Smith 2 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-28 9:54 UTC (permalink / raw) To: Will Dyson Cc: Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list I think there are two ways to analyze the code boundary issue. One is "does it belong in the kernel?" Another is, "does it belong in the filesystem. and if so should name resolution in a filesystem be split into two parts, one in kernel, and one in user space." In ten years I might have the knowledge needed to make such a split, but I know for sure that I don't know how to do it today without regretting it tomorrow, and I don't really have confidence that I will ever be able to do it without losing performance. Glad that BeFS finds the new model better.:) Hans Will Dyson wrote: > Hans Reiser wrote: > >> Will Dyson wrote: >> >>> >>> In the original BeOS, they solved the problem by having the >>> filesystem driver itself take a text query string and parse it, >>> returning a list of inodes that match. The whole business of parsing >>> a query string in the kernel (let alone in the filesystem driver) >>> has always seemed ugly to me. >> >> >> Why? > > > Hmm. Trying to explain aesthetic judgments is always fun, but I'll try. > > String parsing bloats the kernel with code that will be called rarely, > and doing it inside the filesystem module makes for duplicate code if > more than one filesystem does it. But a good common parser routine (or > a kernel api that takes a pre-compiled parse tree) would reduce the > bad taste. > > The real objection I have is that it puts the kernel in the position > of doing more work than it has to for the system to operate correctly > and efficiently. The user wants to know what files match a certain set > of criteria. The filesystem provides special features which can > greatly accelerate some searches. Does it make more sense to move the > functionality of /usr/bin/find into the kernel, or to export the index > information so that an enhanced version of find can make use of it? > > I don't think having find in the kernel would be worth the cost or > complexity. > > However, now that I've had to think about it while writing the above, > it occurs to me that if we wanted to offer something like the beos's > live queries, doing it in the kernel might be the way to go. > > For the 99.9% of people who were not beos users back in the day, the > "live query" was a way to register a query string with the kernel, so > that you got notifications when the list of files matching the query > changed. It was pretty cool, but this is really just blue-skying since > I am not stepping up to write such a beast any time soon. > >>> However, the best alternative I've come up with is to simply export >>> the index data as a special file (perhaps in sysfs?) and have >>> userspace responsible for searching the index. >> >> >> That is the best implementation suggestion I've heard for splitting >> the filesystem into two parts, one in user space and one in kernel, >> but I still don't trust it to work well. > > > Why not? And if it really splits the filesystem into two parts depends > on which of the two following statements you agree with more: > > "Fast searching is a feature of the filesystem" > "Maintaining indexes is a feature of the filesystem" > > > PS. > The file-as-directory attribute/stream/whatever model is a much better > fit for befs's internals than the xattr api. If the kernel supported > it, I would enable support for it in befs. > ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 9:54 ` Hans Reiser @ 2004-08-31 3:14 ` Will Dyson 0 siblings, 0 replies; 1392+ messages in thread From: Will Dyson @ 2004-08-31 3:14 UTC (permalink / raw) To: Hans Reiser Cc: Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Hans Reiser wrote: > I think there are two ways to analyze the code boundary issue. One is > "does it belong in the kernel?" Another is, "does it belong in the > filesystem. and if so should name resolution in a filesystem be split > into two parts, one in kernel, and one in user space." In ten years I > might have the knowledge needed to make such a split, but I know for > sure that I don't know how to do it today without regretting it > tomorrow, and I don't really have confidence that I will ever be able to > do it without losing performance. I don't see how exporting a set of indices on file attributes to userspace constitutes putting part of the name resolution into userspace. A file's name (or names, in the case of hardlinks) would still be determined entirely within the filesystem. The more I think about it, the more I am convinced that "the index" is the correct primative to use for exposing any filesystem's fast searching features. -- Will Dyson "Back off man, I'm a scientist!" -Dr. Peter Venkman ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Separating Indexing and Searching (was silent semantic changes with reiser4) 2004-08-27 18:20 ` Will Dyson 2004-08-27 18:38 ` David Ford 2004-08-28 9:54 ` Hans Reiser @ 2004-08-28 15:18 ` Alexander G. M. Smith 2004-08-28 22:11 ` Hans Reiser ` (2 more replies) 2 siblings, 3 replies; 1392+ messages in thread From: Alexander G. M. Smith @ 2004-08-28 15:18 UTC (permalink / raw) To: Will Dyson Cc: akpm, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list, reiser Will Dyson wrote on Fri, 27 Aug 2004 14:20:29 -0400: > Hans Reiser wrote: > > Will Dyson wrote: > >> In the original BeOS, they solved the problem by having the filesystem > >> driver itself take a text query string and parse it, returning a list > >> of inodes that match. The whole business of parsing a query string in > >> the kernel (let alone in the filesystem driver) has always seemed ugly > >> to me. > > > > Why? > > Hmm. Trying to explain aesthetic judgments is always fun, but I'll try. > > String parsing bloats the kernel with code that will be called rarely, > and doing it inside the filesystem module makes for duplicate code if > more than one filesystem does it. But a good common parser routine (or a > kernel api that takes a pre-compiled parse tree) would reduce the bad taste. Not only bloat, but time wasted reinventing the wheel. For my experimental file system (http://members.rogers.com/agmsbeos/AGMSRAMFileSystem.readme - see the Nifty Features chapter), I had to figure out the BFS query language (it was documented but didn't have every little detail explained - some experimentation was needed), then write a tokenizer and parser for it, to convert it down to a parse tree that could be used during the query's life. Good exercise for the programmer, but duplication of effort. I also added a few new things to the query language, but because the parsing is done by each file system separately, my "improvements" won't show up in other file systems. > For the 99.9% of people who were not beos users back in the day, the > "live query" was a way to register a query string with the kernel, so > that you got notifications when the list of files matching the query > changed. It was pretty cool, but this is really just blue-skying since I > am not stepping up to write such a beast any time soon. Kind of like that. For performance, the query string is converted into a parse tree and then all file operations that touch any attribute mentioned in the query get evaluated against the parse tree to see if the result (in or out of the query result set) has changed. If it changes, then a kernel facility is used to broadcast a message describing the change and file involved to all parties monitoring that query. Of course, before becoming live, the existing files that match the query are found by iterating over an index (choosing the appropriate index is an optimization trick). > >> However, the best alternative I've come up with is to simply export > >> the index data as a special file (perhaps in sysfs?) and have > >> userspace responsible for searching the index. > > > > That is the best implementation suggestion I've heard for splitting the > > filesystem into two parts, one in user space and one in kernel, but I > > still don't trust it to work well. > > Why not? And if it really splits the filesystem into two parts depends > on which of the two following statements you agree with more: > > "Fast searching is a feature of the filesystem" > "Maintaining indexes is a feature of the filesystem" I like that. Put the indexing and a related attribute change update notification mechanism into the kernel. User land libraries can build on that to implement whatever query language you want, thus saving work and having a common language for all file system varieties that support indices. However, one of my (unfinished) experiments was to have magic directories that show query results as their contents. One attribute of the directory (or even its name) would be the query string. That way even old software (like "ls") could use queries! Implementing queries-as-directories might require moving some things back into the kernel, or at least into some plug-in level. - Alex ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: Separating Indexing and Searching (was silent semantic changes with reiser4) 2004-08-28 15:18 ` Separating Indexing and Searching (was silent semantic changes with reiser4) Alexander G. M. Smith @ 2004-08-28 22:11 ` Hans Reiser 2004-08-28 22:12 ` Hans Reiser 2004-08-31 1:57 ` Will Dyson 2 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2004-08-28 22:11 UTC (permalink / raw) To: Alexander G. M. Smith Cc: Will Dyson, akpm, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Alexander G. M. Smith wrote: > > >However, one of my (unfinished) experiments was to have magic directories that show query results as their contents. One attribute of the directory (or even its name) would be the query string. That way even old software (like "ls") could use queries! Implementing queries-as-directories might require moving some things back into the kernel, or at least into some plug-in level. > > Beautiful example of the sort of thing that I don't want to be locked out from doing by having the parser in user space. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: Separating Indexing and Searching (was silent semantic changes with reiser4) 2004-08-28 15:18 ` Separating Indexing and Searching (was silent semantic changes with reiser4) Alexander G. M. Smith 2004-08-28 22:11 ` Hans Reiser @ 2004-08-28 22:12 ` Hans Reiser 2004-09-01 20:41 ` Jamie Lokier 2004-08-31 1:57 ` Will Dyson 2 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-28 22:12 UTC (permalink / raw) To: Alexander G. M. Smith Cc: Will Dyson, akpm, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Alexander G. M. Smith wrote: > > >However, one of my (unfinished) experiments was to have magic directories that show query results as their contents. One attribute of the directory (or even its name) would be the query string. That way even old software (like "ls") could use queries! Implementing queries-as-directories might require moving some things back into the kernel, or at least into some plug-in level. > >- Alex > > > > Symlinks also. Symlinks with powerful queries in them would require a parser in the kernel. Thanks for helping me to distill my incoherent reasons for the parser being in the kernel. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: Separating Indexing and Searching (was silent semantic changes with reiser4) 2004-08-28 22:12 ` Hans Reiser @ 2004-09-01 20:41 ` Jamie Lokier 0 siblings, 0 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-09-01 20:41 UTC (permalink / raw) To: Hans Reiser Cc: Alexander G. M. Smith, Will Dyson, akpm, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Hans Reiser wrote: > Symlinks also. Symlinks with powerful queries in them would require a > parser in the kernel. No they wouldn't. You can do symlinks with powerful queries in userspace _today_ (as well as directories which list queries). Making queries be up to date in real-time is still a problem, but that has nothing to do with symlinks or directories, or where the parsers live. > Thanks for helping me to distill my incoherent reasons for the > parser being in the kernel. This is one occasion where you're mistaken :) There may be other reasons for parsing query strings in the kernel (though I'm not convinced there are any), but this isn't one. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: Separating Indexing and Searching (was silent semantic changes with reiser4) 2004-08-28 15:18 ` Separating Indexing and Searching (was silent semantic changes with reiser4) Alexander G. M. Smith 2004-08-28 22:11 ` Hans Reiser 2004-08-28 22:12 ` Hans Reiser @ 2004-08-31 1:57 ` Will Dyson 2 siblings, 0 replies; 1392+ messages in thread From: Will Dyson @ 2004-08-31 1:57 UTC (permalink / raw) To: Alexander G. M. Smith Cc: akpm, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list, reiser Alexander G. M. Smith wrote: > Not only bloat, but time wasted reinventing the wheel. For my > experimental file system > (http://members.rogers.com/agmsbeos/AGMSRAMFileSystem.readme - see > the Nifty Features chapter), I had to figure out the BFS query I had a look. Cool stuff! > I like that. Put the indexing and a related attribute change update > notification mechanism into the kernel. User land libraries can > build on that to implement whatever query language you want, thus > saving work and having a common language for all file system > varieties that support indices. Yeah, thats basicly what I was thinking. Right now I'm trying to think of any kind of filesystem that could support the general concept of "Fast Searching", but be somehow unable to export this capability as a set of indices on file attributes.... Ok. I give up. It seems like the right abstraction to me. > However, one of my (unfinished) experiments was to have magic > directories that show query results as their contents. One attribute > of the directory (or even its name) would be the query string. That > way even old software (like "ls") could use queries! Implementing > queries-as-directories might require moving some things back into the > kernel, or at least into some plug-in level. That is a pretty awesome idea. It should be done in its own specialized filesystem though. QueryFS. If the change-notification mechanism in the kernel was robust enough (a big if), the rest could be done with a userspace filesystem (via FUSE). Mount a query, cd into it, run grep, write a little bash script... Very unixy. -- Will Dyson "Back off man, I'm a scientist!" -Dr. Peter Venkman ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:31 ` Hans Reiser ` (2 preceding siblings ...) 2004-08-26 16:32 ` Will Dyson @ 2004-08-26 18:12 ` Horst von Brand 2004-08-27 20:32 ` Markus Törnqvist ` (2 more replies) 2004-08-26 20:04 ` Martin J. Bligh 2004-08-27 21:23 ` vfs2 (was Re: silent semantic changes with reiser4) Pavel Machek 5 siblings, 3 replies; 1392+ messages in thread From: Horst von Brand @ 2004-08-26 18:12 UTC (permalink / raw) To: Hans Reiser Cc: Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Hans Reiser <reiser@namesys.com> said: > Andrew Morton wrote: [...] > > The fact that one filesystem will offer features which other > > filesystems do not and cannot offer makes me queasy for some reason. Not me. As long as it is clearly experimental stuff that can be removed at the head hacker's whim, that is. 2.7 material. > Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem > for Apple, Says who? > and that means we need to put search engine and database > functionality into the filesystem. Please don't. Unix works and is extremely popular because it _doesn't_ try to be smart (policy vs mechanism distinction, simple abstractions that can be combined endlessly). If you add this to the filesystems, you either redo _all_ userland to understand _one_ particular way of doing "smart things" (which will turn out to be almost exactly the dumbest possible for some uses, and then you are stuck), or get lots of shards from broken apps (and users, and sysadmins). > It takes 11 years of serious > research to build a clean storage layer able to handle doing that. Great! Build an experimental OS showing how to use it, and through that see if the ideas hold any water _in real, day-to-day, down-to-earth, practice_. > Reiser4 has done that, finally. None of the other Linux filesystems > have. How come nobody before you in the almost 30 years of Unix has ever seen the need for this indispensable feature? > The next major release of ReiserFS is going to be bursting with > semantic enhancements, because the prerequisites for them are in place > now. None of the other Linux filesystems have those prerequisites. To me that would mean that ReiserFS has no place in Linux. > They won't be able to keep up with the semantic enhancements. This > metafiles and file-directories stuff is actually fairly trivial stuff. Maybe. But the question of it being of any use aren't trivially answered "yes". My gut feeling is that the answer is a resounding "no", but that's just me. Direcories are directories, if you want to ship directory-like stuff around, use directories (or tarfiles, or whatever). Just look what happened to the much, much easier stuff of splitting SUID/SGID into capabilities: Nothing at all whatsoever, because they have no decent place to stay in the hallowed Unix APIs. Sure, Linux is now large enough to be able to _define_ APIs to follow, but even so it isn't at all easy to do so. > Look guys, in 1993 I anticipated the battle would be here, and I build > the foundation for a defensive tower right at the spot MS and Apple are > now maneuvering towards. Why place a protective tower around the pit into which they are trying desperately to throw themselves? ;-) > Help me get the next level on the tower before > they get here. It is one hell of a foundation, they won't be able to > shake it, their trees are not as powerful. Don't move reiser4 into vfs, > use reiser4 as the vfs. Don't write filesystems, write file plugins and > disk format plugins and all the other kinds of plugins, and you won't be > missing any expressive power that you really want.... Better write libraries handling whatever weird format you have in mind for each use. Even applications. I don't expect to be able to use emacs to edit a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit a poem. The current trend (which I heartily agree with) is to take stuff _out_ of the kernel (for flexibility, access, development ease, even performance). -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:12 ` silent semantic changes with reiser4 Horst von Brand @ 2004-08-27 20:32 ` Markus Törnqvist 2004-08-27 20:38 ` Linus Torvalds 2004-08-27 20:44 ` Spam 2004-09-01 21:43 ` Jamie Lokier 2 siblings, 1 reply; 1392+ messages in thread From: Markus Törnqvist @ 2004-08-27 20:32 UTC (permalink / raw) To: Horst von Brand Cc: Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list On Thu, Aug 26, 2004 at 02:12:32PM -0400, Horst von Brand wrote: >> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem >> for Apple, >Says who? People will say it when people stop using Linux on servers because they can integrate metadata easier in other operating systems ;) >Please don't. Unix works and is extremely popular because it _doesn't_ try >to be smart (policy vs mechanism distinction, simple abstractions that can >be combined endlessly). If you add this to the filesystems, you either redo Just more moving little pieces to be combined endlessly. I think it looks like Reiser4 semantics should go into VFS and not break legacy posix compatibility. If you don't want to use a filesystem like that, don't. Maybe the reason Linux hasn't developed this yet is the fact that Linux supports n+1 file systems and above and beyond that many userspace ones. So there has been enough work in maintaining and competing the old ones and not coming up with new ones. >_all_ userland to understand _one_ particular way of doing "smart things" >(which will turn out to be almost exactly the dumbest possible for some >uses, and then you are stuck), or get lots of shards from broken apps (and >users, and sysadmins). This is a question if implementation. What if the files-as-dirs have a ..metas/mainstream[1] file and are handled by legacy applications as directories? Nothing breaks, but may look a bit ugly. Nothing one line of shell scripting would not cure. OTOH, the user space progs are quickly patched to support all this. >Great! Build an experimental OS showing how to use it, and through that see >if the ideas hold any water _in real, day-to-day, down-to-earth, practice_. Why should he? Parts of Linux are already here and susceptible to experiments in evolution. (Break out 2.7 for this one, the people want code! :) >How come nobody before you in the almost 30 years of Unix has ever seen the >need for this indispensable feature? Lack of imagination? Lack of need? Lack of drive space? And maybe someone did think among these lines back in the days, it's not like the current dogma of file systems is the only one that has ever been used. >To me that would mean that ReiserFS has no place in Linux. This is the type of argumentation I can not understand. Do you not have the freedom not to use it? And this is not personal, but please say Reiser4, as ReiserFS is coupled with versions 3.x. They are quite different. >Maybe. But the question of it being of any use aren't trivially answered >"yes". My gut feeling is that the answer is a resounding "no", but that's I think it's a pretty big yes, even if the technical details do need to be clear before anything is set in stone. That's the reason for the on-going discussion. >just me. Direcories are directories, if you want to ship directory-like >stuff around, use directories (or tarfiles, or whatever). Just look what Without a common user-space API to deal in them. Oh, joy. Make the common user-space API? Will never happen. >Better write libraries handling whatever weird format you have in mind for >each use. Even applications. I don't expect to be able to use emacs to edit >a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit >a poem. The current trend (which I heartily agree with) is to take stuff >_out_ of the kernel (for flexibility, access, development ease, even >performance). I don't think anyone's talking about using emacs to edit a picture, but emacs could be used to edit the picture's description.[2] Maybe with some userspace-hook that exports the exif data via ..metas/ so the kernel won't have to parse it. Linus wrote about why TCP is not in userspace and I think this is a similar situation. Technically. But not everything has to be in kernel space in Reiser4. Or that's what they say and I'm not sure where I stand on an exif parser in the kernel... On the social side, I may be wrong in this, but not many userspace filesystems have the possibility of becoming a de-facto standard. One might say that's because they suck, or they suck because they're implemented in userspace or they are just met with prejudice for the same reason. In the kernel something like this would have to be done right and people would put effort into doing it right. I for one would welcome that. [1] Sorry, guys, just had to use that name.. [2] You can edit a picture in emacs already, it's just prone to break ;) -- mjt ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 20:32 ` Markus Törnqvist @ 2004-08-27 20:38 ` Linus Torvalds 2004-08-27 20:48 ` Re[2]: " Spam ` (3 more replies) 0 siblings, 4 replies; 1392+ messages in thread From: Linus Torvalds @ 2004-08-27 20:38 UTC (permalink / raw) To: Markus Törnqvist Cc: Horst von Brand, Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, 27 Aug 2004, Markus Törnqvist wrote: > > People will say it when people stop using Linux on servers because > they can integrate metadata easier in other operating systems ;) Heh. Considering that WinFS seems to be delayed yet more, I don't think that's a very strong argument. Hell will freeze over before Microsoft does a filesystem right. Besides, WinFS is likely almost in user mode anyway, ie mostly a library, rather like the gnome people are already doing with nome storage. So there's really no point in trying to push your agenda by trying to scare people with MS activities. Linux kernel developers do what's right because it is _right_, not because somebody else does it. Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re[2]: silent semantic changes with reiser4 2004-08-27 20:38 ` Linus Torvalds @ 2004-08-27 20:48 ` Spam 2004-08-27 20:53 ` Markus Törnqvist ` (2 subsequent siblings) 3 siblings, 0 replies; 1392+ messages in thread From: Spam @ 2004-08-27 20:48 UTC (permalink / raw) To: Linus Torvalds Cc: Markus Törnqvist, Horst von Brand, Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list > On Fri, 27 Aug 2004, Markus Törnqvist wrote: >> >> People will say it when people stop using Linux on servers because >> they can integrate metadata easier in other operating systems ;) > Heh. Considering that WinFS seems to be delayed yet more, I don't think > that's a very strong argument. > Hell will freeze over before Microsoft does a filesystem right. Besides, > WinFS is likely almost in user mode anyway, ie mostly a library, rather > like the gnome people are already doing with nome storage. Yes, it is mostly in user land. But, It is still integrated with the Windows API. This means that normal programs that use the Windows API will be able to use most of the new features (like displaying thumbnails, filtered views, etc when you do file operations) Besides, NTFS is not that bad. It does have both encryption and compression and a big set of ACL features that aren't in most Linux filesystems today. > So there's really no point in trying to push your agenda by trying to > scare people with MS activities. Linux kernel developers do what's right > because it is _right_, not because somebody else does it. I agree. Things should be done because they are good, not because you are threatened to. It is a shame really that so many people hate Microsoft (or Apple) so much they fail to see things that actually are good. > Linus ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 20:38 ` Linus Torvalds 2004-08-27 20:48 ` Re[2]: " Spam @ 2004-08-27 20:53 ` Markus Törnqvist 2004-08-27 21:33 ` Bill Huey 2004-08-28 9:55 ` Hans Reiser 3 siblings, 0 replies; 1392+ messages in thread From: Markus Törnqvist @ 2004-08-27 20:53 UTC (permalink / raw) To: Linus Torvalds Cc: Horst von Brand, Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Fri, Aug 27, 2004 at 01:38:28PM -0700, Linus Torvalds wrote: > >Heh. Considering that WinFS seems to be delayed yet more, I don't think >that's a very strong argument. But there's Mac OS and isn't Solaris doing something along these lines? Microsoft isn't alone there ;) >scare people with MS activities. Linux kernel developers do what's right >because it is _right_, not because somebody else does it. Never really doubted that, mind you. I just think this is both right and a competition situation. -- mjt ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 20:38 ` Linus Torvalds 2004-08-27 20:48 ` Re[2]: " Spam 2004-08-27 20:53 ` Markus Törnqvist @ 2004-08-27 21:33 ` Bill Huey 2004-08-28 9:55 ` Hans Reiser 3 siblings, 0 replies; 1392+ messages in thread From: Bill Huey @ 2004-08-27 21:33 UTC (permalink / raw) To: Linus Torvalds Cc: Markus T?rnqvist, Horst von Brand, Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list, bhuey On Fri, Aug 27, 2004 at 01:38:28PM -0700, Linus Torvalds wrote: > Heh. Considering that WinFS seems to be delayed yet more, I don't think > that's a very strong argument. Which is another argument, and bragging rights, to integrate something like Reiser FS4. :) > Hell will freeze over before Microsoft does a filesystem right. Besides, > WinFS is likely almost in user mode anyway, ie mostly a library, rather > like the gnome people are already doing with nome storage. They're trying to push their low level storage model into the kernel after they saw how slow their userspace version of that is. It makes sense to have this kind of stuff directly apart of a lower level FS later > So there's really no point in trying to push your agenda by trying to > scare people with MS activities. Linux kernel developers do what's right > because it is _right_, not because somebody else does it. I tried to stay out of this thread, but I couldn't resist after this post. :) MS has other problems, namely, they're a large company trying to basically replicated Apple's (NeXT) OS X stuff, but onl 10x larger and slower than it is. It's really silly what's going on with MS development as well as the greater open source X community. The sooner folks realize that all rendering models are moving in the direction of Apple's OS X, the sooner they can just flat out rip it off. It's going to require both kernel and userspace changes for this to happen. bill ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-27 20:38 ` Linus Torvalds ` (2 preceding siblings ...) 2004-08-27 21:33 ` Bill Huey @ 2004-08-28 9:55 ` Hans Reiser 2004-09-01 22:45 ` Bill Huey 3 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2004-08-28 9:55 UTC (permalink / raw) To: Linus Torvalds Cc: Markus Törnqvist, Horst von Brand, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list Linus Torvalds wrote: >On Fri, 27 Aug 2004, Markus Törnqvist wrote: > > >>People will say it when people stop using Linux on servers because >>they can integrate metadata easier in other operating systems ;) >> >> > >Heh. Considering that WinFS seems to be delayed yet more, I don't think >that's a very strong argument. > >Hell will freeze over before Microsoft does a filesystem right. Besides, >WinFS is likely almost in user mode anyway, ie mostly a library, rather >like the gnome people are already doing with nome storage. > >So there's really no point in trying to push your agenda by trying to >scare people with MS activities. Linux kernel developers do what's right >because it is _right_, not because somebody else does it. > > Linus > > > > Apple will get it right. I promise it. I have met Dominic, and he is very very sharp. Look at the Tiger demos on their website. Simple interface, looks nice to me.... The one area he might screw up is performance, but I don't care to count on that. WinFS first tried to put it all in the FS, and then it became a user mode library almost certainly because they are making the standard mistakes the database guys make when they try to emulate file systems without changing the core balanced tree algorithms, and their performance sucked and they had to back off. It took 11 years for me to get it right, and they aren't as crazy-err-persistent as I am.;-) We might get lucky and have them produce another NTFS, but then again, when Microsoft focuses on a task, they do much better at it than they do most of the time, and they are focused on WinFS. They have hired very sharp people. We can hope that they don't know how to use them, but when they hire people like Gerard Salton for $1 million a year, there is just possibly a chance that they might try to get their money's worth out of him. You should not be complacent about WinFS being delayed to 2007, because even if I get funding for enhanced ReiserFS semantics tomorrow we also can't get the job done before 2007. This is big science, not writing a device driver. Finally, how much harm will it be if we do it right and it is important and they fail? Suppose I am wrong about them, and we create a powerful unifying namespace for Linux before any other OS does? Is that so bad? Creating a powerful namespace at the heart of Linux is the most important enhancement you can make to the OS. Finally the storage layer is good enough to support putting the relationship between keywords (actually keyobjects in my scheme....) and their documents directly into the FS without losing performance for traditional file system usage patterns, and I get to stop tweaking performance and go have fun with semantics in the next major release. Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-28 9:55 ` Hans Reiser @ 2004-09-01 22:45 ` Bill Huey 0 siblings, 0 replies; 1392+ messages in thread From: Bill Huey @ 2004-09-01 22:45 UTC (permalink / raw) To: Hans Reiser Cc: Linus Torvalds, Markus T?rnqvist, Horst von Brand, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, reiserfs-list On Sat, Aug 28, 2004 at 02:55:00AM -0700, Hans Reiser wrote: > mistakes the database guys make when they try to emulate file systems > without changing the core balanced tree algorithms, and their > performance sucked and they had to back off. It took 11 years for me to > get it right, and they aren't as crazy-err-persistent as I am.;-) > > We might get lucky and have them produce another NTFS, but then again, > when Microsoft focuses on a task, they do much better at it than they do > most of the time, and they are focused on WinFS. They have hired very > sharp people. We can hope that they don't know how to use them, but > when they hire people like Gerard Salton for $1 million a year, there is > just possibly a chance that they might try to get their money's worth > out of him. I realize this is a relatively late reply, but... MS's problem is that they hire very smart folks, but the compartimentalize them so they don't influence the overall architecture of the entire system. I guarantee you that even if they did get all of this in place and working, their arrogance in this matter will screw up how this will be properly used and never expressed the power of a system like this. Apple is another story. They're the only other folks that might get this right. Your system is a dream of mine since I was dorking around with persistent objects in Smalltalk systems. But getting other folks to understand something like this and then use it in this way is very difficult and I don't expect MS to be able to do this nor get .NET to compete with Java for server side applications. I don't expect traditional Unix folks to understand why something like this is important also. It's not about a single technology as much as the system as a whole, larger picture... That's were a system starts to show it's superiority. That's my two cents on the matter. bill ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:12 ` silent semantic changes with reiser4 Horst von Brand 2004-08-27 20:32 ` Markus Törnqvist @ 2004-08-27 20:44 ` Spam [not found] ` <200408291521.i7TFLsQk028363@localhost.localdomain> 2004-09-01 21:43 ` Jamie Lokier 2 siblings, 1 reply; 1392+ messages in thread From: Spam @ 2004-08-27 20:44 UTC (permalink / raw) To: Horst von Brand Cc: Hans Reiser, Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list > Hans Reiser <reiser@namesys.com> said: >> Andrew Morton wrote: > [...] >> > The fact that one filesystem will offer features which other >> > filesystems do not and cannot offer makes me queasy for some reason. > Not me. As long as it is clearly experimental stuff that can be removed at > the head hacker's whim, that is. 2.7 material. >> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem >> for Apple, > Says who? At least I as a user see the benefits largely. The workstation / desktop user has much more need for such features, than perhaps the normal Linux sysadmin. >> and that means we need to put search engine and database >> functionality into the filesystem. > Please don't. Unix works and is extremely popular because it _doesn't_ try > to be smart (policy vs mechanism distinction, simple abstractions that can > be combined endlessly). If you add this to the filesystems, you either redo > _all_ userland to understand _one_ particular way of doing "smart things" > (which will turn out to be almost exactly the dumbest possible for some > uses, and then you are stuck), or get lots of shards from broken apps (and > users, and sysadmins). Indeed. But you forget one thing. Combining lots of small userland tools can actually make things more complex - at least for the user. That said, I do not see that anyone has suggested to remove this way of working with applications. Are the new and old features mutually exclusive? >> It takes 11 years of serious >> research to build a clean storage layer able to handle doing that. > Great! Build an experimental OS showing how to use it, and through that see > if the ideas hold any water _in real, day-to-day, down-to-earth, practice_. There are many uses already. It could be encryption, compression, meta-data, file streams, sorting, searching and a multitude of uses and ideas that haven't been discovered yet. >> Reiser4 has done that, finally. None of the other Linux filesystems >> have. > How come nobody before you in the almost 30 years of Unix has ever seen the > need for this indispensable feature? I, as a user, have for a long time. Do not drag everyone over the same edge. Users have different needs. We should try to be as forthcoming as possible to provide the tools and options for them. >> The next major release of ReiserFS is going to be bursting with >> semantic enhancements, because the prerequisites for them are in place >> now. None of the other Linux filesystems have those prerequisites. > To me that would mean that ReiserFS has no place in Linux. Why do you say this? I thought Linux embraced advancements and new technologies. >> They won't be able to keep up with the semantic enhancements. This >> metafiles and file-directories stuff is actually fairly trivial stuff. > Maybe. But the question of it being of any use aren't trivially answered > "yes". My gut feeling is that the answer is a resounding "no", but that's > just me. Direcories are directories, if you want to ship directory-like > stuff around, use directories (or tarfiles, or whatever). Just look what > happened to the much, much easier stuff of splitting SUID/SGID into > capabilities: Nothing at all whatsoever, because they have no decent place > to stay in the hallowed Unix APIs. Sure, Linux is now large enough to be > able to _define_ APIs to follow, but even so it isn't at all easy to do so. Exactly. The problem I see in this discussion is the lack of just this - APIs. Because the lack of good APIs we cannot make any larger changes or bring new features because old applications will break. I think that Linux should make new API that will completely separate applications from the storage media. This way we will not as easily break applications when new features are developed. This should be the main priority. Perhaps over a transition period the old ways will be allowed while applications are ported to the new APIs. The change is certainly going to be needed at one point or Linux won't be able to evolve as quickly as its users are demanding it. Application breakage has happened before between virtually every kernel change. It isn't new. Just take devfs and sysfs. >> Look guys, in 1993 I anticipated the battle would be here, and I build >> the foundation for a defensive tower right at the spot MS and Apple are >> now maneuvering towards. > Why place a protective tower around the pit into which they are trying > desperately to throw themselves? ;-) It is easy to say this. But the fact is that users can actually use these new features for good things. >> Help me get the next level on the tower before >> they get here. It is one hell of a foundation, they won't be able to >> shake it, their trees are not as powerful. Don't move reiser4 into vfs, >> use reiser4 as the vfs. Don't write filesystems, write file plugins and >> disk format plugins and all the other kinds of plugins, and you won't be >> missing any expressive power that you really want.... > Better write libraries handling whatever weird format you have in mind for > each use. Even applications. I don't expect to be able to use emacs to edit > a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit > a poem. The current trend (which I heartily agree with) is to take stuff > _out_ of the kernel (for flexibility, access, development ease, even > performance). If I want to be able to quickly find files by a certain author I could search the meta-data fields (if they are used) with normal tools such as grep. Compression or encryption is another example. Would you rather patch every application there is for Linux to enable support for reading these files and directories? ^ permalink raw reply [flat|nested] 1392+ messages in thread
[parent not found: <200408291521.i7TFLsQk028363@localhost.localdomain>]
* Re: silent semantic changes with reiser4 [not found] ` <200408291521.i7TFLsQk028363@localhost.localdomain> @ 2004-08-29 16:52 ` Spam 2004-08-29 17:22 ` Nikita Danilov 2004-08-29 16:53 ` Spam 2004-08-29 16:54 ` Spam 2 siblings, 1 reply; 1392+ messages in thread From: Spam @ 2004-08-29 16:52 UTC (permalink / raw) To: Horst von Brand Cc: Hans Reiser, Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list > Spam <spam@tnonline.net> said: >> Horst von Brand <vonbrand@inf.utfsm.cl> said: >> > Hans Reiser <reiser@namesys.com> said: >> >> Andrew Morton wrote: > [...] >> >> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem >> >> for Apple, >> >> > Says who? >> >> At least I as a user see the benefits largely. The workstation / >> desktop user has much more need for such features, than perhaps the >> normal Linux sysadmin. > Show benefits, show there are no downsides. Show that the same can't > possibly be done in userland. Then I'll consider buying it. I am not saying it couldn't be done in userland. Most things can. The problem with userland is that it doesn't provide generic access across most applications. For example GnomeVFS is limited to Gnome applications only. The same for KDE. to be able to coordinate application developer on anything is just almost impossible. Still. Why do you oppose plugins, streams and meta files? The could be valuable and easy to use tools for many purposes. One could be would be advanced ACLs defined as a meta-file using XML format. > AFAICS: Smallish benefits for some narrow application niche. Mayor problems > (no decent multiuser way of using it, extremely complex stuff placed in the > kernel). Userland solutions are all around us right now, and work just > fine. No, placing the same gunk in the kernel won't make it magically use > no time or memory, more like exactly the opposite. Besides, all the visions > waved around here have very little in common (except the idea that they > make all restrictions magically dissapear). > Get a list of required features, show concrete ways of using it, outline > possible implementations, discuss resource usage. I.e., do "engineering". > Then we have something to discuss. I am not sure what you refer to as "required features". Do you mean example of usage and contents for streams, plugins and meta files? >> >> and that means we need to put search engine and database >> >> functionality into the filesystem. >> > Please don't. Unix works and is extremely popular because it _doesn't_ try >> > to be smart (policy vs mechanism distinction, simple abstractions that can >> > be combined endlessly). If you add this to the filesystems, you either redo >> > _all_ userland to understand _one_ particular way of doing "smart things" >> > (which will turn out to be almost exactly the dumbest possible for some >> > uses, and then you are stuck), or get lots of shards from broken apps (and >> > users, and sysadmins). >> Indeed. But you forget one thing. Combining lots of small userland >> tools can actually make things more complex - at least for the user. > So? Placing lots of little things in kernel does the same, plus screws over > sysadmins, and makes kernel hacker's job harder. I prefer the former alone. Difference being that it could be transparent in a way otherwise not possible. >> That said, I do not see that anyone has suggested to remove this way >> of working with applications. > Excuse me, if files suddenly become directories, and you can't have some > "reserved" file/directory names, and... you are breaking lots of stuff. >> Are the new and old features mutually exclusive? > Yep. >> >> It takes 11 years of serious >> >> research to build a clean storage layer able to handle doing that. >> >> > Great! Build an experimental OS showing how to use it, and through that see >> > if the ideas hold any water _in real, day-to-day, down-to-earth, practice_. >> There are many uses already. > Where? >> It could be encryption, compression, >> meta-data, file streams, sorting, searching and a multitude of uses >> and ideas that haven't been discovered yet. > "Could", "multitude ... haven't been discovered yet" is _not_ "many current > uses". >> >> Reiser4 has done that, finally. None of the other Linux filesystems >> >> have. >> > How come nobody before you in the almost 30 years of Unix has ever seen the >> > need for this indispensable feature? >> I, as a user, have for a long time. Do not drag everyone over the >> same edge. Users have different needs. We should try to be as >> forthcoming as possible to provide the tools and options for them. > Why? Breaking the basic paradigms for some user whim is not good OS design > in my book, sorry. Many "bright ideas" have been thrown out over the years > for just the same reasons. >> >> The next major release of ReiserFS is going to be bursting with >> >> semantic enhancements, because the prerequisites for them are in place >> >> now. None of the other Linux filesystems have those prerequisites. >> > To me that would mean that ReiserFS has no place in Linux. >> Why do you say this? I thought Linux embraced advancements and new >> technologies. > If they are right. I think breaking the base of the OS is not a good > idea. Perhaps ReiserFS has a place elsewhere (in MacOS, or Windows, where > this is supposedly so so badly needed?), but not here. You mean in Windows and MacOS where much of this functionality already exists? >> >> They won't be able to keep up with the semantic enhancements. This >> >> metafiles and file-directories stuff is actually fairly trivial stuff. >> > Maybe. But the question of it being of any use aren't trivially answered >> > "yes". My gut feeling is that the answer is a resounding "no", but that's >> > just me. Direcories are directories, if you want to ship directory-like >> > stuff around, use directories (or tarfiles, or whatever). Just look what >> > happened to the much, much easier stuff of splitting SUID/SGID into >> > capabilities: Nothing at all whatsoever, because they have no decent place >> > to stay in the hallowed Unix APIs. Sure, Linux is now large enough to be >> > able to _define_ APIs to follow, but even so it isn't at all easy to do so. >> Exactly. The problem I see in this discussion is the lack of just >> this - APIs. Because the lack of good APIs we cannot make any larger >> changes or bring new features because old applications will break. I'll chop this up... > No. There is a complete lack of answers to why it should be done ("I think > it would be nice", "MSFT is trying to do it", "Look at MacOS" is far from > enough) No perhaps not by itself. But it certainly should be considered. These two OS'es have most of the market share. They have great resources trying to make things that work for their users. Users are more important than you think. Linux and any OS can do what it wants, but if it wants to keep users then it has to listen to them. > , on what the alternatives are ("Sorry, it must be in the kernel > because it is impossible to get application writers to agree on how to do > it" This is a perfectly valid reason. If it is necessary to move something into the core of Linux because it couldn't otherwise be done then it should seriously be considered. > will just add another incompatible way of "doing almost the same", this > time in the kernel), People on this list are actually trying to make it to be fully compatible with old legacy applications. If you hadn't noticed, Reiser4 does work. The meta files do work. The discussion is wether parts of Reiser4 should be made generic and fit into the VFS. > and last but not least what the impact is (performance > wise, application breakage, user/sysadmin expectation breakage, > incompatibility with other systems, kernel development impact, to list a > few). Hash this out, and we can talk. You paint up a doomsday picture here. Nothing of this magnitude has happened and won't. besides, what is wrong if sysadmins would have to learn something new? >> I think that Linux should make new API that will completely separate >> applications from the storage media. This way we will not as easily >> break applications when new features are developed. This should be >> the main priority. > The "files + directories" stuff is a nice abstraction, and works today on > all random-access storage media. You don't care much if it is on a CD, a > floppy, a disk, a RAID array, or a flash card, do you? It won't work on > tapes, or printers; but that is to be expected. you didn't actually comment on what I said. I mentioned API's nothing about directories and files. API being the keyword here. If there are good API's that applications can use, for example to access filesystems, then it will be easier and much more possible to introduce new things into the filesystem without breaking existing applications. One example of preventing application breakage could be to have a versioning system. Applications could query which version is supported, or vice versa. From what I understand it is much harder to do anything today because of the fixed ways. This is why we have this discussion. > [...] >> Application breakage has happened before between virtually every >> kernel change. It isn't new. Just take devfs and sysfs. > Oops, now emacs stopped working because devfs was added? gcc needed a mayor > overhaul because of /sys vs /proc? Gimme a break. That was _basic system tool_ > breakage, not all-around application breakage. Emacs seem to work fine with the file-as-dir on my Reiser4 partition. And so does actually any other application I have used on it. Be it in Gnome, KDE or in plain shell's. >> >> Look guys, in 1993 I anticipated the battle would be here, and I build >> >> the foundation for a defensive tower right at the spot MS and Apple are >> >> now maneuvering towards. >> >> > Why place a protective tower around the pit into which they are trying >> > desperately to throw themselves? ;-) >> >> It is easy to say this. But the fact is that users can actually use >> these new features for good things. > And can abuse them for bad things, and the features aren't free. Balance > benefits (very minor) against costs (quite large). Abuse depends on permissions and security policies which is your job as admin to look after. It would be possible to limit, for example), the types of plugins that a user can either use or load. Do you give your users access to modprobe or tools like that? >> >> Help me get the next level on the tower before >> >> they get here. It is one hell of a foundation, they won't be able to >> >> shake it, their trees are not as powerful. Don't move reiser4 into vfs, >> >> use reiser4 as the vfs. Don't write filesystems, write file plugins and >> >> disk format plugins and all the other kinds of plugins, and you won't be >> >> missing any expressive power that you really want.... >> >> > Better write libraries handling whatever weird format you have in mind for >> > each use. Even applications. I don't expect to be able to use emacs to edit >> > a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit >> > a poem. The current trend (which I heartily agree with) is to take stuff >> > _out_ of the kernel (for flexibility, access, development ease, even >> > performance). >> If I want to be able to quickly find files by a certain author I >> could search the meta-data fields (if they are used) with normal >> tools such as grep. > Iff the metadata is there. It won't (everybody would have to agree on > placing that on the files for it to work), so it is useless. No it is not useless. It just means the file has no meta information. Just as it is today. No change. >> Compression or encryption is another example. >> >> Would you rather patch every application there is for Linux to >> enable support for reading these files and directories? > No. But an encrypted or compressed filessytem doesn't need weird files. > Easiest on applications is just to serve the decrypted/uncompressed files > for use via the traditional APIs. Next easiest is provide programs that > handle cryptography and compression and can be combined with whatever > unaware program you care to use. Funny, that is exactly the way things work > now... So you can setup a directory that has some files compressed and some other encrypted and yet they will be available to use with tar, cp, ls, mc, Nautilus, Emacs etc etc? ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 16:52 ` Spam @ 2004-08-29 17:22 ` Nikita Danilov 2004-08-29 18:12 ` Markus Törnqvist 0 siblings, 1 reply; 1392+ messages in thread From: Nikita Danilov @ 2004-08-29 17:22 UTC (permalink / raw) To: Spam Cc: Horst von Brand, Hans Reiser, Helge Hafting, Rik van Riel, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list Spam writes: > > > Spam <spam@tnonline.net> said: > >> Horst von Brand <vonbrand@inf.utfsm.cl> said: [...] > > > Show benefits, show there are no downsides. Show that the same can't > > possibly be done in userland. Then I'll consider buying it. > > I am not saying it couldn't be done in userland. Most things can. > > The problem with userland is that it doesn't provide generic access > across most applications. For example GnomeVFS is limited to Gnome > applications only. The same for KDE. to be able to coordinate > application developer on anything is just almost impossible. Hmm... drag-and-drop also doesn't work constently over all applications. Let's put it into kernel. Seriously, kernel prformance is critical to the system, and to achieve high performance all kernel code runs in single address space (actually, in portion of address spaces of user processes shared by all processes). All shared system state is located there. This means that bug in a kernel affects whole system. This means that kernel should be kept as simple as possible (and a bit more simple). This is why only things that cannot be done efficiently in the user level are put into kernel. And political agendas of various camps of user-level developers change nothing here. > > Still. Why do you oppose plugins, streams and meta files? The could > be valuable and easy to use tools for many purposes. One could > be would be advanced ACLs defined as a meta-file using XML format. POSIX has standard ACL API in C (well, "eternal draft" only), one can use it to extract ACLs from kernel and convert it to any format to one's heart content. Advantage of this is that when XML goes out of fashion (I hope this wouldn't yet happen when you will read this message), kernel API will remain intact. > Nikita. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-29 17:22 ` Nikita Danilov @ 2004-08-29 18:12 ` Markus Törnqvist 0 siblings, 0 replies; 1392+ messages in thread From: Markus Törnqvist @ 2004-08-29 18:12 UTC (permalink / raw) To: Nikita Danilov Cc: Spam, Horst von Brand, Hans Reiser, Helge Hafting, Rik van Riel, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list [ This will probably go down in history as the most pointless message ever, I just told Nik on IRC I'd take it up on the list :) ] On Sun, Aug 29, 2004 at 09:22:08PM +0400, Nikita Danilov wrote: >Spam writes: >Hmm... drag-and-drop also doesn't work constently over all >applications. Let's put it into kernel. What I'm after is that the file system should be able to store any arbitrary metadata. No matter what the format is. Maybe I was a bit harsh on XML in my previous email, but whatever. >Seriously, kernel prformance is critical to the system, and to achieve >high performance all kernel code runs in single address space (actually, >in portion of address spaces of user processes shared by all >processes). All shared system state is located there. This means that >bug in a kernel affects whole system. This means that kernel should be >kept as simple as possible (and a bit more simple). I do agree here. But is it seriously such a big issue to implement the files-as-dirs kind of thing? Be it overloading MAY_READ so it works with chdir or implementing openat or whatever it takes. I don't think this should be much more, if any more, than the capability to store this data. Which Reiser4 does. But moved from the Reiser4 level to the VFS level. Right? >This is why only things that cannot be done efficiently in the user >level are put into kernel. And political agendas of various camps of >user-level developers change nothing here. What I'd want to do is to be able to write a Reiser4 plugin that suits my needs for something. Not that I'd even know how to, but that's a matter of time. Are there other means of doing this than Reiser? I mean Reiser4 plugin functionality by other means. >> Still. Why do you oppose plugins, streams and meta files? The could >> be valuable and easy to use tools for many purposes. One could >> be would be advanced ACLs defined as a meta-file using XML format. >POSIX has standard ACL API in C (well, "eternal draft" only), one can >use it to extract ACLs from kernel and convert it to any format to one's >heart content. Advantage of this is that when XML goes out of fashion (I >hope this wouldn't yet happen when you will read this message), kernel >API will remain intact. Aa, this apparently is an argument against XML parsing in the kernel, not the idea that if someone wanted ACLs, that are in XML format, and could not use them (because the kernel doesn't parse), he should be able to write such a plugin. Besides, let's say the Reiser4 XML-ACL plugin has a parser that actually DOES something. For argument's sake. Would it not be up to the individual user to disable this plugin? I'd assume it's possible to write plugins so that they do not affect anything but themselves, meaning this parser. Of course the problem of copying a file with XML-ACLs to another Reiser4 fs without support for them should be thought of. But I guess this is where you take the shortest route; the XML-ACL files are intact in the file's metadata, but totally unusable? -- mjt ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 [not found] ` <200408291521.i7TFLsQk028363@localhost.localdomain> 2004-08-29 16:52 ` Spam @ 2004-08-29 16:53 ` Spam 2004-08-29 16:54 ` Spam 2 siblings, 0 replies; 1392+ messages in thread From: Spam @ 2004-08-29 16:53 UTC (permalink / raw) To: Horst von Brand Cc: Hans Reiser, Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list > Spam <spam@tnonline.net> said: >> Horst von Brand <vonbrand@inf.utfsm.cl> said: >> > Hans Reiser <reiser@namesys.com> said: >> >> Andrew Morton wrote: > [...] >> >> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem >> >> for Apple, >> >> > Says who? >> >> At least I as a user see the benefits largely. The workstation / >> desktop user has much more need for such features, than perhaps the >> normal Linux sysadmin. > Show benefits, show there are no downsides. Show that the same can't > possibly be done in userland. Then I'll consider buying it. I am not saying it couldn't be done in userland. Most things can. The problem with userland is that it doesn't provide generic access across most applications. For example GnomeVFS is limited to Gnome applications only. The same for KDE. to be able to coordinate application developer on anything is just almost impossible. Still. Why do you oppose plugins, streams and meta files? The could be valuable and easy to use tools for many purposes. One could be would be advanced ACLs defined as a meta-file using XML format. > AFAICS: Smallish benefits for some narrow application niche. Mayor problems > (no decent multiuser way of using it, extremely complex stuff placed in the > kernel). Userland solutions are all around us right now, and work just > fine. No, placing the same gunk in the kernel won't make it magically use > no time or memory, more like exactly the opposite. Besides, all the visions > waved around here have very little in common (except the idea that they > make all restrictions magically dissapear). > Get a list of required features, show concrete ways of using it, outline > possible implementations, discuss resource usage. I.e., do "engineering". > Then we have something to discuss. I am not sure what you refer to as "required features". Do you mean example of usage and contents for streams, plugins and meta files? >> >> and that means we need to put search engine and database >> >> functionality into the filesystem. >> > Please don't. Unix works and is extremely popular because it _doesn't_ try >> > to be smart (policy vs mechanism distinction, simple abstractions that can >> > be combined endlessly). If you add this to the filesystems, you either redo >> > _all_ userland to understand _one_ particular way of doing "smart things" >> > (which will turn out to be almost exactly the dumbest possible for some >> > uses, and then you are stuck), or get lots of shards from broken apps (and >> > users, and sysadmins). >> Indeed. But you forget one thing. Combining lots of small userland >> tools can actually make things more complex - at least for the user. > So? Placing lots of little things in kernel does the same, plus screws over > sysadmins, and makes kernel hacker's job harder. I prefer the former alone. Difference being that it could be transparent in a way otherwise not possible. >> That said, I do not see that anyone has suggested to remove this way >> of working with applications. > Excuse me, if files suddenly become directories, and you can't have some > "reserved" file/directory names, and... you are breaking lots of stuff. >> Are the new and old features mutually exclusive? > Yep. >> >> It takes 11 years of serious >> >> research to build a clean storage layer able to handle doing that. >> >> > Great! Build an experimental OS showing how to use it, and through that see >> > if the ideas hold any water _in real, day-to-day, down-to-earth, practice_. >> There are many uses already. > Where? >> It could be encryption, compression, >> meta-data, file streams, sorting, searching and a multitude of uses >> and ideas that haven't been discovered yet. > "Could", "multitude ... haven't been discovered yet" is _not_ "many current > uses". >> >> Reiser4 has done that, finally. None of the other Linux filesystems >> >> have. >> > How come nobody before you in the almost 30 years of Unix has ever seen the >> > need for this indispensable feature? >> I, as a user, have for a long time. Do not drag everyone over the >> same edge. Users have different needs. We should try to be as >> forthcoming as possible to provide the tools and options for them. > Why? Breaking the basic paradigms for some user whim is not good OS design > in my book, sorry. Many "bright ideas" have been thrown out over the years > for just the same reasons. >> >> The next major release of ReiserFS is going to be bursting with >> >> semantic enhancements, because the prerequisites for them are in place >> >> now. None of the other Linux filesystems have those prerequisites. >> > To me that would mean that ReiserFS has no place in Linux. >> Why do you say this? I thought Linux embraced advancements and new >> technologies. > If they are right. I think breaking the base of the OS is not a good > idea. Perhaps ReiserFS has a place elsewhere (in MacOS, or Windows, where > this is supposedly so so badly needed?), but not here. You mean in Windows and MacOS where much of this functionality already exists? >> >> They won't be able to keep up with the semantic enhancements. This >> >> metafiles and file-directories stuff is actually fairly trivial stuff. >> > Maybe. But the question of it being of any use aren't trivially answered >> > "yes". My gut feeling is that the answer is a resounding "no", but that's >> > just me. Direcories are directories, if you want to ship directory-like >> > stuff around, use directories (or tarfiles, or whatever). Just look what >> > happened to the much, much easier stuff of splitting SUID/SGID into >> > capabilities: Nothing at all whatsoever, because they have no decent place >> > to stay in the hallowed Unix APIs. Sure, Linux is now large enough to be >> > able to _define_ APIs to follow, but even so it isn't at all easy to do so. >> Exactly. The problem I see in this discussion is the lack of just >> this - APIs. Because the lack of good APIs we cannot make any larger >> changes or bring new features because old applications will break. I'll chop this up... > No. There is a complete lack of answers to why it should be done ("I think > it would be nice", "MSFT is trying to do it", "Look at MacOS" is far from > enough) No perhaps not by itself. But it certainly should be considered. These two OS'es have most of the market share. They have great resources trying to make things that work for their users. Users are more important than you think. Linux and any OS can do what it wants, but if it wants to keep users then it has to listen to them. > , on what the alternatives are ("Sorry, it must be in the kernel > because it is impossible to get application writers to agree on how to do > it" This is a perfectly valid reason. If it is necessary to move something into the core of Linux because it couldn't otherwise be done then it should seriously be considered. > will just add another incompatible way of "doing almost the same", this > time in the kernel), People on this list are actually trying to make it to be fully compatible with old legacy applications. If you hadn't noticed, Reiser4 does work. The meta files do work. The discussion is wether parts of Reiser4 should be made generic and fit into the VFS. > and last but not least what the impact is (performance > wise, application breakage, user/sysadmin expectation breakage, > incompatibility with other systems, kernel development impact, to list a > few). Hash this out, and we can talk. You paint up a doomsday picture here. Nothing of this magnitude has happened and won't. besides, what is wrong if sysadmins would have to learn something new? >> I think that Linux should make new API that will completely separate >> applications from the storage media. This way we will not as easily >> break applications when new features are developed. This should be >> the main priority. > The "files + directories" stuff is a nice abstraction, and works today on > all random-access storage media. You don't care much if it is on a CD, a > floppy, a disk, a RAID array, or a flash card, do you? It won't work on > tapes, or printers; but that is to be expected. you didn't actually comment on what I said. I mentioned API's nothing about directories and files. API being the keyword here. If there are good API's that applications can use, for example to access filesystems, then it will be easier and much more possible to introduce new things into the filesystem without breaking existing applications. One example of preventing application breakage could be to have a versioning system. Applications could query which version is supported, or vice versa. From what I understand it is much harder to do anything today because of the fixed ways. This is why we have this discussion. > [...] >> Application breakage has happened before between virtually every >> kernel change. It isn't new. Just take devfs and sysfs. > Oops, now emacs stopped working because devfs was added? gcc needed a mayor > overhaul because of /sys vs /proc? Gimme a break. That was _basic system tool_ > breakage, not all-around application breakage. Emacs seem to work fine with the file-as-dir on my Reiser4 partition. And so does actually any other application I have used on it. Be it in Gnome, KDE or in plain shell's. >> >> Look guys, in 1993 I anticipated the battle would be here, and I build >> >> the foundation for a defensive tower right at the spot MS and Apple are >> >> now maneuvering towards. >> >> > Why place a protective tower around the pit into which they are trying >> > desperately to throw themselves? ;-) >> >> It is easy to say this. But the fact is that users can actually use >> these new features for good things. > And can abuse them for bad things, and the features aren't free. Balance > benefits (very minor) against costs (quite large). Abuse depends on permissions and security policies which is your job as admin to look after. It would be possible to limit, for example), the types of plugins that a user can either use or load. Do you give your users access to modprobe or tools like that? >> >> Help me get the next level on the tower before >> >> they get here. It is one hell of a foundation, they won't be able to >> >> shake it, their trees are not as powerful. Don't move reiser4 into vfs, >> >> use reiser4 as the vfs. Don't write filesystems, write file plugins and >> >> disk format plugins and all the other kinds of plugins, and you won't be >> >> missing any expressive power that you really want.... >> >> > Better write libraries handling whatever weird format you have in mind for >> > each use. Even applications. I don't expect to be able to use emacs to edit >> > a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit >> > a poem. The current trend (which I heartily agree with) is to take stuff >> > _out_ of the kernel (for flexibility, access, development ease, even >> > performance). >> If I want to be able to quickly find files by a certain author I >> could search the meta-data fields (if they are used) with normal >> tools such as grep. > Iff the metadata is there. It won't (everybody would have to agree on > placing that on the files for it to work), so it is useless. No it is not useless. It just means the file has no meta information. Just as it is today. No change. >> Compression or encryption is another example. >> >> Would you rather patch every application there is for Linux to >> enable support for reading these files and directories? > No. But an encrypted or compressed filessytem doesn't need weird files. > Easiest on applications is just to serve the decrypted/uncompressed files > for use via the traditional APIs. Next easiest is provide programs that > handle cryptography and compression and can be combined with whatever > unaware program you care to use. Funny, that is exactly the way things work > now... So you can setup a directory that has some files compressed and some other encrypted and yet they will be available to use with tar, cp, ls, mc, Nautilus, Emacs etc etc? ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 [not found] ` <200408291521.i7TFLsQk028363@localhost.localdomain> 2004-08-29 16:52 ` Spam 2004-08-29 16:53 ` Spam @ 2004-08-29 16:54 ` Spam 2 siblings, 0 replies; 1392+ messages in thread From: Spam @ 2004-08-29 16:54 UTC (permalink / raw) To: Horst von Brand Cc: Hans Reiser, Helge Hafting, Rik van Riel, Spam, Jamie Lokier, David Masover, Diego Calleja, christophe, vda, christer, Andrew Morton, wichert, jra, hch, linux-fsdevel, Kernel Mailing List, flx, reiserfs-list > Spam <spam@tnonline.net> said: >> Horst von Brand <vonbrand@inf.utfsm.cl> said: >> > Hans Reiser <reiser@namesys.com> said: >> >> Andrew Morton wrote: > [...] >> >> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem >> >> for Apple, >> >> > Says who? >> >> At least I as a user see the benefits largely. The workstation / >> desktop user has much more need for such features, than perhaps the >> normal Linux sysadmin. > Show benefits, show there are no downsides. Show that the same can't > possibly be done in userland. Then I'll consider buying it. I am not saying it couldn't be done in userland. Most things can. The problem with userland is that it doesn't provide generic access across most applications. For example GnomeVFS is limited to Gnome applications only. The same for KDE. to be able to coordinate application developer on anything is just almost impossible. Still. Why do you oppose plugins, streams and meta files? The could be valuable and easy to use tools for many purposes. One could be would be advanced ACLs defined as a meta-file using XML format. > AFAICS: Smallish benefits for some narrow application niche. Mayor problems > (no decent multiuser way of using it, extremely complex stuff placed in the > kernel). Userland solutions are all around us right now, and work just > fine. No, placing the same gunk in the kernel won't make it magically use > no time or memory, more like exactly the opposite. Besides, all the visions > waved around here have very little in common (except the idea that they > make all restrictions magically dissapear). > Get a list of required features, show concrete ways of using it, outline > possible implementations, discuss resource usage. I.e., do "engineering". > Then we have something to discuss. I am not sure what you refer to as "required features". Do you mean example of usage and contents for streams, plugins and meta files? >> >> and that means we need to put search engine and database >> >> functionality into the filesystem. >> > Please don't. Unix works and is extremely popular because it _doesn't_ try >> > to be smart (policy vs mechanism distinction, simple abstractions that can >> > be combined endlessly). If you add this to the filesystems, you either redo >> > _all_ userland to understand _one_ particular way of doing "smart things" >> > (which will turn out to be almost exactly the dumbest possible for some >> > uses, and then you are stuck), or get lots of shards from broken apps (and >> > users, and sysadmins). >> Indeed. But you forget one thing. Combining lots of small userland >> tools can actually make things more complex - at least for the user. > So? Placing lots of little things in kernel does the same, plus screws over > sysadmins, and makes kernel hacker's job harder. I prefer the former alone. Difference being that it could be transparent in a way otherwise not possible. >> That said, I do not see that anyone has suggested to remove this way >> of working with applications. > Excuse me, if files suddenly become directories, and you can't have some > "reserved" file/directory names, and... you are breaking lots of stuff. >> Are the new and old features mutually exclusive? > Yep. >> >> It takes 11 years of serious >> >> research to build a clean storage layer able to handle doing that. >> >> > Great! Build an experimental OS showing how to use it, and through that see >> > if the ideas hold any water _in real, day-to-day, down-to-earth, practice_. >> There are many uses already. > Where? >> It could be encryption, compression, >> meta-data, file streams, sorting, searching and a multitude of uses >> and ideas that haven't been discovered yet. > "Could", "multitude ... haven't been discovered yet" is _not_ "many current > uses". >> >> Reiser4 has done that, finally. None of the other Linux filesystems >> >> have. >> > How come nobody before you in the almost 30 years of Unix has ever seen the >> > need for this indispensable feature? >> I, as a user, have for a long time. Do not drag everyone over the >> same edge. Users have different needs. We should try to be as >> forthcoming as possible to provide the tools and options for them. > Why? Breaking the basic paradigms for some user whim is not good OS design > in my book, sorry. Many "bright ideas" have been thrown out over the years > for just the same reasons. >> >> The next major release of ReiserFS is going to be bursting with >> >> semantic enhancements, because the prerequisites for them are in place >> >> now. None of the other Linux filesystems have those prerequisites. >> > To me that would mean that ReiserFS has no place in Linux. >> Why do you say this? I thought Linux embraced advancements and new >> technologies. > If they are right. I think breaking the base of the OS is not a good > idea. Perhaps ReiserFS has a place elsewhere (in MacOS, or Windows, where > this is supposedly so so badly needed?), but not here. You mean in Windows and MacOS where much of this functionality already exists? >> >> They won't be able to keep up with the semantic enhancements. This >> >> metafiles and file-directories stuff is actually fairly trivial stuff. >> > Maybe. But the question of it being of any use aren't trivially answered >> > "yes". My gut feeling is that the answer is a resounding "no", but that's >> > just me. Direcories are directories, if you want to ship directory-like >> > stuff around, use directories (or tarfiles, or whatever). Just look what >> > happened to the much, much easier stuff of splitting SUID/SGID into >> > capabilities: Nothing at all whatsoever, because they have no decent place >> > to stay in the hallowed Unix APIs. Sure, Linux is now large enough to be >> > able to _define_ APIs to follow, but even so it isn't at all easy to do so. >> Exactly. The problem I see in this discussion is the lack of just >> this - APIs. Because the lack of good APIs we cannot make any larger >> changes or bring new features because old applications will break. I'll chop this up... > No. There is a complete lack of answers to why it should be done ("I think > it would be nice", "MSFT is trying to do it", "Look at MacOS" is far from > enough) No perhaps not by itself. But it certainly should be considered. These two OS'es have most of the market share. They have great resources trying to make things that work for their users. Users are more important than you think. Linux and any OS can do what it wants, but if it wants to keep users then it has to listen to them. > , on what the alternatives are ("Sorry, it must be in the kernel > because it is impossible to get application writers to agree on how to do > it" This is a perfectly valid reason. If it is necessary to move something into the core of Linux because it couldn't otherwise be done then it should seriously be considered. > will just add another incompatible way of "doing almost the same", this > time in the kernel), People on this list are actually trying to make it to be fully compatible with old legacy applications. If you hadn't noticed, Reiser4 does work. The meta files do work. The discussion is wether parts of Reiser4 should be made generic and fit into the VFS. > and last but not least what the impact is (performance > wise, application breakage, user/sysadmin expectation breakage, > incompatibility with other systems, kernel development impact, to list a > few). Hash this out, and we can talk. You paint up a doomsday picture here. Nothing of this magnitude has happened and won't. besides, what is wrong if sysadmins would have to learn something new? >> I think that Linux should make new API that will completely separate >> applications from the storage media. This way we will not as easily >> break applications when new features are developed. This should be >> the main priority. > The "files + directories" stuff is a nice abstraction, and works today on > all random-access storage media. You don't care much if it is on a CD, a > floppy, a disk, a RAID array, or a flash card, do you? It won't work on > tapes, or printers; but that is to be expected. you didn't actually comment on what I said. I mentioned API's nothing about directories and files. API being the keyword here. If there are good API's that applications can use, for example to access filesystems, then it will be easier and much more possible to introduce new things into the filesystem without breaking existing applications. One example of preventing application breakage could be to have a versioning system. Applications could query which version is supported, or vice versa. From what I understand it is much harder to do anything today because of the fixed ways. This is why we have this discussion. > [...] >> Application breakage has happened before between virtually every >> kernel change. It isn't new. Just take devfs and sysfs. > Oops, now emacs stopped working because devfs was added? gcc needed a mayor > overhaul because of /sys vs /proc? Gimme a break. That was _basic system tool_ > breakage, not all-around application breakage. Emacs seem to work fine with the file-as-dir on my Reiser4 partition. And so does actually any other application I have used on it. Be it in Gnome, KDE or in plain shell's. >> >> Look guys, in 1993 I anticipated the battle would be here, and I build >> >> the foundation for a defensive tower right at the spot MS and Apple are >> >> now maneuvering towards. >> >> > Why place a protective tower around the pit into which they are trying >> > desperately to throw themselves? ;-) >> >> It is easy to say this. But the fact is that users can actually use >> these new features for good things. > And can abuse them for bad things, and the features aren't free. Balance > benefits (very minor) against costs (quite large). Abuse depends on permissions and security policies which is your job as admin to look after. It would be possible to limit, for example), the types of plugins that a user can either use or load. Do you give your users access to modprobe or tools like that? >> >> Help me get the next level on the tower before >> >> they get here. It is one hell of a foundation, they won't be able to >> >> shake it, their trees are not as powerful. Don't move reiser4 into vfs, >> >> use reiser4 as the vfs. Don't write filesystems, write file plugins and >> >> disk format plugins and all the other kinds of plugins, and you won't be >> >> missing any expressive power that you really want.... >> >> > Better write libraries handling whatever weird format you have in mind for >> > each use. Even applications. I don't expect to be able to use emacs to edit >> > a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit >> > a poem. The current trend (which I heartily agree with) is to take stuff >> > _out_ of the kernel (for flexibility, access, development ease, even >> > performance). >> If I want to be able to quickly find files by a certain author I >> could search the meta-data fields (if they are used) with normal >> tools such as grep. > Iff the metadata is there. It won't (everybody would have to agree on > placing that on the files for it to work), so it is useless. No it is not useless. It just means the file has no meta information. Just as it is today. No change. >> Compression or encryption is another example. >> >> Would you rather patch every application there is for Linux to >> enable support for reading these files and directories? > No. But an encrypted or compressed filessytem doesn't need weird files. > Easiest on applications is just to serve the decrypted/uncompressed files > for use via the traditional APIs. Next easiest is provide programs that > handle cryptography and compression and can be combined with whatever > unaware program you care to use. Funny, that is exactly the way things work > now... So you can setup a directory that has some files compressed and some other encrypted and yet they will be available to use with tar, cp, ls, mc, Nautilus, Emacs etc etc? ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 18:12 ` silent semantic changes with reiser4 Horst von Brand 2004-08-27 20:32 ` Markus Törnqvist 2004-08-27 20:44 ` Spam @ 2004-09-01 21:43 ` Jamie Lokier 2 siblings, 0 replies; 1392+ messages in thread From: Jamie Lokier @ 2004-09-01 21:43 UTC (permalink / raw) To: Horst von Brand; +Cc: linux-kernel, linux-fsdevel, reiserfs-list Horst von Brand wrote: > Build an experimental OS showing how to use it, and through that see > if the ideas hold any water _in real, day-to-day, down-to-earth, practice_. Hey, if you can think of a way to build an experimental OS and get it used in real, day-to-day, down-to-earth, practice - i.e. used by a diverse audience of thousands of vocal groups doing different things with it, please let me know. Until you do we have to lobby for some things to be tried in Linux before we know if they really work. -- Jamie ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: silent semantic changes with reiser4 2004-08-26 8:31 ` Hans Reiser ` (3 preceding siblings ...) 2004-08-26 18:12 ` silent semantic changes with reiser4 Horst von Brand @ 2004-08-26 20:04 ` Martin J. Bligh 2004-08-27 21:23 ` vfs2 (was Re: silent semantic changes with reiser4) Pavel Machek 5 siblings, 0 replies; 1392+ messages in thread From: Martin J. Bligh @ 2004-08-26 20:04 UTC (permalink / raw) To: Hans Reiser, Andrew Morton Cc: hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list > Don't move reiser4 into vfs, use reiser4 as the vfs. Ah ... that seems to be a good insight into what you're driving for. That would seem to be good for you ... and bad for other filesystems. So whilst I can see why you'd want it - but I don't think it's a good idea. Competition is good. Andrew wrote: > And what are the licensing implications of plugins? Are they derived > works? Must they be GPL'ed? You replied: >>And what are the licensing implications of plugins? Are they derived >>works? > > Yes. I find it odd that you cut out the last part of his question. So must they be GPL'ed or not? M. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* vfs2 (was Re: silent semantic changes with reiser4) 2004-08-26 8:31 ` Hans Reiser ` (4 preceding siblings ...) 2004-08-26 20:04 ` Martin J. Bligh @ 2004-08-27 21:23 ` Pavel Machek 5 siblings, 0 replies; 1392+ messages in thread From: Pavel Machek @ 2004-08-27 21:23 UTC (permalink / raw) To: Hans Reiser Cc: Andrew Morton, hch, linux-fsdevel, linux-kernel, flx, torvalds, reiserfs-list Hi! > >Yes, but I didn't say "flame Christoph and ignore the issues" ;) > > > > > Oh....;-) ... > enhancements. This metafiles and file-directories stuff is actually > fairly trivial stuff. Now I'm scared. not as powerful. Don't > move reiser4 into vfs, use reiser4 as the vfs. Don't write > filesystems, write file plugins and disk format plugins and all the > other kinds of plugins, and you won't be missing any expressive power > that you really want.... So you want to replace vfs. Fine, but you should have told us before. Also vfs replacement probably should not be hidden in fs/reiserfs4... -- 64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms ^ permalink raw reply [flat|nested] 1392+ messages in thread
* file-as-dir vs. dir? (was Re: silent semantic changes with reiser4) 2004-08-24 20:25 silent semantic changes with reiser4 Christoph Hellwig ` (2 preceding siblings ...) 2004-08-25 19:53 ` Hans Reiser @ 2004-09-14 4:27 ` Andy Lutomirski 3 siblings, 0 replies; 1392+ messages in thread From: Andy Lutomirski @ 2004-09-14 4:27 UTC (permalink / raw) To: linux-fsdevel, linux-kernel (Apologies for beating on a dead horse here...) I seems like one confusion here is that everyone has a different idea of what the semantics we're talking about are. I see two main ones: Hans's: A file, a directory, and an attribute are functionally equivalent (except for S_ISxxx and hardlinks). That is, /usr/bin/metas makes sense, and it's not talking about a program called metas. This also means that /foo/metas/metas might exist and needs dealing with. Linus's (I think): A directory is just a directory (no attributes and no read()able data). A file can contain attributes, where attributes can be "file" attributes or "directory" attributes. That is, a file is also a subtree with posix-like semantics (except for hardlink stuff). So doing "touch /tmp/foo; cat /tmp/foo/metas" fails, rather than doing something that's probably useless. "touch /tmp/foo; touch /tmp/foo/bar; touch /tmp/foo/bar/baz" fails on the last touch because bar is a file attribute and recursive magic is disallowed. Which one are we talking about? FWIW, I like the latter version a lot better, as it removes a lot of ambiguity. If I see a path like /tmp/foo/metas/uid, it is either a uid attribute (i.e. writing it has security implications) or it is a standard file (i.e. writing it just writes it). But _I can tell which one_ by fstat()ing /tmp/foo! If it's a directory, than I have either a named stream called uid or a genuine file called uid (and I can tell which by fstat()ing /tmp/foo/metas), but if it's a file then I have the magic uid. And I'm guaranteed that there's no other funny business, because /tmp/foo is a "file with a subtree," which means that it's not an attribute. This way I know exactly what I'm dealing with. As an added bonus, we could have O_NOMETAS which means that "files" may not be traversed. Then someone who wants to make sure they get a real file can do it. If recursive files-as-dirs were allowed, that might not do quite what the caller expected. If you really wanted a directory with some data attached, but it in dir/.mystuff, because that's probably what you meant, and any existing tool will do exactly the right thing with it. In the former version, AFAIK, I have a mess. In UNIX, a path is a path. It's trivial to understand and there's no funny business parsing it. But now a path has a slippery meaning where the token "metas" could be a plain old token or it could be magic or it could be magic some other way. The security implications and the possbility for something to break horribly are scary. --Andy ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins @ 2005-06-29 5:09 Horst von Brand 2005-06-29 20:40 ` Hubert Chan 0 siblings, 1 reply; 1392+ messages in thread From: Horst von Brand @ 2005-06-29 5:09 UTC (permalink / raw) To: Hubert Chan Cc: Kyle Moffett, David Masover, Valdis.Kletnieks, Lincoln Dale, Gregory Maxwell, Hans Reiser, Horst von Brand, Jeff Garzik, Christoph Hellwig, Andrew Morton, linux-kernel, ReiserFS List Hubert Chan <hubert@uhoreg.ca> wrote: [...] > Of course. With file-as-dir, you can "cd /usr/games/tetris/..." and > mess with the game files, or you can run "/usr/games/tetris" and get on > with ... stacking blocks. And doing "tar cf /dev/tape /usr/games/tetris" gives you a nice tangle of undecipherable junk. > The advantage of doing it in the kernel is > that you don't need extra support from the applications (or GnomeVFS or > KDE). I don't see any advantage here... it has to be implemented, and doing it in-kernel for one filesystem is /much/ harder than doing it in userland, where it'll probably work whatever the underlying filesystem is. Besides, why should the Tetris scores (or icon, or whatever) reside inside the executable, when said executable is perhaps shared by a bunch of machines on the network, each of which would like to keep its own? Or perhaps the sysadmin is terminally paranoid, and mounts /usr read-only (Hey, the FHS people defined what goes in there with /exactly/ that in mind too!). > So typing "/usr/games/tetris" from the command prompt does the > same thing as double-clicking it in the file manager. Right. And what should happen if I run the (logically equivalent) directory /home? Or /etc? What if I want to shove a directory into the tetris executable? Symlinks? Hard links to files inside? Outside? Symlinks from the outside in? Hardlinks? And if you now move that to another filesystem, or ship it to another machine? No, the answer "All that will be forbidden" is /not/ allowed, you want this stuff working "normally". "Unpack" the contents into real files and directories how? "Pack" a directory into one of those structured objects? What would be the /practical/ difference between the packed and unpacked forms? (If none, why do it in the first place? If the packed version has significant restrictions, why use it? If the packed version does have significant extra capabilities, why not just give those to regular objects?) Yet again, what somebody (I forget who) called the "Zero - one - infinity principle" (I think it was in the area of programming languages; which arguably are very complex user interfaces, just like what an operating system provides): It only makes sense giving zero, exactly one, or how many you wish for of some item/nesting. Here: Either files got /no/ strange things inside, or /exactly one/ (and then it becomes questionable, as there is no space for regular data anymore...), or whatever complex (sub)structure you want. But in the last case, there is /no/ difference between a file and a directory... and the whole setup is just mess for mess sake, nothing else. > Of course, this > change does require file managers to understand default actions when > it's ambiguous what to do on a double-click -- but MacOS X has that too, > in the form of the "Open as Folder" option (or whatever it's called). Right. For the sake of a filesystem among many on a minority operating system /all/ GUI programs have to be rewritten. And all command-line stuff. Just because. Please /do/ think the above through, giving reasonable answers for /all/ of the operations mentioned. Tell what the advantage of all this would be on a multiuser operating system, when files are shared via the net, when files are being handled from read-only media (or filesystems). When different users want different metadata (I'm interested in the names of members of the band playing a song, you might want a high-resolution image of the album cover, the next guy wants the song's lyrics translated into swahili, all for the MP3s on the shared server or on the CD each of us bought separately). Consider the scenario where all this works only on /a very few/ machines (no Sun or *BSD box will have this for a very long time, and to get a significant fraction of just Linux systems will take some time), for only a limited selection of tools (existing tools will have to be rewritten or at least adapted, that doesn't happen overnight). Factor in space and processing requirements for several, even hundreds, of concurrent users (or versions of metadata at least). How do you account for that? The 1 byte file with GiB of metadata where everybody and their pet iguana store their junk is handled how by quota systems? What should happen if I email such a file with several versions of metadata to someone? Do they get just my metadata, or everybody's? If I copy it to, say, a CD for safekeeping for myself? For system backup purposes? What if I copy such a file (with only my, or everybody's) metadata back from a backup? Or try to merge it with the version I took with my notebook on a trip, changing my parts while the otghers change theirs? I'd assume all of those will requiere special tools, or at least flags selecting the right operation or some other unspecified magic, and the "simple to use" just went out for lunch. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2005-06-29 5:09 reiser4 plugins Horst von Brand @ 2005-06-29 20:40 ` Hubert Chan 2005-06-29 21:34 ` Ross Biro 0 siblings, 1 reply; 1392+ messages in thread From: Hubert Chan @ 2005-06-29 20:40 UTC (permalink / raw) To: Horst von Brand Cc: Kyle Moffett, David Masover, Valdis.Kletnieks, Lincoln Dale, Gregory Maxwell, Hans Reiser, Jeff Garzik, Christoph Hellwig, Andrew Morton, linux-kernel, ReiserFS List On Wed, 29 Jun 2005 01:09:05 -0400, Horst von Brand <vonbrand@inf.utfsm.cl> said: > Hubert Chan <hubert@uhoreg.ca> wrote: [...] >> Of course. With file-as-dir, you can "cd /usr/games/tetris/..." and >> mess with the game files, or you can run "/usr/games/tetris" and get >> on with ... stacking blocks. > And doing "tar cf /dev/tape /usr/games/tetris" gives you a nice tangle > of undecipherable junk. Uh, yeah. I thought we already discussed this in the last flamewar about merging Reiser4. You can patch tar/cp/backup programs if you want to go the file-as-dir route, or you can try to patch all the other programs if you want to go the xattr route and you really want to edit you extended attributes using $EDITOR (which I'll admit not everyone wants to do). >> The advantage of doing it in the kernel is that you don't need extra >> support from the applications (or GnomeVFS or KDE). > I don't see any advantage here... it has to be implemented, and doing > it in-kernel for one filesystem is /much/ harder than doing it in > userland, where it'll probably work whatever the underlying filesystem > is. Only filesystems that support the capability, or if you have an appropriate filename munging system (see below) or some other system to emulate the capability. > Besides, why should the Tetris scores (or icon, or whatever) reside > inside the executable, when said executable is perhaps shared by a > bunch of machines on the network, each of which would like to keep its > own? Or perhaps the sysadmin is terminally paranoid, and mounts /usr > read-only (Hey, the FHS people defined what goes in there with > /exactly/ that in mind too!). Scores, no. But Tetris still may have resource data that logically belongs to it. Like icons, internationalization data, UI data (say a Glade file), etc. It's all part of the same package; it would be nice (IMHO) if it was grouped together in the filesystem. I don't think MacOS X stores scores in the app file, but they have a lot of other stuff in there -- stuff that I mentioned in the previous paragraph. >> So typing "/usr/games/tetris" from the command prompt does the same >> thing as double-clicking it in the file manager. > Right. And what should happen if I run the (logically equivalent) > directory /home? Or /etc? If there's no binary data attached to /home or /etc (i.e. it only has directory contents), then nothing happens (other than an error message). That's like asking what happens if I do "cat nonexistent.file". If there's no data there to work with, then nothing happens. > What if I want to shove a directory into the tetris executable? > Symlinks? Hard links to files inside? Outside? Symlinks from the > outside in? Hardlinks? What problems do you see with those? > And if you now move that to another filesystem, or ship it to another > machine? What happens when you copy a file from NTFS that has multiple streams? Or a file that has extended attributes onto a filesystem that doesn't support xattr? > No, the answer "All that will be forbidden" is /not/ allowed, you want > this stuff working "normally". "Unpack" the contents into real files > and directories how? If you really want to, you can do some filename munging. The contents of the /usr/games/tetris/ directory could be put in /usr/games/...tetris, or something like that. > "Pack" a directory into one of those structured objects? What would be > the /practical/ difference between the packed and unpacked forms? It's really, really ugly. ;-) It's possible to just put all my files all in a single directory. But sorting things into different directories makes things look much nicer. > (If none, why do it in the first place? If the packed version has > significant restrictions, why use it? If the packed version does have > significant extra capabilities, why not just give those to regular > objects?) > Yet again, what somebody (I forget who) called the "Zero - one - > infinity principle" (I think it was in the area of programming > languages; which arguably are very complex user interfaces, just like > what an operating system provides): It only makes sense giving zero, > exactly one, or how many you wish for of some item/nesting. Here: > Either files got /no/ strange things inside, or /exactly one/ (and > then it becomes questionable, as there is no space for regular data > anymore...), or whatever complex (sub)structure you want. But in the > last case, there is /no/ difference between a file and a > directory... and the whole setup is just mess for mess sake, nothing > else. Files would have two "forks" (can't think of any better terminology right now). A data fork, and a directory fork. --- Begin useless analogy --- Sort of like a real folder in meatspace. I can talk about what I've written on the folder (because I can write stuff on real physical folders), or I can talk about what's inside the folder. Or if I have a document, I can talk about the document itself, or the random post-its, notes, and paperclips that I have attached to it. ---- End useless analogy ---- Right now, most Linux filesystems have xattrs and acls (stored as xattrs). Maybe in the future, the Samba people will get support for multiple streams. If I want to copy my NTFS file to ext3 and preserve the multiple streams, then ext3 will need to support multiple streams now. Now we have two strange things inside. Why not just unify the interface using something like file-as-dir, or metafs that was proposed in this thread, or openat, or something like that? >> Of course, this change does require file managers to understand >> default actions when it's ambiguous what to do on a double-click -- >> but MacOS X has that too, in the form of the "Open as Folder" option >> (or whatever it's called). > Right. For the sake of a filesystem among many on a minority operating > system /all/ GUI programs have to be rewritten. And all command-line > stuff. Just because. Not rewritten; patched. And not all programs. Actually, most programs would work just fine. The Namesys people can tell you how many programs didn't work properly when they tried out the metas directory, and how much patching was required to get them to work properly again. (Not counting tar/cp/backup programs. Everyone already knows about the problems there.) > Please /do/ think the above through, giving reasonable answers for > /all/ of the operations mentioned. Tell what the advantage of all this > would be on a multiuser operating system, when files are shared via > the net, when files are being handled from read-only media (or > filesystems). When different users want different metadata (I'm > interested in the names of members of the band playing a song, you > might want a high-resolution image of the album cover, the next guy > wants the song's lyrics translated into swahili, all for the MP3s on > the shared server or on the CD each of us bought separately). Consider > the scenario where all this works only on /a very few/ machines (no > Sun or *BSD box will have this for a very long time, and to get a > significant fraction of just Linux systems will take some time), for > only a limited selection of tools (existing tools will have to be > rewritten or at least adapted, that doesn't happen overnight). xattr is being added to most filesystems in Linux. It doesn't work everywhere, in all operating systems. Programs will have to be patched to work with it. Does that mean that xattr is a bad idea? > Factor in space and processing requirements for several, even > hundreds, of concurrent users (or versions of metadata at least). How > do you account for that? The 1 byte file with GiB of metadata where > everybody and their pet iguana store their junk is handled how by > quota systems? How do you handle it now when everyone wants different information on a song? I don't really see what the problem is here. Note: I'm not saying that keeping everyone's pet metadata with shared files is a good idea. This is your example. The local library doesn't allow me to make random annotations to their books, but I can do whatever I want with my own books. > What should happen if I email such a file with several versions of > metadata to someone? Do they get just my metadata, or everybody's? If I email a picture that I took to a friend, I don't send them the whole 6 megapixel picture in RAW format. I convert to JPEG and scale it down. If I email a music file that has the lyrics translated into swahili, I can strip that bit out. Nobody's forcing you to keep everything. > If I copy it to, say, a CD for safekeeping for myself? For system > backup purposes? What if I copy such a file (with only my, or > everybody's) metadata back from a backup? You do whatever you want to do with your data. If you want to keep the pet iguana's metadata, go ahead. > Or try to merge it with the version I took with my notebook on a trip, > changing my parts while the otghers change theirs? I'd assume all of > those will requiere special tools, or at least flags selecting the > right operation or some other unspecified magic, and the "simple to > use" just went out for lunch. So what do you do right now when you want to do something like that? File-as-dir (or the other proposals) won't prevent you from doing that. It just allows you more options. -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2005-06-29 20:40 ` Hubert Chan @ 2005-06-29 21:34 ` Ross Biro 2005-06-29 23:29 ` Hubert Chan 0 siblings, 1 reply; 1392+ messages in thread From: Ross Biro @ 2005-06-29 21:34 UTC (permalink / raw) To: Hubert Chan Cc: Horst von Brand, Kyle Moffett, David Masover, Valdis.Kletnieks, Lincoln Dale, Gregory Maxwell, Hans Reiser, Jeff Garzik, Christoph Hellwig, Andrew Morton, linux-kernel, ReiserFS List On 6/29/05, Hubert Chan <hubert@uhoreg.ca> wrote: > On Wed, 29 Jun 2005 01:09:05 -0400, Horst von Brand <vonbrand@inf.utfsm.cl> said: > > > Hubert Chan <hubert@uhoreg.ca> wrote: [...] > > And doing "tar cf /dev/tape /usr/games/tetris" gives you a nice tangle > > of undecipherable junk. > I'm confused. Can someone on one of these lists enlighten me? How is directories as files logically any different than putting all data into .data files and making all files directories (yes you would need some sort of special handling for files that were really called .data). Then it's just a matter of deciding what happens when you call open and stat on one of these files? For backwards compatibility, current existing system calls have to treat these things as directories. Perhaps an exception could be made for exec. But we could have a whole new set of system calls that treat things as magic, and if files as directories is as cool as many people think, apps will start using the new api. If not, they won't and the new api can be deprecated. Ross ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2005-06-29 21:34 ` Ross Biro @ 2005-06-29 23:29 ` Hubert Chan 2005-07-01 8:06 ` David Masover 0 siblings, 1 reply; 1392+ messages in thread From: Hubert Chan @ 2005-06-29 23:29 UTC (permalink / raw) To: Ross Biro Cc: Horst von Brand, Kyle Moffett, David Masover, Valdis.Kletnieks, Lincoln Dale, Gregory Maxwell, Hans Reiser, Jeff Garzik, Christoph Hellwig, Andrew Morton, linux-kernel, ReiserFS List On Wed, 29 Jun 2005 17:34:41 -0400, Ross Biro <ross.biro@gmail.com> said: > I'm confused. Can someone on one of these lists enlighten me? > How is directories as files logically any different than putting all > data into .data files and making all files directories (yes you would > need some sort of special handling for files that were really called > .data). Then it's just a matter of deciding what happens when you > call open and stat on one of these files? Logically, I don't think there is a difference. A filesystem that doesn't support file-as-dir could implement the same functionality that way. [1] In fact, that's essentially what MacOS X/NeXTSTEP does with its bundle format -- it's just a regular directory with regular files inside. The Reiser4 implementation would probably look very similar to that structurally, except that since it understands file-as-dir, it doesn't need to use the .data name, and it should still know where to find it. (Of course, I'm not a Reiser4 developer, and I don't speak for them.) [1] (Well, except that file-as-dir as some people would like it to operate, can do a whole lot of other magic. (e.g. transparent compress, etc. In Reiser4, it's also supposed to allow an interface to filesystem stuff so that you can manipulate parameters such as whether it should use the cryptocompress plugin for that file.) But I think this is a sort of separate issue.) > For backwards compatibility, current existing system calls have to > treat these things as directories. Perhaps an exception could be made > for exec. Which makes the system really quite ugly. I certainly wouldn't enjoy having to type "$EDITOR foo.txt/.data" instead of "$EDITOR foo.txt". If you want to introduce new system calls, you would have to patch all the programs pretty much overnight in order for the system to be useful. It also makes portability a big pain if you want to support systems that don't offer the same system calls. You also get a chicken-and-egg problem. Application developers don't patch their programs because nobody's using the system. Nobody's using the system because no applications support it. > But we could have a whole new set of system calls that treat things as > magic, and if files as directories is as cool as many people think, > apps will start using the new api. If not, they won't and the new api > can be deprecated. File-as-dir doesn't require new system calls (that I know of), which is the whole point of the idea. Existing programs can edit the strange new attributes without being modified. The main thing blocking file-as-dir is that there are some locking(IIRC?) issues. And, of course, some people wouldn't want it to be merged into the mainline kernel. (Of course, the latter doesn't prevent Namesys from maintaining their own patches for people to play around with.) I would like to see file-as-dir so that we can try it out to see if it's as useful as some of us think it is. It might be less useful than I expect, or it could exceed my expectations and someone might come up with some killer use for it, once we can start playing with it. And I think it would be nice if it was merged into mainline so that it gets more exposure, and we have more people trying it out. People like Horst (and probably others, who are less vocal), I think, don't think that it's even worth trying it out because they don't see any major advantages. Or at least they think that the potential negatives outweigh the potential positives. I respect that they have different opinions, but I of course disagree and attempt to convince them otherwise. -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2005-06-29 23:29 ` Hubert Chan @ 2005-07-01 8:06 ` David Masover 2005-07-05 21:01 ` Hubert Chan 0 siblings, 1 reply; 1392+ messages in thread From: David Masover @ 2005-07-01 8:06 UTC (permalink / raw) To: Hubert Chan Cc: Ross Biro, Horst von Brand, Kyle Moffett, Valdis.Kletnieks, Lincoln Dale, Gregory Maxwell, Hans Reiser, Jeff Garzik, Christoph Hellwig, Andrew Morton, linux-kernel, ReiserFS List Hubert Chan wrote: > On Wed, 29 Jun 2005 17:34:41 -0400, Ross Biro <ross.biro@gmail.com> said: > > >>I'm confused. Can someone on one of these lists enlighten me? > > >>How is directories as files logically any different than putting all >>data into .data files and making all files directories (yes you would >>need some sort of special handling for files that were really called >>.data). Then it's just a matter of deciding what happens when you >>call open and stat on one of these files? > > > Logically, I don't think there is a difference. A filesystem that > doesn't support file-as-dir could implement the same functionality that > way. [1] In fact, that's essentially what MacOS X/NeXTSTEP does with its > bundle format -- it's just a regular directory with regular files > inside. I, personally, would hate it if everything in my /bin suddenly became a directory, mainly because everything would stop working. Is that the kind of thing you're suggesting? I'm a little confused about the .data idea, I guess. >>But we could have a whole new set of system calls that treat things as >>magic, and if files as directories is as cool as many people think, >>apps will start using the new api. If not, they won't and the new api >>can be deprecated. > > > File-as-dir doesn't require new system calls (that I know of), which is > the whole point of the idea. Existing programs can edit the strange new > attributes without being modified. That is indeed the point, but scroll down. > The main thing blocking file-as-dir is that there are some > locking(IIRC?) issues. And, of course, some people wouldn't want it to > be merged into the mainline kernel. (Of course, the latter doesn't > prevent Namesys from maintaining their own patches for people to play > around with.) What's the locking issue? I think that was more about transactions... [...] > People like Horst (and probably others, who are less vocal), I think, > don't think that it's even worth trying it out because they don't see > any major advantages. Or at least they think that the potential > negatives outweigh the potential positives. I respect that they have > different opinions, but I of course disagree and attempt to convince > them otherwise. Did the /meta (metafs) idea get killed while I was out? Using that approach, your potential negatives are that apps which crawl the entire FS tree, starting at /, with hardcoded apps for /proc and /sys, are now broken -- but then, /sys already broke them once, so I don't particularly care if we break them again. Potential positives? I think even just because we like the idea is enough, because it doesn't break anything and doesn't really affect anyone who doesn't use it. Maybe there are coding standards, but I think others are working that out now. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2005-07-01 8:06 ` David Masover @ 2005-07-05 21:01 ` Hubert Chan 2005-07-05 22:01 ` Hans Reiser 0 siblings, 1 reply; 1392+ messages in thread From: Hubert Chan @ 2005-07-05 21:01 UTC (permalink / raw) To: David Masover Cc: Ross Biro, Horst von Brand, Kyle Moffett, Valdis.Kletnieks, Lincoln Dale, Gregory Maxwell, Hans Reiser, Jeff Garzik, Christoph Hellwig, Andrew Morton, linux-kernel, ReiserFS List On Fri, 01 Jul 2005 03:06:19 -0500, David Masover <ninja@slaphack.com> said: > Hubert Chan wrote: >> The main thing blocking file-as-dir is that there are some >> locking(IIRC?) issues. And, of course, some people wouldn't want it >> to be merged into the mainline kernel. (Of course, the latter >> doesn't prevent Namesys from maintaining their own patches for people >> to play around with.) > What's the locking issue? I think that was more about transactions... It was whatever was Al Viro's (technical) complaint about file-as-dir. I don't remember exactly what it was. The technical people know what it is (and the Namesys guys are probably working on it), and the exact issue doesn't concern us non-technical people that much, so I don't feel like looking it up. But if you want to, just look for Al Viro's message in this thread. -- Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2005-07-05 21:01 ` Hubert Chan @ 2005-07-05 22:01 ` Hans Reiser 2005-07-05 22:21 ` David Masover 0 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2005-07-05 22:01 UTC (permalink / raw) To: Hubert Chan Cc: David Masover, Ross Biro, Horst von Brand, Kyle Moffett, Valdis.Kletnieks, Lincoln Dale, Gregory Maxwell, Jeff Garzik, Christoph Hellwig, Andrew Morton, linux-kernel, ReiserFS List Hubert Chan wrote: >On Fri, 01 Jul 2005 03:06:19 -0500, David Masover <ninja@slaphack.com> said: > > > >>Hubert Chan wrote: >> >> > > > >>>The main thing blocking file-as-dir is that there are some >>>locking(IIRC?) issues. And, of course, some people wouldn't want it >>>to be merged into the mainline kernel. (Of course, the latter >>>doesn't prevent Namesys from maintaining their own patches for people >>>to play around with.) >>> >>> > > > >>What's the locking issue? I think that was more about transactions... >> >> > >It was whatever was Al Viro's (technical) complaint about file-as-dir. >I don't remember exactly what it was. The technical people know what it >is (and the Namesys guys are probably working on it), and the exact >issue doesn't concern us non-technical people that much, so I don't feel >like looking it up. But if you want to, just look for Al Viro's message >in this thread. > > > Cycle detection when hard links to directories are allowed. There is a debate over whether cycle detection is feasible that can only be resolved by working code or a formal proof that it is not computationally feasible. Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2005-07-05 22:01 ` Hans Reiser @ 2005-07-05 22:21 ` David Masover 2005-07-05 22:51 ` Hans Reiser 0 siblings, 1 reply; 1392+ messages in thread From: David Masover @ 2005-07-05 22:21 UTC (permalink / raw) To: Hans Reiser Cc: Hubert Chan, Ross Biro, Horst von Brand, Kyle Moffett, Valdis.Kletnieks, Lincoln Dale, Gregory Maxwell, Jeff Garzik, Christoph Hellwig, Andrew Morton, linux-kernel, ReiserFS List Hans Reiser wrote: > Hubert Chan wrote: > > >>On Fri, 01 Jul 2005 03:06:19 -0500, David Masover <ninja@slaphack.com> said: >> >> >> >> >>>Hubert Chan wrote: >>> >>> >> >> >> >> >>>>The main thing blocking file-as-dir is that there are some >>>>locking(IIRC?) issues. And, of course, some people wouldn't want it >>>>to be merged into the mainline kernel. (Of course, the latter >>>>doesn't prevent Namesys from maintaining their own patches for people >>>>to play around with.) >>>> >>>> >> >> >> >> >>>What's the locking issue? I think that was more about transactions... >>> >>> >> >>It was whatever was Al Viro's (technical) complaint about file-as-dir. >>I don't remember exactly what it was. The technical people know what it >>is (and the Namesys guys are probably working on it), and the exact >>issue doesn't concern us non-technical people that much, so I don't feel >>like looking it up. But if you want to, just look for Al Viro's message >>in this thread. >> >> >> > > Cycle detection when hard links to directories are allowed. There is a > debate over whether cycle detection is feasible that can only be > resolved by working code or a formal proof that it is not > computationally feasible. Ah. But then, one solution was to avoid the issue at all, and have the directory inside a file act as a mountpoint. After all, mount --bind doesn't cause problems... Hey! This sounds like metafs (/meta) already! I wonder if we can do file-as-dir in /meta, and just not support user-created hardlinks there? (other than creating brand-new files, of course...) ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2005-07-05 22:21 ` David Masover @ 2005-07-05 22:51 ` Hans Reiser 2005-07-05 23:00 ` David Masover 0 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2005-07-05 22:51 UTC (permalink / raw) To: David Masover Cc: Hubert Chan, Ross Biro, Horst von Brand, Kyle Moffett, Valdis.Kletnieks, Lincoln Dale, Gregory Maxwell, Jeff Garzik, Christoph Hellwig, Andrew Morton, linux-kernel, ReiserFS List David Masover wrote: >Hans Reiser wrote: > > >>Hubert Chan wrote: >> >> >> >> >>>On Fri, 01 Jul 2005 03:06:19 -0500, David Masover <ninja@slaphack.com> said: >>> >>> >>> >>> >>> >>> >>>>Hubert Chan wrote: >>>> >>>> >>>> >>>> >>> >>> >>> >>> >>>>>The main thing blocking file-as-dir is that there are some >>>>>locking(IIRC?) issues. And, of course, some people wouldn't want it >>>>>to be merged into the mainline kernel. (Of course, the latter >>>>>doesn't prevent Namesys from maintaining their own patches for people >>>>>to play around with.) >>>>> >>>>> >>>>> >>>>> >>> >>> >>> >>> >>>>What's the locking issue? I think that was more about transactions... >>>> >>>> >>>> >>>> >>>It was whatever was Al Viro's (technical) complaint about file-as-dir. >>>I don't remember exactly what it was. The technical people know what it >>>is (and the Namesys guys are probably working on it), and the exact >>>issue doesn't concern us non-technical people that much, so I don't feel >>>like looking it up. But if you want to, just look for Al Viro's message >>>in this thread. >>> >>> >>> >>> >>> >>Cycle detection when hard links to directories are allowed. There is a >>debate over whether cycle detection is feasible that can only be >>resolved by working code or a formal proof that it is not >>computationally feasible. >> >> > >Ah. But then, one solution was to avoid the issue at all, and have the >directory inside a file act as a mountpoint. After all, mount --bind >doesn't cause problems... > > Can you explain this idea at greater length? >Hey! This sounds like metafs (/meta) already! I wonder if we can do >file-as-dir in /meta, and just not support user-created hardlinks there? > (other than creating brand-new files, of course...) > > > > ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2005-07-05 22:51 ` Hans Reiser @ 2005-07-05 23:00 ` David Masover 2005-07-05 23:47 ` Hans Reiser 0 siblings, 1 reply; 1392+ messages in thread From: David Masover @ 2005-07-05 23:00 UTC (permalink / raw) To: Hans Reiser Cc: Hubert Chan, Ross Biro, Horst von Brand, Kyle Moffett, Valdis.Kletnieks, Lincoln Dale, Gregory Maxwell, Jeff Garzik, Christoph Hellwig, Andrew Morton, linux-kernel, ReiserFS List Hans Reiser wrote: > David Masover wrote: > > >>Hans Reiser wrote: >> >> >> >>>Hubert Chan wrote: >>> >>> >>> >>> >>> >>>>On Fri, 01 Jul 2005 03:06:19 -0500, David Masover <ninja@slaphack.com> said: >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>>>Hubert Chan wrote: >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>>> >>>> >>>>>>The main thing blocking file-as-dir is that there are some >>>>>>locking(IIRC?) issues. And, of course, some people wouldn't want it >>>>>>to be merged into the mainline kernel. (Of course, the latter >>>>>>doesn't prevent Namesys from maintaining their own patches for people >>>>>>to play around with.) >>>>>> >>>>>> >>>>>> >>>>>> >>>> >>>> >>>> >>>> >>>> >>>>>What's the locking issue? I think that was more about transactions... >>>>> >>>>> >>>>> >>>>> >>>> >>>>It was whatever was Al Viro's (technical) complaint about file-as-dir. >>>>I don't remember exactly what it was. The technical people know what it >>>>is (and the Namesys guys are probably working on it), and the exact >>>>issue doesn't concern us non-technical people that much, so I don't feel >>>>like looking it up. But if you want to, just look for Al Viro's message >>>>in this thread. >>>> >>>> >>>> >>>> >>>> >>> >>>Cycle detection when hard links to directories are allowed. There is a >>>debate over whether cycle detection is feasible that can only be >>>resolved by working code or a formal proof that it is not >>>computationally feasible. >>> >>> >> >>Ah. But then, one solution was to avoid the issue at all, and have the >>directory inside a file act as a mountpoint. After all, mount --bind >>doesn't cause problems... >> >> > > Can you explain this idea at greater length? Just don't allow user-created hardlinks inside either metafs (/meta) or the magical meta directory inside files. In order to do that, one way would be to have "file/..." appear as a mountpoint. I don't know if this is feasable, performance-wise, but I think it would work. >>Hey! This sounds like metafs (/meta) already! I wonder if we can do >>file-as-dir in /meta, and just not support user-created hardlinks there? >>(other than creating brand-new files, of course...) This is still my preferred solution, because it's not any harder or less efficient to develop new apps based on metafs than on file-as-dir, but it means that old apps will see something *entirely* POSIX-compliant, and the strangeness will be confined to /meta. Basically, you have to allow hardlinks in /meta, in case some plugin wants to do that, but if you have files that are also directories, you also have to make sure that users can't create hardlinks. I don't think this would be particularly hard, though. Pretend it's a read-only filesystem unless the user is doing something safe, like creating a brand-new file, deleting an old one, or modifying the contents of an existing one, all assuming that the plugin responsible for the file/directory you're doing this in allows it. But then, if we're doing metafs, I don't think you need file-as-directory, and certain existing tools that we'd like to be able to point at metafs might not like file-as-directory anyway. Now, can anyone think of a situation where we want user-created hardlinks inside metadata? More importantly, what do we do about it? ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2005-07-05 23:00 ` David Masover @ 2005-07-05 23:47 ` Hans Reiser 2005-07-06 0:15 ` Hans Reiser 0 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2005-07-05 23:47 UTC (permalink / raw) To: David Masover Cc: Hubert Chan, Ross Biro, Horst von Brand, Kyle Moffett, Valdis.Kletnieks, Lincoln Dale, Gregory Maxwell, Jeff Garzik, Christoph Hellwig, Andrew Morton, linux-kernel, ReiserFS List, Alexander Zarochentcev, vs, Nate Diller David Masover wrote: >Now, can anyone think of a situation where we want user-created >hardlinks inside metadata? More importantly, what do we do about it? > > I think the equivalent of symlinks would be good enough to get by on for now for most linking of metafiles. Maybe some years from now somebody can fault me for saying this and write a patch to fix it to be better, at which point I will be happy to concede the point. So the basic principal here is, one can have hardlinks to directories without cycles provided that one does not allow any child of the directory to have a hardlink. The question is, how cleanly can that relaxed restriction be coded? Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2005-07-05 23:47 ` Hans Reiser @ 2005-07-06 0:15 ` Hans Reiser 2005-07-06 14:01 ` David Masover 0 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2005-07-06 0:15 UTC (permalink / raw) To: Hans Reiser Cc: David Masover, Hubert Chan, Ross Biro, Horst von Brand, Kyle Moffett, Valdis.Kletnieks, Lincoln Dale, Gregory Maxwell, Jeff Garzik, Christoph Hellwig, Andrew Morton, linux-kernel, ReiserFS List, Alexander Zarochentcev, vs, Nate Diller If we also add to this the restriction that once you create the file part of a file-directory, you can never hardlink to a child of it, it should then all work, yes? So, /filename/..../owner should be able to avoid colliding with any common names by virtue of the '....', and not letting any filedir (file-directory) have children with links to them should also work. The one thing that seems inelegant is that when you create the file part of a filedir, you must check all its children for hardlinks and fail if they exist, and you must flag all its directory children so that the plugins for them will disallow hardlinks to their children from that point onward. Still, seems workable.... Thanks David, Hans Hans Reiser wrote: >David Masover wrote: > > > >>Now, can anyone think of a situation where we want user-created >>hardlinks inside metadata? More importantly, what do we do about it? >> >> >> >> >I think the equivalent of symlinks would be good enough to get by on for >now for most linking of metafiles. Maybe some years from now somebody >can fault me for saying this and write a patch to fix it to be better, >at which point I will be happy to concede the point. > >So the basic principal here is, one can have hardlinks to directories >without cycles provided that one does not allow any child of the >directory to have a hardlink. The question is, how cleanly can that >relaxed restriction be coded? > >Hans > > > ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2005-07-06 0:15 ` Hans Reiser @ 2005-07-06 14:01 ` David Masover 2005-07-11 4:07 ` Stefan Smietanowski 0 siblings, 1 reply; 1392+ messages in thread From: David Masover @ 2005-07-06 14:01 UTC (permalink / raw) To: Hans Reiser Cc: Hubert Chan, Ross Biro, Horst von Brand, Kyle Moffett, Valdis.Kletnieks, Lincoln Dale, Gregory Maxwell, Jeff Garzik, Christoph Hellwig, Andrew Morton, linux-kernel, ReiserFS List, Alexander Zarochentcev, vs, Nate Diller Hans Reiser wrote: > If we also add to this the restriction that once you create the file > part of a file-directory, you can never hardlink to a child of it, it > should then all work, yes? > > So, /filename/..../owner should be able to avoid colliding with any > common names by virtue of the '....', and not letting any filedir > (file-directory) have children with links to them should also work. The > one thing that seems inelegant is that when you create the file part of > a filedir, you must check all its children for hardlinks and fail if > they exist, and you must flag all its directory children so that the > plugins for them will disallow hardlinks to their children from that > point onward. Still, seems workable.... Ok, still haven't heard much discussion of metafs vs file-as-directory, but it seems like it'd be easier in metafs. Basically, we are entirely POSIX compliant outside of metafs, so that /filename is a file and may be hardlinked, and /directory/ is a directory and may not. No problems yet. Inside /meta/inode, we have no problems yet because any hardlinks created outside /meta won't even show up as hardlinks here, so we don't get hardlinked directories. Inside /meta/vfs, where we can find the metadata of '/filename' under '/meta/vfs/filename/...', we have what looks like a problem -- we may have hardlinks created outside metafs, which will show up as two directories inside metafs, so those directories are hardlinks. I don't think that's such a problem, since we won't allow users to change anything in /meta/vfs except when it's inside metadata, such as '/meta/vfs/some/where/...'. Thus, it's impossible to create a hardlink to a directory unless we do it *outside* metafs, as a hardlink to a file. Now, the question I asked is, what if we want hardlinks *inside* metafs, *inside* the metadata for a given file, and would we ever want such a thing? Because we can avoid the whole problem if we just disallow any sys_link calls inside metafs. Of course, sometimes we want to have a chunk of metadata that appears *exactly* as if it were a normal, POSIX-compliant directory tree, such as the contents of a tarball or a zipfile. In that case, we might want to have the "zip" plugin allow hardlinks inside "/meta/file.zip/.../contents" -- the only restriction is that any hardlink made to a file inside 'contents' must be made to another file inside 'contents'. > Hans Reiser wrote: > > >>David Masover wrote: >> >> >> >> >>>Now, can anyone think of a situation where we want user-created >>>hardlinks inside metadata? More importantly, what do we do about it? >>> >>> >>> >>> >> >>I think the equivalent of symlinks would be good enough to get by on for >>now for most linking of metafiles. Maybe some years from now somebody >>can fault me for saying this and write a patch to fix it to be better, >>at which point I will be happy to concede the point. >> >>So the basic principal here is, one can have hardlinks to directories >>without cycles provided that one does not allow any child of the >>directory to have a hardlink. The question is, how cleanly can that >>relaxed restriction be coded? >> >>Hans >> >> >> > > ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2005-07-06 14:01 ` David Masover @ 2005-07-11 4:07 ` Stefan Smietanowski 2005-07-11 20:50 ` David Masover 0 siblings, 1 reply; 1392+ messages in thread From: Stefan Smietanowski @ 2005-07-11 4:07 UTC (permalink / raw) To: David Masover Cc: Hans Reiser, Hubert Chan, Ross Biro, Horst von Brand, Kyle Moffett, Valdis.Kletnieks, Lincoln Dale, Gregory Maxwell, Jeff Garzik, Christoph Hellwig, Andrew Morton, linux-kernel, ReiserFS List, Alexander Zarochentcev, vs, Nate Diller -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 > Ok, still haven't heard much discussion of metafs vs file-as-directory, > but it seems like it'd be easier in metafs. Why not implement it inside the directory containg the file ? Ie the metadata for /home/stesmi/foo is in /home/stesmi/.meta/foo This should be suit both camps I'd think? I mean, editing something is easy and you don't have to "know" how to navigate /meta and you don't have the clash of files vs metadata (is /meta/vfs/home/stesmi/foo a file or an attribute called foo of the dir stesmi ?). /home/stesmi/foo <- dir /home/stesmi/.meta/foo <- "dir" containing all metadata /home/stesmi/.meta/foo/attrib <- some attribute called attrib /home/stesmi/foo/bar <- file /home/stesmi/foo/.meta/bar <- "dir" containg all metadata /home/stesmi/foo/.meta/bar/attrib <- some attribute called attrib The file is $dir/$file. The attrib dir is $dir/.meta/$file. The attribute is $dir/.meta/$file/$attribute. If $attrib is something user-editable it's easy to $EDITOR $dir/.meta/$file/$attrib If this has already been taken up, I must've missed it. // Stefan -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (MingW32) iD8DBQFC0fBsBrn2kJu9P78RAlt4AJ4qWik6hA4oXBNZMdZ1TkweYrJHmgCdFAY+ m+Qtg9uqBq9m0qKfRkK6iUI= =ghWb -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2005-07-11 4:07 ` Stefan Smietanowski @ 2005-07-11 20:50 ` David Masover 2005-07-12 7:19 ` Neil Brown 0 siblings, 1 reply; 1392+ messages in thread From: David Masover @ 2005-07-11 20:50 UTC (permalink / raw) To: Stefan Smietanowski Cc: Hans Reiser, Hubert Chan, Ross Biro, Horst von Brand, Kyle Moffett, Valdis.Kletnieks, Lincoln Dale, Gregory Maxwell, Jeff Garzik, Christoph Hellwig, Andrew Morton, linux-kernel, ReiserFS List, Alexander Zarochentcev, vs, Nate Diller Stefan Smietanowski wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > >>Ok, still haven't heard much discussion of metafs vs file-as-directory, >>but it seems like it'd be easier in metafs. > > > Why not implement it inside the directory containg the file ? > > Ie the metadata for /home/stesmi/foo is in /home/stesmi/.meta/foo > > This should be suit both camps I'd think? You still need to figure out the parent of "foo", which isn't necessarily easy, especially considering that even if we store a link to the parent, it will be inside the metadata. Then you have a chicken-and-egg situation. Both camps seem to want to allow easy access to the metadata of a file, whether we're given that file as an inode or as a pathname. That's why I suggested /meta/vfs and /meta/inode -- sometimes I want to look up a file by name, and sometimes by inode, but either way, I should be able to get its metadata. > I mean, editing something is easy and you don't have to "know" how > to navigate /meta But then you have to "know" how to navigate .meta, and more importantly, how to find .meta > and you don't have the clash of files vs metadata > (is /meta/vfs/home/stesmi/foo a file or an attribute called foo of > the dir stesmi ?). So, how do I get metadata of a directory? If the metadata for /home/me is in /home/.meta/me, and the metadata for /home is in /.meta/home, then where is the metadata for / ? > /home/stesmi/foo <- dir > /home/stesmi/.meta/foo <- "dir" containing all metadata > /home/stesmi/.meta/foo/attrib <- some attribute called attrib > /home/stesmi/foo/bar <- file > /home/stesmi/foo/.meta/bar <- "dir" containg all metadata > /home/stesmi/foo/.meta/bar/attrib <- some attribute called attrib [...] > If this has already been taken up, I must've missed it. It looks a lot like how I suggested we resolve the ambiguity within /meta/vfs, only I called it '...' instead of '.meta'. Either way, the major challenge to this method is not so much whether it'd work, but how to choose a suitable delimiter? The delimiter must be standard if we're going to have apps develop for it, and it must not be used in the name of any existing file or directory. I had a long essay here on how '.meta' breaks every recursive operation on directories (rm -rf, tar, mkisofs), but I deleted it when I remembered that I played around with the alpha/beta reiser4 which had a 'metas' dir that did the same thing, but was hidden from the normal directory listing. 'cd metas' worked, 'ls metas' worked, but 'ls' showed no directory called 'metas'. I don't *think* there are any apps that will break if you tell them to open a path that doesn't exist in a directory listing. That is, typing 'vim /home/metas/acl' should always let me edit the acl on /home, and vim should never complain about how /home/metas doesn't show up in /home. A program would have to be pretty retarded to fail on something like that. But, we have to support some pretty retarded programs. That is why I like /meta -- the default view is a completely POSIX-compliant tree that works with tar, and even the /meta view is POSIX-compliant, even if it'd be a bad idea to tar it. Then we don't have to worry at all about stupid programs. How much should we be worrying about this particular brand of stupidity? And what's a good delimiter? -- No virus found in this outgoing message. Checked by AVG Anti-Virus. Version: 7.0.323 / Virus Database: 267.8.12/46 - Release Date: 7/11/2005 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2005-07-11 20:50 ` David Masover @ 2005-07-12 7:19 ` Neil Brown 2005-07-12 7:45 ` Hans Reiser 2005-07-12 23:13 ` David Masover 0 siblings, 2 replies; 1392+ messages in thread From: Neil Brown @ 2005-07-12 7:19 UTC (permalink / raw) To: David Masover Cc: Stefan Smietanowski, Hans Reiser, Hubert Chan, Ross Biro, Horst von Brand, Kyle Moffett, Valdis.Kletnieks, Lincoln Dale, Gregory Maxwell, Jeff Garzik, Christoph Hellwig, Andrew Morton, linux-kernel, ReiserFS List, Alexander Zarochentcev, vs, Nate Diller On Monday July 11, ninja@slaphack.com wrote: > Stefan Smietanowski wrote: > > -----BEGIN PGP SIGNED MESSAGE----- > > Hash: SHA1 > > > > > >>Ok, still haven't heard much discussion of metafs vs file-as-directory, > >>but it seems like it'd be easier in metafs. > > > > > > Why not implement it inside the directory containg the file ? > > > > Ie the metadata for /home/stesmi/foo is in /home/stesmi/.meta/foo > > > > This should be suit both camps I'd think? > > You still need to figure out the parent of "foo", which isn't > necessarily easy, especially considering that even if we store a link to > the parent, it will be inside the metadata. Then you have a > chicken-and-egg situation. > > Both camps seem to want to allow easy access to the metadata of a file, > whether we're given that file as an inode or as a pathname. That's why > I suggested /meta/vfs and /meta/inode -- sometimes I want to look up a > file by name, and sometimes by inode, but either way, I should be able > to get its metadata. Well, it's not really 'as an inode or as a pathname'. It is 'as an open file descriptor or as a path name' which is an important difference. Maybe it is worth repeating Al Viro's suggestion at this point. I don't have a reference but the idea was basically that if you open "/foo" and get filedescriptor N, then /proc/self/fds/N-meta is a directory which contains all the meta stuff for "/foo". Then it is trivial to get the 'meta' stuff given a filedescriptor and if you have a pathname, you can always get yourself a filedescriptor. Then to allow cat /home/friend/foo/.meta.../perms you write a little .so which redefines open, stat, and a few others to scan for ".meta..." in the pathname and to the necessary magic, and export LD_PRELOAD=/that/.so Finally you write a killer app which fundamentally relies on this functionality, get everyone addicted to it, and *then* (and only then) start trying to get this functionality into the kernel. i.e. prototype new semantics in userspace. NeilBrown ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2005-07-12 7:19 ` Neil Brown @ 2005-07-12 7:45 ` Hans Reiser 2005-07-13 0:05 ` Neil Brown 2005-07-12 23:13 ` David Masover 1 sibling, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2005-07-12 7:45 UTC (permalink / raw) To: Neil Brown Cc: David Masover, Stefan Smietanowski, Hubert Chan, Ross Biro, Horst von Brand, Kyle Moffett, Valdis.Kletnieks, Lincoln Dale, Gregory Maxwell, Jeff Garzik, Christoph Hellwig, Andrew Morton, linux-kernel, ReiserFS List, Alexander Zarochentcev, vs, Nate Diller Neil Brown wrote: > > >Maybe it is worth repeating Al Viro's suggestion at this point. I >don't have a reference but the idea was basically that if you open >"/foo" and get filedescriptor N, then > /proc/self/fds/N-meta >is a directory which contains all the meta stuff for "/foo". >Then it is trivial to get the 'meta' stuff given a filedescriptor and >if you have a pathname, you can always get yourself a filedescriptor. > > This sound like it might be cute, but filedescriptors are too heavy weight for stat data accesses in quantity. In general, the whole file handle paradigm is too heavy for lightweight files. Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2005-07-12 7:45 ` Hans Reiser @ 2005-07-13 0:05 ` Neil Brown 2005-07-13 1:20 ` Hans Reiser 0 siblings, 1 reply; 1392+ messages in thread From: Neil Brown @ 2005-07-13 0:05 UTC (permalink / raw) To: Hans Reiser Cc: David Masover, Stefan Smietanowski, Hubert Chan, Ross Biro, Horst von Brand, Kyle Moffett, Valdis.Kletnieks, Lincoln Dale, Gregory Maxwell, Jeff Garzik, Christoph Hellwig, Andrew Morton, linux-kernel, ReiserFS List, Alexander Zarochentcev, vs, Nate Diller On Tuesday July 12, reiser@namesys.com wrote: > Neil Brown wrote: > > > > > > >Maybe it is worth repeating Al Viro's suggestion at this point. I > >don't have a reference but the idea was basically that if you open > >"/foo" and get filedescriptor N, then > > /proc/self/fds/N-meta > >is a directory which contains all the meta stuff for "/foo". > >Then it is trivial to get the 'meta' stuff given a filedescriptor and > >if you have a pathname, you can always get yourself a filedescriptor. > > > > > This sound like it might be cute, but filedescriptors are too heavy > weight for stat data accesses in quantity. > > In general, the whole file handle paradigm is too heavy for lightweight > files. That may well be true, but is completely orthogonal to filesystem name semantics. If you find file descriptors too slow, come up with an alternate (I suspect you have in the reiser4 syscall, but I haven't looked at that yet), implement it in the VFS, and show the world benchmarks of real-world applications that go faster with this new interface. I doubt that you would then have a great deal of trouble in getting the interface accepted (some trouble of course as you will need to convince a few people, but numbers speak quite loudly). I suspect that there might need to be a new internal interface into filesystems, and filesystems which don't provide that will not get the same speed benefit, but that is perfectly acceptable. NeilBrown ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2005-07-13 0:05 ` Neil Brown @ 2005-07-13 1:20 ` Hans Reiser 0 siblings, 0 replies; 1392+ messages in thread From: Hans Reiser @ 2005-07-13 1:20 UTC (permalink / raw) To: Neil Brown Cc: David Masover, Stefan Smietanowski, Hubert Chan, Ross Biro, Horst von Brand, Kyle Moffett, Valdis.Kletnieks, Lincoln Dale, Gregory Maxwell, Jeff Garzik, Christoph Hellwig, Andrew Morton, linux-kernel, ReiserFS List, Alexander Zarochentcev, vs, Nate Diller, Vladimir Demidov Neil Brown wrote: >On Tuesday July 12, reiser@namesys.com wrote: > > >>Neil Brown wrote: >> >> >> >>>Maybe it is worth repeating Al Viro's suggestion at this point. I >>>don't have a reference but the idea was basically that if you open >>>"/foo" and get filedescriptor N, then >>> /proc/self/fds/N-meta >>>is a directory which contains all the meta stuff for "/foo". >>>Then it is trivial to get the 'meta' stuff given a filedescriptor and >>>if you have a pathname, you can always get yourself a filedescriptor. >>> >>> >>> >>> >>This sound like it might be cute, but filedescriptors are too heavy >>weight for stat data accesses in quantity. >> >>In general, the whole file handle paradigm is too heavy for lightweight >>files. >> >> > >That may well be true, but is completely orthogonal to filesystem name >semantics. > >If you find file descriptors too slow, come up with an alternate (I >suspect you have in the reiser4 syscall, but I haven't looked at >that yet), implement it in the VFS, and show the world benchmarks of >real-world applications that go faster with this new interface. > >I doubt that you would then have a great deal of trouble in getting >the interface accepted (some trouble of course as you will need to >convince a few people, but numbers speak quite loudly). > >I suspect that there might need to be a new internal interface into >filesystems, and filesystems which don't provide that will not get the >same speed benefit, but that is perfectly acceptable. > >NeilBrown > > > > We need time, and then we can demonstrate sys_reiser4, it is not ready for showing yet..... Hans ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2005-07-12 7:19 ` Neil Brown 2005-07-12 7:45 ` Hans Reiser @ 2005-07-12 23:13 ` David Masover 2005-07-12 23:40 ` Neil Brown 1 sibling, 1 reply; 1392+ messages in thread From: David Masover @ 2005-07-12 23:13 UTC (permalink / raw) To: Neil Brown Cc: Stefan Smietanowski, Hans Reiser, Hubert Chan, Ross Biro, Horst von Brand, Kyle Moffett, Valdis.Kletnieks, Lincoln Dale, Gregory Maxwell, Jeff Garzik, Christoph Hellwig, Andrew Morton, linux-kernel, ReiserFS List, Alexander Zarochentcev, vs, Nate Diller -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Neil Brown wrote: > On Monday July 11, ninja@slaphack.com wrote: > >>Stefan Smietanowski wrote: >> >>>-----BEGIN PGP SIGNED MESSAGE----- >>>Hash: SHA1 >>> >>> >>> >>>>Ok, still haven't heard much discussion of metafs vs file-as-directory, >>>>but it seems like it'd be easier in metafs. >>> >>> >>>Why not implement it inside the directory containg the file ? >>> >>>Ie the metadata for /home/stesmi/foo is in /home/stesmi/.meta/foo >>> >>>This should be suit both camps I'd think? >> >>You still need to figure out the parent of "foo", which isn't >>necessarily easy, especially considering that even if we store a link to >>the parent, it will be inside the metadata. Then you have a >>chicken-and-egg situation. >> >>Both camps seem to want to allow easy access to the metadata of a file, >>whether we're given that file as an inode or as a pathname. That's why >>I suggested /meta/vfs and /meta/inode -- sometimes I want to look up a >>file by name, and sometimes by inode, but either way, I should be able >>to get its metadata. > > > Well, it's not really 'as an inode or as a pathname'. It is 'as an > open file descriptor or as a path name' which is an important > difference. > > Maybe it is worth repeating Al Viro's suggestion at this point. I > don't have a reference but the idea was basically that if you open > "/foo" and get filedescriptor N, then > /proc/self/fds/N-meta How am I supposed to get there with a shell script? -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQtROqXgHNmZLgCUhAQLRcg/+I9PWSmFXRwKtj7pnEeMXOCjiTo6GQE3O 61fjH3f6aL9Ydkip4eXum3S14cioiU9bzC11GA5kRIM+W1DKcYex1dIpivrtF9Ht Rvozn9x2TP5tacDmSfqRJXvAB+xTRtZOu+M/rDjXdLsriDJDA0AdyDH8Yo/8WMbU 6i1DWzLTO0vHS3kEb/8oqgBj7sQ63sS/4KVszBx6+bN0KOikXbORDu6efKjC9w21 3DZPnBG0B03smhdCygd0j0Zmh0JEaZHfuFgNK1ZmRzipbvvUBDtdKY5MJ6f4pHnA GBO8ybsXp9qxNQr6DNenF/wbAT6n3dMyE/AWuql+qx3iumSwx/Prh7xDAhBZBMXp Oin7hOa1i583cdju4ErSBPaciRzumGluY6gbFvVA8Yva+IjPxxjPtfLwalK11cH1 k4oQO5Par1W0TmMOpc0PQ/bEeEHHxUcn1ToeJa4NYJWIiBe+UHMb/AyI4hKjSIkt Kp0wrCPBjRfuBCHXXL89bWZoSeSFkN8fAyOxBV928naxxr8e+vCPUX1/H3ca7UsB Nxg0Vzt4V9tz4xCw4QAy810Uya8/HSm3aVhqEzjHKBoKboHrMVDJvxRQxfkqQcnC 4jIFYPBdHgGw7OONyhfbgTPLIm1OCNPpcRkS4aidHqg0DkDU50h6zFQkhG5Xwh5Z x5REgxbqD+A= =FGm4 -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2005-07-12 23:13 ` David Masover @ 2005-07-12 23:40 ` Neil Brown 0 siblings, 0 replies; 1392+ messages in thread From: Neil Brown @ 2005-07-12 23:40 UTC (permalink / raw) To: David Masover Cc: Stefan Smietanowski, Hans Reiser, Hubert Chan, Ross Biro, Horst von Brand, Kyle Moffett, Valdis.Kletnieks, Lincoln Dale, Gregory Maxwell, Jeff Garzik, Christoph Hellwig, Andrew Morton, linux-kernel, ReiserFS List, Alexander Zarochentcev, vs, Nate Diller On Tuesday July 12, ninja@slaphack.com wrote: > > > > Maybe it is worth repeating Al Viro's suggestion at this point. I > > don't have a reference but the idea was basically that if you open > > "/foo" and get filedescriptor N, then > > /proc/self/fds/N-meta > > How am I supposed to get there with a shell script? function get_meta() { var=$1 file=$2 meta=$3 val=`cat /proc/self/fd/3-meta/$meta < $file` eval var=\$val } then get_meta varname /home/foo/bar username will read the 'username' meta-file of 'home/foo/bar' and place it in varname. Is that what you wanted? NeilBrown ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins @ 2005-07-12 2:33 Horst von Brand 2005-07-12 5:53 ` Hans Reiser 0 siblings, 1 reply; 1392+ messages in thread From: Horst von Brand @ 2005-07-12 2:33 UTC (permalink / raw) To: Hans Reiser Cc: Stefan Smietanowski, David Masover, Hubert Chan, Ross Biro, Horst von Brand, Kyle Moffett, Valdis.Kletnieks, Lincoln Dale, Gregory Maxwell, Jeff Garzik, Christoph Hellwig, Andrew Morton, linux-kernel, ReiserFS List, Alexander Zarochentcev, vs, Nate Diller Hans Reiser <reiser@namesys.com> wrote: > Stefan Smietanowski wrote: > > I think "..." and ".meta" both serve as a logical delimiter. However > > some programs implement their own "..." which would make it clash with > > them. Naturally if some program created a directory called .meta we're > > equally screwed. > I chose '....' (four dots) because it clashes with less, not three dots. Is this some kind of "My dots are more than yours" contest?! /None/ of them is safe. ".meta", "...", "....", ".this.has.five.dots." are all perfectly legal file (or directory) names, POSIXly. If any one of them won't do, none will. -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2005-07-12 2:33 Horst von Brand @ 2005-07-12 5:53 ` Hans Reiser 2005-07-12 23:22 ` David Masover 0 siblings, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2005-07-12 5:53 UTC (permalink / raw) To: Horst von Brand Cc: Stefan Smietanowski, David Masover, Hubert Chan, Ross Biro, Kyle Moffett, Valdis.Kletnieks, Lincoln Dale, Gregory Maxwell, Jeff Garzik, Christoph Hellwig, Andrew Morton, linux-kernel, ReiserFS List, Alexander Zarochentcev, vs, Nate Diller Horst von Brand wrote: >Hans Reiser <reiser@namesys.com> wrote: > > >>Stefan Smietanowski wrote: >> >> >>>I think "..." and ".meta" both serve as a logical delimiter. However >>>some programs implement their own "..." which would make it clash with >>>them. Naturally if some program created a directory called .meta we're >>>equally screwed. >>> >>> > > > >>I chose '....' (four dots) because it clashes with less, not three dots. >> >> > >Is this some kind of "My dots are more than yours" contest?! > >/None/ of them is safe. ".meta", "...", "....", ".this.has.five.dots." are >all perfectly legal file (or directory) names, POSIXly. If any one of them >won't do, none will. > > There is a long history of encroaching namespaces by creating new keywords in CS, and it is a survivable problem. Clearcase (the version control filesystem) is an excellent example. They have special meanings for @ and some other common characters, and you have to learn how to escape those characters if you use them in Clearcase. It works, it makes users happy, in practice it is far less of a problem than one might think. I think there were two or three times I had to remember how to escape things in 2 years of using it..... Better to spend one's mind looking for bugs instead of this issue..... ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2005-07-12 5:53 ` Hans Reiser @ 2005-07-12 23:22 ` David Masover 2005-07-12 23:38 ` Hans Reiser 2005-07-13 2:09 ` Horst von Brand 0 siblings, 2 replies; 1392+ messages in thread From: David Masover @ 2005-07-12 23:22 UTC (permalink / raw) To: Hans Reiser Cc: Horst von Brand, Stefan Smietanowski, Hubert Chan, Ross Biro, Kyle Moffett, Valdis.Kletnieks, Lincoln Dale, Gregory Maxwell, Jeff Garzik, Christoph Hellwig, Andrew Morton, linux-kernel, ReiserFS List, Alexander Zarochentcev, vs, Nate Diller -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hans Reiser wrote: > Horst von Brand wrote: > > >>Hans Reiser <reiser@namesys.com> wrote: >> >> >> >>>Stefan Smietanowski wrote: >>> >>> >>> >>>>I think "..." and ".meta" both serve as a logical delimiter. However >>>>some programs implement their own "..." which would make it clash with >>>>them. Naturally if some program created a directory called .meta we're >>>>equally screwed. >>>> >>>> >> >> >> >> >>>I chose '....' (four dots) because it clashes with less, not three dots. >>> >>> >> >>Is this some kind of "My dots are more than yours" contest?! >> >>/None/ of them is safe. ".meta", "...", "....", ".this.has.five.dots." are >>all perfectly legal file (or directory) names, POSIXly. If any one of them >>won't do, none will. And, conversely, if any one of them makes sense, they all do. Well, with the exception that some of them have never, ever been used before, and that's what we should be aiming for. No conflicts with existing programs, then new programs can be written to avoid conflicting with us. > There is a long history of encroaching namespaces by creating new > keywords in CS, and it is a survivable problem. Agreed, but not one to be taken too lightly. > Clearcase (the version control filesystem) is an excellent example. > They have special meanings for @ and some other common characters, and > you have to learn how to escape those characters if you use them in > Clearcase. I like using @, though. Without having to escape it. That's why we're trying to find something that people won't actually touch, especially since if we design it right, this will be the last delimiter introduced at the fs/vfs level. > It works, it makes users happy, in practice it is far less of a problem > than one might think. I think there were two or three times I had to > remember how to escape things in 2 years of using it..... > > Better to spend one's mind looking for bugs instead of this issue..... .....if bugs were seen as such a big deal. I think it's far easier to get into the kernel with something ludicrously buggy than something which actually changes fundamental behavior. That is, you can put in an FS which actually corrupts data (such as the old NTFS write support), so long as it doesn't break POSIX, or cause other weird restrictions like "No files named 'metas'" Now, if we can decide that we don't care about being in the vanilla kernel, then we can just call it ".metas" or "lost+found" or whatever and get to work on bug fixes and other much-needed features such as a repacker. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iQIVAwUBQtRQvXgHNmZLgCUhAQJXWBAAlZggJQGNi2lEZh0MAqnz+rNVT1JYcY/b adHgvVOZZNiJw2GmVjGLIiG5cqSqML1//f0+4XOxjYED2rbfOwBJiDqcq/0IsKPp 5JJflJV2uWkEZukJ+oA5mspfSeKof2N5egBoD1Ije079VBKXdjoN5Kprkbe4ZYZ2 +Yu+w3FpdcaGvSqVlTRHPWsS0je4z8ieh0u6ql+HNR4ze/hKwMw4nrb2sARW9NOQ EZ8Ot5NDhVaxz9Rj72rLVuQrUZEF8b0ihkpmzTauVV/nysEGi33SPqYTF7nYGBnM 5mVY63uXG4zxmThMDpn+J/iofA/hS1fd1bY9tCgF1ZAPu9HrCTnVzKaTYoOq+ciD sY0m7HEc49JfaiyZ/SJGH02WUH3JqLQAVGQftEkqAQLyYdVRbzdBHUVUR2i8hX2M ofFLM6DGJgFK784PfO0sjNboQT5ay4B8js8NltdfytsOVwDzjMST7dZAWcnMgTZU jrMCJuMYeYh8468fy1C9pCMUPahVXvmF4O5Xazt7m9HvV4lxRJIUDZsaR0Volg2K OAQIO3rtT9heK/+/PaUuXX8RYwLQUlhdUmdHfdtjRiLtKRcbw/fYBOqjkgXZfOfy RzyBgD463sVCVAE8qMbAVnnHNGKZ25tyRTYITiTZ2kfnIeURJawj2SZVmn5ezGQG xRnI3+Ir5e0= =moA6 -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2005-07-12 23:22 ` David Masover @ 2005-07-12 23:38 ` Hans Reiser 2005-07-13 3:43 ` David Masover 2005-07-13 2:09 ` Horst von Brand 1 sibling, 1 reply; 1392+ messages in thread From: Hans Reiser @ 2005-07-12 23:38 UTC (permalink / raw) To: David Masover Cc: Horst von Brand, Stefan Smietanowski, Hubert Chan, Ross Biro, Kyle Moffett, Valdis.Kletnieks, Lincoln Dale, Gregory Maxwell, Jeff Garzik, Christoph Hellwig, Andrew Morton, linux-kernel, ReiserFS List, Alexander Zarochentcev, vs, Nate Diller David Masover wrote: > > That's why we're trying to find something that people won't actually > touch, especially since if we design it right, this will be the last > delimiter introduced at the fs/vfs level. Uh, no, there needs to be about a dozen or so more. But not this year. ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2005-07-12 23:38 ` Hans Reiser @ 2005-07-13 3:43 ` David Masover 0 siblings, 0 replies; 1392+ messages in thread From: David Masover @ 2005-07-13 3:43 UTC (permalink / raw) To: Hans Reiser Cc: Horst von Brand, Stefan Smietanowski, Hubert Chan, Ross Biro, Kyle Moffett, Valdis.Kletnieks, Lincoln Dale, Gregory Maxwell, Jeff Garzik, Christoph Hellwig, Andrew Morton, linux-kernel, ReiserFS List, Alexander Zarochentcev, vs, Nate Diller Hans Reiser wrote: > David Masover wrote: > > >>That's why we're trying to find something that people won't actually >>touch, especially since if we design it right, this will be the last >>delimiter introduced at the fs/vfs level. > > > Uh, no, there needs to be about a dozen or so more. Where? From what I (vaguely) remember of the future-vision paper, having the meta delimiter lets us do everything else from inside the metas. We can certainly add delimiters to stuff in a meta-dir... -- No virus found in this outgoing message. Checked by AVG Anti-Virus. Version: 7.0.323 / Virus Database: 267.8.12/46 - Release Date: 7/11/2005 ^ permalink raw reply [flat|nested] 1392+ messages in thread
* Re: reiser4 plugins 2005-07-12 23:22 ` David Masover 2005-07-12 23:38 ` Hans Reiser @ 2005-07-13 2:09 ` Horst von Brand 1 sibling, 0 replies; 1392+ messages in thread From: Horst von Brand @ 2005-07-13 2:09 UTC (permalink / raw) To: David Masover Cc: Hans Reiser, Horst von Brand, Stefan Smietanowski, Hubert Chan, Ross Biro, Kyle Moffett, Valdis.Kletnieks, Lincoln Dale, Gregory Maxwell, Jeff Garzik, Christoph Hellwig, Andrew Morton, linux-kernel, ReiserFS List, Alexander Zarochentcev, vs, Nate Diller David Masover <ninja@slaphack.com> wrote: > Hans Reiser wrote: > > Horst von Brand wrote: > >>Hans Reiser <reiser@namesys.com> wrote: > >>>Stefan Smietanowski wrote: [...] > > Better to spend one's mind looking for bugs instead of this issue..... > > .....if bugs were seen as such a big deal. > I think it's far easier to get into the kernel with something > ludicrously buggy than something which actually changes fundamental > behavior. Wonder why.... [Fixing bugs in the $FOO driver or the $BAR filesystem is /easy/, fixing bugs in "fundamental behaviour changes" is /extremely hard/.] > That is, you can put in an FS which actually corrupts data > (such as the old NTFS write support), so long as it doesn't break POSIX, > or cause other weird restrictions like "No files named 'metas'" Because that kind of problems are isolated. If you introduce a change that affects /all/ filesystems, and that change later on has unfixable bugs, or fundamental design issues, it is /a lot/ of work. > Now, if we can decide that we don't care about being in the vanilla > kernel, then we can just call it ".metas" or "lost+found" or whatever > and get to work on bug fixes and other much-needed features such as a > repacker. Great! -- Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513 ^ permalink raw reply [flat|nested] 1392+ messages in thread
end of thread, other threads:[~2005-07-13 2:44 UTC | newest] Thread overview: 1392+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2004-08-24 20:25 silent semantic changes with reiser4 Christoph Hellwig 2004-08-24 20:35 ` Lee Revell 2004-08-24 20:38 ` Christoph Hellwig 2004-08-24 20:42 ` Lee Revell 2004-08-24 21:18 ` Jamie Lokier 2004-08-24 20:38 ` Jeff Garzik 2004-08-24 20:53 ` viro 2004-08-24 21:22 ` viro 2004-08-25 18:28 ` Hans Reiser 2004-08-25 18:45 ` Christoph Hellwig 2004-08-26 9:02 ` Hans Reiser 2004-08-26 18:38 ` William Lee Irwin III 2004-08-26 18:49 ` Linus Torvalds 2004-08-26 18:56 ` William Lee Irwin III 2004-08-27 8:14 ` Hans Reiser [not found] ` <MPG.1b9926e15d166dfe9896d8@news.gmane.org> 2004-08-27 15:25 ` William Lee Irwin III [not found] ` <20040827161539.GC28876@khan.acc.umu.se> 2004-08-27 16:18 ` William Lee Irwin III 2004-08-25 19:53 ` Hans Reiser 2004-08-25 20:06 ` Matthew Wilcox 2004-08-26 8:41 ` Hans Reiser 2004-08-25 20:08 ` Christoph Hellwig 2004-08-25 20:19 ` Christoph Hellwig 2004-08-25 20:24 ` Linus Torvalds 2004-08-25 20:25 ` Christoph Hellwig 2004-08-25 20:59 ` viro 2004-08-26 8:43 ` Hans Reiser 2004-08-26 8:42 ` Hans Reiser 2004-08-26 9:24 ` Christoph Hellwig 2004-08-26 18:45 ` Hans Reiser 2004-08-26 13:38 ` Chris Mason 2004-08-26 23:53 ` Hans Reiser 2004-08-27 9:16 ` Christoph Hellwig 2004-08-25 20:22 ` Linus Torvalds 2004-08-25 20:35 ` Christoph Hellwig 2004-08-25 20:41 ` Hans Reiser 2004-08-25 20:51 ` Chris Mason 2004-08-25 20:58 ` Markus Törnqvist 2004-08-25 21:03 ` Rik van Riel 2004-08-26 9:00 ` Hans Reiser 2004-08-25 20:42 ` viro 2004-08-25 21:00 ` Christophe Saout 2004-08-25 22:59 ` Andrea Arcangeli 2004-08-26 8:35 ` Hans Reiser 2004-08-26 11:28 ` Andrea Arcangeli 2004-08-26 11:44 ` Christophe Saout 2004-08-26 12:16 ` Andrea Arcangeli 2004-08-26 12:23 ` Christophe Saout 2004-08-27 18:38 ` Alex Zarochentsev 2004-08-28 6:17 ` Hans Reiser 2004-08-26 23:53 ` Hans Reiser 2004-08-27 0:18 ` Andrea Arcangeli 2004-08-28 10:49 ` Yury Umanets 2004-08-28 11:11 ` Yury Umanets 2004-08-25 21:00 ` Linus Torvalds 2004-08-25 21:25 ` viro 2004-08-26 0:11 ` Jamie Lokier 2004-08-26 0:30 ` viro 2004-08-26 1:00 ` Jamie Lokier 2004-08-26 3:13 ` viro 2004-08-26 8:49 ` Hans Reiser 2004-08-26 14:05 ` Adrian Bunk 2004-08-26 14:25 ` Christophe Saout 2004-08-26 14:45 ` Matthew Wilcox 2004-08-26 15:04 ` Jamie Lokier 2004-08-26 15:12 ` Christophe Saout 2004-08-26 16:00 ` Jamie Lokier 2004-08-26 15:41 ` Nikita Danilov 2004-08-29 12:34 ` Alex Zarochentsev 2004-08-29 15:52 ` Shaya Potter 2004-09-01 22:05 ` Jamie Lokier 2004-08-26 15:02 ` Jamie Lokier 2004-08-28 23:14 ` Horst von Brand 2004-09-01 20:08 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Jamie Lokier 2004-09-01 20:50 ` Linus Torvalds [not found] ` <20040902002431.GN31934@mail.shareable.org> [not found] ` <413694E6.7010606@slaphack.com> [not found] ` <Pine.LNX.4.58.0409012037300.2295@ppc970.osdl.org> 2004-09-02 5:53 ` The argument for fs assistance in handling archives Hans Reiser [not found] ` <4136A14E.9010303@slaphack.com> 2004-09-02 6:04 ` Linus Torvalds 2004-09-02 7:15 ` Hans Reiser 2004-09-02 7:36 ` Linus Torvalds 2004-09-02 8:58 ` Hans Reiser 2004-09-02 9:12 ` Oliver Hunt 2004-09-02 9:26 ` Helge Hafting 2004-09-02 9:45 ` Oliver Hunt 2004-09-02 10:02 ` Spam 2004-09-02 11:00 ` Giuliano Pochini 2004-09-02 14:38 ` Christian Mayrhuber 2004-09-02 22:39 ` Valdis.Kletnieks 2004-09-02 23:06 ` Spam 2004-09-03 8:28 ` Helge Hafting 2004-09-03 8:22 ` Helge Hafting 2004-09-03 17:41 ` Horst von Brand 2004-09-03 19:30 ` Spam 2004-09-03 21:07 ` Chris Dukes 2004-09-03 21:50 ` Spam 2004-09-04 2:27 ` Horst von Brand 2004-09-04 13:20 ` Alexander G. M. Smith 2004-09-06 13:43 ` Helge Hafting 2004-09-06 13:25 ` Helge Hafting 2004-09-05 13:44 ` Tonnerre 2004-09-05 14:13 ` Spam 2004-09-08 9:24 ` Helge Hafting 2004-09-08 16:02 ` Tonnerre 2004-09-09 11:22 ` Helge Hafting 2004-09-05 13:28 ` Tonnerre 2004-09-02 9:52 ` Spam 2004-09-02 9:32 ` Alan Cox 2004-09-02 10:27 ` V13 2004-09-02 14:35 ` John Stoffel 2004-09-02 11:09 ` Oliver Neukum 2004-09-02 11:11 ` Spam 2004-09-03 0:43 ` David Masover 2004-09-03 0:47 ` Spam 2004-09-03 1:36 ` David Masover 2004-09-03 1:18 ` Valdis.Kletnieks 2004-09-03 1:39 ` David Masover 2004-09-03 8:13 ` Christoph Hellwig 2004-09-03 8:30 ` Oliver Neukum 2004-09-02 15:40 ` Horst von Brand 2004-09-02 18:21 ` Lee Revell 2004-09-03 8:30 ` Helge Hafting 2004-09-02 14:24 ` John Stoffel 2004-09-02 15:36 ` Alan Cox 2004-09-02 15:32 ` Horst von Brand 2004-09-05 15:16 ` Tonnerre 2004-09-02 17:43 ` Linus Torvalds 2004-09-02 18:22 ` Martin J. Bligh 2004-09-02 18:38 ` Christer Weinigel 2004-09-03 13:16 ` Brian Beattie 2004-09-02 19:52 ` Steve Bergman 2004-09-02 20:06 ` Spam 2004-09-02 20:16 ` Martin J. Bligh 2004-09-03 4:07 ` Hacksaw 2004-09-03 8:49 ` Bernd Petrovitsch 2004-09-03 9:53 ` Spam 2004-09-02 20:35 ` Erik Hensema 2004-09-06 14:26 ` Tonnerre 2004-09-03 0:35 ` David Masover 2004-09-03 6:35 ` Hans Reiser 2004-09-03 16:02 ` Chris Mason 2004-09-03 17:26 ` Horst von Brand 2004-09-03 0:41 ` David Masover 2004-09-03 0:46 ` Spam 2004-09-03 1:34 ` David Masover 2004-09-03 5:59 ` Hans Reiser 2004-09-15 17:23 ` Timothy Miller 2004-09-16 8:29 ` Helge Hafting 2004-09-02 9:04 ` Hans Reiser 2004-09-02 12:50 ` James Bruce 2004-09-02 17:50 ` Linus Torvalds 2004-09-05 15:12 ` Tonnerre 2004-09-03 1:11 ` David Masover 2004-09-03 1:35 ` Valdis.Kletnieks 2004-09-03 5:57 ` Hans Reiser 2004-09-02 7:55 ` Robin Rosenberg 2004-09-02 9:46 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Alan Cox 2004-09-02 16:11 ` Jamie Lokier 2004-09-02 17:41 ` Dave Kleikamp 2004-09-03 0:25 ` The argument for fs assistance in handling archives David Masover 2004-09-03 12:50 ` Dave Kleikamp 2004-09-04 0:25 ` David Masover 2004-09-05 14:42 ` Tonnerre 2004-09-03 15:48 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Pavel Machek 2004-09-02 23:33 ` Paul Jakma 2004-09-02 23:49 ` Spam 2004-09-02 22:55 ` Alan Cox 2004-09-03 0:03 ` Spam 2004-09-03 0:14 ` Paul Jakma 2004-09-03 0:39 ` Spam 2004-09-03 13:05 ` Dave Kleikamp 2004-09-03 13:16 ` Spam 2004-09-03 13:33 ` Dave Kleikamp 2004-09-03 21:45 ` Horst von Brand 2004-09-04 5:37 ` The argument for fs assistance in handling archives David Masover 2004-09-04 6:21 ` Hans Reiser 2004-09-04 11:07 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Spam 2004-09-04 0:13 ` The argument for fs assistance in handling archives David Masover 2004-09-04 11:42 ` Stephan von Krawczynski 2004-09-04 18:27 ` David Masover 2004-09-03 16:43 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Paul Jakma 2004-09-03 23:45 ` The argument for fs assistance in handling archives David Masover 2004-09-06 15:07 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Tonnerre 2004-09-06 15:15 ` Spam 2004-09-02 17:46 ` Linus Torvalds 2004-09-02 17:50 ` Christoph Hellwig 2004-09-02 18:03 ` Linus Torvalds 2004-09-03 0:57 ` The argument for fs assistance in handling archives Linh Dang 2004-09-04 2:58 ` Giuseppe Bilotta 2004-09-02 18:46 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Alan Cox 2004-09-02 20:22 ` Linus Torvalds 2004-09-05 7:27 ` Stephen Rothwell 2004-09-02 21:56 ` Bill Huey 2004-09-02 22:48 ` Alan Cox 2004-09-07 21:44 ` Bill Huey 2004-09-02 19:50 ` Spam 2004-09-03 0:17 ` The argument for fs assistance in handling archives David Masover 2004-09-06 7:45 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Tonnerre 2004-09-06 8:05 ` Spam 2004-09-06 8:56 ` Tonnerre 2004-09-06 9:15 ` The argument for fs assistance in handling archives Dr. Giovanni A. Orlando 2004-09-06 12:55 ` Grzegorz Jaśkiewicz 2004-09-02 20:38 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Frank van Maarseveen 2004-09-02 21:36 ` Dave Kleikamp 2004-09-02 21:48 ` Frank van Maarseveen 2004-09-02 22:00 ` viro 2004-09-02 22:02 ` Frank van Maarseveen 2004-09-02 22:03 ` Christoph Hellwig 2004-09-02 22:04 ` Christoph Hellwig 2004-09-02 22:06 ` viro 2004-09-02 22:11 ` Frank van Maarseveen 2004-09-02 22:17 ` viro 2004-09-02 22:26 ` Frank van Maarseveen 2004-09-02 22:33 ` viro 2004-09-02 22:56 ` Frank van Maarseveen 2004-09-02 23:02 ` viro 2004-09-03 8:43 ` Helge Hafting 2004-09-03 8:50 ` Frank van Maarseveen 2004-09-03 8:55 ` Frank van Maarseveen 2004-09-03 23:55 ` The argument for fs assistance in handling archives David Masover 2004-09-08 9:56 ` Helge Hafting 2004-09-06 8:04 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Tonnerre 2004-09-06 8:41 ` Frank van Maarseveen 2004-09-03 0:08 ` The argument for fs assistance in handling archives David Masover 2004-09-03 0:18 ` Linus Torvalds 2004-09-03 1:25 ` David Masover 2004-09-03 5:22 ` Hans Reiser 2004-09-03 0:39 ` viro 2004-09-03 0:44 ` Spam 2004-09-03 1:28 ` David Masover 2004-09-03 3:44 ` Chris Dukes 2004-09-03 4:37 ` David Masover 2004-09-03 13:09 ` Horst von Brand 2004-09-03 6:16 ` Hans Reiser 2004-09-03 13:55 ` John Stoffel 2004-09-03 14:01 ` Spam 2004-09-05 11:37 ` Michelle Konzack 2004-09-06 8:58 ` Tonnerre 2004-09-06 12:53 ` Clemens Schwaighofer 2004-09-06 12:57 ` Spam 2004-09-06 12:59 ` Clemens Schwaighofer 2004-09-06 13:01 ` Frank van Maarseveen 2004-09-06 13:03 ` Spam 2004-09-06 13:17 ` Clemens Schwaighofer 2004-09-06 13:45 ` Tonnerre 2004-09-02 23:19 ` The argument for fs assistance in handling archives (was: silent semantic changes with reiser4) Valdis.Kletnieks 2004-09-02 23:41 ` Frank van Maarseveen 2004-09-02 23:43 ` Spam 2004-09-02 23:52 ` Frank van Maarseveen 2004-09-06 14:43 ` Tonnerre 2004-09-06 14:59 ` Spam 2004-09-03 7:13 ` Jan Harkes 2004-09-03 19:30 ` Valdis.Kletnieks 2004-09-06 7:56 ` Tonnerre 2004-09-06 8:08 ` Frank van Maarseveen 2004-09-06 12:43 ` Herbert Poetzl 2004-09-06 12:54 ` Frank van Maarseveen 2004-09-03 15:44 ` Pavel Machek 2004-09-05 13:16 ` Tonnerre 2004-09-02 14:06 ` Horst von Brand 2004-09-02 17:32 ` Jamie Lokier 2004-09-02 18:23 ` Christer Weinigel 2004-09-02 21:47 ` Jamie Lokier 2004-09-02 22:27 ` Alan Cox 2004-09-03 15:17 ` Jon Smirl 2004-09-06 15:55 ` Christer Weinigel 2004-09-06 15:56 ` Chris Dukes 2004-09-10 2:13 ` silent semantic changes with reiser4 Timothy Miller 2004-09-10 5:22 ` Hans Reiser 2004-09-10 6:32 ` Peter Foldiak 2004-09-10 6:57 ` Hans Reiser 2004-09-10 15:47 ` Timothy Miller 2004-09-10 15:50 ` Wayne Scott 2004-09-10 17:50 ` Horst von Brand 2004-09-10 19:54 ` Uwe Ohse 2004-09-10 17:51 ` Hans Reiser 2004-09-10 18:15 ` Horst von Brand 2004-09-11 0:00 ` Timothy Miller 2004-09-10 15:33 ` Horst von Brand 2004-08-26 18:35 ` Joel Becker 2004-08-26 19:53 ` Jamie Lokier 2004-08-27 9:19 ` Markus Törnqvist 2004-08-27 12:26 ` Matthew Wilcox 2004-08-26 10:05 ` Chris Wedgwood 2004-08-26 11:02 ` Jamie Lokier 2004-08-26 11:19 ` Spam 2004-08-28 0:28 ` Horst von Brand 2004-08-26 15:50 ` Will Dyson 2004-08-26 20:38 ` Lee Revell 2004-08-26 20:50 ` Christophe Saout 2004-08-26 20:57 ` Lee Revell 2004-08-27 9:21 ` Markus Törnqvist 2004-08-27 11:34 ` Spam 2004-08-26 21:00 ` Jamie Lokier 2004-08-27 21:06 ` Pavel Machek 2004-08-31 1:00 ` David Masover 2004-08-31 8:21 ` Pavel Machek 2004-08-31 19:31 ` Horst von Brand 2004-08-31 20:05 ` Linus Torvalds 2004-08-31 20:29 ` Spam 2004-08-31 20:32 ` Pavel Machek 2004-08-31 20:38 ` Linus Torvalds 2004-08-31 20:54 ` Pavel Machek 2004-08-31 20:59 ` Linus Torvalds 2004-08-31 22:07 ` Pavel Machek 2004-08-31 23:19 ` Christer Weinigel 2004-09-01 1:43 ` David Masover 2004-09-01 5:02 ` Pavel Machek 2004-09-05 11:22 ` Tonnerre 2004-08-31 23:02 ` Christer Weinigel 2004-08-31 23:20 ` Spam 2004-09-05 11:17 ` Tonnerre 2004-09-05 11:40 ` Grzegorz Jaśkiewicz 2004-09-05 11:57 ` Spam 2004-09-05 11:58 ` Tonnerre 2004-09-05 12:23 ` Spam 2004-09-05 12:30 ` Spam 2004-09-06 10:50 ` Pavel Machek 2004-09-06 12:32 ` Spam 2004-09-06 14:52 ` Christer Weinigel 2004-09-06 15:13 ` Spam 2004-09-06 15:44 ` Christer Weinigel 2004-09-06 15:54 ` Pavel Machek 2004-09-06 16:07 ` Christer Weinigel 2004-09-06 16:56 ` Pavel Machek 2004-09-08 6:11 ` Tonnerre 2004-09-08 12:57 ` Alan Cox 2004-09-06 16:14 ` Spam 2004-09-06 19:14 ` David Masover 2004-09-06 18:14 ` Horst von Brand 2004-09-06 18:53 ` David Masover 2004-09-06 22:37 ` Christer Weinigel 2004-09-07 2:06 ` Horst von Brand 2004-09-07 5:50 ` David Masover 2004-09-07 6:02 ` Hans Reiser 2004-09-07 6:28 ` William Lee Irwin III 2004-09-07 22:38 ` William Lee Irwin III 2004-09-08 2:43 ` William Lee Irwin III 2004-09-08 3:03 ` William Lee Irwin III 2004-09-07 11:55 ` Christer Weinigel 2004-09-07 12:26 ` Gunnar Ritter 2004-09-07 18:20 ` Hans Reiser 2004-09-07 19:22 ` Gunnar Ritter 2004-09-07 12:30 ` Herbert Poetzl 2004-09-07 13:44 ` Oliver Neukum 2004-09-07 12:33 ` Spam 2004-09-07 13:40 ` Christer Weinigel 2004-09-07 13:52 ` Spam 2004-09-07 15:37 ` Horst von Brand 2004-09-07 18:11 ` Hans Reiser 2004-09-07 21:41 ` Christer Weinigel 2004-09-07 15:30 ` Horst von Brand 2004-09-07 18:05 ` Hans Reiser 2004-09-07 18:15 ` Horst von Brand 2004-09-07 19:14 ` Hans Reiser 2004-09-07 21:02 ` Horst von Brand 2004-09-07 21:29 ` Spam 2004-09-07 22:01 ` Horst von Brand 2004-09-08 1:38 ` Alexander G. M. Smith 2004-09-07 22:28 ` Christer Weinigel 2004-09-07 15:37 ` Gunnar Ritter 2004-09-07 16:25 ` Spam 2004-09-07 16:55 ` Gunnar Ritter 2004-09-07 21:39 ` Christer Weinigel 2004-09-07 22:13 ` Gunnar Ritter 2004-09-07 22:44 ` Christer Weinigel 2004-09-07 23:45 ` Gunnar Ritter 2004-09-08 5:48 ` Christer Weinigel 2004-09-08 6:29 ` Gunnar Ritter 2004-09-07 18:34 ` Hans Reiser 2004-09-07 22:13 ` Christer Weinigel 2004-09-07 22:36 ` David Lang 2004-09-07 22:47 ` Christer Weinigel 2004-09-07 22:56 ` David Lang 2004-09-08 0:15 ` Horst von Brand 2004-09-07 23:14 ` Gunnar Ritter 2004-09-08 1:29 ` Julian Blake Kongslie 2004-09-08 0:07 ` Horst von Brand 2004-09-08 6:51 ` David Lang 2004-09-08 6:20 ` Tonnerre 2004-09-07 6:32 ` Hans Reiser 2004-09-01 12:10 ` Alan Cox 2004-09-01 19:44 ` Markus Törnqvist 2004-09-07 6:43 ` Neil Brown 2004-09-07 6:47 ` Chris Wedgwood 2004-09-01 1:14 ` David Masover 2004-09-01 4:59 ` Pavel Machek 2004-09-01 16:14 ` Jamie Lokier 2004-09-01 20:18 ` Pavel Machek 2004-09-01 21:59 ` Jamie Lokier 2004-09-01 22:51 ` Lee Revell 2004-09-02 14:25 ` Horst von Brand 2004-09-02 17:47 ` Lee Revell 2004-09-02 19:41 ` Spam 2004-09-02 18:55 ` Alan Cox 2004-09-02 20:07 ` Spam 2004-09-02 19:17 ` Alan Cox 2004-09-02 20:28 ` Spam 2004-09-02 20:42 ` Pavel Machek 2004-09-02 19:49 ` Pavel Machek 2004-09-02 20:01 ` Lee Revell 2004-09-02 19:16 ` Alan Cox 2004-09-05 12:07 ` Tonnerre 2004-09-05 12:27 ` Spam 2004-09-06 10:54 ` Pavel Machek 2004-09-06 18:08 ` Horst von Brand 2004-09-08 6:00 ` Tonnerre 2004-09-08 13:26 ` Horst von Brand 2004-09-02 20:04 ` Chris Wedgwood 2004-09-02 20:10 ` Spam 2004-09-02 20:43 ` Pavel Machek 2004-09-02 20:47 ` Lee Revell 2004-09-02 20:49 ` Chris Wedgwood 2004-09-02 20:57 ` Lee Revell 2004-09-02 20:58 ` Chris Wedgwood 2004-09-02 20:58 ` Pavel Machek 2004-09-02 22:33 ` Alan Cox 2004-09-03 3:22 ` Gianni Tedesco 2004-09-03 9:24 ` Userspace framework (was: Re: silent semantic changes with reiser4) Luca Ferroni 2004-09-03 20:31 ` Gianni Tedesco 2004-09-03 9:42 ` silent semantic changes with reiser4 Pavel Machek 2004-09-02 22:00 ` Horst von Brand 2004-09-02 23:23 ` Jamie Lokier 2004-09-02 22:51 ` Alan Cox 2004-09-03 0:02 ` Spam 2004-09-03 13:46 ` John Stoffel 2004-09-05 11:54 ` Tonnerre 2004-09-03 1:25 ` Gianni Tedesco 2004-09-01 5:50 ` Hans Reiser 2004-09-01 6:06 ` Linus Torvalds 2004-09-01 7:22 ` Hans Reiser 2004-09-01 11:51 ` Alexander G. M. Smith 2004-09-15 17:06 ` Timothy Miller 2004-09-16 8:23 ` Helge Hafting 2004-09-01 11:49 ` Alan Cox 2004-09-02 23:40 ` Paul Jakma 2004-09-03 6:06 ` Hans Reiser 2004-09-03 7:00 ` viro 2004-09-08 7:51 ` Sriram Karra 2004-09-08 9:36 ` viro 2004-09-05 11:33 ` Tonnerre 2004-09-01 1:10 ` David Masover 2004-08-27 21:01 ` Pavel Machek 2004-08-26 1:13 ` Chris Wright 2004-08-26 8:43 ` Hans Reiser 2004-08-25 21:52 ` Matt Mackall 2004-08-25 22:21 ` Linus Torvalds 2004-08-26 0:18 ` Mikulas Patocka 2004-08-26 0:27 ` Linus Torvalds 2004-08-26 0:51 ` Mikulas Patocka 2004-08-26 8:36 ` Hans Reiser 2004-08-26 10:53 ` Jamie Lokier 2004-08-26 10:59 ` Christoph Hellwig 2004-08-26 11:07 ` Jamie Lokier 2004-08-26 11:17 ` Spam 2004-08-26 0:57 ` Rik van Riel 2004-08-26 7:51 ` Helge Hafting 2004-08-26 9:21 ` Paul Jackson 2004-08-26 10:47 ` Jamie Lokier 2004-08-26 11:19 ` Paul Jackson 2004-08-27 12:33 ` Helge Hafting 2004-08-26 8:40 ` Hans Reiser 2004-08-26 14:46 ` Lars Marowsky-Bree 2004-08-26 9:44 ` Paul Jackson 2004-08-26 8:43 ` Hans Reiser 2004-08-26 13:53 ` Michael Halcrow 2004-08-26 22:26 ` Hans Reiser 2004-08-29 14:20 ` Alan Cox 2004-08-25 20:35 ` Alex Zarochentsev 2004-08-25 20:51 ` Christoph Hellwig 2004-08-25 23:54 ` Jamie Lokier 2004-08-26 8:44 ` Hans Reiser 2004-08-26 12:43 ` Christoph Hellwig 2004-08-26 8:43 ` Hans Reiser 2004-08-26 12:41 ` Christoph Hellwig 2004-08-26 13:48 ` Jamie Lokier 2004-08-26 15:57 ` Christoph Hellwig 2004-08-26 16:06 ` Jamie Lokier 2004-08-26 16:13 ` Christoph Hellwig 2004-08-26 16:35 ` Linus Torvalds 2004-08-26 17:20 ` Jamie Lokier 2004-08-26 17:29 ` Linus Torvalds 2004-08-26 17:54 ` viro 2004-08-26 18:16 ` Jamie Lokier 2004-08-26 18:27 ` Linus Torvalds 2004-08-26 19:40 ` Jamie Lokier 2004-08-30 13:25 ` Alex Zarochentsev 2004-08-27 9:18 ` Giuliano Pochini 2004-08-30 14:10 ` Alex Zarochentsev 2004-08-29 18:43 ` Alex Zarochentsev 2004-08-25 20:20 ` Jeremy Allison 2004-08-26 8:42 ` Hans Reiser 2004-08-26 13:27 ` Rik van Riel 2004-08-26 13:56 ` Markus Törnqvist 2004-08-26 19:58 ` Paul Jackson 2004-08-26 23:53 ` Hans Reiser 2004-08-25 20:22 ` Chris Mason 2004-08-26 8:42 ` Hans Reiser 2004-08-26 9:36 ` Christoph Hellwig 2004-08-25 20:23 ` Chris Friesen 2004-08-25 22:28 ` Andrew Morton 2004-08-25 22:51 ` Spam 2004-08-25 22:51 ` Christoph Hellwig 2004-08-25 22:59 ` Linus Torvalds 2004-08-25 23:19 ` Spam 2004-08-25 23:32 ` Andrew Morton 2004-08-25 23:37 ` Jeremy Allison 2004-08-25 23:46 ` Wichert Akkerman 2004-08-26 0:42 ` Nicholas Miell 2004-08-26 1:03 ` Jamie Lokier 2004-08-26 1:26 ` Nicholas Miell 2004-08-26 1:53 ` Jamie Lokier 2004-08-26 2:02 ` Nicholas Miell 2004-08-26 8:41 ` Hans Reiser 2004-08-26 19:06 ` Giuseppe Bilotta 2004-08-26 1:22 ` Jeremy Allison 2004-08-26 11:01 ` Wichert Akkerman [not found] ` <c7ef32a70408260942c8efa54@mail.gmail.com> 2004-08-26 16:44 ` Tierra 2004-08-26 4:44 ` Matt Mackall 2004-08-26 5:09 ` Nicholas Miell 2004-08-26 5:17 ` James Morris 2004-08-26 5:32 ` Matt Mackall 2004-08-26 7:34 ` Denis Vlasenko 2004-08-26 11:42 ` Rik van Riel 2004-08-26 12:00 ` Christophe Saout 2004-08-28 11:00 ` Yury Umanets 2004-08-28 11:12 ` Yury Umanets 2004-08-26 12:12 ` Jamie Lokier 2004-08-26 12:25 ` Wichert Akkerman 2004-09-03 4:36 ` Grzegorz Jaśkiewicz 2004-09-03 8:01 ` Frank van Maarseveen 2004-09-03 17:12 ` Horst von Brand 2004-09-03 4:40 ` Grzegorz Jaśkiewicz 2004-08-26 20:35 ` Lee Revell 2004-08-27 0:02 ` Hans Reiser 2004-08-26 7:53 ` Markus Törnqvist 2004-08-26 16:32 ` Jan Harkes 2004-08-26 16:48 ` Linus Torvalds 2004-08-26 18:07 ` Jan Harkes 2004-08-31 3:39 ` Tom Vier 2004-08-31 4:08 ` Linus Torvalds 2004-08-31 4:38 ` Jeff Garzik 2004-08-31 5:30 ` Userspace file systems & MKs (Re: silent semantic changes with reiser4) Bill Huey 2004-08-31 5:56 ` Jeff Garzik 2004-08-31 6:50 ` Bill Huey [not found] ` <20040831155613.2b25df1e.diegocg@teleline.es> 2004-08-31 20:52 ` Bill Huey 2004-09-02 17:20 ` Diego Calleja 2004-09-02 21:37 ` Bill Huey 2004-08-31 21:23 ` Bill Huey 2004-08-31 9:01 ` Tonnerre 2004-08-31 11:15 ` Userspace file systems Luca Ferroni 2004-08-31 20:53 ` Userspace file systems & MKs (Re: silent semantic changes with reiser4) Bill Huey 2004-08-31 19:01 ` silent semantic changes with reiser4 Ryan Breen 2004-08-31 19:12 ` Jeff Garzik 2004-09-01 15:37 ` Clay Haapala 2004-09-03 12:50 ` Yury Umanets 2004-08-31 10:57 ` Alan Cox 2004-08-31 17:15 ` Linus Torvalds 2004-08-31 21:13 ` Frank van Maarseveen 2004-08-31 21:25 ` Linus Torvalds 2004-08-31 19:45 ` Toon van der Pas 2004-09-02 22:35 ` Tom Vier 2004-09-02 22:53 ` Linus Torvalds 2004-09-10 19:22 ` Timothy Miller 2004-09-10 22:18 ` Elladan 2004-09-12 20:11 ` Pavel Machek 2004-09-12 20:02 ` Pavel Machek 2004-09-03 4:43 ` Grzegorz Jaśkiewicz 2004-09-03 5:05 ` viro 2004-09-03 5:40 ` Grzegorz Jaśkiewicz 2004-09-05 11:10 ` Tonnerre 2004-08-26 8:31 ` Helge Hafting 2004-08-26 18:17 ` Horst von Brand 2004-08-28 16:49 ` Helge Hafting 2004-09-02 23:05 ` Tom Vier 2004-09-03 8:07 ` Helge Hafting 2004-09-03 8:19 ` Jan Dittmer 2004-09-03 9:54 ` Erik Hensema 2004-09-05 11:13 ` Tonnerre 2004-09-08 9:13 ` Helge Hafting 2004-08-26 9:59 ` Spam 2004-08-26 13:05 ` Joseph Fannin 2004-08-26 6:53 ` Paul Jackson 2004-08-26 9:54 ` Spam 2004-08-26 5:23 ` Jamie Lokier 2004-08-26 8:16 ` Helge Hafting 2004-08-26 9:51 ` Spam 2004-08-26 9:40 ` Spam 2004-08-26 9:49 ` Andrew Morton 2004-08-26 10:20 ` Spam 2004-08-26 10:24 ` Andrew Morton 2004-08-26 10:51 ` Spam 2004-08-26 10:55 ` Andrew Morton 2004-08-26 11:15 ` Spam 2004-08-26 11:20 ` Andrew Morton 2004-08-26 11:29 ` Wichert Akkerman 2004-09-02 23:18 ` Tom Vier 2004-09-09 16:46 ` Timothy Miller 2004-08-28 0:24 ` Horst von Brand 2004-08-26 11:27 ` Wichert Akkerman 2004-08-26 13:39 ` Rik van Riel 2004-08-26 13:59 ` Spam 2004-08-26 14:39 ` Felipe Alfaro Solana 2004-08-27 23:58 ` Horst von Brand 2004-08-29 15:17 ` Markus Törnqvist 2004-08-29 16:18 ` Spam 2004-08-29 19:10 ` Tonnerre 2004-08-30 0:39 ` Alexander G. M. Smith 2004-08-30 18:39 ` Spam 2004-08-31 8:15 ` Tonnerre 2004-09-01 20:16 ` Jamie Lokier [not found] ` <1094070506.28509.24.camel@shaggy.austin.ibm.com> 2004-09-02 5:57 ` Jamie Lokier 2004-09-02 9:50 ` Alan Cox 2004-09-02 20:25 ` Jamie Lokier 2004-09-02 18:16 ` Christer Weinigel 2004-09-02 19:58 ` Jamie Lokier 2004-08-26 11:48 ` Jamie Lokier 2004-08-26 16:07 ` James Bruce 2004-08-26 16:58 ` Jamie Lokier 2004-08-26 21:34 ` Felipe Alfaro Solana 2004-08-26 23:52 ` Hans Reiser 2004-08-26 11:11 ` Bernd Petrovitsch 2004-08-26 23:59 ` Hans Reiser 2004-08-26 10:54 ` Wichert Akkerman 2004-08-26 23:48 ` Hans Reiser 2004-08-26 11:16 ` Christer Weinigel 2004-08-26 11:35 ` Spam 2004-08-26 11:33 ` Wichert Akkerman 2004-08-26 11:43 ` Spam [not found] ` <1093592467.18603.6.camel@tara.firmix.at> 2004-08-27 8:05 ` Hans Reiser 2004-08-27 8:51 ` Christoph Hellwig 2004-08-27 8:49 ` Christoph Hellwig 2004-08-27 9:03 ` Bernd Petrovitsch 2004-08-26 23:56 ` Hans Reiser 2004-08-27 1:05 ` Andreas Dilger 2004-08-27 0:00 ` Stelian Ionescu 2004-08-26 11:58 ` Jamie Lokier 2004-08-26 14:00 ` Denis Vlasenko 2004-08-26 14:12 ` Rik van Riel 2004-08-26 14:24 ` Christophe Saout 2004-08-26 15:44 ` Jamie Lokier 2004-08-26 15:54 ` Rik van Riel 2004-08-26 16:04 ` Christophe Saout 2004-08-26 16:07 ` Christoph Hellwig 2004-08-26 16:44 ` Jamie Lokier 2004-08-26 18:40 ` David Masover 2004-08-26 21:17 ` David Lang 2004-08-26 21:05 ` David Lang 2004-08-26 21:19 ` Jamie Lokier 2004-08-26 21:31 ` Linus Torvalds 2004-08-26 22:30 ` Felipe Alfaro Solana 2004-08-26 22:32 ` Chris Dukes 2004-08-26 22:52 ` David Lang 2004-08-26 16:53 ` Jamie Lokier 2004-08-26 19:32 ` John Stoffel 2004-08-26 19:51 ` Denis Vlasenko 2004-08-26 20:13 ` Jamie Lokier 2004-08-30 17:37 ` Alex Zarochentsev 2004-08-30 18:51 ` Hans Reiser 2004-08-26 17:05 ` Diego Calleja 2004-08-26 17:16 ` Rik van Riel 2004-08-26 17:20 ` Linus Torvalds 2004-08-26 17:27 ` Jamie Lokier 2004-08-26 17:32 ` Linus Torvalds 2004-08-26 19:09 ` Jamie Lokier 2004-08-26 17:57 ` Rik van Riel 2004-08-26 18:15 ` Linus Torvalds 2004-08-26 18:41 ` Rik van Riel 2004-08-26 18:55 ` Linus Torvalds 2004-08-26 18:59 ` Rik van Riel 2004-08-26 19:14 ` Christophe Saout 2004-08-26 19:23 ` Linus Torvalds 2004-08-26 19:44 ` Rik van Riel 2004-08-26 19:47 ` Christophe Saout 2004-08-26 20:10 ` Rik van Riel 2004-08-26 20:17 ` Christophe Saout 2004-08-26 20:21 ` Jamie Lokier 2004-08-26 20:35 ` Stephen Wille Padnos 2004-08-31 11:01 ` Claus Färber 2004-09-01 15:13 ` Alan Cox 2004-09-03 13:23 ` Claus Färber 2004-08-26 20:40 ` Linus Torvalds 2004-08-26 22:53 ` Hans Reiser 2004-08-26 23:27 ` Linus Torvalds 2004-08-26 23:47 ` Hans Reiser [not found] ` <20040827020350.GF21964@parcelfarce.linux.theplanet.co.uk> 2004-08-27 8:10 ` Hans Reiser 2004-08-26 20:48 ` Jamie Lokier 2004-08-26 20:52 ` Jeremy Allison 2004-08-27 9:19 ` Christoph Hellwig 2004-08-28 3:56 ` Jeremy Allison 2004-08-28 19:18 ` Brad Boyer 2004-08-26 22:20 ` Felipe Alfaro Solana 2004-08-29 0:04 ` Horst von Brand 2004-09-01 22:45 ` Jamie Lokier 2004-09-02 5:23 ` Olaf Hering 2004-09-03 8:22 ` Greg KH 2004-09-03 8:32 ` Grzegorz Jaśkiewicz 2004-09-03 8:54 ` Helge Hafting 2004-09-03 9:14 ` Grzegorz Jaśkiewicz 2004-09-03 9:58 ` Miquel van Smoorenburg 2004-09-03 15:18 ` Grzegorz Jaśkiewicz 2004-09-03 15:29 ` Alan Cox 2004-09-03 16:41 ` Grzegorz Jaśkiewicz 2004-09-03 23:46 ` Paul Jackson 2004-09-03 10:43 ` Greg KH 2004-09-08 9:09 ` Helge Hafting 2004-09-08 15:25 ` Tonnerre 2004-08-27 8:06 ` Hans Reiser 2004-08-26 19:29 ` Jamie Lokier 2004-08-26 19:42 ` Christophe Saout 2004-08-27 8:46 ` Herbert Poetzl 2004-08-26 20:44 ` Martin J. Bligh 2004-08-26 20:54 ` Linus Torvalds 2004-08-26 21:00 ` Martin J. Bligh 2004-08-26 21:09 ` Christophe Saout 2004-08-26 21:13 ` Linus Torvalds 2004-08-26 21:24 ` Jamie Lokier 2004-08-27 9:36 ` David Greaves 2004-08-26 22:44 ` Miquel van Smoorenburg 2004-08-26 21:29 ` Chris Wright 2004-08-27 8:19 ` Anton Altaparmakov [not found] ` <412E33D6.5020500@slaphack.com> 2004-08-27 8:06 ` Hans Reiser 2004-08-27 14:12 ` Rik van Riel 2004-08-27 18:27 ` Hans Reiser 2004-08-27 18:44 ` Lee Revell 2004-08-27 19:13 ` Rik van Riel 2004-08-28 6:10 ` Hans Reiser 2004-08-27 20:56 ` Christian Mayrhuber 2004-08-27 14:45 ` Rik van Riel 2004-08-27 15:15 ` Spam [not found] ` <87llg0mnl0.fsf@uhoreg.ca> 2004-08-28 0:02 ` Spam [not found] ` <20040827154156.GA31757@mail.shareable.org> 2004-08-28 0:19 ` Spam 2004-08-28 1:59 ` Rik van Riel 2004-08-28 2:05 ` Linus Torvalds 2004-08-28 9:53 ` Hans Reiser 2004-08-28 11:21 ` Spam 2004-08-28 17:05 ` Helge Hafting 2004-08-28 18:09 ` Linus Torvalds 2004-08-28 18:20 ` Rik van Riel 2004-08-28 18:29 ` viro 2004-08-28 18:44 ` Linus Torvalds 2004-08-28 18:56 ` viro 2004-08-28 19:16 ` Linus Torvalds 2004-08-28 19:41 ` Andries Brouwer 2004-08-28 19:46 ` Linus Torvalds 2004-08-29 13:56 ` Andries Brouwer 2004-08-28 19:45 ` viro 2004-08-29 5:12 ` Linus Torvalds 2004-08-29 5:44 ` Jeremy Allison 2004-08-29 0:46 ` Paul Jakma 2004-08-28 22:29 ` Hans Reiser 2004-08-29 0:40 ` The Necessity of File Types (was silent semantic changes with reiser4) Alexander G. M. Smith 2004-08-31 0:42 ` David Masover 2004-08-29 5:01 ` silent semantic changes with reiser4 Linus Torvalds 2004-08-29 9:36 ` Hans Reiser 2004-08-29 10:44 ` CaT [not found] ` <87fz65wt82.fsf@uhoreg.ca> 2004-08-30 6:31 ` Markus Törnqvist 2004-08-29 17:57 ` Linus Torvalds 2004-08-29 19:53 ` Hans Reiser 2004-08-29 20:14 ` Nicholas Miell 2004-08-29 21:01 ` Linus Torvalds 2004-08-29 15:12 ` Alexander Lyamin 2004-08-29 16:20 ` viro 2004-08-30 15:00 ` John Stoffel 2004-08-28 23:47 ` Jeremy Allison 2004-08-30 13:56 ` Helge Hafting 2004-08-27 18:28 ` Hans Reiser 2004-08-28 6:08 ` Paul Jackson 2004-08-29 15:02 ` Alexander Lyamin 2004-08-29 18:28 ` Hans Reiser 2004-08-29 18:36 ` viro 2004-08-29 18:57 ` viro 2004-08-29 20:06 ` Hans Reiser 2004-08-29 21:27 ` viro 2004-08-29 21:50 ` Linus Torvalds [not found] ` <Pine.LNX.4.60.0408300009001.10533@alpha.polcom.net> 2004-08-29 22:37 ` Linus Torvalds 2004-08-29 23:28 ` viro 2004-08-30 1:43 ` Neil Brown 2004-09-02 10:51 ` Stephan von Krawczynski 2004-09-02 11:26 ` Frank van Maarseveen 2004-09-02 12:58 ` Stephan von Krawczynski 2004-08-29 23:17 ` Trond Myklebust 2004-08-29 23:25 ` viro 2004-08-29 23:54 ` Linus Torvalds 2004-08-30 1:42 ` Trond Myklebust 2004-08-30 2:31 ` Linus Torvalds 2004-08-30 3:01 ` viro 2004-08-30 3:55 ` Linus Torvalds 2004-08-30 4:46 ` viro 2004-08-30 6:33 ` viro 2004-08-30 3:12 ` Trond Myklebust 2004-08-30 5:42 ` Brad Boyer 2004-09-07 20:08 ` Tom Vier 2004-08-30 0:07 ` Alexander G. M. Smith 2004-08-30 2:18 ` Hans Reiser 2004-08-30 7:42 ` Hans Reiser 2004-08-30 13:17 ` Alexander G. M. Smith 2004-08-30 17:43 ` Paul Stewart 2004-08-30 17:58 ` Linus Torvalds 2004-08-29 15:00 ` Alexander Lyamin 2004-08-31 0:46 ` David Masover 2004-08-31 6:47 ` Hans Reiser 2004-09-01 2:04 ` David Masover 2004-08-26 18:28 ` Denis Vlasenko 2004-08-26 18:46 ` Linus Torvalds 2004-08-26 19:13 ` viro 2004-08-26 20:32 ` viro 2004-08-26 20:36 ` Christophe Saout 2004-08-26 20:47 ` Linus Torvalds 2004-08-26 21:28 ` [some sanity for a change] possible design issues for hybrids viro 2004-08-26 22:04 ` Linus Torvalds 2004-08-26 22:36 ` viro 2004-08-26 22:45 ` Linus Torvalds 2004-08-26 22:53 ` viro 2004-08-26 23:24 ` Linus Torvalds 2004-08-26 23:40 ` viro 2004-08-26 23:57 ` Linus Torvalds 2004-08-27 1:01 ` viro 2004-08-27 8:56 ` Anton Altaparmakov 2004-08-27 9:00 ` Christoph Hellwig 2004-08-26 19:36 ` silent semantic changes with reiser4 Jonathan Abbey 2004-08-26 20:16 ` Jamie Lokier 2004-08-26 20:25 ` Christophe Saout 2004-08-26 21:45 ` Nikita Danilov 2004-08-26 21:48 ` Christophe Saout 2004-08-26 22:17 ` Emil Larsson 2004-08-26 22:21 ` Nikita Danilov 2004-08-26 22:29 ` Christophe Saout 2004-08-26 23:35 ` Hans Reiser 2004-08-27 2:00 ` Stephen Wille Padnos 2004-08-27 8:08 ` Hans Reiser 2004-08-26 17:40 ` Diego Calleja 2004-08-26 17:55 ` Christophe Saout 2004-08-26 17:59 ` Rik van Riel 2004-08-26 18:20 ` Jamie Lokier 2004-08-27 8:06 ` Hans Reiser 2004-08-26 15:19 ` Denis Vlasenko 2004-08-29 14:30 ` Alan Cox 2004-08-29 17:42 ` Jeremy Allison 2004-08-29 18:41 ` Trond Myklebust 2004-08-29 19:38 ` Jeremy Allison 2004-09-01 20:19 ` Jamie Lokier 2004-09-01 20:26 ` Jeremy Allison 2004-09-01 20:31 ` Jamie Lokier 2004-09-01 20:35 ` Jeremy Allison 2004-09-01 20:47 ` Jamie Lokier 2004-09-01 20:51 ` Jeremy Allison 2004-09-01 21:35 ` Lee Revell 2004-09-01 23:22 ` Oliver Hunt 2004-09-07 11:48 ` Herbert Poetzl 2004-09-02 12:54 ` Theodore Ts'o 2004-09-02 14:45 ` Stuart Young 2004-09-03 7:53 ` Helge Hafting 2004-09-03 10:41 ` Stuart Young 2004-09-03 19:17 ` Horst von Brand 2004-09-04 5:03 ` Stuart Young 2004-09-01 21:09 ` David Lang 2004-09-02 9:48 ` Alan Cox 2004-09-02 11:13 ` Oliver Neukum 2004-09-02 11:01 ` Alan Cox 2004-09-04 1:10 ` Brad Boyer 2004-09-04 6:03 ` David Masover 2004-09-01 20:25 ` Jamie Lokier 2004-08-26 14:42 ` Felipe Alfaro Solana 2004-08-26 13:36 ` Rik van Riel 2004-08-26 13:59 ` Jamie Lokier 2004-08-26 14:07 ` Rik van Riel 2004-08-26 23:54 ` Hans Reiser 2004-08-26 17:32 ` Jeremy Allison 2004-08-26 18:17 ` Rik van Riel 2004-08-26 22:03 ` Felipe Alfaro Solana [not found] ` <20040827052804.GD2736@localhost.localdomain> 2004-08-27 8:05 ` Hans Reiser 2004-08-27 8:13 ` Hans Reiser 2004-08-26 9:03 ` Anton Altaparmakov 2004-08-26 10:45 ` Christian Mayrhuber 2004-08-26 10:52 ` Christoph Hellwig 2004-08-26 12:43 ` Olivier Galibert 2004-08-26 12:44 ` Christoph Hellwig 2004-08-26 12:49 ` Markus Törnqvist 2004-08-26 12:52 ` Christoph Hellwig 2004-08-26 15:15 ` Adrian Bunk 2004-08-26 8:44 ` Hans Reiser 2004-08-26 8:31 ` Hans Reiser 2004-08-26 8:45 ` Andrew Morton 2004-08-26 9:24 ` Hans Reiser 2004-08-26 19:26 ` Lee Revell 2004-08-26 19:34 ` Christoph Hellwig 2004-08-26 19:45 ` Lee Revell 2004-08-26 20:30 ` Dmitry Baryshkov 2004-08-26 20:38 ` Christophe Saout 2004-08-26 20:49 ` Dmitry Baryshkov 2004-08-26 21:33 ` Nikita Danilov 2004-08-26 21:46 ` Christophe Saout 2004-08-26 22:03 ` Nikita Danilov 2004-08-26 22:05 ` Christoph Hellwig 2004-08-27 8:12 ` Hans Reiser 2004-08-27 8:57 ` Christoph Hellwig [not found] ` <1093548815.13881.10.camel@leto.cs.pocnet.net> 2004-08-27 8:05 ` Hans Reiser 2004-08-27 9:24 ` Christoph Hellwig 2004-08-26 12:18 ` Christophe Saout 2004-08-26 12:49 ` Christoph Hellwig 2004-08-26 13:00 ` Christophe Saout 2004-08-26 13:07 ` Christoph Hellwig 2004-08-26 13:17 ` reiser4 plugins (was: silent semantic changes with reiser4) Christophe Saout 2004-08-26 13:24 ` Christoph Hellwig 2004-08-26 13:35 ` Christophe Saout 2004-08-26 13:40 ` Christoph Hellwig 2004-08-26 13:58 ` Christophe Saout 2004-08-26 15:37 ` Christoph Hellwig 2004-08-26 15:48 ` Christophe Saout 2004-08-26 16:06 ` Christoph Hellwig 2004-08-26 16:14 ` Christophe Saout 2004-08-26 16:25 ` Christoph Hellwig 2004-08-26 16:39 ` Christophe Saout 2004-08-26 23:55 ` reiser4 plugins Hans Reiser 2004-08-27 12:04 ` Nikita Danilov 2004-08-27 18:15 ` Hans Reiser 2004-08-27 18:55 ` Nikita Danilov 2004-08-28 9:53 ` Hans Reiser 2004-08-28 13:47 ` Nikita Danilov 2004-08-28 23:45 ` Hans Reiser 2004-08-29 9:35 ` Nikita Danilov 2004-08-29 11:17 ` Alex Zarochentsev 2004-08-27 22:29 ` Steve Bergman 2004-08-28 6:54 ` Hans Reiser [not found] ` <1093687768.8097.25.camel@voyager.localdomain> 2004-08-29 11:55 ` Alex Zarochentsev 2004-08-29 11:42 ` Alex Zarochentsev 2004-08-26 23:54 ` Hans Reiser 2004-08-26 13:52 ` reiser4 plugins (was: silent semantic changes with reiser4) Rik van Riel 2004-08-26 13:58 ` Chris Mason 2004-08-26 14:02 ` Christophe Saout 2004-08-28 11:18 ` Alexander Lyamin 2004-08-28 11:22 ` Christoph Hellwig 2004-08-28 11:46 ` Alexander Lyamin 2004-08-28 11:51 ` Christoph Hellwig 2004-08-28 12:18 ` Alexander Lyamin 2004-08-28 11:58 ` Markus Törnqvist 2004-08-28 16:11 ` Jan Harkes 2004-08-28 16:26 ` Nikita Danilov 2004-08-28 19:41 ` Alexander Lyamin 2004-08-28 10:59 ` Alexander Lyamin 2004-08-28 11:12 ` Christoph Hellwig 2004-08-28 12:05 ` Alexander Lyamin 2004-08-28 13:56 ` Christoph Hellwig 2004-08-28 19:23 ` Alexander Lyamin 2004-08-28 19:27 ` Alexander Lyamin 2004-08-28 22:36 ` reiser4 plugins Hans Reiser 2004-08-28 11:14 ` reiser4 plugins (was: silent semantic changes with reiser4) Markus Törnqvist 2004-08-28 17:18 ` Linus Torvalds 2004-08-28 19:03 ` Alexander Lyamin 2004-08-28 19:09 ` Christoph Hellwig 2004-08-28 21:41 ` reiser4 plugins Hans Reiser 2004-08-30 16:02 ` Herbert Poetzl 2004-08-30 18:55 ` Hans Reiser 2004-08-28 19:26 ` reiser4 plugins (was: silent semantic changes with reiser4) Linus Torvalds 2004-08-26 13:27 ` silent semantic changes with reiser4 Christophe Saout 2004-08-26 14:44 ` Jamie Lokier 2004-08-26 16:03 ` Christoph Hellwig 2004-08-26 16:19 ` Jamie Lokier 2004-08-26 16:23 ` Christoph Hellwig 2004-08-26 17:08 ` Jamie Lokier 2004-08-26 23:54 ` Hans Reiser 2004-08-26 9:34 ` Christoph Hellwig 2004-08-26 9:37 ` Markus Törnqvist 2004-08-26 18:40 ` Thomas Cataldo 2004-08-26 16:32 ` Will Dyson 2004-08-26 16:44 ` Christophe Saout 2004-08-26 17:22 ` Jamie Lokier 2004-08-27 8:08 ` Hans Reiser 2004-08-27 18:20 ` Will Dyson 2004-08-27 18:38 ` David Ford 2004-08-28 6:14 ` Hans Reiser 2004-08-28 9:54 ` Hans Reiser 2004-08-31 3:14 ` Will Dyson 2004-08-28 15:18 ` Separating Indexing and Searching (was silent semantic changes with reiser4) Alexander G. M. Smith 2004-08-28 22:11 ` Hans Reiser 2004-08-28 22:12 ` Hans Reiser 2004-09-01 20:41 ` Jamie Lokier 2004-08-31 1:57 ` Will Dyson 2004-08-26 18:12 ` silent semantic changes with reiser4 Horst von Brand 2004-08-27 20:32 ` Markus Törnqvist 2004-08-27 20:38 ` Linus Torvalds 2004-08-27 20:48 ` Re[2]: " Spam 2004-08-27 20:53 ` Markus Törnqvist 2004-08-27 21:33 ` Bill Huey 2004-08-28 9:55 ` Hans Reiser 2004-09-01 22:45 ` Bill Huey 2004-08-27 20:44 ` Spam [not found] ` <200408291521.i7TFLsQk028363@localhost.localdomain> 2004-08-29 16:52 ` Spam 2004-08-29 17:22 ` Nikita Danilov 2004-08-29 18:12 ` Markus Törnqvist 2004-08-29 16:53 ` Spam 2004-08-29 16:54 ` Spam 2004-09-01 21:43 ` Jamie Lokier 2004-08-26 20:04 ` Martin J. Bligh 2004-08-27 21:23 ` vfs2 (was Re: silent semantic changes with reiser4) Pavel Machek 2004-09-14 4:27 ` file-as-dir vs. dir? " Andy Lutomirski 2005-06-29 5:09 reiser4 plugins Horst von Brand 2005-06-29 20:40 ` Hubert Chan 2005-06-29 21:34 ` Ross Biro 2005-06-29 23:29 ` Hubert Chan 2005-07-01 8:06 ` David Masover 2005-07-05 21:01 ` Hubert Chan 2005-07-05 22:01 ` Hans Reiser 2005-07-05 22:21 ` David Masover 2005-07-05 22:51 ` Hans Reiser 2005-07-05 23:00 ` David Masover 2005-07-05 23:47 ` Hans Reiser 2005-07-06 0:15 ` Hans Reiser 2005-07-06 14:01 ` David Masover 2005-07-11 4:07 ` Stefan Smietanowski 2005-07-11 20:50 ` David Masover 2005-07-12 7:19 ` Neil Brown 2005-07-12 7:45 ` Hans Reiser 2005-07-13 0:05 ` Neil Brown 2005-07-13 1:20 ` Hans Reiser 2005-07-12 23:13 ` David Masover 2005-07-12 23:40 ` Neil Brown 2005-07-12 2:33 Horst von Brand 2005-07-12 5:53 ` Hans Reiser 2005-07-12 23:22 ` David Masover 2005-07-12 23:38 ` Hans Reiser 2005-07-13 3:43 ` David Masover 2005-07-13 2:09 ` Horst von Brand
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).